Learning agents that interact with complex environments often cannot predict the exact outcome of their actions due to noisy sensors or incomplete knowledge of the world. Learning the internal representation of such partially observable environments has proven to be a difficult problem. In order to simplify this task, the agent can choose to give up building an exact model which is able to predict all possible future behaviours, and replace it with a more modest goal of predicting only specific quantities of interest. This work is primarily concerned with ways of representing the agent's state that allows it to predict the conditional probability of a restricted set of future events, given the agent's past experience. Because of memory limitations, the agent's experience must be summarized in such a way as to make these restricted predictions possible. We introduce the idea of history representations, which allow us to condition the predictions on "interesting" behaviour, and present a simple algorithmic implementation of this framework. The learned model abstracts away the unnecessary details of the agent's experience and focuses only on making certain predictions of interest. We illustrate our approach empirically in small computational examples, demonstrating the data efficiency of the algorithm. [PDF]
Abstract: We study a double dual construction which gives a minimal and deterministic representation of a POMDP, while retaining the same behaviour. We discuss two types of equivalence relations on states, namely linear and branching ones, and construct a hierarchy of them. Additionally, we show that bisimulation is the strongest equivalence relations on states, and that just traces of a system are not enough to capture its behaviour. [PDF]
Abstract: Recent work by Still and Bialek proposes an information theoretic approach that compresses the available history into an internal representation and maximizes its predictive power. In order to validate the asymptotic nature of the theoretical algorithm, we present the first empirical results in the field, demonstrating the accuracy of the internal representation as well as its predictive powers. In addition, we propose two alternative approaches that may ensure faster convergence times and more accurate optimal action strategies. [PDF]
Monica Dinculescu, Approximate Predictive Representations of Partially Observable Systems, M.Sc. Thesis, McGill University, 2010. [PDF]
Monica Dinculescu, Doina Precup, Approximate Predictive Representations of Partially Observable Systems, ICML 2010. [PDF]
Susanna Still, Monica Dinculescu, Doina Precup, An Information Theoretic Approach for Building Approximate Predictive Models, NIPS Workshop on Grounding Perception, Knowledge and Cognition in Sensori-Motor Experience, 2006.
Learning Approximate Representations of Partially Observable Systems, Multidisciplinary Symposium on Reinforcement Learning (MSRL), Montreal, June 2009. [PDF]
The Duality of State and Observations, 4th International Conference on the Quantitative Evaluation of SysTems (QEST), Edinburgh, September 2007. [PDF]
Can be found here.