1. Presented by Alp Sardağ Algorithms for POMDP

2. Monahan Enumeration Phase Generate all vectors: Number of gen. Vectors = |A|M |  | where M vectors of previous state

3. Monahan Reduction Phase All vectors can be kept: Each time maximize over all vectors. Lot of excess baggage The number of vectors in next step will be even large. LP used to trim away useless vectors

4. Monahan Reduction Phase For a vector to be useful, there must be at least one belief point it gives larger value than others:

5. Monahan Algorithm

6. Monahan’s LP Complication Formulate LP and check for  :

7. Eagle’s Variant of Monahan The optimization occurs in enumaration phase. If, in the enumaration process, a vector’s components are completely dominated by another vector’s component, discard it. Generate  j i (t) and following condition holds: Discard  j i (t). Can be applied to check new vector dominates any vector previously enumarated.

8. Sondik’s One-Pass Algorithm Find theproper set of belief states to plug into the below formula to get all necessary vectors: The algorithm is guaranteed to visit finite number of regions. The union of these regions is the entire belief space.

9. Simplified version of Sondik’s algorithm: Sondik’s One-Pass Algorithm

10. How to define a region around this belief state where that vector is guaranteed to be true linear portion of the value function? Construct a series of constraints when satisfied, region is found. Then go step (5) Sondik’s One-Pass Algorithm

11. The condition  *(t), generated at , larger for all other  a (t), as  varies: Variations in  can cause changes in  a (t). Need a new constraint to ensure components of  a (t) stay the same. Sondik’s One-Pass Algorithm

12. What affects  *(t) and  a (t)? To ensure that every part of the function does not change, these constraint exists for every combination of a and  Sondik’s One-Pass Algorithm

13. Constraints restrict belief states to lie on the belief state space simplex: Sondik’s One-Pass Algorithm

14. A constraint consists of a region with all the points on one side of the line: Sondik’s One-Pass Algorithm

15. The LP constraints at step (4): Sondik’s One-Pass Algorithm

16. In step (5), find belief states guaranteed not to be in region defined in step (4). With the new point proceed exactly as step (4). The algorithm goes until a complete partition of the belief space found. Sondik’s One-Pass Algorithm

17. To find points in the neighboring regions, points lying on the edge of the region defined by the constraints is used: Sondik’s One-Pass Algorithm

18. Which constraints are binding: For each constraint, change its inequality into an equality, Solve this LP. If the LP has solution, it is a binding constraint, a non-binding constraint can not pass through the region defined by all other constraints. Sondik’s One-Pass Algorithm

19. Cheng’s Relaxed Region Same as Sondik’s One Pass algorithm except each region specified with fewer constraints. Defines regions that will typically be larger than the actual vectors’ regions.

20. Set of constraints for the relaxed regions of Cheng: Cheng’s Relaxed Region

21. Corners found with interior algorithm:

22. The algorithm defines an approximate value function over the entire belief space. Refine this approximation until it reaches the optimal value function. Cheng’s Linear Support

23. Difference between two algorithms: Cheng’s Linear Support

24. Initiliaze a search list with extreme points on the belief simplex(e.g. [1,0,0...],[0,1,0,0...]), and an empty set of vectors. For each of these points the true  (t) vector calculated, and added to the set of vectors. Cheng’s Linear Support

25. Since both the true and the approximation are PWLC, the largest difference must occur at a corner point. Cheng then finds all the corner points of the regionsinduced by the approximation. Disregard the corner points seen before and add those not seen before to search list. Pick a point from the search list, generate the vector. If it is different all the other approximation, add it to the approximation set. Repeat whole procedure with the new approximation Cheng’s Linear Support