1. 1 Temporal Uncertainty Computation, Fusion, and Visualization in Multisensor Environments Pramod K. Varshney Kishan G. Mehrotra C. Krishna Mohan Electrical Engineering and Computer Science Dept. Syracuse University Syracuse, NY 13244 Phone: (315) 443-4013 Email: varshney@syr.edu

2. 2 Outline Introduction Temporal Update Mechanisms for Decision Making in Probabilistic Networks Sensor and Bandwidth Management in Distributed Sensor Networks Temporal Fusion in Multi-Sensor Target Tracking Systems Uncertainty Computation and Visualization Concluding Remarks

3. 3 Information Acquisition and Fusion Model for Visualization Dynamic network connectivity with varying bandwidths Heterogeneous mobile agents in terms of resources and capabilities

4. 4 Sample Military Scenario

5. 5 Technical Objectives Decentralized inferencing algorithms Data/information fusion models and algorithms Algorithms for uncertainty computation and integration Methods for uncertainty representation and visualization Experimentation with real data and testbeds

6. 6 Main Accomplishments Development of information fusion and visualization algorithms that take temporal effects into account –Decision making in Bayesian networks –Sequential detection problems –Target tracking –Uncertainty visualization of mobile objects

7. 7 Temporal Effects Multiple mobile observers with different reliability characteristics send in reports at different points in time Target being observed is itself changing in observable or inferable characteristics Information arriving later is expected to be more reliable and relevant than earlier information

8. 8 Temporal Update Mechanisms for Decision Making with Aging Observations in Probabilistic Networks

9. 9 Background Bayesian causal networks are being used for modeling many important uncertainty-related problems (cf. current work by Decision-Making under Uncertainty MURIs) Practical battlefield management tasks involve reasoning with uncertainty that varies over time, e.g., observations lose their predictive power as time elapses, and visual observations are more reliable in daytime (better visibility conditions).

10. 10 Objectives To incorporate time-dependence of observations and evidence in Bayesian inference networks. To model a wide range of time-dependent uncertainty computations using few parameters that can be queried or learned based on past data. To develop an easily usable tool that visualizes and updates time-dependent uncertainty measures in multisensor hierarchical decision-making environments.

11. 11 Related Work Dean and Kanazawa, 1989: Survivor functions used to represent changing beliefs –Limited modeling power Kjaerulff 1995 and others: Causal networks with nodes duplicated for different time slices –Networks become very large and are difficult to compute with –Darwiche, 2001 proposes algorithms to improve their space and time complexity Tawfik and Neufeld, 1996: Markov chain representations used to analyze the degeneration of relevance of information with time –Difficult to use in practice, especially when computations must also depend on actual time points at which observations are made

12. 12 Detection/Recognition of an Object Sensor 1 Sensor 2 Sensor 3 Object Processor 1 Processor 2 Central Decision Maker

13. 13 Information Flow Central decision maker generates the global inference while accounting for time delays

14. 14 Causal Network Model These arrows represent the causal links between nodes

15. 15 Conditional Independence Inferences about the probability of B at time t B >t A are made based on the priors and the pairwise conditional probabilities associated with the links in the figure P(B:t B |A:t A )= P(B:t B |B:t A ). P(B:t A |A:t A ) + P(B:t B |~B:t A ). P(~B:t A |A:t A )

16. 16 Temporal Belief Updates We have developed temporal belief update algorithms that address: –Dependence of conditional probabilities on absolute times t A and t B –Dependence of conditional probabilities on relative time delays (t A – t B )

17. 17 Relative Time-Decay Model linear Exponential Juxtaposition of f and g models a large variety of practical scenarios

18. 18 Two Temporal Update Models Lazy (Belief update on demand) Non-Lazy (Steady updates)

19. 19 Lazy Belief Updating Computation by B needs to be carried out only when node C requests the latest belief of B, given the most recent observation at A The conditional probability associated with an evidential edge does not require temporal updating until an observation is actually made at that node

20. 20 Non-Lazy Belief Updating Time-dependent updates are restricted to edges between non-evidential nodes and are performed on a periodic basis The belief at each node decays steadily using a fixed multiplicative decay constant, e.g., P(C:t C +1|B:t B )=k.P(C:t C |B:t B )+(1-k).P(C) for t C >t B

21. 21 Implementation of Relative Case A tool was developed in Matlab, implementing the relative time-delay model with lazy belief updating A graphical user interface facilitates updating and viewing of results

22. 22 Single Target Example Target Reading of sensor one Reading of sensor two Reading of sensor three Report from processor one Report from processor two

23. 23 Synchronous Reports (Single Target) The simulation shows that the probability of uppermost node decays toward 0.5 (the pre- assigned prior probability)

24. 24 Decay of Inference Hypothesis Probability (Single Target)

25. 25 Asynchronous Reports (Single Target) At time 0, no information is available from either processor At time 1, the first processor reports a positive sighting At time 2, the second processor reports a positive sighting

26. 26 Temporal Updates of Inference Hypothesis Probability: Asynchronous Reports (Single Target)

27. 27 Asynchronous Reports (Single Target) At time 0, both processors report a negative sighting. At time 1, the first processor reports a positive sighting At time 2, the second processor reports a positive sighting

28. 28 Temporal Updates of Inference Hypothesis Probability: Asynchronous Reports (Single Target)

29. 29 Multiple Targets Example Target 1 Reading of sensor one Reading of sensor two Reading of sensor three Report from processor one Report from processor two Target 2

30. 30 Processors with Different Temporal Decay Parameters Thicker lines indicate stronger links (higher conditional probs.) Info. from first observer decays imperceptibly. Info. from observer 2 decays fast with time

31. 31 Multiple Targets Case

32. 32 Future Work (1) Position uncertainty modeling using hierarchical spatial grids along with the network models Target classification using the network model (non-binary hypothesis nodes) Modeling practical large-sized problems using the new tool Applying data-driven learning algorithms to determine time-dependence of conditional and prior probabilities, based on data Knowledge-elicitation process to develop the right time- dependent uncertainty model. Improving network visualization and user interface (UCSC) Test with mobile visualization testbeds (Ga Tech and USC)

33. 33 Sensor and Bandwidth Management in Distributed Sensor Networks

34. 34 Bandwidth and Energy Considerations Reduction of communication cost is a key focus of distributed sensor networks –Bandwidth –Energy Bandwidth constraints necessitate the compression of data collected at local sensors

35. 35 Key Questions What is the relationship between data compression and the resulting system performance? If a fixed amount of total bandwidth is available, then what is the optimal allocation of bandwidth (bits) to heterogeneous sensors?

36. 36 Tradeoff Tradeoff between the bandwidth, decision quality (QoS) and time-to- decide –Fixed sample size (FSS) detection problems Bayesian criterion: optimal bandwidth distribution across sensors to achieve minimum probability of error –Sequential detection problems Optimal bandwidth distribution across sensors to achieve minimum time delay of decision making for specified detection performance

37. 37 Distributed Sequential Detection denotes the number of bits assigned to sensor i=1,2,…,M Local Sensor #1 Local Sensor #2 Local Sensor #M Fusion Center

38. 38 Quantization and Decision- Making Local sensor, Q i, quantizes into m-ary variables,, prior to transmission Quantized data,, are sent to the fusion center where a sequential data fusion scheme is implemented to reach a global decision

39. 39 Sequential Prob. Ratio Test At time t, fusion center performs the SPRT as follows: where

40. 40 Average Sample Number Neglecting the excesses over the test thresholds, the average sample number (ASN) when is true is where

41. 41 Bandwidth Management Goals: Partition available bandwidth B optimally into Optimally quantize each sensor’s observation space Optimality criterion: minimization of ASN

42. 42 Bandwidth Allocation Algorithm Optimization algorithm –Sort the sensors in decreasing order of SNR –For b=1 to B, do: Scan the sensor in the above sorted order and assign the bth digit to the sensor that minimizes ASN Assignment of incremental bandwidth to more informative sensors results in better performance in terms of ASN Because of the concavity of ASN as a function of B, this systematic approach based on marginal analysis generates an optimal bit allocation

43. 43 Target Detection Example A distributed sensor network consists of ten sensors of different capabilities in terms of SNR Task: detect if there is a target or not, which is assumed to be equiprobable Constraint: Total available bandwidth is limited Goal: Make a decision as quickly as possible while still satisfying the specified probabilities of false alarm and missed detections

44. 44 Bit Allocation for Different Bandwidth Constraints # of available bits S1 S2 S3 S4 S5 S6 S7 S8 S9 S10# of available bits S1 S2 S3 S4 S5 S6 S7 S8 S9 S10 1 2 3 4 5 6 7 8 9 10 11 12 13 1 0 0 0 0 0 0 0 0 0 1 1 0 0 0 0 0 0 0 0 1 1 1 0 0 0 0 0 0 0 1 1 1 1 0 0 0 0 0 0 1 1 1 1 1 0 0 0 0 0 2 1 1 1 1 0 0 0 0 0 2 1 1 1 1 1 0 0 0 0 2 1 1 1 1 1 1 0 0 0 2 2 1 1 1 1 1 0 0 0 2 2 1 1 1 1 1 1 0 0 2 2 1 1 1 1 1 1 1 0 2 2 2 1 1 1 1 1 1 0 2 2 2 1 1 1 1 1 1 1 14 15 16 17 18 19 20 21 22 23 24 25 2 2 2 2 1 1 1 1 1 1 2 2 2 2 2 1 1 1 1 1 2 2 2 2 2 2 1 1 1 1 3 2 2 2 2 2 1 1 1 1 3 2 2 2 2 2 2 1 1 1 3 2 2 2 2 2 2 2 1 1 3 3 2 2 2 2 2 2 1 1 3 3 2 2 2 2 2 2 2 1 3 3 3 2 2 2 2 2 2 1 3 3 3 2 2 2 2 2 2 2 3 3 3 3 2 2 2 2 2 2 3 3 3 3 3 2 2 2 2 2

45. 45 ASN as a Function of Total Available Bandwidth Pf = Pm = 10e-5, 10 sensors with sigma=[1, 1.1, 1.2, 1.3, 1.4, 1.5, 1.6, 1.7, 1.8, 1.9]

46. 46 Time-dependent Cost Formulation SPRT cost: where c(k) is a time-dependent cost per- digit Determine B* that minimizes C. Also, find bandwidth distribution along with quantizer parameters

47. 47 Time-dependent Cost as a Function of Total Bandwidth Pf=Pm=10e-5, 10 sensors with sigma = [1, 1.1, 1.2, 1.3, 1.4, 1.5, 1.6, 1.7, 1.8, 1.9]

48. 48 Future Work (2) Improved accuracy –Renewal theory Dynamic environment –Dynamic bandwidth allocation in distributed sensor networks –Sensor selection Multiple hypotheses—classification and recognition

49. 49 Temporal Fusion in Multi-Sensor Target Tracking Systems

50. 50 Key Issues How does the estimation uncertainty evolve temporally? What are the effects of the asynchronous sensors on tracking system performance? Can we benefit by using asynchronous sensors? If so, how can we design asynchronous or temporal staggering pattern to maximize the benefit?

51. 51 Synchronous vs. Asynchronous Measurement Patterns For a multi-sensor tracking system, sensors can be either synchronous or asynchronous (temporally staggered) T: Sampling interval of synchronous sensors T1: Time difference between sensor 1 and sensor 2 in asynchronous-sensor case T=T1+T2

52. 52 Estimation Error as a Function of Time The system with temporally staggered sensors is a better choice when the major concern is to keep maximum prediction error or average estimation error low

53. 53 System Performance Metrics To capture the system performance over time, we construct a family of metrics. The average error variance, AEV is defined as where w(t) is a weighting function which satisfies and V(t) is the estimation error variance at time t

54. 54 Two Special Cases of AEV At the time of Observation: Averaged over time: : AEV for position estimation : AEV for velocity estimation

55. 55 AEV vs. Staggering Interval Length

56. 56 Optimal Staggering Pattern To get the lowest AEV, we numerically calculate steady state covariance matrices and use optimization techniques. We find it is best to uniformly stagger sensors with same measurement noise variances. For sensors with same measurement noise variances, we analytically prove that the and of the system with uniformly staggered sensors always outperform those of the system with synchronous sensors.

57. 57 AEV P vs. Target Maneuvering Index measures the degree of elusiveness of the target to be tracked.

58. 58 AEV V vs. Target Maneuvering Index

59. 59 Staggering Time for Minimum AEV P for Two Heterogeneous Sensors r : the ratio between the two sensors’ measurement noise variances

60. 60 Staggering Time for Minimum AEV V for Two Heterogeneous Sensors

61. 61 Future Work (3) Investigate the optimal staggering pattern for systems with more than two sensors with different measurement noise variances. Take into account the false alarms and missed detections. Study the effect of staggered sensors in multiple-target scenarios.

62. 62 Uncertainty Computation and Visualization

63. 63 Particle Movement Model Uncertainty in initial position, direction and speed Uncertainty modeled by Gaussian distribution Joint work with Suresh Lodha of UCSC

64. 64 Constrained Target Tracking

65. 65 Future Work (4) Ground target –Limited speed –Low Maneuverability –On road or in the open field –Road junctions –Varying obscuration conditions (tunnels, hills, etc.) Tracking algorithm –Constrained vs. unconstrained problem –Particle filter (sequential Monte Carlo method) Uncertainty in terms of covariance matrices Joint work with Christian Fruh and Avideh Zakhor: Using constrained tracking techniques, digital road maps and aerial photographs to improve the localization of a moving vehicle in a city.

66. 66 Some Technical Outreach Activities Collaborative project on information fusion, visualization, and integrated display systems –Andro Consulting Services, Rome, NY. –AFRL, Information Directorate –The NYS Center for Advanced Technology (CAT) in Computer Applications and Software Engineering (CASE) Technical exchange with the Decision Fusion MURI –Alan Willsky, MIT –Sanjeev Kulkarni, Princeton

67. 67 Concluding Remarks Highlights of accomplishments –Decision making with aging observations in probabilistic networks –Temporal sensor staggering in multi-sensor target tracking Plans for next year –Information fusion for heterogeneous sources in dynamic environments –Uncertainty computation models and algorithms –Collaborative research Uncertainty visualization with UCSC Estimation and tracking with UCB Mobile visualization and experimentation with Ga Tech and USC Information fusion with MIT and Princeton