What is the temporal feature in video sequences? It refers to the temporal relationships of video events in the time space For example, such a typical basketball video sequence consists of a finite number of scenes such as left court, middle court, right court, close-up. A basketball video sequence generally goes through a routine that enters one scene at a time, remains for some duration time and transits into another scene. You can make a model to describe this scene movement.

First problem: Markov Process a1 a2 a3 a4 We first attempt to model the process that could have generated the pattern. What I was doing is to use a first order markov chain in which the current state always depends on the previous state.. For any frame in left court scene, it may continue staying at the left court, or it may next go to the middle court or right court with different probabilities. So we can build this matrix to describe this markov process. Say, in this matrix, a11 denotes the probability the left court scene remain itself, a12 denotes the pro. Of transition from ….

Second problem: Hidden Markov Model Construct a model to explain a time sequence of events Use the model to identify other observation sequence In some case the pattern we wish to find are not described sufficiently by a single Markov process. In this diagram, ….., in the markov process, there are 3 states. right court scene and left court scene are combined to one state because they have similar color feature. Return to the second problem, we want the model have the capacity to identify the given video sequence. We can not have access to evaluate its video topic with the markov process. But We see that the pattern and the given video sequence are probabilistically related. You can calculate the visual similarity between the frames and states. You also can match the underlying temporal feature in the given video and transition pattern of the markov chain. Here we model such a process using a hidden markov model where there is an underlying hidden markov process changing over time, and a set of observable states that are related somehow to the hidden states. Modeled markov states are hidden states, and to compare with the hidden states, we call the given video sequence observations or observable stats. The essence of hmm is to first construct a model to describe the occurrence of events in a time sequence and then use this model to identify the new observations.

HMM Elements Observation sequence: O={O1,O2,…,OT} Hidden states: S={S1,S2,…,SN} State transition probability A={aij} Observation probability B={bij} Initial state distribution Based on the above analysis, we give the elements of a HMM here. A hmm has 5 elements: In this case, observations sequence refers to the given video frame sequence that will be input to the hidden markov model to be classify, Hidden states describe the states of markov process. We extract a 12-vector for each frame and cluster them to find the hidden states State transition probability characterizes the temporal relationship between the hidden states. The connections between the observation and hidden states represent the probability of their visual similarity, which is called observation probability. Initial state distribution contains the probability of the model being in a particular hidden state at time t=1, that is the start point of a HMM

Chromaticity removes shading and magnitude: Here is the definition of chromaticity, changing the image from r.g.b 3 channel to the 2d chromaticity in such way , that can be used to remove the shading and magnitude.

This is a flow showing the image processing.

Observable State Sequence for a given video Temporal keyframe-based Summarized Video (TSV) Efficient Removing frame noise Video frame sequence Now we want to generate the observtion sequence for a test video. Rather than directly using the given video frame sequence, we define a … as input of observation sequence to HMM. What I was doing is first segment scenes, and extract the keyframe from each scene, then repeat the every keyframe as a number times as the number of frames in the scenes. Like, this scene has 3 frames, for tsv, its keyframe will be repeated 3 times. So you can input the 12 vector sequence of tsv into HMM. This way has 2 adv. It is effi, because you don’t need to do the matching computation for every frame, the second is it can help remove the frame noise, obviously it remove the transition part between scene that would cause confusion. ... … TSV