1. Video Analysis Tool Box for Digital Video Forensics By
Susinda Perera
Department of Computer Science and Engineering,
University of Moratuwa,
Supervised by
Dr. Chathura De Silva
PhD (NUS-Singapore), MEng (NTU-Singapore), BSc Eng.(Hons) (Moratuwa)
Senior Lecturer

2. Agenda Project Intro Implementation Demo Video Stream Analyzer
Video Stabilizer
Compressed domain video analysis
Demo

3. Problem Statement Can we trust digital videos?
Are they real, computer generated or tampered
Extract some wanted Information from video
Difficult due to unclearness of video

4. Can we trust digital videos?
Figure 1‑1 : A still from controversial video aired on Channel 4
Source – YouTube

5. Can we trust digital videos?
Figure 1‑2 : A still from eagle catching kid
Source –

6. Unclear Videos

7. Solution Video Stream analysis tool Video Enhancement tool
To ensure the authenticity and integrity
Video Enhancement tool
To improve the visual quality

8. Market Analysis Video Stream Analyzing Tools Elecard StreamEye
Tektronix MPEG Software Tools
MPEG-2 Transport Stream packet analyser
TSReader
MTS4EA Elementary Stream Analyzer

9. Screenshots from Elecard StreamEye

10. Some Features Of Video Stream Analyzing Tools
Navigation and display of media stream picture-by-picture (I, P, B).
Display of the current frame.
Display of the time, type, size and number of a current frame in a stream, decoding order and offset from the file beginning.
Display of the bit rate (declared in the sequence header) and a calculated bit rate.
Display of detailed information about macroblocks in MPEG-1 (ISO/IEC ), MPEG 2 (ISO/IEC ), MPEG-4 (ISO/IEC ) and AVC/H.264 (ISO/IEC ) video streams.
Information about motion vectors
Frame-accurate positioning.
Display of the stream and gathering of statistics relating to the entire file.

11. What is missing in existing tools
All high end tools are commercial ones, Expensive
Almost all tools are designed for DVB purposes
Lack of detailed analyzing features for video forensic
Lack of textual representation

12. Video Forecsic /Enhancement Software
Cognitech
Ocean Systems dTective
Salient Stills VideoFOCUS
StarWitness
Avid Technology, Inc.
Intergraph Video Analyst
TREC, Inc.
Forevid
MotionDSP Ikena
Amped FIVE
Kinesense

14. Features Of Video Enhancement Tools
Video Stabilization
Denoising
Deblur Filters
Detection Filters
Enhancement
Histogram Editor
Segmentation
Tracking
Transform
Zoom
Velocity
Reconstruction

15. Implementation of Video Stream Analyzer

16. Literature Review Market Analysis Possible resources [31- 35]
Most of tools are for DVB, not for forensic purposes
Lack of textual output
Possible resources [31- 35]
Video Player libraries / source codes
Decoder Libraries/Source Codes
MPEG2Event[33]is a possible candidate

17. Video Stream Analyzer Implementation Based on MPEG2Event[33] library
Mpeg demuxer
ffmpeg for stream information
A simple thread sync model to get motion vectors
Quartztype library for video player implementation
C# inbuilt xml libraries for xml manipulation

18. What is MPEG stream
MPEG Bit-Stream Structure (Source [1] )

19. GOP Structure and Display order
GOP Structure and Transmission Order. (Source: [2])

20. Video Stream Analyzer

21. Video Stream Analyzer Features Free and open source
Textual output (xml) of analysis reports
For further analysis
All most all the features in slide 10
Motion vector display
Frame navigation
Bit stream index/ Display index view
User data display
Motion vector summary – xml output
Full video summary – xml output

22. Video Stream Analyzer -outputs

23. Motion Vectors Summary <Pictures>
<MPicture>
<PicType>I</PicType>
<MBlocks>
<MVector>
<Row>0</Row>
<Col>0</Col>
<ForwardX>0</ForwardX>
<ForwardY>0</ForwardY>
<BackwardX>0</BackwardX>
<BackwardY>0</BackwardY>
<IsForwardPresent>false</IsForwardPresent>
<IsbackwardPresent>false</IsbackwardPresent>
<IsSkipped>true</IsSkipped>
</MVector>
<Col>1</Col>
<Col>2</Col>

24. MacroBlocks Summary

25. Implementation of Video Enhancement tool

26. Video Enhancement Tool
Objective
Enhance the visual quality
Video Stabilization
Noise removal
Color Correction

27. Video Stabilization Removing annoying shaky motion from videos
helpful in identifying people, number plates, etc. from low-quality video cameras
Three aspects
Inter frame motion estimation
Motion smoothing and compensation
Filling up the missing image areas.
Main references [3], [4] , [5]

28. Video Stabilization Inter Frame Motion Estimation
Result of video stabilization. Top row: Original input sequence, middle row: stabilized sequence which still has missing image areas, and bottom row: stabilized and completed sequence. The grid is overlaid for better visualization. (Source [3])

29. Handling camera motion
a) Natural scene, b) when camera is moved, c) when image is shifted to compensate the motion.

30. Motion Model 𝑓 𝑥, 𝑦, 𝑡 = frame at time t
𝑇 = affine transform
𝑓 𝑥, 𝑦, 𝑡 =𝑓 𝑚 1 𝑥+ 𝑚 2 𝑦+ 𝑚 5 , 𝑚 3 𝑥+ 𝑚 4 𝑦+ 𝑚 6 , 𝑡−1
𝑓 𝑥, 𝑦, 𝑡 = T * 𝑓 𝑥, 𝑦, 𝑡 −1
How to find T ?
There papers ([3], [4], [5]) mentioned above use diffèrent mechanisms to find the transform matrix parametrs
𝑇= 𝑚 1 𝑚 2 𝑚 3 𝑚 𝑚 𝑚 6

31. Motion Estimation Computing inter frame motion
Use of object recognition
Scale Invariant Feature Transform(SIFT) features
Minimizing quadratic error function with a proposed model [5]
𝐸 𝑚 = 𝑅𝑂𝐼 𝑓 𝑥, 𝑦, 𝑡 −𝑓 𝑚 1 𝑥+ 𝑚 2 𝑦+ 𝑚 5 , 𝑚 3 𝑥+ 𝑚 4 𝑦+ 𝑚 6 , 𝑡− 𝑅𝑂𝐼 𝑓 𝑥, 𝑦, 𝑡 −𝑓 𝑚 1 𝑥+ 𝑚 2 𝑦+ 𝑚 5 , 𝑚 3 𝑥+ 𝑚 4 𝑦+ 𝑚 6 , 𝑡−1 2

32. Stabilization

33. Motion Smoothing
A stabilized motion path is obtained by removing undesired motion fluctuation.
Assumed that the intentional motion in the video is usually slow and smooth
Uses Gaussian kernel in most literatures
Applies Gaussian kernel to neighboring N frames
Gaussian kernel + curve fitting methods

34. Motion Smoothing
Let Nt = {j|t-k<=j<=t+k} be the neighboring frames
And It is the frame at the origin
Calculate the position of each neighboring frame Is, relative to frame It using transform matrixes defined above ( lets say Tst)
Find the correcting transformation S from the original frame It to the motion-compensated frame I’t according to
Where G is a Gaussian kernel of size k

35. Motion Smoothing
The global transformation chain T defined over the original video frames Ii, and the transformation from the original path to the smoothed path S. (Source [3])

36. Filling up missing image areas
Not addressed here
Due to time limitations and complexity
Some techniques used in research literature
Motion Inpainting [3]
the local motion data in the known image areas is propagated into the missing image areas.
The propagation starts at pixels on the boundary of the missing image area. Using motion values of neighboring known pixels
Use of dynamic programming [3],[6]

37. Noise Removal (Denoising Filters)
Denoising Filter Categories [7]
Nonmotion compensated spatiotemporal
Motion compensated spatiotemporal
Nonmotion compensated temporal
Motion compensated temporal filters
What is first? motion compensation or denoising?
Each have their procs and cons
Depends on the video
References
[7], [8], [9], [10], [11]

38. Implementation of Video Enhancement Tool

39. Implementation of Video Enhancement Tool
Video Stabilization
Based on the algorithm described in [5]
+ Improvements to reduce error propagation
Makes a additional pass (slow)
+ Improvements to support color video
Noise Removal and color Correction
Based on openCV filters

40. A Compressed Domain Approach for Video Tamper Detection

41. Common Modes of Tampering
Classification
Spatial, Temporal or Both (Spatio-temporal)
Temporal
Frame insertion, deletion, modification and reordering
Spatial
Similar, but operates on objects within a frame
Based on MPEG's layer structure ([12])
GOP level, frame level and single- or multi-block level
GOP jittering, inter-GOP frame and intra-GOP frame jittering, or their combinations

42. Tamper detection techniques
Video Authentication
Pasive
Pixel Domain
Pixel Statistics Based
Detecting resampling
Double MPEG compresion
Inconsistant Noise patterns
Duplication detection
Physics Based
Motion Analysis
Shadow Analysis
Compressed Domain
Active
Watermarking
Digital Signature
References – [12], [13], [14], [15], [16], [17]

43. Compressed Domain approaches
Many researches can be found on detecting alterations by image processing means.
But very few initiatives in Compressed Domain
Due to the lack of techniques to grab the frame data from compressed video
Possibilities
Analyze stream structure headers w.r.t. its contents
Analyze userdata
Analyze inconsistencies in frame sizes, block sizes
Analyze motion vectors

44. Our Approach Based on motion vectors Assumption
For an unaltered video the motions present in the video need to be smooth and therefore should not contain sudden changes
Tampering disturbs the motion flow in a video
Extract the motion information from video and compare them with the above assumption.
Motion information is derived from the motion vectors [17], [18]

45. Concept of Motion Vector
MPEG achieves it its high compression rate by the use of motion estimation and compensation
Instead of encoding each block in the current frame, matching block is search in the past frame and if suitable block is found, the difference between the two macroblocks is encoded and transmitted
The displacement between the two macroblocks is represented using motion vector

46. Concept Of Motion Vector cont..
Suppose that the macroblock “x” is the macroblock we wish to encode and macroblock “y” is the counterpart in the past frame.
A search is done around “y” (search area) to find the best match for “x”.
The displacement between the two macroblocks gives the motion vector associated with “x”.

47. Motion Vectors vs MacroBlocks
As a part of MPEG encoding process motion vectors are calculated and inserted to the bitstream.
In mpeg bitstream motion vectors are associated with macroblocks. But not every macroblock will contain a motion vector
There are four basic types of macroblock types
Intra
Skipped
Forward predicted
Backward predicted

48. Forward and Backward predictions
Picture Type
Possible Macroblock Types
Motion Vectors I
Intra
No motion vectors present P
Skipped, Forward predicted
Only forward motion vectors may present B
Skipped, Forward predicted, Backward predicted
Both Forward and backward motion vectors may present
Summary of motion vectors and macroblocks

49. Deriving Motion Field from Motion Vectors
Motion vectors represent the motions ,But not directly (therefore we need to derive)
Steps
Reorder the bitstream into display order
A simple algorithm
Generate Motion felid from motion vectors
[18] defines set of rules.
Simplified mechanism and rule set is described in the [19].
Filter out the outliers
Median filter (3*3 as in [18])

50. Simplified rule set for deriving motion field
Macroblocks with no motion vector have the same movement as in the previous image.
This is required since there is a small percentage of intra-coded blocks in B- and P-frames that would otherwise have unknown motion. Also, I-frames have no motion at all so this rule will provide a smooth motion field for them. The justification for this is step is that consecutive images are strongly correlated therefore there is a high probability that the motion field has not changed much.
When a macroblock has two motion vectors, the one pointing back is reversed and added to the one pointing forward.
This step accomplishes both noise reduction (due to the averaging effect) and normalizes vector magnitude (so they span three images in B-frames which is what happens in P-frames).
Motion vector magnitude is normalized.
If a macroblock in a B-frame only has one motion vector then it has to be multiplied by a factor so its magnitude corresponds to a vector spanning three images.
Skipped macroblocks in I-frames have no movement, while in P-frames, they have movement similar to the previous block.
This is a rule obtained directly from the MPEG-2 standard.

51. Implementation of rules
Let’s consider a transition from frame F1 to F2, Lets define mbF1 and mbF2 be the corresponding macroblocks in F1 and F2.
Transition
Motion Vectors Present
I to B
P to B
P to P
motion = mbF2.FwdMV
B to B
B to P
If only forward vectors present: motion = mbF2.FwdMV – mbF1.FwdMV
If only backward vectors present: motion = mbF2.BwdMV –mbF1.BwdMV
If both forward and backward vectors are present:
motion = ((mbF2.FwdMV – mbF1.FwdMV) + (mbF2.BwdMV –mbF1.BwdMV) )/ 2
P to I
Not defined

52. Motion field
The derived motion field is considered as the motion for the frame to frame transition.
Now the derived motion field contains the motion information for every macroblock
Motion vectors of the left image

53. Derived Motion field for the above motion vector field and next frame

54. Smoothed motion field of the above

55. Detecting Motion Inconsistencies
If we consider a typical video, most of pixel data in two adjacent frames are same.
Differences are present due to moving objects of the scene, but these differences are small in magnitude.
Similar assumption is made in [13]
In simple terms, for three adjacent frames, motion between second and third frame is very much similar to motion between first and second frames

56. Mathematical definition of the assumption
Consider three frames 𝑓 𝑡 , 𝑓 𝑡+1 𝑎𝑛𝑑 𝑓 𝑡+2 , and a block in row j and column i in frame 𝑓 𝑡 is denoted by 𝑏 (𝑖,𝑗,𝑡) . for every block in these frames the following inequalities should hold.
𝐸𝑥= 𝑚𝑜𝑡𝑖𝑜𝑛𝑋 𝑏 𝑖,𝑗,𝑡 , 𝑏 𝑖,𝑗, 𝑡+1 − 𝑚𝑜𝑡𝑖𝑜𝑛𝑋 𝑏 𝑖,𝑗,𝑡+1 , 𝑏 𝑖,𝑗, 𝑡 < 𝜀
𝐸𝑦= 𝑚𝑜𝑡𝑖𝑜𝑛𝑌 𝑏 𝑖,𝑗,𝑡 , 𝑏 𝑖,𝑗, 𝑡+1 − 𝑚𝑜𝑡𝑖𝑜𝑛𝑌 𝑏 𝑖,𝑗,𝑡+1 , 𝑏 𝑖,𝑗, 𝑡 < 𝜀
Where 𝑚𝑜𝑡𝑖𝑜𝑛𝑋 𝑎𝑛𝑑 𝑚𝑜𝑡𝑖𝑜𝑛𝑌 would be the functions which output the motion in X and Y directions respectively and 𝜀 is a very small positive constant.
Here the 𝑚𝑜𝑡𝑖𝑜𝑛𝑋 𝑎𝑛𝑑 𝑚𝑜𝑡𝑖𝑜𝑛𝑌 functions can be simply replaced by motion field since the motion field we derived has the exact meaning.
𝐸𝑥= 𝑚𝑜𝑡𝑖𝑜𝑛𝐹𝑒𝑖𝑙𝑑𝑋 𝑖, 𝑗, 𝑡 − 𝑚𝑜𝑡𝑖𝑜𝑛𝐹𝑒𝑖𝑙𝑑𝑋 𝑖, 𝑗, 𝑡+1 < 𝜀
𝐸𝑦= 𝑚𝑜𝑡𝑖𝑜𝑛𝐹𝑒𝑖𝑙𝑑𝑌 𝑖, 𝑗, 𝑡 − 𝑚𝑜𝑡𝑖𝑜𝑛𝐹𝑒𝑖𝑙𝑑𝑌 𝑖, 𝑗, 𝑡+1 < 𝜀

57. How to detect tampering
Get motion vectors for every frame
Derive motion fields for each transition
Check whether motion fields satisfies the above two equations, for every transition (i.e every motion field with its next)
Based on our assumption if it does not satisfy above it is more likely to be a manipulated video.

58. Results
For low resolution videos the assumption we made fails 90% of the time(success for large 𝜀)
It is observed that derived motion field itself has discontinuities and noise.
For high resolution videos assumption satisfies but fails when there are multiple flows of motion is there.
It is difficult to fully automate the tamper detection process, need to have manual intervention.

59. Summary Video Stream Analysing tool Video enhancement tool
Existing tools are for DVB and not for forensic purposes
Implemented using open stack of tools and libraries
Supports detailed view of bit-stream as well as textual outputs fro further processing
Video enhancement tool
Two major issues in video footprints are unclearness and shakes
This tool helps to stabilize a video and reduce noise
Order of stabilize and denoise, has their own pros and cons, and also may depends on video
Compressed domain approach for tamper detection
Deriving motion fields from motion vectors
Rule set
Consistency in motion Assumption and mathematical derivation
Detecting motion inconsistencies
Showed that video bit-stream data (compressed domain) gathered by stream analyzing tool can be use as an aid for video forensic analysis

60. Future Work
Video Stabilization algorithm can be improved to Fill the missing image areas.
Video stream analyzer need to support new formats of video.
Compressed domain tamper detection mechanisms need to be provided as tools and can be integrate with stream analyzer tool.
Motion object segmentation methods can be used in combination with proposed motion field method
detect motions of individual objects and then apply our assumption
which would give better results since we apply our assumption on individual motion fields

61. References
[1] MPEG-2 White Paper. [Online].
[2] MPEG Encoding basics, 1st ed.: Snell & Wilcox , 2007 [Online]
[3] Rongjie Shi, I-fan Shen, Wenbin Chen Rong Hu, "Video Stabilization Using Scale-Invariant Features," in Proc. of the 11th International Conference Information Visualization, 2007, pp
[4] Eyal Ofek, Weina Ge, Xiaoou Tang, Heung-Yeung Shum Y. Matsushita, "Full-Frame Video Stabilization with Motion Inpainting," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 28, no. 7, pp , July 2006.
[5] Jeffrey B. Woodward Hany Farid, "Video Stabilization and Enhancement," Department of Computer Science , Dartmouth College , TR , 1997.

62. [6] E. Shechtman, and M. Irani Y
[6] E. Shechtman, and M. Irani Y.Wexler, "Space-time video completion," in IEEE Conf. on Computer Vision and Pattern, vol. 1, 2004, pp
[7] R. P. Kleihorst, S. Efsratiadis, A. K. Katsaggelos, and R. L. Lagendijk J. C. Brailean, "Noise reduction filters for dynamic image sequences: A review," Proceedings of the IEEE, vol. 83, pp , 1995.
[8] B. Coll, J. M. Morel A. Buades, "Denoising image sequences does not require motion estimation," Dpt. Matematiquesi Informatica, Universitat Illes Balears.
[9] William T. Freeman 2 Ce Liu 1, "A High-Quality Video Denoising Algorithm based on Motion Estimation," 1Microsoft Research New England, 2 Massachusetts Institute of Technology
[10] M.I.Sezan, A.M. Tekalp M.K. Ozkan, "Adaptive motion compensated filtering of noisy image sequences," IEEE Trans, circuits, vol. CSVT-3, pp , Aug 1993.
[11] B. Coll, J. M. Morel A. Buades, "A non-local algorithm for image denoising," IEEE International Conference on Computer Vision and Pattern Recognition, 2005

63. [12] Hong Heather Yu Peng yj, "Classification Of Video Tampering Methods and Countermeasures Using Digital Watermarking," SPIE, Multimedia Systems and Applications, vol. IV, pp
[13] Wang Weihong, "Digital Video Forensics," DARTMOUTH COLLEGE, Hanover, New Hampshire, Thesis 2009.
[14] Farid H Wang W, "Exposing Digital Forgeries in Video by Detecting Double MPEG Compression," ACM Multimedia and security workshop, 2006.
[15] S.K. Singh S. Upadhyay, "Video Authentication: Issues and Challenges," IJCSI International Journal of Computer Science Issues, Vol. 9, Issue 1, No 3, January 2012.
[16] Farid H Wang W, "Exposing Digital Forgeries in Interlaced and De-Interlaced Video," EEE Transactions on Information Forensics and Security, pp , 2007.
[17] Farid H Wang W, "Exposing digital forgeries in video by detecting duplication," ACM Multimedia and security workshop, 2007.

64. [18] M.T.Coimbra and M. Davies, "Approximating optical flow within the MPEG-2 Compressed Domain," IEEE Trans. Circuit Syst. Video Tech., vol. 15, no. 1, pp , Jan 2005.
[19] Joseph Gilvarry, "Extraction of Motion Vectors from an MPEG Stream," School of Electronic Engineering, Dublin City University, 1999.
[20] MPEG2Event Class Documentation. [Online].
[21] mpegdemux. [Online].
[22] Information technology – Generic coding of moving pictures and associated audio information, ITU-T, Standard Recommendation ITU-T H.222.0, 2012.
[23] Video coding for low bit rate communication , ITU-T, Standard ITU-T Recommendation H.263, 2005.
[24] SERIES H: AUDIOVISUAL AND MULTIMEDIA SYSTEMS - Advanced video coding for generic audiovisual services," ITU-T, Recommendation ITU-T H.264, 2013.

65. [25] SalientStills VFPro 4. [Online]. http://www. salientstills
[26] Tektronix. [Online].
[27] Elecard Streameye Studio. [Online].
[28] MediaInfo. [Online].
Sliq Media’s WMSnoop. [Online].
[29] MPEG-2 Transport Stream packet analyzer. [Online].
[30] StreamXpert TS Analysis Software. [Online].
[31] Libmpeg2. [Online].
[32] FFMpeg. [Online].
[33] MPEG2Event. [Online].
[34] ooMPEG. [Online].
[35] MPEG-1/2 DirectShow Decoder Filter. [Online].