Video Analysis Tool Box for Digital Video Forensics By Susinda Perera Department of Computer Science and Engineering, University of Moratuwa, Supervised by Dr. Chathura De Silva PhD (NUS-Singapore), MEng (NTU-Singapore), BSc Eng.(Hons) (Moratuwa) Senior Lecturer 02-02-2014

Agenda Project Intro Implementation Demo Video Stream Analyzer Video Stabilizer Compressed domain video analysis Demo

Problem Statement Can we trust digital videos? Are they real, computer generated or tampered Extract some wanted Information from video Difficult due to unclearness of video

Can we trust digital videos? Figure 1‑1 : A still from controversial video aired on Channel 4 Source – YouTube http://www.youtube.com/watch?v=rjVxyG_7mTk

Can we trust digital videos? Figure 1‑2 : A still from eagle catching kid Source – http://www.cbc.ca/news/canada/montreal/child-snatching-eagle-video-created-as-student-project-1.1268125

Unclear Videos

Solution Video Stream analysis tool Video Enhancement tool To ensure the authenticity and integrity Video Enhancement tool To improve the visual quality

Market Analysis Video Stream Analyzing Tools Elecard StreamEye Tektronix MPEG Software Tools MPEG-2 Transport Stream packet analyser TSReader MTS4EA Elementary Stream Analyzer

Screenshots from Elecard StreamEye

Some Features Of Video Stream Analyzing Tools Navigation and display of media stream picture-by-picture (I, P, B). Display of the current frame. Display of the time, type, size and number of a current frame in a stream, decoding order and offset from the file beginning. Display of the bit rate (declared in the sequence header) and a calculated bit rate. Display of detailed information about macroblocks in MPEG-1 (ISO/IEC 11172-2), MPEG 2 (ISO/IEC 13818-2), MPEG-4 (ISO/IEC 14496-2) and AVC/H.264 (ISO/IEC 14496-10) video streams. Information about motion vectors Frame-accurate positioning. Display of the stream and gathering of statistics relating to the entire file.

What is missing in existing tools All high end tools are commercial ones, Expensive Almost all tools are designed for DVB purposes Lack of detailed analyzing features for video forensic Lack of textual representation

Video Forecsic /Enhancement Software Cognitech Ocean Systems dTective Salient Stills VideoFOCUS StarWitness Avid Technology, Inc. Intergraph Video Analyst TREC, Inc. Forevid MotionDSP Ikena Amped FIVE Kinesense

Features Of Video Enhancement Tools Video Stabilization Denoising Deblur Filters Detection Filters Enhancement Histogram Editor Segmentation Tracking Transform Zoom Velocity Reconstruction

Implementation of Video Stream Analyzer

Literature Review Market Analysis Possible resources [31- 35] Most of tools are for DVB, not for forensic purposes Lack of textual output Possible resources [31- 35] Video Player libraries / source codes Decoder Libraries/Source Codes MPEG2Event[33]is a possible candidate

Video Stream Analyzer Implementation Based on MPEG2Event[33] library Mpeg demuxer ffmpeg for stream information A simple thread sync model to get motion vectors Quartztype library for video player implementation C# inbuilt xml libraries for xml manipulation

What is MPEG stream MPEG Bit-Stream Structure (Source [1] )

GOP Structure and Display order GOP Structure and Transmission Order. (Source: [2])

Video Stream Analyzer https://code.google.com/p/mymscproject/source/browse/trunk/Experiments/Mpeg2Event/Downloads/VideoAnalyzerSetup.msi?r=174

Video Stream Analyzer Features Free and open source Textual output (xml) of analysis reports For further analysis All most all the features in slide 10 Motion vector display Frame navigation Bit stream index/ Display index view User data display Motion vector summary – xml output Full video summary – xml output

Video Stream Analyzer -outputs

Motion Vectors Summary <Pictures> <MPicture> <PicType>I</PicType> <MBlocks> <MVector> <Row>0</Row> <Col>0</Col> <ForwardX>0</ForwardX> <ForwardY>0</ForwardY> <BackwardX>0</BackwardX> <BackwardY>0</BackwardY> <IsForwardPresent>false</IsForwardPresent> <IsbackwardPresent>false</IsbackwardPresent> <IsSkipped>true</IsSkipped> </MVector> <Col>1</Col> <Col>2</Col>

MacroBlocks Summary

Implementation of Video Enhancement tool

Video Enhancement Tool Objective Enhance the visual quality Video Stabilization Noise removal Color Correction

Video Stabilization Removing annoying shaky motion from videos helpful in identifying people, number plates, etc. from low-quality video cameras Three aspects Inter frame motion estimation Motion smoothing and compensation Filling up the missing image areas. Main references [3], [4] , [5]

Video Stabilization Inter Frame Motion Estimation Result of video stabilization. Top row: Original input sequence, middle row: stabilized sequence which still has missing image areas, and bottom row: stabilized and completed sequence. The grid is overlaid for better visualization. (Source [3])

Handling camera motion a) Natural scene, b) when camera is moved, c) when image is shifted to compensate the motion.

Motion Model 𝑓 𝑥, 𝑦, 𝑡 = frame at time t 𝑇 = affine transform 𝑓 𝑥, 𝑦, 𝑡 =𝑓 𝑚 1 𝑥+ 𝑚 2 𝑦+ 𝑚 5 , 𝑚 3 𝑥+ 𝑚 4 𝑦+ 𝑚 6 , 𝑡−1 𝑓 𝑥, 𝑦, 𝑡 = T * 𝑓 𝑥, 𝑦, 𝑡 −1 How to find T ? There papers ([3], [4], [5]) mentioned above use diffèrent mechanisms to find the transform matrix parametrs 𝑇= 𝑚 1 𝑚 2 𝑚 3 𝑚 4 𝑚 5 𝑚 6

Motion Estimation Computing inter frame motion Use of object recognition Scale Invariant Feature Transform(SIFT) features Minimizing quadratic error function with a proposed model [5] 𝐸 𝑚 = 𝑅𝑂𝐼 𝑓 𝑥, 𝑦, 𝑡 −𝑓 𝑚 1 𝑥+ 𝑚 2 𝑦+ 𝑚 5 , 𝑚 3 𝑥+ 𝑚 4 𝑦+ 𝑚 6 , 𝑡−1 2 𝑅𝑂𝐼 𝑓 𝑥, 𝑦, 𝑡 −𝑓 𝑚 1 𝑥+ 𝑚 2 𝑦+ 𝑚 5 , 𝑚 3 𝑥+ 𝑚 4 𝑦+ 𝑚 6 , 𝑡−1 2

Stabilization

Motion Smoothing A stabilized motion path is obtained by removing undesired motion fluctuation. Assumed that the intentional motion in the video is usually slow and smooth Uses Gaussian kernel in most literatures Applies Gaussian kernel to neighboring N frames Gaussian kernel + curve fitting methods

Motion Smoothing Let Nt = {j|t-k<=j<=t+k} be the neighboring frames And It is the frame at the origin Calculate the position of each neighboring frame Is, relative to frame It using transform matrixes defined above ( lets say Tst) Find the correcting transformation S from the original frame It to the motion-compensated frame I’t according to Where G is a Gaussian kernel of size k

Motion Smoothing The global transformation chain T defined over the original video frames Ii, and the transformation from the original path to the smoothed path S. (Source [3])

Filling up missing image areas Not addressed here Due to time limitations and complexity Some techniques used in research literature Motion Inpainting [3] the local motion data in the known image areas is propagated into the missing image areas. The propagation starts at pixels on the boundary of the missing image area. Using motion values of neighboring known pixels Use of dynamic programming [3],[6]

Noise Removal (Denoising Filters) Denoising Filter Categories [7] Nonmotion compensated spatiotemporal Motion compensated spatiotemporal Nonmotion compensated temporal Motion compensated temporal filters What is first? motion compensation or denoising? Each have their procs and cons Depends on the video References [7], [8], [9], [10], [11]

Implementation of Video Enhancement Tool

Implementation of Video Enhancement Tool Video Stabilization Based on the algorithm described in [5] + Improvements to reduce error propagation Makes a additional pass (slow) + Improvements to support color video Noise Removal and color Correction Based on openCV filters

A Compressed Domain Approach for Video Tamper Detection

Common Modes of Tampering Classification Spatial, Temporal or Both (Spatio-temporal) Temporal Frame insertion, deletion, modification and reordering Spatial Similar, but operates on objects within a frame Based on MPEG's layer structure ([12]) GOP level, frame level and single- or multi-block level GOP jittering, inter-GOP frame and intra-GOP frame jittering, or their combinations

Tamper detection techniques Video Authentication Pasive Pixel Domain Pixel Statistics Based Detecting resampling Double MPEG compresion Inconsistant Noise patterns Duplication detection Physics Based Motion Analysis Shadow Analysis Compressed Domain Active Watermarking Digital Signature References – [12], [13], [14], [15], [16], [17]

Compressed Domain approaches Many researches can be found on detecting alterations by image processing means. But very few initiatives in Compressed Domain Due to the lack of techniques to grab the frame data from compressed video Possibilities Analyze stream structure headers w.r.t. its contents Analyze userdata Analyze inconsistencies in frame sizes, block sizes Analyze motion vectors

Our Approach Based on motion vectors Assumption For an unaltered video the motions present in the video need to be smooth and therefore should not contain sudden changes Tampering disturbs the motion flow in a video Extract the motion information from video and compare them with the above assumption. Motion information is derived from the motion vectors [17], [18]

Concept of Motion Vector MPEG achieves it its high compression rate by the use of motion estimation and compensation Instead of encoding each block in the current frame, matching block is search in the past frame and if suitable block is found, the difference between the two macroblocks is encoded and transmitted The displacement between the two macroblocks is represented using motion vector

Concept Of Motion Vector cont.. Suppose that the macroblock “x” is the macroblock we wish to encode and macroblock “y” is the counterpart in the past frame. A search is done around “y” (search area) to find the best match for “x”. The displacement between the two macroblocks gives the motion vector associated with “x”.

Motion Vectors vs MacroBlocks As a part of MPEG encoding process motion vectors are calculated and inserted to the bitstream. In mpeg bitstream motion vectors are associated with macroblocks. But not every macroblock will contain a motion vector There are four basic types of macroblock types Intra Skipped Forward predicted Backward predicted

Forward and Backward predictions Picture Type Possible Macroblock Types Motion Vectors I Intra No motion vectors present P Skipped, Forward predicted Only forward motion vectors may present B Skipped, Forward predicted, Backward predicted Both Forward and backward motion vectors may present Summary of motion vectors and macroblocks

Deriving Motion Field from Motion Vectors Motion vectors represent the motions ,But not directly (therefore we need to derive) Steps Reorder the bitstream into display order A simple algorithm Generate Motion felid from motion vectors [18] defines set of rules. Simplified mechanism and rule set is described in the [19]. Filter out the outliers Median filter (3*3 as in [18])

Simplified rule set for deriving motion field Macroblocks with no motion vector have the same movement as in the previous image. This is required since there is a small percentage of intra-coded blocks in B- and P-frames that would otherwise have unknown motion. Also, I-frames have no motion at all so this rule will provide a smooth motion field for them. The justification for this is step is that consecutive images are strongly correlated therefore there is a high probability that the motion field has not changed much. When a macroblock has two motion vectors, the one pointing back is reversed and added to the one pointing forward. This step accomplishes both noise reduction (due to the averaging effect) and normalizes vector magnitude (so they span three images in B-frames which is what happens in P-frames). Motion vector magnitude is normalized. If a macroblock in a B-frame only has one motion vector then it has to be multiplied by a factor so its magnitude corresponds to a vector spanning three images. Skipped macroblocks in I-frames have no movement, while in P-frames, they have movement similar to the previous block. This is a rule obtained directly from the MPEG-2 standard.

Implementation of rules Let’s consider a transition from frame F1 to F2, Lets define mbF1 and mbF2 be the corresponding macroblocks in F1 and F2. Transition Motion Vectors Present I to B P to B P to P   motion = mbF2.FwdMV B to B B to P If only forward vectors present: motion = mbF2.FwdMV – mbF1.FwdMV If only backward vectors present: motion = mbF2.BwdMV –mbF1.BwdMV If both forward and backward vectors are present: motion = ((mbF2.FwdMV – mbF1.FwdMV) + (mbF2.BwdMV –mbF1.BwdMV) )/ 2 P to I Not defined

Motion field The derived motion field is considered as the motion for the frame to frame transition. Now the derived motion field contains the motion information for every macroblock Motion vectors of the left image

Derived Motion field for the above motion vector field and next frame

Smoothed motion field of the above

Detecting Motion Inconsistencies If we consider a typical video, most of pixel data in two adjacent frames are same. Differences are present due to moving objects of the scene, but these differences are small in magnitude. Similar assumption is made in [13] In simple terms, for three adjacent frames, motion between second and third frame is very much similar to motion between first and second frames

Mathematical definition of the assumption Consider three frames 𝑓 𝑡 , 𝑓 𝑡+1 𝑎𝑛𝑑 𝑓 𝑡+2 , and a block in row j and column i in frame 𝑓 𝑡 is denoted by 𝑏 (𝑖,𝑗,𝑡) . for every block in these frames the following inequalities should hold. 𝐸𝑥= 𝑚𝑜𝑡𝑖𝑜𝑛𝑋 𝑏 𝑖,𝑗,𝑡 , 𝑏 𝑖,𝑗, 𝑡+1 − 𝑚𝑜𝑡𝑖𝑜𝑛𝑋 𝑏 𝑖,𝑗,𝑡+1 , 𝑏 𝑖,𝑗, 𝑡+2 < 𝜀 𝐸𝑦= 𝑚𝑜𝑡𝑖𝑜𝑛𝑌 𝑏 𝑖,𝑗,𝑡 , 𝑏 𝑖,𝑗, 𝑡+1 − 𝑚𝑜𝑡𝑖𝑜𝑛𝑌 𝑏 𝑖,𝑗,𝑡+1 , 𝑏 𝑖,𝑗, 𝑡+2 < 𝜀 Where 𝑚𝑜𝑡𝑖𝑜𝑛𝑋 𝑎𝑛𝑑 𝑚𝑜𝑡𝑖𝑜𝑛𝑌 would be the functions which output the motion in X and Y directions respectively and 𝜀 is a very small positive constant. Here the 𝑚𝑜𝑡𝑖𝑜𝑛𝑋 𝑎𝑛𝑑 𝑚𝑜𝑡𝑖𝑜𝑛𝑌 functions can be simply replaced by motion field since the motion field we derived has the exact meaning. 𝐸𝑥= 𝑚𝑜𝑡𝑖𝑜𝑛𝐹𝑒𝑖𝑙𝑑𝑋 𝑖, 𝑗, 𝑡 − 𝑚𝑜𝑡𝑖𝑜𝑛𝐹𝑒𝑖𝑙𝑑𝑋 𝑖, 𝑗, 𝑡+1 < 𝜀 𝐸𝑦= 𝑚𝑜𝑡𝑖𝑜𝑛𝐹𝑒𝑖𝑙𝑑𝑌 𝑖, 𝑗, 𝑡 − 𝑚𝑜𝑡𝑖𝑜𝑛𝐹𝑒𝑖𝑙𝑑𝑌 𝑖, 𝑗, 𝑡+1 < 𝜀

How to detect tampering Get motion vectors for every frame Derive motion fields for each transition Check whether motion fields satisfies the above two equations, for every transition (i.e every motion field with its next) Based on our assumption if it does not satisfy above it is more likely to be a manipulated video.

Results For low resolution videos the assumption we made fails 90% of the time(success for large 𝜀) It is observed that derived motion field itself has discontinuities and noise. For high resolution videos assumption satisfies but fails when there are multiple flows of motion is there. It is difficult to fully automate the tamper detection process, need to have manual intervention.

Summary Video Stream Analysing tool Video enhancement tool Existing tools are for DVB and not for forensic purposes Implemented using open stack of tools and libraries Supports detailed view of bit-stream as well as textual outputs fro further processing Video enhancement tool Two major issues in video footprints are unclearness and shakes This tool helps to stabilize a video and reduce noise Order of stabilize and denoise, has their own pros and cons, and also may depends on video Compressed domain approach for tamper detection Deriving motion fields from motion vectors Rule set Consistency in motion Assumption and mathematical derivation Detecting motion inconsistencies Showed that video bit-stream data (compressed domain) gathered by stream analyzing tool can be use as an aid for video forensic analysis

Future Work Video Stabilization algorithm can be improved to Fill the missing image areas. Video stream analyzer need to support new formats of video. Compressed domain tamper detection mechanisms need to be provided as tools and can be integrate with stream analyzer tool. Motion object segmentation methods can be used in combination with proposed motion field method detect motions of individual objects and then apply our assumption which would give better results since we apply our assumption on individual motion fields

References [1] MPEG-2 White Paper. [Online]. http://www.pinnaclesys.com/files/MainPage/Professional/TopTabItems/products/dc1000/WhitePapers/DC1000-DVD1000MPEG2whitepaper.pdf [2] MPEG Encoding basics, 1st ed.: Snell & Wilcox , 2007 [Online] http://www.media-matters.net/docs/resources/Digital%20Files/MPEG/MPEG%20Encoding%20Basics.pdf [3] Rongjie Shi, I-fan Shen, Wenbin Chen Rong Hu, "Video Stabilization Using Scale-Invariant Features," in Proc. of the 11th International Conference Information Visualization, 2007, pp. 871-877. [4] Eyal Ofek, Weina Ge, Xiaoou Tang, Heung-Yeung Shum Y. Matsushita, "Full-Frame Video Stabilization with Motion Inpainting," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 28, no. 7, pp. 1150-1163, July 2006. [5] Jeffrey B. Woodward Hany Farid, "Video Stabilization and Enhancement," Department of Computer Science , Dartmouth College , TR2007- 605, 1997.

[6] E. Shechtman, and M. Irani Y [6] E. Shechtman, and M. Irani Y.Wexler, "Space-time video completion," in IEEE Conf. on Computer Vision and Pattern, vol. 1, 2004, pp. 120-127. [7] R. P. Kleihorst, S. Efsratiadis, A. K. Katsaggelos, and R. L. Lagendijk J. C. Brailean, "Noise reduction filters for dynamic image sequences: A review," Proceedings of the IEEE, vol. 83, pp. 1272-1292, 1995. [8] B. Coll, J. M. Morel A. Buades, "Denoising image sequences does not require motion estimation," Dpt. Matematiquesi Informatica, Universitat Illes Balears. [9] William T. Freeman 2 Ce Liu 1, "A High-Quality Video Denoising Algorithm based on Motion Estimation," 1Microsoft Research New England, 2 Massachusetts Institute of Technology [10] M.I.Sezan, A.M. Tekalp M.K. Ozkan, "Adaptive motion compensated filtering of noisy image sequences," IEEE Trans, circuits, vol. CSVT-3, pp. 277-294, Aug 1993. [11] B. Coll, J. M. Morel A. Buades, "A non-local algorithm for image denoising," IEEE International Conference on Computer Vision and Pattern Recognition, 2005

[12] Hong Heather Yu Peng yj, "Classification Of Video Tampering Methods and Countermeasures Using Digital Watermarking," SPIE, Multimedia Systems and Applications, vol. IV, pp. 239-246. [13] Wang Weihong, "Digital Video Forensics," DARTMOUTH COLLEGE, Hanover, New Hampshire, Thesis 2009. [14] Farid H Wang W, "Exposing Digital Forgeries in Video by Detecting Double MPEG Compression," ACM Multimedia and security workshop, 2006. [15] S.K. Singh S. Upadhyay, "Video Authentication: Issues and Challenges," IJCSI International Journal of Computer Science Issues, Vol. 9, Issue 1, No 3, January 2012. [16] Farid H Wang W, "Exposing Digital Forgeries in Interlaced and De-Interlaced Video," EEE Transactions on Information Forensics and Security, pp. 428-449, 2007. [17] Farid H Wang W, "Exposing digital forgeries in video by detecting duplication," ACM Multimedia and security workshop, 2007.

[18] M.T.Coimbra and M. Davies, "Approximating optical flow within the MPEG-2 Compressed Domain," IEEE Trans. Circuit Syst. Video Tech., vol. 15, no. 1, pp. 103-107, Jan 2005. [19] Joseph Gilvarry, "Extraction of Motion Vectors from an MPEG Stream," School of Electronic Engineering, Dublin City University, 1999. [20] MPEG2Event Class Documentation. [Online]. http://www.cs.unc.edu/~kmp/mpeg2event/classdocs/index.html [21] mpegdemux. [Online]. http://www.hampa.ch/mpegdemux/ [22] Information technology – Generic coding of moving pictures and associated audio information, ITU-T, Standard Recommendation ITU-T H.222.0, 2012. [23] Video coding for low bit rate communication , ITU-T, Standard ITU-T Recommendation H.263, 2005. [24] SERIES H: AUDIOVISUAL AND MULTIMEDIA SYSTEMS - Advanced video coding for generic audiovisual services," ITU-T, Recommendation ITU-T H.264, 2013.

[25] SalientStills VFPro 4. [Online]. http://www. salientstills [26] Tektronix. [Online]. http://www.tek.com/mpeg-test-video/mpeg-software [27] Elecard Streameye Studio. [Online]. http://www.elecard.comwww.elecard.com/en/products/professional/analysis/streameye-studio.html [28] MediaInfo. [Online]. http://mediainfo.sourceforge.net/en Sliq Media’s WMSnoop. [Online]. http://www.sliq.com/default.asp?view=wmsnoop [29] MPEG-2 Transport Stream packet analyzer. [Online]. http://www.pjdaniel.org.uk/mpeg/ [30] StreamXpert TS Analysis Software. [Online]. http://www.dektec.com/products/Apps/DTC-320/ [31] Libmpeg2. [Online]. http://libmpeg2.sourceforge.net/ [32] FFMpeg. [Online]. http://ffmpeg.org/ [33] MPEG2Event. [Online]. http://wwwx.cs.unc.edu/~kmp/mpeg2event/blosxom.cgi/overview.html [34] ooMPEG. [Online]. http://www.cs.brown.edu/software/ooMPEG/ [35] MPEG-1/2 DirectShow Decoder Filter. [Online]. http://gplmpgdec.sourceforge.net/