Seamless Video Stitching from Hand-held Camera Inputs Kaimo Lin, Shuaicheng Liu, Loong-Fah Cheong, Bing Zeng National University of Singapore University of Electronic Science and Technology of China

Seamless Video Stitching from Hand-held Camera Inputs 2 Motivation Create video content with a wide field of view Avoid expensive wide-angle lens and camera rigs Use consumer-grade cameras and allow free movement Collaborative video recording for social activity Coadjutant UAV capturing or multiple robots co-vision tasks

Seamless Video Stitching from Hand-held Camera Inputs 3 Hardware-based Solutions Panocast ( Point Gray Ladybug5 ( FullView (

Seamless Video Stitching from Hand-held Camera Inputs 4 Other Post-processing Solutions Perazzi et al. - “Panoramic video from unstructured camera arrays” [1]  Unstructured camera arrays on a fixed support  Per-pixel 2D image alignment [1] PERAZZI F., ALEXANDER S.-H., ZIMMER H., KAUFMANN P., WANG O., WATSON S., GROSS M.: Panoramic video from unstructured camera arrays. Comput. Graph. Forum (Proc. of Eurographics 2015) 32, 2 (2015). [2] JIANG W., GU J.: Video stitching with spatial-temporal contentpreserving warping. In CVPR Workshops (2015). Jiang and Gu - “Video stitching with spatial-temporal content-preserving warping” [2]  Relative positions of the cameras are fixed  Spatial-temporal image warping

Seamless Video Stitching from Hand-held Camera Inputs 5 Difficulties of Freely-moving Camera Inputs Camera frame stitching with parallax  Images with large parallax  Ghosting effects, misalignment, distortions Temporal smoothness of the stitched frames  Feature selection  Seam estimation

Seamless Video Stitching from Hand-held Camera Inputs 6 Pipeline of Our Framework Input video frames 3D Reconstruct & Virtual path Line-preserving Video Warp Final composition

Seamless Video Stitching from Hand-held Camera Inputs 7 3D Reconstruction Video Frame Calibration [3] D. Zou and P. Tan. Coslam: Collaborative visual slam in dynamic environments. IEEE Trans. Pattern Anal. Mach. Intell., 35(2),  Video Capture Settings Each person hold one camera and can move freely Focal lengths are fixed and the intrinsic parameters are pre-calibrated  Video Synchronization Use a manually triggered flash  Camera Motion Estimation by CoSLAM System[3] A SLAM system in dynamic environments with multiple cameras

Seamless Video Stitching from Hand-held Camera Inputs 8 3D Reconstruction Dense Reconstruction  Multi-view Stereo [4] [4] C. Rhemann, A. Hosni, M. Bleyer, C. Rother, and M. Gelautz. Fast cost-volume filtering for visual correspondence and beyond. In Proc. CVPR, Virtual Path Generation  Centers and rotations  Smoothing

Seamless Video Stitching from Hand-held Camera Inputs 9 Line-preserving Video Warp (LPVW) Frame Warp Energy Function Feature Term Line TermEpipolar TermCoherence Term frame t-1 frame t frame t+1

Seamless Video Stitching from Hand-held Camera Inputs 10 Line-preserving Video Warp (LPVW) Feature Term

Seamless Video Stitching from Hand-held Camera Inputs 11 Line-preserving Video Warp (LPVW) Line Term

Seamless Video Stitching from Hand-held Camera Inputs 12 Line-preserving Video Warp (LPVW) Epipolar Term

Seamless Video Stitching from Hand-held Camera Inputs 13 Line-preserving Video Warp (LPVW) Coherence Term

Seamless Video Stitching from Hand-held Camera Inputs 14 Line-preserving Video Warp (LPVW) Coherence Term frame t-1 frame tframe t+1 Meshes in previous iteration Mesh “t” to be optimized in current iteration

Seamless Video Stitching from Hand-held Camera Inputs 15 Line-preserving Video Warp (LPVW) Optimization  Energy function is quadratic  Sequentially optimize the frame meshes  One or two passes is usually sufficient

Seamless Video Stitching from Hand-held Camera Inputs 16 Final Composition Spatial-temporal Seam Estimation [5]  Find spatial-temporal consistent stitching seams across multiple frames  Performed inside the overlapping boundary regions  Restrict potential artifacts to overlapping boundary regions [5] JIANG W., Gu J.: Video Stitching with spatial-temporal content-preserving warping. In CVPR Workshops (2015). Multiband Blending

Seamless Video Stitching from Hand-held Camera Inputs 17 Discussion Overlaying vs. Blending Multi-band blendingUse all features

Seamless Video Stitching from Hand-held Camera Inputs 18 Discussion Overlaying vs. Blending Linear blendingOverlaying

Seamless Video Stitching from Hand-held Camera Inputs 19 Discussion Dense vs. Sparse Sparse features Dense features

Seamless Video Stitching from Hand-held Camera Inputs 20 Experiments The role of each component in our LPVW Comparison with other image stitching methods Video stitching results of challenging scenarios Failure cases

Seamless Video Stitching from Hand-held Camera Inputs 21 Experiments Evaluation of LPVW (Line Term) Without line term With line term

Seamless Video Stitching from Hand-held Camera Inputs 22 Experiments Evaluation of LPVW (Epipolar Term) Without epipolar term With epipolar term

Seamless Video Stitching from Hand-held Camera Inputs 23 Experiments Comparison with other image stitching methods Single homography APAP [5] [5] ZARAGOZA J., CHIN T.-J., BROWN M. S., SUTER D.: Asprojective-as-possible image stitching with moving dlt. In Proc. CVPR(2013), pp. 2339–2346.

Seamless Video Stitching from Hand-held Camera Inputs 24 Experiments Comparison with other image stitching methods SPHP [5]Our warp [5] CHANG C.-H., SATO Y., CHUANG Y.-Y.: Shape-preserving half-projective warps for image stitching. In Proc. CVPR (2014), pp. 3254–3261.

Seamless Video Stitching from Hand-held Camera Inputs 25 Experiments Failure cases Bad 3D reconstructionSevere depth discontinuity

Seamless Video Stitching from Hand-held Camera Inputs 26 Conclusion A video stitching framework for freely moving camera inputs Solve video stitching and stabilization in a unified framework A novel Line-preserving Video Warp (LPVW) Handle a variety of challenging scenarios

Seamless Video Stitching from Hand-held Camera Inputs 27 Q & A