1. Weihong Li, Hao Tang and Zhigang Zhu
Automated Registration of High Resolution Images from Slide Presentation and Whiteboard Handwriting via a Video Camera
Weihong Li, Hao Tang and Zhigang Zhu
Video Computing Lab
Department of Computer Science
City College and Graduate Center
The City University of New York
My name is Weihong Li (in fact you will be introduced by Chairs…) The title of our paper is “……”
(The authors are Weihong Li, Hao Tang, and Prof. Zhigang Zhu)

2. Outline of the Talk Background and Motivation
Virtualized Classroom Basic Components
Media Integration Algorithms
Slide & Video Registration
Video & Whiteboard Registration
Slide & Board Integration (Results)
Discussion and Future Work
12/8/2018
IVR2004

3. Outline of the Talk Background and Motivation
Virtualized Classroom Basic Components
Media Integration Algorithms
Slide & Video Registration
Video & Whiteboard Registration
Slide & Board Integration (Results)
Discussion and Future Work
12/8/2018
IVR2004

4. Background “Record-and-playback” approach
Georgia Tech – Classroom 2000
CMU – Just-In-Time system
UMass – RIPPLES/MANIC
More automated, flexible and interactive systems are needed
The first generation e-learning systems primarily adopt a “record-and-playback” approach.
Including classroom 2000 project conducted in Georgia Tech.
Just-In-Time System conducted in CMU and
Ripples/Manic project conducted in Umass-Amherst.
The new generation system should be more automated, flexible and interactive.
12/8/2018
IVR2004

5. Motivation Rich information in multimedia materials
Need to be captured
Need to be organized
Powerpoint slides combined with whiteboard
A better way to present a lecture
Deliver effective e-lectures
Allow instructor to add handwriting notes on-the- fly
Our project is motivated by the following considerations.
Multimedia materials from classrooms and seminars are rich sources of information.
These materials need to be better organized to provide useful information effectively
2. A PowerPoint (PPT) slide presentation combined with a whiteboard handwriting capture system
can provide a better means to present a lecture and later can deliver more effective e-lectures for
off-campus students.
3. This will allow the instructor to dynamically add handwritten material generated in a different
medium (the whiteboard) onto the projected slides.
12/8/2018
IVR2004

6. Outline of the Talk Background and Motivation
Virtualized Classroom Basic Components
Media Integration Algorithms
Slide & Video Registration
Video & Whiteboard Registration
Slide & Board Integration (Results)
Discussion and Future Work
12/8/2018
IVR2004

7. CCNY Virtualized Classroom
Three Basic Components:
Automated data collection
Intelligent media processing and integration
User-customized presentation interface
Focus of This Paper
Media Synchronization and Integration
1. By using a portable presentation system with very cost-effective multimedia sensors, we have developed some basic components for our Virtualized Classroom project – automated data collection, intelligent media processing and integration algorithms and user-customized presentation interface design.
2. We mainly focus on alignment and integration of images with PPT printing notes and images with whiteboard handwriting contents.
3. Media capturers and synchronization is important for media integration
12/8/2018
IVR2004

8. Used for registration of powerpoint slides with whiteboard handwriting
video V1 V2
PPT slides S
Handwriting pages H
combined
(Why use camera here?)
Used for registration of powerpoint slides with whiteboard handwriting
Record the video and audio materials for later review and delivery

9. Outline of the Talk Background and Motivation
Virtualized Classroom Basic Components
Media Integration Algorithms
Slide & Video Registration
Video & Whiteboard Registration
Slide & Board Integration (Results)
Discussion and Future Work
12/8/2018
IVR2004

10. Outline of the Talk Background and Motivation
Virtualized Classroom Basic Components
Media Integration Algorithms
Slide & Video Registration
Video & Whiteboard Registration
Slide & Board Integration (Results)
Discussion and Future Work
First I will discuss registration of a PPT slide and its video image
12/8/2018
IVR2004

11. Slide and Video Registration
Distortion Removal
Feature Extraction
Image Transformation
There are 3 steps to register a slide and its corresponding video image(s).
First, since video images have radial distortion as well as perspective distortion, we need to preprocess the video frames, that is, to remove the radial distortion.
Second, since the projection area is usually significantly brighter than other areas, we can extract the boundary of the projection area based on the difference of the illumination
Third ???
12/8/2018
IVR2004

12. Slide and Video Registration
Step 1: Radial Distortion Removal
We applied the calibration method proposed by Dr. Zhang at Microsoft Research to remove the radial distortion.
Radial Removal: Zhang, PAMI 2000
12/8/2018
IVR2004

13. Slide and Video Registration
Step 2: Feature Extraction
To detect the boundary, we:
First generate a binary image from the video frame,
Then, use the Laplacian-of-Gaussian operator to obtain the edges.
Finally, we use the Hough transform to extract the four boundary lines and calculate the coordinates of the four corners of the projection area.
12/8/2018
IVR2004

14. Slide and Video Registration
Step 3: Image Transformation
This figure shows the video and powerpoint slide alignment result that uses the projective mapping matrix A1.
The video frame is transformed to the PPT slide coordinates so that the orthogonal view of the PPT slide remains.
Please note: only the fonts , not the background , on the PPT slide are superimposed on the transformed video frame for showing the accuracy of the alignment.
Mapping between slide and video: homography
12/8/2018
IVR2004

15. Outline of the Talk Background and Motivation
Virtualized Classroom Basic Components
Media Integration Algorithms
Slide & Video Registration
Video & Whiteboard Registration
Slide & Board Integration (Results)
Discussion and Future Work
12/8/2018
IVR2004

16. Video and Board Registration a two stage algorithm
(1) Initial Matching
handwriting extraction
Translation via centroid
Scaling via polar profiles
(2) Fine Registration
connected components
content-based matching
Projection transformation
Since the camera cannot “see” the invisible frame boundary of the whiteboard page, the matching of the whiteboard handwriting page and video frame is more challenging.
We propose to match the handwriting contents from the Mimio pages and video frames.
12/8/2018
IVR2004

17. Video and Board Registration a two stage algorithm
(1) Initial Matching
handwriting extraction
Translation via centroid
Scaling via polar profiles
(2) Fine Registration
connected components
content-based matching
Projection transformation
Since the camera cannot “see” the invisible frame boundary of the whiteboard page, the matching of the whiteboard handwriting page and video frame is more challenging.
We propose to match the handwriting contents from the Mimio pages and video frames.
12/8/2018
IVR2004

18. Video and Board Registration (1) Handwriting Extraction
To obtain the handwriting contents from video frames, we subtract the video frame with PPT projection only from video frame with both PPT projection and Mimio handwriting
??? Shouldn’t we do the radial distortion removal first? (Yes)
Then, we use the video slide mapping matrix A1 to transform the handwriting image to rectified image
12/8/2018
IVR2004

19. Video and Board Registration (1) Translation and Scaling
Translation from centroids
To initial matching the two image, we need know the translation and scale. We can ignore the rotation here since both of them are almost orthogonal view.
First, we calculate the centroids for both of the image, which is indicated by the red dot, the difference of the two centroids will give us the translation of this two images
To determine the scale factor, we generate a centroid-centered polar profile fro each image by measuring the distances of all the outmost stroke points in all directions from each centroid.
The scale factor will be given by the ratio of the areas under these two polar profiles.
Scale from polar profiles
12/8/2018
IVR2004

20. Video and Board Registration (1) Initial Match Result
After image translation and scaling, the two image are roughly aligned.
The red image is from mimio, and black one is from video difference image. They are almost matched.
Please note:
Though the video difference image didn’t capture the “using mimio” written in green color,The two images
Can still get a good match, that shows that our method is robust.
(What if Using Mimio was written at the place of 1st Step! ???)
12/8/2018
IVR2004

21. Video and Board Registration a two stage algorithm
(1) Initial Matching
handwriting extraction
Translation via centroid
Scaling via polar profiles
(2) Fine Registration
connected components
content-based matching
Projection transformation
Since the camera cannot “see” the invisible frame boundary of the whiteboard page, the matching of the whiteboard handwriting page and video frame is more challenging.
We propose to match the handwriting contents from the Mimio pages and video frames.
12/8/2018
IVR2004

22. Video and Board Registration (2) Connected Component (CC) Extraction
We extract the the connected components from Mimio image, which are indicated by the rectangle boxes.
12/8/2018
IVR2004

23. Video and Board Registration (2) Content-based matching
Select each large CC in Mimio as template
Match pixel pattern under the bounding box in video.
Transformation:
First, we select each of those connected components in Mimio pages whose size is sufficiently large for a robust match.
Then, we use the pixel pattern under the bounding box of the each selected primitive as the matching template to search for the best matched rectangle region in the video handwriting image using the normalized cross-correlation measures. (Weihong- double check what were the templates you used)
Finally, we choose those matches whose maximum normalized correlation values are above a threshold to calculate the final projective transformation.
12/8/2018
IVR2004

24. Outline of the Talk Background and Motivation
Virtualized Classroom Basic Components
Media Integration Algorithms
Slide & Video Registration
Video & Whiteboard Registration
Slide & Board Integration (Results)
Discussion and Future Work
12/8/2018
IVR2004

25. Slide-Board Integration (Ex 1) via video images
We can see that the result of registration is very good.
The two circle are perfected registered from mimio image to Powerpoint slide.
There is only about one-pixel misalignment can be found
12/8/2018
IVR2004

26. Slide-Board Integration (Ex 2) Device may move
The devices may move.
12/8/2018
IVR2004

27. Slide-Board Integration (Ex 1) Keystone Distortion
Keystone can be found:
The sensor is vertically attached to the wihteboard is not shown vertically.
The rectified Mimio whiteboard page has perspective distortion
12/8/2018
IVR2004

28. Outline of the Talk Background and Motivation
Virtualized Classroom Basic Components
Media Integration Algorithms
Slide & Video Registration
Video & Whiteboard Registration
Slide & Board Integration (Results)
Discussion and Future Work
12/8/2018
IVR2004

29. Discussion and Future Work
When should the calibration be conducted
At the beginning
When devices are moved – how to detect movements?
Usability study
CCNY
UMass-Amherst
Registering slides and instructor images
Immersive presentation?
When should we do the calibration?
At the beginning, the instructor can draw four sufficiently separated marks as the calibration step.
The camera will monitor if there are any changes of locations of the projection area, the capture bar or the camera itself.
We realize it is important to have a usability study of our Virtualized Classroom system in real classroom use. We are actively pushing this at both CCNY and UMass-Amherst
The position and gestures of the instructor are a very effective way to attract the attention of students and help them to recall what they have learned in the class.
12/8/2018
IVR2004