Download presentation
Presentation is loading. Please wait.
Published byTyler Burns Modified over 9 years ago
1
© Copyright National University of Singapore. All Rights Reserved. AUTOMATED LINK GENERATION FOR SENSOR- ENRICHED SMARTPHONE IMAGES IBM Interview talk, Sept 29 th 2015, Singapore Presenter: Seshadri Padmanabha Venkatagiri Advisor: Prof. Chan Mun Choon Collaborating Advisor: Prof. Ooi Wei Tsang
2
© Copyright National University of Singapore. All Rights Reserved. MOBILE USER GENERATED CONTENT: PHOTO SLIDE: 2 More than 1.8 Billion photo uploads in 2014 till date Source: Kleiner Perkins Caufield & Byers, Internet Trends 2014
3
© Copyright National University of Singapore. All Rights Reserved. ADHOC EVENTS: IMPORTANT SOURCE OF MOBILE UGC SLIDE: 3 Exhibitions Street Performances Mishaps Social events Well attended Attendees share same event context Lack of prior information. Eg. Training data Lack of planned infrastructure. Eg. GPS, camera deployments etc.
4
© Copyright National University of Singapore. All Rights Reserved. OBJECTIVE OF AUTOLINK SLIDE: 4 To organize noisy unstructured photo collections to improve user interaction
5
© Copyright National University of Singapore. All Rights Reserved. APPLICATIONS OF STRUCTURED PHOTO COLLECTIONS SLIDE: 5 Content Analytics/ Discovery Crowd-sourced surveillance Photograph this! Scene recommendation
6
© Copyright National University of Singapore. All Rights Reserved. CHALLENGE 1: NOISE IN CONTENT SLIDE: 6 occlusion Varying lighting conditions Diverse views Diverse regions of interest Diversity in moments captured Scene-localization issues Redundancy
7
© Copyright National University of Singapore. All Rights Reserved. CHALLENGE 2: RESOURCE BOTTLENECK SLIDE: 7 Resource Bottleneck UGC from Ad-hoc events Exhibitions Street Performances Mishaps Battery Bandwidth Applications Photograph this ! Content Analytics/ Discovery Scene recommendation Surveillance
8
© Copyright National University of Singapore. All Rights Reserved. OUR APPROACH: DETAIL-ON-DEMAND PARADIGM SLIDE: 8 Does not overwhelm users with lots of content. Suited for small-factor devices. Retrieves specific content, on-demand. Consumes less bandwidth and power Organizing content by providing progressively detailed content, helps analytics and elicits user interest
9
© Copyright National University of Singapore. All Rights Reserved. DETAIL-ON-DEMAND PARADIGM: EXISTING SOLUTIONS SLIDE: 9 Hyperlinking Provides progressive content Could be adapted to link content from multiple sources Zooming Provides progressively detailed content Not suitable for relating content from multiple sources WE CHOOSE HYPERLINK BASED DETAIL-ON-DEMAND TO PROVIDE PROGRESSIVELY DETAILED CONTENT
10
© Copyright National University of Singapore. All Rights Reserved. COMPARISON WITH STATE-OF-THE-ART SLIDE: 10 TechniqueAutoma ted Prior information (Eg. Training, visual dictionary) Camera Calibration Distinguishes “same” from “similar” scenes Requires GPS Hyper-HitchcockNo YesNo Photo Tourism (Photo Synth) Google Street View YesNoYesNoYes ImageWebsYes No Geo-taggingYesNo Yes AutoLinkYesNo YesNo Manual technique
11
© Copyright National University of Singapore. All Rights Reserved. TechniqueAutoma ted Prior information (Eg. Training, visual dictionary) Camera Calibration Distinguishes “same” from “similar” scenes Requires GPS Hyper-HitchcockNo YesNo Photo Tourism (Photo Synth) Google Street View YesNoYesNoYes ImageWebsYes No Geo-taggingYesNo Yes AutoLinkYesNo YesNo COMPARISON WITH STATE-OF-THE-ART SLIDE: 11 GPS Error/Non-availability, Camera calibration difficult in ad-hoc events
12
© Copyright National University of Singapore. All Rights Reserved. COMPARISON WITH STATE-OF-THE-ART SLIDE: 12 TechniqueAutoma ted Prior information (Eg. Training, visual dictionary) Camera Calibration Distinguishes “same” from “similar” scenes Requires GPS Hyper-HitchcockNo YesNo Photo Tourism (Photo Synth) Google Street View YesNoYesNoYes ImageWebsYes No Geo-taggingYesNo Yes AutoLinkYesNo YesNo Visual vocabulary not always available in ad-hoc events
13
© Copyright National University of Singapore. All Rights Reserved. COMPARISON WITH STATE-OF-THE-ART SLIDE: 13 TechniqueAutoma ted Prior information (Eg. Training, visual dictionary) Camera Calibration Distinguishes “same” from “similar” scenes Specialized equipment (Eg. GPS, laser range finding) Hyper-HitchcockNo YesNo Photo Tourism (Photo Synth) Google street view YesNoYesNoYes ImageWebsYes No Geo-taggingYesNo Yes AutoLinkYesNo YesNo GPS Error/Non-availability
14
© Copyright National University of Singapore. All Rights Reserved. TechniqueAutoma ted Prior information (Eg. Training, visual dictionary) Camera Calibration Distinguishes “same” from “similar” scenes Requires GPS Hyper-HitchcockNo YesNo Photo Tourism (Photo Synth) Google Street View YesNoYesNoYes ImageWebsYes No Geo-taggingYesNo Yes AutoLinkYesNo YesNo COMPARISON WITH STATE-OF-THE-ART SLIDE: 14 Avoids GPS, Visual vocabulary and camera calibration
15
© Copyright National University of Singapore. All Rights Reserved. APPLICATION CONTEXT SLIDE: 15 Multiple scenes Located indoor/outdoor or both People move between scenes and capture photos Capture photos with different orientations, and regions of interest
16
© Copyright National University of Singapore. All Rights Reserved. PROBLEM SLIDE: 16 Image collection Inertial Sensor log Set of scenes user is interested in ? Detail-on-demand Image hierarchy High Context High Detail
17
© Copyright National University of Singapore. All Rights Reserved. ARCHITECTURE SLIDE: 17 AutoLink Server Mobile Wireless Network AutoLink Client 1 2 Photos uploaded on- demand by smart phones Client uploads Metadata in the form of sensor log and content characteristics extracted from photo Servers runs AutoLink and performs inter-scene and intra-scene clustering. Users could request photos by navigating through these links. 2 1 Image content hierarchy High Context High Detail
18
© Copyright National University of Singapore. All Rights Reserved. AUTOLINK OUTLINE SLIDE: 18 Image features (4) Sensor-assisted hybrid scene classification (6) Hierarchy creation (5) Region estimation Photo Scene Region (3) User selected scenes Angle Time Step (2) Sensor log (1) Capture photo AutoLink
19
© Copyright National University of Singapore. All Rights Reserved. STEP 1: RATIONALE OF SCENE CLASSIFICATION SLIDE: 19 Order of reliability: Use content features to find matching scene for an image. If content does not provide a match, use sensors and already-labeled images, to improve the match possibility.
20
© Copyright National University of Singapore. All Rights Reserved. STEP 1: SCENE CLASSIFICATION SLIDE: 20 Content features User specified scenes Match content features Apply Naïve Bayes Nearest Neighbor classification No scene match? Label image with scene Scene match? Remains unlabeled
21
© Copyright National University of Singapore. All Rights Reserved. CONTENT FEATURES SLIDE: 21 Color-SIFT [Abdel2006] Captures scale-invariant and color-invariant characteristics Maximally Stable Extremal Regions [Forssen2007] Describes region level features ORB [Ethan2011] Rotation and noise resistant features Color Global color histogram description [Ethan2011] Ethan Rublee, Vincent Rabaud, Kurt Konolige, and Gary Bradski. ORB: An efficient alternative to SIFT or SURF. ICCV 2011 [Abdel2006] A. Abdel-Hakim and A. Farag, CSIFT: A SIFT descriptor with color invariant characteristics. CVPR 2006 [Forssen2007] P.-E. Forssen, Maximally stable colour regions for recognition and matching. CVPR 2007
22
© Copyright National University of Singapore. All Rights Reserved. NAÏVE BAYES NEAREST NEIGHBOUR SLIDE: 22 1.Compute descriptors d 1, d 2, …, d n of the photo 2.Compute descriptors d 1, d 2, …, d n for all the scenes (Preprocessing step, done only once) 3.For each, d i and for each scenes, compute the nearest neighbour of d i : NN(di). 4.Choose matching scene with ARG MIN { SUM([d i – NN(d i )] 2 ) }
23
© Copyright National University of Singapore. All Rights Reserved. STEP 1: SCENE CLASSIFICATION SLIDE: 23 Content (interest points) User specified scenes Match content features Apply Naïve Bayes Nearest Neighbor classification No scene match? Label image with scene scene match? Combine content, time, sensor features to find matching scene label Angle Time Step Content Remains unlabeled
24
© Copyright National University of Singapore. All Rights Reserved. TIME FEATURE: TEMPORAL CLUSTERING SLIDE: 24 Inter-photo time gaps Photo-clusters
25
© Copyright National University of Singapore. All Rights Reserved. TIME FEATURE: GAP LABELING SLIDE: 25 t 1 = 0min t 2 = 1.5min t 3 = 2min t 4 = 5min t 5 = 5.1min Time t gap Max{t gap } = 3 min i 1 = Scene 1 i 5 = Scene 2 i 2 = No label i 3 = No label i 4 = No label t 1 = 0min t 2 = 1.5min t 3 = 2min t 4 = 5min t 5 = 5.1min Time t gap Max{t gap } = 3 min i 1 = Scene 1 i 5 = Scene 2 i 2 = Scene 1 i 3 = Scene 1 i 4 = Scene 2 (a) (b) If time-gaps is less than a ΔT, which is a threshold obtained from the traces collected from dataset, then time-gap labeling is not applied because, photo as are too close to distinguish cluster boundaries
26
© Copyright National University of Singapore. All Rights Reserved. ACCELEROMETER FEATURE: STEP COUNT BETWEEN PHOTO CAPTURES SLIDE: 26 Photo capture Steps Accelerometer + Step counting algorithm
27
© Copyright National University of Singapore. All Rights Reserved. PHOTO ORIENTATION SLIDE: 27 Photo orientation is obtained from sensor fusion of magnetic field sensor + Gyroscope + Accelerometer
28
© Copyright National University of Singapore. All Rights Reserved. STEP 2: REGION ESTIMATION SLIDE: 28 Image Scene Window-based estimation Super-pixel based estimation Super-pixel based estimation
29
© Copyright National University of Singapore. All Rights Reserved. STEP 2: REGION ESTIMATION SLIDE: 29 Image Scene Window-based estimation Super-pixel based estimation 1.Apply SEEDS super-pixel to image and scene. Super-pixel based estimation 1.Apply SEEDS super-pixel to image and scene.
30
© Copyright National University of Singapore. All Rights Reserved. STEP 2: REGION ESTIMATION SLIDE: 30 Image Scene Window-based estimation Super-pixel based estimation 1.Apply SEEDS super-pixel to image and scene. 2.Compute super-pixels which have matching features. 3.Estimate bounding box around it. Super-pixel based estimation 1.Apply SEEDS super-pixel to image and scene. 2.Compute super-pixels which have matching features. 3.Estimate bounding box around it.
31
© Copyright National University of Singapore. All Rights Reserved. STEP 2: REGION ESTIMATION SLIDE: 31 Image Scene Window-based estimation 1.Estimate region center using horizontal and vertical shifts in matching features of image and scene 2.Correct estimation using compass sensor z-axis Window-based estimation 1.Estimate region center using horizontal and vertical shifts in matching features of image and scene 2.Correct estimation using compass sensor z-axis Super-pixel based estimation 1.Apply SEEDS super-pixel to image and scene. 2.Compute super-pixels which have matching features. 3.Estimate bounding box around it. Super-pixel based estimation 1.Apply SEEDS super-pixel to image and scene. 2.Compute super-pixels which have matching features. 3.Estimate bounding box around it.
32
© Copyright National University of Singapore. All Rights Reserved. ESTIMATING REGION CENTER SLIDE: 32 Region Center is estimated by computing horizontal and vertical shift in image features between candidate image and a scene reference image. Scene Reference imageCandidate Image
33
© Copyright National University of Singapore. All Rights Reserved. ENERGY-DELAY-ACCURACY PERFORMANCE OF REGION CENTER ESTIMATION SLIDE: 33 Image Resize Factor (%) Delay (seconds) Energy (mJoules) Accuracy (%) 200.0983.675.7 400.31112.182.5 600.89939.391.5 802.558117.493.4 1006.494292.693.5 Subsampling to 60% of the original image size: Only a 2% reduction in accuracy compared to using the original image Power reduction by approximately 86% computation time reduction by 86%
34
© Copyright National University of Singapore. All Rights Reserved. STEP 2: REGION ESTIMATION SLIDE: 34 Image Scene Find region center Super-pixel estimation 1.Iterate through bounding box around the region center 2.Find best match bounding box with the scene. 1.Iterate through bounding box around the region center 2.Find best match bounding box with the scene.
35
© Copyright National University of Singapore. All Rights Reserved. STEP 2: REGION ESTIMATION SLIDE: 35 Image Scene Find region center Super-pixel estimation 1.Iterate through bounding box around the region center 2.Find best match bounding box with the scene. 1.Iterate through bounding box around the region center 2.Find best match bounding box with the scene.
36
© Copyright National University of Singapore. All Rights Reserved. STEP 2: REGION ESTIMATION SLIDE: 36 Image Scene Find region center Super-pixel estimation 1.Iterate through bounding box around the region center 2.Find best match bounding box with the scene. 1.Iterate through bounding box around the region center 2.Find best match bounding box with the scene.
37
© Copyright National University of Singapore. All Rights Reserved. STEP 2: REGION ESTIMATION SLIDE: 37 Image Scene Window-based estimation Super-pixel based estimation Use average of both estimates. Red: Our estimate Blue: Ground truth
38
© Copyright National University of Singapore. All Rights Reserved. DEMO: REGION CENTER AND REGION WINDOW ESTIMATION SLIDE: 38
39
© Copyright National University of Singapore. All Rights Reserved. STEP 3: ADDING TO IMAGE HIERARCHY USING BOUNDING BOX (BB) SLIDE: 39 High Context High Detail Image ? BB L2? L1 L2 L3 L4
40
© Copyright National University of Singapore. All Rights Reserved. STEP 3: ADDING TO IMAGE HIERARCHY USING BOUNDING BOX(BB) SLIDE: 40 High Context High Detail BB L3? L1 L2 L3 L4
41
© Copyright National University of Singapore. All Rights Reserved. STEP 3: ADDING TO IMAGE HIERARCHY SLIDE: 41 High Context High Detail BB L4? L1 L2 L3 L4
42
© Copyright National University of Singapore. All Rights Reserved. STEP 3: ADDING TO IMAGE HIERARCHY USING BOUNDING BOX(BB) SLIDE: 42 High Context High Detail BB matches L4 ! L1 L2 L3 L4
43
© Copyright National University of Singapore. All Rights Reserved. DEMO: AUTOLINK IMAGE HIERARCHY SLIDE: 43
44
© Copyright National University of Singapore. All Rights Reserved. EVALUATION: METHODOLOGY Precision and Recall Running time Bounding box accuracy Ranking accuracy SLIDE: 44 Metrics NBNN [Boiman2008] Bag-of-words Structure-from-motion [Wu2013] NBNN + Time only AutoLink compared with… Evaluation with 2 datasets: 1.Single participant 2.6 participant Evaluation with 2 datasets: 1.Single participant 2.6 participant [Wu2013] Changchang Wu, "Towards Linear-time Incremental Structure from Motion", 3DV 2013 [Boiman2008] O. Boiman, E. Shechtman, M. Irani, ” In Defense of Nearest-Neighbor Based Image Classification”, CVPR 2008
45
© Copyright National University of Singapore. All Rights Reserved. EVALUATION: ACCURACY SLIDE: 45 DatasetSingle userMultiple user ApproachPrecisionRecallPrecisionRecall AutoLink0.70.780.710.51 NBNN + Sensors0.760.380.740.37 NBNN + Time0.640.730.640.32 NBNN0.780.10.780.25 Structure-from- Motion 0.53 0.37 Bag-of-Visual words 0.190.0290.19 Upto 70% precision and 78% recall BOW poor performance without training SfM does not classify scenes with similar features
46
© Copyright National University of Singapore. All Rights Reserved. EVALUATION: RUNNING TIME SLIDE: 46 ApproachAverage running time per image (in milliseconds) AutoLink298 NBNN + Sensors290 NBNN + Time292 NBNN287 Structure-from-Motion5946.7 Bag-of-Visual words35.9 20 times faster than SfM BOW faster but poor accuracy
47
© Copyright National University of Singapore. All Rights Reserved. EVALUATION: REGION MATCHING SLIDE: 47 58% of photos have atleast 50% overlap 40% of photos have atleast 65% overlap
48
© Copyright National University of Singapore. All Rights Reserved. EVALUATION: TOP-M PREDICTIONS FOR RANDOM BOUNDING BOXES SLIDE: 48 Top-M predictionsPercentage accuracy 156.6 270.4 374.9 576.9 1080.14 1581.11 2081.11 Top-2 results have the best match to requested bounding box 70% of the time.
49
© Copyright National University of Singapore. All Rights Reserved. THANK YOU
50
© Copyright National University of Singapore. All Rights Reserved. CONTENT + TIME + SENSOR 1.For every unlabeled and labeled image pair (i,*), estimate: – Number of steps traversed s – Time gap t 2.For all scenes j, get |s – s i,j | and |t – t i,j | using the transition maps. Rank scenes j in decreasing order of this difference. 3.Combine content ranks with the above ranks using mean reciprocal ranking 4.Label, unlabeled image * with the high ranking scene. SLIDE: 50
51
© Copyright National University of Singapore. All Rights Reserved. TRANSITION MAPS FOR TIME AND STEPS SLIDE: 51 3 2 4 1 (s 1,2, t 1,2 ) s 3,4, t 3,4 (s 2,3, t 2,3 ) (s 3,4, t 3,4 ) (s 1,4, t 1,4 ) [s i,j ]: Matrix of steps taken from every scene i to every other scene j [t i,j ]: Matrix of time taken from every scene i to every other scene j
52
© Copyright National University of Singapore. All Rights Reserved. COURTESY CREDITS TO IMAGES http://tech.co/wp-content/uploads/2013/01/vine-twitter.gif http://tech.co/wp-content/uploads/2013/01/vine-twitter.gif http://pad3.whstatic.com/images/thumb/6/61/Have-a-Birthday-Party-Step-7.jpg/670px-Have-a-Birthday-Party-Step-7.jpg http://pad3.whstatic.com/images/thumb/6/61/Have-a-Birthday-Party-Step-7.jpg/670px-Have-a-Birthday-Party-Step-7.jpg http://xtasea.com/wp-content/uploads/2014/02/a-really-cool-looking-compass.jpg http://xtasea.com/wp-content/uploads/2014/02/a-really-cool-looking-compass.jpg https://play.google.com/store/apps/details?id=edu.umich.PowerTutor https://play.google.com/store/apps/details?id=edu.umich.PowerTutor http://www.robots.ox.ac.uk/~vgg/share/images/image07.png http://www.robots.ox.ac.uk/~vgg/share/images/image07.png http://www.cool-smileys.com/blog/wp-content/uploads/2009/11/cool-text-emoticons.jpg http://www.cool-smileys.com/blog/wp-content/uploads/2009/11/cool-text-emoticons.jpg http://4.bp.blogspot.com/-0yBATGzDbV0/Ugk7nLc59QI/AAAAAAAAAeY/rwXXjiIvcYE/s1600/cell-tower.jpg http://4.bp.blogspot.com/-0yBATGzDbV0/Ugk7nLc59QI/AAAAAAAAAeY/rwXXjiIvcYE/s1600/cell-tower.jpg http://www.wpclipart.com/computer/PCs/more_computers/server.png http://www.wpclipart.com/computer/PCs/more_computers/server.png http://www.wireless-social.com/assets/Uploads/_resampled/SetWidth800-router-small2.png http://www.wireless-social.com/assets/Uploads/_resampled/SetWidth800-router-small2.png http://poulingail.edublogs.org/files/2012/09/laptop-kid-18fhs8b.jpg http://poulingail.edublogs.org/files/2012/09/laptop-kid-18fhs8b.jpg https://static-secure.guim.co.uk/sys-images/Guardian/Pix/audio/video/2013/11/6/1383774447878/Smartphone-camera-street--009.jpg https://static-secure.guim.co.uk/sys-images/Guardian/Pix/audio/video/2013/11/6/1383774447878/Smartphone-camera-street--009.jpg http://www.clker.com/cliparts/F/G/b/g/z/I/rf-signal-wave.svg http://www.clker.com/cliparts/F/G/b/g/z/I/rf-signal-wave.svg http://gpsmaestro.com/wp-content/uploads/2012/05/smart-phone-icon.jpg http://gpsmaestro.com/wp-content/uploads/2012/05/smart-phone-icon.jpg http://pndblog.typepad.com/.a/6a00e0099631d08833013486239200970c-800wi http://pndblog.typepad.com/.a/6a00e0099631d08833013486239200970c-800wi http://www.droid-life.com/wp-content/uploads/2013/04/whatsapp1.jpg http://www.droid-life.com/wp-content/uploads/2013/04/whatsapp1.jpg SLIDE: 52
Similar presentations
© 2024 SlidePlayer.com Inc.
All rights reserved.