1. TagPix Automatic Real-time Landscape Photo Tagging For Smartphones
Hillol Debnath
Cristian Borcea
Department of Computer Science
New Jersey Institute of Technology

2. Why TagPix?
People take huge numbers of photos with smartphones, especially while traveling
Many landscape photos including unknown landmarks
How to tag the photos with the names of these landmarks?
Mammoth task to manually tag thousand of photos
Helpful if the name is automatically tagged

3. TagPix Goals High accuracy automatic tagging User privacy protection
Sensing the correct context (what exactly is in the photo)
User privacy protection
Not sending the photo to the server (compute everything locally)
Real-time tagging
User may want to share the photo to FB/Twitter after capturing it
Acceptable tag generation latency should be < 1 second
Modest phone resource usage
Avoiding processor hungry techniques
No training and indexing

4. Related Work Cannot Satisfy The 4 Goals
Computer Vision
If used locally
Drains battery heavily
Slow
If used at server side
Privacy problem
Needs extensive training
Augmented Reality Browsers
Only provide navigational help, but don’t pin-point landmarks
Examples: Argon, Wikitude
Google Goggles
Works decent only for very famous landmarks
Needs a properly oriented and lighted photo
Doesn’t work well in low light/for blurred photos

5. Outline Why TagPix? Related Works Overview Design and Implementation
Goals
Related Works
Overview
Design and Implementation
Results
Conclusion

6. TagPix Overview
How about using angle of view of lens?
How about considering Euclidean distance?
Euclidean distance estimation used to filter out Champ de Mars
Current location?
Tag suggestions: Tour Eiffel & Champ de Mars αv
Champ
de Mars
Trocadero
Tour Eiffel
Musee du
Quai Branly
10m
Pont d'Iena
300m
Best tag selected: Tour Eiffel
Using only distance gives wrong Tag: Trocadero
500m

7. Design & Implementation
Context
Sensors
System
Manager
Tag
Generator
Prototype implemented using Android SDK

8. Step By Step Algorithm Acquire current location
Fetch nearby landmarks list
Calculate angular distance
Generate tags
Euclidean distance estimation
Select best tag
Remove false positives

9. Determining Current Location
Android location provider service is used intelligently to reduce energy consumption
Based on the battery condition, location can be acquired using
GPS (accuracy: within several meters, energy: very high)
WiFi (accuracy: <200m, energy: decent)
Cellular Network (accuracy can be >1000m, energy: low)
GPS is used to acquire location in normal battery level
However, the trade-off between energy consumption and location accuracy in considered during location query
Location
Landmarks
Angular Distance
Tag Gen
Euclidean Dist

10. Collecting Tags From Landmark DB
Google Places API is used
Gigantic collection of landmark info
HTTP request is used to fetch data
Data retrieved in JSON/XML
How many landmarks are fetched is configurable
Requires to send current location to server
Alternative: using local DB such as GeoNames
Location
Landmarks
Angular Distance
Tag Gen
Euclidean Dist

11. Determining Where The Target Is
Angle of View (α) of the lens is calculated
Orientation sensor is used to find rotation angle (z)
Rotation around Y (for portrait mode)
Rotation around X (for landscape mode)
Location
Landmarks
Angular Distance
Tag Gen
Euclidean Dist

12. Calculating Angular Distance
Based on the orientation angle (z) and the nearby landmark info
Angular distance ϒ is calculated for each landmark: ϒ = β – z + δ
δ is the magnetic declination
Angular distance between true north and magnetic north
In ideal case, ϒ = 0
The lower the ϒ is, the higher the chance the landmark is the target
Choose landmark 1
Location
Landmarks
Angular Distance
Tag Gen
Euclidean Dist

13. Tag Generation
Using only angular distance calculations, tags will be generated based on 4 classes of angular distances (e.g., ϒ<θ, ϒ<α/2, etc.)
There might be multiple tags suggested if there is more than one landmark present in the photo
User can configure this behavior
Since the Euclidean distance to the target is unknown at this point, TagPix might generate unwanted tags (false positive tags)
Not visible in the photo, hidden behind any visible landmark
Location
Landmarks
Angular Distance
Tag Gen
Euclidean Dist

14. Problem With Using Only Angular Distance
Camera
Both landmarks are within angular distance θ
Manor Bar will be the 1st suggested tag (15° < 25°)
But, Manor Bar is far and not visible in the photo
Only angular distance based tag generation will result in false positives θ d1
20° d2
15°
Burger
King
Manor Bar
(Not visible
to the user)
Location
Landmarks
Angular Distance
Tag Gen
Euclidean Dist

15. Euclidean Distance Could Help With Tag Selection
Formula used in Optics cannot be used on smartphones to calculate the distance of the target
It uses the focal length and the magnification
The distance cannot be calculated due to digital zooming technique and small sensor size
We propose 3 methods to estimate the Euclidean distance
If(|d1-dest|<|d2-dest|) select landmark1
Using Euclidean distance
estimation to select best tag
Location
Landmarks
Angular Distance
Tag Gen
Euclidean Dist

16. Euclidean Distance Estimation I
Simplest; User only needs to take the photo once
Formula:
d = h * tan(x)
As the object is targeted only once, the accuracy is not very good, especially for long distances
Location
Landmarks
Angular Distance
Tag Gen
Euclidean Dist

17. Euclidean Distance Estimation II
The user needs to target the object twice
The user captures the photo, then moves forward/backward for few steps and targets the object again
Formula:
As the object is targeted twice, the accuracy increases compared to the previous method
Location
Landmarks
Angular Distance
Tag Gen
Euclidean Dist

18. Euclidean Distance Estimation III
The user needs to target the object twice
The user captures the photo, then moves left/right for few steps and targets the object again
Formula used:
The best accuracy
Location
Landmarks
Angular Distance
Tag Gen
Euclidean Dist

19. Outline Why TagPix? Related Works Design and Implementation Results
Goals
Related Works
Design and Implementation
Results
Conclusion

20. Results Tested using 5 different Android phones
Evaluated in 8 cities of USA
89 photos
Accuracy: 86.52% using only angular distance
Only class 1 and class 2 tags were considered

21. Improved Results with Euclidean Distance Estimation
Combining both angular distance and Euclidean distance can accurately pin-point to the target object
Accuracy increased to 93%
81% false positives removed

22. Blurry Photos & Photos In Low Light
Vision based systems work very poorly for blurred photos or taken in low light
As our system works based on sensor data, it doesn’t have a problem with such conditions

23. Summary (Features)
Accuracy: meaningful and relevant tags for the photo content
Real time: tags are generated within < 1 second
Photos are shareable in real-time to other apps (FB, Google+, Twitter, etc.)
Privacy: no need to send photos to the server; fully local computations
Modest resource consumption:
Does not harm the battery life of the phone
Consumes very little data bandwidth

24. Thank You!
Happy Clicking!

25. Threshold θ and Tag classes
θ is a threshold angle used to tune the resulting tags
Any landmark tag with ϒ < θ is considered to be highly accurate tag
Different classes of tags (the higher the class is, lower the accuracy is)
Class 1: if ϒ < θ
Class 2: if ϒ < α/2 (α is the angle of view of the lens)
Class 3: if ϒ < α
Class 4: if ϒ < 90°
If the user configures θ to be as large as α or 90°, then whatever falls within the camera’s viewing angle would be suggested as tags
If the user configures θ to be very low (<15°), then only most accurate ones would be suggested as tags