Introducing of handwritten icons Ralph Niels, Don Willems and Louis Vuurpijl

Introducing the NicIcon database of handwritten icons Ralph Niels Don Willems Louis Vuurpijl Icon NicI

Introduction Domain: crisis management Data collection Icon design Method Data Classification experiment Method Results Conclusion and URL Overview = Introducing the NicIcon database of handwritten icons Ralph Niels Don Willems Louis Vuurpijl NEW!

Crisis management (CM) Scenario: tunnel disaster Distributed computer systems to support CM Multiple modalities: speech, gestures, and pen Introducing the NicIcon database of handwritten icons Ralph Niels Don Willems Louis Vuurpijl The car is here

Iconic pen gestures Faster than handwriting Easy to learn & remember Visual meaningful shape Introducing the NicIcon database of handwritten icons Ralph Niels Don Willems Louis Vuurpijl

Database Public databases available for several handwriting applications Not for iconic pen gestures Based on symbology reference by USA Homeland Security Workgroup –Used in e.g., USA, Australia, New-Zealand Introducing the NicIcon database of handwritten icons Ralph Niels Don Willems Louis Vuurpijl

Icons Introducing the NicIcon database of handwritten icons Ralph Niels Don Willems Louis Vuurpijl FloodAccidentCarBombRoadblock ElectricityCasualtyFireFire brigadePolice InjuryGasParamedicsPerson

Data collection 35 participants Online and offline Variation in size Per person: –22 pages –55 instances / icon Introducing the NicIcon database of handwritten icons Ralph Niels Don Willems Louis Vuurpijl

Database content Online: 26,163 iconic gestures (24,144 reported in paper) Introducing the NicIcon database of handwritten icons Ralph Niels Don Willems Louis Vuurpijl Tablet: Wacom Intuos2 A4 oversize

Database content Offline: 770 scanned pages Introducing the NicIcon database of handwritten icons Ralph Niels Don Willems Louis Vuurpijl Scanner: HP Scanjet 7400C flatbed 300dpi, 24 bit colour)

Classification experiment Data sub sets Stratified Writer dependent (WD), writer independent (WI) Complete data set (26,163 icons) Evaluation set (40%) Train set (36%) Test set (24%) WD Evaluation set (40%) WD Train set (36%) WD Test set (24%) WI Evaluation set (40%) WI Train set (36%) WI Test set (24%) Introducing the NicIcon database of handwritten icons Ralph Niels Don Willems Louis Vuurpijl

Classification experiment Multi classifier system Feature sets 28 geometric features (Willems & Vuurpijl) 30/60 coordinates running features (Schomaker & Vuurpijl) 1185 features (Willems, Niels, Van Gerven, Vuurpijl) Feature classifiers Support Vector Machine Multi-Layered Perceptron Template matching Dynamic Time Warping (Niels & Vuurpijl) Introducing the NicIcon database of handwritten icons Ralph Niels Don Willems Louis Vuurpijl NEW! on online data

New feature set (‘m-fs’) Features from literature –Geometrical, temporal, pressure Introducing the NicIcon database of handwritten icons Ralph Niels Don Willems Louis Vuurpijl D. Rubine, Specifying gestures by example, Computer Graphics. 25 (4) (1991) 329–337. J. LaViola Jr. & R. Zeleznik, A practical approach for writer- dependent symbol recognition using a writer-independent symbol recognizer, IEEE Transactions on pattern analysis and machine intelligence. 29 (11) (2007) 1917–1926. L. Zhang & Z. Sun, An experimental comparison of machine learning for adaptive sketch recognition, Applied Mathematics and Computation. 185 (2) (2007) 1138–1148. and many others… D. Rubine, Specifying gestures by example, Computer Graphics. 25 (4) (1991) 329–337. J. LaViola Jr. & R. Zeleznik, A practical approach for writer- dependent symbol recognition using a writer-independent symbol recognizer, IEEE Transactions on pattern analysis and machine intelligence. 29 (11) (2007) 1917–1926. L. Zhang & Z. Sun, An experimental comparison of machine learning for adaptive sketch recognition, Applied Mathematics and Computation. 185 (2) (2007) 1138–1148. and many others…

New feature set (‘m-fs’) Over complete icon, but also… –Mean over strokes –Std. dev. over strokes Introducing the NicIcon database of handwritten icons Ralph Niels Don Willems Louis Vuurpijl Feature definitions in technical report at website

New feature set (‘m-fs’) Feature selection –1185 features, which are the best? –Sort them on best individual performance –Add 1 by 1 to classifier Performance maximizes at: –545 features for WI –660 features for WD Selected features: –± 1/3 full icon –± 1/3 mean –± 1/3 standard deviation Introducing the NicIcon database of handwritten icons Ralph Niels Don Willems Louis Vuurpijl The best features

Breaking news: the best features Area Sine first / last sample Length of diagonal Vertical offsets Average centroidal radius Introducing the NicIcon database of handwritten icons Ralph Niels Don Willems Louis Vuurpijl

Classification results g-28: 28 geometric features (Willems & Vuurpijl) m-fs: 1185 features (Willems, Niels, Van Gerven, Vuurpijl) af-30/af-60: 30/60 running features (Schomaker & Vuurpijl) DTW: Dynamic Time Warping (Niels & Vuurpijl) MCS: Multiple classifier system (majority voting) Introducing the NicIcon database of handwritten icons Ralph Niels Don Willems Louis Vuurpijl NEW!

Misclassifications Wrong box Sloppy drawing Retracing Introducing the NicIcon database of handwritten icons Ralph Niels Don Willems Louis Vuurpijl

Discussion Results already quite good (WD: 99.5%, WI: 97.8%), but gain is still possible Which features are important? –For different domains Performance in interactive experiment Offline data still open Mapping online -> offline Introducing the NicIcon database of handwritten icons Ralph Niels Don Willems Louis Vuurpijl

Introducing the NicIcon database of handwritten icons Ralph Niels Don Willems Louis Vuurpijl Freely available: Online (Unipen) Offline (PNG) Technical report about new feature set