Combining geometry and domain knowledge to interpret hand-drawn diagrams As Presented By: Andrew Campbell Christopher Dahlberg

Lecture Overview ● About the Authors ● Introduction to Sketch Recognition ● History on the Topic ● Paper Overview ● Details ● Contributions and Conclusion

About the Authors ● Stahovich and Shimana were professors in the mechanical engineering department of CMU ● Kara is a post-doctoral researcher there. ● Gennari is student research assistant. ● Stahovich has since moved to UC Riverside ● Very little has changed due to the recency of the paper

Sketch Recognition ● The process of translating human drawn figures into computer readable information. ● How does this challenge differ from vision problems we have seen so far?

Differences ● The task requires parsing the image based on some grammar. ● Human handwriting results in high variability of symbol appearance. ● Additional data is available- stroke order, pen speed.

Recent Work in Sketch Recognition ● Predefined pen stroke sequences ● Shilman- hierarchal definitions based on low level objects ● Kara- “Marker Symbols” ● Costagliola & Deufemia- prerecognized shapes ● Narayanaswamy- fixed drawing order, pauses between symbols

Problems ● Most prior work creates constraints on the order or manner in which the user draws the symbols. ● Many methods also rely on hard-coded recognition of low-level building blocks. ● This leads to a system that is sensitive to any changes from the expected input (size, orientation, legibility)

Paper Overview ● Segmentation ● Symbol location ● Classification ● Culling ● Error correction

Segmentation ● Change pen strokes into a set of lines and arcs. ● Some single pen strokes need to be split into several lines. ● Example:

Segmentation ● The authors used an algorithm from an earlier paper. ● It uses pen speed and changes in curve inflection. ● Differs from the segmentation algorithms we've seen because it depends on getting extra information from how the sketch was drawn.

Locating the Symbols ● Two methods are used: – Locating symbols based on changes in hand coded features. This method is similar to that used in previous work. – A novel domain-specific method based on measuring ink density.

Locating the Symbols Common Features ● Assign a type (line or arc), length, orientation and intersection type (L, T, or X) to each segment. ● Group segments with similar characteristics as potential symbols.

Locating the Symbols Ink Density ● In electrical circuit diagrams, circuit components are identified with fairly dense symbols connected by long stretches of single lines for the wires. ● Ink density is high for the components and low for the wires.

Locating the Symbols Ink Density ● For each segment: – Consider the next segment. If adding it will cause a big drop in ink density, mark this segment as a possible symbol endpoint. – Repeat until a user-specified number of segments are considered. ● For each potential endpoint: – Consider the previous segment. If adding it will cause a big drop in density, mark the whole range as a potential symbol. – Repeat for a user-specified number of segments.

Locating the Symbols ● Why both methods?

Locating the Symbols ● Some symbols can be found easily with one method but not the other.

Classify the Symbols ● A set of features for each potential symbol is calculated. ● A gaussian distribution is fit to each feature and forced to have a standard deviation of at least.3 so that zero counts don't translate to zero probability.

Classify the Symbols ● A naive bayes classifier is used to find the most likely component for each symbol. ● This assumes that each feature is independent (they aren't) and that the gaussians are a good estimation of how the features are distributed.

Culling ● The symbol locater steps are designed to pick up all possible symbols along with many non-symbols. ● A set of domain-specific heuristics are applied to each possible symbol. Symbols with scores above a threshold are accepted. If some symbols overlap, the one with the greatest score is accepted.

Error Correction ● After all the symbols in a sketch have been located and classified, the whole sketch is reexamined to see if it makes sense. ● If there are any components with unattached connections, dangling wires, wires drawn with many segments, or other evidence of parse errors, the threshold values for those segments are modified and they are reprocessed.

Analysis- Paper Quality ● How effectively do the authors frame their presentation of their work? ● Do they adequately introduce the problem? ● Are their examples relevant and clear?

Analysis- Technique Quality ● How does the work presented in this paper advance the field of computer vision?

Contributions ● Ink density model ● Lack of stroke constraints ● Error correction step

Conclusion ● The work presented here advances the power of sketch recognition techniques ● The system still relies on unique sketch data (stroke speed, order) ● Difficult to judge overall effectiveness due to lack of standard corpus of test problems