Topes: Meeting the Challenges of User Input Validation Christopher Scaffidi Key collaborators: Brad Myers, Mary Shaw Carnegie Mellon University

2 Hurricane Katrina “Person Locator” site: Many inputs unvalidated... and error-ful Introduction  Challenges  Topes  Tools  Evaluation  Conclusion

3 Data errors reduce the usefulness of data. Even little typos impede data de-duplication. Age is not useful for flying my helicopter to come rescue you. Nor is a “city name” with 1 letter. Introduction  Challenges  Topes  Tools  Evaluation  Conclusion

4 Hurricane Katrina sites are not alone in lacking input validation. Eg: Google Base web application –13 primary web forms –Even numeric fields accept unreasonable inputs (such as a salary of “-45”) Eg: Spreadsheets –40% of cells are non-numeric, non-date textual data –Often used to gather/organize textual data for reports Introduction  Challenges  Topes  Tools  Evaluation  Conclusion

5Outline 1.Challenges of data validation 2.Topes Model for describing data Tools for creating/using topes 3.Evaluation 4.Conclusion Introduction  Challenges  Topes  Tools  Evaluation  Conclusion

6 Digging into the details: real user inputs that need validation. Sources: –Interviews of Hurricane Katrina website creators –Survey of Information Week readers –Contextual inquiry of information workers who created and used websites –Logs of what admin assistants typed into browsers –Exploration of the EUSES spreadsheet corpus Validating user inputs has 3 primary challenges… Introduction  Challenges  Topes  Tools  Evaluation  Conclusion

7 1. Inputs don’t always conform well to the simple “binary” validation model. Data is sometimes questionable… yet valid. –Eg: a suspiciously long address –In practice, person names and other proper nouns are never validated with regexps… too brittle. –Life is full of corner cases and exceptions. If code can identify questionable data, then it can double-check the data: –Ask an application end user to confirm the input –Flag the input for checking by a system administrator –Compare the value to a list of known exceptions –Call up a server and see if it can confirm the value Introduction  Challenges  Topes  Tools  Evaluation  Conclusion

8 2. User inputs often can occur in multiple different formats. Two different strings can be equivalent. –How many ways can you write a date? –What if an end user types a date in the wrong format? –“Jan ” and “1/1/2007” mean the same thing because of the category that they are in: date. –Sometimes the interpretation is ambiguous. In real life, preferences and experience guide interpretation. If code can transform among formats (ie: not just recognize formats with regexps), then it can put data in an unambiguous format as needed. –Display result so users can check/fix interpretation Introduction  Challenges  Topes  Tools  Evaluation  Conclusion

9 3. The meaning of data is often tied to its “parts”, not directly to its characters. Data often has parts, each with a meaning. –What are the parts of a date, 12/31/2008? –Valid data obeys intra- and inter-part constraints. –Constraints are usually platform-independent –Writing regexps requires you to translate constraints into a character sequence… tough in many cases, practically or truly impossible in others. If code could succinctly state the parts, as well as mandatory and optional constraints on the parts, wouldn’t the code be easier to write and maintain? –Especially if it was platform-independent! Introduction  Challenges  Topes  Tools  Evaluation  Conclusion

10 Limitations of existing approaches Types do not support questionable values Grammars do not, either, nor can they reformat Information extraction algorithms rely on grammatical cues that are absent during validation Cues, Forms/3,  -calculus, Slate, pollution markers, etc, infer numerical constraints but not constraints on strings, nor are they platform-independent Introduction  Challenges  Topes  Tools  Evaluation  Conclusion

11 Imagine a world where… Code can ask an oracle, “Is this a company name?”, and the oracle replies yes, no, almost definitely, probably not, and other shades of gray. Code allows input in any reasonable format, since the code can ask the oracle to put the input into the format that is actually needed. People teach the oracle about a new data category by concisely stating its parts and constraints. Introduction  Challenges  Topes  Tools  Evaluation  Conclusion

12 New Approach: Topes A tope = a platform-independent abstraction describing how to recognize and transform strings in one category of data Greek word for “place,” because each corresponds to a data category with a natural place in the problem domain Validating with topes improves –Accuracy of validation –Reusability of validation code –Subsequent duplicate identification Introduction  Challenges  Topes  Tools  Evaluation  Conclusion

13 A tope is a graph. Node = format, edge = transformation Notional representation for a CMU room number tope… Formal building name & room number Elliot Dunlap Smith Hall 225 Colloquial building name & room number Smith 225 Building abbreviation & room number EDSH 225 Introduction  Challenges  Topes  Tools  Evaluation  Conclusion

14 A tope is a conceptual abstraction. A tope implementation is code. Each tope implementation has executable functions: –1 isa:string  [0,1] function per format, for recognizing instances of the format (a fuzzy set) –0 or more trf:string  string functions linking formats, for transforming values from one format to another Validation function:  (str) = max(isa f (str)) where f ranges over tope’s formats –Valid when  (str) = 1 –Invalid when  (str) = 0 –Questionable when 0 <  (str) < 1 Introduction  Challenges  Topes  Tools  Evaluation  Conclusion

15 Common kinds of topes: enumerations and proper nouns Multi-format Enumerations, e.g: US states –“New York”, “CA”, maybe “Guam” Open-set proper nouns, e.g.: Company names –Whitelist of definitely valid names (“Google”), with alternate formats (e.g. “Google Corp”, “GOOG”) –Augmented with a pattern for promising inputs that are not yet on the whitelist Introduction  Challenges  Topes  Tools  Evaluation  Conclusion

16 Two other common kinds of topes: numeric and hierarchical Numeric, e.g.: human masses –Numeric and in a certain range –Values slightly outside range might be questionable –(Very rarely) labeled with an explicit unit –Transformation usually by multiplication Hierarchical, e.g.: address lines –Parts described with other topes (e.g.: “100 Main St.” uses a numeric, a proper noun, and an enum) –Simple isas can be implemented with regexps. –Transformations involve permutation of parts, changes to separators, arithmetic, and lookup tables. Introduction  Challenges  Topes  Tools  Evaluation  Conclusion

17 Tope Development Environment (TDE) Topei Module Infers tope from examples Toped Module Enables EUPs to create/edit topes Topeg Module Generates context-free grammars and transformations Topep Module Parses data against grammars, performs transformations Plug-ins Read/write program data Robofox Web macros Vegemite/CoScripter Web macros Microsoft Excel Spreadsheets Visual Studio.NET Web applications … Introduction  Challenges  Topes  Tools  Evaluation  Conclusion

18 Toped User Interface Features Format inference Format/part names Soft constraints Testing features Format reusability Introduction  Challenges  Topes  Tools  Evaluation  Conclusion User Study EUPs are fast & accurate at creating tope formats

19 Integration with programming platforms Microsoft Excel: buttons and menus Visual Studio: drag-and drop code generation Introduction  Challenges  Topes  Tools  Evaluation  Conclusion

20 Other integrations to date: CoScripter, Robofox, XML/HTML library Introduction  Challenges  Topes  Tools  Evaluation  Conclusion

21 Evaluating accuracy, reusability, and usefulness for data cleaning Implemented topes for spreadsheet data –32 topes based on 720 online spreadsheets –Tested accuracy Reused topes on web application data –8 data categories in Google Base and 5 data categories in Hurricane Katrina site –Tested accuracy Used transformations to reformat data –5 data categories in Hurricane Katrina site –Measured increase in number of duplicates identified Introduction  Challenges  Topes  Tools  Evaluation  Conclusion

22 Extracting spreadsheet test data Cluster spreadsheet columns based on data category –EUSES spreadsheet corpus “database” section –Hierarchical agglomerative clustering –Manual inspection –Result = 1713 columns in 246 clusters (1 cluster per data category) Created 1 tope for each of 32 most common categories –Yielding 32 topes –Covered 70% of clustered columns Introduction  Challenges  Topes  Tools  Evaluation  Conclusion

23 We considered 5 validation strategies Strategy 1: Current spreadsheet practice (accept all inputs) Strategy 2: Current webapp practice (validate with regexp or fixed list, when available; accept all other inputs) –36 regexps + 35 fixed lists, in 7 categories Strategy 3A: Tope rejecting questionable (accept when  (str)=1) Strategy 3B: Tope accepting questionable (accept when  (str)>0) Strategy 4: Tope warn on questionable (simulate double-check by user when 0<  (str)<1) Introduction  Challenges  Topes  Tools  Evaluation  Conclusion

24Measurements Based on 100 random values per category Used F1 to measure accuracy –standard measure of accuracy for classifiers = (precision*recall)/avg(precision,recall) Considered topes with 1, 2, 3, 4, or 5 formats Introduction  Challenges  Topes  Tools  Evaluation  Conclusion

25 Recognizing multiple formats and questionable inputs raises accuracy Condition 4: Hypothetical user has to help on ~ 3% of inputs Condition 1: Recall = 0 (fails to identify any invalid inputs) Introduction  Challenges  Topes  Tools  Evaluation  Conclusion

26 Topes based on spreadsheet data were accurate on web application data. Google Base Introduction  Challenges  Topes  Tools  Evaluation  Conclusion Hurricane Katrina

27 Putting data in a consistent format improves duplicate identification. Randomly extracted values for each of 5 Hurricane Katrina data categories Implemented transformations for each 5-format tope from the less commonly used formats to the most commonly used Found approximately 8% more duplicates after transformation Introduction  Challenges  Topes  Tools  Evaluation  Conclusion

28 Conclusion: Topes improve data validation Validating with topes improves –Accuracy of validation –Reusability of validation code –Subsequent duplicate identification Contributions: –Support for ambiguous data categories –Support for transforming values –Platform-independent validation Introduction  Challenges  Topes  Tools  Evaluation  Conclusion

29 Primary Limitations Topes are only appropriate… For string-ish, categorical data –Not for validating images, audio files, etc –Values must appear in a single field or variable –Validation rules derive from categorical constraints When validation rules are known by programmer –Who must label the field/variable with a tope –Who must implement the tope, which runs locally (future work…) Introduction  Challenges  Topes  Tools  Evaluation  Conclusion

30 Future Work: Sharing topes Introduction  Challenges  Topes  Tools  Evaluation  Conclusion Future topes development/use process: 1.People implement new topes by using the basic tope editor (or another language such as JavaScript) 2.People publish tope implementations on repositories. 3.People download tope implementations to local cache 4.Tool plug-ins let people browse their local cache and associate topes with variables and input fields. 5.Plug-ins use tope implementations to validate data. Stay tuned (or come collaborate !!)

31 Thank You… To Margaret Burnett, Martin Erwig, and many others for suggestions over the past 3 years To Oregon State University for this opportunity To NSF for funding Introduction  Challenges  Topes  Tools  Evaluation  Conclusion

32 Professional programmers use lots of tricks to simplify validation code. Eg: njtransit.com Split inputs into many easy-to-validate fields. Who cares if the user has to type tabs now, or if he can’t just copy-paste into one field? Make users pick from drop-downs. Who cares if it’s faster for users to type “NJ” or “1/2007”? (Disclaimer: drop-downs sometimes are good!) I implemented this site in Introduction  Challenges  Topes  Tools  Evaluation  Conclusion

33 Even with these tricks, writing validation is still very time-consuming. Overall, the site had over 1100 lines of JavaScript just for validation…. Plus equivalent server-side Java code (too bad code isn’t platform-independent) if (!rfcCheck (frm.primary .value)) return messageHelper(frm.primary , "Please enter a valid Primary address."); var atloc = if (atloc > 31 || atloc < frm.primary .value.length-33) return messageHelper(frm.primary , "Sorry. You may only enter 32 characters or less for your name\r\n”+ ”and 32 characters or less for your domain Introduction  Challenges  Topes  Tools  Evaluation  Conclusion

34 That was worst case. Best case: reusable regexps. Many IDEs allow the programmer to enter one regular expression for validating each input field. –Usually, this drastically reduces the amount of code, since most validation ain’t fancy. –So why don’t programmers validate most inputs? Introduction  Challenges  Topes  Tools  Evaluation  Conclusion