Field Based Data Validation: a very real experience in wrangling data, taxonomic names, and photos Moorea Biocode Project, supported by the Gordon and Betty Moore Foundation Presentation by John Deck, University of California at Berkeley

Outline Part 1: Background on Moorea Biocode Project Part 2: bioValidator: field based data validation Part 3: A case study in handling taxonomic names in a field based client application

Part 1: Background on Moorea Biocode Project

Moorea Biocode: The Collecting

The Sorting Moorea Biocode: Sorting Specimens

Moorea Biocode: Tissue Sampling

Moorea Biocode LIMS: Binning, Trimming, & Assembly of Sequence Data

Challenges Facing the Moorea Biocode Project IT Team Multiple taxa & a different team for each group. Various cultures and workflow for each team. Everyone in a hurry, non-technical biologists entering data Specimens (& metadata) ultimately owned by multiple host institutions. Multiple labs processing genetic data (w/ different equipment, processes, and workflows). Final taxonomic determination made using Lab and/or Host Institution (Often much later than collecting event) No internet or bad internet in field. *Need to associate photos/standardized higher taxonomy in the field (before accession into any db)

Field Based System Requirements Spreadsheets for data entry Extensible validation rules (each project or sub- project has its own requirements) Match specimen data to Photos Tag photos and load to external system (e.g. Flickr) Query multiple taxonomic authorities (each TaxonTeam selects its own authority) Updates online database periodically.

Part 2: bioValidator: a Field based Data Validation Tool Validate data using extensible validation rules Search multiple taxonomies built in Lucene Specimen to photo matching Upload to Flickr using machine tags No internet required Java based

Part 3: A case study in handling taxonomic names in a field based client application Uniform Lucene Indices, Cached, and searchable offline GenbankITISWoRMS

Why Lucene? Java-based, cross platform Indexes can be delivered to client apps (can run offline) Ability to build a standardized interface to multiple taxonomies.

Higher Taxonomic Name Handling in the Field Initial Spreadsheet: Just assign the lowest taxon name and lowest taxon level. bioValidator: Suggest a higher taxonomy based off name and level provided. Revised Spreadsheet: update with suggested higher taxonomic hierarchy.

Lucene Indexer Implementation for Taxonomy Taxonomic ConceptLucene Class UnitDocument RankField Taxonomic DatabaseIndexWriter String sql = "SELECT tsn from taxonomic_units”; … obtain resultset … while (resultset.next()) { Document doc = new Document(); // itisUnit is class that abstracts ITIS Schema itisRanks ir = new itisRanks(resultset.getString("tsn”)); while (ir.next()) { doc.add(new Field(ir.rank, ir.name)); } IndexWriter.addDocument(doc); Example of Lucene Index built on ITIS

Lucene Search Implementation for Taxonomy public static Hashtable searchIndex(String taxonLevel, String taxonName) { // Construct query Query query = new QueryParser(taxonLevel, taxonName); // Possible multiple matches TopFieldDocs hits = new IndexSearcher().search(query); // Loop through each taxonomic Unit for (int taxonUnit = 0; i < hits.totalHits; taxonUnit++) { Document doc = searcher.doc(hits.scoreDocs[taxonUnit].doc); // Loop each rank to assign to map for (int rank = 0; rank < taxonLevels.getNumLevels(); rank++) { Object value = doc.get(taxonLevels.getLevel(rank)); // Populate a simple table with taxon ranks & values map.put(level, value); } return map; } Example of (a simplified!) Lucene Search:

Further Work Standardization in validation protocols (expand on CRIA work). As we push the envelop in field-based data collection this will become more of an issue. Network of Lucene indexes for taxonomies? GUID implementation in spreadsheets? How to track and update data as it changes in dependent systems (LIMS Systems, Genbank, BOLD, CalPhotos). See BiSciCol Grant (NSF)

More Information John Deck Moorea Biocode Project – bioValidator – bioTaxonomy (Lucene index/search) –