Hackathon Challenge: (Semi-) Automating DNA Collection Sara Farmer Noah Hofmann-Smith Jonathan Undy.

Slides:

Advertisements

Similar presentations

A gentle introduction to R – how to load in data and produce summary statistics BRC MH Bioinformatics group.

Advertisements

A Toolbox for Blackboard Tim Roberts

DL Windows Software “Rules” Import a CSV File From Excel

Project level information Structure of IATI XML file Includes: Activity identifier (project id) Reporting organization Participating organization Activity.

AgMIP SSA Meeting Accra, Ghana 12 September, 2012 Importing and translating crop model data.

‘On-the-Ground’ Test Execution Challenges

Newsletter Plugin The newsletter plugin allows you to create and send newsletters to a managed list or multiple lists of users. Your users can subscribe.

04b | Manage Test Execution (2 of 2) Steven Borg | Co-founder & Strategist, Northwest Cadence Anthony Borton | ALM Consultant, Enhance ALM.

ACAT 2008 Erice, Sicily WebDat: Bridging the Gap between Unstructured and Structured Data Jerzy M. Nogiec, Kelley Trombly-Freytag, Ruben Carcagno Fermilab,

The Sixth Form College Farnborough Microsoft® Office OpenXML Jim Lyle Data Analyst The Sixth Form College Farnborough Presented at the Sixth Form Colleges’

ANU Archives Maggie Shapley, University Archivist.

A web based Project Management and Tracking System Zheng Wang, Yuntian Zhao, Yanhong Li Biostatistics & Statistical programming.

Context\Context.wb Library Functions.  Data Driven Programming.  Enhance Existing programs.  Quickly Develop new Programs.  Powerful Set of Library.

QWise software engineering – refactored! Testing, testing A first-look at the new testing capabilities in Visual Studio 2010 Mathias Olausson.

Selecting and Combining Tools F. Duveau 02/03/12 F. Duveau 02/03/12 Chapter 14.

Reporting and Build Statistics Using Business Intelligence By Naga Sowjanya Karumuri Build Team, VMware, Cambridge Summer Internship 2008.

Auto Board Power Testing George Madden Principle Engineer.

Data on the Web Life Cycle Bernadette Farias Lóscio March, 2014.

Integrating and managing your Engaging Networks data Top ten data features.

1 Country report 2014 – Statistics Norway PC-Axis Reference Group meeting

© 2008 Ocean Data Systems Ltd - Do not reproduce without permission - exakom.com creation Dream Report O CEAN D ATA S YSTEMS O CEAN D ATA S YSTEMS The.

Copyright © Eric Liria Web Site Builder This application allows you to build and manage web sites. It provides the following functionnalities: use.

DBSi 5.0 Data Cleansing January Agenda Introduction Customer Information Special Characters & Validity Checks Sample Sequel Related documentation.

SRDR Quarterly Training Brown Evidence-based Practice Center Brown University June 20 th, :00pm-2:00pm Entering Data Retrospectively into SRDR The.

ABSTRACT Dual classification systems (Dewey and LC) and a complex floor plan presented challenges for patrons in the main campus library at the University.

Step by Step Instruction: How to Conduct Direct Certification using File Upload: Standard Format Released January 2014 “How to Conduct Direct Certification.

A STEP-BY-STEP GUIDE FOR TEACHERS AND STUDENTS How to Use Google Documents.

Walk through the reporting process for Barcelona Convention using Reportnet Miruna Badescu, Giuseppe Aristei.

Hampshire Hub Data Platform Progress update 1 October Bill Roberts Swirrl.

SMS Experiment Work as a Group Do not modify the password Send sms to your group mates and yourself 1.Type the phone numbers in the box 2.Use csv or xls.

Online Library of Knowledge Juro4C – Introduction.

11 TRAINING COURSE ON MALARIA ELIMINATION FOR THE GMS Databases Ryan Williams Chang Mai, August 2015.

Estimate Job Specific Play Book Production Worksheet Production Database Actual Estimate Actual Estimate Remote Time Entry Field Reporter OfficeField Accu-Crete.

Powered by Microsoft Azure, PointMatter Is a Flexible Solution to Move and Share Data between Business Groups and IT MICROSOFT AZURE ISV PROFILE: LOGICMATTER.

Start smart finish wise The Kiel Marine Science Provenance- Aware Data Management Approach Peer Brauer 1, Andreas Czerniak 2, Wilhelm Hasselbring 1 1 Software.

Core Cluster IM Products & Humanitarian Datasets IMWG Meeting 21 December 2015.

DEVELOPMENT GATEWAY Solutions that empower. Overview Program Description and Learning Agenda Activities Performed Progress Made Lessons Learned Recommendations.

What’s Next? Generation Challenge Programme. 1. Create list of germplasm 2. Create study 3. Create trial using wizard 4. Save & Export Fieldbook How to.

Web Scraping with Python and Selenium. What is Web Scraping?  Software technique for extracting info from websites Get information programmatically that.

In order to survive in the era of competition a business firm needs market research. Researching market involves thorough analysis and gathering of data.

UNEP Live. What is UNEP Live? - An on-line knowledge management platform - Focuses on open access to global, regional and national data and knowledge.

Forum to improve your experience entering data into SRDR 1 SRDR is being developed and maintained by the Brown EPC under contract with the Agency for Healthcare.

Medway: Here we David Whiting SEPHIG, 16 June, 2016.

ONS API Progress / Plans July 2010 Census Web Services Working Group.

1 New Perspectives on Access 2016 Module 8: Sharing, Integrating, and Analyzing Data.

HCAI Information for ACtion 2010

Best Data Mining, Web Scraping and ebay Template Services

LSI Business Intelligence Initiative

Managing Large Data Sets For Finance officers Sean p. Canning mpa, qpa

Kelly Romirowsky, PsyD Evaluation & Research Manager,

Statistical database Debbie Becker Developed by.

22-INTEGRATION HUB

The Shortest Distance…

Removing Duplicate Job Ads

SEO can excel your website.

Matrix Template and Example

How to Run a DataOnDemand Report

Integrating Survey data into a geographic information system

RDF123 RDF123 is an application and web service to generate RDF data from spreadsheets Graphically create/edit spreadsheet to RDF map MAP map + spreadsheet.

SDMX: Enabling World Bank to automate data ingestion

Data Extraction using Web Scraping

Whakatāne District Business Recovery Grant application process

This module Provides some tips for data management

eDAMIS Status for UA collection

Course Introduction CSC 576: Data Mining.

SDMX: an Overview Abdulla Gozalov UNSD.

Importing Transmittals Using the Excel Spreadsheet Template

DATA MANIPULATION Wendy Harrison Mari Morgan Dafydd Williams

Recitation on AdFisher

Presentation transcript:

Hackathon Challenge: (Semi-) Automating DNA Collection Sara Farmer Noah Hofmann-Smith Jonathan Undy

Outline Need to assess country preparedness on onset of disaster QUICKLY. Lots of sources, but is not machine accessible.

Motivation Websites: Html, xls, csv, apis etc Template Creator Partially-filled indicators spreadsheet Researchers Completed indicators spreadsheet DNA Analyst

Outline 2 Process for automation: Scrape data from webpages Transform scraped data into CSV files Automatically load data from CSV files into standard Excel report Sara and team (partially completed already) Noah and Jonathan

Scraping data and CSV files (Sara)

Scrapers

CSV Data Files

Loading from CSV files to Excel (Noah & Jonathan) Challenges: Key indicators referred to differently by different sources Several years’ worth of data Countries not included in all datasets

Challenges going forward Improving data quality. (E.g. unpacking compound data items from the same field.) Continue to develop the standard list of indicators. “Close the loop”. Eliminate manual cleaning of the scraped data.