Crowdsourcing Innovation: Open Data and APIs Jonathan Young December 10, 2012

About the Presenter Currently: Financial Analyst, Oakland Housing Authority MPA Student, SF State Experience: 7 years of consulting and public sector work in infrastructure planning and growth management finance

What is the management innovation? Public agencies at all levels of government generate and collect data on all kinds of activities (everything from bus arrivals to weather forecasts to crime reports to medical treatment outcomes). By making this data increasingly open and available, people can use the data to develop applications that can benefit businesses, citizens, and public agencies. The data can be more usable if agencies provide it in standardized formats and develop application programming interfaces (APIs) for new software to interact with the data feeds.

Big Picture Implementation of ideas found in emerging public administration theories: governance, public value, Government 2.0 – Crowdsourcing – Networks & Coproduction – New ways to create public value

Definitions Open data is data that meets these criteria: accessible (ideally via the internet) at no more than the cost of reproduction, without limitations based on user identity or intent; in a digital, machine readable format for interoperation with other data; and free of restriction on use or redistribution in its licensing conditions. – UK Open Data White Paper

Definitions An application programming interface (API) is a particular set of rules and specifications that a software program can follow to access and make use of the services and resources provided by another software program that implements that API. – 3Scale Networks

Examples of Applications Using Open Data transparency on 401(k) fees and profiles of financial advisors using SEC data web and mobile app providing data on building permits pulled in San Francisco identifies sites in New York City for potential community project – public vacant lots, community gardens, etc.

History & Origin No clear starting point – government has always been creating and publishing data used by businesses and individuals. – e.g. maps, geologic surveys, property tax assessments Early 2000s: with Web 2.0 technologies increased interactivity through the web – Increased interest in app development, data mashups and crowdsourcing 2007: iPhone introduced. With mobile devices, more situations when data can provide value 2009: and data.gov.uk launched as clearinghouses for government data in machine- readable formatswww.data.govdata.gov.uk

API Explosion (mostly private sector)

The Future Data storage and computer power continue to get cheaper Computing moves to the cloud More and more data will be created by government, and developers will have the capability to do more and more useful things with the data Shift from government to governance, coproduction of public value

Significance “Open data is the raw material of the new industrial revolution.” – Francis Maude, UK Minister for the Cabinet Office Open data used in health, life sciences, transport, real estate, weather, GPS, etc. Government data systems generate billions of dollars of value, with the potential for more – NOAA weather data: $10 billion annually – U.S. GPS systems: $90 billion annually – Potential value from open health data: $350 billion Source: Howard, 2012

Significance New data applications can increase the value of public services (e.g. next bus apps, parking spot locators) Public agencies do not have the resources to develop every app that could be useful or the creativity to envision every useful application of data – open data and APIs allows others to create value with the data U.S. government, states (Missouri, Colorado), and cities (Baltimore, Chicago, NYC, Seattle, SF, Palo Alto, etc.) have established data policies and websites to collect standardized, open data sets and make them available to application developers

Stakeholders Data generators Public agencies Effort and resources required to make data available and usable Application developers Some companies’ business model is organizing data and creating useful applications e.g. Google, BrightScope Application users Business Citizens Government

Trends Political Open data is a newly prominent goal in federal government and large cities Open Data Initiatives launched by feds, states, cities Establishment of CTO and CIO offices in governments Economic Increasingly data- driven economy Web 2.0 Silicon Valley weathered Great Recession relatively well Social Customers joining in to coproduction of useful data e.g. Yelp.com Technical Cheaper processing and storage Cloud computing Mobile devices Environmental Big data applications driving energy efficiency Smart meters and smart grids e.g. Nest learning thermostat

Design Steps in the process from data generation to open data app: 1.Agency generates a data set or a stream of data 2.Formatting standards (for data sets) or APIs (for continuous interactions between data systems and apps) are developed and published A.May be developed by agency or jointly with an app developer 3.Agency publishes data in a stable location 4.Developers write applications that use data or incorporate it to enrich existing apps (e.g. Google Maps and traffic congestion data) A.Ideas for apps may come from outside developers, discussions between developers and agency, or stimulated by developer challenges and cash prizes

Data Standardization Data can vary from city to city because they don’t always collect the same information, their databases do not use the same format, and they use different codes to categorize information. Governments need to make it easier for companies to use their numbers by creating data standards. e.g. General Transit Feed Specification, developed in 2005 by Portland’s TriMet transit agency and Google, now used by 500+ cities. Allows web developers to integrate transit information.

8 Principles of Open Government Data Data shall be considered open if it is made public in a way that complies with these principles: 1. Complete All public data is made available. Public data is data that is not subject to valid privacy, security or privilege limitations. 2. Primary Data is as collected at the source, with the highest possible level of granularity, not in aggregate or modified forms. 3. Timely Data is made available as quickly as necessary to preserve the value of the data. 4. Accessible Data is available to the widest range of users for the widest range of purposes. 5. Machine processable Data is reasonably structured to allow automated processing. 6. Non-discriminatory Data is available to anyone, with no requirement of registration. 7. Non-proprietary Data is available in a format over which no entity has exclusive control. 8. License-free Data is not subject to any copyright, patent, trademark or trade secret regulation. Reasonable privacy, security and privilege restrictions may be allowed. Source: Open Government Working Group

What resources are required/helpful stimulate app development from open data? Required Raw data Formatting rules Web hosting IT expertise to publish data Interested developers Helpful Developer collaboration to design useful data specs Formatting follows interagency standards An API Developer challenges and prizes

Metrics and Timeframe Relevant Accessible Reliable Well formatted Developers know it exists Open data initiatives are successful if the data are actually incorporated into applications that are used by citizens, businesses or public agencies. Success depends on having data that are: Data can sit waiting for developers to use it for an indefinite amount of time. But if the agency has specific goals for the data, it needs to become a developer itself or proactively engage developers.

Interorganizational Validity – Someone has to care Unless there is a payoff, you can release the data but nothing will happen. There needs to be – Commercial interest – Nonprofit willing to invest resources, or – Volunteers wanting to solve a problem (Code for America) Scale matters. For some types of data and some agencies, the scope may be too small for it to be worthwhile to invest effort in using data. – Initiatives to standardize data formats and create APIs that work across jurisdictions build scale and can make development of apps using data from smaller cities/agencies more feasible

Intraorganizational Validity – Not going to magically solve your agency’s IT problems Opening up data allows the public to benefit from your agency’s data. BUT… Just releasing data does not mean someone will build a useful application with it. If your organization or your clients have a need for a particular application, you still need to built it yourself, contract it out, or actively engage with an organization that might want to do it. Agencies have used developer challenges and cash prizes to encourage people to build desired apps.

Sources of Resistance Public vs. private value – Developers may want exclusive use of data to increase paybacks of their efforts – Citizens and agencies may dislike using public data for commercial benefit Privacy: concerns about private data from government services becoming widely available Sunshine: cities and other agencies may hesitate to publish data that facilitates better performance evaluation

References UK Government. (2012). Open data white paper: Unleashing the potential Black, A. (2012, September 5). The most important information missing from yelp. Slate.com. The Presidential Innovation Fellows: Open Data Initiatives. Howard, A. (2012, February 12). Data for the public good. Howard, A. B. (2012, September 10). Here comes the data economy. Slate.com. What is an API? 3Scale USA. content/uploads/2012/06/What-is-an-API-1.0.pdf

References data.baltimorecity.gov data.cityofchicago.org data.cityofnewyork.us data.hawaii.gov data.honolulu.gov data.mo.gov data.seattle.gov data.sfgov.org data.somervillema.gov data.gov.uk explore.data.gov opencolorado.org paloalto.opendata.junar. com paloalto.opendata.junar. com

Frequently Asked Questions Q: Why should private companies be allowed to profit from public data? A: If companies are able to develop uses of data that generate a profit, it means they are creating value for someone, and that is generally good for society. The use of open data is non-exclusive, so they aren’t stopping anyone else from benefitting from the use of the data. Q: Why should agencies invest resources in developing APIs? A: APIs allow separate computer programs to work together. If data isn’t just a static list, but instead is a dynamic stream generated by government systems, and API might be needed to facilitate beneficial uses of the data by third party developers.