Presentation on theme: "Statewide Real-time Data Hub Update Presented by Marullus Williams April19, 2012."— Presentation transcript:
Statewide Real-time Data Hub Update Presented by Marullus Williams April19, 2012
Background The transit community in Virginia is looking at transit traveler information and is discussing standards, trends and applications ITS Virginia along with the Virginia Department of Rail and Public Transportation (DRPT) led an effort to create a technology community for transit operators statewide To accomplish the task, a working group was formed to; Discuss, develop and promote the use of transit technology standards, Act as a resource for the exchange of ideas and general technology discourse, Promote the systems engineering process for the development, procurement and deployment of transit ITS projects The working group is an open group for anyone interested in transit technology
2011 ITS Program Update Survey *Survey conducted in 2010 and published in 2011 ITS Program Update - ITS participants rated the needs of transit operators
Background Transit Traveler Information is primarily in two forms Static Data – most static transit data is now available in electronic form provided in; Trip planners via the web, and In standard formats like GTFS Real-time Information – some larger transit providers are now providing in; Various form via web tools and applications In various formats There are nearly 30 transit operators supported by State funds in Virginia There are over a dozen urban, small urban and rural transit agencies engaged in this traveler information discussion in Virginia Virginia has an active community discussing transit ITS issues through the support of ITSVA and the Commonwealth
Approach The working group is interested in making real-time and historical data available to the public and to 3rd party developers in order to Improve passenger information, Improve government transparency, and Improve multimodal transportation options A reasonable approach is to use the standards working group to define Virginia transit traveler information goals and leverage the work and approach undertaken by WMATA, Blacksburg and other national leaders The potential benefits of this approach include; Strengthening of standards-based sharing, Out of the box interoperability, and Cost efficiencies to agencies by leveraging the existing investment
Progress The working group has met numerous times over a number of months to discuss Agreements and potential public policy Standard formats, including the possible creation of a standard real-time data format for Virginia transit agencies to follow Lead and participating agencies Hosting locations for static and real-time data Lead efforts; Recently, WMATA released real-time and historical data to the public through the use of very inexpensive, commercially available, cloud computing technologies through a standards based application programming interface (API). This API has been extremely well received by the transit, software, and passenger communities Blacksburg Transit in the Virginia Tech community has also provided their real-time transit data in open API In August 2011, DRPT sponsored the development of a ConOps to guide the development of the Statewide Real-time Data Hub. The ConOps was completed November Future implementation plans are yet to be determined.
ConOps FINDINGS and RECOMMENDATIONS
ConOps Study Team Washington, DC –based SIRC created the ConOps Team Members: Jamey Harvey, Subject Matter Expert Marullus Williams, Project Manager Kunmi Ayanbule, Technical Architect
Important Notes These recommendations are currently being reviewed by DRPT No decisions have been made on how to proceed with these recommendations This presentation is only an update on the findings presented to DRPT by the ConOps Team
The Regional API Concept Needed to decide on the approach to the Regional API. Is the goal to facilitate each transit agencys ability to publish its own standards based API, or is the idea to have all of the regional data fed into a regional API? If each local agency publishes its own API, the regional responsibility would be to ensure that each agency is truly standards-based and interoperable. There would be a regional directory of agency API feeds, but each agency would be responsible for building and maintaining its own API. There could be a hybrid approach in which the Regional API would aggregate data from each local API. The Regional API would in essence become a consumer of each local APIs data and in turn provide that information to the public 8/8/201110
Four Regional Approaches Local Transit Agency A Local Transit Agency B API Local Transit Agency A Local Transit Agency B Regional Data Store and API Developer Community / Public Data Consumers / Transit Agencies 1: Direct from Local API to Developer with Regional Directory Regional Directory of API Feeds 2: Local API to Regional API 3: All data aggregated and served at regional level 8/8/ : Hybrid approach of 1-3
Stakeholder Interview Topics Describe your agencys organizational structure. Provide details on your transportation system. Modes of travel Types of service schedules Number of riders Number and types of incidents Geographic range Interconnections with other systems Describe your current IT environment. Describe your current transportation technology. What data does your organization currently make available to the public (including paper-based info)? Which data elements would be easiest to provide to the public via the API? What is your role in providing real time data for the API? 8/8/201112
Agencies Interviewed Fredericksburg: Arnold Levine PRTC/Omniride: Doris Chism, Ryan Jones and Eric Marx Williamsburg/James-City: Kevan Danker Blacksburg: Tim Witten and Aneil Samuel Arlington: Bee Buergler and Tom Scherer VDOT: Scott Cowherd and Noah Goodall Loudoun County: Scott Gross University of MD (RITIS): Michael Pack 13
Key Interview Findings All local agencies have data available to create static GTFS feeds With the exception of WMATA and Hampton Roads, all agencies have bus service only. Most agencies do not have dedicated information technology departments. They are heavily dependent upon city/county IT resources or outside contractors Many agencies have either recently procured, currently evaluating or soon issuing RFPs for AVL technology All agencies are interested in participating in the real time API The three most critical issues facing local agencies in providing real time data: Integrating information from the many disparate transit systems that are in place within each agency Encouraging vendors to provide data in an open, standards-based format Obtaining technical help given the lack of IT resources within most transit agencies Agencies need guidance from the real time API team on how to ensure AVL vendors provide data in the proper format Real-time and static data collection regionally is needed as much for transit planning purposes as for creation of public-facing applications. The scope of this project is to develop the real time API, not the data warehouse specifications RITIS is an important stakeholder in the API development. Most agencies underscored the importance of ensuring that providing data to the API and RITIS are as similar as possible 14
Concept of Operations OV-1
Rolling Out a Successful API Project Agreeing to data sets to be published Implementing a standards-based approach Connecting all required data elements to the API Creating a fast, reliable infrastructure by leveraging cloud services and API-specific solutions like Mashery Publicizing the API Communicate regularly with the developer community Building an API forum / community using tools such as Facebook, Twitter Managing updates to the API. Good documentation is key. Identifying and managing all legal, policy and security risks. Monitoring the use of transit data by developers and the public.
Local Agency Data Collection It is the responsibility of the participating local agencies to integrate the required data and provide a location (or locations) within each agencys infrastructure for retrieving the data required for the API. In order for the data to be collected consistently and uniformly from each local agency, it is important that all local data be formatted as defined in the API specification. The data must be made available via csv files, xml files or Excel spreadsheets. Depending on the type of data that is contained for each file, the data will be updated by the local transit agency and provided to the VTA at varying frequencies. The data retrieval layer will be built within the VTA infrastructure (whether cloud or on-premises). In order to support all agencies with varying technology infrastructures, the data retrieval layer will offer a push and pull service.
Concept of Operations OV-1
Database Considerations Data from the local transit agencies must be stored in the VTA database. The ConOps OV-1 illustrates the need for Data Translation and Integration to accommodate any semantic or syntactical differences in data collected from the regional transit agencies. The VTA Database is intended to be a temporary storage with current data and limited historical data. For example, the Database can keep four hours of transit data after which that data will be pushed to a data warehouse. The Database will be based on a real-time database systems or an in-memory data-store. To improve scalability, several traffic management, rate-limiting and smart time-sensitive data caching strategies will be implemented. Caching will reduce the latency between HTTP requests to the application server and the fetching of data feeds
Concept of Operations OV-1
API Assembly Layer The Feed Assembly Layer packages data that will be provided to Data Consumers. The interface to this layer will be HTTP- based REST protocol, which will respond in one of the supported output formats, SIRI and GTFS/GTFS-RT. This layer will have specific modules for converting to XML, Protobuf and JSON formats depending on the request. Protocol buffers (Protobuf) is a binary format used by GTFS- RT and is a flexible, efficient, automated mechanism for serializing structured data. It is smaller, faster, and simpler than XML. JSON is also a small footprint format that is simpler, less verbose than XML and widely used by application developers. JSON is not natively supported by SIRI and GTFS-RT, but the API Assembly Layer will be able to produce JSON formatted responses based on the structure of GTFS-RT.
API Management An API Management tool like Mashery would provide the following benefits to the API: Eliminates the need to internally develop API gatekeeping functionality Well-supported and currently employed by WMATA, Best Buy, Netflix, Cnet and others to support publication of APIs for third-party developer use Provides API registration, access and self-service provisioning Provides key issuance and credential management Allows usage control: throttling and limiting tied to key, user, method or group Caches frequently used calls Supports business rules configuration based on filters, parameters, and methods Provides real-time insight to all activity and data export available for independent analysis Provides reports that measure uptime, track errors, and show cache activity Provides API usage information including call volumes, top method calls, and top user activity Includes content management, versioning and documentation change control
Portal The Portal must provide information and documentation for Data Consumers The term Data Consumers refers to computer applications (and the users of those applications) that retrieve data via the VTA. The most popular applications that will use the API data can generally be divided into the following two types: Traveler Applications built for desktop, web and mobile platforms Transit Agency Operations and Planning Applications that leverage the data to improve safety, efficiency, and customer satisfaction of transit operations
The User Community Third Party Developer Portal 400+ developers have registered 380,000+ successful API calls per week
The User Community – Window Unit WMATA Signboard Example 27
Data Set Definition Standard data sets foster subsystem and multi-agency communication Proprietary formats can be restrictive or cost prohibitive to convert to a non-proprietary format The national trend is for transit agencies and others to make static and real-time information openly available to developers at no charge Information clearinghouses like Regional Integrated Transportation Information System (RITIS) and VA 511 can also be data receivers Google transit information data standard, general transit feed specification (GTFS) has emerged as a national standard for static information and for the most part is the standard in Virginia Real-time data standards have yet to formally emerge The working group reviewed local existing data formats including; Washington Metropolitan Transit Authority (WMATA) real-time data format, SIRI – transit-specific, highly extensible, and Virginia Tech Bus Tracker
GTFS GTFS transit feed specification defines a common format for public transportation schedules and associated geographic information. GTFS- RT is a feed specification that allows public transportation agencies to provide real-time updates about their fleet to application developers. GTFS Advantages: Supported by Google. Google provides significant marketing resources for publicizing the availability of agencies GTFS data feeds. Easy for agencies to adopt standard and quickly display data via the popular Google Maps service. Robust online documentation and forums to provide support to transit agencies Free to connect to GTFS Many transit technology vendors have adopted GTFS There is a large community of developers familiar with Googles API specifications GTFS Disadvantages: Completely dependent upon Googles support; if Google ceases support for GTFS, the standard would be in jeopardy of obsolescence Google does not provide access to raw data that it collects from agencies Must agree to Googles inflexible legal terms regarding indemnification
SIRI SIRI is managed by a CEN Working Group - TC278 WG3 SG7. SIRI allows pairs of server computers to exchange structured real-time information about schedules, vehicles, and connections, together with general informational messages related to the operation of the services. The information can be used for many different purposes, for example: To provide real time-departure from stop information for display on stops, internet and mobile delivery systems. To provide real-time progress information about individual vehicles. To manage the movement of buses roaming between areas covered by different servers. To manage the synchronization of guaranteed connections between fetcher and feeder services. To exchange planned and real-time timetable updates. To distribute status messages about the operation of the services. To provide performance information to operational history and other management systems SIRI Advantages: Vendor-neutral standard Supports significantly more data elements than GTFS Widely used Internationally Extensible; agencies can create their own custom data fields SIRI Disadvantages: Complex to implement Not used as much in the US as in Europe
Proposed API Technical Specification The API will provide data access via three interfaces: SIRI, GTFS RT and GTFS. Only data elements that are part of a standard can be delivered via that standards interface. The goal is to have SIRI provide access to all data elements. Mode of Transportation (Bus, Rail) Information Type (Static, Real-time, Support) Data Category (Groups similar information, e.g., Agency Information, Stop Information, Route Information) Data Element (Defines individual data elements available via the API) The following information is provided for each VTA Data Element: VTA Name: The unique name assigned by VTA for each Data Element. Participating local agencies will provide data to the API using the VTA names. Description: Explains the information provided VTA Data Type: The data type required by VTA for local agencies to provide the Data Element Transmodel/SIRI Equivalent: The SIRI name that Data Consumers will use to access the Data Element Transmodel / SIRI Module Source: The SIRI module in which Data Consumers will find the Data Element GTFS-RT Equivalent: The GTFS-RT name that Data Consumers will use to access the Data Element GTFS-RT Module Source: The GTFS-RT module in which Data Consumers will find the Data Element GTFS Equivalent: The GTFS name that Data Consumers will use to access the Data Element GTFS Module Source: The GTFS module in which Data Consumers will find the Data Element
ModeStatic / RT Datapoint Bus and Rail Static Agency Information Agency Identifier Agency Name Agency URL Timezone Fare URL Stop Information Stop identifier Stop code Name of stop Stop description Latitude Longitude Zone identifier Stop URL Type of location Parent station identifier Route Information Route Identifier Agency Identifier Short name Long name Description Type of route Trip Information Route Identifier Service Identifier Trip identifier Headsign text Short name Direction Block Shape Stop times Trip identifier Arrival time Departure time Stop identifier Stop sequence Headsign text Pick off type Drop off type Shape distance traveled
Data- point Data Type TransModel/ SIRI Equivalent TransModel/ SIRI Module Source GTFS-RT Equivalent GTFS-RT Module SourceGTFS Equivalent GTFS Module SourceDescription Agency Infor- mation Agency Identifier uint6 4 authorityID Authority --agency_idAgency This field is an ID that uniquely identifies a transit agency. Agency Name StringauthorityName Authority --agency_nameAgency The full name of the transit agency Agency URL StringauthorityURL Authority --agency_urlAgency This field contains the URL of the transit agency. Example: v TimezoneString authorityTimez one Authority --agency_timezoneAgency The timezone where the transit agency is located. Example: UTC+02 LanguageStringauthorityLang Authority --agency_langAgency This field contains a two-letter ISO code for the primary language used by this transit agency. Example: EN Phone Number StringauthorityPhone Authority --agency_phoneAgency The agency's phone number Fare URLString----agency_fare_urlAgency This specifies the URL of a web page that allows a rider to purchase tickets or other fare instruments for that agency online
Future Considerations Finalize phased plan for rollout of the real time data hub. Who will build and manage the infrastructure? What type of governance will be implemented? How will local agencies obtain the funding and technical support required to connect to the data hub?