DDI – Metadata for social science data Wolfgang Zenk-Möltgen GESIS – Leibniz Institute for the Social Sciences DataCite Summer Meeting 2010 – Making datasets visible and accessible Hannover, 7-8 June 2010

About DDI Basic DDI Concepts Identification and Citation Application Examples Topics Acknowledgement: DDI Alliance TIC members, namely Wendy Thomas, Arofan Gregory, Joachim Wackerow

About DDI DDI – Data Documentation Initiative The Data Documentation Initiative (DDI) is an effort to create an international standard for describing social science data. Expressed in XML, the DDI metadata specification now supports the entire life cycle of social science datasets. DDI metadata accompanies and enables data conceptualization, collection, processing, distribution, discovery, analysis, repurposing, and archiving. (Stefan Kramer)

History of DDI Concept of DDI and definition of needs grew out of the data archival community Established in 1995 as a grant funded project, initiated and organized by ICPSR February 2003 – Formation of DDI Alliance –Membership based alliance –Formalized development procedures

Members of DDI Initial members –Social science data archives –Statistical data producers Actual membership expanded by –Research data centers –Data producers –Commercial organizations University of Alberta, Canada Australian Bureau of Statistics (ABS) Australian Social Science Data Archive (ASSDA) University of California, Berkeley -- Computer-Assisted Survey Methods Program and UCDATA University of California, California Digital Library Centro De Investigaciones Sociologicas (CIS), Spain CEPS/INSTEAD -- Luxembourg Cornell University (CISER) Danish Data Archive Data Archiving and Networked Services (DANS), The Netherlands Finnish Social Science Data Archive German Socio-Economic Panel Study (SOEP) GESIS - Leibniz Institute for the Social Sciences University of Guelph Institute for Quantitative Social Science (IQSS) at Harvard University Institute for the Study of Labor (IZA) Inter-university Consortium for Political and Social Research (ICPSR) Massachusetts Institute of Technology (MIT) University of Minnesota, Minnesota Population Center National Opinion Research Center (NORC) Norwegian Social Science Data Service (NSD) Open Data Foundation Princeton University Research Data Centre of the German Federal Employment Agency, Institute for Employment Research (IAB) Roper Center Stanford University Survey Research Operations, University of Michigan Swedish National Data Service (SND) Swiss Foundation for Research in Social Sciences (FORS) United Kingdom Data Archive University of Toronto University of Wisconsin U.S. Bureau of Labor Statistics (Associate Member) World Bank, Development Data Group (DECDG) Yale University

DDI is being used around the world Archives and Data Libraries Research Institutes and Data Service Centers International Organizations and National Statistical Agencies

DDI Versions 2000 – DDI 1.0 –Documentation of simple surveys, microdata only 2003 – DDI 2.0 and 2.1 –Extension to aggregate data –Support for geographic material 2008 – DDI 3.0 –Lifecycle model: Shift from the codebook centric / variable centric model to capturing the lifecycle of data –Focus on metadata creation and re-use –Machine-actionable aspects of DDI to support programming –CAI instruments supported by expanded description of the questionnaire –Data series support (longitudinal surveys, panel studies, etc.) –Support comparison by design and comparison-after-the-fact –Improved support for describing complex data files 2009 – DDI 3.1 –Correction of bugs –Introduction of final URN structure to ensure persistent URNs for all identified elements

Basic DDI 3 Concepts Lifecycle Concept Re-usable documentation –Modules –Maintainables, versionables, identifiables –Scheme-based (maintainable lists) Relations to other standards Controlled Vocabularies

The Data Life Cycle CollectionConceptProcessingDistributionDiscoveryAnalysis Archiving Repurposing

DDI 3 versus earlier versions Previous versions had the codebook idea that creates a documentation of a social science dataset DDI 3 with its lifecycle model allows for documentation at all stages from study conception and data processing until analysis and repurposing of data DDI 3 uses XML Schemas instead of XML Data Type Definition (DTD) to have a stronger definition of metadata types, to make better reuse of content and to reach the goal of machine actionability A DDI 3 instance includes now the simple instance from previous DDI versions. Multiple data products can be included for a single study.

DDI 3.1 Modules Contain groups of related documentation elements Some are related to the Lifecycle model, some are technically grouped Archive module Comparative module Conceptual components module Data collection module Dataset module Dublin Core Elements module DDI profile module Grouping module Instance module Logical product module Physical data product module –(plus inline n-cube, normal n-cube, tabular n-cube module and proprietary module) Physical instance module Reusable module Study unit module

Usage of DDI 3 Modules Study Unit Identification Coverage –Topical –Temporal –Spatial Conceptual Components –Universe –Concept –Representation (optional replication) Purpose, Abstract, Proposal, Funding Data Collection Methodology Question Scheme –Question –Response domain Instrument –using Control Construct Scheme Coding Instructions –question to raw data –raw data to public file Interviewer Instructions Logical Product Category Schemes Coding Schemes Variables NCubes Variable and NCube Groups Data Relationships Physical Data Structure Links to Data Relationships Links to Variable or NCube Coordinate Description of physical storage structure –in-line, fixed, delimited or proprietary Physical Instance One-to-one relationship with a data file Coverage constraints Variable and category statistics Archive Organization or individual which has control over the metadata Lifecycle events Archive specific information etc…

Maintainables, Versionables, Identifiables Inheritance Maintainables (may be maintained separately, need agency) Versionables (may be versioned in the form 1.0.0) Identifiables (may be identified and be referenced, either by ID or URN) Other DDI elements Inheritance

DDI Schemes Schemes = Lists of elements of one type Examples archive –OrganizationScheme datacollection –QuestionScheme –ControlConstructScheme –InterviewerInstructionScheme conceptualcomponent –ConceptScheme –UniverseScheme –GeographicStructureScheme –GeographicLocationScheme logicalproduct –CategoryScheme –CodeScheme –VariableScheme –NCubeScheme physicaldataproduct –PhysicalStructureScheme –RecordLayoutScheme

Relationship to Other Standards Dublin Core –Basic bibliographic citation information –Basic holdings and format information METS –Upper level descriptive information for managing digital objects –Provides specified structures for domain specific metadata OAIS –Reference model for the archival lifecycle PREMIS –Supports and documents the digital preservation process ISO – Geography (FGDC) –Metadata structure for describing geographic feature files such as shape, boundary, or map image files and their associated attributes ISO/IEC –International standard for representing metadata in a Metadata Registry –Consists of a hierarchy of concepts with associated properties for each concept SDMX –Exchange of statistical information (time series/indicators) –Supports metadata capture as well as implementation of registries

Contr. Vocab Not part of standard Recommendations on: Example: TimeMethod may be –Longitudinal (Cohort or Trend) –Panel (Continuous or Interval) –TimeSeries (Continuous or Discrete) –CrossSectional –CrossSectionalAdHocFollowUp –Other LifeCycleEventType CommonalityTypeCoded TimeMethod ResponseUnit AggregationMethodsType DataType SoftwarePackage CharacterSet CategoryStatistic SummaryStatistic AnalysisUnit

Identification in DDI 3 Two possibilities to identify an element: –Specify the Tag Agency and Version are inherited –Use the specially-structured URN Agency and Version must be included The structured URN approach is preferred These IDs/URNs can be referenced Both ways need a resolver service that turns the names into locations to make effective re-use possible DDI Alliance ist currently working on that, based on the DNS (Domain Name System) infrastructure approach

URN Identification Examples URN of a maintained object To identify of a variable scheme in DDI 3 via a URN would be as follows: urn=urn:ddi:us.icpsr:VariableScheme.V_GENDER_SCHEME URN of an versionable object All versionable objects are contained within maintainable objects. To identify a variable in DDI 3 via a URN would be as follows: urn=urn:ddi:us.icpsr.VariableScheme. V_GENDER_SCHEME.1.0.0:Variable.Gender URN of an identifiable object An identifiable object may be a direct child of a maintainable object or be contained by a versionable object within a maintainable object. The full path should be provided to facilitate locating the item when referenced. To identify the identifiable object in the above hierarchy in DDI 3 via a URN would be as follows: urn=urn:ddi:us.icpsr:DataCollection.DC_ :TimeMethod.TM_ (from the DDI Technical Specification Part I)

Citation in DDI

OtherMaterial Elements Citationholds full citation information for the external object ExternalURLReferencelocation of the external object ExternalURNReferenceURN expression for the external object MIMETypethe standard internet MIME type for applications Relationshipreference to DDI object and description of relation to it Segmentspecifies part of external object (e.g. with audio/video files) UserID unique ID of other types, e.g. DOI Attributes Actionused for local overrides in case of inheritance ("Add" | "Update" | "Delete") id DDI ID of the element isIdentifiable fixed value of "true" objectSourcesource name or location typerequired type code for type of the external object urnDDI URN of the element xml:langoptional identification of the language of the external object

DOIs and DDI URNs Relationship still unclear DDI URN resolution service still needed Every identifiable element could be registered with a DOI, that would result in huge amounts of DOIs Only study level could be registered with a DOI, e.g. each StudyUnit In DDI all registered DOIs should be documented Vice versa each DOI should contain the DDI URN in the metadata Diverse software applications will make use of them

Application Examples Enhanced Publications –Providing Information to connect Publications with the underlying datasets/variables used –Making retrieval of research with specific datasets/variables possible Version History of Datasets –Documenting errata and correction history –Making it easy to cite used data

Supporting Enhanced Publications Publications with References to Data: DDI 3.1 URN contains: Agency Object Version URL of Documentatio n and/or Data URL of Documentatio n and/or Data DDI Alliance find agency gesis.de.ddi return resolver address find object return URL request document return document Publication with References (URNs) urn:ddi:de.gesis:VariableScheme.ZA3811_VarSch.1.0.0:Variable.V

Supporting Enhanced Publications DSDM DDI 3 EPE Simple Export Wizard 1.2.0

Enhancing Publications - DatapluS A University of Tilburg and Centerdata project, supported by GESIS and the European Values Study

Version History of Datasets The GESIS data catalogue holds study descriptions with links to data access GESIS currently introduces a common versioning policy for datasets Starting with version and increasing the major, minor or revision number according to change in the dataset Corresponding to each published version a DOI will be created That gives transparancy in the history of data processing Citation of used datasets will include the specific version to ease replication

Data Catalogue

Thank you!