1. Fundamental Practices for Preparing Data Sets
Bob Cook
Environmental Sciences Division
Oak Ridge National Laboratory

2. The 20-Year Rule
The metadata accompanying a data set should be written for a user 20 years into the future--what does that investigator need to know to use the data?
Prepare the data and documentation for a user who is unfamiliar with your project, methods, and observations
NRC (1991)

3. Information Entropy Information Content Time Paper publication
Specific details are lost
General details are lost
Information Content
Retirement or career change
This graph illustrates the phenomenon of “information entropy”, associated with research. At the time of the research project, a scientists memory is fresh. Details about the development of the dataset are easily recalled, and it is a good time to document information about the process. Over time, memory of the details begins to fade. A variety of circumstances can intervene, and eventually detailed knowledge about the dataset fades. Without a metadata record, this data might be unusable. A dataset it not considered complete without a metadata record to accompany it.
Accident or technology change
Loss of data developer
Time
(From Michener et al 1997)

4. Metadata needed to Understand Data
The details of the data ….
Parameter
name
Measurement
date
Sample ID
location
An investigator and their team need only a small amount of metadata. They know that the steamwater sample was collected at the weir near the orchard and was analyzed by the ion chromatograph in Room 220. But for others outside the team, more information is needed.
Courtesy of Raymond McCord, ORNL

5. Metadata Needed to Understand Data
Units
method
Parameter def.
lab
field
Method def.
method
Units def.
parameter
name
Units
media
date
words
QA def.
QA flag
Measurement
Record
system
records
generator
To share the data more broadly more information is needed:
analytical method, uncertainty, and units
Geocoordinates, elevation, datum
Sample ID
location
date
GIS
org.type
name
custodian
address, etc.
coord.
elev. type
depth
Sample def.
Type, date
location
generator 5

6. Fundamental Data Practices
Define the contents of your data files
Use consistent data organization
Use stable file formats
Assign descriptive file names
Preserve information
Perform basic quality assurance
Provide documentation
Protect your data
Main part of my talk will be to describe these 8 fundamental data practices. Basic habits that will help improve the information content of your data and make it easier to share with others

7. 1. Define the contents of your data files
Content flows from science plan (hypotheses) and is informed from requirements of final archive.
Keep a set of similar measurements together in one file
same investigator,
methods,
time basis, and
instrument
No hard and fast rules about contents of each file.
Researchers who later use your relatively large data file won't have to process many small files individually.

8. 1. Define the Contents of Your Data Files Define the parameters
Ehleringer, et al LBA-ECO CD-02 Carbon, Nitrogen, Oxygen Stable Isotopes in Organic Material, Brazil. Data set. Available on-line [ from Oak Ridge National Laboratory Distributed Active Archive Center, Oak Ridge, Tennessee, U.S.A. doi: /ORNLDAAC/983
NACP Data Management Practices, February 3, 2013

9. 1. Define the Contents of Your Data Files Define the parameters (cont)
Be consistent
Choose a format for each parameter,
Explain the format in the metadata, and
Use that format throughout the file
Use commonly accepted parameter names and units (SI Units)
e.g., use yyyymmdd; January 2, 1999 is
Use 24-hour notation (13:30 hrs instead of 1:30 p.m. and 04:30 instead of 4:30 a.m.)
Report in both local time and Coordinated Universal Time (UTC)
See Hook et al. (2010) for additional examples of parameter formats
Check your discipline and adopt commonly used parameter names and units

10. 1. Define the Contents of Your Data Files Define the parameters (cont)
Check your discipline and adopt commonly used parameter names and units
Global Change Master Directory
Names only
Climate and Forecast Standard Names
FLUXNET Standards
AmeriFlux

11. 1. Define the contents of your data files Site Table……
Ehleringer, et al LBA-ECO CD-02 Carbon, Nitrogen, Oxygen Stable Isotopes in Organic Material, Brazil. Data set. Available on-line [ from Oak Ridge National Laboratory Distributed Active Archive Center, Oak Ridge, Tennessee, U.S.A. doi: /ORNLDAAC/983

12. 2. Use consistent data organization (one good approach)
Each row in a file represents a complete record, and the columns represent all the parameters that make up the record.
Station
Date
Temp
Precip
Units
YYYYMMDD C mm
HOGI 12 14 3 19
-9999
Choose one way to organize your data and stick to that way throughout the file
Make month or site a parameter and have all the data in one large file.
There is an upper limit to the size of files, though.
Note: is a missing value code for the data set

13. 2. Use consistent data organization (a 2nd good approach)
Parameter name, value, and units are placed in individual rows.
This approach is used in relational databases.
Station
Date
Parameter
Value
Unit
HOGI
Temp 12 C 14
Precip mm 3
For data that is sparse (a sparsely populated matrix)

14. 2. Use consistent data organization (cont)
Be consistent in file organization and formatting
don’t change or re-arrange columns
Include header rows (first row should contain file name, data set title, author, date, and companion file names)
column headings should describe content of each column, including one row for parameter names and one for parameter units

15. Collaboration and Data Sharing
2. Use consistent data organization (cont)
Collaboration and Data Sharing
A personal example of bad practice…
Courtesy of Stefanie Hampton, NCEAS

16. 3. Use stable file formats
Los[e] years of critical
knowledge because
modern PCs could not
always open old
file formats.
Lesson: Avoid proprietary formats.
They may not be readable in the future 16

17. 3. Use stable file formats (cont)
Use text-based comma separated values (csv)
Aranibar, J. N. and S. A. Macko SAFARI 2000 Plant and Soil C and N Isotopes, Southern Africa, Data set. Available on-line [ from Oak Ridge National Laboratory Distributed Active Archive Center, Oak Ridge, Tennessee, U.S.A. doi: /ORNLDAAC/783

18. 4. Assign descriptive file names
Use descriptive file names
Unique
Reflect contents
ASCII characters only
Avoid spaces
Bad: Mydata.xls
2001_data.csv
best version.txt
Better: bigfoot_agro_2000_gpp.tiff
Project Name
File Format
Year
Site name
What was measured 18

19. Courtesy of PhD Comics

20. 4. Assign descriptive file names Organize files logically
Make sure your file system is logical and efficient
Biodiversity
Lake
Biodiv_H20_heatExp_2005_2008.csv
Experiments
Biodiv_H20_predatorExp_2001_2003.csv …
As a corollary, need to make sure that your well-named files are organized in a logical directory structure
Biodiv_H20_planktonCount_start2001_active.csv
Field work
Biodiv_H20_chla_profiles_2003.csv …
Grassland
From S. Hampton

21. 5. Preserve information Keep your raw data raw
No transformations, interpolations, etc, in raw file
Raw Data File
Processing Script (R)
### Giles_zoop_temp_regress_4jun08.r
### Load data
Giles<-read.csv("Giles_zoopCount_Diel_2001_2003.csv")
### Look at the data
Giles
plot(COUNT~ TEMPC, data=Giles)
### Log Transform the independent variable (x+1)
Giles$Lcount<-log(Giles$COUNT+1)
### Plot the log-transformed y against x
plot(Lcount ~ TEMPC, data=Giles)
Giles_zoopCount_Diel_2001_2003.csv
TAX COUNT TEMPC C F M F C F M N …
Do actions such as transformations, sorts and analyses in separate files – e.g., here I’m using R to call on this data set and to make plots of the data and do a log transform – this way the changes won’t be retained in the original, raw data file.
You may want to make your raw data file “read only”.
From S. Hampton

22. 5. Preserve information (cont)
Use a scripted language to process data
R Statistical package (free, powerful)
SAS
MATLAB
Processing scripts are records of processing
Scripts can be revised, rerun
Graphical User Interface-based analyses may seem easy, but don’t leave a record

23. 6. Perform basic quality assurance
Assure that data are delimited and line up in proper columns
Check that there no missing values (blank cells) for key parameters
Scan for impossible and anomalous values
Perform and review statistical summaries
Map location data (lat/long) and assess errors
No better QA than to analyze data

24. 6. Perform basic quality assurance (con’t)
Place geographic data on a map to ensure that geographic coordinates are correct.
Place geographic data on a map to ensure that geographic coordinates are correct. Example is from OBFS Web site.
Illinois site (longitude has the wrong sign, should be negative)
Brazil Site (latitude is positive, but should be negative (should be south of the equator).

25. Model-Observation Intercomparison
6. Perform basic quality assurance (con’t) Plot information to examine outliers
NACP Site Synthesis
Model-Observation Intercomparison
Model X uses UTC time, all others use Eastern Time
Data from the North American Carbon Program Site Synthesis
(Courtesy of Dan Ricciuto and Yaxing Wei, ORNL)

26. Model-Observation Intercomparison
6. Perform basic quality assurance (con’t) Plot information to examine outliers
NACP Site Synthesis
Model-Observation Intercomparison
Data from the North American Carbon Program Site Synthesis
(Courtesy of Dan Ricciuto and Yaxing Wei, ORNL)

27. 7. Provide Documentation / Metadata
What does the data set describe?
Why was the data set created?
Who produced the data set and Who prepared the metadata?
When and how frequently were the data collected?
Where were the data collected and with what spatial resolution? (include coordinate reference system)
How was each parameter measured?
How reliable are the data?; what is the uncertainty, measurement accuracy?; what problems remain in the data set?
What assumptions were used to create the data set?
What is the use and distribution policy of the data set? How can someone get a copy of the data set?
Provide any references to use of data in publication(s)

28. 8. Protect data Ensure that file transfers are done without error
Compare checksums before and after transfers
Example tools to generate checksums
Checksums to verify that files received are the same as those sent

29. 8. Protect data (cont) Create back-up copies often
Ideally three copies
original, one on-site (external), and one off-site
Frequency based on need / risk
Remember that disaster may strike not just your computer, but also your building or city. Public repositories provide mirroring of your data at multiple, managed sites.

30. 8. Protect data (cont) Use reliable devices for backups
Removable storage device
Managed network drive
Raid, tape system
Managed cloud file-server
DropBox, Amazon Simple Storage Service (S3), Carbonite 30

31. 8. Protect data (cont) Test your backups
Automatically test backup copies
Media degrade over time
Annually test copies using checksums or file compare
Know that you can recover from a data loss
Periodically test your ability to restore information (at least once a year)
Each year simulate an actual loss, by trying to recover solely from the backed up copies 31

32. Fundamental Data Practices
Define the contents of your data files
Use consistent data organization
Use stable file formats
Assign descriptive file names
Preserve information
Perform basic quality assurance
Provide documentation
Protect your data

33. Proper Curation Enables Data Reuse
Time
Information Content
Planning
Collection
Assure
Documentation
Archive
Sufficient for Sharing and Reuse
Compile information during different parts of the data life cycle. Need to create sufficient information content so that users can find, understand, and use data 20 years into the future. The time axis should be shorter than one year to fully capture the information.

34. Best Practices: Conclusions
Data management is important in today’s science
Well organized data:
enables researchers to work more efficiently
can be shared easily by collaborators
can potentially be re-used in ways not imagined when originally collected

35. Bibliography
Cook, Robert B., Richard J. Olson, Paul Kanciruk, and Leslie A. Hook Best Practices for Preparing Ecological Data Sets to Share and Archive. Bulletin of the Ecological Society of America, Vol. 82, No. 2, April 2001.
Hook, L. A., T. W. Beaty, S. Santhana-Vannan, L. Baskaran, and R. B. Cook Best Practices for Preparing Environmental Data Sets to Share and Archive
Michener, W. K., J. W. Brunt, J. Helly, T. B. Kirchner, and S. G. Stafford Non- Geospatial Metadata for Ecology. Ecological Applications. 7:

36. Questions?

37. Additional Slides