1. Open Dialogue on Digital Data management
Pat Burns, Dean
Dawn Paschal, Assistant Dean
CSU Libraries
Open Dialogue on Digital Data management
October 13, 2010
Open Dialogue, Data Mgmt.

2. Open Dialogue, Data Mgmt.
Background
NSF requires proposals submitted as of Jan. 18, 2011 to include plans for data management:
NIH & USDA also have similar requirements
Other agencies looming: ‘Federal Research Public Access Act’
Maximizing the value of data by sharing
Discoverability
Access
Preservation
Management
October 13, 2010
Open Dialogue, Data Mgmt.

3. Science ‘Then’ (5-10 years ago)
Theory
Computation
Experiment
October 13, 2010
Open Dialogue, Data Mgmt.

4. Open Dialogue, Data Mgmt.
Science ‘Now’
Data
Theory
Computation
Experiment
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
October 13, 2010
Open Dialogue, Data Mgmt.

5. Open Dialogue, Data Mgmt.
Science ‘Emerging’
Theory
Experiment
Data
Computation
October 13, 2010
Open Dialogue, Data Mgmt.

6. Open Dialogue, Data Mgmt.
Science 2.0 ‘Now’?
Data
Theory
Experiment
Data
Computation
October 13, 2010
Open Dialogue, Data Mgmt.

7. Large Digital Data Sets
Satellite imagery can generate > 1 petabyte (1015 bytes) of data per day!
Supercomputers also generate massive data sets
Can we transport them? E.g., at 10 Gbits per second (note bits, not bytes: 1 byte = 8 bits)
Time = 8x1015 bits/(1010 bits/sec) = 8x105 secs = 222 hours = 1 week, 2 days, 6 hours, 13 mins
Can we store them?
Requires 500 ea. 2 TByte $250 ea. = 5 year lifetime = $2,500/yr.
Requires 1 full rack in a data center: space, power, cooling, …
October 13, 2010
Open Dialogue, Data Mgmt.

8. Open Dialogue, Data Mgmt.
Incoming!
An individual researcher can generate many data sets
We have many researchers who generate large data sets
Number: Many x many = Very many!
Size: Very many x Very big = Enormous!
October 13, 2010
Open Dialogue, Data Mgmt.

9. Projected Needs (2009 CSU Survey)
CSU-DR = 3 Tbytes!!!
October 13, 2010
Open Dialogue, Data Mgmt.

10. Open Dialogue, Data Mgmt.
How Can We Help? IT
Libraries
Storage capacity
Transport capacity
Back-up
Sysadmin
IT security/privacy
Transcoding
Data organization & structure
IP issues
Metadata
Discoverability
Preservation
Joint
Operations
“there comes a time in one’s life, where one must grab the bull by the tail, and face the situation.
System Stewards
Data/Info Stewards
The ‘front end’
Interactions w/ researchers
The ‘back end’
October 13, 2010
Open Dialogue, Data Mgmt.

11. How Can We Help (cont’d)?
Agreement upon a framework
Draft of a framework, present to faculty
Language for our faculty to include in their proposals
Strategy, policy, procedures
Definition of work flow(s)
Architectures for operations & preservation
Back-up vs. preservation, LOCKSS?
October 13, 2010
Open Dialogue, Data Mgmt.

12. Policies: The ‘Front End’
DRM: IP/ownership issues: data sets not ‘copyrightable’ (not creative works)
But there may be local, institutional IP policies that override this
Note that IP ≠ copyright
Creative Commons or Science Commons licensing may apply
An embargo period is required
What are the preservation periods?
October 13, 2010
Open Dialogue, Data Mgmt.

13. NSB Data Type Definitions*
Research collections (small, useful to individuals/teams for life of a project, limited curation, standards typically lacking)
Resource collections (medium, useful to a community, follow group’s standards, mid- to long-term utility)
Reference collections (large, serve many segments of science/engineering, conform to robust standards, indefinite support)
*National Science Board
October 13, 2010
Open Dialogue, Data Mgmt.

14. Open Dialogue, Data Mgmt.
Workflow
Faculty provide data/information
Enter metadata, user’s manuals, select embargo period, select licensing options, enter pubs, point to or supply data sets, …
Librarians manage data/information
Review metadata, ingest and make accessible, review periodically, deaccession periodically (annually?), manage data, interact w/ faculty
IT staff implement and operate systems
Operate system, backups , security, upgrading storage, transport, move to LOCKSS, etc.
October 13, 2010
Open Dialogue, Data Mgmt.

15. Digital Assets - the 4 Pieces
The Metadata, ideally on the CSU-DR
Typical, what we collect today, e.g. lightweight metadata (probably not copyrightable)
Contextual, e.g., user’s manuals (yes, copyrightable)
Scholarly publications associated with the data – ideally on the CSU-DR
The data itself – should be in the most appropriate place (pointers?)
October 13, 2010
Open Dialogue, Data Mgmt.

16. Digital Assets Management
4. Data Sets
Small
Medium
Large
2. User’s
Manuals
1. Metadata
3. Pubs
Disciplinary
Repositories,
SC Centers, etc.
Local
Storage
Libraries-DR
“The Cloud”
“Pointers”
October 13, 2010
Open Dialogue, Data Mgmt.

17. Architecture LOCKSS High-speed Networks Primary System
The Digital Repository
Preservation System
October 13, 2010
Open Dialogue, Data Mgmt.

18. Open Dialogue, Data Mgmt.
8/19/2009
CSU Storage Project
45 TBytes (raw) for ~$8k
October 13, 2010
Open Dialogue, Data Mgmt.

19. Strategy for Storage of Data Sets
Small, < 100 GB, we would agree to store on the DR, but not forever
Medium, we would agree to store on the DR for a limited time at a cost, or on a local server somewhere and we point to it
Large, stored on a disciplinary DR somewhere, at a supercomputer center, or at a large instrument center
We point to it (persistent URL?)
How do we deal with exceptions?
October 13, 2010
Open Dialogue, Data Mgmt.

20. What CSUL will Store & at What Cost
PRESERVATION PERIOD
SIZE
(+ means beyond end of grant period)
SMALL
0.1 TB
MEDIUM TB
LARGE
> 10 TB
Short (1 yr.+)
Free
Maybe +
Medium (2 yrs. +)
$500/TB
Maybe -
Long (> 5 yrs. +)
$1,000/TB No
Forever is a long time…..
October 13, 2010
Open Dialogue, Data Mgmt.

21. Open Dialogue, Data Mgmt.
Needs
Libraries-IT partnership
Define policies for usage
Define practice for usage
Definition of workflows
Operations
Develop needed tools
Build an on-line, self-service submission tool + requirements for review of user-created metadata
Establish systems
Develop preservation infrastructure
October 13, 2010
Open Dialogue, Data Mgmt.

22. Open Dialogue, Data Mgmt.
Issues
Will the DR become a ‘Trusted Digital Repository?’
Will this enhance our proposals?
What will be stored where?
Will disciplinary digital repositories emerge, e.g. at NCAR and elsewhere?
Flexibility is key
How best to engage
The VPR (probably already accomplished)
The faculty
Library staff: faculty and operational (DM Librarians at UNM?)
October 13, 2010
Open Dialogue, Data Mgmt.

23. Open Dialogue, Data Mgmt.
Discussion
Is most welcome.
October 13, 2010
Open Dialogue, Data Mgmt.