1. Peter Fox (TWC/RPI) ESIP, Jan 4. 2011
Why and how you value your plumber: establishing and conveying value in the outcomes of a data virtual organization.
Credentials?
Peter Fox (TWC/RPI)
ESIP, Jan

2. What you will hear A bit of history – plumbers (PG13)
Why – VOs: Outcomes and value
How
Evaluating – really, really?
Provenance for a virtual organization?
Opportunities for ESIP and Type Is
A rough attempt to pull it all together

3. ESIP history
11th + year – the honeymoon is well over, past the 7-year itch, now it’s time for ESIP and those that fund it to get comfortable
Plumbers – I promise not to over-do this analogy but it arose (surprise) from the early years of DODS, nka OPeNDAP
The way to make data matter is to get it from those that have to those that want it
Inlets and outlets
Infrastructure is required
Computers and people
Fittings, pipes and yes plumbers
Sources: Scott Adams ©

4. Virtual Organizations, i.e. ESIP
‘A VO is a group of individuals whose members and resources may be dispersed geographically and institutionally, yet who function as a coherent unit with a common goal.’ (Beyond Being There 2008) or a long-term common interest (Ahuja)
ESIP has VOs within VOs, i.e. some organizational structure; executive committee, committees, working groups, clusters
Many goals, different levels of coherency
Source: BBT, 2008 and NSF and Virtual does not always mean most dealings are conducted via CI/IT.

5. Reminder: roles and relationships
‘The virtual research organization: in which members of various corporate and academic research units voluntarily come together to advance a technology on an ongoing basis.
These members assume well defined roles and status relationships within the context of the virtual group that may be independent of their role and status in the organization employing them’ (Ahuja et al., 1998).
I.e. do your goals match ESIPs?

6. Contrasting a Virtual Information Organization with a Traditional one
Source: Rouse (BEVO presentation) 2008

7. Successes focus on value
Source: Rouse, BEVO talk, 2008

8. Evaluation of a VO (1)?
An assessment of the overall effectiveness of an effort, ideally yielding a numeric measure by which informed cost-benefit analysis of investment decisions can be made.
An assessment of the degree to which the effort fulfills its specification in terms of whatever measures were pre-specified.
Adapted from Twidale et al.

9. Evaluation of a VO (2)?
An assessment of whether the effort fulfills the purpose for which it was intended
An assessment of whether the artifacts or processes embodied in the effort have been proved to be superior to an alternative, where that alternative is frequently the traditional solution
An assessment of whether funds allocated to an effort have been productively used, yielding useful generalizable results
Adapted from Twidale et al.

10. Evaluation of a VO (3)?
An assessment of whether the effort proves acceptable to the intended end-users
An assessment of whether end-users continue to use it in their normal work
An assessment of where the effort fails to perform as desired or as is now seen to be desirable
An assessment of the relative importance of the inadequacies of the effort
Adapted from Twidale et al.

11. (Orthogonal) Dimensions of evaluations
Structured
Less structured
Quantitative
Qualitative
Summative
Formative
Controlled experiments
Ethnographic observations
Formal and rigorous
Informal and opportunistic

12. Formative and Summative
Evaluation carried out for two reasons:
giving feedback = formative evaluation
grading the outcome = summative evaluation
“When the cook tastes the soup, that’s formative; when the guests taste the soup, that’s summative." (Stakes)
What you use to evaluate (formative) is often different from what others use (summative)

13. Summative == Metrics Things you can measure (numerical)
Things that are categorical
Could not do before
Faster, more complete, less mistakes, etc.
Wider range of users
What others for ESIP?
Measure or estimate the baseline before you start

14. Evaluation questions and associated data collection methods
Evaluation questions: To what extent does ESIP
Interviews/ Focus Group
Surveys
Document Analysis
Observation
Activities enhance end-user access to and use of data to advance science and education needs? X
Activities enable higher levels of semantic capability and interoperability such as explanation, reasoning on rules, and semantic query?
Contribute to the development and support of community resources, data systems and provision of results from diverse observing systems using suitable technologies? 15

15. Evaluation questions and associated data collection methods
Interviews/ Focus Group
Surveys
Document Analysis
Observation
What factors impede or facilitate progress toward ESIP goals? X
What progress has been made toward sustaining and ‘scaling up’ ESIP activities?
Does ESIP contribute to the reports on modern data frameworks, user interfaces, and science progress achieved?
Does ESIP incorporate user experiences in the evolution of its organization? 16

16. VOs - artifacts or processes?
Both – i.e. what is created, documented, saved as well as the things that were done along the way and by whom, and how, etc.
E.g. community building, inter-agency working activities, tutorials, …
VOs tend to deal better with artifacts than processes

17. Recall VO- coupled elements
Technology
Organizational
Structure
Source: DeSanctis and Monge
Communication Patterns

18. Some of this is, er…
Provenance - Origin or source from which something comes, intention for use, who/what generated for, manner of manufacture, history of subsequent owners, sense of place and time of manufacture, production or discovery, documented in detail sufficient to allow reproducibility (or assess value!)
Yes, apply this to ESIP – VO provenance
Adapted from Fox AGU FM 10 talk on translucency

19. And the reality? It’s about the questions that are being asked, e.g.
What exactly happened during that technical workshop last summer and how has it helped our agency?
Exactly what value do you (or we) get by you being the chair of a Committee (Working Group, Cluster)?
Adapted from Fox AGU FM 10 talk on translucency

20. The ecosystem?
These are what enable anyone to explore/ confirm/ deny their value perceptions or get quality answers to direct questions…
Accountability
Identity
Provenance
- Internal/
External
Explanation
Justification
Verifiability
Proof
Trust
Adapted from Fox AGU FM 10 talk on translucency
‘Transparency’ (the illusion of it)

21. Why an illusion?
It’s not that the word transparency is wrong, it is what it is being used for –
“If I let you see everything, you can get answers to your questions”
Nope, not unless you are very lucky…
It depends on
Who is asking the question (and why)
What the answer will be used for
CONTEXT and ROLE (poorly represented)
Producer solution to consumer need - Adapted from Fox AGU FM 10 talk on translucency

22. So … translucency See just what is necessary and sufficient
Practical definition
As close to the relevant data, information and knowledge artifacts presented in an appropriate form
Yes, I mean curation for evaluations and value assessments
Methodological means - many
Adapted from Fox AGU FM 10 talk on translucency

23. And some … Identity Accountability Documentation – are you happy Ted?
YOUR identity
Friends, organizations
Communities
Peer and non-peer relations
Accountability
By whom, to whom
When and how often
Documentation – are you happy Ted?
Adapted from Fox AGU FM 10 talk on translucency

24. Just remember
Plumbers fix things, they show up, the do the work, they leave and then you get the bill… they work in an analog world
Thus the value proposition is different (you all know that) in a digital world
So, we need a …

25. We need a Knowledge Base
Questions?
Who?
What/when/why/ how?
Answer: explain value of outcomes
VO Provenance
Descriptions of the artifacts
Specific terms/ language
Adapted from Fox AGU FM 10 talk on translucency

26. Populating the KB: Communication
A key feature of virtual organizations is a high degree of informal communication
Because of a lack of formal rules, procedures, clear reporting relationships, and norms, more extensive informal communication is required
-> as a result a lot of VO provenance is either
Not collected
Collected but fragmented, disconnected
Not identified as provenance (when it is created)

27. Opportunities
To undertake both formative (ESIP) and summative (Type Is) evaluations
Give some level of formalization to the meaning/ terms and relations – progress
An incentive to identify provenance of the VO that is ESIP
‘Rules’ to define the combinations of evidence required – value – ESIP and Is
Type Is - some use cases (state+value)
A few test cases, I/O metrics
Adapted from Fox AGU FM 10 talk on translucency
IMNSHO: VOs succeed more often when they are use case driven
Can be used for big goals, small goals, science, technical, policy, education, …

28. Comment: Translucent VOs
Along the spectrum from opaque (bad) to transparent (unsustainable) is a balance
Applies to leadership, communication, feedback
Posit: balance arises out of flowing requirements consistently to the working level of the VO and percolating direct experience and value of outcomes up
Usually depends on the right people – not everyone can evaluate (or be evaluated)

29. Bringing this together
A persistent presence and record of artifacts, activities and processes is needed: state and metrics/ value
No, this is currently not being done (or feasible)…
Facilitate sustained activities - support the notion of nested VOs by establishing their value (may change funding model)
But really - take ESIP from an organization to a ‘brand’ (like AMA) – then you know who they are gonna call (w/ apologies to Ghostbusters – that was the other story I could have told)
Light touch, roles responsibilities, outcomes, value – make them explicit

30. Thanks