1. 1 NIIF-Hungarnet Assessing the Service Portfolio and Evaluating User Experiences/Demands by Dedicated Static On-Line Surveys at NIIFI Lajos Balint lajos.balint@niif.hu NIIFI TF-MSP meeting Zagreb, 09-10.05.2012

2. 2 Dilemmas and options The need of surveys (assessment and/or evaluation) The way of surveying (motivation and methodology) Living with and without surveys (NREN vs. users) The big questions: What? (coverage, complexity, extent) Why? (direct, indirect, and final objectives; external/internal) When? (before-during-after preparing/executing activities) Who? (who talks to whom, what expertise/responsibility level) How? (traditional/online and static/dynamic/interactive)

3. 3 Some crucial details Level of granularity (high-low: topics/users) Width of coverage (wide-narrow) Technique (traditional-online) Adaptivity (static-dynamic-interactive) Complexity (coverage vs. expertise) Output (availaility, accessibility) Frequency (single – repeating – periodical)

4. 4 Recent NIIF/Hungarnet surveys (NIIFI) Background: ISO 9000 relations (external) NDP project relations (external + internal) Internal intentions (internal) Execution: Wide vs. narrow topical coverage (overall – VoIP) High vs. low granularity (3000 – 30 invitees) Single vs. periodical surveying (2010 – 2005+2n) Static/dynamic on-line non-expert/expert responses Internal exploitation, on-demand external access

5. 5 Survey I – 2010-2011 Wide SP coverage – discovering demands (RN [IP, λ], HPC, storage, cloud, VC, VoIP) Single execution (2010: NDP, ~2005: ISO) 2700 invited respondents (Y/N + annotation) (institutes, departments, research groups …) 90 incoming responses, mostly moderate value (applicable evaluation well supported) Started in 2010, concluded in 2011 External motivation (meeting internal interest) (request by funding organisation)

6. 6 Survey II – 2005,2007,2009,2011 Narrow coverage – discovering directions (VoIP only, extended to collab.in 2009 by VC) Periodic execution (commensurable results  fine tuning possible) 20-30 invited respondents (institutes, sometimes departments) 80-100 % incoming high value responses (expert, responsible respondents) Motivated by internal interest (+ high respondent willingness)

7. 7 Survey II – additional info Evaluation (VoIP experiences): (net: size, call types in/out, technology, additional operator, cost, …) (use:, quality, support, invoicing, errors/failures, repair cycles, …) Development needs (VoIP demands) (improve support; improve invoicing; involve more institutes; provide directory; apply selective promotion – user by user; introduce novel/additional technologies; …)

8. 8 Experiences - corollaries Result depends on motivation-devotion(both sides) Improper objective or preparation prohibits success Experts need demanding surveys, unlike non-experts Even very low (3%) response level can be usable Complex surveys mustn’t be frequently repeated Repeated surveys should be commensurable Interactive on-line surveys should have multilevel memory Allocate time-energy-cost also to due post-processing Resource demand: prepare + n * complete + n k * evaluate Consider overall cost/benefit Surveys sometimes can save but always do cost money!