1. 1 TF-MSP May 2012 NREN Surveys for Assessing the Service Portfolio and Evaluating User Experiences/Demands (brief overview) Lajos Balint lajos.balint@niif.hu NIIFI TF-CPR meeting Reykjavik, 20.05.2012

2. 2 TF-MSP meeting at CARnet Present: fi, uk, dk, gr, nl, hu, fr, hr, ie, be, lv, si, nordunet, dante, terena 9 May: NRENs Brokering of Services for their customers 10 May: Client Surveys / User Satisfaction Assessment / Methodologies The big questions:What? Why? When? Who? How?

3. 3 FUNET Surveys- Harri Kuusisto Bi-annual survey – third one ended just now (42 questions) (+ feedback from annual Funet conf. + web pages eg.for polls) Respondent coverage: admin, tech, security, AAI 69-72-68% responses, web based (72% = 65 orgs, 100 answers) (from 80 orgs, 400 potential respondents) (universities better response: 88%) Topical coverage: quality, satisfaction, requirements wrt. network, service portfolio, features, security, AAI, admin. Comparison with earlier responses (trends, new requirements …) NPS first time: % of 9-10 pts - % of 0-6 pts = 65,5 % (good, but …) Definite difference between university vs. polytechnics answers Completion: 15…60 mins (39 crossings + 3 short texts) Specific reports automatically generated

4. 4 CARNet Surveys - Barbara Kolarek Annual survey since 1997 (contractual obligation for the members!) Appointed representative at each respondent 241 / 432 and 1383 / 1798 organisations / sites (academic and schoolnet) Structure: like TERENA Compendium (but also qualification/evaluation - 6 thematic areas) Output: automatic graphs (go to web, CARnet depts, institutes, ministry) Forward and backward information 150 questions (80% of respondees respond but mostly incomplete answers!)

5. 5 SURFnet Surveys - Walter van Dijk Annual survey (responded by ICT mgmt of connected 156 institutions) Since 2009, standard set of questions, 10 mins to answer online About 80 responses (~50%, at each category between 40 and 60 %) Acutal work outsourced Mostly quality and satisfaction related qualitative questions (”quantified measures”) NPS = 34 % (= 36% - 2%), universities just about 20 % … (but average 2011 score is 8.2!) Corollaries: more consultancy, attention, support, excellence needed + campus level activities to be improved Uptake of 14 services also measured (from 100% SURFcert to 20% SURFconnect)

6. 6 HEANET Surveys - Peter O'Halloran 65 clients, 1M users incl.schools Diverse voices and perspectives (Universities, institutes, schools, others) 33 products/services (connectivity + hosting incl.MM + MW + contract mgmt) Multiple ways of communicating with customers (Board, requirements, reviews, polls, WGs, conf., WSs, NOC …) Goal: evaluation/improvement: performance, services (evaluate+introduce), issues, common knowledge All clients (one per each), e-mail, anonymous to the public Define (Sep 11), Collect (Oct 11), Communicate (Nov 11 – March 12!) Method: Survey-Monkey, consolidated by XL sheets and graphs Questions about: HEAnet, services, NOC, user satisfaction 68 % response rate (58% very satified, 37% satisfied, 5% neutral wrt. quality) (38% strongly satisfied, 57% satisfied, 5% neutral with value for money) (+ many more results) Challenges/issues mentioned (by priority order): Budget/resources, mobility, cloud, demand for new services, data, others Continue, simplify, third party (?), timing (?), engage non-respondents

7. 7 NIIF/HungarNet Surveys - Lajos Balint Picture is less pleasing than with the previous presentations (Responses: sometimes valueless, sometimes nothing new) (+ many outside surveys - TERENA, DANTE, EU projects …) No personnel for surveying, no budget for outsourcing – difficulties Dilemmas and crucial aspects - motivation, frequency, exploitation - coverage (thematic/respondent), granularity, complexity Survey I: Wide SP coverage, external request, single execution (2010-11) 2700 invited units (Y/N + annotation) - 90 moderate responses (~3%) (however well exploitable output) Survey II: Narrow (VoIP+VC) coverage, biannual execution (commensurable results) 20-30 invited institutes - 80-100 % high value responses (expert, responsible, motivated respondents)

8. 8 BELNET Surveys – Koen Schelkens 102 from 187 (55%) responses Completing: ~30 mins Overall Net Promoter Score: NPS = 65 % (68-3) Customer Experience Index: >85 % Satisfaction results: excellent Strengths and weaknesses evaluated SWOT analysis performed Strategic conclusions derived Many more questions involved (percentage results on importance and satisfaction derived)

9. 9 Experiences - corollaries Result depends on motivation-devotion(both sides) Proper objective and careful preparation allow success Experts prefer demanding surveys, unlike non-experts Even very low (<10%) response level can be usable Repeated surveys – commensurability is a crucial Interactive on-line surveys should have multilevel memory Allocate time-energy-cost also to due post-processing Consider overall cost/benefit (resources vs. influence) Successful survey not yet guarantees attained impact!

10. 10 Consequences for TF-CPR Surveying is just one way of communicating with customers (Board, requirements, reviews, polls, WGs, conf., WSs, NOC …) CPR can/should well complement other ways of interaction Common goal: improving: performance, services (evaluate+introduce), issues, common knowledge While surveys ask, CPR should inform the users/stakeholders Diverse target groups need different handling when surveyed (research vs. education … communities, experts vs. non-experts …) CPR also should apply different (audience-dependent) approaches Small results also can help a lot (low response rate in surveying – narrow segment of potential reachable community) Gradual extension of target community leads to stable CPR impact Allocate time-energy-cost to all activity components (preparation - execution - post-processing both in surveying and CPR) Duly prepare-disseminate-evaluate CPR materials and effects Consider overall cost/benefit also wrt. CPR resource spending vs. influence Successfully completed CPR actions not yet guarantee attained impact!

11. 11 Conclusion common goals complementary activities equivalent principles same target communities similar problems and solutions … Another reason why TF-MSP and TF-CPR should cooperate