1. Seeing the universities’ digital environment through the eyes of staff and students Advance HE’s Surveys Conference 8th May 2019
Ruth Drysdale, Jisc

2. A quick fun survey http://bit.ly/AHE2019
Name:
University:
Job title:
Has your University run Insights (aka Tracker) before?
Digital super power:

3. investments in digital learning as well as physical campus
Why evaluate the impact of transformational technologies with the student and staff users?
investments in digital learning as well as physical campus
within 20 years, 90% of jobs will require digital skills
users need to understand and consent to how data about them is captured and used
Universities’ make large investments in their digital learning and teaching environments as well as their physical campus, but how is the digital environment evaluated to ensure it enhances the overall student experience and the institution gets its return on investment?
It is anticipated that, within 20 years, 90% of jobs will require digital skills, so it’s important that universities are in a good position to prepare today’s students for tomorrow’s workplace. Understanding how students use technology and their attitudes towards its use in learning is a good place to start.
With the emergence of initiatives like the learner analytics, we need to ensure that students and staff are aware of, and engaged with the Universities’ digital environment. This is in part, so they understand and consent to how data about them could be captured and used.

4. How do you evaluate the impact of transformational technologies with the student and staff users?
With the high investment in the digital teaching and learning environment and supporting infrastructure, how do you assess if the investment is being made in the right places and realising benefits to your overall student experience?
This was a question asked by the UCISA’s Heads of IT community to Jisc’s Student experience team. Based on our previous research, we developed and piloted over 3 years with over 100 UK and FE institutions a set of questions for students and teaching staff about their holistic experience of the institution’s digital learning environment.

5. Are not an audit of tech tools
Insights surveys are; .
Are not an audit of tech tools
In 2018 issues included providing support for students to develop the digital skills they need to thrive in the workplace, providing health and wellbeing services online (including online safety), and transparency about the use of students’ personal data.

6. What is digital experience insights?
Student and staff surveys, made up of:
Questions with answers that can be benchmarked
Questions with free text answers for local analysis
Add a few of your own local questions
Compare student feedback with teaching staff views and organisational factors
A student and staff engagement process, supported by our guidance
A community of practice around the insights process and findings (including student representatives)

7. Teaching staff insights 2018 pilot report http://bit.ly/DEI-reports
1,921 teachers surveyed across 15 institutions as a pilot
Four colleges and 11 universities across England, Scotland and Wales piloted the surveys
376 college responses
1,545 university responses
The survey contained 17 items (mini-survey) or 47 items (full version)
Core questions were benchmarkable
Customisable questions
Sarah to present

8. UK Student insights report 2018
37,720 students surveyed across 83 UK institutions
Our student survey collected data from a total of 37,720 students
14,292 FE learners
23,428 HE students
36 FE colleges,4 sixth form colleges and 43 universities – approximately 16% of UK colleges and 30% of UK universities
Report of 2018 insights findings: ji.sc/dig-exp-insights- survey-18
At a glance: summary of 2018 insights findings: ji.sc/dig- exp-insights-summary-18
Digital experience insights
As an outcome of the digital student work and the need to gather quantitative data on students digital experience at an organisational level and at a sector level, we developed the student digital experience tracker as a survey tool with a robust set of student tested questions delivered in BOS. See
This evidence supports discussions with senior managers
The report containing the summary findings from 2017 surveys will be available from 20th June from web link on this slide.

9. How do you avoid ‘survey fatigue’?
In groups discuss:
How do you gather student and staff feedback, especially about their experience of the digital learning and teaching environment?
How do you avoid ‘survey fatigue’?
How do you analysis the different data sources?

10. What the students are telling us
Although 50% of FE and 69% of HE students thought digital skills were important for their chosen career…..
Only 41% of FE and HE students felt their course prepared them for the digital workplace.
Only about a third of students agreed that they were told what digital skills they would need before starting their course.
About 40% of FE and HE students agreed that they had regular opportunities to review and update their digital skills.
These findings from our 2018 Jisc digital experience insights report provide the evidence of what students are telling us - although 69% of HE learners & 50% FE learners recognise that digital skills are important in their future career, only 41% feel that their course is preparing them for the digital workplace.
And only a third felt that they were aware of the digital skills their course required.
Only 41% of FE and HE students felt they had regular opportunities to review and update their skills.

11. Using persona analysis to compare student social behaviours with institutional digital provision
quantitative analysis of their typical applicant pool (geographic location, average GPA, average SAT scores, ethnicity, family size, family income, etc). This is a great start on the who, but doesn’t cover the what, where, why or how of

12. Question and answer text
Question focus
Question and answer text
Student opinions
In your own learning time, how often do you use digital tools or apps to ...discuss your learning informally on social media (weekly or more, monthly or less, never)
Digital skills are important in my chosen career (agree, neutral, disagree)
When digital technologies are used on my course ...I understand things better (agree, neutral, disagree)
When digital technologies are used on my course ...I feel more connected with my lecturers (agree, neutral, disagree)
When digital technologies are used on my course ...I feel more connected with other learners (agree, neutral, disagree)
Which best describes your preferences as a learner? (on my own, mix, group learning)
How much would you like digital technologies to be used on your course? (more, same, less)
When digital technologies are used on my course ...I am more easily distracted (agree, neutral, disagree)
Course opportunities for digital learning
As part of your course, how often do you do the following digital activities? Work online with others (weekly or more, monthly or less, never)
Student demographics
How old are you? (Under 19, 19-24, over 25)
What gender do you identify as? (male, female, other)
Do you use any assistive technologies to meet your learning needs? (e.g. screen readers, voice recognition, switches) (Yes vital to learning needs, optional to learning no)
Institutional ratings
Overall, how would you rate the quality of this institution's digital provision (software, hardware, learning environment)? (Best imaginable to worst imaginable)
Overall, how would you rate the quality of digital teaching and learning on your course? (Best imaginable to worst imaginable)
Using persona analysis to compare student social behaviours with institutional digital provision

13. UK HE digital student personas
Data were re-coded and analysed using a multiple correspondence analysis (MCA) using R
Positive connecteds
Neutral, happy with status quo
Negative and technophopic
Digitally distracteds
The MCA variables plot for 4,000 randomly selected university (HE) students from the insights surveys
The analysis identified three potential ‘digital persona’ groups visible in both the university and college student data, and an additional one on the university data plot. These are summarised as follows:
Positive and connected (HE and FE)
They rated teaching and learning AND digital infrastructure very highly (choosing to answer either ‘excellent’ or ‘best imaginable’ from the seven-point scale)
They agreed that use of digital technologies on their course helped them feel more connected with their lecturers and with other learners
They reported that on their course they work online with others ‘weekly or more’
When digital technologies are used on their course they agreed that they understand things better
They want digital technologies to be used more often on their course
They tend to discuss learning informally on social media ‘weekly or more’
They feel that digital is important for their chosen career
Users of assistive technologies fell within this persona group
Neutral and happy with the status quo (HE and FE)
They feel that teaching and learning AND digital infrastructure are good or average
They have a neutral opinion when asked whether digital is important for their future career, whether it helps them to understand things better, whether they find technology use more distracting, and whether use of digital technologies on their course makes them feel more connected to lecturers and other students
They reported that on their course they work online with others ‘monthly or less’
Negative and technophobic (HE and FE)
They don’t feel technology is useful for their career
They do not feel more connected with learners or lecturers when digital technologies are used on their course
They give low rating scores to institutional digital teaching and learning and digital infrastructure (choosing either poor, awful or worst imaginable)
They want less technology used on their course, and they don’t feel it helps them to learn
It was of interest to see that student age, student gender, and desire for working solo or a mix of solo and group work didn’t fall within any of these three groups and instead were largely independent of them (with one exception, identified below).
It was also of interest to identify a fourth persona cluster in the University data that included students with a preference for group work. This was not present in the college student data set. This group can be described as follows:
Digitally distracteds (HE only)
When digital is used on their course, these students feel more easily distracted from their learning
They have a preference for learning in groups (rather than solo or in a mix), but they say that their course ‘never’ provides them with opportunities to work with others online
They also report that they never informally discuss their learning on social media
They tend to rate institutional digital teaching and learning and digital infrastructure as good or average
They tend to be aged over 25
Other than this additional persona type, the variable clusters were very similar for university and college students, reflecting previous findings from the initial survey analysis and subsequent Factor Analysis.

14. UK HE digital student personas
Positive and connected
Neutral and content with the status quo
Negative and technophobic
Digitally distracted
The analysis identified three potential ‘digital persona’ groups visible in both the university and college student data, and an additional one on the university data plot. These are summarised as follows:
Positive and connected (HE and FE)
They rated teaching and learning AND digital infrastructure very highly (choosing to answer either ‘excellent’ or ‘best imaginable’ from the seven-point scale)
They agreed that use of digital technologies on their course helped them feel more connected with their lecturers and with other learners
They reported that on their course they work online with others ‘weekly or more’
When digital technologies are used on their course they agreed that they understand things better
They want digital technologies to be used more often on their course
They tend to discuss learning informally on social media ‘weekly or more’
They feel that digital is important for their chosen career
Users of assistive technologies fell within this persona group
Neutral and happy with the status quo (HE and FE)
They feel that teaching and learning AND digital infrastructure are good or average
They have a neutral opinion when asked whether digital is important for their future career, whether it helps them to understand things better, whether they find technology use more distracting, and whether use of digital technologies on their course makes them feel more connected to lecturers and other students
They reported that on their course they work online with others ‘monthly or less’
Negative and technophobic (HE and FE)
They don’t feel technology is useful for their career
They do not feel more connected with learners or lecturers when digital technologies are used on their course
They give low rating scores to institutional digital teaching and learning and digital infrastructure (choosing either poor, awful or worst imaginable)
They want less technology used on their course, and they don’t feel it helps them to learn
It was of interest to see that student age, student gender, and desire for working solo or a mix of solo and group work didn’t fall within any of these three groups and instead were largely independent of them (with one exception, identified below).
It was also of interest to identify a fourth persona cluster in the University data that included students with a preference for group work. This was not present in the college student data set. This group can be described as follows:
Digitally distracteds (HE only)
When digital is used on their course, these students feel more easily distracted from their learning
They have a preference for learning in groups (rather than solo or in a mix), but they say that their course ‘never’ provides them with opportunities to work with others online
They also report that they never informally discuss their learning on social media
They tend to rate institutional digital teaching and learning and digital infrastructure as good or average
They tend to be aged over 25
Other than this additional persona type, the variable clusters were very similar for university and college students, reflecting previous findings from the initial survey analysis and subsequent Factor Analysis.

15. Who does it? (role/department) When do they do it?
In groups discuss:
How do you analyse student and staff feedback especially about their experience of the digital learning and teaching environment?
Who does it? (role/department)
When do they do it?
Who do they report to? (which committees)
How do you monitor year on year progress?

16. Jisc NUS roadmap for supporting students in improving their digital study experience
Outstanding
Developed
Developing
First Step

17. Who pays for it? (role/department) How do they evaluate it?
In groups discuss:
How do you make enhancements to the student and staff experience of the digital learning and teaching environment?
Who pays for it? (role/department)
How do they evaluate it?

18. Institutional insights project – fully guided
4. Enhance student
digital experience
1. Plan
3. Analyse
Digital Experience Insights
Student
Teaching Staff
Senior Champion
Project team
Working group
Stakeholders: Student Union, IT, library, academics
Benchmark
Recommendations
Change initiated
Feedback to respondents on changes
Enhance services
2. Survey
Report to
Student experience
Governance
groups
Governance
Communication plan
Budget
Sampling cohort
Prizes?
Surveys start
& end dates
Design local
questions
Link to other datasets eg NSS, modular feedback
Survey set up
in BOS
Quantitative and Qualitative analysis
Lunch surveys
BOS or export
Into Excel/
SPSS/NVIVO
Monitor responses
Why – what is your rational?
Who – key stakeholders to benefits from the findings?
How – survey set up and communications plan?
When – best time of year to run the surveys?
What – run surveys, analyse that data, report back
Our guidance will help you at every stage of your insights surveys – from planning and implementation to engaging respondents, data collection and analysis as well as responding to your findings go to;

19. Join our community
next community of practice event on 22 May 2019 at the Studio, Birmingham
lists:
for
for Follow #digitalstudent
and
HE students
Teaching staff
Digital experience insights

20. Get in touch… Digital experience insights
Ruth Drysdale
(put ‘digital insights’ in the subject line)
Call
between 07: :00 (Mon - Fri)
Or your Jisc account manager
Except where otherwise noted, this work is licensed under CC-BY
Digital experience insights