1. Caught in the Web- How young people use the Internet to resolve civil justice problems Dr Catrina Denvir University College London

2. 2011 LSRC research looking at use of the Internet for civil justice problems Findings were counterintuitive (for some groups, i.e. young people) – High levels of Internet access but low levels of use for problems – Lesser success when used – Even worse for disadvantaged young people Fit with existing research: – Resource or ‘toy’ (Nicholas et al., 2003; EdComms 2007) – Willingness correlated with level of educational attainment (Iske, Klein and Kutscher, 2005) – Difficulty searching/obtaining information related to use of key words (Landauer 1992) Research Context

3. Aims of this Study What is going wrong? Finding appropriate information Getting to the right websites? Using appropriate search terms? Assessing trustworthiness of information / jurisdictional relevance? Cross-checking information from a number of sources? Search behaviour Getting frustrated or giving up easily? What the Internet was delivering Could the information found replace the need for advice? Was the internet acting as a standalone self-help resource?

4. Limitations of survey approaches People may think they remember: How long they spent online The websites they used (although most don’t remember this either) BUT….not reliable Detailed search history / order of search impossible Not possible to identify errors being made (e.g. inappropriate sources) Online experiments An experiment was devised in which young people were presented with legal problems and asked to use the Internet to resolve them Able to track exactly what they did, the errors they made and the impact of the Internet on knowledge and problem solving behaviour

5. Structure of Presentation 1.Introduce Experiment 2.How Young People searched online 3.Website behaviour 4.Impact of Internet use on knowledge of rights/strategy 1.What this means for policy 1.Moving Forward – Exploring the content/capability dynamic

6. Internet Experiment Two groups targeted School aged participants undertaking GCSE’s/A Levels (15-18 years old) University Students (from a range of disciplines) 208 participants in total Procedure Individuals were supplied with a username and password and assigned a specific ‘virtual desktop’ number Required to read instructions, log into the virtual desktop page, select and run their desktop, and open up the survey NO time limit Treat the hypothetical as if it was a problem they were facing themselves

7. Questions - Housing Alisha rents a house for one year. Discovers the bath has been leaking, causing damp. Landlord is asked to repair. Without providing any notice, the landlord visits the house one afternoon and lets himself in to inspect the leak. Q1: Is the landlord entitled to enter the house in this way? Q2: Is the landlord obliged to repair the leak? The landlord refuses to repair the leak. So, after three months ALISHA pays for the repairs and deducts the cost from the next rent payment. ALISHA encloses a note with the rent payment explaining what she has done. After the next rent becomes due, the landlord calls ALISHA and says that she must leave the house in 28 days time. He says she is in breach of the tenancy agreement by not paying the rent in full. Q3: Has ALISHA breached her tenancy agreement by not paying the rent in full? Q4: If ALISHA refuses to leave, will the landlord be able to evict ALISHA without first obtaining a Court Order saying that Alisha must leave? After 28 days, two employees of the landlord arrive at the house to help ALISHA move out. Q5: Do the two employees have the legal right to enter the property to remove ALISHA’S belongings? Suppose that before the 28 th day, the landlord had obtained a Court Order stating that ALISHA must leave the house by the 28 th day Q6: Would the two employees now have the legal right to enter the property to remove ALISHA’S belongings after 28 days have passed?

8. Questions - Employment ALISHA is 19 years old. She has been working 48 hours per week as an employee ZAP COMPUTERS for 1 year. She earns £5.50 per hour. Her manager, PAUL, says he needs her to increase her hours to 50 hours per week. ALISHA does not want to work the extra hours. PAUL shows her a part of her contract which says she can be asked to work up to 50 hours per week. Q1: Does ALISHA have to work 50 hours per week? Q2: Is ALISHA’S salary, £5.50 per hour, above, below or the same as the National Minimum Wage? Q3: Does the National Minimum Wage vary according to how old you are? ALISHA has been asking to see details of the main terms of her contract of employment since she started at ZAP Q4: Does ALISHA have a legal right to see the main terms of her contract of employment? One month later – when ALISHA has been working at ZAP for just over a year – PAUL tells her she is going to lose her job Q5: Is ALISHA covered by the full range of unfair dismissal laws? Zap’s personnel manager explains that ZAP is reducing the number of technicians it employs, and that ALISHA is going to be made redundant. The personnel manager tells her it is only fair that ‘the younger staff go first’. Q6: Is ZAP allowed to consider ALISHA’S age in deciding who is to be made redundant?

9. Internet Experiment- Virtual Desktop

10. Internet Experiment - stages Internet use and access Socio- demographics Housing scenario Questions on rights (1) Hint website Questions on rights (2) Advice preferences No hint Questions on rights (2) Advice preferences Employment scenario Questions on rights (1) Hint website Questions on rights (2) Advice preferences No hint Questions on rights (2) Advice preferences Tracked Activity

11. Overview Time Spent < 1 minute to < 43 minutes Mean = 11:46 (SD = 7.56) Only small differences (<1 minute) between groups on the basis of group assignment Searches Minimum = 0 : Max: 14 Mean 4 (SD = 2.9) 19 individuals performed no searches (going directly to the hint website, only one of these individuals did not receive the hint) Small differences between searching for those in hint/no-hint groups with the hint group searching less Number of searches did not differ between housing/employment groups

12. Simple Two to three words, lack of familiarity with the subject-area, or likely to yield very broad results, e.g. ‘average working week’ Commonly used by those in the housing group Specific Familiarity either with the subject area or with the way in which search engines handled search terms. Typically four terms long, ‘specific’ search terms were distinguished by their sophistication, e.g. ‘national minimum wage UK’, or ‘EU Working Time Directive’ More often used by those studying law at university Question Question format, closed-ended questions aimed at answering a specific element of the hypothetical problem the protagonist faced, e.g. if my contract says I can work up to 50 hours a week do I have a choice More common among those with housing questions, those in no-hint group and school students Types of Searches

13. 54% refined search terms one or more times (on initial failure of original search term) No participant went beyond the first page of search results 85.6% did not use search boxes within websites Respondents search terms belied a lack of understanding of the nature of the problem Law Students were often hindered by their knowledge Trial and error with little thought given to search terms that might yield the best results Search Behaviour If you are not first, you don’t feature……

14. Average of 9 webpages visited Employment group visited (slightly) more pages than the housing group Hint group visited a greater number of pages than the no-hint group University (law) students visited the most number of pages Websites Used 51.5% of individuals went to commercial websites 29.5% went to direct.gov.uk www.gov.uk which 22.5% of participants visited which transitioned to halfway through fieldwork 39% went to AdviceGuide.org.uk 27.5% visited the Shelter website 30% ‘other government website’ 30% ‘other third sector website’ 16.5% Yahoo Answers/ ‘e-How’/ Discussion Boards Smaller numbers (<7%) visiting online forums, news sites, blogs and acas.gov.uk School students more often used ‘mostly commercial’ websites

15. 35.5 per cent of participants were ‘browsers’ 44.2 per cent of participants were ‘dippers’ 8.6 per cent ‘mixed Users’ School students more commonly engaged in ‘dipping’ behaviour – inability to specify informational needs? Where individuals returned to the same website at a latter point in their browsing, this was often because it had shown up in search results again - brand loyalty? Website Interaction Use of the Hint Employment Hint Group - 66.7% of those supplied with the hint used it No-Hint Group - 44% went on to find the hint website during their searching Housing Hint Group - 70% of those supplied with the hint used it No-Hint Group - 40% went on to find the hint website during their searching

16. 19.6% visited a website with irrelevant content 41.5% visited a website with legal content intended for another jurisdiction Some realised their mistake and added ‘UK’ to the end of search terms Errors more common among school students age/inexperience? School students significantly more likely to arrive at websites with foreign content. 53 % of school aged respondents would seek information from one or more webpages outside of the relevant jurisdiction, v 29.7% of university (law) students and 36.3% of university (other) students Errors

17. Impact of the Internet on Knowledge Housing Before - Knowledge poor in regards to eviction without a court order, and whether the landlord’s employees can remove you from the property Uncertainty as to what constituted a breach of the lease After - improvement across the board, but uncertainty about who can evict a tenant Employment Before - Lack of knowledge in relation to maximum working hours, the minimum wage and who is covered by unfair dismissal After - Improvements in knowledge after use of the Internet, yet still unsure as to the rules surrounding unfair dismissal Improvements, but question dependent

18. Knowledge of rights and impact of the Internet

19. Impact of the hint Hint did not make a difference to score improvement Housing - hint group achieved lower mean scores after using the Internet than no-hint group Hint group viewed more webpages and went to a site where all the information was contained yet did not improve their scores as much no-hint group. How can we explain this? Those with the hint often did not go to other websites Searching for information in a direct fashion may be beneficial – assuming the right kind of websites appear in search results Those using the hint website sometimes used discussion boards – content issue? What is the purpose of directing people to a site? Avoid the pitfalls they would otherwise encounter if left to search alone – assuming that content on the site is delivered in an accessible format

20. Predictors of improvement Predicting score improvement on the basis of participant characteristics, search behaviour and website visitation found that: School students least likely to improve their scores University ‘other’ students most likely to improve their scores While not statistically significant, those adding UK to the end of search terms were associated with an increase in score, although its contribution was small. Other characteristics of searching were not statistically significant (e.g. hint/search type (simple, specific, question), search engine and interaction type (dipper/browser) Employment hypothetical, visiting direct.gov.uk was associated with score improvement, gov.uk was not. Housing hypothetical, visiting shelter.org.uk was associated with score improvement, other websites were not.

21. Where should Alisha go (if seeking advice)? Housing Broad range of sources of advice

22. Where should Alisha go (if seeking advice)? Employment Some changes, though no more coherent

23. Appropriate action Before using the Internet Good knowledge of CAB and the need to get legal advice But…. Expectation that there are other agencies/ombudsmen/courts that handle problems within an appropriate time frame Employment ‘Report it to an authority (home office?)’, ‘Talk to a government agency that deals with issues of workers' rights’ Landlord ‘Contact either the police or the local court and explain her reasons for not moving out or letting the employees in’ ‘Contact a solicitor or ombudsmen, keep any records or proof they tried to enter’ ‘Get ready to move out and find a landlord who isn't an [expletive]’

24. Appropriate action After using the Internet Greater emphasis on informal resolution in respect of employment Respondents fail to understand legal processes (particularly for housing) Answers do not reflect urgency (interpret with caution) Employment ‘Ask her trade union for help’ ‘She should ask a tribunal to decide if she has been discriminated against because of her age’ Landlord ‘Get an injunction from the court to stop the eviction notice’ ‘Contact Savilles’ (the estate agents) ’Leasehold Advisory Service’ ‘Consult the Rent Act’ ‘Get help from your tenants' association…bring up situation at next meeting.’ ‘…..She should contact a property lawyer….’ Increased knowledge of rights not translating to appropriate action

25. Confidence dealing with problems Internet may increase knowledge of rights, but does not increase confidence Knowledge does not equate to confidence/ability to handle alone…

26. Problem Handling Preferences How would you handle the problem? Retain preference for family/friends and advisers Little appetite for handling problems alone 62% would speak to one or both of their parents about the problem 88.5% said they would use the Internet to obtain information about rights

27. Summary Internet increased knowledge of rights Question dependent Many still got answers wrong, despite availability of resources Searches were structures around search engines Not as much searching within sites as might be expected Errors stemming from search terms Some lack of awareness regarding jurisdiction Little attempt to discriminate between sources of information Rigid adherence to order of search results In addition... Knowledge does not equate to appropriate action Little change in knowledge of relevant processes / appropriate action Internet did not increase understanding of advice sources Knowledge does not equate to confidence Continue to rely on family / friends / advisers, valuing their experience Seemingly little appetite for self-help

28. Implications For public legal education and self-help (in light of policy changes) Internet blunt instrument for PLE in the context of this experiment Knowledge does not equal confidence or competence with regard to action For design of websites The best designed websites will receive little attention if they are not first in search results (and young people did not take direction well) Future developments and website functionality Phrasing of search terms has implications for website designers Policy findings Problems likely to be exacerbated for vulnerable young people 1 st and 2 nd digital divide Less reliable support structures Internet appears to be being used as a diagnostic tool to confirm the nature of a problem before advice is sought from another source (parent/advisor). We do not know if this is to do with the content available online or the way it is perceived by audiences

29. Capability or Content – Search Experiment Final part of study looked at whether varying search terms (in relation to the hypothetical questions) led to different websites and websites of different quality. The results emerging from the internet experiment offer less comfort than might be hoped in respect of the capacity of a technologically proficient group of young people to devise search terminology. A lack of familiarity with the context of the hypothetical problem rendered participants predisposed to relying on cues drawn from the rights-based questions they were asked. Where such cues are not available (as may well be the case in everyday problem solving) young people may struggle to conceive of search keywords.

30. We know that ‘simple’, ‘specific’ and ‘question’ based search terms did not have a (statistically) significant influence on whether or not participants’ improved their knowledge following Internet use. But, do the different search terms lead to search results of a different nature or quality? In the search experiment study, 580 web pages examined – some of which were duplicates ‘Simple’, ‘Specific’ and ‘Question’ based search terms were devised for each one of the hypothetical questions Top ten Google and Yahoo search results for each search term/phrase were assessed against an evaluation framework Framework based on similar framework used by Advicenow in 2006 and more general indicators of quality/credibility drawn from the literature Search Experiment

31. Additionally Classified According to whether: Government/Third Sector/Commercial/Union/News Site/Discussion Board/Blog/Other

32. Mean problem support, skills support and credibility scores were typically higher for government and third sector websites than for other website types. Credibility and problem support scores were far higher for union websites than for any other type of website in relation to the employment hypothetical. In respect of the housing hypothetical, commercial websites tended to score far lower on problem support than commercial websites evaluated for the purposes of the employment hypothetical. Overall, irrespective of the type of website, all fared quite poorly in terms of skills support. Only a small number of websites (20.1%) met the criteria for ‘providing information specific to young people’, most often, these websites were provided by the government (58.1%). Search Results- Quality

33. Compared to use of a ‘simple’ search term, a ‘specific’ search related to a significant increase in the likelihood of a relevant result. There was also an increase in the likelihood of a relevant result for question based searches (when compared to ‘simple’ searches) though the difference fell well short of statistical significance. Compared to Google, Yahoo was significantly less likely to yield relevant results. A Google result could be expected to be relevant around 70 per cent of the time compared to 59 per cent for Yahoo. Google/Yahoo relevance is likely tied to the fact that the presence of commercial websites in the Yahoo result listings was in some cases more than double that of the Google listings for both the employment and housing searches. Question based search terms more commonly led to discussion board, ‘communities of knowledge’ type information. Search Results- Relevance

34. Search Engines are absorbing some of the ‘bad habits’ of young people when it comes to searching. Ten years ago a question based search would have been rejected, now search algorithms are being designed around these behaviours Although search behaviour does not have as much of an influence as expected in this field, this might be due to the relatively small number of websites operating in the legal advice space. Nevertheless, search behaviour can lead to less desirable websites and if individuals lack the capacity to distinguish between what is and what is not a reputable site, then this is not ideal. If it is expected that more young people will be using the Internet to self-help, then PLE and technological capability go hand in hand The idea that technology will compensate for any lack of capacity on the part of the user is appealing in its simplicity, but this simplicity ultimately underestimates the centrality of the user to the process of civil justice self- help. What can we conclude?