Copyright © Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Jeff Place, US Sales Manager Greg Pope, Analytics and Psychometric Manager NCTA – San Antonio Thursday Aug. 6, 2009, 3-4pm

Copyright © Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. 1. Brief introduction to Questionmark products and services 2. Case studies from four of our customers a.What their test security needs were b.How our software addressed those needs c.Demo of solutions 3. Questions and closing

Copyright © Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.  There are no shortage of test security challenges  At each stage of the assessment process (Authoring, Scheduling, Delivery, and Reporting) solutions exist to meet security challenges  We hope to show you today how our product can meet your test security needs by walking through a typical examination process and some real life examples Slide 3

Copyright © Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Slide 4  Founded in 1988  Technologies and services that enable organizations to measure knowledge, skills, and attitudes securely for certification, regulatory compliance and improved learning outcomes – Assessment Management System  More than 2,500 customers have purchased Perception ◦ For Corporate, Government, and Education markets  More than 14,000 authoring systems installed  Offices in US, UK, Belgium, Germany, Mexico  Available in English, Spanish, German, French, Dutch and Japanese

Copyright © Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Slide 5 Create questions and group into quizzes, tests, exams and surveys using a web browser or Windows PC Add, import, change, and delete participants, groups and their schedules Deliver assessment using a standard or secure browser, PDA’s, or using a Windows PC Report on the results. Reports include coaching reports, item analysis, gap analysis, and others.

Copyright © Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Slide 6 Financial Services Health / Pharma RetailHigh Tech / Manufacturing TelecommunicationsEnergy / Utilities Public SectorEducation

Copyright © Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Slide 7  Security impacts ◦ Validity of assessment results ◦ Fairness to participants  One approach: security rights that are role-based ◦ Set access permissions by role or profile ◦ Limit an author’s or reviewer’s access to the various functions ◦ Limit access to topic (item) folders ◦ Limit what the author may do in those folders

Copyright © Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Slide 8 Wizards to create 22 item types  Drag and Drop  Essay  Explanation  Fill in Blanks  Hotspot  *Likert  Adobe Flash  Adobe Captivate  Matching  Knowledge Matrix  Survey Matrix  *Multiple Choice  *Multiple Response  Numeric  File Upload  Pull-down List  Ranking  Select a Blank  Spoken Response  *Text Match  *True/False  *Yes/No  Enable authors to create more interesting and probing questions  Reveals more information during an assessment in a shorter time  Not offered by simpler assessment tools in LMS’s  Enable authors to create more interesting and probing questions  Reveals more information during an assessment in a shorter time  Not offered by simpler assessment tools in LMS’s

Copyright © Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Slide 9  Create participants and schedule assessment delivery  Manage participants individually or in hierarchical groups  Schedule access to assessments: ◦ Limit dates ◦ Limit attempts ◦ Require proctor/invigilator ◦ Deliver to certified test center ◦ Online or offline delivery  Broadcast: Notifications to Participants

Copyright © Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Slide 10 Secure Browser Blended Delivery Options Web Browser Questionmark to Go Player Questionmark to Go Player Integration Windows / CD Test Centers Printing and Scanning Mobile Device

Copyright © Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Slide 11  Stops people from printing questions  Stops people from typing in their own URL  Always display in full screen and it’s not possible to maximize or minimize  Stops people starting a new task  Does not provide menu options or icons  Disables control keys and task switching  Disables right-click menu options  Disables the ability to start new applications  Prevents going backwards to a previous page  Stops people exiting in a high stakes, proctored, environment  Hides the HTML source  Provides an API to control certain functions of a browser from the server  Server can command Questionmark Secure to display a toolbar Secure Browser Regular Browser

Copyright © Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Slide 12 Ensures Test Takers use a compatible browser

Copyright © Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.  All participant scoring is conducted on the Perception server (rather than on client/participant PCs)  Scoring algorithms are not sent in any way to participant PCs  Perception can be configured to use SSL to ensure communication between participants and the server are secure  Data in item repositories are encrypted (QML is encrypted) providing added item bank security Slide 13

Copyright © Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

 Question by Question (QxQ) delivery options limit the amount of content shown at one time  Randomization of questions and choices to prevent cheating

Copyright © Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.  Score List Report ◦ IP addresses (as expected?) ◦ Time taken (too quick, too long?) ◦ Easy flagging of participants who get perfect scores for further investigation  Coaching Report ◦ Drill down on an individual participant, comparison to benchmark, use in appeal process  Item Analysis Report ◦ Determine psychometric performance and examine item difficulty drift over time  Test Analysis Report ◦ Assessment defensibility review and test difficulty drift Slide 16

Copyright © Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Case studies from four of our customers a.What their test security needs were b.How our software addressed those needs c.Demo of solutions Slide 17

Copyright © Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.  Multi-campus community college based in Iowa with about 15,000 students  Kirkwood offered surgical technician programs to a large base of students through distance learning  Security requirements arose related to the standardized, secure, administration of assessments across the multiple remote campuses 1.Authoring to be controlled by faculty at the main campus 2.Delivery of assessments to be standardized across the remote locations to ensure that student experiences were comparable 3.Assessments must be locked-down and protected so that content cannot be shared/exposed  rgical.aspx rgical.aspx

Copyright © Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.  Solutions offered to meet requirements: 1.Authoring: Centralize secure authoring platform (Authoring Manager), Easy remote authoring tools (Questionmark Live) 2.Delivery: Templates ensure equal participant experience regardless of location 3.Locked down/protected: Monitor/proctor and participant logins Demos ◦ Questionmark Live ◦ Monitor/proctor/test center login and participant logins ◦ Version 5 templates -

Copyright © Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.  Technical college located in Rome, GA  The home to many of the health education programs offered by the Georgia Dept. of Tech and Adult Education system including Ultrasound / Sonography  Learn how to read Ultrasounds and recognize problems  After the 18-month curriculum, students must be prepared to take their National Certification Exam  Issues to resolve and security requirements 1.Online testing capability to closely resemble the National Exam 2.Paper and pencil were too expensive and difficult to add graphics 3.Ability for students to “mark” an identified area of a graphic 4.Control who sees what test and when 5.Lock down the computer to eliminate printing, instant messaging, browsing the internet during the tests

Copyright © Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Solutions offered to meet requirements: 1.Provided online testing solution and eliminated all paper and pencil tests to reduce costs 2.22 Question types including Hotspot to resemble the National exam with the ability provide feedback. 3.Manage large item banks of questions including security for collaborative authoring among faculty (5,000 items for mock exams) 4.Lock down browser 5.Scheduling (right test to right student), Demos 1.Hotspot question type & feedback text/images 2.Item Banking and security for collaborative authoring 3.Questionmark Secure 4.Scheduling  Slide 21

Copyright © Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.  14,000 students located in Mobile  College of Medicine offers courses in various specialties to students in their first two years of medical school  Students must be prepared to take medical certification exams  Security requirements: 1.P&P to online test security 2.Secure testing in computer labs 3.Reporting to facilitate security  uth_alabama.aspx uth_alabama.aspx

Copyright © Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.  Solutions offered to meet requirements: 1.P&P to online test security: Central secure storage of item banks, controlled delivery via secure channels 2.Secure testing in computer labs: Questionmark Secure, shuffling, randomization, proctor/student logins 3.Reporting: Many reports to ensure secure testing Demos 1.Score list report: Time taken, IP address 2.Coaching Report: Appeals, comparison to benchmark 3.Item Analysis Report: Item difficulty drift over time and psychometric performance 4.Test Analysis Report: Defensibility and test difficulty drift

Copyright © Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.  Department of Nursing in Syracuse, NY  2 year program  Mission to prepare certification exam (NCLEX)  Issues to resolve and security requirements 1.Remote testing 2.Lock down the computer to eliminate printing, word processing programs, browsing the internet during the tests 3.Randomization 4.Remediation capabilities

Copyright © Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Solutions offered to meet requirements: 1.Had to switch from P&P to online (QM used for over 100 exams) 2.Secure browser provides more security in non-proctored locations (uses video surveillance) 3.Shuffling choices 4.Random administration of questions 5.Randomize within block 6.Limiting the amount of attempts to take tests (4 tries) 7.Host with us or in your secure environment (accessible) Demo 1.Shuffling choices 2.Random administration of questions 3.Randomize within block  Slide 25

Copyright © Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.  Thank you for your time and attention!  Questions/Comments?  Visit our booth to talk to us more  We have a whitepaper on this topic (Delivering Computerized Assessments Safely and Securely) ◦ Visit our website to download:

Copyright © Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Jeff Place Greg Pope