1. Trusted Tester and Agile
Presented by:
DHS OAST: Al Hoffman, Cynthia Clinton- Brown
DOJ: Kathy Eng
DHS USCIS: Ken Moser
2016 Section 508 Interagency Accessibility Forum- NIH October 11, 2016

2. Federal Government 508 Unification Agile Implementation
Agenda
DHS Trusted Tester
Trusted Tester Beyond DHS
Federal Government 508 Unification
Agile Implementation
Section 508 Community of Interest

3. The Why, What, and How about
DHS Trusted Tester

4. Common Section 508 Problems
Inconsistent testing within one approach
Different test results for the same standard
Unable to accept other’s results
Mixed messages for vendors and developers

5. How was the VPAT tested? What test approach was followed?
What tools were used?
Assistive technology
Code evaluation
Both?
None of the above??
In 2008, would your agency accept these test results?

6. One Product: 2 Agencies, 2 Results
“JAWS reads text, images and table on the Web page.”
“Data table doesn’t have TH for column and row headers.”
Assistive Technology Test Result
What 508 standard is this?
What version of JAWS?
What criteria determined that JAWS read the table?
Result: Compliant
Code Review Test Result
Was this found by viewing the source code?
What tools were used?
Result: Not Compliant

7. DHS Test Process and Training History
Developed a DHS Test Process
Needed a 508 test process for OAST testers
Something reliable and generated repeatable test results
Software and Web Application standards
Created Training for DHS
2 day instructor led, hands-on training
Manual Test tools included JAWS
No exam

8. DHS Trusted Tester (version 2.x)
DHS Test Process instructions rewritten for larger training audience
Minimize subjective outcomes
Easier to use testing tools
Code-based inspection (no Assistive Technology)
Training updated:
Added pre-requisites courses
“Trusted Tester” was born
Passing grade on exam meant the Tester had the acceptable skill level to test and report accurate results.

9. DHS Test Process (Version 3.x)
What you're testing?
Why you're testing?
Specific step-by-step test instructions
Failure conditions, along with the standard(s) that fail
Clear conditions for marking results:
Does Not Apply; Not Compliant;
or Compliant

10. Trusted Tester Tools Tool Purpose Object Inspector 32 Java Ferret
Reveals the Name, Role, State and Value of software elements.
Java Ferret
Reveals the Name, Role, State and Value of Java software elements
Web Accessibility Toolbar (WAT)
Adds a toolbar to Internet Explorer to aid manual inspection of accessibility related elements on web pages.
ARIA Markup Favelet
Adds a JavaScript function that highlights all ARIA code on the page
Frames Favelet
Adds a JavaScript function that marks up frames and iFrames on the page with name/title; added for use in IE

11. Name, Role, State, Value of Windows SW
Object Inspector
Name, Role, State, Value of Windows SW

12. Web Accessibility Toolbar (WAT)
Reveals web element properties for review

13. Identifies elements with ARIA markup
ARIA Markup Favelet
Identifies elements with ARIA markup

14. Identifies Frames, iFrames and their descriptions
Frames Favelet
Identifies Frames, iFrames and their descriptions

15. Trusted Tester Training Series
Course ID
Course Name
FAC 049
Section 508: What is it and Why is it Important to You?
FAC 050
Section 508 Testing Tools Installation*
FAC 051
Section 508 Standards for Applications
FTE 250
Section 508 Trusted Tester Training
FTE 251
Section 508 Trusted Tester Certification Exam
* Requires tools to be installed before taking the FTE250

16. OFAC 049: Section 508: What is It and Why is It Important to you?
Introduction course to educate learners on the reason Section 508 is important and required by law
Prerequisite: None
Duration: approximate one hour

17. OFAC 049: Course Objectives
Explain what Section 508 is and why it is important
Explain how Section 508 conformance can make EIT more accessible to persons with disabilities
List job-related responsibilities for meeting Section 508 Standards
List available resources to help you meet Section 508 Standards

18. OFAC 051: Section 508 Standards for Applications
Basic course to educate learners on the functional and technical standards that apply to web and software applications
Prerequisite: OFAC 049 Section 508: What is It and Why is It Important to you?
Duration: Four to six hours

19. OFAC 051: Course Objectives
Identify the purpose and criteria of Section 508 standards for applications including the Law, Purpose and which disabled persons are impacted
Identify methods that make interactive and non-interactive elements accessible based on functional performance criteria
Identify standards that software and web applications have in common:
Interactive Interface Elements
Non-Text Interface Elements
Color and Flickering
Identify Software specific and Web application-only standards

20. OFAC 050: Section 508 Testing Tools Installation
Web-based job aid to ensure students successfully install the tools required to operate the advance courses and perform as a Trusted Tester
Prerequisites:
OFAC 049 Section 508: What is It and Why is It Important to you?
OFAC 051 Section 508 Standards for Applications
Duration: Tool installation time varies due to Admin access requirement for most students

21. OFAC 050: Course Objectives
Identify and install the tools used to test web content and software applications:
Inspect
Java Ferret
Web Accessibility Toolbar (WAT)
ARIA Markup Favelet
Frames Favelet
Named Anchor Bookmarklet
Successful completion of the approved tools enables users to enroll and complete the testing examples for the advanced Trusted Tester course (OFTE 250) and Certification Exam (OFTE 251)

22. OFTE 250: Section 508 Trusted Tester Training
Advanced course to educate students on detailed requirements of the DHS Section 508 Compliance Test Process for Applications
Prerequisites:
OFAC 049 Section 508: What is It and Why is It Important to you?
OFAC 051 Section 508 Standards for Applications
OFAC 050 Section 508 Testing Tools Installation
Duration: Over 40 hours

23. OFTE 250: Course Objectives
Identify the tools, testing procedures, and reporting results for testing web and software applications for Section 508 conformance
Determine whether applications are web or software to apply the proper testing methodology
Using the DHS Section 508 Compliance Test Process for Applications, students learn how to apply test processes 1 to 15 to evaluate elements of web and software applications for Section 508 conformance

24. Example of Web Page Testing

25. Student Enters Results

26. Select NC Response Details

27. View Correct Results for each page tested (OFTE 250 only)

28. Submit Answers

29. OFTE 251: Section 508 Trusted Tester Certification Exam
Online Certification Exam
Prerequisites
OFAC 049: Section 508: What is It and Why is It Important to You?
OFAC 051: Section 508 Standards for Applications
OFAC 050: Section 508 Testing Tools Installation
OFTE 250: Section 508 Trusted Tester Training

30. OFTE 251: Course Objectives
Demonstrate proficiency in testing a multi-page application for a fictional web site using DHS Section 508 Compliance Test Process and installed test tools.
Meet Certification requirements by achieving a score of 90% or more
Duration: Ten day timeframe, with up to three attempts allowed

31. People with Disabilities can be Trusted Testers
People with disabilities test as part of normal life and often can easily understand the basic concepts of accessibility
Assistive-technology interoperability testing is a different evaluation task than Trusted Tester testing
Assistive technology can, and does, interfere with expected results in various ways and in some cases must be turned off 

32. People with Disabilities can be Trusted Testers (cont.)
It is a fundamental alteration for people with disabilities to independently perform some of the evaluation tasks, e.g. interpret picture content, evaluate caption synchronization and accuracy, etc.
To help students with disabilities, guidance is provided for students -- identifying for each lesson and test-ID what assistance is needed, and why

33. Keyboard Access Testing Guidance
Failure Condition: A meaningful image does not have an equivalent text description (purpose and function)
Guidance for Students who are Blind
Guidance for Students who have Low Vision
Guidance for Students with a Motor Disability
Screen readers often modify standard keyboard behavior. Screen reading software must be disabled and assistance is needed to identify any interactive elements or functions that cannot be accessed via keyboard.
If screen magnification software modifies standard keyboard behavior it must be disabled. Assistance is needed to identify any interactive elements or functions that cannot be accessed via keyboard.
Speech input or control software often modifies standard keyboard behavior. Speech input software should be disabled and assistance may be needed to navigate the application with the keyboard.

34. Web Images Testing Guidance
Failure Condition: A meaningful image does not have an equivalent text description (purpose and function)
Guidance for Students who are Blind
A screen reader is needed to read this information from the testing tool. Relying on screen reader output alone, without confirmation by using testing tool output, is insufficient to complete this test. Assistance is needed to compare image and any descriptive elements found.

35. Multimedia Testing Guidance
Failure Condition: The provided captions for multimedia are not equivalent
Guidance for Students who are Blind
Guidance for Students who are Deaf
Guidance for Students who are Hard of Hearing
Assistance is needed to evaluate if on-screen text is equivalent with multimedia.

36. Benefits of Trusted Tester
Section 508 acceptance criteria identified
Consistent test approach
Easy to use testing tools
Minimizes subjectivity
Accept shared results
Developer Remediation
Created a Trusted Tester pool (600+)
Established TT Community of Practice (CoP)

37. How to Sign Up
To sign up for Section 508 courses and Trusted Tester training, send an to

38. Federal Government Section 508 Unification
How can we get there?

39. Unified Testing for Accessibility Project (UTAP) Pilot
A program aimed at assisting four agencies in adopting the Trusted Tester approach for conformance with Section 508 standards.
Timeline: FY14 started
Pilot: 1.5 years
Participating agencies: US Mint, FDIC, Dept. of Ed and DOL
OAST Team: Norman Robinson and Chris Law

40. Why conduct a UTAP?
To improve the accessibility of IT systems at your agency, to benefit people with disabilities.

41. UTAP Pilot Coverage Category Description Policy
Section 508 and/or 504/508 accessibility development and testing policy in place.
Section 508 Team Staffing
508 Management; 508 team.
Acquisition
Conduct validation of procurement solicitations to ensure incorporation of Section 508 contract language into Statements of Work and Performance Work Statements.
IT Development Lifecycles
activities
Conduct validation of Section 508 requirements to ensure incorporation into EIT lifecycle, enterprise architecture, design, development, testing, deployment, and ongoing maintenance activities.
Testing and Validation
Testing and validation of Section 508 conformance claims.

42. UTAP Pilot Coverage (Continued)
Category
Description
Training
Training for stakeholders on roles and responsibilities related to Section 508 compliance.
Outreach
Awareness raising throughout the organization.
Technical Assistance
508 integrated with current IT technical assistance and/or 504/508 accessibility-specific technical assistance resource.
Complaints Process
Track and resolve incoming Section 508 complaints.

43. UTAP Stages 1.Executive support & Goals
2. DHS familiarization with agency
3. Agency familiarization with options 4.
Organizational design
5. Implementation Plan

44. Stage 1 A: Executive Support
Executive support helps to:
Clarify strategic objectives and expectations so that the project serves the overall business purpose.
Authorize the Project funding.
Secure project resources for effective and efficient project execution by influencing by getting the right people assigned for the project.

45. Conduct a program maturity assessment
Stage 1 B: Goals
Conduct a program maturity assessment
Identify the goals for the key stakeholders:
Section 508 Coordinator(s) / Program Manager(s)
Procurement personnel
Design and development teams

46. Program Maturity Assessment
0-None
1 –
Ad Hoc
2 - Planned
3 – Resourced
4 – Measured
5 - Optimized
Measures:
None (nonexistent).
Ad Hoc: No formal policies, process or procedures defined.
Planned: Policies, processes and procedures defined and communicated.
Measures:
Resourced: Resources committed and/or staff trained to implement policies, processes and procedures.
Measured: Validation is performed; results are measured and tracked.
Optimized: Developed into a sophisticated approach using effective and efficient techniques.

47. Audience poll: Stage 1: Exercise
Who are some other Section 508 stakeholders that should be considered when identifying UTAP goals?
Who should be on a UTAP Project Team?

48. Stage 2: Agency Self-Assessment
Goal: Understand your agency Section 508 culture. Helps define your “baseline” for change.
Diagram the “AS-IS” Section 508 environment for each area identified as a goal.
Gather artifacts: Policies, Procedures, etc.
Conduct interviews of agency stakeholders.

49. Ask your neighbor the following questions:
Stage 2: Exercise
Ask your neighbor the following questions:
Who should participate in the interviews?
Who does Section 508 testing in your organization?
What test method is used?
Who reviews the test results?
If a product is not accessible, what happens?

50. Stage 3: Familiarize available options
Leverage available resources:
Consult with DHS
Review UTAP Case Study artifacts
Collaborate with UTAP Pilot participants

51. Stage 4: Design the System
Goals & Exec Support
Agency
Self-Assessment
Evaluate Options
Policy
Training
Staffing
Outreach
Acquisition
Technical Assistance
IT Development Lifecycles
Complaints Process
IT Testing
The Design Plan will cover “What” areas will be impacted.

52. Stage 4: Exercise
Based on responses from Stage 2 (Agency Assessment) and Stage 3 (Available Resources)
What did you learn about what others are doing that could possibly be leveraged at your agency?
What 508 areas should be addressed in your agency’s UTAP Design Plan?

53. Stage 5: Develop Implementation Plan
Document changes and create timeline for:
Updates to Policies and Procedures
Communication Plan
Rollout Plan
Training
Acquisition
Software Development Lifecycle
Testing

54. UTAP Lessons Learned
Senior Leadership buy-in: Leadership buy-in is critical to the success of adopting TT. (Stage 1)
Treat UTAP like a project: Assign PM, team members, establish goals, a timeline, level of work commitment and priority. (Stage 1)
Agency 508 culture: It is vital to collect information from all stakeholders, especially those outside the traditional Section 508 Program or office. Complete a 508 Maturity Survey. (Stage 3)
Implement and Monitor: Execute the Implementation Plan developed in Stage 5. Monitor the components of the plan. (Stage 6)
Note: Stage 6 was not part of the original scope of UTAP, but is vital to agencies conducting their own UTAP.

55. Interagency Trusted Tester Program (ITTP)
One step closer towards Federal Government Section 508 Unification

56. ITTP Overview
The ITTP is the strategic solution modeled after the DHS OAST Trusted Tester Program (TTP) with a multi-pronged approach for:
Facilitating the adoption of Trusted Tester (TT) within other federal agencies.
Promoting a common evaluation processes, tools and procedures for Section 508 testing of electronic and information technology (EIT) across the federal government.
Increasing the workforce skillset and capability by implementing a certification program for Section 508 conformance testing.
Creating a central repository to collect trusted tester test results that can be shared amongst government agencies to reduce redundant testing.

57. ITTP Mission
The mission of the Interagency Trusted Tester Program is to promote a unified, consistent, sharable, and repeatable test and evaluation approach for Section 508 standards conformance that federal agencies can implement throughout their information technology lifecycle to reduce redundant testing and improve cost savings.
Draft – May 2015

58. ITTP Vision
Improving IT accessibility across government through unified requirements, reviews [test and evaluation], reporting, remediation, and reuse.
Draft – May 2015

59. ITTP Goals
Goal
Name
Description 1
Baseline Requirements
Establish as set of accessibility design requirements and validation tests that have been agreed upon and adopted by representatives of federal agencies as part of an effort to harmonize their testing processes for Section 508. 2
Review (Streamlined Test Process)
Develop a streamlined test process based on the Baseline Tests. 3
Report (Test Results and Metrics)
Establish consistent test result and metric reporting standards and formats.

60. ITTP Goals (Continued)
Name
Description 4
Remediation
Establish processes and procedures for managing and tracking the remediation of accessibility defects. 5
Reuse (Share Trusted Tester Results)
Establish mechanism to reuse trusted tester results with stakeholders. 6
Resources (Policies, People, Training, Tools, Tips, and Techniques)
Create, Use, and Promote accessibility resources across the federal government.

61. ITTP Change Control Board
Baseline and Test Process Updates
Added Browsers: Chrome, Firefox
Added Windows 10
Added new tools: Web Accessibility Favelets
WAT alternative
Compatible with IE, Chrome and Firefox
Modified Test Process to align with baseline changes

62. ITTP Opportunities
Develop and deploy a test repository outside DHS firewalls
Partner with federal agencies to adopt TT
Interagency funding of ITTP via Interagency Agency Agreement (IAA)
Partner with Industry (i.e. Develop Open-Source Tools)

63. Alignment with WCAG Testing
The Trusted Tester program partially aligns with WCAG, but needs further updates in these areas
Success Criteria
Description
1.1.1
Non-text content: images – allow Longdesc, link to equivalents
2.3.1
Red-flash thresholds
1.4.1
Use of color
2.4.4
Link text
1.4.4
Resize text
3.2.1
On focus
1.4.5
Images of text
3.2.3
Consistent navigation
2.2.1
Timeout - other options
3.2.4
Consistent identification of elements
2.2.2
Blinking, auto-updating content

64. Other WCAG Requirements
The Trusted Tester program does not cover some WCAG requirements and needs to be updated
Success Criteria
Description
1.3.2
Reading Order
3.3.1
Errors identified using text
1.3.3
Sensory characteristics: instructions not dependent on shape, size, etc
3.3.3
Error suggestion: provide correction help
1.4.2
Audio controls
3.3.4
Error prevention: for legal, financial, etc. changes, make it reversible
1.4.3
Contrast of large text
4.1.1
Parsing, valid mark-up
2.4.5
Multiple ways

65. Recommended Next Steps to Join ITTP
Review UTAP info published on
Assemble a UTAP Project Team
Conduct a UTAP within your agency
Partner with DHS and other federal agencies to adopt the Trusted Tester program

66. Available Resources Resource Location
DHS Accessibility Help Desk
Trusted Tester training enrollment
Send to DHS Accessibility Help Desk
DHS Accessibility Guidance
Compliance Test Processes
DHS Trusted Tester Program
Interagency Trusted Tester Program (ITTP)

67. Section 508 in an Agile Environment

68. Introduction Who am I? What am I? Ken Moser
Section 508 Coordinator (also Delivery Assurance Branch Chief)
U.S. Citizenship and Immigration Services
(desk) or (Mobile)
What am I?
NOT a 508 expert, but involved with it for 10 years
Privileged to lead an exceptional team of independent testers
Building a development-based community of 50+ Trusted Testers
Fortunate to have the support of the DHS OAST

69. Background
How did we get here?

70. In the beginning… Do you recognize this?
God created the universe. When the earth was as yet unformed and desolate, with the surface of the ocean depths shrouded in darkness, and while the Spirit of God was hovering over the surface of the waters, God said, “Let there be light!” So there was light. God saw that the light was beautiful. He separated the light from the darkness, calling the light “day,” and the darkness “night.” The twilight and the dawn were day one. 
Then God said, “Let there be a canopy between bodies of water, separating bodies of water from bodies of water!” So God made a canopy that separated the water beneath the canopy from the water above it. And that is what happened: God called the canopy “sky.” The twilight and the dawn were the second day. 
Then God said, “Let the water beneath the sky come together into one area, and let dry ground appear!” And that is what happened…
Do you recognize this?

71. The Waterfall Methodology
Planning
Requirements
Design
Develop
Test
Deploy
Maintain
Local methodologies often include additional steps but these are usually cited as the essential steps in the process.
Steps are usually gated by formal reviews.
The defining feature of the waterfall methodology is that each step or gate must be completed before moving to the next step.
Once a step is completed, change there are discouraged.

72. The DHS SELC
Section 508 Problem: This is where Section 508 testing & compliance determination usually took place.
How effective do you suppose that was?
General Problem: The time lag from Solution Engineering to Implementation and deployment averaged 18 months or more and this is an eternity for IT requirements.

73. Let’s talk briefly about Agile Development
How are we addressing this?
Let’s talk briefly about Agile Development

74. What is Agile? These are all wrong!
Many people will assert that certain methodologies such as scrum are agile.
Others will assert that certain practices such as working in sprints are agile.
Still others will call out a special role such as Product Owner.
These are all wrong!
First and foremost, Agile is a set of values.
If you violate these values then you are NOT agile.

75. Common Agile Practices
Individuals and interactions over processes and tools
Real-time communication
Co-location
Cross-functional teams
Dedicated team members
Real teamwork
Working software over comprehensive documentation
Frequent deployments that are:
Customer collaboration over contract negotiation
Frequent client meetings
Real-time feedback and adjustment
Acceptance that requirements can change
Ability to adjust
Responding to change over following a plan
Acceptance that:
Coded
Unit & Integration Tested
User Accepted
Regression Tested
Performance Tested
508 Compliant
Secure
Done
Priorities change
Needs change
Requirements evolve
Changes will be discovered
Planning is continuous

76. Common Agile Methodologies

77. Example: Traditional Scrum

78. Sprint Planning Meeting
Where does 508 fit in Scrum?
Features
Sprint Planning Meeting
Tasks
Features
Release Planning
Daily Scrum
Daily Scrum
Retrospective
Daily Scrum
Sprint Demo/Review
Daily Scrum

79. Did Agile solve all our problems?
No!
In fact, it just created a whole new set of challenges for us.

80. Agile Challenges Section 508 testing in waterfall was easy:
We just had to have testers available as planned
This usually took place at the end of the test phase
How do you provision 508 testers when:
There is no long-term plan?
Plans and requirements are allowed to change?
UI changes can appear anywhere in a sprint?
Development teams test continuously?
Deployments take place monthly, weekly, or even daily?
We still have only 8 IV&V testers for 80+ active projects?

81. Section 508 plays a role throughout, but this is just the beginning…
Section 508 at USCIS
Phase
Activity
Typical Artifacts
Release Planning
Define Needs
Where we first discuss UI changes
Project Oversight Plan
Capabilities And Constraints
Test Plan
Team Process Agreement
EIT Accessibility Plan
Blanket CDF
Section 508 exception requests
Development
Repeating as authorized
Planning
Define user stories
Release Backlog
Code
Test
Re-factor
Develop and test code
Development TAR
Section 508 TAR
Independent 508 TAR
Review/demo
Demo and deploy
Updated CDF
End Release cycle
Final CDF
Remediation Plan
Accommodation Plan
RPR
RRR
Section 508 plays a role throughout, but this is just the beginning…

82. Enabling Section 508 in agile projects
Section 508 Trusted Testers
Program created and maintained by DHS OAST
Increased 508 resources from 10 to 30 people in two years
Contract changes
Requiring agile methodologies
Requiring a Section 508 Trusted Tester for every project
Increased 508 resources at USCIS from <20 to 50+ people in two years
Engagement model for Independent Test
Focuses independent test resources on projects with the greatest risk
Allows the independent test team to focus on larger, longer-term issues
Better testing tools
Moves Section 508 left in the development process, to UI development
Automates some testing
Allows testers to focus on issues that cannot be easily automated

83. Matching resources to risks
Our Engagement Model
Matching resources to risks

84. Four new requests have arrived…
Several low and two medium-severity defects were found in updated style sheets for the latest release of a public-facing application – a release that includes new functionality of interest to senior leadership.
Multiple medium-severity defects have been identified in the latest release of an internal application that supports approximately 250 users.
High-severity defects have been identified in the supervisory panel – a function available to less than a hundred supervisor – for an enterprise application.
A new application that supports an entire service center – roughly 2,000 employees – is ready to roll out but the PDF documentation created for it is completely inaccessible. And 12 of the employees are known to have vision challenges that will be severely impacted.
Every one of these teams wants to deploy this week, they are waiting for your evaluation, you have only one tester available, and the CIO wants a recommendation by noon. Go!

85. Engagement model factors
Exceptions
Known defects
Application exposure
Scope of planned changes
Team risk
Impacted users

86. Factor: Exceptions RISK DESCRIPTION LOW
National security or Undue Burden Exception authorized by DHS OAST, or - Incidental to Contract Exception documented and authorized by me. These are good until cancelled and call for no further review.
MODERATE
Fundamental Alteration or Back-Office Exception. These require review to ensure they are still applicable.
HIGH
Most Compliant Product Exception. These also require review to ensure they are still applicable. Since good market research plans are rare, I don’t approve many of these. Where I do approve them, I usually require business owners to conduct market research to seek alternatives every months.

87. Factor: Known Defects RISK DESCRIPTION
LOW
Applications exhibiting no outstanding defects.
MODERATE
Applications exhibiting only low-severity defects.
HIGH
Applications exhibiting a handful of low to moderate severity defects which are likely to be fixed within 90 days.
CRITICAL
Applications with high-severity defects or moderate-severity defects that cannot be fixed within 90 days.*
If promised remediation is overdue then release risk goes to CRITICAL regardless of other factors; this calls for immediate referral to the CIO.
* This actually should not happen but business requirements sometimes dictate a temporary variance. A good example was PIV compliance, mandated by the Department for fast action.

88. Factor: Application exposure
RISK
DESCRIPTION
TINY
One of the following:
Headless applications that have no User Interface (UI);
COTS applications with no significant defects and no customizations;
Internal applications or UI elements within a larger application that support less than 50 people;
Examples: Service bus applications connecting systems; network scanning applications used by a few security teams; or a set of administrative screens within a larger application*.
LOW
Internal applications supporting people.
MODERATE
Internal applications supporting 251-2,500 people.
HIGH
Enterprise applications supporting more than 2,500 people (e.g.: WebTA, our timesheet application for federal employees)
CRITICAL
Public-facing or mission-critical applications (e.g.: ELIS)
If the application exposure is TINY then release risk drops to TINY
* With a little effort, these could qualify for an exception – probably back office, fundamental alteration, or incidental to contract.

89. Factor: Scope of planned changes
RISK
DESCRIPTION
LOW
Minor changes to a few existing UI elements – no more than one new screen, some pop-up changes, new fields, etc.
MODERATE
Multiple and significant UI changes – many new screens.
HIGH
Migration to a completely new UI design.
CRITICAL
Use of new technologies – especially when they may not be supported by approved test standards and tools. (e.g.: For 2015, cell phone and tablet-based applications.)
If there are no defects due for remediation and no planned UI changes then release risk drops to TINY.

90. Factor: Team Risk RISK DESCRIPTION LOW
Teams that conduct extensive re-use in a strong framework and which employ experienced Trusted Testers who have a proven track record.
MODERATE
New teams with limited experience in USCIS.
HIGH
Teams that do not employ or currently lack a certified Trusted Tester.
CRITICAL
Teams with a proven history of serious or outstanding defects that have not been remediated timely.

91. Factor: Known users impacted*
RISK
DESCRIPTION
LOW
None.
MODERATE
Known internal people.
HIGH
Known internal people who have submitted valid complaints.
CRITICAL
Known external stakeholders who have submitted valid complaints.
* People with disabilities who either would be or are impacted by defects, especially known defects.

92. Ruminations
I have a short paper and checklist to help me calculate this and I continue to refine it.
It may be dumb, but I am testing an Excel calculator to help me bang through quickly when I am busy and distracted.
Ideas would be most welcome.
Exceptions
LOW 1
Known defects
MODERATE 10
Application scope
Scope of planned changes
HIGH
100
Team risk
Impacted users
113

93. Developing a Section 508 Community of Interest

94. Department Support
Essentials
Written, approved policies for Section 508 compliance
Defined & documented set of testing standards and tools
Compliance Determination Forms and other reporting templates
Training & appointment of Section 508 Coordinators
Training programs for Section 508 compliance & testing
Important
Compliance database
Component dashboard
Support for FAQs, templates, and other resources
Accessibility Help Desk,
Certification program for Trusted Testers
Helpful
Ongoing participation on the U.S. Access Board
Coordination of pilot testing for emerging tools & technologies
Clearinghouse for COTS test results

95. CIO Support Awareness Introduce yourself to the CIO
Educate the CIO on department policy
Support
Explain your approach and build trust
Work hard to become a trusted advisor
Action
Develop a Section 508 program
Present a budget and secure funding

96. Strong Section 508 teams Com mon Prob lems Potential Solutions
Too many 508 testers do double duty as functional testers
Coordinators can’t know it all and be everywhere
A dedicated Section 508 team can seem threatening to programs and developers
Capacity for Section 508 testing is still insufficient
Potential Solutions
Build a dedicated team that does nothing but 508 work and has the capacity to meet critical needs
Ensure your team has skills you may lack, including emerging technologies
Bring a core team on under an IV&V contract to avoid bias and conflicts of interest
Add a requirement for Trusted Testers to every application development contract

97. Encourage Trusted Testers and recognize them for good work
Develop Incentives
Carr ots
Adopt a defined, risk-based approach to determine testing requirements for each deployment
Limit IV&V to sample testing when applications are tested by certified Trusted Testers
Encourage Trusted Testers and recognize them for good work
Sticks
Require that all UI changes be tested for Section 508 compliance using department-approved standards & tools
Accept Section 508 test results only from independent testers and certified development Trusted Testers

98. Encourage and Recognize Testers

99. Building a Community of Interest
ECN (SharePoint) Page
Set up an distribution list and breaking news
Minutes from the monthly Coordinator’s meeting
Job announcements
Updates on new tools, templates, guidance
COI meeting announcements
Quarterly COI meetings
Greet and introduce new members
Discuss challenges, announce training opportunities, etc.
Drag the CIO to one meeting each year!
Local training for new tools and emerging technologies
Maybe even pilot a Slack channel…

100. Action Items Move further left into the procurement process
Head off non-compliant COTS procurements
Encourage teams to seek compliant alternatives
Address known deficiencies
Improve document assessment & compliance
Deliver more automated testing tools
Bridge developing and emerging gaps
Macintosh, Chrome, mobile platforms…
Changing regulations for compliance

101. Questions