DEVELOPING MOBILE APPS FOR THE VISUALLY IMPAIRED HEARTLAND COMMUNITY COLLEGE Nicolaas tenBroek Todd Simeone

WHO ARE YOU? And why are you here? And why are you here?

HCCS MOBILE CURRICULUM Computer Science I iOSiOSAndroidAndroid Windows Phone AppsAppsGamesGamesCommComm Apps for BLV

HOW WE BECAME INVOLVED IN DEVELOPING APPS FOR THE BLIND Independence Science Independence Science Independence Science Independence Science Desire for classroom apps on iOS Desire for classroom apps on iOS Chemistry Chemistry Basic Mathematics Basic Mathematics Graphing Graphing Advanced Mathematics Advanced Mathematics Other Educational Apps Other Educational Apps

ISSUES CONVERTING VISUAL INTERFACES TO SUPPORT VOICEOVER The amount of information conveyed by a visual interface is tremendous. Graphics and the relationship between controls instantly convey functional information that would take paragraphs of text to explain. The amount of information conveyed by a visual interface is tremendous. Graphics and the relationship between controls instantly convey functional information that would take paragraphs of text to explain. Animation and graphical gloss are often used to explain what is happening within an app. Animation and graphical gloss are often used to explain what is happening within an app. Many common idioms simply do not transfer. For instance: Many common idioms simply do not transfer. For instance: What does a Back button mean to someone who did not see the screen move forward? What does a Back button mean to someone who did not see the screen move forward? Pull down to refresh is only meaningful because you saw the hidden setting when the table bounced after scrolling to the top.Pull down to refresh is only meaningful because you saw the hidden setting when the table bounced after scrolling to the top. Items partially obscured by a translucent control instantly tell a sighted person that more things are available, but overlapping controls will confuse a screen reader Items partially obscured by a translucent control instantly tell a sighted person that more things are available, but overlapping controls will confuse a screen reader

ISSUES CONVERTING VISUAL INTERFACES TO SUPPORT VOICEOVER Most apps are designed to be used quickly and in non-ideal circumstances, but screen readers take a long time to read, and can interfere with a blind users interaction with the app, increasing frustration and confusion. Most apps are designed to be used quickly and in non-ideal circumstances, but screen readers take a long time to read, and can interfere with a blind users interaction with the app, increasing frustration and confusion. Gestures that make sense when tied to the visual display (i.e. those that mimic real world interactions) have to change when the screen reader is active, and often lose all real world meaning. For instance, if you used a swipe gesture to push a ball around a screen, that would mimic a real-world interaction. With the screen reader turned on, that same gesture will cycle between elements on the screen (from top to bottom and left to right). Gestures that make sense when tied to the visual display (i.e. those that mimic real world interactions) have to change when the screen reader is active, and often lose all real world meaning. For instance, if you used a swipe gesture to push a ball around a screen, that would mimic a real-world interaction. With the screen reader turned on, that same gesture will cycle between elements on the screen (from top to bottom and left to right).

HOW PEOPLE WITH BLV INTERACT WITH MOBILE DEVICES Often listen to screen reader Often listen to screen reader Explore app by moving finger around Explore app by moving finger around Use List Mode to cycle between items on screen Use List Mode to cycle between items on screen Tap off the selected control to use it Tap off the selected control to use it

COMMON DESIGN APPROACHES AND THE PROBLEMS WE FOUND Just add screen reader support to existing design Just add screen reader support to existing design Can make it very difficult to reliably hit controls Can make it very difficult to reliably hit controls Lack of spacing can cause incorrect controls to be selected Lack of spacing can cause incorrect controls to be selected List Mode often rearranges the controls which can quickly remove all meaning as positioning of labels and images are often used to inform the user of the meaning of other controls List Mode often rearranges the controls which can quickly remove all meaning as positioning of labels and images are often used to inform the user of the meaning of other controls Constant screen reading slows down the users interaction with the app, making it even more difficult to use Constant screen reading slows down the users interaction with the app, making it even more difficult to use

COMMON DESIGN APPROACHES AND THE PROBLEMS WE FOUND Add menus Add menus Complicates the interaction of the app Complicates the interaction of the app Tends to lead to desktop style designs which do not work in the mobile world Tends to lead to desktop style designs which do not work in the mobile world Nested menus can cause users to become lost in the structure, and makes finding features even more difficult Nested menus can cause users to become lost in the structure, and makes finding features even more difficult Contextual menus require users to see when and where they appear Contextual menus require users to see when and where they appear

COMMON DESIGN APPROACHES AND THE PROBLEMS WE FOUND Add vibration Add vibration Used too often it will lose its meaning Used too often it will lose its meaning Causes users fingers to go numb Causes users fingers to go numb Speak commands Speak commands Not an ideal computer interface in a classroom, nor frankly, any public location Not an ideal computer interface in a classroom, nor frankly, any public location Computer comprehension of speech is not good enough to work on its own. When it fails we fall back to on-screen controls, which a blind person might easily miss. Computer comprehension of speech is not good enough to work on its own. When it fails we fall back to on-screen controls, which a blind person might easily miss.

BETTER DESIGN CHOICES Create two completely separate interfaces, one for use with the screen reader, and another for use without. This can drastically reduce the need to compromise on the design. It also simplifies coding by reducing the need to constantly test the environment and select the appropriate display or action based on the state of the screen reader. Create two completely separate interfaces, one for use with the screen reader, and another for use without. This can drastically reduce the need to compromise on the design. It also simplifies coding by reducing the need to constantly test the environment and select the appropriate display or action based on the state of the screen reader. Always test with screen off! Always test with screen off! Larger controls and much more space between controls than used with designs for the sighted Larger controls and much more space between controls than used with designs for the sighted Reduce colour, increase contrast Reduce colour, increase contrast Reduce the amount of information read by the screen reader. If possible, add beginner and pro modes that change the amount of information read. Reduce the amount of information read by the screen reader. If possible, add beginner and pro modes that change the amount of information read.

BETTER DESIGN CHOICES Do everything you can to speed the users interaction with the app. Remove steps whenever possible. Do everything you can to speed the users interaction with the app. Remove steps whenever possible. Add sound, but carefully Add sound, but carefully Best used with headphones Best used with headphones Map sounds to 3D space (we use OpenAL) Map sounds to 3D space (we use OpenAL) Use sparingly Use sparingly Dont make user learn meanings of random sounds. Sounds should convey meaning and match the physical world as much as is practical. Dont make user learn meanings of random sounds. Sounds should convey meaning and match the physical world as much as is practical. Add vibration, but sparingly Add vibration, but sparingly

BETTER DESIGN CHOICES Invert the position of important items. Layouts for the sighted often place important items on the top of the view, but they should be at the bottom for the blind. Invert the position of important items. Layouts for the sighted often place important items on the top of the view, but they should be at the bottom for the blind. Contextual menu display must be optimised to reduce the distance from the control that initiated it. This can cause issues when scrolling leaves a control near the edge of the screen. Contextual menu display must be optimised to reduce the distance from the control that initiated it. This can cause issues when scrolling leaves a control near the edge of the screen. Consider adding unique actions to the interface for the blind user. For instance, you might relocate the currently selected item so that it is always where the user taps if they tap an empty part of the screen. Consider adding unique actions to the interface for the blind user. For instance, you might relocate the currently selected item so that it is always where the user taps if they tap an empty part of the screen. Sleeves which cover the bezel are very useful. Sleeves which cover the bezel are very useful.

INTERESTING APPROACHES THAT SURPRISED US BY WORKING Drawing - Used within a constrained environment,drawing elements onto the screen works well, but introduces issues if the drawn item does not exactly match the finger movements. Drawing - Used within a constrained environment,drawing elements onto the screen works well, but introduces issues if the drawn item does not exactly match the finger movements. Scrolling - When paired with other elements like sound, user-driven scrolling can be effective. However, you must ensure that scrolling ends with complete elements still on the screen. Scrolling - When paired with other elements like sound, user-driven scrolling can be effective. However, you must ensure that scrolling ends with complete elements still on the screen.

ISSUES WE ENCOUNTERED There are some serious issues with VoiceOver There are some serious issues with VoiceOver An item always selected, there is no ability to have nothing selected, which means that something will always respond. This is not the behaviour when VO is off An item always selected, there is no ability to have nothing selected, which means that something will always respond. This is not the behaviour when VO is off You cannot locally disable accessibility. Often things we needed to do within our apps would have been much easier to do ourselves rather than try to work within the constraints of VO You cannot locally disable accessibility. Often things we needed to do within our apps would have been much easier to do ourselves rather than try to work within the constraints of VO VO intercepts every user interaction before passing it to the app. This makes overriding default gestures impossible, even when they dont make sense. VO intercepts every user interaction before passing it to the app. This makes overriding default gestures impossible, even when they dont make sense.

BENEFITS OF THE COURSES Educational Community Educational Community Heartland Community College Heartland Community College Students Students Employers Employers