Unity Accessibility Plugin – Update 8 – Supported UI Elements

Contract Work has been keeping me busy the past two months, so I haven’t found the time to post any updates on this blog until today.

Still Alive, just not Blogging!

The internet is short lived – and people ask if you gave up on a project the moment you aren’t blogging about it daily (and spend your time working on it instead for example).
So here is a quick check-in-slash-update-slash-still-alive ping!

The post will be short and possibly full of typos – I’m still knee deep in contracting work and hard pressed to divide my time equally between work, play, work on side projects and sleep. The latter is drawing the short straw almost every night.

Full Windows Screen Reader Support

I recently added proper Screen Reader support for Windows to my feature wishlist after reading this excellent article by Ian Hamilton: Screen Readers and Game Engines

Ian and I had a great chat on Twitter and thanks to his article I learned about Tolk. Tolk is a screen reader abstraction library.

See, on Windows there is more than just one screen reader – users get to choose which one they prefer. Or which one they can afford. JAWS for example has a price tag around $1000.

Instead of writing custom code to support every potential screen reader out there, software could simply include Tolk. This greatly reduces the amount of code one has to write. Tolk then does the actual interfacing with the various screen readers. It currently supports six different screen readers.

This is on my future wishlist for now and I will have to investigate it further. Most likely I will release the plugin only for mobile first and then add Windows support at a later time. And then I will have to look into screen reader support for MAC. (No Linux planned at this point.)

Most UI elements now supported

As for the core of the plugin, the basic functionality of the plugin is complete for mobile. That doesn’t mean the plugin is ready to be released, just that it is working and usable. While that sounds like almost the same thing, it really is not. There is lots of code cleanup and refactoring to be done.

 

A menu screen with a label, buttons, a dropdown list, a toggle, a slider, a text edit box and a scroll list.

A screenshot of my test user interface.

Here is a complete list of UI elements that are now supported:

  • Labels
  • Drop-down Lists
  • Buttons
  • Toggles
  • Sliders
  • Scroll Lists
  • Text Edit Boxes (Windows and iOS only at the moment)

The scroll lists actually turned out to be quite tricky and need a little more coding love to handle special use cases. But that might actually deserve a blog post all on its own.

That’s it for today.
Still alive, still eating cake and still working on this plugin!

Surprise: Blind users have established interface conventions, too!

Imagine the following scenario:
You want to delete a file from your computer. A dialog pops up on your screen and a text message asks: “Are you sure?”. Below the text are two rectangular shaped gray areas. One reads “OK” and the other “Cancel”.

An alert asking the user whether he really wants to delete the file. The options are "OK" and "Cancel".

Raise your hand if you know how to click the OK button.

Did you know that you need to move your mouse cursor over the rectangle that reads “OK”, – and then, while hovering over it, click the left button on your mouse exactly one time?

Yes? You did?

Well of course you did!

We all know the basics of navigating user interfaces. We know to left-click on buttons, and to right-click for context menus. We use the mouse wheel and expect a page to scroll. We instinctively search for the close button in the upper corner of a window.

Those are established user interface conventions – and they are a good thing.

As a developer, it means not having to explain the interface basics to your users. You can safely assume that players know how to activate the button labeled “Play” in your main menu. You can just focus on teaching your players the rules of your actual gameplay.

Surprise: Blind users have interface conventions, too!

It shouldn’t come as a big surprise that such interface navigation conventions exist for blind users as well.
It shouldn’t – but it seems that it does.

Often enough, when developers do go the extra mile to add blind accessibility to their games, they create their own interface navigation rules. One of their sighted programmers will spend an afternoon thinking about how it could work and then implement it. If he’s pressed for time he might not even look at how other games did it before. As a result, these games start out by reading a page of instructions to their users at the beginning. And that is about as exciting as reading a phone book.

Why break established conventions?

Not only is it boring to listen to instructions. Breaking conventions can also lead to losing users and turning them away from your app. Consider this: Every app that does this has their own navigation scheme. A blind user has to learn the quirks of each one and then try not to forget as they switch apps. Maybe they will remember the important basics, but if a game implements special gestures for things not used very often, let’s say for Pause for example, it will quickly be forgotten.

You don’t ask your sighted users to press Ctrl-K to click buttons, or right-click to scroll text, do you?
The same should be true for making your menus accessible. Don’t reinvent the wheel, don’t invent your own navigation scheme. Go with what’s already established.

Remember – in order to even start your app, the blind user will have to navigate to it and activate somehow. They know how to do that. They are already using an interface. If your app could be navigated the same way that the user navigated their operating system, you wouldn’t need instructions at all.

Not building on what users already know is creating an unnecessary entry barrier. And for blind users especially, more barriers is the last thing they need.

Made by Apple.

For mobile accessibility, the conventions have pretty much been established by Apple, because their VoiceOver screen reader is the gold standard. The majority of blind smart phone and tablet users are using Apple devices, and these devices are running VoiceOver.

The folks at Apple have done a great job with VoiceOver. Apple received an Access Award from the American Foundation for the Blind in 2009 and a Helen Keller Achievement Award in 2015, and the Bray Award for Accessibility Innovation in 2016 from the American Council of the Blind.
Whether you are an Apple fan or hater, following the really simple interface conventions from VoiceOver is a good place to start.

The freedom to be impatient

Whether you implement your own system or go with established conventions is ultimately up to you, of course. If there is one thing to be said about blind mobile users, it’s that they are incredibly patient and resilient. If your game is good, heck, even if it is just OK, they will probably be willing to jump through the hoops of learning how to navigate your menus either way.
For now.

A sighted user will uninstall a game within seconds if he finds the controls frustrating. There are enough other games out there – the Google Play store counted 2 million apps in February 2016 (Source: Statista.com). Blind users don’t have that option. It won’t surprise you to hear that the number of accessible games is extremely limited.

But that number is growing.
And here’s why:

There around 21 million blind or visually impaired people over the age of 18 in the US alone (Source: AFB). In my opinion it’s one of the few customer groups still left in the mobile market that aren’t yet over-saturated with games. In other words, there is money to be made. The challenge to create games specifically for the blind is also attractive to game designers. If it’s financially viable and artistically interesting… all that’s left is the technical barrier.

And the easier it gets to create accessible apps and games, the more developers will happily take up the task. I hope that my plugin will help with that as well. This is a good thing, because everybody wins. Blind people get more software and more developers can make a living doing what they love.

There will come a time when blind users have a choice. They will have the freedom that we sighted users already have: The freedom to be impatient.

And unless a game has outstanding gameplay – those apps with unusual and tricky interfaces will be the first to be ignored.

I’m guilty

Remember that time pressed programmer I mentioned, who implemented her own accessible menus without regards to existing conventions? Yeah, that was me. I created an elaborate interface system, based on long taps and circular menus. But I’ve learned my lesson earlier this year and deleted all that code. I found a forum with lots of blind gamers and asked them for their help and feedback, and got plenty. It’s when my idea to write a plugin for Unity was born. If you would like to see just how helpful this huge community is, here is the link to the original forum thread.

All apps made accessible with my plugin will automatically be working similar to VoiceOver. Like I said, I am not a fan of reinventing the wheel. You can read more about the plugin and how to make games for sighted and blind users for Windows, iOS and Android in Unity here: Accessibility Plugin for Unity

Further Reading

Unity Accessibility Plugin – Update 7 – Explore By Touch

As of today, the Unity Accessibility Plugin has a new feature: Explore By Touch!

Explore By What Now?

Explore By Touch is an accessibility feature on smartphones that let’s blind users run their finger over the screen – and the screen reader will read out what’s under their fingertip. The screen reader also makes sure that no buttons are accidentally pressed or an app is started unintentionally. This works on both Google TalkBack and iOS VoiceOver.

A blindfolded man is swinging a stick at a pinata. He misses and hits his girlfriend instead.

This was my original impression of Explore By Touch. Typical rookie mistake of a sighted person.

I originally thought that this is one feature that I wouldn’t need to re-create for the accessibility plugin, because I figured that no one would be using it much. I assumed no one wanted to blindly poke around on the screen to discover buttons. Why would anyone do that when swiping left and right navigates safely through the user interface?

Think again, you sighted fool!

As we all know, making blind assumptions is a bad thing. Pun intended.

Why would anyone use Explore By Touch?
The answer is simple. Because it is faster!

Navigating menus is a necessity, not fun. Doing it slowly by stepping through all elements on the screen one by one doesn’t make it any better.
If you already roughly know where a button is located, then using Explore By Touch is the next best thing to actually seeing where it is and clicking it directly.

Sure, the first time you get hit with a new menu, you might just trigger the Read From Top function and have the entire screen read out to you. But after that, you can poke around and find out where things are. And being blind doesn’t mean you cannot remember roughly where a button is located on the screen.

Thanks for the Feedback

I wouldn’t have come to this conclusion if I hadn’t received some very clear feedback from an actually blind person, who straight up told me the plugin would be pretty much useless without Explore By Touch. His opinion matched what I read in an article by Matt Gemmel in which he covers Myths about visually impaired users. After that I put my phone back in accessibility mode for a while and ultimately had to agree. It made me want that feature, too.

I am happy to announce that my plugin’s Explore By Touch is now fully functional.
Also: Thank you for your honest feedback, Scott!

3D Audio with Stereo Headphones

Today I want to write about an audio experiment I did for a game that I’m developing using my Unity Accessibility Plugin. It’s an accessible cooking game in which you prepare food for customers.

Problem 

The game pans the voices of customers to the left or right, depending on where they are standing. But when playing the game blind, I find it hard to quickly tell where a customer is. It always takes me a second and it requires a little concentration. This might not be a show-stopper, but it keeps bugging me.

Spatial Audio could be the solution.

Woman with headphones on has her eyes closed and focuses on the whether the sounds are left, right or center.

One burger to the left, two hot dogs to the right…

3D Vision and 3D Hearing

Never heard of spatial audio? Don’t worry, I won’t delve into the details. In a nutshell, it is a clever way of playing back audio so that the listener has a 3D audio effect using only stereo headphones. It’s not even that complicated. Humans only have two ears, so stereo is technically all we need.

Just like your brain uses the differences in the images from your two eyes and gives you 3D vision, it uses the differences in sound between your two ears and gives you 3D hearing.

The picture shows sound sources that are placed at different locations around a head.

Even with only two ears, you can always tell where a sound is coming from. Image Source

Here Is The General Idea

Because sound waves travel through the air (at the speed of sound), a sound coming from your left would reach your left ear first, and then your right ear. Furthermore, your head is in the way of the sound and will block part of it. This is called the head shadow, which dampens some frequencies from the sound before it reaches your other ear. While the differences are very small, your brain picks up on them and uses this very subtle information to place the sound in the world around you. There’s more playing into it, like sound reflected off walls close to you etc, but I said I wouldn’t delve into the science too far.

If you haven’t yet, listen to the famous barber shop example (skip to 2:54 to get the idea):

 

Famous Games using Spatial Audio

A game can reproduce this 3D effect artificially, by delaying the playback on the far ear and filtering frequencies. The only requirement for it to work is that the player wears headphones.

Games for visually impaired or blind players sometimes use spatial audio to allow players to find their way in the dark. Famous examples are the Papa Sangre games and The Nightjar. If you want to check them out, get The Nightjar if you like Benedict Cumberbatch, or Papa Sangre II if you are a fan of Sean Bean (no need to play Papa Sange I first).

Actors Benedict Cumberbatch and Sean Bean inside a voice recording studio. They look cute.

The Nightjar and Papa Sangre II have pretty darn good voice acting in my totally biased opinion.

Unity and Spatial Audio

Somethin’ Else, the developers of Papa Sange and Nightjar, have written their own audio engine and their results are impressive. I wanted to give spatial audio a try in my Sandbox dev environment, before bringing it into my game. But I am fan of not reinventing the wheel. I didn’t want to implement my own spatial audio system when there are some out there already.

There are plugins for spatial audio available for Unity. Here are the three that I found in the Asset Store: Spatial Audio, AstoundSound and dearVR. The last one looked the best, but the first one was the cheapest. For just a quick test that would do. But dearVR is now on my WishList and I will be hoping for a sale in the future.

Direct Comparison

Which one works better?
The majority of people should be able to tell the direction of the voice much faster when listening to the spatial audio example.
But have a listen for yourself.

Put on your headphones and try to determine whether the voice is coming from the left, the right or from the front. How fast are you able to tell the direction? Do you need to focus on it?

Tip: After you’ve listened to the spatial audio example, listen to the stereo one for a second time. It seems a lot worse now, doesn’t it?

I know the spatial audio sounds like it is panned more to the side than the stereo one, but that is not the case – it’s just the effect that the brain creates from the sounds.

EDIT: I just learned that the SoundCloud player above doesn’t work with all screen readers. Here are alternative links to the audio files:
Stereo Results
Spatial Results

Conclusion

While the stereo panning works “good enough” for the simplistic setting of my cooking game, it seems to me that the spatial audio requires less effort to determine where the sound is coming from. For that reason, I will put in the work and switch over to the spatial audio library. The difference might not be huge, but I am hoping it will make the gameplay feel more intuitive, because blind players need to pay less attention to locating the customer. After all, the gameplay is supposed to be about cooking, not about telling direction.

Notes for those interested in the topic:

  • If you want to play around with spatial audio in Unity but are unable to spend money on plugins, good news: Unity has just released a demo Spatialization SDK. You can find it here. I gave it a quick test and it seems to be working fine. Hint: If you check out the demos, you need to select the HRTF demo.
  • If you want to read more about the topic of spatial audio, but are bummed out by all the “binaural beats for better sleep” nonsense search results, try searching for ‘HRTF’ instead.
  • The developer Somethin’ Else used to license their audio engine (Papa Engine) to other developers. While this is no longer the case, their website hints that there might be a Unity compatible version in the future (see here). There is also a forum thread that claims the engine might become open source entirely. You can read about it here.

 

My phone dropped into the bathtub… and lives!

Last night my phone fell into the bathtub.
It slid off the towel on the shelf at the side of the tub, made a splash and disappeared. Complete submersion, for at least 4 seconds.

After realizing what that unexpected splash must have been, I used the headphones like a fishing line to pull it out. Amazingly, the phone was still on, but the touch screen no longer worked.

A previous drop saved it

My phone is working again, 12 hours later. What saved it, at least in part, was the fact that I had dropped it before – on asphalt. That drop had my screen shatter into a million pieces, and forced me to completely disassemble it, replace the touchscreen and digitizer with a replacement ($40 on the internet) and reassemble it.

gas oven

Because of that previous surgery, it was easy to pry the back of the phone off again. Sure enough, there was water inside the phone. I shook it out until it stopped feeling wet and then put the still open phone and a paper towel in the oven for a while. No, I didn’t bake it – I just pre-heated the oven for a few minutes, then turned it off and let the phone lie in the remaining heat for the night.

Withdrawal Symptoms

I spend the night going through the usual phone withdrawal symptoms.
There was no casual gaming, no bedtime audio book and I couldn’t google those random questions that pop into your head at night. I wanted to message my sister and tell her about my mishap, but couldn’t (not without leaving the bed, anyway). I desperately wanted to take a picture of my disassembled phone in the oven, but of course, my phone is my only camera, so no luck. And I very nearly overslept, because my phone is also my alarm clock.

But this morning my phone turned on without complaint and the touchscreen is working again, too.

Not bad, little phone, not bad.

 

Just in case you were wondering, this was a Nexus 5, by the way.

Unity Accessibility Plugin – Update 6 – Text Editing

This week I extended the list of UI element types that plugin supports. The basic functionality of the Unity Accessibility Plugin is almost complete. I will give a complete overview of what is supported in my next post. Today, I want to write about text edit boxes.

Android is refusing to cooperate.

Android Logo standing with crossed arms, looking at me with crossed arms, telling me I can't do what I am trying.Android TalkBack or BackTalk?

Text Editing is complex

Text editing is a huge and complicated topic, with lots of functionality. Just think about everything you can do in a text editor, from moving the caret (insertion point) forward and backward, selecting, copying, deleting and pasting text, changing font size, color and so on.

There is no way I could re-create all of that functionality – which of course also works differently with iOS VoiceOver and Android TalkBack. But luckily I don’t have to.

The Keyboard is Native

On Windows, you can use an actual keyboard to type in the text. The plugin reads out what you typed to make it a little safer. On mobile devices, blind users will have to use the virtual onscreen keyboard.

Luckily, the onscreen keyboard on iOS and Android is a native overlay. When the edit field receives focus, the onscreen keyboard is brought up automatically. And it’s input is automatically directed into the edit field in Unity. On iOS, this was a breeze. If VoiceOver is running, it will spring into action and read out what the user types. On Android? Not so much.

Google TalkBack – Unsupported

Unlike VoiceOver, which is happy to stay in the background until needed, TalkBack will have to be disabled or suspended for my plugin to work. Which means it won’t spring into action automatically once the keyboard comes up. The user won’t even know that the keyboard is active.

This had me running around in circles for the past few days. I cannot access what key is currently under the finger on the onscreen keyboard, so I cannot even have the plugin read it out. There’s no way to access TalkBack directly to resume or suspend it. The plugin doesn’t receive any input from the touchscreen, because it is filtered out. Heck, the system doesn’t even let me detect whether TalkBack is suspended. I can tell when it is off completely, but not when it is on, but currently suspended.

Of course Android cannot allow apps to suspend TalkBack. Apple has a clear advantage since they control the hardward and the software. Android devices don’t have that physical Home button that all iOS devices have. There is no way a user could quit an app if it suspended TalkBack and then didn’t offer enough decent accessibility by itself. The device would be stuck.

The only solution I could come up with would be if TalkBack had a Limited mode. This mode would allow direct input through, but still listened for the magic gestures for Home and the global or local context menu. It would ensure that users could always quit an app but allow most apps to function fairly normally.

Compromise

Unfortunately there is no TalkBack Limited mode. There is a compromise however, though I don’t like it.

TalkBack will allow multi-finger gestures through to the apps, and pretend they’re one-finger gestures. So, technically, TalkBack wouldn’t have to be disabled for Unity. The user just would have to awkwardly use two fingers for all gestures (and three fingers for a double tap for some reason). I tried it, and it isn’t comfortable. But at least it would work.

A street sign pointing into the direction of "Good Enough"Do I really want to go down that road?

Currently I don’t really see a way to make edit boxes work on Android with TalkBack suspended. It’s either forcing the user to use two fingers for everything, or asking him to resume TalkBack for the edit field, then disable it again when done. That is a far cry from the user friendly experience I am aiming for.

But for the time being, I will postpone this for another day, so I don’t start pulling my hair out.

On a personal note

I know this isn’t the first post in which I focus on the issues with Google’s TalkBack. This wasn’t an intentional choice, it just so happened that I came across these issues and wrote about them. It doesn’t say anything about my general attitude towards Android. This is not supposed to be a rant, and I am certainly not an Apple fan girl just jumping at every opportunity to bash on their competition. I exchanged my personal iPhone for an Android phone a few years ago, and found that both operating systems have their advantages and disadvantages. All I care about in the scope of these posts is how easy or difficult it is for me to develop the features I want for them.

Unity Accessibility Plugin – Update 5 – Text to Speech vs. Voice Recordings

In one of my last posts I went on a crusade about how important it is to require as little manual setup as possible from the users of your software. I stand by that, but I want to add a footnote: “Advanced Settings”.

A control panel with about one hundred buttons that all look the same.

Just turn the third dial from the right in the second row-counter clockwise by 45 degrees.

A screen full of settings and options seems intimidating, sure. And people will end up using them wrong. But take 95% of those settings and hide them in a tab called “Advanced Settings”, and all is well. People that are easily scared off by this many choices will leave it well alone (and subsequently not break the system). And those who would like a little more control will be excited that they can get it. Everybody is happy. Now let’s apply this to my plugin.

Customization Ability

The accessibility plugin needs very little setup so far. Import the package into Unity, put the Accessibility Manager prefab into your scene and add an Accessibility component to every UI element that you would like handled. I might even automate this last step. Provided you have given your buttons, toggles and sliders decent names, you are good to go. It’s fast and simple and I would like to keep it this way.

However, it wouldn’t be a good plugin if this wasn’t customizable ad absurdum.

Text to Speech vs. Voice Recordings

One of the most important things I imagine users will want to customize is the voice output. By default, the native text-to-speech engine installed on the device will read out the text. And honestly, this might be good enough for your app.

Text To Speech has come a long way since the old days. They now include different intonations and take context into account. The good systems can even occasionally fool you for a second into believing that it’s real. Here are a few examples, in ascending quality:

They can even do accents now:

By default, the accessibility plugin will use the native text-to-speech available on the device. In the spirit of customizability, all calls to the speech synthesis are wrapped into four separate, dedicated functions. That will make it easy to use a different TTS engine. As I mentioned in a previous post, there are a number of TTS engines available on the asset store. I’ll make a section in the documentation on how to change the TTS engine. (Yes, I have already started on the documentation.)

Alternative to Artificial Speech

And in the spirit of not just regular, but maximum customizability, the plugin will also support custom pre-recorded audio instead of synthesized speech.
There are good reasons for going this way.

  • Stand out: Users get tired of hearing the same voice all the time, in all their accessible apps
  • Improve immersion: Synthesized voices are often neutral, which is not great for story telling. Also, if you have multiple characters, you don’t want them all having the same voice.
  • Avoid Fatigue: It’s hard to listen to artificial speech for longer texts. Especially since the speech rate is often set to high.
  • It cheaper than you might think: You can get good to professional voice acting on platforms such as Fiverr.com, and semi-professional acting on various Audio Forums. Or simply grab a decent microphone and some friends. It might still be better than artificial speech.

Other Customizations

This post is already quite long, but here is a quick rundown of other optional customizations I am working on.

  • Custom Hints – Customize the hint text/audio either for an individual UI element, or for all elements of that type
  • Custom Values – Customize default value text/audio. This will come in especially handy for various toggles. For example, instead of just reading “On” or “Off”, the UI element can be customized to read “Hints are enabled/disabled”.
  • Custom Label text – Customize what the label reads and what is being read by the TTS. Sometimes the text just needs to be different. Maybe you need to add a non-visual description, or replace the word read with hear etc..
  • Keyboard – On Windows, you can choose the keys which will be used to navigate the menus and trigger buttons (although I recommend to leave the defaults alone)

If you have any other ideas for customization, please drop me a line!