Unity Accessibility Plugin – Update 7 – Explore By Touch

As of today, the Unity Accessibility Plugin has a new feature: Explore By Touch!

Explore By What Now?

Explore By Touch is an accessibility feature on smartphones that let’s blind users run their finger over the screen – and the screen reader will read out what’s under their fingertip. The screen reader also makes sure that no buttons are accidentally pressed or an app is started unintentionally. This works on both Google TalkBack and iOS VoiceOver.

A blindfolded man is swinging a stick at a pinata. He misses and hits his girlfriend instead.

This was my original impression of Explore By Touch. Typical rookie mistake of a sighted person.

I originally thought that this is one feature that I wouldn’t need to re-create for the accessibility plugin, because I figured that no one would be using it much. I assumed no one wanted to blindly poke around on the screen to discover buttons. Why would anyone do that when swiping left and right navigates safely through the user interface?

Think again, you sighted fool!

As we all know, making blind assumptions is a bad thing. Pun intended.

Why would anyone use Explore By Touch?
The answer is simple. Because it is faster!

Navigating menus is a necessity, not fun. Doing it slowly by stepping through all elements on the screen one by one doesn’t make it any better.
If you already roughly know where a button is located, then using Explore By Touch is the next best thing to actually seeing where it is and clicking it directly.

Sure, the first time you get hit with a new menu, you might just trigger the Read From Top function and have the entire screen read out to you. But after that, you can poke around and find out where things are. And being blind doesn’t mean you cannot remember roughly where a button is located on the screen.

Thanks for the Feedback

I wouldn’t have come to this conclusion if I hadn’t received some very clear feedback from an actually blind person, who straight up told me the plugin would be pretty much useless without Explore By Touch. His opinion matched what I read in an article by Matt Gemmel in which he covers Myths about visually impaired users. After that I put my phone back in accessibility mode for a while and ultimately had to agree. It made me want that feature, too.

I am happy to announce that my plugin’s Explore By Touch is now fully functional.
Also: Thank you for your honest feedback, Scott!

3D Audio with Stereo Headphones

Today I want to write about an audio experiment I did for a game that I’m developing using my Unity Accessibility Plugin. It’s an accessible cooking game in which you prepare food for customers.

Problem 

The game pans the voices of customers to the left or right, depending on where they are standing. But when playing the game blind, I find it hard to quickly tell where a customer is. It always takes me a second and it requires a little concentration. This might not be a show-stopper, but it keeps bugging me.

Spatial Audio could be the solution.

Woman with headphones on has her eyes closed and focuses on the whether the sounds are left, right or center.

One burger to the left, two hot dogs to the right…

3D Vision and 3D Hearing

Never heard of spatial audio? Don’t worry, I won’t delve into the details. In a nutshell, it is a clever way of playing back audio so that the listener has a 3D audio effect using only stereo headphones. It’s not even that complicated. Humans only have two ears, so stereo is technically all we need.

Just like your brain uses the differences in the images from your two eyes and gives you 3D vision, it uses the differences in sound between your two ears and gives you 3D hearing.

The picture shows sound sources that are placed at different locations around a head.

Even with only two ears, you can always tell where a sound is coming from. Image Source

Here Is The General Idea

Because sound waves travel through the air (at the speed of sound), a sound coming from your left would reach your left ear first, and then your right ear. Furthermore, your head is in the way of the sound and will block part of it. This is called the head shadow, which dampens some frequencies from the sound before it reaches your other ear. While the differences are very small, your brain picks up on them and uses this very subtle information to place the sound in the world around you. There’s more playing into it, like sound reflected off walls close to you etc, but I said I wouldn’t delve into the science too far.

If you haven’t yet, listen to the famous barber shop example (skip to 2:54 to get the idea):

 

Famous Games using Spatial Audio

A game can reproduce this 3D effect artificially, by delaying the playback on the far ear and filtering frequencies. The only requirement for it to work is that the player wears headphones.

Games for visually impaired or blind players sometimes use spatial audio to allow players to find their way in the dark. Famous examples are the Papa Sangre games and The Nightjar. If you want to check them out, get The Nightjar if you like Benedict Cumberbatch, or Papa Sangre II if you are a fan of Sean Bean (no need to play Papa Sange I first).

Actors Benedict Cumberbatch and Sean Bean inside a voice recording studio. They look cute.

The Nightjar and Papa Sangre II have pretty darn good voice acting in my totally biased opinion.

Unity and Spatial Audio

Somethin’ Else, the developers of Papa Sange and Nightjar, have written their own audio engine and their results are impressive. I wanted to give spatial audio a try in my Sandbox dev environment, before bringing it into my game. But I am fan of not reinventing the wheel. I didn’t want to implement my own spatial audio system when there are some out there already.

There are plugins for spatial audio available for Unity. Here are the three that I found in the Asset Store: Spatial Audio, AstoundSound and dearVR. The last one looked the best, but the first one was the cheapest. For just a quick test that would do. But dearVR is now on my WishList and I will be hoping for a sale in the future.

Direct Comparison

Which one works better?
The majority of people should be able to tell the direction of the voice much faster when listening to the spatial audio example.
But have a listen for yourself.

Put on your headphones and try to determine whether the voice is coming from the left, the right or from the front. How fast are you able to tell the direction? Do you need to focus on it?

Tip: After you’ve listened to the spatial audio example, listen to the stereo one for a second time. It seems a lot worse now, doesn’t it?

I know the spatial audio sounds like it is panned more to the side than the stereo one, but that is not the case – it’s just the effect that the brain creates from the sounds.

EDIT: I just learned that the SoundCloud player above doesn’t work with all screen readers. Here are alternative links to the audio files:
Stereo Results
Spatial Results

Conclusion

While the stereo panning works “good enough” for the simplistic setting of my cooking game, it seems to me that the spatial audio requires less effort to determine where the sound is coming from. For that reason, I will put in the work and switch over to the spatial audio library. The difference might not be huge, but I am hoping it will make the gameplay feel more intuitive, because blind players need to pay less attention to locating the customer. After all, the gameplay is supposed to be about cooking, not about telling direction.

Notes for those interested in the topic:

  • If you want to play around with spatial audio in Unity but are unable to spend money on plugins, good news: Unity has just released a demo Spatialization SDK. You can find it here. I gave it a quick test and it seems to be working fine. Hint: If you check out the demos, you need to select the HRTF demo.
  • If you want to read more about the topic of spatial audio, but are bummed out by all the “binaural beats for better sleep” nonsense search results, try searching for ‘HRTF’ instead.
  • The developer Somethin’ Else used to license their audio engine (Papa Engine) to other developers. While this is no longer the case, their website hints that there might be a Unity compatible version in the future (see here). There is also a forum thread that claims the engine might become open source entirely. You can read about it here.

 

My phone dropped into the bathtub… and lives!

Last night my phone fell into the bathtub.
It slid off the towel on the shelf at the side of the tub, made a splash and disappeared. Complete submersion, for at least 4 seconds.

After realizing what that unexpected splash must have been, I used the headphones like a fishing line to pull it out. Amazingly, the phone was still on, but the touch screen no longer worked.

A previous drop saved it

My phone is working again, 12 hours later. What saved it, at least in part, was the fact that I had dropped it before – on asphalt. That drop had my screen shatter into a million pieces, and forced me to completely disassemble it, replace the touchscreen and digitizer with a replacement ($40 on the internet) and reassemble it.

gas oven

Because of that previous surgery, it was easy to pry the back of the phone off again. Sure enough, there was water inside the phone. I shook it out until it stopped feeling wet and then put the still open phone and a paper towel in the oven for a while. No, I didn’t bake it – I just pre-heated the oven for a few minutes, then turned it off and let the phone lie in the remaining heat for the night.

Withdrawal Symptoms

I spend the night going through the usual phone withdrawal symptoms.
There was no casual gaming, no bedtime audio book and I couldn’t google those random questions that pop into your head at night. I wanted to message my sister and tell her about my mishap, but couldn’t (not without leaving the bed, anyway). I desperately wanted to take a picture of my disassembled phone in the oven, but of course, my phone is my only camera, so no luck. And I very nearly overslept, because my phone is also my alarm clock.

But this morning my phone turned on without complaint and the touchscreen is working again, too.

Not bad, little phone, not bad.

 

Just in case you were wondering, this was a Nexus 5, by the way.

Unity Accessibility Plugin – Update 6 – Text Editing

This week I extended the list of UI element types that plugin supports. The basic functionality of the Unity Accessibility Plugin is almost complete. I will give a complete overview of what is supported in my next post. Today, I want to write about text edit boxes.

Android is refusing to cooperate.

Android Logo standing with crossed arms, looking at me with crossed arms, telling me I can't do what I am trying.Android TalkBack or BackTalk?

Text Editing is complex

Text editing is a huge and complicated topic, with lots of functionality. Just think about everything you can do in a text editor, from moving the caret (insertion point) forward and backward, selecting, copying, deleting and pasting text, changing font size, color and so on.

There is no way I could re-create all of that functionality – which of course also works differently with iOS VoiceOver and Android TalkBack. But luckily I don’t have to.

The Keyboard is Native

On Windows, you can use an actual keyboard to type in the text. The plugin reads out what you typed to make it a little safer. On mobile devices, blind users will have to use the virtual onscreen keyboard.

Luckily, the onscreen keyboard on iOS and Android is a native overlay. When the edit field receives focus, the onscreen keyboard is brought up automatically. And it’s input is automatically directed into the edit field in Unity. On iOS, this was a breeze. If VoiceOver is running, it will spring into action and read out what the user types. On Android? Not so much.

Google TalkBack – Unsupported

Unlike VoiceOver, which is happy to stay in the background until needed, TalkBack will have to be disabled or suspended for my plugin to work. Which means it won’t spring into action automatically once the keyboard comes up. The user won’t even know that the keyboard is active.

This had me running around in circles for the past few days. I cannot access what key is currently under the finger on the onscreen keyboard, so I cannot even have the plugin read it out. There’s no way to access TalkBack directly to resume or suspend it. The plugin doesn’t receive any input from the touchscreen, because it is filtered out. Heck, the system doesn’t even let me detect whether TalkBack is suspended. I can tell when it is off completely, but not when it is on, but currently suspended.

Of course Android cannot allow apps to suspend TalkBack. Apple has a clear advantage since they control the hardward and the software. Android devices don’t have that physical Home button that all iOS devices have. There is no way a user could quit an app if it suspended TalkBack and then didn’t offer enough decent accessibility by itself. The device would be stuck.

The only solution I could come up with would be if TalkBack had a Limited mode. This mode would allow direct input through, but still listened for the magic gestures for Home and the global or local context menu. It would ensure that users could always quit an app but allow most apps to function fairly normally.

Compromise

Unfortunately there is no TalkBack Limited mode. There is a compromise however, though I don’t like it.

TalkBack will allow multi-finger gestures through to the apps, and pretend they’re one-finger gestures. So, technically, TalkBack wouldn’t have to be disabled for Unity. The user just would have to awkwardly use two fingers for all gestures (and three fingers for a double tap for some reason). I tried it, and it isn’t comfortable. But at least it would work.

A street sign pointing into the direction of "Good Enough"Do I really want to go down that road?

Currently I don’t really see a way to make edit boxes work on Android with TalkBack suspended. It’s either forcing the user to use two fingers for everything, or asking him to resume TalkBack for the edit field, then disable it again when done. That is a far cry from the user friendly experience I am aiming for.

But for the time being, I will postpone this for another day, so I don’t start pulling my hair out.

On a personal note

I know this isn’t the first post in which I focus on the issues with Google’s TalkBack. This wasn’t an intentional choice, it just so happened that I came across these issues and wrote about them. It doesn’t say anything about my general attitude towards Android. This is not supposed to be a rant, and I am certainly not an Apple fan girl just jumping at every opportunity to bash on their competition. I exchanged my personal iPhone for an Android phone a few years ago, and found that both operating systems have their advantages and disadvantages. All I care about in the scope of these posts is how easy or difficult it is for me to develop the features I want for them.

Unity Accessibility Plugin – Update 5 – Text to Speech vs. Voice Recordings

In one of my last posts I went on a crusade about how important it is to require as little manual setup as possible from the users of your software. I stand by that, but I want to add a footnote: “Advanced Settings”.

A control panel with about one hundred buttons that all look the same.

Just turn the third dial from the right in the second row-counter clockwise by 45 degrees.

A screen full of settings and options seems intimidating, sure. And people will end up using them wrong. But take 95% of those settings and hide them in a tab called “Advanced Settings”, and all is well. People that are easily scared off by this many choices will leave it well alone (and subsequently not break the system). And those who would like a little more control will be excited that they can get it. Everybody is happy. Now let’s apply this to my plugin.

Customization Ability

The accessibility plugin needs very little setup so far. Import the package into Unity, put the Accessibility Manager prefab into your scene and add an Accessibility component to every UI element that you would like handled. I might even automate this last step. Provided you have given your buttons, toggles and sliders decent names, you are good to go. It’s fast and simple and I would like to keep it this way.

However, it wouldn’t be a good plugin if this wasn’t customizable ad absurdum.

Text to Speech vs. Voice Recordings

One of the most important things I imagine users will want to customize is the voice output. By default, the native text-to-speech engine installed on the device will read out the text. And honestly, this might be good enough for your app.

Text To Speech has come a long way since the old days. They now include different intonations and take context into account. The good systems can even occasionally fool you for a second into believing that it’s real. Here are a few examples, in ascending quality:

They can even do accents now:

By default, the accessibility plugin will use the native text-to-speech available on the device. In the spirit of customizability, all calls to the speech synthesis are wrapped into four separate, dedicated functions. That will make it easy to use a different TTS engine. As I mentioned in a previous post, there are a number of TTS engines available on the asset store. I’ll make a section in the documentation on how to change the TTS engine. (Yes, I have already started on the documentation.)

Alternative to Artificial Speech

And in the spirit of not just regular, but maximum customizability, the plugin will also support custom pre-recorded audio instead of synthesized speech.
There are good reasons for going this way.

  • Stand out: Users get tired of hearing the same voice all the time, in all their accessible apps
  • Improve immersion: Synthesized voices are often neutral, which is not great for story telling. Also, if you have multiple characters, you don’t want them all having the same voice.
  • Avoid Fatigue: It’s hard to listen to artificial speech for longer texts. Especially since the speech rate is often set to high.
  • It cheaper than you might think: You can get good to professional voice acting on platforms such as Fiverr.com, and semi-professional acting on various Audio Forums. Or simply grab a decent microphone and some friends. It might still be better than artificial speech.

Other Customizations

This post is already quite long, but here is a quick rundown of other optional customizations I am working on.

  • Custom Hints – Customize the hint text/audio either for an individual UI element, or for all elements of that type
  • Custom Values – Customize default value text/audio. This will come in especially handy for various toggles. For example, instead of just reading “On” or “Off”, the UI element can be customized to read “Hints are enabled/disabled”.
  • Custom Label text – Customize what the label reads and what is being read by the TTS. Sometimes the text just needs to be different. Maybe you need to add a non-visual description, or replace the word read with hear etc..
  • Keyboard – On Windows, you can choose the keys which will be used to navigate the menus and trigger buttons (although I recommend to leave the defaults alone)

If you have any other ideas for customization, please drop me a line!

Unity Accessibility Plugin – Update 4 – Popup Windows

Thanks to the power of Antibiotics I’m back on my feet and back working on the Unity Accessibility Plugin. This week, I implemented support for popup dialogs.

If you want to start reading about the plugin from the beginning, you can find the first post here.

Popup Menus and Overlays

Not all user interfaces in an app are full screen menus with one clear list of items. There are also partial screen overlays and popup menus. Just think of a level end screen that grays out the game screen in the background and overlays itself in the front. In an app for seeing users this is easy to achieve. Simply put a full screen panel behind the popup and voila – the buttons and elements below it cannot be pressed.

A notification window asking the user for confirmation. There are other buttons grayed out in the back behind the confirmation window..

An example of a popup confirmation window.

Automatic vs. Manual Setup

However – the accessibility plugin can’t know that a button cannot be pressed because something is laying over it. Yes, I know, I could do raycasts and see if there is something blocking it. But that wouldn’t end up being a very clean solution. I would have to do those checks every frame on every UI component – and even then it might be tricky in special cases (which every project always has). It also would contradict the ideally very small footprint the plugin should have.

When a popup menu or dialog window opens, the expected behavior is very clear. The input should focus to that window, and that window alone.

The simplest solution I found was to simply add a little checkbox to my UI Container component: [ ] IsPopup.
This tells the plugin that the other UI elements on the screen are no longer accessible for as long as this popup is open. The plugin will then push all those elements onto a stack, and then restrict focus to the newly opened dialog. It will then get those elements back once the popup closes. All of that happens automatically. The checkbox is the only manual action needed from the developer. Seems like a fair trade off between performance, clean functionality and required manual setup.

 

Continue Reading with the next part here

Sick Again

Almost to the date one year ago everyone here in the office and at home was sick – except for Sascha. I blogged about it back then (see here). Back then I swore to get a flu shot like him this time.

Woman sitting in bed, looking ill and unhappy. She holds an ice bag to her head.

Well, I did, even at the same place as him. And now I am sick – again. And guess what? Sascha is healthy as a horse – again. How does he do it?

I know a flu shot doesn’t protect you from the common cold, but when you’re feeling as horrible as I am right now, you’re willing to try out everything.