Unity Accessibility Plugin – Update 7 – Explore By Touch

As of today, the Unity Accessibility Plugin has a new feature: Explore By Touch!

Explore By What Now?

Explore By Touch is an accessibility feature on smartphones that let’s blind users run their finger over the screen – and the screen reader will read out what’s under their fingertip. The screen reader also makes sure that no buttons are accidentally pressed or an app is started unintentionally. This works on both Google TalkBack and iOS VoiceOver.

A blindfolded man is swinging a stick at a pinata. He misses and hits his girlfriend instead.

This was my original impression of Explore By Touch. Typical rookie mistake of a sighted person.

I originally thought that this is one feature that I wouldn’t need to re-create for the accessibility plugin, because I figured that no one would be using it much. I assumed no one wanted to blindly poke around on the screen to discover buttons. Why would anyone do that when swiping left and right navigates safely through the user interface?

Think again, you sighted fool!

As we all know, making blind assumptions is a bad thing. Pun intended.

Why would anyone use Explore By Touch?
The answer is simple. Because it is faster!

Navigating menus is a necessity, not fun. Doing it slowly by stepping through all elements on the screen one by one doesn’t make it any better.
If you already roughly know where a button is located, then using Explore By Touch is the next best thing to actually seeing where it is and clicking it directly.

Sure, the first time you get hit with a new menu, you might just trigger the Read From Top function and have the entire screen read out to you. But after that, you can poke around and find out where things are. And being blind doesn’t mean you cannot remember roughly where a button is located on the screen.

Thanks for the Feedback

I wouldn’t have come to this conclusion if I hadn’t received some very clear feedback from an actually blind person, who straight up told me the plugin would be pretty much useless without Explore By Touch. His opinion matched what I read in an article by Matt Gemmel in which he covers Myths about visually impaired users. After that I put my phone back in accessibility mode for a while and ultimately had to agree. It made me want that feature, too.

I am happy to announce that my plugin’s Explore By Touch is now fully functional.
Also: Thank you for your honest feedback, Scott!

3D Audio with Stereo Headphones

Today I want to write about an audio experiment I did for a game that I’m developing using my Unity Accessibility Plugin. It’s an accessible cooking game in which you prepare food for customers.

Problem 

The game pans the voices of customers to the left or right, depending on where they are standing. But when playing the game blind, I find it hard to quickly tell where a customer is. It always takes me a second and it requires a little concentration. This might not be a show-stopper, but it keeps bugging me.

Spatial Audio could be the solution.

Woman with headphones on has her eyes closed and focuses on the whether the sounds are left, right or center.

One burger to the left, two hot dogs to the right…

3D Vision and 3D Hearing

Never heard of spatial audio? Don’t worry, I won’t delve into the details. In a nutshell, it is a clever way of playing back audio so that the listener has a 3D audio effect using only stereo headphones. It’s not even that complicated. Humans only have two ears, so stereo is technically all we need.

Just like your brain uses the differences in the images from your two eyes and gives you 3D vision, it uses the differences in sound between your two ears and gives you 3D hearing.

The picture shows sound sources that are placed at different locations around a head.

Even with only two ears, you can always tell where a sound is coming from. Image Source

Here Is The General Idea

Because sound waves travel through the air (at the speed of sound), a sound coming from your left would reach your left ear first, and then your right ear. Furthermore, your head is in the way of the sound and will block part of it. This is called the head shadow, which dampens some frequencies from the sound before it reaches your other ear. While the differences are very small, your brain picks up on them and uses this very subtle information to place the sound in the world around you. There’s more playing into it, like sound reflected off walls close to you etc, but I said I wouldn’t delve into the science too far.

If you haven’t yet, listen to the famous barber shop example (skip to 2:54 to get the idea):

 

Famous Games using Spatial Audio

A game can reproduce this 3D effect artificially, by delaying the playback on the far ear and filtering frequencies. The only requirement for it to work is that the player wears headphones.

Games for visually impaired or blind players sometimes use spatial audio to allow players to find their way in the dark. Famous examples are the Papa Sangre games and The Nightjar. If you want to check them out, get The Nightjar if you like Benedict Cumberbatch, or Papa Sangre II if you are a fan of Sean Bean (no need to play Papa Sange I first).

Actors Benedict Cumberbatch and Sean Bean inside a voice recording studio. They look cute.

The Nightjar and Papa Sangre II have pretty darn good voice acting in my totally biased opinion.

Unity and Spatial Audio

Somethin’ Else, the developers of Papa Sange and Nightjar, have written their own audio engine and their results are impressive. I wanted to give spatial audio a try in my Sandbox dev environment, before bringing it into my game. But I am fan of not reinventing the wheel. I didn’t want to implement my own spatial audio system when there are some out there already.

There are plugins for spatial audio available for Unity. Here are the three that I found in the Asset Store: Spatial Audio, AstoundSound and dearVR. The last one looked the best, but the first one was the cheapest. For just a quick test that would do. But dearVR is now on my WishList and I will be hoping for a sale in the future.

Direct Comparison

Which one works better?
The majority of people should be able to tell the direction of the voice much faster when listening to the spatial audio example.
But have a listen for yourself.

Put on your headphones and try to determine whether the voice is coming from the left, the right or from the front. How fast are you able to tell the direction? Do you need to focus on it?

Tip: After you’ve listened to the spatial audio example, listen to the stereo one for a second time. It seems a lot worse now, doesn’t it?

I know the spatial audio sounds like it is panned more to the side than the stereo one, but that is not the case – it’s just the effect that the brain creates from the sounds.

EDIT: I just learned that the SoundCloud player above doesn’t work with all screen readers. Here are alternative links to the audio files:
Stereo Results
Spatial Results

Conclusion

While the stereo panning works “good enough” for the simplistic setting of my cooking game, it seems to me that the spatial audio requires less effort to determine where the sound is coming from. For that reason, I will put in the work and switch over to the spatial audio library. The difference might not be huge, but I am hoping it will make the gameplay feel more intuitive, because blind players need to pay less attention to locating the customer. After all, the gameplay is supposed to be about cooking, not about telling direction.

Notes for those interested in the topic:

  • If you want to play around with spatial audio in Unity but are unable to spend money on plugins, good news: Unity has just released a demo Spatialization SDK. You can find it here. I gave it a quick test and it seems to be working fine. Hint: If you check out the demos, you need to select the HRTF demo.
  • If you want to read more about the topic of spatial audio, but are bummed out by all the “binaural beats for better sleep” nonsense search results, try searching for ‘HRTF’ instead.
  • The developer Somethin’ Else used to license their audio engine (Papa Engine) to other developers. While this is no longer the case, their website hints that there might be a Unity compatible version in the future (see here). There is also a forum thread that claims the engine might become open source entirely. You can read about it here.

 

My phone dropped into the bathtub… and lives!

Last night my phone fell into the bathtub.
It slid off the towel on the shelf at the side of the tub, made a splash and disappeared. Complete submersion, for at least 4 seconds.

After realizing what that unexpected splash must have been, I used the headphones like a fishing line to pull it out. Amazingly, the phone was still on, but the touch screen no longer worked.

A previous drop saved it

My phone is working again, 12 hours later. What saved it, at least in part, was the fact that I had dropped it before – on asphalt. That drop had my screen shatter into a million pieces, and forced me to completely disassemble it, replace the touchscreen and digitizer with a replacement ($40 on the internet) and reassemble it.

gas oven

Because of that previous surgery, it was easy to pry the back of the phone off again. Sure enough, there was water inside the phone. I shook it out until it stopped feeling wet and then put the still open phone and a paper towel in the oven for a while. No, I didn’t bake it – I just pre-heated the oven for a few minutes, then turned it off and let the phone lie in the remaining heat for the night.

Withdrawal Symptoms

I spend the night going through the usual phone withdrawal symptoms.
There was no casual gaming, no bedtime audio book and I couldn’t google those random questions that pop into your head at night. I wanted to message my sister and tell her about my mishap, but couldn’t (not without leaving the bed, anyway). I desperately wanted to take a picture of my disassembled phone in the oven, but of course, my phone is my only camera, so no luck. And I very nearly overslept, because my phone is also my alarm clock.

But this morning my phone turned on without complaint and the touchscreen is working again, too.

Not bad, little phone, not bad.

 

Just in case you were wondering, this was a Nexus 5, by the way.

Unity Accessibility Plugin – Update 6 – Text Editing

This week I extended the list of UI element types that plugin supports. The basic functionality of the Unity Accessibility Plugin is almost complete. I will give a complete overview of what is supported in my next post. Today, I want to write about text edit boxes.

Android is refusing to cooperate.

Android Logo standing with crossed arms, looking at me with crossed arms, telling me I can't do what I am trying.Android TalkBack or BackTalk?

Text Editing is complex

Text editing is a huge and complicated topic, with lots of functionality. Just think about everything you can do in a text editor, from moving the caret (insertion point) forward and backward, selecting, copying, deleting and pasting text, changing font size, color and so on.

There is no way I could re-create all of that functionality – which of course also works differently with iOS VoiceOver and Android TalkBack. But luckily I don’t have to.

The Keyboard is Native

On Windows, you can use an actual keyboard to type in the text. The plugin reads out what you typed to make it a little safer. On mobile devices, blind users will have to use the virtual onscreen keyboard.

Luckily, the onscreen keyboard on iOS and Android is a native overlay. When the edit field receives focus, the onscreen keyboard is brought up automatically. And it’s input is automatically directed into the edit field in Unity. On iOS, this was a breeze. If VoiceOver is running, it will spring into action and read out what the user types. On Android? Not so much.

Google TalkBack – Unsupported

Unlike VoiceOver, which is happy to stay in the background until needed, TalkBack will have to be disabled or suspended for my plugin to work. Which means it won’t spring into action automatically once the keyboard comes up. The user won’t even know that the keyboard is active.

This had me running around in circles for the past few days. I cannot access what key is currently under the finger on the onscreen keyboard, so I cannot even have the plugin read it out. There’s no way to access TalkBack directly to resume or suspend it. The plugin doesn’t receive any input from the touchscreen, because it is filtered out. Heck, the system doesn’t even let me detect whether TalkBack is suspended. I can tell when it is off completely, but not when it is on, but currently suspended.

Of course Android cannot allow apps to suspend TalkBack. Apple has a clear advantage since they control the hardward and the software. Android devices don’t have that physical Home button that all iOS devices have. There is no way a user could quit an app if it suspended TalkBack and then didn’t offer enough decent accessibility by itself. The device would be stuck.

The only solution I could come up with would be if TalkBack had a Limited mode. This mode would allow direct input through, but still listened for the magic gestures for Home and the global or local context menu. It would ensure that users could always quit an app but allow most apps to function fairly normally.

Compromise

Unfortunately there is no TalkBack Limited mode. There is a compromise however, though I don’t like it.

TalkBack will allow multi-finger gestures through to the apps, and pretend they’re one-finger gestures. So, technically, TalkBack wouldn’t have to be disabled for Unity. The user just would have to awkwardly use two fingers for all gestures (and three fingers for a double tap for some reason). I tried it, and it isn’t comfortable. But at least it would work.

A street sign pointing into the direction of "Good Enough"Do I really want to go down that road?

Currently I don’t really see a way to make edit boxes work on Android with TalkBack suspended. It’s either forcing the user to use two fingers for everything, or asking him to resume TalkBack for the edit field, then disable it again when done. That is a far cry from the user friendly experience I am aiming for.

But for the time being, I will postpone this for another day, so I don’t start pulling my hair out.

On a personal note

I know this isn’t the first post in which I focus on the issues with Google’s TalkBack. This wasn’t an intentional choice, it just so happened that I came across these issues and wrote about them. It doesn’t say anything about my general attitude towards Android. This is not supposed to be a rant, and I am certainly not an Apple fan girl just jumping at every opportunity to bash on their competition. I exchanged my personal iPhone for an Android phone a few years ago, and found that both operating systems have their advantages and disadvantages. All I care about in the scope of these posts is how easy or difficult it is for me to develop the features I want for them.