Unity Accessibility Plugin – Update 11 – Editor Accessibility vs App Accessibility

About once every month I receive an email or Twitter message asking – usually outstandingly nicely – whether my Unity Accessibility Plugin will make it possible for blind developers to create games with Unity. It’s happened often enough that I think it warrants a short post to clarify what my plugin does, and what it does not do.

On the left side is the logo of the Unity Editor, on the opposite side is the Made-With-Unity splash screen that is displayed when starting games created with Unity. They look almost identical.

So similar and yet so different.

It’s a screen reader

In a nutshell, the plugin is a screen reader, specifically tailored to work with apps and games created with Unity. Neither VoiceOver nor TalkBack can recognize the UI elements that Unity renders, so all apps created with Unity are automatically inaccessible otherwise. The important part is that the plugin makes the apps created with Unity accessible, not Unity itself.

It does not make Unity itself accessible

The Unity Editor – at least on Windows – is not very accessible to screen reader software. NVDA will read the menus and the names of the individual panels, but nothing else. JAWS apparently fares not much better. For development, this is useless. This particular plugin doesn’t change that, unfortunately.

However…

Experimental accessibility for the Editor

A few weeks ago, over Christmas, I was playing around with making the Editor itself accessible. Inspired by a blind developer who wanted to use Unity to make a Go-Fish game I started to create a plugin that adds accessibility functionality to Unity. But this project is so early in its infancy that I feel almost uncomfortable writing about it at all.

Currently, this plugin adds keyboard shortcuts to read out the errors in the console, it plays sound effects when entering or leaving game mode, and notifies the developer if the compilation fails. It let’s the developer tab through the game objects in the scene hierarchy, reading out their names, how many children they have and reads out hints on how to add new children. It also makes the project view a little more accessible, reading out the names of files and folders.

But I haven’t found any solution for managing the components on a game object. Unity’s Inspector window supports keyboard navigation, but I can’t find a way to query what is currently highlighted and focused, so that I could tell NVDA (or any other screen reader) to read it out. Managing components and their values, usually in the form of prefabs, is probably the most important core feature of Unity. That makes this a major road block at the moment

Interested in joining the project?

In a finished version I would love this plugin to allow blind developers to do as much as possible with the Editor, including the creation and management of prefabs. It should also be possible to create builds for the various target platforms. And it should include all kinds shortcuts to make the most common tasks quick to do. And I would also want it to include a stack of documentation and tutorials on how to use Unity and create games with it without sight. Then throw in a demo project or two. All neatly wrapped in a free, easy-to-download-and-install package.

I would love to see this work. But I am enough of a realist to know that I don’t have a lot of time left over to put into this – at least not while I’m still working on the other accessibility plugin. If I tried to split my time between the two, neither one would ever get finished.

For that reason, this is an open invitation to other developers willing to help with this. I’d be happy to put what I have up on Git Hub if anyone was interested in joining in.
Just contact me: michelle@metalpopgames.com

 

Unity Accessibility Plugin – Update 10 – A First Game

A few weekends ago I did my own personal Game Jam. The plugin is in a good place and I wanted to give it a test run.
If you have no idea what I am talking about, I recommend quickly reading the first post on this project and then come back. You can find it here.

A woman sits deeply focused at her desk, typing at her computer.

My One-Person Game Jam

I started at 7 pm, got some sugar (aka chocolate!) and some caffeine in me – and shortly after midnight I had a game up and running. It would have been sooner, but my dogs insisted that I feed them and then even demanded a walk around the neighborhood. What can I say – puppy eyes are my big weakness.

The result is an accessible Match 3 game.
Yes, you read correctly – another Match 3 game. This is a technology test, so stop moaning!

Screenshot of the puzzle game, showing a grid of different colored gems.

The world really needed another Match 3 game, didn’t it?

The game is quite simplistic without a deep story line. Or any story line, really. I wanted to test the plugin, so going for something simplistic made the most sense – and I didn’t have time for more anyway.

The plugin isn’t fully complete yet, there are a dozen small issues, and just as many missing features. Most importantly, many of the special gestures don’t work yet, like “Read From Top” or “Go Back”. But still, this is a first live action test for the plugin and I am a little excited about that!

Go check it out!

If anyone wants to see the plugin in action in its current state, please give it a go. And if you try it out, I would be grateful if you gave me your feedback. I’m not talking about a review (although I won’t stop you), I mean feedback about the navigation, the controls and the overall accessibility of everything. Just post it into a comment here or write me an email: mika@metalpopgames.com

Banner for Blindie Match Puzzle Game showing a few of the gems and the subtitle 'a fully accessible puzzle game'.

If VoiceOver or TalkBack are detected to be active, accessibility will turn on automatically. But you can turn the accessibility mode on or off with the button in the main menu.

The game is available on Android and iOS. Here are the links:

Blindie Match for iOS – (Update: Speech rate and voice issues are fixed now)

Blindie Match for Android

Note to sighted players:
If you have never before used a phone with VoiceOver or TalkBack enabled, the navigation might be a tad confusing – at least if you choose to turn on the accessibility mode. Sighted people tend to intuitively swipe into the direction that they want the UI focus to move, for example down to get to the next button in a list. This is not how screen readers work – since that wouldn’t make much sense for non-sighted people. Try swiping left and right instead, and double tap to press a button.

Why are there graphics? 

That’s a valid question.
Both the accessibility plugin’s code and actually blind players couldn’t care less whether there are beautifully animated, hand-drawn images bouncing around, or whether I just show you a blank screen. In fact, that is what I did at first.

Here is how the game actually looked that night after my game jam. Next to it is a screenshot of how it looks now.

The left picture shows the game's main menu with a light gray background and rectangular white buttons. The right picture shows the main menu with a strong purple background, and colorful round buttons in different sizes.

The game’s title was shortened from Blindie Match 3 to just Blindie Match

I spent a week cleaning up the code and adding in some minimal level of polish. The point of this plugin is not just to enable developers to make games specifically targeted at blind or sight-impaired players. It is equally supposed to give them a chance to make their regular, graphics-based games accessible. I don’t want to send the wrong message by making the first game I release with this plugin something that looks like it is actively uninviting sighted players.

Yes, I’m still a crappy artist

As you might know, I have no sense for colors or graphical design, and I can’t draw well. I’ve written about it before in a previous post. It’s OK, I like to believe I make up for my lack of skills in the graphics department by being really good at coding instead. So if you are wondering how I made the game look pretty, the answer is: I cheated. I bought a pre-made set of Match 3 graphics online. Luckily we live in an indie developer heaven, where you can buy what you can’t create yourself for very reasonable prices.

Unity Accessibility Plugin – Update 7 – Explore By Touch

As of today, the Unity Accessibility Plugin has a new feature: Explore By Touch!

Explore By What Now?

Explore By Touch is an accessibility feature on smartphones that let’s blind users run their finger over the screen – and the screen reader will read out what’s under their fingertip. The screen reader also makes sure that no buttons are accidentally pressed or an app is started unintentionally. This works on both Google TalkBack and iOS VoiceOver.

A blindfolded man is swinging a stick at a pinata. He misses and hits his girlfriend instead.

This was my original impression of Explore By Touch. Typical rookie mistake of a sighted person.

I originally thought that this is one feature that I wouldn’t need to re-create for the accessibility plugin, because I figured that no one would be using it much. I assumed no one wanted to blindly poke around on the screen to discover buttons. Why would anyone do that when swiping left and right navigates safely through the user interface?

Think again, you sighted fool!

As we all know, making blind assumptions is a bad thing. Pun intended.

Why would anyone use Explore By Touch?
The answer is simple. Because it is faster!

Navigating menus is a necessity, not fun. Doing it slowly by stepping through all elements on the screen one by one doesn’t make it any better.
If you already roughly know where a button is located, then using Explore By Touch is the next best thing to actually seeing where it is and clicking it directly.

Sure, the first time you get hit with a new menu, you might just trigger the Read From Top function and have the entire screen read out to you. But after that, you can poke around and find out where things are. And being blind doesn’t mean you cannot remember roughly where a button is located on the screen.

Thanks for the Feedback

I wouldn’t have come to this conclusion if I hadn’t received some very clear feedback from an actually blind person, who straight up told me the plugin would be pretty much useless without Explore By Touch. His opinion matched what I read in an article by Matt Gemmel in which he covers Myths about visually impaired users. After that I put my phone back in accessibility mode for a while and ultimately had to agree. It made me want that feature, too.

I am happy to announce that my plugin’s Explore By Touch is now fully functional.
Also: Thank you for your honest feedback, Scott!

3D Audio with Stereo Headphones

Today I want to write about an audio experiment I did for a game that I’m developing using my Unity Accessibility Plugin. It’s an accessible cooking game in which you prepare food for customers.

Problem 

The game pans the voices of customers to the left or right, depending on where they are standing. But when playing the game blind, I find it hard to quickly tell where a customer is. It always takes me a second and it requires a little concentration. This might not be a show-stopper, but it keeps bugging me.

Spatial Audio could be the solution.

Woman with headphones on has her eyes closed and focuses on the whether the sounds are left, right or center.

One burger to the left, two hot dogs to the right…

3D Vision and 3D Hearing

Never heard of spatial audio? Don’t worry, I won’t delve into the details. In a nutshell, it is a clever way of playing back audio so that the listener has a 3D audio effect using only stereo headphones. It’s not even that complicated. Humans only have two ears, so stereo is technically all we need.

Just like your brain uses the differences in the images from your two eyes and gives you 3D vision, it uses the differences in sound between your two ears and gives you 3D hearing.

The picture shows sound sources that are placed at different locations around a head.

Even with only two ears, you can always tell where a sound is coming from. Image Source

Here Is The General Idea

Because sound waves travel through the air (at the speed of sound), a sound coming from your left would reach your left ear first, and then your right ear. Furthermore, your head is in the way of the sound and will block part of it. This is called the head shadow, which dampens some frequencies from the sound before it reaches your other ear. While the differences are very small, your brain picks up on them and uses this very subtle information to place the sound in the world around you. There’s more playing into it, like sound reflected off walls close to you etc, but I said I wouldn’t delve into the science too far.

If you haven’t yet, listen to the famous barber shop example (skip to 2:54 to get the idea):

 

Famous Games using Spatial Audio

A game can reproduce this 3D effect artificially, by delaying the playback on the far ear and filtering frequencies. The only requirement for it to work is that the player wears headphones.

Games for visually impaired or blind players sometimes use spatial audio to allow players to find their way in the dark. Famous examples are the Papa Sangre games and The Nightjar. If you want to check them out, get The Nightjar if you like Benedict Cumberbatch, or Papa Sangre II if you are a fan of Sean Bean (no need to play Papa Sange I first).

Actors Benedict Cumberbatch and Sean Bean inside a voice recording studio. They look cute.

The Nightjar and Papa Sangre II have pretty darn good voice acting in my totally biased opinion.

Unity and Spatial Audio

Somethin’ Else, the developers of Papa Sange and Nightjar, have written their own audio engine and their results are impressive. I wanted to give spatial audio a try in my Sandbox dev environment, before bringing it into my game. But I am fan of not reinventing the wheel. I didn’t want to implement my own spatial audio system when there are some out there already.

There are plugins for spatial audio available for Unity. Here are the three that I found in the Asset Store: Spatial Audio, AstoundSound and dearVR. The last one looked the best, but the first one was the cheapest. For just a quick test that would do. But dearVR is now on my WishList and I will be hoping for a sale in the future.

Direct Comparison

Which one works better?
The majority of people should be able to tell the direction of the voice much faster when listening to the spatial audio example.
But have a listen for yourself.

Put on your headphones and try to determine whether the voice is coming from the left, the right or from the front. How fast are you able to tell the direction? Do you need to focus on it?

Tip: After you’ve listened to the spatial audio example, listen to the stereo one for a second time. It seems a lot worse now, doesn’t it?

I know the spatial audio sounds like it is panned more to the side than the stereo one, but that is not the case – it’s just the effect that the brain creates from the sounds.

EDIT: I just learned that the SoundCloud player above doesn’t work with all screen readers. Here are alternative links to the audio files:
Stereo Results
Spatial Results

Conclusion

While the stereo panning works “good enough” for the simplistic setting of my cooking game, it seems to me that the spatial audio requires less effort to determine where the sound is coming from. For that reason, I will put in the work and switch over to the spatial audio library. The difference might not be huge, but I am hoping it will make the gameplay feel more intuitive, because blind players need to pay less attention to locating the customer. After all, the gameplay is supposed to be about cooking, not about telling direction.

Notes for those interested in the topic:

  • If you want to play around with spatial audio in Unity but are unable to spend money on plugins, good news: Unity has just released a demo Spatialization SDK. You can find it here. I gave it a quick test and it seems to be working fine. Hint: If you check out the demos, you need to select the HRTF demo.
  • If you want to read more about the topic of spatial audio, but are bummed out by all the “binaural beats for better sleep” nonsense search results, try searching for ‘HRTF’ instead.
  • The developer Somethin’ Else used to license their audio engine (Papa Engine) to other developers. While this is no longer the case, their website hints that there might be a Unity compatible version in the future (see here). There is also a forum thread that claims the engine might become open source entirely. You can read about it here.