Unity Accessibility Plugin – Update 13 – How fast can you make an existing app accessible?

With my plugin now in beta testing stage, I was wondering how long it actually takes to make an existing app accessible using the plugin.

So I did a little experiment.

How long does it take?

For this test, I used a game from our company that had just come out of beta testing and was about to be released. Aside from some final bug fixes, the game was done and just waiting for the localization to finish. Not a single thought had been given to accessibility during development.

Result:
It took me a little less than four days to make it totally accessible.

A forest backdrop with the words 'Crafting Kingdom' written in front. A hammer and a pickaxe hover on either side of the text.

If you want to skip the rest of this blog and just check out the game, you can download it here:
Crafting Kingdom on the iOS App Store
Crafting Kingdom on the Google Play Store

Some Perspective

Maybe you have a game in mind that you’d like to make accessible and wonder how much work it would be.
Allow me to give you some additional context, so that you can put my result into perspective and draw your own conclusions.

The game I used is neither small nor simple, and has a ton of UI. It has a scrolling world map, over a dozen different locations with custom UI screens and dialogs, and almost each of them has a custom help screen. There are tutorial windows and statistics screens. In comparison, the Match 3 game I made had only a main menu screen, the game screen and a pause menu.

A medieval world map. Several locations are highlighted, such as a castle and a city.

This map can scroll in two directions

And while this game has a lot of text, it also makes heavy use of icons instead of text labels. It uses 32 different resource icons, a coin image to indicate a price, an X on cancel buttons, a check mark on confirm buttons, an image of a left arrow to indicate a back button… The list goes on. 

Many games use icons in lieu of text. Using images is prettier, uses less screen space, it makes the app playable by people who can’t read, and it saves money on localization. It is a win-win-win-win for the developer. Really the only drawback is that it makes accessibility a little harder.

One of the game screen showing a production pipeline. Graphics show that a ladder can be crafted from two pieces of timber and two pieces of leather.

A screen full of information, but barely any written text.

But not that much harder. Adding accessibility text to an image is easy enough, it just means it isn’t a one-click deal to add an accessibility component to the UI element and be done. You have to type in a name into the appropriate field of the accessibility component. For the product icons I had to modify the code to write the correct text into the accessibility component dynamically.
However, this particular game is localized in 12 languages, and I didn’t want to break with that. So I had to generate localization keys in our database as well which slowed me down a little.

The optional Extra Mile

I also didn’t just slap on the accessibility components and call it a day. That would have worked, but I knew that with a little effort the playability could be much improved.
For example, when a help text was useless for a blind player in its original form – e.g. “Click on the white house” – I created an alternate version of the text and had that read out instead.

I added in additional sound effects for extra feedback which are played only when accessibility mode is active. When a treasure chest appears somewhere on the map, a jingle sound will play. When the player cannot afford to buy something, he will be informed verbally that he doesn’t have enough money. Similarly, the game will speak up and tell the player when the warehouse has run full. Sighted players are notified of this with an animated, bright red colored text at the top of the screen.

Game screen showing the farm. It has four work stations, which are busy producing grain, meat and wool.

When a product is finished, a sound effect will be played.

Because this game has so many buttons and labels on the screen all the time, I grouped the UI elements by context, to make navigation a little quicker. The plugin supports up and down swipes (instead of the normal left and right swipes) to skip directly between groups. This means blind players can get to where they want faster, instead of having to cycle through every single UI element on screen. In a few dialogs I also set up a custom manual traversal order for the individual elements, because the most important information wasn’t always at the top. For example, the game always displays the name of the location that you are in at the bottom of the screen.

In one screen I actually decided to change the interface a little. I added a new layer of UI which offered buttons where there was a slider for sighted people. In that particular context it was the better choice. An accessible slider wouldn’t have given the same fine control to blind players as it gave to sighted ones in that particular context.

Results

With all that extra work it stands to reason whether I could have been done sooner, had I just done the bare minimum. On the other hand, I know my plugin better than anyone else. The speed advantage I gained from that probably more than made up for the difference.

You know your games best, so you can judge whether my results are applicable to you. In any case, they are a good indicator of the general scope. It doesn’t take months or weeks to make an app accessible. The work is more in the realm of days.

Just to be on the safe side – I am definitely not promising that you can make your game completely accessible within four days using my plugin. So don’t sue me.
(Write to me for help instead!)

How fast you will actually be will probably strongly depend on my ability to write good documentation. That’s what I am currently working on.

Afterthought: Not everybody does accessibility from the start, and that’s OK

As a general rule of thumb, I’d say if accessibility is added in early in development, it will be a smoother ride for everyone.

If you do it from the beginning, you quickly get into the habit of attaching an accessibility component to every UI right when you create it. And you implement any additional feedback for blind players at the same time that you actually implement the features. At no time during development will you be more familiar with the feature’s code. Thus, you are less likely to forget any special cases and can make sure everything runs smoothly in accessibility mode as well.

But in reality accessibility is often added later – possibly only after the game is finished. I really want to avoid saying ‘accessibility as an afterthought’ here in this context, which I stumble on a lot when browsing. It just sounds so negative. That’s unfair to any small game developer who wants to put in the work to make their app accessible. They are trying to do something good and don’t deserve to be berated for not having thought about it sooner.

Also consider this – everybody who adds in accessibility late in their current game might add it in early in their next one.

Unity Accessibility Plugin – Update 12 – Release Date and Anniversary

Roughly a year ago I started work on the Unity Accessibility Plugin. That doesn’t mean I have been working on it for one year – far from it. I honestly have no way of telling how many weekend and evening hours went into it. (It was a lot, though.)

Which is why I am a little excited to see that the plugin is finally so close to release!

What’s the holdup?

If you’ve followed this blog a little, you might remember that I already released a game with the plugin.

It might seem counter-intuitive, but having released a game with it doesn’t mean the plugin itself is ready for release. It’s one thing for me to use my own plugin, and another to have the plugin usable by others. It needs rigorous testing, documentation, a nice UI, demo projects and a support forum. And then everything needs to be bubble wrapped and uploaded to the Unity Asset Store.

When?

I hate disappointing people. So I won’t give an official release date. I couldn’t anyway, even if I tried.

Remember the old saying that the last 5% of a software project take 95% of the time? And I still need to do actual (I mean paid) work, too. It is impossible to predict anything with any reasonable certainty.

WHEN???

Ok, so here’s the jist.

The documentation just isn’t done. There’s not actually all that much left to do on the code before the initial release. Oh, yes, I have a long list of things on my wish list (from localization to Braille support) – but all of those are merely nice-to-have features that can wait until after release. The documentation can not.

Documentation, including at least one tutorial video, needs to be there from the start and it needs to be good.

I’m also in the process of finding a few developers that will beta test the plugin (and documentation) with their own projects. I don’t know how long that will take, either.

Sooon…ish?

So that’s where things are at and that’s why I can’t give a release date prediction. If you’re a developer, you probably understand this anyway, buy I just felt like I should explain why I am so evasive when asked for a date.

Unity Accessibility Plugin – Update 10 – A First Game

A few weekends ago I did my own personal Game Jam. The plugin is in a good place and I wanted to give it a test run.
If you have no idea what I am talking about, I recommend quickly reading the first post on this project and then come back. You can find it here.

A woman sits deeply focused at her desk, typing at her computer.

My One-Person Game Jam

I started at 7 pm, got some sugar (aka chocolate!) and some caffeine in me – and shortly after midnight I had a game up and running. It would have been sooner, but my dogs insisted that I feed them and then even demanded a walk around the neighborhood. What can I say – puppy eyes are my big weakness.

The result is an accessible Match 3 game.
Yes, you read correctly – another Match 3 game. This is a technology test, so stop moaning!

Screenshot of the puzzle game, showing a grid of different colored gems.

The world really needed another Match 3 game, didn’t it?

The game is quite simplistic without a deep story line. Or any story line, really. I wanted to test the plugin, so going for something simplistic made the most sense – and I didn’t have time for more anyway.

The plugin isn’t fully complete yet, there are a dozen small issues, and just as many missing features. Most importantly, many of the special gestures don’t work yet, like “Read From Top” or “Go Back”. But still, this is a first live action test for the plugin and I am a little excited about that!

Go check it out!

If anyone wants to see the plugin in action in its current state, please give it a go. And if you try it out, I would be grateful if you gave me your feedback. I’m not talking about a review (although I won’t stop you), I mean feedback about the navigation, the controls and the overall accessibility of everything. Just post it into a comment here or write me an email: mika@metalpopgames.com

Banner for Blindie Match Puzzle Game showing a few of the gems and the subtitle 'a fully accessible puzzle game'.

If VoiceOver or TalkBack are detected to be active, accessibility will turn on automatically. But you can turn the accessibility mode on or off with the button in the main menu.

The game is available on Android and iOS. Here are the links:

Blindie Match for iOS – (Update: Speech rate and voice issues are fixed now)

Blindie Match for Android

Note to sighted players:
If you have never before used a phone with VoiceOver or TalkBack enabled, the navigation might be a tad confusing – at least if you choose to turn on the accessibility mode. Sighted people tend to intuitively swipe into the direction that they want the UI focus to move, for example down to get to the next button in a list. This is not how screen readers work – since that wouldn’t make much sense for non-sighted people. Try swiping left and right instead, and double tap to press a button.

Why are there graphics? 

That’s a valid question.
Both the accessibility plugin’s code and actually blind players couldn’t care less whether there are beautifully animated, hand-drawn images bouncing around, or whether I just show you a blank screen. In fact, that is what I did at first.

Here is how the game actually looked that night after my game jam. Next to it is a screenshot of how it looks now.

The left picture shows the game's main menu with a light gray background and rectangular white buttons. The right picture shows the main menu with a strong purple background, and colorful round buttons in different sizes.

The game’s title was shortened from Blindie Match 3 to just Blindie Match

I spent a week cleaning up the code and adding in some minimal level of polish. The point of this plugin is not just to enable developers to make games specifically targeted at blind or sight-impaired players. It is equally supposed to give them a chance to make their regular, graphics-based games accessible. I don’t want to send the wrong message by making the first game I release with this plugin something that looks like it is actively uninviting sighted players.

Yes, I’m still a crappy artist

As you might know, I have no sense for colors or graphical design, and I can’t draw well. I’ve written about it before in a previous post. It’s OK, I like to believe I make up for my lack of skills in the graphics department by being really good at coding instead. So if you are wondering how I made the game look pretty, the answer is: I cheated. I bought a pre-made set of Match 3 graphics online. Luckily we live in an indie developer heaven, where you can buy what you can’t create yourself for very reasonable prices.

Unity Accessibility Plugin – Update 9 – Is It Worth Doing?

After having spent months working on an accessibility plugin for Unity, it might be time to discuss the most basic question:

Is it even worth doing?

Are there Unity developers out there interested in using this plugin? Is it attractive for devs to make games accessible? Are there enough blind gamers out there? Is there money in this? And are there other reasons to do it?

Arnold Schwarzenegger looking at a skull like Hamlet, pondering a big question.

To Be Or Not To Be?

The Big Question: Is it financially viable?

The logic is simple: If it isn’t financially viable for developers to make games for the blind community, then it wouldn’t be financially viable to make an accessibility plugin.

Generally speaking, if enough people need or demand a specific Unity plugin, then someone will already have created one. Considering there currently is no accessibility plugin already available on the asset store, I have a feeling I already know the answer to this one.

Woman standing excitedly in front of the Unity Asset Store.

You want it? The asset store has it. Usually.

Let me say right away that I didn’t start working on this plugin to make money. But I want to look at this from all angles, including the financial side. So let’s look at it in detail.

Is there a Market for Blind Gamers?

The World Health Organization estimates just below 300 million people worldwide to be visually impaired (Source). To give you something to compare, that is a little less than the current total population of the US. The American Foundation for the Blind estimates that 22 million adult Americans have some form of visual impairment, and in total roughly 10% of the population suffer some kind of vision loss. (Source)

Those numbers are high enough to present a very viable market share, but being visually impaired and being blind is not the same thing. Out of those 300 million people, only 40 million are estimated to be legally blind, with the remainder having some vision left.

A woman wearing a blindfold is trying to use an iPad

Surprisingly, I couldn’t find any statistics on how many sighted people play on their phones while wearing a blindfold.

There is no way to tell how many people in this group would benefit from a screen reader plugin, or are able to play games just fine. For the purpose of my calculations, I will go with the most pessimistic approach and assume that currently the market is capped at 40 million people worldwide.

40 Million Blind Gamers? Really?

While 40 million still sounds like a decent market, that number is misleading. Are those 40 million actually interested in games, and would there be a way to monetize games targeted at them?

About 20% of the general population play video games (Source). Assuming that the blind population shares this percentage, that would bring the number down to 8 million potential blind gamers. Those stand opposite a staggering 1.2 billion sighted gamers. That means the blind gaming market is only 0.7% the size of the one for the sighted.

Considering the size and that this number is worldwide, including many non-English speaking countries, the market just became a lot less attractive. Now we’re talking about not just adding accessibility, but localization as well, if a developer wants a chance to reach that market.

Blind man with a white cane

What’s more, according to the WHO, more than 80% of the visually impaired are 50+ years old. I was going to discount this age group at first, but according to a representative survey, this age group actually represents a fourth of all gamers. You can find that survey here.

Marketing to Blind Gamers

While age might not impact the market size, discoverability of games made for the blind does. What good is a game made for blind people, if they can’t find it? This is a real problem. I recently heard from a colleague visiting a convention for blind people that many aren’t even aware that games for the blind exist at all.

Currently, there is no way for a developer to reach the blind target audience on a broad spectrum. AdMob doesn’t offer a check box to show a game ad only to people with visually impairment. Neither Apple’s App Store nor the Google Play Store offer a way to find apps that are accessible. There are no categories or tags to sort or filter them. Blind users rely on word of mouth recommendations, podcast Let’s Plays and a number of websites that try their best to maintain lists and recommendations of accessible apps.

Logo of the Steam Game Store

Steam has great discoverability compared to Google and Apple.

Oddly enough, a simple user tag system like Steam has successfully implemented would instantly fix this problem. I mentioned this to a friend and he noted that the app stores might actually not want to make discoverability too easy. Google and until recently Apple are also selling advertisements via their ad networks, so it would directly cut down on their income.

Do They Have Smartphones? Tablets? …or Internet?

This one is a tricky question – though it seems that with the exception of Cuba and North Korea, this doesn’t seem to be a problem. The statistics indicate that we are well on the way to everybody on the planet having a cellphone: Countries Population and Number of Mobile Phones. When looking at those numbers, keep in mind that they can be misleading. Even though statistically every man, woman and child in Hong Kong owns 2.4 mobile phones – it is safe to assume that there are plenty of people there that don’t have even a single one.

In a nutshell, don’t be fooled into assuming that all blind people in all countries have phones, internet or computers. I couldn’t find any good numbers for this, the closest I found was some statistics on how many people use a screen reader. But the survey was small and possibly not representative. I don’t know if the numbers outside of the US are even statistically relevant, due to the small number of participants. But you can find the survey here: Screen Reader Survey

At least for the US, the numbers are looking good. The majority of blind people seem to use or have access to a phone or computer and internet.

Monetization – Do They Have Money?

The question might sound cruel, but I said I was going to look at this from all angles.

Scrooge McDuck carrying a briefcase with money falling out.

The WHO states the 90% of people with visual impairments live in low-income settings. This can mean a number of things, like living in an underdeveloped country with inadequate access to health care. It could also simply mean being unemployed, possibly because of the visually impairment. The numbers on the National Federation of the Blind website point to an unemployment rate of near 20% for visually impaired adults in the US alone. (Source)

Monetization – Whale Fishing

Low income might not mean poor, but it means that this group will likely not spend huge amounts of money on games. In-app purchases in mobile games usually start at 99 cents a pop. This alone will hardly ruin anyone, but those 99 cents are not where a developer’s money comes from. The money is made through whales. And with 90% of a group being classified as low-income, it’s reasonable to expect there to be less whales in this group than in the general population.

Picture of a whale smiling lazily.

If you are unfamiliar with the term whale, here is an article that explains it: 0.15% Of Gamers Bring In 50% Of The Revenue

There are other means of monetization, such as simply charging a premium for a game, and showing advertisements. The premium game approach might work well with this group, but it doesn’t fly well with the rest of the gaming world, where everything has to be free. Advertisements are even trickier. Ad providers that pay-per-click will be useless since the chances of a blind user clicking on an ad are relatively slim. Rewarded video ads however could well work, as they usually pay for watched videos and not clicks.

If there was a way to include radio ads in mobile games, it would be a great way to gain some of that earning potential back.

An Unattractive Target Audience?

Let’s conclude that the majority of blind people worldwide has or could get access to a cellphone, but can’t spend a huge amount of money on games.

The market is small, and there are few marketing options. The app discoverability is poor and the monetization options aren’t as plentiful.

UPDATE: Before you read on, you might want to check out Ian Hamilton’s comment to this blog post. He raises some excellent points as to why and how blind gamers can actually be quite an attractive audience, even financially speaking. I won’t steal his words, so this link will take you directly to the comments: Comments

Big Publishers vs. Small Developers

That makes blind gamers into an unattractive target audience for the big players – if you look at it from purely a financial standpoint. Large user bases and whale fishing are important to big developers and publishers. Our own studio has been turned down by publishers because our games didn’t offer the ability to spend endless amounts of money.

Having said that – it’s possible that publishers and big development studios are interested in accessibility for other reasons. Such reasons could be maintaining a good image. It could be due to public pressure. Or because it is the right thing to do in an inclusive, civilized, western society. Or simply because the CEO has a relative who’s blind.

Smaller developers and indies on the other hand might be drawn to this group a lot more. The market might be smaller, but there is potential for growth, and it isn’t yet overcrowded with games. In other words: There is a LOT less competition. And smaller devs need a LOT less money to be financially viable.

A homeless man sitting on the street begging - next to a sign that reads: "will code for food".

Indie developers need less than big developers to be profitable.

The design challenge of making games for a blind audience could also be attractive to indies. If the blind are targeted exclusively, games can be made without any graphics at all, which could further entice small developers, as audio can be cheaper than graphics. In fact, I’m surprised not more indies have flooded into this niche yet. Maybe my plugin could give them the push they need. (Well – one can dream, right?)

The problem of discoverability for accessible apps and games remains. It is potentially the most important issue that’s blocking developers from entering this market niche. Incidentally, it is also the easiest to fix, if Google or Apple were willing to make a minor addition to their stores, in the form of a tagging system for example.

Conclusion

Summing this up is not an easy task.
It seems that estimating the earnings potential of this market is simply impossible. Which means it’s impossible to judge whether anyone would actually buy my plugin on the asset store.

So… the prospect of earning lots of money obviously won’t be the force driving me.

So? Is It Worth Doing?

Spoilers: Yeah, I think it is.
There are other things to consider aside from financial feasibility. Here are some of them:

Self-interest.
I have often wondered what would happen if I lost my eyesight. Maybe every programmer has wondered this at some point. I need my eyes for work. Sure, it wouldn’t be the end of the world, but personally, I would rather lose an arm or a leg than my eyes. If I go blind one day, I want to still be able to play games. Heck, I want to still make games. But one thing at a time. Self-interest is definitely at play here.

It might make people happy.
Who doesn’t love that? The feedback I have gotten so far is great and has motivated me like nothing else. Every time I read a positive comment on my thread at AppleVis or talk to someone on Twitter about this project, I feel the urge to drop everything else and work on the plugin some more. This is the kind of motivation that will keep you coding away the entire weekend without it ever feeling like work.

I want to see if I can do it.
That to me is a perfectly valid reason to do a lot of things, and maybe you can relate. Also, this is a missing feature, a hole that needs closing, and that keeps bugging me,

I really don’t expect to make heaps of money with it, and the numbers back me up on this. I’d be happy if the time I will have to spend on support is covered. Because that would mean I get to spend time on fixing bugs and improving the plugin

Final Thoughts – and numbers aren’t everything

Remember the Ice Bucket Challenge?

Actor Benedict Cumberbatch sits on a bench shivering, while being doused with ice water from above.

Only 0.01% of Americans have ALS, an estimated 30,000 people (Source). That number obviously is far FAR smaller than the 22 million adults with visual impairments (more than 700 times, to be exact). Yet, the ALS Ice Bucket challenge raised over 100 million dollars. And what’s even more, it raised public awareness. I bet if I talked to ten random people on the street, four of them will have heard of ALS. Or at least that crazy time when everybody filmed themselves pouring ice-cold water over their heads for some charity thing. But all of them will stare at me with wide eyes when told that blind people can actually use smartphones.

Numbers and financial feasibility aren’t everything. Developers, especially the big ones, have other good reasons to consider accessibility, and public image is important to them. Did you know there is actually a petition underway to entice the Pokemon GO developers to make the game accessible? It’s worth a try, since public awareness can move mountains.

My final take on the topic is this:
If it was easy, comfortable and convenient, more people would consider making their games accessible.
And that is one more reason to make this plugin.

 

Afterthoughts

Public Exposure

Unity currently has close to 6 million registered users (Source). Of course, not all of those actually will ever develop or release anything with the engine, but many will create multiple games, again and again.

34% of the Top 1000 Free Mobile Apps in 2016 Q1 were made with Unity.
This is relevant, because it says that if there is a popular game out there that everybody is playing, there is a 1 in 3 chance that it was made with Unity. But if it was made with Unity, then it will automatically be not accessible with screen readers. Case and Point: Pokemon GO!

Imagine the public exposure the whole accessibility issue could be getting if those popular and visible apps start caring about accessibility? Even if they only care about it because I offered them a convenient and inexpensive way to do so.

Non-Gaming Developers

I also want to mention use cases of the plugin outside of gaming.

There is probably a number of users currently not using Unity, because their software has to be Section 508 compliant. I imagine this includes a lot of public institutions as well as government subcontractors, the military and serious games developers. The plugin could introduce Unity to a whole new range of users. I am not sure that the plugin is enough to make apps actually 508 compliant, but it is a start.

Unity Accessibility Plugin – Update 7 – Explore By Touch

As of today, the Unity Accessibility Plugin has a new feature: Explore By Touch!

Explore By What Now?

Explore By Touch is an accessibility feature on smartphones that let’s blind users run their finger over the screen – and the screen reader will read out what’s under their fingertip. The screen reader also makes sure that no buttons are accidentally pressed or an app is started unintentionally. This works on both Google TalkBack and iOS VoiceOver.

A blindfolded man is swinging a stick at a pinata. He misses and hits his girlfriend instead.

This was my original impression of Explore By Touch. Typical rookie mistake of a sighted person.

I originally thought that this is one feature that I wouldn’t need to re-create for the accessibility plugin, because I figured that no one would be using it much. I assumed no one wanted to blindly poke around on the screen to discover buttons. Why would anyone do that when swiping left and right navigates safely through the user interface?

Think again, you sighted fool!

As we all know, making blind assumptions is a bad thing. Pun intended.

Why would anyone use Explore By Touch?
The answer is simple. Because it is faster!

Navigating menus is a necessity, not fun. Doing it slowly by stepping through all elements on the screen one by one doesn’t make it any better.
If you already roughly know where a button is located, then using Explore By Touch is the next best thing to actually seeing where it is and clicking it directly.

Sure, the first time you get hit with a new menu, you might just trigger the Read From Top function and have the entire screen read out to you. But after that, you can poke around and find out where things are. And being blind doesn’t mean you cannot remember roughly where a button is located on the screen.

Thanks for the Feedback

I wouldn’t have come to this conclusion if I hadn’t received some very clear feedback from an actually blind person, who straight up told me the plugin would be pretty much useless without Explore By Touch. His opinion matched what I read in an article by Matt Gemmel in which he covers Myths about visually impaired users. After that I put my phone back in accessibility mode for a while and ultimately had to agree. It made me want that feature, too.

I am happy to announce that my plugin’s Explore By Touch is now fully functional.
Also: Thank you for your honest feedback, Scott!

3D Audio with Stereo Headphones

Today I want to write about an audio experiment I did for a game that I’m developing using my Unity Accessibility Plugin. It’s an accessible cooking game in which you prepare food for customers.

Problem 

The game pans the voices of customers to the left or right, depending on where they are standing. But when playing the game blind, I find it hard to quickly tell where a customer is. It always takes me a second and it requires a little concentration. This might not be a show-stopper, but it keeps bugging me.

Spatial Audio could be the solution.

Woman with headphones on has her eyes closed and focuses on the whether the sounds are left, right or center.

One burger to the left, two hot dogs to the right…

3D Vision and 3D Hearing

Never heard of spatial audio? Don’t worry, I won’t delve into the details. In a nutshell, it is a clever way of playing back audio so that the listener has a 3D audio effect using only stereo headphones. It’s not even that complicated. Humans only have two ears, so stereo is technically all we need.

Just like your brain uses the differences in the images from your two eyes and gives you 3D vision, it uses the differences in sound between your two ears and gives you 3D hearing.

The picture shows sound sources that are placed at different locations around a head.

Even with only two ears, you can always tell where a sound is coming from. Image Source

Here Is The General Idea

Because sound waves travel through the air (at the speed of sound), a sound coming from your left would reach your left ear first, and then your right ear. Furthermore, your head is in the way of the sound and will block part of it. This is called the head shadow, which dampens some frequencies from the sound before it reaches your other ear. While the differences are very small, your brain picks up on them and uses this very subtle information to place the sound in the world around you. There’s more playing into it, like sound reflected off walls close to you etc, but I said I wouldn’t delve into the science too far.

If you haven’t yet, listen to the famous barber shop example (skip to 2:54 to get the idea):

 

Famous Games using Spatial Audio

A game can reproduce this 3D effect artificially, by delaying the playback on the far ear and filtering frequencies. The only requirement for it to work is that the player wears headphones.

Games for visually impaired or blind players sometimes use spatial audio to allow players to find their way in the dark. Famous examples are the Papa Sangre games and The Nightjar. If you want to check them out, get The Nightjar if you like Benedict Cumberbatch, or Papa Sangre II if you are a fan of Sean Bean (no need to play Papa Sange I first).

Actors Benedict Cumberbatch and Sean Bean inside a voice recording studio. They look cute.

The Nightjar and Papa Sangre II have pretty darn good voice acting in my totally biased opinion.

Unity and Spatial Audio

Somethin’ Else, the developers of Papa Sange and Nightjar, have written their own audio engine and their results are impressive. I wanted to give spatial audio a try in my Sandbox dev environment, before bringing it into my game. But I am fan of not reinventing the wheel. I didn’t want to implement my own spatial audio system when there are some out there already.

There are plugins for spatial audio available for Unity. Here are the three that I found in the Asset Store: Spatial Audio, AstoundSound and dearVR. The last one looked the best, but the first one was the cheapest. For just a quick test that would do. But dearVR is now on my WishList and I will be hoping for a sale in the future.

Direct Comparison

Which one works better?
The majority of people should be able to tell the direction of the voice much faster when listening to the spatial audio example.
But have a listen for yourself.

Put on your headphones and try to determine whether the voice is coming from the left, the right or from the front. How fast are you able to tell the direction? Do you need to focus on it?

Tip: After you’ve listened to the spatial audio example, listen to the stereo one for a second time. It seems a lot worse now, doesn’t it?

I know the spatial audio sounds like it is panned more to the side than the stereo one, but that is not the case – it’s just the effect that the brain creates from the sounds.

EDIT: I just learned that the SoundCloud player above doesn’t work with all screen readers. Here are alternative links to the audio files:
Stereo Results
Spatial Results

Conclusion

While the stereo panning works “good enough” for the simplistic setting of my cooking game, it seems to me that the spatial audio requires less effort to determine where the sound is coming from. For that reason, I will put in the work and switch over to the spatial audio library. The difference might not be huge, but I am hoping it will make the gameplay feel more intuitive, because blind players need to pay less attention to locating the customer. After all, the gameplay is supposed to be about cooking, not about telling direction.

Notes for those interested in the topic:

  • If you want to play around with spatial audio in Unity but are unable to spend money on plugins, good news: Unity has just released a demo Spatialization SDK. You can find it here. I gave it a quick test and it seems to be working fine. Hint: If you check out the demos, you need to select the HRTF demo.
  • If you want to read more about the topic of spatial audio, but are bummed out by all the “binaural beats for better sleep” nonsense search results, try searching for ‘HRTF’ instead.
  • The developer Somethin’ Else used to license their audio engine (Papa Engine) to other developers. While this is no longer the case, their website hints that there might be a Unity compatible version in the future (see here). There is also a forum thread that claims the engine might become open source entirely. You can read about it here.

 

Dev Update #14 – Concept Graphics and Aspect Ratios

Have your base (classes) covered!

In spite of what I wrote in my previous blog post, I realized I need to create one more department before I can finally tackle the Build Menu: The Management department. Like the Design department, this one works differently from the production floors. This makes a difference from the coding point of view – as I want a base class to take care of all floors, and then handle only specialized stuff in derived classes. And like I said, I want all my bases covered.

Here is the mockup graphic for the Management floor:
Management

The idea was that a Management floor can employ up to four development directors. Each director would allow for one game to be in production. This way the player can work on multiple games at the same time. But I made a pretty noobish mistake when creating the above mockup – I hadn’t given any thought to the aspect ratio of my floors.

Aspect Ratio, anyone?

When creating the actual game graphics for this floor based on this mockup, I ran into some trouble. I hadn’t really thought about aspect ratios and screen width when I created those images. That was a big mistake, as it turns out. The mockup screens were too wide. There was simply no room left on the right side of the screen for the two big purple buttons, not unless I scaled the height down.  But then everything else in the floor would become tiny. To counteract, I would have to scale up the characters. But then I didn’t have enough room above the head of the managers to put the actual game icons. My characters were simply too big.

So I redesigned that department completely. Here’s the result (middle floor):

BlogPost15

Instead of placing the icons above the characters’ heads, they were placed next to them, sorta like a PowerPoint screen in the background. It actually gives it a nice manager look and feel. And as a bonus, this will free up some screen space at the top of each floor where I have been wanting to put a floor number. In another change, the Release and Trash buttons on the right are done away with. Instead, scripts can be released or trashed by clicking on them and opening up their details dialog.

I definitely learned my lesson and will create future mockups in a sensible aspect ratio.

Now, this time for real – next up is going to be the build menu.