Categories
AI Powered Gaming Assistant Gaming Assistant Video games

Making Red Dead Redemption 2 accessible with Fridai, the voice assistant

Accessibility and making games more inclusive has always been a high priority for our team, ever since the first community engagement we had about how Fridai was really helpful for our members to access information for their favorite games. We even had a documentary video made about our approach as part of the Digital Defenders Series, if you are interested you can watch the video here:

Voice access for games – our journey

The underlying technology of Fridai has always been voice, the way we process audio streams and use them to control the games and provide guides or strategies made our software unique in the industry. 

This ambient technology however is capable of so much more than we initially thought. The original pain point we set out to solve was to provide a way for gamers to eliminate distractions from their gameplay, access guides and walkthroughs without opening any other apps, or looking at other screens. While it remained as a selling point of Fridai, the main focus has been shifted towards an even bigger pain point we know in gaming: accessibility.

 

The current state of gaming in terms of accessibility

After thorough research we have found that even though games today include accessibility settings, they are most of the time far from covering the angles gamers really need. We have seen games where the only accessibility setting you could access was the subtitle settings, if you wanted to increase or decrease the size of the fonts or their colors. There was no mention of customizing keybinds, using a special controller or even providing a different way of interacting with the actual game.

Going even deeper into the numbers (US market) we can see that around 10% of the population actually has a form of impairment that would stop them from playing games. More interestingly 92% of gamers play games in spite of their impairment. The question here of course is which games can they play?

There are currently several solutions for disabled gamers to get more information about games, if they can play them or not. There are game review sites, coming from the community who test games from the accessibility perspective and after reading through them it is possible to see if a game is playable according to one’s disability or not. One of many problems with this is that you get to read a review after the game comes out, so in case you wanted to pre-order any of the games you could but risk not being able to play it. There is a great article about game previews and the lack of these actually affecting disabled gamers by our friends at Can I Play That, I recommend you to read that, too.

We have come to a conclusion that in many cases accessibility feels like an afterthought for game studios and we simply can not accept that not every game can be played by anyone.

Our contribution: NLP powered voice control

The way we approached this problem was setting a north star or a vision for our features: a person who has at least access to a headmouse and can press at least 1 button should be able to play any game they like. This might look like a limited vision, however for the first set of features we wanted to develop this set the bar very high already – since we are talking about voice technology and not neural links or eye tracking – just yet.

We know three aspects of Fridai that can make it a very powerful assistant when it comes to accessibility tools:

  1. Fridai can understand many accents and can figure out what was said even with very noisy backgrounds, pays attention to mispronunciations and gets the intent right even if the sentence being said was just remotely accurate
  2. We already have applied voice control for FIFA 20 as a multi tasking tool, where you could change players by voice
  3. Voice provides a new interaction layer when it comes to the players and their PCs or consoles and the only thing you need is the ability to speak, therefore it could help a huge amount of players.

Naturally at first we took a look at how players use current voice solutions, to create macros, setup voice recognition and so on. We had interviews with many players who were using any of these to really dig deep into the user experience. What we have found is that current voice solutions use a technique that can be described as strict match, so only if an exact word is understood will the associated action activate. In comparison, with Fridai, we could activate one action for a wider range of expressions, so as an example, you will not need to map “mount, mount horse, mount the horse etc.” one by one, once one of these is said Fridai can understand the intent of the player and work right then.

Once we saw how the natural language processing algorithm we build can aid users and can improve their gaming experience, compared to what services they use today, we set out to work out the weaknesses we had – again compared to the currently available solutions. One of these being the fact that Fridai needed to be activated using a wake word – so whenever you say “Hey Fridai” the AI wakes up waiting for the command. Now while this provides a hands-free solution, it actually can take a couple seconds to activate the action, hence might not provide a great gameplay experience. So we added keypress wakeup to Fridai, as a result you can quickly wake it up by pressing one key. (Remember our vision about only having to access one key and a head mouse? Now this is part of it.)

Once both of these were completed we set out to find a game that a lot of people want to play, but were previously unable to thanks to the complexity of the controls.

For a better explanation of how NLP works in games, here is one of our videos for the Witcher 3. It shows how different expressions activate the same intent.

SpecialEffect and Red Dead Redemption 2

Our approach about picking a game was simple: we asked ourselves, which game is the one that tons of people would want to play but they can not for some reason. So we started reaching out to accessibility focused organizations, to understand their needs and ask them to provide feedback about Fridai from their users’ perspective.

This is how we’ve found the team at SpecialEffect, who responded to our query right away. We focused first around collecting their feedback about voice control and current solutions out there and we got a great overview of the market and needs. Even during our first call, we saw that bringing our NLP into the world of voice control, could provide the seamless interface to games that we set out to build and more importantly what service users of SpecialEffect were looking for. We have settled on building the first version of voice control powered by Fridai for Red Dead Redemption 2, as that is a game that is as complex as it gets from the control perspective and many in the audience of SpecialEffect were looking forward to a solution that enables them to play this game.

Now it was about time we built Fridai into Red Dead Redemption 2, with the goal of making it playable and enjoyable through voice access.

Implementing voice control for Red Dead Redemption 2

We already have laid the foundations for supporting voice control with Fridai, therefore it did not take us a long time to implement the actual voice based game control into the game. The first task we had to do was going through the keyboard settings and current keybinds and assign intents to each one of those.

Once it was completed, we started playing the game and constantly followed the instructions on the screens during mission and free roam gameplay, so we could capture the actions and understand what users would say as voice commands while playing. As an example, when it comes to the player interacting with the horse, we added the following intents that Fridai would later recognize:
Mount horse, dismount horse, call horse, hitch horse, brush horse, feed horse etc.

All of these commands activate a corresponding intent and thanks to our NLP engine even different variations of these work when players are speaking to Fridai, so we completely removed the necessity of having to remember exact commands.

Even though we received constant feedback from SpecialEffect as we progressed with development, we made time every week to check-in for a live testing of Fridai. Each of these sessions resulted in more patterns and expressions we needed to teach Fridai, however we got closer and closer to a solution, when it finally looked like this:


As a summary, now we are proud to say you can now play through most of the story missions and do free roam using only your voice. You only need access to a piece of hardware that can control your camera – but we will fix this soon, too.