Tag: game play counselor

Categories
AI Powered Gaming Assistant Gaming Assistant Video games

Making Red Dead Redemption 2 accessible with Fridai, the voice assistant

Download Fridai Now

Accessibility and making games more inclusive has always been a high priority for our team, ever since the first community engagement we had about how Fridai was really helpful for our members to access information for their favorite games. We even had a documentary video made about our approach as part of the Digital Defenders Series, if you are interested you can watch the video here:

Voice access for games – our journey

The underlying technology of Fridai has always been voice, the way we process audio streams and use them to control the games and provide guides or strategies made our software unique in the industry. 

This ambient technology however is capable of so much more than we initially thought. The original pain point we set out to solve was to provide a way for gamers to eliminate distractions from their gameplay, access guides and walkthroughs without opening any other apps, or looking at other screens. While it remained as a selling point of Fridai, the main focus has been shifted towards an even bigger pain point we know in gaming: accessibility.

 

The current state of gaming in terms of accessibility

After thorough research we have found that even though games today include accessibility settings, they are most of the time far from covering the angles gamers really need. We have seen games where the only accessibility setting you could access was the subtitle settings, if you wanted to increase or decrease the size of the fonts or their colors. There was no mention of customizing keybinds, using a special controller or even providing a different way of interacting with the actual game.

Going even deeper into the numbers (US market) we can see that around 10% of the population actually has a form of impairment that would stop them from playing games. More interestingly 92% of gamers play games in spite of their impairment. The question here of course is which games can they play?

There are currently several solutions for disabled gamers to get more information about games, if they can play them or not. There are game review sites, coming from the community who test games from the accessibility perspective and after reading through them it is possible to see if a game is playable according to one’s disability or not. One of many problems with this is that you get to read a review after the game comes out, so in case you wanted to pre-order any of the games you could but risk not being able to play it. There is a great article about game previews and the lack of these actually affecting disabled gamers by our friends at Can I Play That, I recommend you to read that, too.

We have come to a conclusion that in many cases accessibility feels like an afterthought for game studios and we simply can not accept that not every game can be played by anyone.

Our contribution: NLP powered voice control

The way we approached this problem was setting a north star or a vision for our features: a person who has at least access to a headmouse and can press at least 1 button should be able to play any game they like. This might look like a limited vision, however for the first set of features we wanted to develop this set the bar very high already – since we are talking about voice technology and not neural links or eye tracking – just yet.

We know three aspects of Fridai that can make it a very powerful assistant when it comes to accessibility tools:

  1. Fridai can understand many accents and can figure out what was said even with very noisy backgrounds, pays attention to mispronunciations and gets the intent right even if the sentence being said was just remotely accurate
  2. We already have applied voice control for FIFA 20 as a multi tasking tool, where you could change players by voice
  3. Voice provides a new interaction layer when it comes to the players and their PCs or consoles and the only thing you need is the ability to speak, therefore it could help a huge amount of players.

Naturally at first we took a look at how players use current voice solutions, to create macros, setup voice recognition and so on. We had interviews with many players who were using any of these to really dig deep into the user experience. What we have found is that current voice solutions use a technique that can be described as strict match, so only if an exact word is understood will the associated action activate. In comparison, with Fridai, we could activate one action for a wider range of expressions, so as an example, you will not need to map “mount, mount horse, mount the horse etc.” one by one, once one of these is said Fridai can understand the intent of the player and work right then.

Once we saw how the natural language processing algorithm we build can aid users and can improve their gaming experience, compared to what services they use today, we set out to work out the weaknesses we had – again compared to the currently available solutions. One of these being the fact that Fridai needed to be activated using a wake word – so whenever you say “Hey Fridai” the AI wakes up waiting for the command. Now while this provides a hands-free solution, it actually can take a couple seconds to activate the action, hence might not provide a great gameplay experience. So we added keypress wakeup to Fridai, as a result you can quickly wake it up by pressing one key. (Remember our vision about only having to access one key and a head mouse? Now this is part of it.)

Once both of these were completed we set out to find a game that a lot of people want to play, but were previously unable to thanks to the complexity of the controls.

For a better explanation of how NLP works in games, here is one of our videos for the Witcher 3. It shows how different expressions activate the same intent.

Download Fridai Now

SpecialEffect and Red Dead Redemption 2

Our approach about picking a game was simple: we asked ourselves, which game is the one that tons of people would want to play but they can not for some reason. So we started reaching out to accessibility focused organizations, to understand their needs and ask them to provide feedback about Fridai from their users’ perspective.

This is how we’ve found the team at SpecialEffect, who responded to our query right away. We focused first around collecting their feedback about voice control and current solutions out there and we got a great overview of the market and needs. Even during our first call, we saw that bringing our NLP into the world of voice control, could provide the seamless interface to games that we set out to build and more importantly what service users of SpecialEffect were looking for. We have settled on building the first version of voice control powered by Fridai for Red Dead Redemption 2, as that is a game that is as complex as it gets from the control perspective and many in the audience of SpecialEffect were looking forward to a solution that enables them to play this game.

Now it was about time we built Fridai into Red Dead Redemption 2, with the goal of making it playable and enjoyable through voice access.

Implementing voice control for Red Dead Redemption 2

We already have laid the foundations for supporting voice control with Fridai, therefore it did not take us a long time to implement the actual voice based game control into the game. The first task we had to do was going through the keyboard settings and current keybinds and assign intents to each one of those.

Once it was completed, we started playing the game and constantly followed the instructions on the screens during mission and free roam gameplay, so we could capture the actions and understand what users would say as voice commands while playing. As an example, when it comes to the player interacting with the horse, we added the following intents that Fridai would later recognize:
Mount horse, dismount horse, call horse, hitch horse, brush horse, feed horse etc.

All of these commands activate a corresponding intent and thanks to our NLP engine even different variations of these work when players are speaking to Fridai, so we completely removed the necessity of having to remember exact commands.

Even though we received constant feedback from SpecialEffect as we progressed with development, we made time every week to check-in for a live testing of Fridai. Each of these sessions resulted in more patterns and expressions we needed to teach Fridai, however we got closer and closer to a solution, when it finally looked like this:


As a summary, now we are proud to say you can now play through most of the story missions and do free roam using only your voice. You only need access to a piece of hardware that can control your camera – but we will fix this soon, too.

Download Fridai Now
Categories
AI Powered Gaming Assistant Gaming Assistant Video games

Rethinking gaming guides with GPT3 – Fridai and the Witcher 3: Wild Hunt

The basic idea behind Fridai has always been to build an AI that behaves like a friend sitting with you while playing your favorite games – because we all remember how much fun it was to play together with our friends as kids. With an AI friend, of course, the experience will be different, but if we can make even a slight resemblance of how that felt, we will have built a truly amazing helper for games.

On the user experience side of this, our goal is to help everyone who enjoys gaming access information in a different way, without having to Alt+Tab out or even look at other screens. The first versions of the assistant delivered on this promise, however as we wanted to add more and more games, we seemed to have hit a barrier in how we could scale it efficiently to other games.

On our journey to find ways to accomplish a constantly improving, more human-like Fridai, we needed the answer to 3 questions:

  1. How can we create a better personality for Fridai?
  2. What is the most efficient way to keep Fridai’s knowledge updated?
  3. Can we make Fridai provide more native advice while gaming?

In this post we will only discuss our answers and methods to the last 2 questions, as they are related to the engine we’ve built and its connection to GPT3.

This post is not going to cover GPT-3 and how it works, so feel free to go to openai.com or read the actual whitepaper here

Ever since the first use cases of GPT-3 I have seen demos for, I’ve been thinking about how to apply the technology to reimagine gaming walkthroughs – and more importantly how gamers can access them today. So let me just insert a little sidetrack here, to explain how I think about game guides.

The idea started from a personal experience and problem, where whilst playing games, I actually enjoyed looking up guides and doing research on certain points (especially items and builds) – to be honest I needed to do this more than I can confess.

From all the content out there, I have found two types to be actually useful – longer websites where I can see the guides for my questions and of course videos with playthroughs. I am not going to go into explaining how much I hated having huge ad banners on the sites or even the fact that I had to wait 15 seconds even for a video to actually get started, we all know this (except of course, Youtube Premium subscribers, kudos).

The way I found these guides to be useful was when I was actually playing the game, for example in the Witcher 3 while collecting the griffin school gear, I constantly switched back and forth and alt+tabbed out so I would be able to find all the pieces. 

Research took time though and even if we disregard the actual search process or having to read multiple sites when looking up these pieces of information, I still had to read the actual content even the parts that were completely irrelevant to my current situation in the game.

And you guessed it right, in the meantime the game was on pause, running in the background. Then when I found the content, I started the process of switching back and forth between the game and the guide. I’ve been doing this for years and with so many games I’ve lost count.

With Fridai and GPT-3 however, I knew we could find the right way of recreating these guides in a way so it would give me filtered through content, only relevant to me, without having to leave the game – or even using my keyboard or mouse.

The marriage of these two products, would result in a human-like conversation that is tailored to the actual problem I would be facing in the game. Is this the answer to question number 2 and 3 above? Well, partially.

GPT-3 is a super creative technology and it has already been trained for usage, but putting it into an actual product requires a lot more than just connecting to it. In a way, you need to tame its creativity to make sure it works with your product. Therefore the first action we needed to take is to figure out an architecture that would serve the right content to GPT-3, so when it parses the unstructured content, it would be able to digest and produce the right piece of advice.

Then we needed to figure out where to get started, in terms of which game we should go after first. A game’s knowledge base can be very complicated, with hundreds of items, monsters, characters, quests and the list goes on. In order to make the experience work, we would need to dissect the game into the smallest parts, so whenever a user requires guidance about a monster or alchemy, we could provide the best possible response. Therefore we needed a game that we all knew, had little upcoming changes to its knowledge base but also had users who could then test it for us. This is why we picked the Witcher 3: Wild Hunt.

After really taking the game apart, in terms of content we identified the moving parts of how many elements exist in the game of which we will need to provide some form of content to the engine.

We had:

  • 132 monsters
  • 521 characters
  • 250 alchemy items
  • 304 pieces of armor
  • 30 witcher contracts

And more…

We collected individual pieces of content of all the elements of the knowledge base we have identified – either by having gamers in our community write something or looking up an actual guide and reformatting it. Then we saved all these pieces, made them searchable and tagged them.

Afterwards we needed to make sure, when the user’s query comes in about a given situation, we find the right piece of content, parse it and generate a response with GPT-3, format it to fit the game experience and feed it to the text-to-speech engine. Thus providing a real time, filtered piece of information about the game to the end-user.

This made us start working on an NLP (Natural Language Processing) layer to understand the user’s query, connect it to elasticsearch to find the correct file and then feed it to GPT-3, where we could take the most advantage of its creativity to craft the best response after parsing the unstructured data. (Note: I will make a post about the architecture of Fridai in a separate post).

And you can see the results in this video below, step by step:

This is a video of Mark playing the Witcher 3 with Fridai

At the end, user experience speaks for itself. GPT-3 works its magic on content that we can feed to it, however if there are no filters or NLP layers preceding the engine, it would just start crafting stories or irrelevant content for the user’s query.

The question of building a scalable, efficient way to keep Fridai’s knowledge base have been solved by the elastic engine that constantly updates itself from online guides and community generated content, while GPT-3 allows users to work with different queries, therefore getting different responses for “List the weaknesses of the griffin” or “Tell me some tips about defeating the griffin” – just the way you would ask your friend about how he solved certain problems while playing.

By the time I am writing this, we will have released Fridai’s version with full Witcher 3 support and I cannot wait to share more of our experiences in how people use this different, more interactive format of a guide or walkthrough to play with the Witcher.

Categories
AI Powered Gaming Assistant Gaming Assistant Video games

Fridai, The New Game Play Counselor

Fridai the new game play counselor2
Fridai the new game play counselor2

Back in the 1980s, Nintendo Entertainment System was on the uphill climb to success and recognized the need to have people readily available to help players get through difficult points in their video titles. The people they employed were known as “Nintendo Game Play Counselors” and, if you think about it, was every gamer’s dream job.

Unfortunately, the advent of the internet was right around the corner, which meant information was readily available at everyone’s fingertips as soon as they started installing dial-up in their home. Once the internet was more accessible in almost everyone’s household by 2005, Nintendo Game Play Counselor’s position was disbanded.

However, it was nice knowing we had people we could rely on in a pinch. Imagine playing the old Nintendo system and getting stuck on trying to beat Mario’s bosses. I ran into this issue with the NES and even with the Bowser fight on the Super Nintendo. Do you know how hard it was without the internet? Nintendo game play wasn’t challenging for a lot of children, but for me, it would have been helpful knowing there was someone I could talk to when I needed help.

As technology started advancing and Nintendo had the capability to put out more challenging games, there was the need for these professional Nintendo Game Play Counselors. Imagine playing thousands of hours worth of games all day so you could answer simple questions about bosses, hidden items, and other issues gamers were having at the time with their Nintendo game play.

Unfortunately, when Nintendo Game Play Counselors were phased out, we were forced to purchase strategy guides or strain our eyes even further on poorly developed websites with black backgrounds and green texts to read walkthroughs. Having that personal touch of hearing another person’s voice, was an experience we were deprived of for so many years. Sure there was YouTube and “Let’s Play” videos, but it never felt personal. No one wants to sit through ten minutes of someone talking about their day as they’re playing when all they want to do is figure out how to get past the part they’re stuck on.

Fortunately, the concept of Nintendo Game Play Counselors is still alive. No, you don’t have to pick up a landline phone to call into someone sitting at a call center ready to help. No, this concept lives on through easily accessible and responsive artificial intelligence known as Fridai, the gamer’s assistant.

How Can Fridai Help Me?

Imagine Fridai as a Nintendo Game Play Counselor, but without having to stop what you’re doing just so you can pick up your phone. Having to stop and minimize a game takes away from the immersion, and it’s just not fun having to sift through every website’s ads just to look for one piece of information.

Fridai can give you animal locations & prices so you can fully immerse yourself into hunting in Red Dead Redemption 2

For example, in Red Dead Redemption 2, there is a lot of information for the game’s content that isn’t readily available in the game. When I encounter a bear, I know it takes the fun away from the game when I have to pause it just to look down at my phone and look up what weapons I need to use to kill it. All I have to say is, “Hey Fridai!” and a comforting “bling” sound assures me that he is live and ready.. “How do I kill a grizzly bear?” I ask it as I don’t have to take my hands off of my keyboard or controller. Fridai then tells me the type of weapon I need to switch to and the ammo I should use for an effective kill.

Now imagine trying to do this with a Nintendo Game Play Counselor. You would have still needed to put your controller down, go to your landline phone, and call their hotline. Imagine Fridai as a similar concept except with more effectiveness and the cool-factor of getting a response as soon as you say something.

You don’t have to switch windows, fumble with your mouse cursor, or accidentally close a game. Fridai is right there, listening to your every need. If you need to step away from your PC for a moment, just say, “Friday, set my Steam status to busy,” and Fridai will set you up. You can also ask Fridai to play your favorite Spotify tunes for you, so you don’t have to stop to do it yourself.

Are You Ready To Give Fridai A Try?

  1. Visit hellofridai.com or search for Hello Fridai through your Microsoft Store 
  2. Download & sign up
  3. After logging in, browse through Fridai’s menu to learn more about what it offers.
  4. Enjoy your gameplay without ever having to leave it.