Looking at Machine Learning in Games

George Dolbier (CTO, IBM Interactive Media) explores how Machine Learning platforms (like IBM's Watson or Google's initiatives) are just now starting to make their way into games. Whether adding conversational interfaces, providing better player behavior modeling, or improving game design, machine learning platforms are going to make games more fun, engaging and natural to play. This talk with go through 2 different case studies - Plight of the Zombie (SparkPlug) and The Suspect - to show how ML platforms have been used in games, and the challenges of interfacing with an ML platform from a game engine.

Related topics:  Machine Learning


Download George's Slides

Thank you all for staying. You have excellent stamina. I'd like to close by talking about something wonderful. So, 20 years ago this year, and AI beat a grand master chess player-- Garry Kasparov. And just a few weeks ago, an AI from Google beat the grandmaster Go player. 

The number of potential moves in a chess game has been equated to the number of atoms, or the number of grains of sand on beaches. The number of moves that are potential on a game of Go are equal to the number of ohms in the universe. 

So within 20 years, the technology of artificial intelligence has progressed from a point where the system that beat Garry Kasparov was a brute force decision tree. Essentially every move in every chess match had been coded, input into a big supercomputer-- that could go through and evaluate every move using a brute force technique pretty quickly. 

And unfortunately, that AI system, although it was a great demonstration of the potential of computing power 20 years ago, was not generally repurposable to solve problems like cancer. Or world hunger. The technologies behind the types of machine learning platforms that are available today are generally repurposable 

So the system AlphaGo from Google that beat a grandmaster and beat quite handily his generally repurposable-- thank you-- to solve other problems. So taking a step back, what is machine learning, and how is it different than a lot of the technologies that we've been talking about today? Machine learning and developing a system, a game, or a machine learning platform is less like programming, and a lot more like testing or teaching a child. And to put it simply, machine learning in a machine learning platform is software that learns and produces-- so produces just like classic software-- but this is a software system that learns and produces without being explicitly programmed. And if you were classically trained as a software engineer or a code developer, learning machine language is a little different. And sometimes a lot different. Because you're not explicitly coding. You're not explicitly controlling the environment. You are-- it's more like you're guiding a very naive child that can think really really fast. 

What I'm going to do today in this presentation is I'm going to go through case studies. And I'm glad I'm following the indie panel. I'm going to go through two case studies where indie game devs have used machine learning in very different ways to solve some very interesting and systemic challenges that we face as an industry. But an interesting thing about machine learning is machine learning platforms-- like from IBM, or Google, or Facebook, or Microsoft, or Tata-- are already very mature technologies. 

And these are technologies that have been used to-- in our case, solve some pretty significant medical problems. Our Watson Technology. The first use case after the Jeopardy challenge in 2011 was to ingest and understand every word that had ever been written about any form of cancer. And then understand that massive body of knowledge. And then be handed over Oncologists to be able to help individual patients and entire populations be treated better, generate better outcomes. 

So whereas the game industry often-- like VR, like 3D, will-- like social media-- will adopt a technology theory very early in it's infancy, and the industry will see the potential of the technology. This is a technology that has direct applicability in the games industry has been matured outside of the industry. And we're going to benefit tremendously. 

If your QR scanner works, while I'm talking, you can play with a particular demo. This demo is a personality insights generator, so this is an AI that you can have read all of your tweets. And then the AI understands language and has been trained on clinical psychological methods, and will actually give you a cycle-- a psychologist personality profile based on your tweets. And it might be something that would be fun to do while you're listening to me talk. 

So I'm going to go through two case studies. I've got lots of case studies. There will be a bunch of links at the end. One of the things that we started out looking at is-- one of the most important challenges in the industry is, how you get game design right? How to create a game that engages player as an individual and keeps them interested? 

So as a game designer, game designers kind of have a mega challenge and [INAUDIBLE] challenges. And in this-- in this context-- we're thinking about the game design of a simple puzzle game. And we chose this case study because it boils down a really fundamental problem. You know, how you engage, how do you retain, how you convert players? And boils it down to something simple that people can get both their heads around, how machine learning can be applied, and that we could actually validate, is this-- you know, can machine learning actually be applied to game design? 

So the big challenge a game designer has is, how do you create a progression curve? An extended experience in your game, so that people start simple, and you're progressively giving them more challenge? Everybody knows that a linear progression curve leads to boredom and abandon pretty quickly. So what the engagement curve look like? Do we give them two hard levels and a simple level? Do you give them one simple level, and then one hard level? What is simple and what is hard? 

And then on an individual level, you're designing individual experiences in a puzzle game. It's-- you know,-- 30-, 60-, 90-second experiences, each puzzle. You give each one of those puzzles kind of a seeded experience level. And then you create all this content, 200, 300 levels, and then you put them in order, that you think the average market would find interesting. And effectively, you're making two pretty big assumptions that the game is going to live or die. And that is this progression curve that you designed is actually interesting. And the content you created actually can fit and deliver that design experience. 

So the first case study here from Sparkplug Games. Shout out to John O'Neil, the CEO of Sparkplug Games. This is a company we work with in a lot of our gaming-- game projects. Independent game developer. Was developing a rather simple puzzle game called Plight of the Zombie. Your job is to go around and eat brains. You know, it's standard puzzle game fare. You know, an ideal test bin, where they have their design progression curve and a couple hundred levels. 

And the challenge was, can we use machine learning? But then, what kind of data do we collect? And how much data do we need to collect? The indie panel talked a lot about when you're working for a big studio, you've got all this behavior data. But what if you're an indie, and you don't have a lot of player data? Or you're working on something really new, and you don't know what people are going to like? 

With machine learning, you have the opportunity with a simple API call, collect all your telemetry data, and send it to a machine learning system. And have the players own behavior tell you what they'd like. And that's kind of the goal for this particular project. 

So we started out with three simple variables. One variable was the seeded difficulty level for each level in this puzzle game. And then two pieces of data that came from the players. Like, how long it took them to complete a level, and how many times they would retry. [INAUDIBLE] Guy Kawasaki never put words on a slide. This is a lot of words. And I'm now so proud. I'm the first and only presenter today that's actually showing code. Or maybe second. 

But this is actually from Unity making a single call to a machine learning platform to have a machine learning platform evaluate all of the players' data-- so all the player's telemetry. And based on that telemetry, pick the best next experience. So some players naturally get the mechanic of a puzzle game. Some players don't. So again, getting back to, I want my players to have an engaging, lasting experience. A player that's having a hard time with my game mechanic-- my progression curve may still be accurate for that player, but it is may be expressed in the simplest sets of puzzles. 

So if I'm scoring level difficultiess-- the difficulty of levels on the scale of one to 100-- it may be that players that are having a hard time with my mechanic are going to be engaged by levels that are scoring a 50. Whereas people that are finding my puzzles really easy may be wanting-- may be engaged by giving them difficulties that range in the 70's to 90's. So one of the-- one of the first things we found out is that, like with Google Maps, and like with many micro-services that are available on the internet, machine learning platforms often express their interfaces as REST based APIs. 

If you tried to integrate a game engine within the the public micro-service or REST based API, you know you need to write a lot of framework. So that was the first-- first challenge. Once we got over that challenge, the APIs we found were really simple to use. Integrating with a machine learning platform was literally 20 lines of code. And that 20 lines of code took an indie game-- a simple puzzle game-- from crossing your fingers and hoping that my levels-- my difficulty levels for my-- were seeded correctly, and my progression curve was accurate. 

Another thing we didn't know when we tried this experiment was, you know, our assumption was, you're going to need lots and lots of data in order to make good decisions, even on an individual basis. And we-- you know, our first assumption was, this is probably not going to work, because you won't be able to make a good decision until the player has played 30 or 40 levels. By then, you've already, you know, won or failed. 

What we found was in this game, there were three tutorial levels, and two dead simple levels. And we found that a machine learning platform-- and in this case we're using an API-- that will not give you the best answer, but give you a range of answers and let you pick the one of the top three [INAUDIBLE] decision analytics. We found that after the fifth bit of telemetry-- so the fifth play through from a player-- the machine learning platform was already able to pick content that was appropriate for a particular players. 

Our big a-ha was that machine learning platforms were able to learn and make good choices very quickly. And this was really good news for us because it means you don't have to use them as often, and they're a lot more powerful that we assumed. 

So some of the next steps. This was a really simple case study that we worked through. Just trying to make sure that we could deliver content that was appropriate to an individual user after sensing what they felt was appropriate. The next steps then became really obvious. 

Let's take all the telemetry from our entire player community and validate the difficulty seeds of our individual levels. And let's validate our entire game design. That progression curve. What type of behavior, what types of levels, are people actually responding to? Which ones are they repeatedly playing over and over, but continuing? So, that showed that there was enough difficulty that they were willing to retry a failure, and felt the game was interesting and continue. Or was it people wanted to play the-- through puzzles as fast as possible? 

We can use the same API and the same technology to validate game design decisions. And with the game where the content is dynamic and can be delivered at run time, you can actually tailor, [? de-design, ?] and tailor levels to an individual's experience. 

The next case study we did-- we did-- we were working on-- is kind of the opposite end of the spectrum in game design. Can you create a character-- a non-- an NPC in a game that, you could say, passes the Turing test? 

But in entertainment, the character has to not [? just pass ?] the Turing test. The character has to interact with you in a way that you find interesting. That you develop an emotional connection with this character. A character that you can hold a conversation with like you would any other person. And that that personality elicits responses and behaviors from the player, but also supports the mechanics in a game. 

So again, the QR code in the link here. The game-- the game is called The Suspect. And if you Google, The Suspect, you're going to go all different ways. It's www.thesuspect.com. And this game the premise is, somebody walks in to a police station with a  bunch of pictures and a bloody hat. And immediately gets put into an interrogation room and just like every episode of Law & Order, you're given the job of finding out what happened. And that's the premise. 

That is the extent of the set-up. You are then given a box, and you hold a conversation with this character. And this character is completely fictional, and completely interactive. And you're interacting with this character in free-form English. And it Is as wide open as Grand Theft Auto, or any other completely un-rule-based game. 

And there are game mechanics wrapped around the interaction, but the idea is, you're trying to extract a set of information from a personality that isn't necessarily cooperative. So, this-- the developer, again Guy Gandy-- was a lead developer for The Suspect. And a leading mind in the development of conversational based interfaces, or chatbot. He has a history of developing conversational based interfaces for primarily television properties. And has some great stories about if you're working with a television-- mass media property-- television or movie-- conversational interfaces can actually grow an audience between productions. 

But the idea here was they wanted to create a game around a simple personality. And using a trope and a mechanic that was very well known-- the police procedural. And they started with a single writer that wrote, effectively, a FAQ-- a frequently asked questions, you know-- if the character was asked this, what would be the response? The kind of thing that you would normally write for a standard decision tree based dialog interface. And they use that basic fact to do the preliminary training of a conversational AI. Commonly called the chatbot. But this conversational AI had more capabilities than the average chatbot. It had the ability to not only understand language as it was being talked to, but also understand free-form language that would inform the personality. 

So the second thing this developer did was brought on a prose writer that wrote the entire back story in free-form prose for this particular character. You know, where they grew up, how old their parents were, what their favorite color was, what their high school experience was. 

So that when you asked a question of this particular character, the answer might be made up of some elements of the fact, or the real piece of hard data that you are trying to extract. And some elements of this character's back story. You know, if they love the color blue, but any mention of rose reminded them of their first bad date. And if you ask them about a shirt, and you described the color as rose, then the personality would then shutdown and stop answering your questions. 

So they got a great back story. And they got the elemental questions for this experience. And then, we all have our own hierarchy of needs, and our own goals, and the way we respond to new input and repeated input. 

The third thing this developer did was actually create a mind map. A standard piece of mind mapping software that described this guy's hierarchy of needs. What his goals were. Why did he walk into this police station? How did he respond to repeated questions? This particular guy was quite snobbish. If you asked him the same question twice, he'd give you a good answer the first time, but if you ask the same question again, he would shut down and stop talking to you. If you rephrased the question-- and this is the power of natural language AI-- if you rephrase the question but it was the same question, you'd get the same result. 

So through this development process, and the focused development of the-- the personality was about four months. Elements of the game also include video, in a second screen experience. So there was a lot of development creating video content. And The secondary screen experience took them about a year and a half. 

As Guy describes it, it is, can we throw everything we can think of-- video, audio, history, mindmap-- as well as straightforward Q&A into an AI, and really give an engaging experience? And I recommend everybody try this out. In my humble opinion, they were tremendously successful. 

And we hoped-- I wish I could talk about it today, but we hope to have a very similar announcement of a conversational interface of a major title within the next month. But this is-- I've got some members from war-gaming. War-gaming was brought up early. 

Think about if you were playing World Worships, and you were piloting a Russian cruiser. And You come across a British battle cruiser. And you turn to your salty 20 year Russian navy vet who is an expert in Russian history, and ask him what you should do, and he rattles off this particular engagement that happened 30 years before, where a-- you know, a Russian hit a British battleship right in the rudder, where all the Russian-- where all British battleships are-- are weak. That's the type of conversational interface and the type of capability that we're looking at just now being able to provide from the games industry. 

So again, some interesting statistics on the screen about using conversational interfaces to support other medium. And talking about doing contract game development. A lot of major media properties are now trying to indie game developers to create these side experiences. Or these second-- second screen experiences for something that's weekly episodic, and has a four month hiatus. And we've seen her only  experiements-- actually from the telenovela industry. Where conversational interfaces on the web and mobile can actually not only retain audiences, but increase them. 

So, if you're an indie out there trying to think about what kind of work you could do and specialize in, and still stay within media and games, this is an area that is growing very rapidly and there's a tremendous amount of demand from other parts of the entertainment industry that would understand games and games development than, say, industrial sector of [INAUDIBLE] 

So I went through two case studies of machine  learning being used very differently in games. I've got five I know of in detail, and there are a lot more. I mentioned the AlphaGo project. Which is a tremendous step forward in just the fundamental capability of machine learning and games. 

Project Aries is another game where the AI actually will throw cyber attacks at you. And it's a game-- a serious game to train people how to respond to cyber attacks in corporate IT. Guy Gandy was-- just wrote a piece for the website, How Do We Get To Next, which is part of the Bill & Melinda Gates Foundations' outreach. And he wrote a great article on the philosophy and some of the underpinning design elements of creating conversational games. I highly recommend going and reading it. 

We did something interesting with a mods-- a Minecraft in an under-privileged high school. Using Watson to train kids to understand the disease so that they can represent it in a game. So what does Measles look like? And what kind of tools would you need to craft-- in Minecraft actually-- to defeat Measles? 

Some other-- Mari/o is a great video tutorial on the programming of neural networks. The author goes through using Mari/o to train a neural network from the ground up. And the neural-- all the neural network can see is what's on screen. And the neural network has something like eight buttons it can press. That tutorial takes you through and gives you a great understanding of genetic algorithms. And how programming these new systems is really different than sitting down and coding. And it shows really visually in a really simply, and actually quite fun way, how to use genetic algorithms to train an AI. In this case, to play a game, but you can think about how AIs can learn to be avatars. 

And then, completely different area this is a company called Artomatix is using AI to make landscapes better. So they started out being able to dynamically and systematically remove scenes and textures. So you know, their demos texturing and desert. With three different tiles. And generating just a  really horrible experience. And then running it through their machine learning platform, to detect edges, and smooth, and using just three simple tiles, create really beautiful landscape. 

So I highly recommend you check out Artomatix. It's the use of AI, and the use of a machine learning platform actually to accelerate the creation of content, and reduce the cost of creating that content. 

And I know I've been standing between you and beer, and I thank you very much for being a great audience.

Product and Performance Information


Performance varies by use, configuration and other factors. Learn more at www.Intel.com/PerformanceIndex.