Learn Tips and Tricks from Intel® Xe Graphics Innovator Christian Mills in This Step-by-Step Machine Learning Tutorial Series
Intro
“If you want to master something, teach it,” said the Nobel Prize-winning theoretical physicist Richard Feynman, whose Caltech lectures remain the gold standard for demonstrating how to clearly explain complex ideas.
That same principle of teaching-to-master-the-subject applies to the work of Intel® Xe Graphics Innovator Christian Mills who had been studying deep learning over the past several years. While he had no previous exposure to game development, Christian recognized the opportunity to apply the Unity* Barracuda inference library to the world of gaming.
“I instantly recognized the library's documentation as a style transfer model, which was one of the first applications I was drawn to when I began to study deep learning,” said Mills.
Christian Mills is an active contributor to the Intel® Software Innovator Program, Intel’s community for forward-thinking developers. Participants share thought leadership and technology expertise to inspire their developer peers by speaking and giving demos of their work at industry events. Now expanding to support graphics developers and creators, the Intel® Software Innovator Program offers an Xe Community track with an emphasis on applications and experiences demanding greater speed and performance even as workloads increase.
Video: Christian Mills Demonstrates In-Game Transfer Style
Working with in-game style transfer methods intrigued Mills so much that he decided to teach himself how to work with the technology. He acknowledged that his first attempts to use the applications were underwhelming, but Mills persevered and gradually he developed a better understanding of the technology behind Unity* and the Barracuda library. And to ensure that he truly understood them, Mills began producing a series of tutorials.
Mills’ tutorial series walks users through the steps for using a single image asset to train a style transfer model and implement it into the Unity* game environment.
Tutorial links:
In-Game Style Transfer Tutorial Leveraging Unity Part 1
Part 1.5 Generating Custom Training Data for In-Game Style Transfer
Part 2 Using Google Collab to Train Style Transfer Model for Unity
Part 3 Implement the Trained Style Transfer Model in Unity
“While I hope the tutorials that I create will help make working with real-time machine learning easier for people new to deep learning or real-time platforms,” said Mills, “I also hope that the process of producing them will make me a more competent developer and practitioner of the technology.”
Process is the Principle
In describing his process Mills says that when he starts a new project, he prioritizes quickly building a basic prototype. He then goes back and iteratively improves the prototype and how it works until it satisfies his goals. As a last step, Mills will often recreate the project from scratch. This ensures that he correctly understands how the technology and project works and provides him with an opportunity to improve it with a fresh eye.
This method, as a developer and teacher, corresponds to Richard Feynman’s technique for quickly and effectively learning any topic or concept. The steps are summarized as follows:
- Identify the topic
- Teach it to a child (or anyone unfamiliar with the material)
- Review your explanation
- Simplify and refine
Video: Christian Mills demonstrates creating a plugin that leverages the OpenVINO™ Toolkit for the Unity game engine.
Bringing it All Together
Having domain expertise is important to being able to solve challenges with deep learning. But according to Mills there currently appears to be a lack of game developers with deep learning engineering skills as well as deep learning engineers with game developing skills. Therefore, as more developers, like Mills, create more tutorials on how to apply machine learning to game development, there will likely be more advances in applying the technology that will benefit the gaming community.
Mills began his series of tutorials with applications to handle pose estimation and basic style transfer because they seem both fairly straightforward and are fun to work with. Moreover, the field of deep learning is constantly expanding and models like GANcraft, OpenAI's GPT-3, and DALL·E demonstrate the exciting possibilities of how the technology is advancing. “Developers who gain an understanding of pose estimation and basic style transfer will have the tools to make real progress in the field,” said Mills.
The Link Between Technology and Performance and the Intel® Software Innovator Program and Support
Deep learning and style transfer applications require high-performance computing. To run them, Mills works with the Intel® Distribution of OpenVINO™ toolkit and the Intel® Iris® Xe and Intel® Iris Xe MAX Graphics—Dedicated GPU for PCs.
“The technology allows me to decide whether the models should run on the CPU, GPU, or on integrated graphics while the application is running,” says Mills. Such optimization makes it possible to run more demanding models on the same hardware. In the future. Mills plans to explore using the Intel® Xe Matrix Extensions (Intel® XMX) engine in the upcoming Intel® Arc™ graphics solutions to free up the primary GPU resources.
A benefit that Mills mentioned is how the optimizations from the Intel Distribution of OpenVINO™ toolkit and other newer hardware make it possible to run models on less expensive computers. “This type of technology will make AI-enabled features available to a wider audience,” said Mills. Mills also believes that the potential for AI-based tools to unlock creativity can reduce barriers, empowering more people to take what is in their heads and turn it into something that other people can experience.
In addition to new technology, Mills finds being a member of the Intel® Software Innovator program community equally inspiring. “Getting together with people who are equally passionate about our work is just as inspiring, if not more so,” said Mills.
Going Forward
In the future, we will see more unique in-game dialogue and story paths based on user input, according to Mills. This user input could take the form of reinforcement learning models, which could generate custom game levels that adjust to a player's current skill level. Models like OpenAI's Codex could also make the field of game development more accessible to beginners by letting them design a game using natural language.
“There are too many possibilities to list here and I’m excited to see what new applications developers come up with as they begin to explore AI-enabled features for their gaming projects,” said Mills.
Mills is just wrapping up a tutorial that explains how to perform object detection in Unity* with the OpenVINO™ Toolkit. He also plans to make tutorials on how to implement other types of deep learning models into Unity*, and his interest in adapting Unity* projects to work with Unreal Engine* will likely find its way into another one of his tutorials.
Check out more of Mills’ work on his website or DevMesh.
Elevate Your Graphics Skills
Ignite inspiration, share achievements, network with peers and get access to experts by becoming an Intel® Software Innovator. Game devs, media developers and other creators can apply to the new Xe Community track – opening a world of opportunity to craft amazing applications, games and experiences running on Xe architecture.
To start connecting and sharing your projects, join the global developer community on Intel® DevMesh.