How the X-Box Kinect tracks your moves

by Bruce Kasanoff on November 20, 2010

NPR’s Ira Flatow spoke with two Microsoft developer/managers and an NYU professor in a fascinating discussion about Kinect’s functionality and potential. In the space below, I’ve combined several observations made by Alex Kipman, Director of Incubation, Xbox at Microsoft…

KIPMAN: In our world, it’s about stepping in front of the sensor. The sensor not only tracks your full body, head to toe, but it also knows who you are. It knows the difference between you, your family and your loved ones as well as it understands your human speech and understands how we’d like to say that if you can see it, you can say it.

For the human body tracking, we had to re-think how to approach development for something of this nature. There is a reason why this level of technology or this level of innovation hasn’t really existed before. And that reason is, because if you think about it, traditional heuristic-based programming is a world of digital land. It’s zeros and ones. It’s yes and nos. It’s trues and falses.

The real world, the world of humans, the world we live in is an analog world. It’s a world where it’s not about yes and no, it’s about maybe. It’s not about black and white. It’s about gray. If you think about it, to understand and track humans – let’s just think about joints on the human body. You have several different joints. Now multiply that number by the degrees of freedom that each one of those joints has. Multiply that by the different proportions of the human body, from the kid to the adult to the slim person to the not so slim person…

So we had to transition from this old world to the new world. The new world is a world of machine learning. It’s a world where you are not writing in the sensor what it sees. You’re teaching it how it can perceive the world. Kinect has set of eyes and a set of ears. The set of eyes allow us to see the room, understand both visually and acoustically what’s going on and that goes to the 360, to the Xbox 360 where there is the equivalent of the Kinect brain.

Now, for just the human tracking part, let’s focus on that, it’s a series of sophisticated algorithms that will range in nature from computer vision to machine learning, to imaging science, to a series of other ones. The key innovation is in the machine learning. And the way that you can think about that working, it works similarly to the human brain.

If you think about, you know, when a baby’s born and you show this baby a lion and a person and you ask the baby, if it could speak, can you tell them apart? The baby would not be able to do it. Fast forward in time and ask that same baby that same question, you would have instantaneously the ability to discern the difference between a lion and a person. Why? Because it has historical data. It has learned. It has burnt the pathways in his or her brain about being able to discern that. Now, show that same baby a male and a female. It won’t be able to do it. Fast forward in time, you’ll have no problems…

So if you think about what Kinect brings to the table, it really brings a new palette, a new set of paint colors and paintbrushes around being to identify who you are and understand what your profile looks like, being able to track your full movement, your head-to-toe movement, and use your voice.

That palette, those paint colors and paintbrushes, get used by our creative game designers, both within Microsoft Game Studios as well as across the entire ecosystem. And in a way, the thing that gets the – everybody that I’ve spoken to really excited about this new palette is that it allows them to tell brand-new stories. At the end of the day, everybody here is a storyteller. And what Kinect allows you to do is tell brand-new stories that haven’t been told before.

So if you think about the fusion of being able to fundamentally understand humans and how you couple that with voice and identity, knowing who you are, what that allows you to do is create extremely personalized experiences that become significantly more emotional and immersive…

This is a shift, monumental shift, where we move the entire computer industry from this old world, where we have to understand technology, into this new world, where technology disappears and it starts more fundamentally understanding us. Now, that world starts with Xbox, with Kinect in the living room across gaming and entertainment, but it’s something that over time, that journey is something that’s a lot more pervasive than just that…

We will, sooner rather than later – and we’re already doing a lot of this – start continue to partner with academic places to make sure that this innovation does make it into academic circles, right? So we started this already with places like USC and other universities some time ago.

And now that the product has volume, we will start increasing that academic program, which we have through Microsoft Research, where at the end of the day, we’re excited about this technology. This technology really allows us to do new things, and we wanted this palette to be available to academics so that they can use the palette to create brand-new pictures we haven’t seen before.

{ 0 comments… add one now }

Leave a Comment

Previous post:

Next post: