The Microsoft Kinect was released in the US on November 4, 2010. It was hacked on November 10, 2010.
For the one or two people on the internet who have not heard about the Xbox Kinect, it is a interactive gaming device for the Microsoft Xbox gaming system. The heart of the Kinect is 3D sensor technology from Primesense. Primesense 3D sensing technology is illustrated above. The technology uses structured light, a infra red camera, and a dedicated processors to measure the distance from the camera to the scene. These measurements are made throughout the view of the Kinect. The result is a point cloud consisting of 307200 distance measurements between the Kinect and the scene.
Structured light is simply light projected in a unique pattern. The Kinect uses a Infra Red (IR) laser and a special lens (shown below on the right) to project a unique dot pattern (shown above). The dots are projected at a precise angle from the laser. The laser dots reflect off objects in the scene and are visible to the IR camera (shown in the center of the image below). Objects in the scene close to the camera reflect dot images that are close together, object far from the camera reflect dot images that are farther apart.
The data from a 3D camera is called a point cloud.
A point cloud is a data structure used to represent a collection of multi-dimensional points and is commonly used to represent three-dimensional data. In a 3D point cloud, the points usually represent the X, Y, and Z geometric coordinates of an underlying sampled surface. When color information is present (see the figure below), the point cloud becomes 4D.
Point clouds can be used to create range images by assigning a color to the Z axis. In the image below white is close, blue is far. The Kinect hardware has a practical range of about 19" to 11 feet. The point cloud has a X,Y resolution of 640x480, and a range depth of 11 bits (0 to 2047). In practice (using the libfreenect driver) the range is 350 to 1045 representing the 19" to 11 feet. The range data is not linear, and anything too close or too far away or in a "shadow" returns a value of 2047 (shown in black below).
The day the Kinect was released (November 4, 2011) Adafruit announced a "Hack the Kinect" contest with a $3000.00 prize. just six days later, Hector Martin made history by being the first to post his hack and source code, and won the prize. Hector created the libfreenect open source project. The libfreenect open source is now part of the Openkinect project.
OpenKinect is an open community of people interested in making use of the amazing Xbox Kinect hardware with our PCs and other devices. We are working on free, open source libraries that will enable the Kinect to be used with Windows, Linux, and Mac.
The OpenKinect community consists of over 2000 members contributing their time and code to the Project. Our members have joined this Project with the mission of creating the best possible suite of applications for the Kinect. OpenKinect is a true "open source" community!
Our primary focus is currently the libfreenect software. Code contributed to OpenKinect where possible is made available under an Apache20 or optional GPL2 license.
The Libfreenect software can be downloaded here: https://github.com/OpenKinect/libfreenect.
It has been ported to Windows, Linux, OSX, and embedded linux. It supports: C++, Python, C# and Java. The library is very easy to use and build. At one time Willow Garage had created a branch of the Libfreenect library (https://github.com/ros-pkg-git/kinect). At that time, the ROS branch had more support than the base Libfreenect library. It does appear that the ROS branch has been merged into the Libfreenect master archive.
Microsoft Kinect SDK
Microsoft originally pushed back on the concept of using the Kinect on a PC, and even threatened lawsuits. As it became obvious that the cat was out of the bag and the open source community was doing incredible things with the Kinect (on Linux) they decided to support it with a SDK.
On February 21, 2011 Microsoft announced that it would release a non-commercial Kinect software development kit (SDK) for Windows in spring 2011, which was released on June 16, 2011; a commercial version is planned for a later release date.
The SDK includes Windows 7 compatible PC drivers for Kinect device. It provides Kinect capabilities to developers to build applications with C++, C#, or Visual Basic by using Microsoft Visual Studio 2010 and includes following features:
- Raw sensor streams: Access to low-level streams from the depth sensor, color camera sensor, and four-element microphone array.
- Skeletal tracking: The capability to track the skeleton image of one or two people moving within the Kinect field of view for gesture-driven applications.
- Advanced audio capabilities: Audio processing capabilities include sophisticated acoustic noise suppression and echo cancellation, beam formation to identify the current sound source, and integration with the Windows speech recognitionAPI.
- Sample code and Documentation.
The Microsoft Kinect SDK (http://kinectforwindows.org/) includes support for skeletal tracking,
which is an application that uses the point cloud data from the Kinect to construct a
mathmatical model of the human body. The model is updated in real-time using the point cloud
data from the Kinect. This software technology makes gesture recognition possible, along
with whole body tracking.
Primesense Official Driver, OpenNI and NITE
Primesense is the creator of the Kinect. Primesense realized early on that the Kinect had more potential than just gaming. The OpenNI consortium is all about bringing the Kinect technology to other devices outside of gaming.
TEL AVIV – December 21, 2010 – PrimeSense (http://www.primesense.com), the industry leader in sensing and recognition solutions, announces the launch of its OpenNI (Open Natural Interaction) consortium, an industry-led, nonprofit organization formed to accelerate the introduction of Natural Interaction applications into the marketplace (http://www.openni.org).
PrimeSense also contributed its first proposal for an OpenNI standard as well as implementation of the OpenNI framework under open source license, royalty free license to PrimeSense NITE middleware modules all designed to work with OpenNI compliant devices such as PrimeSense’s development kits which are going to be available for purchase through PrimeSense’s web site.
As part of the OpeNI initiative, Primesense released an official open source driver for the Kinect (https://github.com/PrimeSense/Sensor#readme) - sort-of. As it turned out the official Primesense driver was for the Primesense referense platform (a little brother to the Kinect). Fortunately, the hacker community came to the rescue and released a branch of the official driver that supports the Kinect.
The Avin2 official/un-official Kinect driver (https://github.com/avin2/SensorKinect) is the most up to date open source driver for the Kinect. The Avin2 branch lags behind the Primesense root by about six months. Both the Primesense and Avin2 drivers are designed for openNI. The drivers use a OpenNI front end interface. You can find examples of how to use the OpenNI API in the OpenCV project.
PRIMESENSE'S NITE MIDDLEWARE IS THE BRAIN OF OUR SOLUTION. IT'S WHAT ALLOWS COMPUTERS OR DIGITAL DEVICES TO PERCEIVE THE WORLD IN 3D, JUST AS WE DO. MORE THAN THAT, NITE COMPREHENDS YOUR MOVEMENTS AND INTERACTIONS WITHIN ITS VIEW, TRANSLATES THEM INTO APPLICATION INPUTS, AND RESPONDS TO THEM, ALL WITHOUT THE NEED FOR ANY WEARABLE EQUIPMENT.
Including computer vision algorithms, NITE identifies users and tracks their movements, and provides the framework API for implementing Natural-Interaction UI controls based on gestures. http://www.primesense.com/technology/nite3
NITE is NOT open source. It is a licensed piece of software that Primesense allows the community to use under a free license. The NITE license is tied to the use of the Primesense chip. Using NITE with any other 3D sensor breaks the licensing agreement.
NITE is released as OpenNI middle ware, and be downloaded from the OpenNI website: http://www.openni.org/Downloads/OpenNIModules.aspx.
The Microsoft Xbox Kinect has created a whole new wave of interest in computer vision.
Thanx to the DIY/Hacker community (Adafruit/Freenect), the true potential of the Kinect was realized early and the corporations were eager to get in on the action. Primesense (the creator of the Kinect) was first to tap into the DIY/Hacker community and created the OpenNI consortium, while also releasing official drivers for hardware nobody had (Primesense reference platform). Again, the hacker/DIY community came to the rescue and made the official Primesense drivers useful by porting them to the Kinect.
Microsoft, being Microsoft, first threatened lawsuits against anyone caught hacking the kinect. Then they realized all the cool Kinect demos were being demonstrated on Linux and decided they better release something. The Microsoft SDK only runs on Windows 7 and requires Visual Studio 2010. If you can live with that, it does provide solid performance and a lot of great features. It is also, by all accounts, easy to use.
The kinect has excited the public about the potential of computer vision. The idea that a computer can interact with a user by strictly "looking" at them has re-activated the imagination of the public, and they are prime to adapt to the new technology. Similar to how the Wii introduced the public to the 3-axis accelerometer which then found its way into everything including our cell phones. The Kinect has the potential to introduce computer vision to a public waiting for the next great thing.