Pranav Mistry demos several tools that help the physical world interact with the world of data – including a deep look at his SixthSense device and a new, paradigm-shifting paper “laptop.” In an onstage Q&A, Mistry says he’ll open-source the software behind SixthSense, to open its possibilities to all.

The SixthSense technology contain a pocket projector, a mirror and a camera contained in a pendant-like, wearable device. Both the projector the camera and sensors are connected to a mobile computing device in the user’s pocket.

The projector projects visual information enabling surfaces, walls and physical objects around us to be used as interfaces; while the camera recognizes and tracks users’ hand gestures and physical objects using computer-vision based techniques.

The software program processes the video stream data captured by the camera and tracks the locations of the colored markers (visual tracking fiducials) at the tips of the user’s fingers.