Here’s the official description of the system from the website of Brett Jones, one of the principal researchers:
The television is the focal point of living room entertainment, and while our TVs have gotten bigger and brighter over the years, our content is still trapped inside a little box in our living room. We introduce IllumiRoom, a proof-of-concept system that augments the area surrounding a television with projected visualizations to enhance traditional viewing experiences. IllumiRoom can directly extend the viewing experience, turning a 40 inch television into a 15 foot television. IllumiRoom can enable augmented reality experiences where virtual objects interact with the physical environment (e.g. furniture). Finally, IllumiRoom can augment and distort the physical environment (e.g making a living room look like a cartoon).
The Illumiroom system can do some pretty cool stuff. The new video details the system further than we saw previously and shows a selection of interesting effects.
The reality-warping ‘radial wobble’ is probably my favorite effect from the limited set shown off in the video, though I have a feeling that there’s even more inventive styles waiting to be created.
For the technically curious out there, Microsoft Research has also published a detailed paper on Illumiroom. Here’s the abstract:
IllumiRoom is a proof-of-concept system that augments the area surrounding a television with projected visualizations to enhance traditional gaming experiences. We investigate how projected visualizations in the periphery can negate, include, or augment the existing physical environment and complement the content displayed on the television screen. Peripheral projected illusions can change the appearance of the room, induce apparent motion, extend the field of view, and enable entirely new physical gaming experiences. Our system is entirely self-calibrating and is designed to work in any room. We present a detailed exploration of the design space of peripheral projected illusions and we demonstrate ways to trigger and drive such illusions from gaming content. We also contribute specific feedback from two groups of target users (10 gamers and 15 game designers); providing insights for enhancing game experiences through peripheral projected illusions.
To give credit where it is due, the paper attributes the following authors: Brett R. Jones, Hrvoje Benko, Eyal Ofek, and Andrew D. Wilson
I was particularly interested in the calibration aspect of the system. The researchers smartly combine the capabilities of the projector and Kinect sensor in a way that allows accurate scanning of the geometry and colors of the scene. I took a look into the paper to learn what I could about the calibration sequence and found something interesting in that the Kinect doesn’t need to be always behind the TV as the projector does, making the system more viable for mainstream use than I previously thought:
The Kinect sensor captures the color and geometry of the scene, and the system renders the illusions using the acquired depth map. Careful calibration of the system is required in order for the illusions to tightly match the onscreen content and the physical environment. The calibration of the IllumiRoom system is fully automatic, determining the relative pose of the projector to the depth sensor and the position of the television in the projector image. Therefore, setup only requires that the projector and depth camera are placed such that they cover the area surrounding the TV. The automatic calibration projects Gray code sequences to establish dense correspondences between the projector and the Kinect’s color camera. Using the default parameters for the Kinect, we transform these correspondences into 3D points. We then solve for the intrinsic and extrinsic parameters of the projector. As there is no depth data for specular surfaces such as that of shiny, black televisions, we recover the position of the TV using a 2D homography, with virtual markers displayed in each corner of the screen. It is important to note that the calibration need only happen once. Thus the Kinect camera could be used to calibrate the projector and then returned to the TV stand for gaming.
If you want to learn more, have a read on projection mapping.
Definitely an interesting system, though the fact that Microsoft is publishing all of this info leads us to believe that we probably won’t see it on the Xbox One any time soon.