With leading high-end VR headsets debuting in 2016, we've had about two years to see what the first-generation VR experience looks like. While there's certainly a range of exciting games and other VR software coming soon to first-generation hardware, it feels like the market is steadily turning its attention toward next-generation technologies, and fostering a renewed sense of excitement and momentum. Here's a smattering of exciting projects that are likely to influence the VR and AR space in the next two years. Oculus Half Dome Prototype [caption id="attachment_77724" align="aligncenter" width="640"] Image courtesy Facebook[/caption] Prior to launch the first consumer Rift headset, Oculus had a long history of showing prototypes and development kits. Following the company's 2012 Kickstarter, enthusiasts and developers watched the subsequent development of the Rift DK1, DKHD, DK2, Crystal Cove, and Crescent Bay headsets before the consumer Rift finally made it to market in 2016. Following the launch of the Rift, Oculus kept their R&D efforts on future PC headsets mostly secretive. That is, until the reveal of the Half Dome prototype less than two months ago. [caption id="attachment_77726" align="aligncenter" width="640"] A Rift-like field of view compared to the Half Dome prototype. | Image courtesy Facebook[/caption] Half Dome introduces a much larger 140 degree field of view (to the Rift's ~100 degree) along with a varifocal display and eye-tracking. The larger field of view will make the virtual world feel much more encompassing and immersive. The varifocal display makes the virtual visuals look more realistic by dynamically changing focus to simulate light coming from objects of varying distances, also making it easier to focus on objects closer to you; in short, it makes the light coming out of the headset act more like light from the real world, which allows your eyes to function more closely to how they would in the real world. Then there's eye-tracking—the ability for the headset to know precisely which direction your eyes are looking—which can enable game-changing capabilities. Oculus has said clearly that we shouldn't expect to see everything from Half Dome in the Rift 2, or whichever PC-based headset is coming next from Oculus, but it gives us a clear direction that the company is looking for the future of its PC headsets. Valve Knuckles [caption id="attachment_79578" align="aligncenter" width="640"] Image courtesy @AntonHand[/caption] It was once thought that controller-less glove & finger tracking technology would be the ideal input for virtual reality, but over the last few years the benefits of having a physical, tracked controller for VR input have become quite apparent. For one, buttons and sticks are more reliable than gestures for binary input (like initiating a 'grab' action, or navigating a menu), and it turns out that having something in your hand while grabbing virtual objects actually feels much more natural than making a mock grabbing posture with nothing to grasp but air. But there's still benefits to the full finger tracking afforded by VR gloves, like added realism and a less abstracted means of fine interaction in VR (like poking and pinching). [irp posts="79583" name="Exclusive: Cloudhead Games Goes In-depth with Knuckles EV2 & Predecessors"] The Knuckles controller, in development by Valve, aims to mashup the capabilities of both VR controllers and VR gloves, by creating a controller which can sense the position of all five fingers. Last month Valve revealed the latest version of Knuckles, dubbed the EV2, which also introduced a force sensor into the controller's handle to detect how hard the user is gripping. Another major part of the controller's design is the strap which can keep it on the user's hand even when they completely release their grip. With finger tracking, force sensing, and the strap design, Knuckles opens up new interactive possibilities like being able to crush things in your hand and more naturally throw objects. Valve hasn't said when Knuckles will ship to consumers, but the latest development kit is looking much more polished than previous versions and could be soon on its way to mass production. Leap Motion North Star [caption id="attachment_76721" align="aligncenter" width="640"] Image courtesy Leap Motion[/caption] The world of AR is poised and waiting for its breakout headset. While HoloLens is impressive in many ways, it's expensive and held back by a limited field of view and sluggish & imprecise input. It will be a few years yet until the market sees an affordable and compact AR headset which feels immersive in both input and output. Until that happens though, it's challenging for developers to design the basic capabilities, interfaces, and interactions that will define AR experiences—just as its taken several years for VR developers to learn how to create compelling VR content. Not wanting to wait for the future to one day show up, Leap Motion has designed its own prototype AR headset which is made to give developers a development platform which represents the input and output experience that future AR headsets will hopefully one day provide. https://youtu.be/SEUa9qJNXMo Called Project North Star, the device eschews any care of form factor, and aims solely at maximizing the end experience with a wide 100 degree field of view, low latency, high resolution, and of course the company's hand-tracking sensor. Having open-sourced the design of the headset, Leap Motion hopes North Star will make it possible for the AR industry to get a head start designing apps and interfaces for a future where everyone is walking around with an immersive AR headset. Oculus Santa Cruz [caption id="attachment_69590" align="aligncenter" width="640"] image courtesy Oculus[/caption] While Oculus launched its low-cost 'Go' headset earlier this year, it offers a very similar experience to the Gear VR headset which has been available for several years now. But Oculus is also working on a more ambitious standalone VR headset called Santa Cruz. You can think of it like an advanced version of Oculus Go which is not only more powerful, but also offers 6DOF tracking, meaning that users can physically move around while using the headset, just like high-end PC VR headsets. And while Santa Cruz won't be the first standalone headset to offer 6DOF tracking, it's poised to be the first to also offer 6DOF motion controllers, giving users both full head and hand tracking, and making it possible to play much more immersive games on the headset. [irp posts="76899" name="Everything We Know About Oculus Santa Cruz (so far)"] Oculus has been working on Santa Cruz for quite some time now. The company first revealed the headset all the way back at the end of 2016. The last time we went hands on with the headset was when it was introduced with motion controllers in 2017. While the company hasn't yet announced a release date or price for the headset, we expect to hear a major update on Santa Cruz at Oculus' developer conference at the end of September. Continued on Page 2 » Ultra Dense Displays [caption id="attachment_78488" align="aligncenter" width="640"] LG & Google's 1,443 PPI VR display | Image courtesy Google & LG[/caption] The resolution of today's VR headsets might sound high on paper, but because the image is stretched across a wide field of view, resolution in today's headsets is still significantly lacking compared to what you'd expect from a typical computer monitor, and even further from what the human eye is actually capable of. In most first-generation headsets it's easy to see aliasing and the 'screen door effect'—a grid-like artifact that's caused by the unlit space between pixels. With enough resolution, VR headsets could eliminate both issues completely. [irp posts="74991" name="Understanding the Difference Between 'Screen Door Effect', 'Mura', & 'Aliasing'"] Display companies are racing to create ultra dense displays which are also inexpensive enough to go inside consumer VR headsets. In the last few months alone Samsung showed that it was developing a 3,840 × 2,160 VR display with 1,200 pixels per inch. Meanwhile, Google & LG revealed a 4,800 × 3,840 display with 1,443 pixels per inch. And then there's INT who is promising a display for VR headsets with a staggering 2,228 pixels per inch. Varjo Retina Resolution Headset [caption id="attachment_78796" align="aligncenter" width="640"] Photo by Road to VR[/caption] Ultra dense displays may not be here just yet, but Varjo has created a promising shortcut to achieving retina resolution in a VR headset by using a combination of macrodisplays and microdisplays to put high density resolution at the center of your view without giving up a wide field of view. While most VR headsets today use one display per eye, Varjo's 'Bionic Display' headset actually uses two. The first display offers a fairly standard resolution compared to today's traditional headsets, but the second display is a pixel-dense microdisplay offering 60 pixels per degree. The image from the second display is projected directly in the center of the first display, putting the pixels smack dab in the place where your eye sees the most detail. The result is pretty impressive: a headset which has an immersive field of view, but also retina resolution at its center where you can't see any pixels or screen door effect. Of course, the peripheral area is standard resolution so if you turn your eye you'll lose the benefits of the high res display, but that's a small tradeoff for the opportunity to see retina resolution in a VR headset today, instead of years down the road. The company is also investigating methods for moving the high res display using eye-tracking input so that you see the maximum quality no matter where you look. Varjo is targeting enterprise use-cases with its Bionic Display headset, but the company has largely proven a novel display design which could find its way into consumer headsets in the near future. DigiLens 150 Degree Waveguide [caption id="attachment_78906" align="aligncenter" width="640"] Image courtesy DigiLens[/caption] AR headsets today can be made fairly small, but generally have a small field of view to match. While 'bird bath' style optics (like those in Meta or Project North Star) offer a large field of view, they are necessarily quite bulky. It will take a different approach to achieve the ultimate vision of AR—a device offering an immersive field of view in a truly glasses-sized device that can be worn all day. DigiLens, a developer of transparent waveguide display technology, says it’s working toward a waveguide display which could bring a 150 degree field of view to AR headsets in a truly glasses-sized form factor. The company's approach involves 'printing' light manipulating structures into thin and transparent material wherein light can be guided along the optic and be made to project perpendicularly, forming an image in the user’s eye. With such precise manipulation of light, the optic effectively acts as both the lens and the display, opening the door to significantly more compact optical designs than what's found in most AR headsets today. [caption id="attachment_79211" align="aligncenter" width="640"] Image courtesy DigiLens[/caption] What's more, DigiLens says that the tech could be paired with a light blocker or even a liquid crystal blackout layer (to make the optic opaque), enabling a single compact headset to be capable of both immersive AR and VR. DigiLens has already developed a AR reference headset with a 50 degree field of view across both eyes. Using a technique combining multiple layers of the optic for a wider field of view, the company expects to be able to achieve a 150 degree field of view in 2019. Continued on Page 3 » Eye-tracking [caption id="attachment_78060" align="aligncenter" width="640"] Photo by Road to VR[/caption] Eye-tracking has been discussed often with regards to foveated rendering—the ability to render sharply at the center of your vision and less sharp in your peripheral vision, to save on processing power—but there's so much more than eye-tracking can do to enhance the VR experience. Beyond foveated rendering, a headset equipped with eye-tracking hardware could automatically detect which user is wearing a headset in order to instantly personalize the experience for that user, including the ability to make automatic adjustments like setting the headset's IPD. Eye-tracking is also a key component in varifocal displays which, using the position of your eyes, can dynamically adjust their focus to eliminate the vergence-accommodation conflict and accurately simulate other effects like depth of field. Eye-tracking could also be used for foveated displays, like Varjo's headset, which concentrates more of the display's pixels at the very center of your vision (where it's the sharpest). And there's still much, much more that a headset equipped with eye-tracking could accomplish. [irp posts="77981" name="Eye-tracking is a Game Changer for VR That Goes Far Beyond Foveated Rendering"] Companies like Tobii are now offering eye-tracking hardware and software to manufacturers and developers; Qualcomm is now offering Tobii’s solution in their VRDK headset. 7invensun is selling the aGlass eye-tracking development kit for Vive headsets. Fove is selling a development kit of their VR headset with inbuilt eye-tracking. Oculus recently showed off a new prototype seen for the first time with eye-tracking. Magic Leap has confirmed eye-tracking on their upcoming development headset. And even Apple is in the game, having reportedly acquired SMI, one of the former leaders in the eye-tracking space, and has drawn up patents implementing the technology. Pimax "8K" Headset [caption id="attachment_73557" align="aligncenter" width="640"] Pimax "8K" V5 prototype | Photo by Road to VR[/caption] While high-end consumer VR headsets available today offer a ~100 degree field of view, Pimax is hoping to take on incumbents with its "8K" VR headset which has a claimed field of view of 200 degrees and a 7,680 × 2,160 resolution. The headset has SteamVR Tracking built in and is planned to be compatible with SteamVR content. The company ran a very successful Kickstarter campaign for the headset, raising $4.2 million, and went on to raise $15 million in venture capital following the crowdfunding. We've watched as the Pimax has made steady progress over the years on its ambitious headset, and while there have been some delays, the company recently said that they're entering the headset's mass production phase, and are on track to ship to Kickstarter backers by the end of the year. If Pimax succeeds, the headset could set a new bar for immersion thanks to its ultra wide field of view and high resolution. Massless [caption id="attachment_76309" align="aligncenter" width="640"] Photo by Road to VR[/caption] While there's a handful of quality VR sculpting, painting, and design apps available today, all are restricted by their reliance on controllers which were meant for general input—like the Oculus and Vive controllers which have grips and triggers that are more suitable for replicating the feel of a gun than a pencil or a paintbrush. Massless wants to change that by developing a VR stylus for applications demanding greater accuracy. Humans have a long history with tools, and when it comes to precision, the stylus is tried and true, thanks to its ability to be carefully manipulated by our fingers instead of just our wrists. The Massless stylus works by using an external camera to track the tip of the device, allowing you to draw in the air (or even against real surfaces) in a very natural way. Massless hasn't announce firm plans for release, but we had a chance to try the latest prototype earlier this year. - - — - - Next-gen AR and VR is steadily coming, but this certainly isn't an exhaustive list. What are you looking forward to?