Steve Jobs once said to a gathering of senior advisers that when it came to the first iPhone, “we’re going to patent it all,” (New York Times reports). While Apple has certainly changed in the years since Job’s passing, the company has undoubtedly continued to aggressively pursue patents, sometimes racking up hundreds of filings in a single month with inventions spanning everything from touch-sensitive smartwatch bands to 3D environmental mapping. This week however, the company was granted two patents that establish basic hardware and software solutions not only pointing towards a prospective Apple AR device, but marking out some fundamental territory in the process.
Both patents, spotted by Apple Insider, were first filed by Metaio GmbH in 2014, a German AR firm later acquired by Apple just a year later.
Patent number 9,560,273, entitled Wearable information system having at least one camera, details a wearable device with a “display attached to his head in front of his eyes” that can use either an onboard SoC or in a wirelessly connected smartphone. The patent goes on to detail the invention’s reactive power mode, letting it switch between a battery-saving low-power mode and an active high-power mode that automatically reacts when a real-world object is matched with a reference object.
In the figure below, the patent shows off a Google Glass-style configuration that is “especially well suited to be used with head mounted displays and a camera pointed at the space in front of the user.”
Patent 9,558,581, entitled Method for representing virtual information in a real environment, delves into the software side of AR, and is centered around virtual info overlays and proper occlusion perception.
The setup proposed in the patent paves way for a device that creates a 3D geometry model of the real environment and shows virtual information in both an occlusion mode and a non-occlusion mode, meaning objects are correctly presented to the user even if they’re hiding behind real-world obstacles. And to that, patent ‘581’ focuses on different ways of making virtual information distinct from the real world so users can quickly differentiate the real environment from the superimposed images. The patent proposes a few ways of doing this, including showing real-world objects as semi-transparent when they’re occluded by virtual info (great for driving) and using dashed lines to indicate a separation from the real and virtual.
While this all seems fairly basic, these sorts of patents help lay an important defensive framework that can help against future disputes from other manufacturers working in the sector; eg. Apple v. Samsung, Apple v. Microsoft, Apple v. Nokia, the list goes on. And if the recent ZeniMax v. Facebook lawsuit has taught us anything, you don’t ever want to be the one writing that half-billion dollar check after you already shelled out $2 billion for the intellectual property in the first place.
So while not giant revelations on their own, this, alongside Apple’s ever-growing library of AR patents, certainly lends credence to the idea that Apple is investing serious time and money in the field, and it doesn’t look to be ‘just in case’.
Apple’s AR Investments
Just one year after the launch of the iPhone, Apple started patenting head mounted displays with features not unlike the VR headsets of today. Of course, at the time it wasn’t feasible to produce such a unit, but again, patents aren’t really for that purpose anyway.
Apple’s recent acquisitions have also pointed towards a heightened interest in AR. Besides acquiring Metaio, Apple has also taken on marker-less facial mapping and animation firm Faceshift, facial expression recognition & analysis company Emotient, and Flyby Media, a large-scale SLAM, indoor navigation, sensor fusion, image recognition, and 3D tracking company.
Another nail in the coffin: around 2014, Apple started hiring specifically for AR/VR software experts at their Cupertino HQ, first for an ‘App Engineer’ who was tasked to “create high performance apps that integrate with Virtual Reality systems for prototyping and user testing,” and then a month later four additional hires with immersive tech talent, including a ‘VR/AR Programmer’.
As we know it, Apple is a perennial ‘johnny-come-lately’ to established tech, preferring to see markets mature before they enter in with a big bet like the iPhone or Apple Watch, and the company seems famously stand-offish when it comes to VR. According to a report by Vanity Fair, Apple CEO Tim Cook says that virtual reality isn’t actually the way to go in his opinion. “Virtual reality sort of encloses and immerses the person into an experience that can be really cool but probably has a lower commercial interest over time,” said Cook.
So if VR isn’t in Apple’s commercial interest currently, the evidence thus far suggests that the company is heading eye balls-first into augmented reality. And we can’t wait to see what the world renowned electronics giant has in store.