This is What Star Wars Looks Like Running on Magic Leap

By
Paul James
-
24

This is a quick demonstration video released by ILMxLab and Magic Leap to mark the announcement of the two company’s newly announced collaboration. They’ve set up a “collaboration lab” in order to work on immersive experiences using Lucasfilm IP and driven by Magic Leap’s augmented reality technology.

It shows Star Wars favourites R2D2 and C3PO as digitally rendered characters, overlaid into reality as filmed through Magic Leap’s visor. Note how the objects in real life occlude the digital creations, a tricky effect to pull off when you’re reliying on computer vision techniques to map to reality.

See also: Magic Leap and ILMxLab Announce “Collaboration Lab” Strategic Partnership

This article may contain affiliate links. If you click an affiliate link and buy a product we may receive a small commission which helps support the publication. See here for more information.


Paul James
Based in the UK, Paul has been immersed in interactive entertainment for the best part of 27 years and has followed advances in gaming with a passionate fervour. His obsession with graphical fidelity over the years has had him branded a ‘graphics whore’ (which he views as the highest compliment) more than once and he holds a particular candle for the dream of the ultimate immersive gaming experience. Having followed and been disappointed by the original VR explosion of the 90s, he then founded RiftVR.com to follow the new and exciting prospect of the rebirth of VR in products like the Oculus Rift. Paul joined forces with Ben to help build the new Road to VR in preparation for what he sees as VR’s coming of age over the next few years.
  • Shields

    Wow. If that clip truly is shot through the device and there was really no post processing, Magic Leap might actually deliver on its claims. This is going to be insanely cool.

    • Bryan Ischo

      Agreed. That looked insanely good. Also, I can’t imagine that ILMxLab would get involved unless they saw the tech and evaluated that it was good enough to devote effort to supporting, which is a very heartening data point.

    • WhatWouldYodaDo

      True, but I’m sure that this room was tested a bunch to ensure that the lighting and furniture worked perfectly for the demo. If you chose to run this demo in a different room, a random room, who knows how well it would perform. But if this is the goal, they clearly will get there!

    • Alx

      I doubt that’s being rendered on the headset, it’ll be using their technology alright, but there will be wires attached. I would make sense to have a inset window with an external perspective of the user otherwise. Looks great though.

  • Byoop

    Every Magic Leap video is filmed in a strangely darkened room. I presume it’s like Hololens FoV trickery all over again and it looks much much worse in daylight.

    • Constantine Chutis

      thats because the UV light from sunlight, or bright light interferes with the magic leap’s infrared camera. if you use it in too bright setting, it throws up an error, and asks you to lower the lights

    • FRIdSUN

      It’s not really “trickery”, but realistic technical limitation. As long as we cannot generate “black light”, we cannot display AR as well in daylight, similar to how backlight LCD fades out in daylight.

    • Tom_Craver

      I believe they use electrochromic diming of the background to increase contrast, which would create a consistent “dark room” effect. So it ought to work in a fairly bright area, as long as you don’t have any strong IR sources to mess up their depth sensing. (Note the lamp that is turned away from the headset, but clearly on. It’d be much brighter if not for the dimming.)

  • Andrew Jakobs

    That’s nice and all, but I still think it’s pretty useless for gaming, at least I think it is, as I don’t want to play a game using my bedroom/livingroom..

    • Bryan Ischo

      You sound like those posters who don’t want a Vive because they “don’t want to stand up while playing a game”. Have an open mind, man. You never know what kind of gaming scenario you might like if you actually try it.

    • yag

      AR will be great for playing outside with backtops. It will be like airsoft parties but “at force 10”.

    • James Friedman

      Fantastic point. A game is supposed to take you to another place in time, not to your shitty basement. Although this is super cool!

      • Bryan Ischo

        Well most people are going to have something nicer than “a shitty basement” to use AR in but … your post does make me think a little bit more about whether or not AR will be more compelling than VR. Consider that AR is just like VR except that the background can’t be any better than the room you’re in. At that point one does have to wonder, why bother with the background of the game being the room you’re in (AR) when it could be anything at all (VR)?

        That being said, I am 100% certain that fun and unique types of games will be present in AR that will make it worthwhile in its own way.

        Oh yeah another point: with AR you can have multiple people in the room interacting with each other and the experience, with much greater fidelity than VR is currently capable of.

    • Jaan

      how much worse is it than being confined to sit in one place staring at a flat box, a.k.a. your tv/screen, for hours in those same rooms you mentioned. freedom to move around could be very interesting

    • [obj-g]

      Check out the movie Her for an interesting use of similar tech in gaming. It doesn’t necessitate using your bedroom/living room as part of the game at all. Not any more than games you already play technically “include” your bedroom/living room outside of the screen…

  • If you watch closely in the very beginning as C3PO crosses the white wall in the background, you can just see it coming through. Same around 0:15 with R2D2 and the little end-table. If there’s light bleed in this dimly lit environment, then even typical indoor lighting will make everything translucent which is to be expected, but confirms this isn’t so much magic as you might hope without those clues. It also makes images like the following seem as if they may be VERY misleading at this point: http://www.roadtovr.com/wp-content/uploads/2015/10/magic-leap-whale-jump.jpg?utm_source=Road+to+VR+Daily+News+Roundup&utm_campaign=90c000e849-RtoVR_RSS_Daily_Newsletter&utm_medium=email&utm_term=0_e2e394ad33-90c000e849-165469661

    The focal plane is done well though. Notice at 0:19-0:20 where the model on the table is in focus and the background is not then at 0:30, the back corner is in focus and the droids are not.

    Also, you can see pixelation on the edges of images at times, but the pixel fill otherwise is very good. I’m also noticing some doubling in certain areas and wonder what that’s about.

    To me it all represents terrific progress in the right direction even if it isn’t “everything we ever hoped for”.

  • veritas

    So is Magic Leap using mini-projectors in the headset shining into our eyes/camera and called it Lightfield technology?

    • No one really knows how it works yet except Magic Leap. A top guess is some kind of projector system that draws the image across the retina, but there is one incredible feature it supports which causes any simple guess at it’s full function to be confounded: it creates images that the systems of the eye recognize as real objects which can be focused on at varying planes. This is what makes it so special, such a “Magic Leap”, as it were.

      • veritas

        The videos we have seen all were recorded through Magic Leap visor without any special effect, so what the video camera recorded and our eye perceived are supposed to be the same.

        • I’m sure it’s true, but I don’t know what it tells us about the technology.

    • Tom_Craver

      I don’t know, but I suspect there is more to it than that. Look at the above video very carefully around 0:16 – where R2 is partly obscured by the straight edge of the table. You’ll see that where the curved parts of R2 interact with that straight edge, the table looks oddly curved – in fact it is the R2 image rendering that has the odd curves at the occlusion edge.

      If it were just a depth calculation issue, such as an error in interpolating depth samples, I think the edge would be straight – the table is much closer than any of the R2 parts that are occluded. So I’m guessing that is somehow due to slight errors in their lightfield calculations…possibly related to the fact that R2’s surface in that area is a curved cylinder.

  • FRIdSUN

    Impressive work. IMHO though, the killer point of Magic Leap is a solid step towards solving VR sickness. Studies from LCD-based VR pilot training programs show that the dissonance between the intended distance of a VR object and the actual distance felt by our focus muscles can harm the trainers’ sense to a point that they are not allowed to drive in the next 72* hours, even if it doesn’t yield severe sickness such as vomit.

    *EDIT: 24