CREAL Raises $7.2 Million to Bring its Light-field Display to AR Glasses

Great to see that the problem is known and solution needed. Two years are realistic for high-end enterprise light-field. We have already first dev. kits. Consumer light-field is more like 4 years from now.

Hi vejmiv738, indeed it is good that you are giving me some navigation on what needs more reasoning when it comes to light-field efficiency. I will try my best to overlook the speculations about dishonesty etc., it really is insulting, but I understand that you see it from a standpoint where some claims are considered fundamentally impossible.

Of course, there are drawbacks of one approach vs another, like always, but this time elsewhere than when you see it here. The fundamentally achievable competitive efficiency is actually one of the most positive features of the near-eye light-field potential.

Few clarifications:
(i) Light-field is typically associated with multiple times or even orders of magnitude more rendering effort and image data. I don’t claim light-field does not need or carry more, but that it can be in most cases very little more, such as a few percent more.

(ii) Most of the talks and people speak about light-field in general terms. Yes, light-field really cannot be comparably efficient generally. The difference for specific cases (such as light-field panels) is gigantic. I speak about near-eye light-field which is limited to an eye pupil and projected in a specific way.

(iii) Other times the discussions are led in the context of some specific HW often assuming you can use only off-the-shelf processing tools. Today’s graphics processing and display drivers do not allow efficient near-eye light-field projection. A lot of work is needed to smartly abuse what is available (and still not have full efficiency) or design the whole pipeline on your own (to reach maximum), including custom HW processing. This partly refers to your statement “Your typical realtime 3d content has a depth buffer. Depth buffer contains 2d depth information”. This is one of the places which need a custom treatment, but not considerably less efficient treatment.

Let’s have a look at a simple example: two small points in space, one closer, one farther. Each of them has a certain color. Always in focus flat image of them on a flat screen would shine two e.g. 24bit pixels to all directions. Their 3D processing (yes I assume the same usecase, e.g. VR/AR) would include their depth unless one obscures the other, more below. To display the two points as a light-field, a light-field projector needs to distribute the color information of each point into a finite number of directions – rays – and give it a depth bias that is realized by multiple real but low color res “display” pixels (defining intersection of rays with a fixed plane in space). These directions can be predetermined (fixed) by HW and set to always enter the eye pupil. The key trick is, that each of these “rays” needs to carry only a fraction of the color information of the virtual point, because what matters is how the point looks when all components are assembled, i.e. when you focus on it. When you focus at a different distance, the rays of one virtual point hit retina at slightly different places, the color information is split and creates a blurred image of the virtual point, but still using the same data. Your eye does the blurring job. For this simple example, you needed the two colors and the distance. The process of shining them through multiple fixed directions does not require more information no matter how many directions your “display” allows to generate (as long as it can provide the right proportion of the total color per “ray”). It uses still the same information only expanded. So when you speak about a higher frame rate, then yes, you can use more frames, but you don’t need more image information, you just expand what you already have to more directions. Once you consider the two points are processed as 3D data, you already have all. To generate the color distribution into fixed rays is simple trigonometry. Light-field projection of more complex scenes follows the same principle. Each virtual point that is visible from the pupil (i.e. not the fully obscured ones) is distributed into a fixed set of rays. The projection process is the same for any scene and defines the light-field quality.
Now to understand the light-field rendering you need to invert the process. The projection needs more rays per point, yes, but since all rays enter the eye simultaneously, it practically doesn’t matter how you distribute the color into them. Hence, you can choose. And since you can choose and the ray directions are defined, you only need the color of the point and its distance.

Here I will only repeat. A typical 3D scene includes objects which obscure each other (even more problems come with transparent objects or smoke etc). The eye pupil can see slightly around closer objects, but this is in a vast majority of cases very little, few percent. The images seen from two extremities of your pupil (e.g. left and right side), are usually almost identical. There are multiple ways of rendering the part “behind” an object, but they all exploit the concept described above. Just think in the reverse order from the projection side.

To a priory answer some possible questions.
Yes, the full efficiency requires that all useful light enters the eye. You either have to secure it dynamically (which may require eye-tracking, but not for the light-field effect itself, only for the efficiency, and it can afford quite big errors) or you have to have some redundancy (exit pupil larger than eye-pupil) as you will lose some “rays” then.
You may say that even slightly different viewpoints should receive different colors from the same point in space. Yes, in reality, but since eye receives them all and mixes them, the information is lost, which means you did not need it from start, you need only the combined color.
We may have some different definitions. What matters in practice is a light-field for a human eye.
There is certainly needed a custom electro-optical system or truly abused existing, but it does not change the efficiency.
Yes, eye-tracking+varifocal is an engineering solution (100% dependent on precise eye-tracking), or even fast varifocal sweep is solution. But there are different pros and cons. Computing efficiency is not the main differentiator here other pros are taken into account. It is rather the robustness or how ultimate solution it is vs readiness level.

I won’t go into more details, there is a lot of tricks that can be done or different ways of looking at it. I only hope that the above describes the fundamental logic. I am afraid I cannot do more.

Hi, I replied already twice here and both comments are “detected as spam” in my disqus account, waiting “to get this corrected” on my request. Strange. I hope you have received it at least by email.

Not to write so long possibly for nothing again, then shortly… I see the core of misunderstanding in the assumption: “a lot of them (frames) = more information”, which is indeed not true in the light-field case we talk about. Because “a lot of frames ≠ more information”, meaning not new information which you would have to render. Since color of each virtual pixel is distributed to multiple low color resolution rays “randomly” and with fixed process, because all (or most) rays enter your pupil and combine at retina, you can expand the image information for free even in the modulator backplane if you design such. For instance, you can double the number of frames/rays at no rendering cost. Ultimately only the pixel color and its coordinates are needed. This should be enough to re-think all the points you introduce. There partial obscuring of objects etc is not trivial, but in this context a technicality.

Hi vejiminv738, thank you for refining the topic. I think we could get to the core of our misunderstanding this way. I hope I answered the projection part and that physical light-field rays can be generated from simple data once we assume that each virtual pixel distributes its color evenly or randomly to a small cone of direction (the cone defined by the pixel and the eyebox).
Rendering is somehow an inverse process of it. We do not have to calculate a new color for a virtual pixel from a new perspective (except those few which were obscured by closer objects) we can assume the same color. We can also calculate only partial information from each perspective (imagine something like short raytracing bursts). There are actually quite many options on how to do it. All of them, however, require entirely custom SW and HW treatment in the whole pipeline. You are certainly right that standard SW and HW tools cannot make this efficiently at all. Practically all must be designed and made custom, that’s why it is a tough job.
In your comment, this is summarized here: “because each perspective in a 3d program has to be rendered separately”. In this case, there is no need to render all perspectives which you plan to project. But even if you do it, you can avoid recalculating the information which you already have. Hence, no need to render “separately”, either. The parallax difference within the pupil (or reasonably big eyebox) is so small that virtual pixels do not change color. Even if we consider that the color changes, many enter the eye simultaneously and make the difference negligible by that.
The misunderstanding could be, therefore, in our definitions. Theoretical light-field should contain unique and kind of full color information in each ray of each pixel. That’s really fundamentally a lot. Practical engineering near-eye light-field display does not need it, because an eye cannot see that.
(Minor points: distribution of image data to dither-like-looking components can be lossless, we care only about their sum. Yes, the rays of a de-focused virtual pixel hit the retina at different places, but enough close and enough messed with other de-focused pixels, that it appears as a very natural blur. This is not an assumption, it already works well … and real-time. The moving eye pupil can be satisfied by a redundant eye-box with redundant image data, or, better, by a primitive pupil tracking – not for the depth cue itself, but for the efficiency).

PS: I have my previous replies, the problem is that it two times disappeared from here without notification or any provided reason. The one you see is the first one which passed and I couldn’t really expect it. However, I think it makes more sense now to narrow down the problem. Anyway, here it is.

Thank you for your feedback. Indeed, I have to be somehow vague, I cannot disclose receipt. I hope this is understandable. Sorry.

Small clarification: “direction of the light cone”, I meant cone of directions/rays, i.e. a cone between virtual point and the pupil with a certain number of rays. The distance of the virtual point from the eye is defined at intersection of the virtual rays with a modulator image plane. The modulator image plane has a fixed position in space. The virtual rays pass through it and that’s where you generate (reflect) the real light-field rays.

About the re-rendering: Displacements between viewpoints inside pupil and between eyes are one to two orders of magnitude different. The objects you see with two eyes are substantially rotated and whole objects can be obscured from eye to eye, this change is entirely negligible even across the whole pupil, and almost unnoticeable between neighboring viewpoints (<1mm apart). This is really not comparable. About the distribution of color information to different perspectives: Even stereo displays used anaglyps at some point. That’s very similar. It just doesn’t work that well as within a single pupil. (Too different images and two different sensors).

About the noticeability of the dither-like images: Of course, if you assume a “low bit-rate image”, or better to say ray-rate, then it may be even very noticeable below some limit. This method needs a very high ray-rate which, however, becomes possible once you are not limited by the amount of rendered data. You can generate theoretically as many rays as you want, in practice you are limited only by the optoelectronic HW. That’s not my fantasy, I know how that light-field looks like at different speeds.

Yes, I do not deny that varifocal is a fast engineering solution to the accommodation cues and that the necessity to remake the whole pipeline is a big disadvantage of the light-field (but it is like not having a railroad. Once it is built, the advantages pay back). Ligh-field cannot compete in a short run. There is substantially more work to do on it. (BTW, as I already mentioned, there exist more types of fast modulator technologies than one, no need to stick to what you write. But, anyway, efficient light-field needs this custom, too. The price is however much more dictated by volumes than by the tech). Nevertheless, no one is eating any lunch, yet.
The main difference is in the level of dependence on the eye-tracking and its reliability. Varifocals need to generate the blur and even the ocular parallax very quickly and precisely while even occasional errors may be very disturbing. I haven’t tested that. Optics is also problematic, the image is not optically flat. Light-field does not need this robustness, indeed it does not need the eye-tracking at all, it is just a bonus. Light-field can correct aberrations.
Fast varifocal sleep may work better and without eye-tracking, but that needs fast bright display and specific rendering, too, while it can technically support only a limited number of focal planes. I saw some, and it was pretty impressive for specific content. The objects were, however, splitting when crossing two or more plains.

I hope with you, that you are wrong :-). Time will show.

Thank you. I understood you meant the FB’s active lens stack with eye-tracking. I just consider the eye-tracking as a big problem in it. I think here Marty Banks went even to conclusion that eye-tracking cannot work perfectly in general and for all.

Sure, there are many risks and uncertainties. If you wait for certainty, you will certainly be late.

Thank you for some additional insights! I tried Varjo, it was good. I didn’t try any of the half-domes. I am curious about the eye-tracking reliability and where the acceptable tolerance of errors is.

Thanks, indeed, we use pupil-labs (among other)

I see. The individual sub-frames have different perspectives, but not different focus (ideally). They are all practically “always-in-focus”, i.e. have very large depth-of-field, ideally infinite. You can imagine that each of the sub-frames is projected sharp on your retina regardless your eye focus. Because they pass through a small aperture, virtually, in reality there is no aperture at your pupil, small is the light-source. The change of focus of the eye just moves positions of the images on your retina. (Yes, in reality, each sub-frame has a finite depth of field, although very large, and a defined focal plane, because of non-zero aperture and diffractions, but it nevertheless works very well. The results prove it.)