We have had this for years. The term is Light Field Camera. There was an attempt several years back to do this with the Lytro camera. It worked but had a few major issues. For one the resolution loss was fairly substantial. There was also a fair few issues around the fact that the only way to share the photos was a cumbersome vendor lock in site iirc.
With more modern sensors you might be able to fix the resolution issue as I recall it being around 2010 or so.
So they built a camera with a microlens array and refocussing... Like this one, which has been commercially available for 10+ years? https://raytrix.de/products/
I think the catch is different on this one. They seem to co-optimize the optics hardware and the processing software. They say that new manufacturing allows this kind of stuff. Which I think is cool.Still I wonder how much difference this will realistically make. In my experience (from printed electronics) new customizable manufacturing processes often also come at some cost and expose a lot of new effects, so is difficult to break even with something that keeps one side stable. Although particularly in optics new smartphone lenses I guess have proven how good you can get with such approaches.
That's true, but it might be mainly a cost issue. You can make up for the resolution loss by using a much larger sensor; since the microlenses are very thin, it won't increase the weight or bulk of the device very much. Just the cost.
The overview of the 48 megapixel camera on the iPhone 14 talked chipperly about the fact that the user-land resolution is already decimated (well, quadrimated?) 4 to 1 to 12mp images, with the subtext "but those are highly determined optimized pixels!"
The 48mp sensor on the iPhone seems like a gimmick - even with purely diffraction-limited optics, I doubt you could seriously increase optical bandwidth at that sensor size beyond, say, 10-20Mp, to say nothing of aberration limiting and shot noise constraints.
Just now we're starting to see 40Mp APS-C sensors, and people are debating if it's worth it to do that on a sensor that's like 10x larger than the biggest iphone sensor.
Some simple calculations suggest otherwise. The iPhone 14 Pro 48MP camera has a pixel size of 1220 nanometers. The diameter of the Airy disc at f1.78 for 500nm light is 2170 nanometers. Seems like Apple has designed things so that they're just on the right side of the Rayleigh criterion.
The Rayleigh criterion is admittedly pretty lax from the point of view of imaging. People often talk about digital cameras being "diffraction limited" if the diameter of the Airy disc is larger than the dimension of a pixel (which is clearly the case for the iPhone). However, AFAIK, there is no strict limit here. It's something of an open question how good smart sharpening algorithms can get – especially when they're allowed to 'cheat' by using knowledge of the statistical properties of typical real world scenes.
The pixel size for a 48MP APS-C sensor is about 2900 nanometers, so it's not 10x bigger (even if you compare area).
> It's something of an open question how good smart sharpening algorithms can get
True, but you're strictly limited by what I assume are microscopic gate capacitances on pixels that small. The maximum number of electrons in an iphone pixel are probably less than 10k, leading to less than 7 bits of maximum theoretical entropy per px from shot noise limits. This is going to sharply curtail (no pun intended) your ability to "back out" airy disk overlap.
> There was also a fair few issues around the fact that the only way to share the photos was a cumbersome vendor lock in site iirc.
That's... not how you make a new product popular and make tons of money. That's how you make a new product a niche product. That's how you kill a product.
Vendor lock-in is for when you have already acquired the mindshare. Users hate vendor lock-in.
To start out having vendor lock-in baked in you'll have to have such an incredibly amazing and useful new product -one that's in its own category- that users will disregard the lock-in for having no choice. The Lytro camera is not such an amazing product.
Mindshare is extremely valuable. Vendor lock-in is a tool for when your competition is rising, but it's very risky, as it may close your product to new customers while making your existing customers want to jump ship as soon as practicable. Vendor lock-in is very risky not just for the customer but also for the vendor.
It's not like they wanted to do this. They had no choice because standard photo viewers can't refocus photos. And a Lytro photo without refocusing is... just a photo, and a low-resolution one at that. So they had to build a UI for refocusing and host the shared photos themselves.
If they actually wanted to build a popular product with significant longevity, they could have done FAR more to enable people to build on their file format etc.
This "they didn't want to pursue full vendor lock-in at every step of their company; they had no choice" apologism just doesn't stand up to any scrutiny.
More likely, they made a bid to capture a greater share of the value, at the cost of limiting the size of their market, and it predictably failed. Or maybe they were prioritizing some kind of acquisition, over building a company with real longevity.
I don't really know anything about the company besides a short skim through Google, but it just doesn't seem remotely plausible to me that they really truly wanted to make this accessible to more people and cultivate a larger market and ecosystem, but somehow had literally no choice in pursuing their vendor lock-in strategy.
They chose to bet on control over mindshare, and this had predictable consequences.
Nah what killed Lytro was that it was very expensive, while also being low resolution, while also not really offering a feature that people really wanted. People basically just want the multi focal plane stuff that modern phones do but cheaper and at higher resolution. I don’t think people have ever really demonstrated a desire to share images that could have the focal plane changed by the viewer, but even when you could with this format the functional number of planes was actually quite small.
Also the device itself looked nice, but it was fairly hard to use and the view finder screen was a fairly bad lcd even at the time.
The fact that they weren’t giving away decoders at the time is pretty much a given, especially given how much emphasis they put on the “algorithms” they used.
The Lytro camera allowed you to choose how to focus the image after taking the exposure. So naturally the only way to do that was with their proprietary software because there was no suitable industry standard. It did then allow for exporting in standard formats like JPEG.
They tried to market to sports and nature photographers since it eliminated the need to focus before taking a picture. Thus they would have a better chance of capturing the perfect image at exactly the right instant. Unfortunately, after doing the software focus post-processing the images always looked a little soft. And the autofocus features on DSLRs got faster and more accurate. So there was no market.
> The Lytro camera allowed you to choose how to focus the image after taking the exposure.
This isn't technically correct. Everything was in focus, because the aperture was so small. The software would add an artificial blur, using the 3d models generated.
So, you chose how to artificially blur the image, after taking the exposure.
Yes, this is a plenoptic camera, the main difference from current methods (e.g., [0]) is that they 'learn' how to reconstruct the images instead on extracting the 3d using optics.
Now the resolution is less a problem with plenoptic 2.0 cameras, in [0] they extract full HD images
What is the basic principle behind this approach. A lens essentially allows you to map light rays (which have an origin and a direction vector) to position on the sensor (a pinhole does the same but lets through much less light).
Did Lytro have a sensor that could actually measure both the position and direction of an incident light ray ?
You more or less described it. If you put a microlens over, say, each 3x3 pixel sub-array, then each of those 3x3 squares now captures a locally incident light ray and its direction (with low resolution). You can now treat that as a single image pixel, but now containing angular information describing how that pixel changes with rotation in the field of view. Effectively you end up with an array of tiny cameras.
Applying the same principle in reverse, you can also create a 3D display by adding a microlens array on top of an image. If you use the same microlens array to capture the image as to display it (swapping out the sensor with a display, or a camera film with its developed photograph) then optical physics does all the necessary image processing for you.
IIRC there was a micro-lens array between the objective lens and the sensor; each of the micro lenses added a separate, overlapping contribution on the sensor and with a bunch of signal processing magic you somehow back out the position/direction.
Reading the article, I don't at all see what the difference is between the Lytro camera and what's described there - although I'm not an optics expert.
One difference may be that, as I recall, Lytro was focused on (2D) photography, while this seems to be targeting 3D, which may be a much more compelling application.
I don't know - it seems pretty much the same - the whole point of the Lytro camera was that you had sufficient information in the recorded image to select your point of focus and depth in post-processing, which is essentially the same as having depth data.
The same information is used but there's a big difference between using it to make prettier photos versus reconstructing 3D scenes. The second application domain seems to have much more potential for revolutionary change than the first (to me at least).
The average person, or probably even the average VC, isn't going to read a pitch about being able to take photos without having to adjust lens focus and say "gee, with that technology you could build a 3D model as easily as taking a photo".
The article says this is an advance in the speed of processing light field images. Also the bit about seeing around an opaque object, which I'm not sure what that means but could be an interesting first?
Applied Science has a good video[1] showing how you can look around objects. He is using a regular lens there, but with a lightfield camera you can emulate basically any lens you want in software as long as the captured lightfield is big enough.
> Also the bit about seeing around an opaque object, which I'm not sure what that means but could be an interesting first?
It's just a property of light field cameras.
Technically, other cameras can do it to some extent too. If there's some opaque object with another object behind it,
1. If you focus on the front object, you'll see the object behind it--blurred.
2. If you focus on the back object, you'll be able to see the back object behind the blur of the front object.
With a light field camera, you can essentially "cut away" the blurry parts and get them in focus. You're not able to see anything new, you're just able to see things in the original image more clearly.
“ To the best of our knowledge, we are the first to demonstrate deep learning data-driven 3D photorealistic reconstruction without system calibration and initializations, and we are the first to demonstrate imaging objects behind obstacles using lensless imagers.”
That’s their actual claim, worth examining.
Also AFAICT their camera has no large traditional lens at all, whereas previous work — ie lytro — put a traditional lens in front of their micro lens array.
Because it's a very different concept from a conventional camera lens. It does not produce a focused image. Each lens corresponds to a single sensor, which is manipulated to receive light from multiple angles.
A regular lens manipulates the light so that each sensor corresponds to one pixel, mimicking the way a piece of film would produce a color at each spot.
With a light field camera the actual image has to be constructed computationally afterwards. In effect, the software computes the effect a lens would have had on the light -- meaning you can set the focus later.
There is a lens-like apparatus built into each sensor to allow it to gather the types of data it needs but it's not at all like a conventional camera lens. You couldn't put on a conventional camera lens, and none of the things that a photographer uses a "lens" for are necesary. "Lensless" seems as good a word as any for that: it accurately conveys a camera that you do not and cannot focus, and has no object corresponding to the lens.
> but it's not at all like a conventional camera lens
The physics is exactly the same as a conventional lens, because each micro lens is conventional lenses.
It's all conventional, up to the point of having to separate all of the real, focused, images to make one clear image.
For proof, if you block all the micro lenses, except one, you would see a boring, focused, real image projected onto the sensor. In fact, the original light field cameras, that you could buy off the shelf, had a single, separate, image per microlens. Looking at the image in this press release, it looks the same.
>“We consider our camera lensless because it replaces the bulk lenses used in conventional cameras with a thin, lightweight microlens array made of flexible polymer,” said research team leader Weijian Yang from the University of California, Davis.
Opinions are opinions, peronally I would consider it multi-lens or poly-lens.
[Oh no, I found a review I wrote of the Lytro camera I bought at the time. The last modified in 2014 so I'm really not sure about the "quality" of the writing, it may well be "kwality", but if anyone wants to read a review of the Lytro written immediately around the time I got/used it: https://nerget.com/Lytro.html]
I bought a Lytro camera that was doing this possibly more than a decade ago. It was a “consumer” device, but it was pretty useless (though kind of pretty to look at?).
Essentially you lose huge amounts of 2d resolution on the sensor so the resultant picture were both very low res and also huge. Coupled with no standard format to be available, I vaguely recall that the only way to “share” the images was via a Facebook applet thing.
I assume a decade of work means they can process the data faster and get better resolution, but the tech itself is far from new.
Also you can title an article “this camera has no lens” and then start with “it has a microarray of thousands of lenses” :D
https://en.wikipedia.org/wiki/Fourier_optics physicists were doing fourier transforms long before they had computers. But it's not an abstraction- it's just a straightforward application of physics (although, I think I understand what you mean).
With more modern sensors you might be able to fix the resolution issue as I recall it being around 2010 or so.