How far can a very simple eye go in solving vision tasks? Like a 1-pixel camera? Humans have one of the greatest eyes in nature, while many animals have significantly simpler eyes and visual systems yet show complex perceptual behavior. In an interesting project, we find that many computer... vision tasks can be solved without a typical camera and with such simple 1-pixel sensors (photoreceptors). We also find that proper design (e.g., where to place the photoreceptors strategically) makes a big difference, so we developed a computational design method to find them. 🌐 👁️[Solving Vision Tasks with Simple Photoreceptors Instead of Cameras] 🧵1/nshow more

Unlike our computer vision pipelines, the visual systems in nature are highly diverse, and even remarkably simple low-resolution eyes enable complex intelligent behaviors. Their diverse designs are believed to have emerged through evolutionary adaptations to the animals’ specific ecological context and play an important role in their effectiveness. 2/n

Inspired by this, we attempt to solve embodied vision tasks (visual navigation and continuous control) using simple photoreceptors (PRs) with resolutions as low as 1x1 pixel. We observe that agents equipped with PRs perform reasonably well compared to typical cameras that are significantly more complex. 3/n

Does this work in the real world, or are the observations a simulator loophole? We tested this by deploying a control policy using 64 PRs (100s of times smaller than camera) on a real robot. It was indeed reasonably capable of navigating to the target in a novel room based on the low-resolution visual signal. 4/n

What kind of behaviors do PR agents display? Are they only simple ones? We find that agents can effectively use the low-dimensional visual signal to avoid collisions, find targets, find more efficient trajectories, and efficiently explore new scenes. 5/n

Is the design important for the effectiveness of PR sensors? We find that the design (placement, orientation, field of view, etc.) is crucial for the agent’s performance. Poorly designed sensors can lead to a significant drop. The plot shows the large spread between the performances of good and bad designs. 6/n

How can we find well-performing designs? We develop a computational design optimization method that can tailor the sensor’s design to a specific agent, environment, and task. It shows promising results, allowing us to find PR designs that perform similarly to the camera. While a computationally found design might look unintuitive to human eyes, it indeed contains a certain structure that improves its performance upon a random design, as experiments show. 7/n

Can we also improve the design of cameras computationally? We observe that computational design can actually outperform the default intuitive design that people typically adopt for their cameras when the complexity of the control network is relatively low. This suggests that a good design could bring efficiency and compensate for a lack of processing power. 8/n

Is it easy to design photoreceptors intuitively? Via a human survey, we collect intuitive designs and find that, although some can achieve high performance, the variance in performance is wide. This makes sense as PRs, compared to cameras, are unintuitive to humans; thus, the world model based on which humans suggest designs doesn’t serve. This signifies the importance of a computational design, especially for less intuitive domains. 9/n

Overall, we show that, similar to nature, simple and well-designed visual sensors can be enough to solve different vision tasks. This suggests the possibility of making effective, low-cost, and low-compute agents tailored to specific uses and environments. More discussions here Joint work w/ @andrew_atanov @JiaweiAcademic @rishubhsingh135 @yukary0t3 Andrew Spielberg @zamir_ar 10/10

[@CVPR Tutorial]: The morphology of embodied agents plays a significant role in exhibiting intelligence. Being their physical (e.g., body) or perceptual (e.g., sensors) morphology. The video showing a fish that appears to be gracefully swimming is a good example. The fish is dead. It’s the clever morphology of its body that does most of the work in interaction with the surrounding environment (water stream) and makes control effortless. We have a tutorial at #CVPR2024 on this topic on Tuesday at 9:00 AM in Summit 344, discussing robots and biological agents, their physical and perceptual morphology, and automated design methods for them. 🌐 📚[Computational Design of Diverse Morphologies and Sensors for Vision and Robotics]