Introducing GRID: the General Robot Intelligence Development platform, designed for prototyping smart and safe robots rapidly using foundation models, LLMs, and simulation. Paper: Try now: GitHub: 🧵👇(1/N)

GRID combines multimodal foundation models with the powerful orchestration capabilities of LLMs. GRID can ingest entire sensor/control APIs of any robot, and for a given task, generate code that goes from sensor -> perception -> reasoning -> control commands. (2/N)

Central to GRID is a foundation mosaic: a collection of models for computer vision, localization, planning, safety, etc. GRID builds upon several existing state-of-the-art models, invoking and chaining them together as needed. You can also bring your own models in! (3/N)

A multi-agent LLM architecture processes user tasks to generate solution plans and deployable code; with LLMs that communicate with each other for plan/code improvement, or handling user feedback. GRID can also learn reusable skills which can be stored and retrieved. (4/N)

GRID contains a high-fidelity simulator AirGen which can generate virtually infinite multimodal, spatiotemporal data. Environmental feedback from the simulation can be leveraged by the LLMs to refine their output code and improve their abilities over time. (5/N)

Users can converse with GRID in natural language to solve complex problems. In a simulated forest fire rescue scenario with a drone, GRID produces code that detects the fire, navigates the drone there, plans a search, and flies the drone identifying people in the forest. (6/N)

Combining LLMs with foundation models ensures effective grounding and reasoning. When asked to detect a safe landing spot for a drone, GRID answers it by internally calling a VQA model, segmenting the pertinent areas, and isolating a flat region using an RGB->depth model. (7/N)

Foundation models enable significant zero-shot capabilities. We implement a vision based landing module for a drone in GRID using an object detector, a point tracker, and an RGB to depth model - which is robust to adverse weather conditions and low light out of the box. (8/N)

GRID integrates machine learning models with robotics primitives. GRID can solve an autonomous inspection problem by importing a mesh into simulation, and then using depth, segmentation and surface normals along with optimal planning to compute a scanning trajectory. (9/N)

GRID aims to lower the barrier to entry into robotics by making foundation models and simulation accessible to any robotics developer/researcher. Our platform is in alpha preview, and is free for academic use. Stay tuned for more updates! (10/10) Docs:

Summary blogpost on the GRID platform and its abilities: Joint work with @shuhang0chen, @Abhinav95_, @dnaraya, @akapoor_av8r

@HartleyAerial check this out!

Amazing, @BradPorter_ 👀

Sounds interesting👌 do you have any plans to integrate automative radars (like LiDAR)? Will there any chance to be open source? We mostly add our custom drones model to airsim or add some custom weather systems (for integration of snow or fall colors) in UE.

This is bad ass.

This is awesome! Have you wondered about creating one to find the sound of trees getting cut down? The Amazon forest will thank you for that. 😊

Is it open source license?

mindblowing

@zahi_kakish Looks great, what engine is this if you don't mind me asking?

@zahi_kakish Our simulator is built on Unreal Engine 5 - more details here:

@zahi_kakish Awesome - I thought so! I love UE

This is really cool but I found it interesting that it had to be prompted to avoid collision. Shouldn’t that be a default setting?

Remarkable, my head is sparking with 1000+ applications!!!

Patented?