1/20 I trained two segmentation models using GPT-6 Astra Ultra as the orchestrator with no human labels supplied. My prompt was very lazy too as I had to catch a flight. RF-DETR-Seg-M predictions: Left: SAM3 training masks Right: Astra training masks Both videos held out.

2/20 I left Astra with these instructions and a Roboflow account with 50 credits, already signed in through my browser. "Get all images and videos of microducks from here Generate masks and train RF-DETR-Seg-M using

3/20 My prompt, continued: "I will be flying in a few hours. I want to make sure the training run is set up and running even after I close this laptop in 6 hours."

4/20 And my follow-up: "Leave a couple of interesting videos out for showing the final results. We can do two runs. One with SAM3 and one with masks you produce." Both cloud-trained models were ready about 93 minutes after my request.

5/20 The data came from Pollen Robotics' public Microduck page and press kit. Astra archived 161 unique assets, including 124 raster images and 31 videos, kept the original source URLs, and deduplicated assets by file hash.

6/20 Those assets became a 233-image dataset: 139 selected frames extracted from launch/demo videos, 14 original press photos, and 80 rendered examples. Blurred frames were excluded. Original media and annotation provenance were preserved.

7/20 We held out the entire living-room video group and my original four-robot screenshot. The train/validation split followed scenes. The stand-up and skating clips stayed untouched until both datasets and models were frozen.

8/20 Two label pipelines: Roboflow's SAM3 mask auto-labeler with the prompt 'robot'; and contours Astra authored from visual inspection. Some Astra vertices were aligned across related frames, then reviewed. No SAM2/SAM3 contours supplied Astra's real-image masks.

9/20 One shared ingredient: 80 examples came from rendered animations with alpha channels. Those existing alpha channels supplied masks for BOTH runs, with provenance recorded. The comparison changes the labels on the 120 real training images.

10/20 The frozen dataset: 200 training images 31 validation frames 2 diagnostic test photos Training = 108 video frames + 12 photos + 80 rendered examples. The Astra dataset contains 306 instance masks in total.

11/20 Masks were saved as COCO RLE, including holes and disconnected parts. The cloud audit confirmed all 233 images matched across runs; all 33 shared reference masks were pixel-identical. Astra's training masks survived import exactly at 432 x 432.

12/20 Both cloud runs used RF-DETR-Seg-M at 432 x 432, up to 100 epochs with early stopping, and no offline augmentation. Once Roboflow accepted the jobs, training was independent of my laptop. Both completed well within the 6-hour window.

13/20 SAM3 had label-count discrepancies on 17 training images: 5 fewer and 62 extra instances versus the reviewed inventory. Some extras were robots depicted on a monitor; our policy targeted physical Microducks. Counts alone don't measure mask quality.

14/20 Evaluation needed a detail fixed: exported COCO 'area' fields contained bounding-box area. Astra made separate evaluation copies using decoded mask area, preserved the originals, and evaluated both models with shared masks and the same confidence floor.

15/20 Standard validation mask metrics, SAM3 labels / Astra labels: AP50-95: 46.85 / 47.35 AP50: 89.56 / 88.72 AP75: 43.53 / 56.34 Astra's overall AP edge was 0.50 points; SAM3 had higher AP50. Different metrics tell different parts of the story.

16/20 These are preliminary results: 31 correlated validation frames from one scene, only 2 diagnostic test photos, and one training trial per condition. Shared Astra-authored references may favor its boundary conventions. This doesn't establish a general winner.

17/20 Every frame of both demo clips was processed: 1,091 frames, totaling 43.64 seconds. Timing and playback checked. The raw masks expose failures too: overlaps can merge or fragment robots, and display colors can change after occlusion.

18/20 Raw weight download was blocked by account permissions. Astra used Roboflow's supported hosted inference API, disabled active learning, and cached raw RLE responses. The demo videos remained excluded from training and model selection.

19/20 The Astra-trained model also detected all 4 robots in my original, excluded image. I received four PNG masks, a colored overlay, two comparison videos, source and annotation archives, and a reproducible report with failures included.

20/20 End-to-end: 1h 56m 56s. Recorded usage across four agents: 122.54M tokens, including 119.49M cached input tokens. That includes repeated context processing. The workflow covered collection, annotation, matched training, evaluation and review.

really cool! I was thinking of trying this out too. have you tried using astra for pose estimation?

@grok what is the context here? what makes this interesting?

Maybe GPT-6 Astra can already segment duck itself, without training a segmentation model.