Pleased to share Spellburst: an LLM-powered creative coding tool, accepted at ACM UIST! Artists can move between semantic (high level) and syntactic (low level) ideas and explore many branches in parallel Paper: More on the Replit ⠕/Stanford Human-Computer Interaction Group collab:

I worked with Miroslav Suzara, @jennyhansolo, @chris_pondoc, and @HariSubramonyam. Our research goal was to understand how creative coders explore conceptual spaces and work across diff levels of abstraction.

The UI is an auto-layout node-based canvas, with p5 sketches connected together by "operators", which can modify the sketch with a prompt, merge sketches together semantically, extract properties, and diff. The point is to visualize how sketches diverge and converge over time

The core of the interface works well independently of the LLM integration. It's a great canvas for branching p5 sketches and quickly testing out changes in parallel

But the real magic is here: a) any connected downstream sketches will update every time you modify the prompt. b) divergent autocomplete helps artists push prompts in alt directions and uses the prev sketch as context, so it knows about vars / concepts used to guide suggestions

Secondly, c) global variables are extracted from sketches, linked to parts of the prompt they're related to, then linked to sliders. d) uses "semantic merging" to take unrelated sketches and combine elements of them together, like physics from 1 and and color palette from another

There's still a lot to do, but it's a promising direction for more expressive UIs for making interactive media. We'll post a full demo vid next week and I'll be demoing this live in San Francisco late October. We aim to make this publicly available to use soon!

This is the first (but not last) official research collab between @Replit and an academic institution. Hopefully this is some proof that you don't need a dedicated research team to contribute to the community! I did this in my free time / balanced it with my normal work.

@ACMUIST @Replit @StanfordHCI This is reeaally neat. Would love to see this for audio too!

@ACMUIST @Replit @StanfordHCI yesss and just text too

@ACMUIST @Replit @StanfordHCI damnnnn this is crazy

@ACMUIST @Replit @StanfordHCI Appreciate it :))