New Experiment: SerendipityLM - Interactive evolutionary exploration of generative design spaces with large language models. Check out the extensive experiment report, incl. many generative art examples & link to open source repo: Find out more in this 🧵

A major HCI challenge with generative AI is the vastness of the design spaces users explore. Human interaction can greatly enhance the novelty & diversity of AI outputs but is often underutilized. Simple methods like manual prompting are inefficient & limit users to local minima

Every designer, artist & researcher knows that lucky accidents - serendipity - often spark breakthrough. Though "Serendipity Engineering" is still in its infancy, diverse research has shown the potential of "Managed Serendipity" to improve collaboration with AI tools.

In AI, there's a debate over if models are just vast databases or can truly create something new. The role of "human-in-the-loop" methods is often overlooked. As AI integrates deeper into our tools, a key question is: Does the model reflect the tech or the user's vision?

In 2007, @kenneth0stanley & co's Picbreeder launched as a study in collaborative evolutionary exploration of design spaces. Despite the vast search space, users quickly discovered recognizable images without explicitly looking for them. How? See insights from Stantley in image

Since then, AI has evolved & is now capable of generating HQ text & code. Many new methods to incorporate user inputs along with nature-inspired concepts were explored. There is a resurgence in interactive evolutionary approaches that are an untapped potential to advance HCI/AI

Generative Art, over a century old, challenges artists with its expansive design spaces and many skills required. Describing art in words is hard & selecting the "best" pieces is subjective, making it a prime field for exploring new approaches in Serendipity Engineering.

SerendipityLM is an experiment in the interactive evolutionary exploration of generative design spaces with large language models. The basic concept is straightforward (see image) but the generative art quickly made with it is complex and varied.

SerendipityLM transform vast design spaces into a playful, game-like experience. By deemphasizes prompts & using evolutionary interactions, it swiftly generates complex generative art without directly searching. It explores how to improve LLM interaction and AI idea variance.

Here is a selection of the examples I generated with SerendipityLM in a few hours. Each was created in under 30 interaction cycles, in just minutes. It uses ThreeJs, GLSL Shaders, Tailwind and related web tech for rendering

SerendipityLM's interaction approach is applicable to related topics, such as generating P5.js code, Webdesign, Cellular Automaton, and short stories. I've explored these experimentally, however refinements are required to make those work great (see blog post).

SerendipityLM effectively makes exploring generative design spaces playful and serendipitous. Yet topics like LLM "in-context learning", preciser user control, and objectively measuring novelty and serendipity remain significant open research questions.

In summary, SerendipityLM is a modest effort to re-imagine how we interact with AI/LLM - not just to generate art, but to foster new ways of interacting, thinking & perceiving. Serendipity Engineering is crucial to gain deeper insights into the nature of intelligence and reality

Respect to the researchers mentioned & linked to in the blog post: @kenneth0stanley @creativeEndvs @SimonGColton @QuantumComrade @fchollet @SakanaAILabs @hardmaru @DoctorJosh @XingyouSong @jeffclune @togelius @WilliamLatham1 @zachlieberman @takawo @nervous_system @beesandbombs

Share and repost this thread if you like it! And If you generate any intriguing examples with SerendipityLM, please share them with me here.

really cool idea! feel like you should talk to @thesephist and @JungleSilicon on account of having very similar goals!

@thesephist @JungleSilicon thanks! fan of the works of @thesephist and @JungleSilicon - curios what they think of this experiment and how it might relate to some of their current research directions.

Great work! I'm a huge fan of Kenneth Stanley and have been thinking along the same lines as you recently. Looking forward to playing around with the repo!

Expand the possibilities of your metabolic research. Resipher tracks real-time cellular oxygen consumption in standard 96-well plates, delivering continuous real-time data directly from your incubator. Request a free virtual demo or quote today >>