Build a realtime AI character/companion is hard, but it shouldn’t be hard RealChar. - all in one open source codebase🎙️🤖 Powered by your favorite AI tech LlamaIndex 🦙, , , Chroma,@openai GPT3.5/4, Anthropic Claude2 Demo first! Repo last! 🧵👇

Who need RealChar - Your Realtime AI Character - If you had enough texting, just want to talk - If you want to bring your AI character everywhere - If you want to full customization - If you are an eng looking for early and unopinionated codebase to start AI Engineer journey

RealChar. allows you to create and customize your own character/companion with low code (just files and prompts) Then? have a conversation with your character/companion everywhere (web, mobile and terminal!) Here's the tech stack. Let's dive into details

Data preparation - Prompts engineering - use ChatGPT to help define character - Voice Clone - @elevenlabsio great tool to clone voice - Data loader - use @llama_index to process data and store into Vector db - Vector DB - use in memory @trychroma to store data for character

Serving - user input - We support all platforms(web, mobile, terminal) - switchable Speech2Text - choose from Open source or OpenAI whisper and Google Cloud speech - LLM Orchestration - use @LangChainAI to pull context from @trychroma, then communicate with GPT3.5/4 or Claude 2

Serving - user output - Text2Speech - currently only support @elevenlabsio. We love their voices! - Stream and stream! - Put stream to the extreme. Use @LangChainAI async call back to stream text. And @elevenlabsio to stream audio chunks

Serving - infra (none AI stuff) - Github actions - for CI/CD - Docker hub - for hosting images - Google cloud - GKE for serving containers - SQLite - locally store interactions

One interesting comparison between RealChar and other products. Yes, RealChar is "product" ready. :D (We are launching mobile version soon)

RealChar is still a very very early open source codebase. We love to build in public and keep it open source. If you love what we are doing and have learned something from this tweet, consider⭐️RealChar on Github, your encouragement is our motivation! 👇

A little more about the codebase philosophy - I want codebase to be extremely simple to start. - I keep everything either on device or in memory. So, you don't have a lot dependencies. - For less tech user, they can still enjoy create characters! Only prompt and data are needed

A little more about the original demo video. 1. It's edit for better watching experience 2. It's powered by GPT4. GPT4 is better than Claude 2. You can also try it yourself 3. It's recored with @screenstudio . Awesome tool! 4. We cloned @elonmusk 's voice with 1 min audio

If you want to give it a try today, here's our web It's powered by GPT3.5 for budget concern :D We set each conversation to be max 10 mins long, just in case we bankrupt Use headphone for best experience (we haven't had chance to optimize audio echo yet)

In the next few days, I will deep dive the codebase and the "product" RealChar. Follow me for more updates, don't forget to like or retweet my first post. Let's build RealChar - your realtime AI character together.

@a16z team definitely beat us on the timing :D But, we believe a conversation will be more natural, only if it's realtime. Glad to see more open source project for AI companion!

Folks has asked me how hard to add a new character? The answer is very simple! Very low code - create a new character folder - put your text it (any format) - add your story in system prompt That's it! Also, we create an AI character helper to talk you through how to add 🤯

Also, I intentionally keep our technology selection simple, well-known and swap-able. It's a great starter project, if you just started to explore AI and LLM space. Or struggle to make stream work. Highly recommend to give it a star, so you don't lose it

Most people missed the fun part of conversation. Here's a short cut with 1.2x - About cage fight between @elonmusk and @finkd ? - About Elon might lose :D

New demo to show the full capability of

See the tutorial for how to setup RealChar locally Next video: how to add AI character!

See the tutorial for how to add new character from @LyncheeO