Video yükleniyor...

Video Yüklenemedi

Ana Sayfaya Dön

I've been experimenting with multiple of these, announcing important messages as they come in, morning briefings, noting down ideas and memos, and browsing agents. I couldn’t resist - here's a playful (unscripted!) video of two talking to each other prompted to be AI’s from "Her

88,095 görüntüleme • 3 yıl önce •via X (Twitter)

13 Yorum

Justin Alvey profil fotoğrafı
Justin Alvey3 yıl önce

I “jailbroke” a Google Nest Mini so that you can run your own LLM’s, agents and voice models. Here’s a demo using it to manage all my messages (with help from @onbeeper) 🔊 on, and wait for surprise guest! I thought hard about how to best tackle this and why, see 🧵

Justin Alvey profil fotoğrafı
Justin Alvey3 yıl önce

After looking into jailbreaking options, I opted to completely replace the PCB. This let’s you use a cheap ($2) but powerful & developer friendly WiFi chip with a highly capable audio framework. This allows a paradigm of multiple cheap edge devices for audio & voice detection…

Justin Alvey profil fotoğrafı
Justin Alvey3 yıl önce

& offloading large models to a more powerful local device (whether your M2 Mac, PC server w/ GPU or even "tinybox"!) In most cases this device is already trusted with your credentials and data so you don’t have to hand these off to some cloud & data need never leave your home

Justin Alvey profil fotoğrafı
Justin Alvey3 yıl önce

The custom PCB uses @EspressifSystem's ESP32-S3 I went through 2 revisions from a module to a SoC package with extra flash, simplifying to single-sided SMT (< $10 BOM) All features such as LED’s, capacitive touch, mute switch are working, & even programmable from Arduino (/IDF)

Justin Alvey profil fotoğrafı
Justin Alvey3 yıl önce

For this demo I used a custom “Maubot” with my @onbeeper credentials (a messaging app which securely bridges your messaging clients using the Matrix protocol & e2e encryption) which runs locally serving an API I’m then using GPT3.5 (for speed) with function calling to query this

Justin Alvey profil fotoğrafı
Justin Alvey3 yıl önce

Fro the prompt I added details such as family & friends, current date, notification preferences & a list additional character voices that GPT can respond in. The response is then parsed and sent to @elevenlabsio

Justin Alvey profil fotoğrafı
Justin Alvey3 yıl önce

I’m working on open sourcing the PCB design, build instructions, firmware, bot & server code - expect something in the next week or so. If you don't want to source Nest Mini's (or shells from AliExpress) it's still a great dev platform for developing an assistant! Stay tuned!

Alterego profil fotoğrafı
Alterego3 yıl önce

Just tell me your price

Jimmy profil fotoğrafı
Jimmy3 yıl önce

Amazing job ! Curious what STT you use for this project ?

Justin Alvey profil fotoğrafı
Justin Alvey3 yıl önce

Voice Activity Detection for filtering out spoken phrases, Whisper for STT running on my Macbook / Linux server (depending), and using the "no speech prob". If you were asking about TTS, using Elevenlabs - tbh best out there for voice cloning (for now...)

Florencio Aldape profil fotoğrafı
Florencio Aldape3 yıl önce

Integrate it with chat gpt and I'll buy 10,000

ethgalaxy.eth profil fotoğrafı
ethgalaxy.eth3 yıl önce

I love how you're using Alan Watts he really is such a master

Alex Hofmann profil fotoğrafı
Alex Hofmann3 yıl önce

I couldn't resist to add some atmo..

Benzer Videolar