正在加载视频...

视频加载失败

I've been experimenting with multiple of these, announcing important messages as they come in, morning briefings, noting down ideas and memos, and browsing agents. I couldn’t resist - here's a playful (unscripted!) video of two talking to each other prompted to be AI’s from "Her

88,095 次观看 • 3 年前 •via X (Twitter)

13 条评论

Justin Alvey 的头像
Justin Alvey3 年前

I “jailbroke” a Google Nest Mini so that you can run your own LLM’s, agents and voice models. Here’s a demo using it to manage all my messages (with help from @onbeeper) 🔊 on, and wait for surprise guest! I thought hard about how to best tackle this and why, see 🧵

Justin Alvey 的头像
Justin Alvey3 年前

After looking into jailbreaking options, I opted to completely replace the PCB. This let’s you use a cheap ($2) but powerful & developer friendly WiFi chip with a highly capable audio framework. This allows a paradigm of multiple cheap edge devices for audio & voice detection…

Justin Alvey 的头像
Justin Alvey3 年前

& offloading large models to a more powerful local device (whether your M2 Mac, PC server w/ GPU or even "tinybox"!) In most cases this device is already trusted with your credentials and data so you don’t have to hand these off to some cloud & data need never leave your home

Justin Alvey 的头像
Justin Alvey3 年前

The custom PCB uses @EspressifSystem's ESP32-S3 I went through 2 revisions from a module to a SoC package with extra flash, simplifying to single-sided SMT (< $10 BOM) All features such as LED’s, capacitive touch, mute switch are working, & even programmable from Arduino (/IDF)

Justin Alvey 的头像
Justin Alvey3 年前

For this demo I used a custom “Maubot” with my @onbeeper credentials (a messaging app which securely bridges your messaging clients using the Matrix protocol & e2e encryption) which runs locally serving an API I’m then using GPT3.5 (for speed) with function calling to query this

Justin Alvey 的头像
Justin Alvey3 年前

Fro the prompt I added details such as family & friends, current date, notification preferences & a list additional character voices that GPT can respond in. The response is then parsed and sent to @elevenlabsio

Justin Alvey 的头像
Justin Alvey3 年前

I’m working on open sourcing the PCB design, build instructions, firmware, bot & server code - expect something in the next week or so. If you don't want to source Nest Mini's (or shells from AliExpress) it's still a great dev platform for developing an assistant! Stay tuned!

Alterego 的头像
Alterego3 年前

Just tell me your price

Jimmy 的头像
Jimmy3 年前

Amazing job ! Curious what STT you use for this project ?

Justin Alvey 的头像
Justin Alvey3 年前

Voice Activity Detection for filtering out spoken phrases, Whisper for STT running on my Macbook / Linux server (depending), and using the "no speech prob". If you were asking about TTS, using Elevenlabs - tbh best out there for voice cloning (for now...)

Florencio Aldape 的头像
Florencio Aldape3 年前

Integrate it with chat gpt and I'll buy 10,000

ethgalaxy.eth 的头像
ethgalaxy.eth3 年前

I love how you're using Alan Watts he really is such a master

Alex Hofmann 的头像
Alex Hofmann3 年前

I couldn't resist to add some atmo..

相关视频