Video wird geladen...

Video konnte nicht geladen werden

Zur Startseite

🚀 Excited to share MiniGPT-4, an open-sourced model performing complex vision-language tasks like GPT-4! ✍️ Write poems for photos 🔍 Pinpoint problems & offers solutions 🌐 Draft ➡️ Website and more! 🔗 Gradio Demo & Project:

tikgiau's profile picture

Deyao Zhu

1,632 subscribers

280,620 Aufrufe • vor 3 Jahren •via X (Twitter)

LIVE

Anya Rossi• Live Now

Private livecam show

38 Kommentare

Profilbild von Steven Hoi
Steven Hoivor 3 Jahren

@Gradio MiniGPT-4 is literally just built based on our #BLIP2 with the latest LM #Vicuna, and it was using our LAVIS library, one of the most comprehensive open-source libraries for multimodal language-and-vision intelligence.

Profilbild von Deyao Zhu
Deyao Zhuvor 3 Jahren

@Gradio Full respect to the amazing BLIP-2! The biggest contribution of MiniGPT-4 is not the method, but the finding that many GPT-4's fancy VL abilities can simply come from a more powerful LLM without fancy VL datasets or secret architectures, and we can make it with opensource models

Profilbild von Shawn Fumo
Shawn Fumovor 3 Jahren

@Gradio This is very cool, but wish people would stop using project names that make things more confusing than they already are. Just call it VisualLLM or some random cool-sounding word, so you don't need to worry about cease and desist from OAI like Dalle Mini had.

Profilbild von Deyao Zhu
Deyao Zhuvor 3 Jahren

@xiaoqian_shen @lix709 @moElhoseiny @KAUSTVisionCAIR

Profilbild von Deyao Zhu
Deyao Zhuvor 3 Jahren

Our code, pretrained model, and dataset are available at

Profilbild von Deyao Zhu
Deyao Zhuvor 3 Jahren

In addition to the 8 demo links on our website we now have a demo on Hugging Face too! Check it out at Huge shoutout to Xiaoqian (@@xiaoqian_shen), AK (@_akhaliq), and Hugging Face (@huggingface) for making this happen!

Profilbild von Deyao Zhu
Deyao Zhuvor 3 Jahren

New aligned checkpoint available with Vicuna 7B. Requires only 12GB GPU memory with beam search width of 1. Launch locally with ease! Check the updated repository:

Profilbild von mikecotic
mikecoticvor 3 Jahren

@Gradio 👋 dropped you an email yesterday . would love to ping-pong some ideas with you . I tried using the iPhone neural engine with swift & hugging face models . but today i saw this : which a step forward into javascript 😎

Profilbild von Jun Chen
Jun Chenvor 3 Jahren

@Gradio great work!!!

Profilbild von iamrobotbear (bk)
iamrobotbear (bk)vor 3 Jahren

First off, excellent work and ty. 1. Can this be used with GPT-3.5 for the chat? Vicuna being based on Llama means no commercial use, if I understand correctly. 2. Would be fantastic to see this use BLIP2 to gen a image text match score too. 3. Now we just need to add SAM to this demo too!

Profilbild von Super AI App
Super AI Appvor 3 Jahren

@Gradio Cool, will try out 👌

Profilbild von XIAOQIAN SHEN
XIAOQIAN SHENvor 3 Jahren

@Gradio Awesome!

Profilbild von visarga
visargavor 3 Jahren

@Gradio Scooped OpenAI's Image-GPT-4 by demonstrating how well it works before they demoed! Very excited to know it will be accessible in smaller models.

Profilbild von XIAOQIAN SHEN
XIAOQIAN SHENvor 3 Jahren

@Gradio Thanks for all your interest in our work. Currently, the number of users has exceeded our expectations. We now provide 8 demo links in total.

Profilbild von Mohamed Zahran
Mohamed Zahranvor 3 Jahren

@Gradio Amazing work 👏

Profilbild von Pasquale Minervini
Pasquale Minervinivor 3 Jahren

@AnnCC12 @Gradio @p_nawrot

Profilbild von Arthur Etchells
Arthur Etchellsvor 3 Jahren

@Gradio Congrats on this! Covered MiniGPT4 and LLaVa in the latest issue 

Profilbild von iamrobotbear (bk)
iamrobotbear (bk)vor 3 Jahren

@Gradio Oh this is fantastic, adding something like SAM to this along with Yolov8 would be even better.. Thank you!

Profilbild von Victor "DeFi Toronto" Li
Victor "DeFi Toronto" Livor 3 Jahren

@Gradio Truly amazing. Congratulations!

Profilbild von Nicole Miller 🏛️ ‍⚖️
Nicole Miller 🏛️ ‍⚖️vor 3 Jahren

@Gradio READ👇

Profilbild von Wrghtxf ✝️
Wrghtxf ✝️vor 3 Jahren

@Gradio Someone said: Requirements? …

Profilbild von Guruprasad Venkataraghavan
Guruprasad Venkataraghavanvor 3 Jahren

@_akhaliq @Gradio @memdotai mem it

Profilbild von Mingchen Zhuge
Mingchen Zhugevor 3 Jahren

@Gradio Nice work! Deyao

Profilbild von realxxx
realxxxvor 3 Jahren

@Gradio @SaveToNotion #图像识别

Profilbild von I N D R A J E E T
I N D R A J E E Tvor 3 Jahren

@Gradio This is open source

Profilbild von wwwwg
wwwwgvor 3 Jahren

@Gradio @memdotai mem it

Profilbild von Ant A. 🇺🇸
Ant A. 🇺🇸vor 3 Jahren

@Gradio fix the project again please

Profilbild von Perry Zhang
Perry Zhangvor 3 Jahren

@Gradio this is blip 2 with a better llm and an extra dataset. I dont get why it is needed to brand it as another model.

Profilbild von Zi-Yuan Hu
Zi-Yuan Huvor 3 Jahren

@Gradio Hi Deyao, could you provide some comparison experiments? (just like OpenAI GPT-4 blog)

Profilbild von shawn.c
shawn.cvor 3 Jahren

@Gradio @SaveToNotion #thread

Profilbild von Ryan | Don't Fear AI
Ryan | Don't Fear AIvor 3 Jahren

@Gradio What happens to the image after upload, are you storing it or using it to train your model further? Use ChatGPT to write a privacy policy 🙂

Profilbild von LLP
LLPvor 3 Jahren

@Gradio @SaveToNotion

Profilbild von 🅻🅾🅵🅸911
🅻🅾🅵🅸911vor 3 Jahren

@Gradio Everyone is excited (and so am I) BUT did you see the requirements ? Once they release the mini-economic ver you will be able to run it on RTX 3090 :D A very (not so) cheap gfx.

Profilbild von Deyao Zhu
Deyao Zhuvor 3 Jahren

@Gradio we now have a pretrained model that aligns with the smaller Vicuna7B. This model consumes about 12 GB memory

Profilbild von Science&Technology🖖
Science&Technology🖖vor 3 Jahren

@Gradio Your tweets are like a breath of fresh air in my feed! 🌬️👌🏼

Profilbild von Arnaud Stiegler
Arnaud Stieglervor 3 Jahren

@Gradio Great work! I mostly tested documents, and from the few tests I did, the model seems to hallucinate quite a bit when asked about specific information on the document (basically just information extraction). Wondering what it would take to make the LM more grounded on the image

Profilbild von sky234234234
sky234234234vor 3 Jahren

@Gradio cool but why did you call it that 💀

Profilbild von Yihan Cao
Yihan Caovor 3 Jahren

@Gradio It seems just like frozen, or is it really just the same structure as frozen only with a stronger decoder?

Ähnliche Videos

🚀 Meta FAIR is releasing several new research artifacts on our road to advanced machine intelligence (AMI). These latest advancements are transforming our understanding of perception. 1️⃣ Meta Perception Encoder: A large-scale vision encoder that excels across several image & video tasks. 2️⃣ Meta Perception Language Model: A fully open & reproducible vision-language model designed to tackle visual recognition tasks. 3️⃣ Meta Locate 3D: An end-to-end model for accurate object localization in 3D environments. 4️⃣ Releasing model weights for our 8B-parameter Dynamic Byte Latent Transformer, an alternative to traditional tokenization methods with the potential to redefine the standards for language model efficiency and reliability. 5️⃣Collaborative Reasoner: A framework for evaluating & improving collaborative reasoning skills in language models. Download the code, datasets, and research papers and learn more about how these artifacts are paving the way for more efficient and accurate AI systems.➡️
2:10

🚀 Meta FAIR is releasing several new research artifacts on our road to advanced machine intelligence (AMI). These latest advancements are transforming our understanding of perception. 1️⃣ Meta Perception Encoder: A large-scale vision encoder that excels across several image & video tasks. 2️⃣ Meta Perception Language Model: A fully open & reproducible vision-language model designed to tackle visual recognition tasks. 3️⃣ Meta Locate 3D: An end-to-end model for accurate object localization in 3D environments. 4️⃣ Releasing model weights for our 8B-parameter Dynamic Byte Latent Transformer, an alternative to traditional tokenization methods with the potential to redefine the standards for language model efficiency and reliability. 5️⃣Collaborative Reasoner: A framework for evaluating & improving collaborative reasoning skills in language models. Download the code, datasets, and research papers and learn more about how these artifacts are paving the way for more efficient and accurate AI systems.➡️

AI at Meta

163,622 Aufrufe • vor 1 Jahr

Excited to introduce SOLE-R1, a video-language reasoning model for zero-shot reward prediction for robot manipulation tasks! SOLE-R1 reasoning can serve as the SOLE signal for learning new tasks (completely from scratch) through online RL - i.e., robots start with random actions and learn previously unseen tasks guided only by SOLE-R1 rewards, without any demonstrations, ground-truth rewards, success indicators, or task-specific tuning. SOLE-R1 significantly outperforms strong baselines (e.g., Robometer, RoboReward, TOPReward, GPT-5, Gemini-3-Pro) in zero-shot online RL when evaluated across 40 tasks - including a real-world tabletop manipulation setting and 4 sim environments (LIBERO, ManiSkill, Meta-World, RoboSuite). We open source all models, training data, and code. Website, demos, and paper at: 🧵 (1/6)
1:15

Excited to introduce SOLE-R1, a video-language reasoning model for zero-shot reward prediction for robot manipulation tasks! SOLE-R1 reasoning can serve as the SOLE signal for learning new tasks (completely from scratch) through online RL - i.e., robots start with random actions and learn previously unseen tasks guided only by SOLE-R1 rewards, without any demonstrations, ground-truth rewards, success indicators, or task-specific tuning. SOLE-R1 significantly outperforms strong baselines (e.g., Robometer, RoboReward, TOPReward, GPT-5, Gemini-3-Pro) in zero-shot online RL when evaluated across 40 tasks - including a real-world tabletop manipulation setting and 4 sim environments (LIBERO, ManiSkill, Meta-World, RoboSuite). We open source all models, training data, and code. Website, demos, and paper at: 🧵 (1/6)

Philip Schroeder

19,755 Aufrufe • vor 4 Monaten

We've officially released and open-sourced HunyuanImage 2.1, our latest text-to-image model. The new model delivers on our commitment to balancing performance and quality. With native 2K image generation, HunyuanImage 2.1 is an advanced open-source text-to-image model.🎨 ✨ New in 2.1: 🔹Advanced Semantics: Supports ultra-long and complex prompts of up to 1000 tokens, and precisely controls the generation of multiple subjects in a single image. 🔹Precise Chinese and English Text Rendering with seamless image–text integration: The model naturally integrates text into images, making it suitable for a wide range of applications such as product covers, illustrations, and poster design to meet the needs of various fields. 🔹Rich Styles and High Aesthetic: Capable of generating images in various styles—including photorealistic portraits, comics, and vinyl figures—it delivers outstanding visual appeal and artistic quality. 🔹High-Quality Generation: Efficiently produces ultra-high-definition (2K) images in the same time other models take to generate a 1K image. HunyuanImage 2.1 uses two text encoders: a multimodal large language model (MLLM) to improve the model's image and text alignment capabilities, and a multi-language character-aware encoder to improve text rendering capabilities. The model is a single- and double-stream diffusion transformer with 17B parameters. We've also open-sourced the weights of the the accelerated version with meanflow which reduces inference steps from 100 to just 8, and PromptEnhancer, the first industrial-grade rewriting model that enhances your prompts for more nuanced and expressive image generation. Now, creators turn complex ideas—like posters with slogans or multi-panel comics—into visuals faster than ever. We’re just getting started. Stay tuned for our native multimodal image generation model coming soon. 🌐Website: 🔗Github: 🤗Hugging Face: ✨Hugging Face Demo:
1:39

We've officially released and open-sourced HunyuanImage 2.1, our latest text-to-image model. The new model delivers on our commitment to balancing performance and quality. With native 2K image generation, HunyuanImage 2.1 is an advanced open-source text-to-image model.🎨 ✨ New in 2.1: 🔹Advanced Semantics: Supports ultra-long and complex prompts of up to 1000 tokens, and precisely controls the generation of multiple subjects in a single image. 🔹Precise Chinese and English Text Rendering with seamless image–text integration: The model naturally integrates text into images, making it suitable for a wide range of applications such as product covers, illustrations, and poster design to meet the needs of various fields. 🔹Rich Styles and High Aesthetic: Capable of generating images in various styles—including photorealistic portraits, comics, and vinyl figures—it delivers outstanding visual appeal and artistic quality. 🔹High-Quality Generation: Efficiently produces ultra-high-definition (2K) images in the same time other models take to generate a 1K image. HunyuanImage 2.1 uses two text encoders: a multimodal large language model (MLLM) to improve the model's image and text alignment capabilities, and a multi-language character-aware encoder to improve text rendering capabilities. The model is a single- and double-stream diffusion transformer with 17B parameters. We've also open-sourced the weights of the the accelerated version with meanflow which reduces inference steps from 100 to just 8, and PromptEnhancer, the first industrial-grade rewriting model that enhances your prompts for more nuanced and expressive image generation. Now, creators turn complex ideas—like posters with slogans or multi-panel comics—into visuals faster than ever. We’re just getting started. Stay tuned for our native multimodal image generation model coming soon. 🌐Website: 🔗Github: 🤗Hugging Face: ✨Hugging Face Demo:

Tencent Hy

89,430 Aufrufe • vor 1 Jahr

Welcome to the era of embodied AI. We are launching AA-Robotics v0.1, our research preview that measures frontier language model performance on understanding, planning, and acting in the physical world. In v0.1, each model is given control of a 6-axis robot arm (5 joints and a gripper) and a simple objective. We’re releasing results for our first eight tasks today. We find that frontier language models in coding agent harnesses have recently become capable of directly manipulating robot arms for simple tasks. Today, they are far too slow and expensive for any kind of production usage - nearly two orders of magnitude worse than human baselines. We expect this to change quickly. GPT-6 Astra places first in AA-Robotics v0.1, followed by GPT-6.1 Sol, Gemini 4 Argon and Claude Opus 5.5. AA-Robotics v0.1 details: ➤ Our first eight tasks test a range of capabilities: control and dexterity (precise grasping and object handling), spatial reasoning (how objects interact with one another) and adaptability (how models respond to the unexpected) ➤ Models get only brief instructions: how to connect to the robot arm (via a simple HTTP API), the arm's name and how to submit an attempt. They can view the scene through four cameras and write and run code to plan and send movements ➤ Once a model submits an attempt, its end state is graded against the task's pass criteria. A pass scores 100; an incomplete attempt earns partial credit. All attempts are zero-shot Key findings: ➤ The GPT family tops the leaderboard: GPT-6 Astra scores 69, ahead of GPT-6.1 Sol at 65 and Gemini 4 Argon at 55. Claude Opus 5.5 (49) and Claude Fable 5.1 (46) trail ➤ Today’s models are incredibly slow at our tasks, taking ~30 minutes on average to complete each tabletop task ➤ Cost varies widely: GPT-6.1 Sol is the cheapest at $1.93 per task on average; GPT-6 Astra, the most expensive, costs $16.36, more than eight times as much ➤ Compared with a human, today's best-performing frontier language models are ~100x slower and ~10x more expensive More results, clips, and how AA-Robotics v0.1 works follow in the thread below.
0:44

Welcome to the era of embodied AI. We are launching AA-Robotics v0.1, our research preview that measures frontier language model performance on understanding, planning, and acting in the physical world. In v0.1, each model is given control of a 6-axis robot arm (5 joints and a gripper) and a simple objective. We’re releasing results for our first eight tasks today. We find that frontier language models in coding agent harnesses have recently become capable of directly manipulating robot arms for simple tasks. Today, they are far too slow and expensive for any kind of production usage - nearly two orders of magnitude worse than human baselines. We expect this to change quickly. GPT-6 Astra places first in AA-Robotics v0.1, followed by GPT-6.1 Sol, Gemini 4 Argon and Claude Opus 5.5. AA-Robotics v0.1 details: ➤ Our first eight tasks test a range of capabilities: control and dexterity (precise grasping and object handling), spatial reasoning (how objects interact with one another) and adaptability (how models respond to the unexpected) ➤ Models get only brief instructions: how to connect to the robot arm (via a simple HTTP API), the arm's name and how to submit an attempt. They can view the scene through four cameras and write and run code to plan and send movements ➤ Once a model submits an attempt, its end state is graded against the task's pass criteria. A pass scores 100; an incomplete attempt earns partial credit. All attempts are zero-shot Key findings: ➤ The GPT family tops the leaderboard: GPT-6 Astra scores 69, ahead of GPT-6.1 Sol at 65 and Gemini 4 Argon at 55. Claude Opus 5.5 (49) and Claude Fable 5.1 (46) trail ➤ Today’s models are incredibly slow at our tasks, taking ~30 minutes on average to complete each tabletop task ➤ Cost varies widely: GPT-6.1 Sol is the cheapest at $1.93 per task on average; GPT-6 Astra, the most expensive, costs $16.36, more than eight times as much ➤ Compared with a human, today's best-performing frontier language models are ~100x slower and ~10x more expensive More results, clips, and how AA-Robotics v0.1 works follow in the thread below.

Artificial Analysis

35,625 Aufrufe • vor 5 Tagen

🚀 Excited to release LongLive 2.0! 🎬 An end-to-end infrastructure for long video generation, with FP4 and parallelism at the core of both training and inference. ⚡45.7 FPS generation speed on 5B model⚡ ✨ LongLive 2.0 supports real-video training, few-step distillation, multi-shot training/inference, sequence-parallel acceleration, NVFP4 KV cache, and async VAE decoding deployment. 🧩 To our knowledge, this is the first open-source 4-bit long video generation infra that covers both training and inference. 🙌 Welcome to check it out, try it, and share feedback! 🔗 Code: 📰 Paper: 🎥 Demo: #LongVideoGeneration #VideoGeneration #Realtime #AIInfra #EfficientAI #FP4 #Parallel #NVIDIA
1:31

🚀 Excited to release LongLive 2.0! 🎬 An end-to-end infrastructure for long video generation, with FP4 and parallelism at the core of both training and inference. ⚡45.7 FPS generation speed on 5B model⚡ ✨ LongLive 2.0 supports real-video training, few-step distillation, multi-shot training/inference, sequence-parallel acceleration, NVFP4 KV cache, and async VAE decoding deployment. 🧩 To our knowledge, this is the first open-source 4-bit long video generation infra that covers both training and inference. 🙌 Welcome to check it out, try it, and share feedback! 🔗 Code: 📰 Paper: 🎥 Demo: #LongVideoGeneration #VideoGeneration #Realtime #AIInfra #EfficientAI #FP4 #Parallel #NVIDIA

Yukang Chen

60,053 Aufrufe • vor 5 Monaten

🚨🚨Can agents earn money, run a business, or even self-organize a society in the physical social world? 🤖🤖 Can agents learn continually to survive and thrive in embodied environments, like how human babies grow? 👶 Super excited to introduce SimWorld, an open-ended simulator of LLM agents in infinite, realistic embodied worlds. SimWorld features 3 key designs: 1⃣Open-ended realistic world simulation - built on Unreal Engine 5, with accurate physical social dynamics - 100+ built-in environments (city, island, wilderness ...) - language-controllable procedural generation - text-to-3D asset generation 2⃣Native interface for LLM/VLM agents - Gym-like agent-environment interaction APIs - plug in any LLMs/VLMs (GPTs, Gemini, Qwen ...) - rich multi-modal perception - open-vocabulary natural-language action outputs 3⃣Diverse physical and social reasoning scenarios - long-horizon embodied reasoning - multi-agent collaboration / competition - easily customizable for any reasoning tasks SimWorld is fully open-sourced, with a hope to become a foundational infrastructure for real-world agent research across disciplines: robotics, economy, public health, education, etc. Project website + more details in the thread👇 ...1/
1:10

🚨🚨Can agents earn money, run a business, or even self-organize a society in the physical social world? 🤖🤖 Can agents learn continually to survive and thrive in embodied environments, like how human babies grow? 👶 Super excited to introduce SimWorld, an open-ended simulator of LLM agents in infinite, realistic embodied worlds. SimWorld features 3 key designs: 1⃣Open-ended realistic world simulation - built on Unreal Engine 5, with accurate physical social dynamics - 100+ built-in environments (city, island, wilderness ...) - language-controllable procedural generation - text-to-3D asset generation 2⃣Native interface for LLM/VLM agents - Gym-like agent-environment interaction APIs - plug in any LLMs/VLMs (GPTs, Gemini, Qwen ...) - rich multi-modal perception - open-vocabulary natural-language action outputs 3⃣Diverse physical and social reasoning scenarios - long-horizon embodied reasoning - multi-agent collaboration / competition - easily customizable for any reasoning tasks SimWorld is fully open-sourced, with a hope to become a foundational infrastructure for real-world agent research across disciplines: robotics, economy, public health, education, etc. Project website + more details in the thread👇 ...1/

Lianhui Qin

65,374 Aufrufe • vor 10 Monaten

🚨 Tencent just dropped a serious open-source AI contender. Meet Hy4 Preview built for productivity, coding & agentic workflows. 770B total parameters. 49B active. 1M+ token context. But the numbers aren’t the most interesting part. Hy4 Preview is designed to handle what actually matters: planning, tool use, coding, research, document workflows and complex multi-step tasks. In Tencent’s internal blind evaluation across 203 engineering tasks, it scored 2.99/4, edging out GLM 5.3 (2.92) and Kimi K3 (2.94). From full-stack development and research to workplace automation and even game development, this looks less like “just another LLM” and more like a model built to get real work done. Worth watching 👀 🔗 Try it with WorkBuddy: Tencent Hy Tencent AI WorkBuddy
1:02

🚨 Tencent just dropped a serious open-source AI contender. Meet Hy4 Preview built for productivity, coding & agentic workflows. 770B total parameters. 49B active. 1M+ token context. But the numbers aren’t the most interesting part. Hy4 Preview is designed to handle what actually matters: planning, tool use, coding, research, document workflows and complex multi-step tasks. In Tencent’s internal blind evaluation across 203 engineering tasks, it scored 2.99/4, edging out GLM 5.3 (2.92) and Kimi K3 (2.94). From full-stack development and research to workplace automation and even game development, this looks less like “just another LLM” and more like a model built to get real work done. Worth watching 👀 🔗 Try it with WorkBuddy: Tencent Hy Tencent AI WorkBuddy

RAVI KUMAR SAHU

63,568 Aufrufe • vor 1 Monat

Excited to share our new project 🦉OWL - an open-source alternative to Manus AI with 6K+ stars and climbing… 🦉OWL: Everyone’s talking about Manus AI (an agent that can research, browse, code, and automate tasks.) But there’s a catch: → All the above agents cost $20 to $200 a month → Manus AI is available via an invite code only OWL by CAMEL-AI is an opensource project that’s breaking records: ⭐ 6K+ GitHub stars in 2 days 🏆 Ranks #1 on the GAIA Benchmark among open-source projects with a 58.18 avg score! 📌 Specialized Multi-Agent Collaboration ↳ AI user agents break down complex tasks ↳ Assistants agents create detailed execution strategies and interact with tools and tool agents ↳ Tools / Tool agents interface with external services and APIs 📌 Real-World Capabilities ↳ Autonomous research, coding, and web browsing ↳ Runs locally on your machine for privacy ↳ Supports top LLMs (easily integrated with GPT-4O, Claude 3.7, Gemini, Mistral, DeepSeek , Qwen, Ollama, Groq more) ↳ Automates complex workflows, from research to execution I 📌 Simple Setup ↳ Fast installs via conda or uv ↳ Docker support for easy deployment ↳ Straightforward config with TOML files No restrictions. No invites. No paywalls. Just powerful multi-agent collaboration. AI autonomy shouldn’t be locked behind paywalls.
1:21

Excited to share our new project 🦉OWL - an open-source alternative to Manus AI with 6K+ stars and climbing… 🦉OWL: Everyone’s talking about Manus AI (an agent that can research, browse, code, and automate tasks.) But there’s a catch: → All the above agents cost $20 to $200 a month → Manus AI is available via an invite code only OWL by CAMEL-AI is an opensource project that’s breaking records: ⭐ 6K+ GitHub stars in 2 days 🏆 Ranks #1 on the GAIA Benchmark among open-source projects with a 58.18 avg score! 📌 Specialized Multi-Agent Collaboration ↳ AI user agents break down complex tasks ↳ Assistants agents create detailed execution strategies and interact with tools and tool agents ↳ Tools / Tool agents interface with external services and APIs 📌 Real-World Capabilities ↳ Autonomous research, coding, and web browsing ↳ Runs locally on your machine for privacy ↳ Supports top LLMs (easily integrated with GPT-4O, Claude 3.7, Gemini, Mistral, DeepSeek , Qwen, Ollama, Groq more) ↳ Automates complex workflows, from research to execution I 📌 Simple Setup ↳ Fast installs via conda or uv ↳ Docker support for easy deployment ↳ Straightforward config with TOML files No restrictions. No invites. No paywalls. Just powerful multi-agent collaboration. AI autonomy shouldn’t be locked behind paywalls.

Guohao Li 🐫

223,361 Aufrufe • vor 1 Jahr

Open-sourcing the first unified LLM routing library 🔥 Meet LLMRouter 16+ routers in ONE framework. Stop reimplementing routing papers from scratch, simply run pip install llmrouter-lib And instantly deploy SOTA LLM routing tailored to your exact needs! 🔗 Get Started with LLMRouter 🚀 Code: 🌐 Project Page: 💸 Why LLMRouter? Why pay GPT-5 prices for "What's the weather?" Smart routing = Significant Savings Simple query → Cheap model Complex query → Powerful model Save 30–50% on inference costs without sacrificing quality 🧭 The 16+ Router Arsenal Switch between state-of-the-art methods with a single flag change. We cover every paradigm: ✅ Single-Round: From classic ML (KNN, SVM) to neural approaches (Graph, Contrastive). ✅ Multi-Round: RL-powered reasoning that "thinks" before routing. ✅ Agentic: Breaks down complex tasks and routes step-by-step. ✅ Personalized: Learns from user history to fit unique needs. 🛠️ Complete "Out-of-the-Box" Toolchain We didn't just ship the code; we shipped the pipeline. ✅ Unified CLI: Train, infer, and chat with any router. ✅ Interactive UI: Full Gradio interface included. ✅ Benchmarks: 11 datasets ready to use + full data generation pipeline. From LLM routing research to production in minutes. 🚀
1:08

Open-sourcing the first unified LLM routing library 🔥 Meet LLMRouter 16+ routers in ONE framework. Stop reimplementing routing papers from scratch, simply run pip install llmrouter-lib And instantly deploy SOTA LLM routing tailored to your exact needs! 🔗 Get Started with LLMRouter 🚀 Code: 🌐 Project Page: 💸 Why LLMRouter? Why pay GPT-5 prices for "What's the weather?" Smart routing = Significant Savings Simple query → Cheap model Complex query → Powerful model Save 30–50% on inference costs without sacrificing quality 🧭 The 16+ Router Arsenal Switch between state-of-the-art methods with a single flag change. We cover every paradigm: ✅ Single-Round: From classic ML (KNN, SVM) to neural approaches (Graph, Contrastive). ✅ Multi-Round: RL-powered reasoning that "thinks" before routing. ✅ Agentic: Breaks down complex tasks and routes step-by-step. ✅ Personalized: Learns from user history to fit unique needs. 🛠️ Complete "Out-of-the-Box" Toolchain We didn't just ship the code; we shipped the pipeline. ✅ Unified CLI: Train, infer, and chat with any router. ✅ Interactive UI: Full Gradio interface included. ✅ Benchmarks: 11 datasets ready to use + full data generation pipeline. From LLM routing research to production in minutes. 🚀

Jiaxuan You

75,596 Aufrufe • vor 9 Monaten

🚨 BREAKING: Microsoft's first robotics foundation model! 🤯 Microsoft just announced Rho-alpha (ρα), their first robotics model derived from the Phi series of vision-language models. Rho-alpha translates natural language commands into control signals for robotic systems performing bimanual manipulation tasks. Commands like "push the green button with the right gripper," "pull out the red wire," "flip the top switch on," or "turn the knob to position 5" get executed directly by dual-arm robots. What makes this different from standard vision-language-action (VLA) models is the additional modalities. Rho-alpha is a VLA+ model that adds tactile sensing to the perceptual mix, with plans to incorporate force feedback. On the learning side, the model is designed to continually improve during deployment by learning from human feedback. The training approach combines trajectories from physical demonstrations and simulated tasks with web-scale visual question answering data. Since teleoperation data is scarce and expensive, Microsoft is using NVIDIA Isaac Sim on Azure to generate physically accurate synthetic datasets via reinforcement learning. These simulated trajectories get combined with commercial and open physical demonstration datasets. The model is currently under evaluation on dual-arm setups and humanoid robots. Microsoft is opening an Early Access Program for organizations interested in evaluating Rho-alpha. Robots that can adapt to dynamic situations and human preferences are more useful in real environments and more trusted by the people operating them. Read more here: ~~ ♻️ Join the weekly robotics newsletter, and never miss any news →
0:34

🚨 BREAKING: Microsoft's first robotics foundation model! 🤯 Microsoft just announced Rho-alpha (ρα), their first robotics model derived from the Phi series of vision-language models. Rho-alpha translates natural language commands into control signals for robotic systems performing bimanual manipulation tasks. Commands like "push the green button with the right gripper," "pull out the red wire," "flip the top switch on," or "turn the knob to position 5" get executed directly by dual-arm robots. What makes this different from standard vision-language-action (VLA) models is the additional modalities. Rho-alpha is a VLA+ model that adds tactile sensing to the perceptual mix, with plans to incorporate force feedback. On the learning side, the model is designed to continually improve during deployment by learning from human feedback. The training approach combines trajectories from physical demonstrations and simulated tasks with web-scale visual question answering data. Since teleoperation data is scarce and expensive, Microsoft is using NVIDIA Isaac Sim on Azure to generate physically accurate synthetic datasets via reinforcement learning. These simulated trajectories get combined with commercial and open physical demonstration datasets. The model is currently under evaluation on dual-arm setups and humanoid robots. Microsoft is opening an Early Access Program for organizations interested in evaluating Rho-alpha. Robots that can adapt to dynamic situations and human preferences are more useful in real environments and more trusted by the people operating them. Read more here: ~~ ♻️ Join the weekly robotics newsletter, and never miss any news →

Lukas Ziegler

61,071 Aufrufe • vor 9 Monaten

Presenting research from Berkeley AI Research Humanoid Intelligence Center. DexTacWAM: A Visuo-Tactile World-Action Model for Dexterous Manipulation.🖐️🤖 Dexterous manipulation requires touch, yet multi-finger tactile data remain scarce and expensive to collect at scale. Through continual vision-to-touch learning, DexTacWAM adapts a pretrained video world model into a visuo-tactile world model for predictive multi-finger contact modeling and action generation. Across six contact-rich dexterous manipulation tasks, DexTacWAM achieves the highest score on every task, averaging 70.6 vs. 38.0 for the strongest baseline. 🔬 Why not simply inject tactile features into the action policy? With the same multi-finger tactile encoder, policy architecture, and training procedure, replacing the tactile world-model latent with direct tactile features drops the 4-task mean from 74.7 to 26.6. Takeaway: the benefit does not come simply from providing tactile observations to the policy, but from making multi-finger contact evolution part of the predicted world state. All code, model weights, and datasets are now open-sourced! 🌐 Project: 📄 Paper: 💻 Code: 🤗 Weights&Data: Author List: Haoran Yuan, Zekai Wang, Boning Shao Haoran Lu, trevordarrell, Ismini Lourentzou, Wei ZHAN
2:24

Presenting research from Berkeley AI Research Humanoid Intelligence Center. DexTacWAM: A Visuo-Tactile World-Action Model for Dexterous Manipulation.🖐️🤖 Dexterous manipulation requires touch, yet multi-finger tactile data remain scarce and expensive to collect at scale. Through continual vision-to-touch learning, DexTacWAM adapts a pretrained video world model into a visuo-tactile world model for predictive multi-finger contact modeling and action generation. Across six contact-rich dexterous manipulation tasks, DexTacWAM achieves the highest score on every task, averaging 70.6 vs. 38.0 for the strongest baseline. 🔬 Why not simply inject tactile features into the action policy? With the same multi-finger tactile encoder, policy architecture, and training procedure, replacing the tactile world-model latent with direct tactile features drops the 4-task mean from 74.7 to 26.6. Takeaway: the benefit does not come simply from providing tactile observations to the policy, but from making multi-finger contact evolution part of the predicted world state. All code, model weights, and datasets are now open-sourced! 🌐 Project: 📄 Paper: 💻 Code: 🤗 Weights&Data: Author List: Haoran Yuan, Zekai Wang, Boning Shao Haoran Lu, trevordarrell, Ismini Lourentzou, Wei ZHAN

Berkeley AI Research

112,249 Aufrufe • vor 20 Tagen

Wrapping up the year and coinciding with #NeurIPS2024, today at Meta FAIR we’re releasing a collection of nine new open source AI research artifacts across our work in developing agents, robustness & safety and new architectures. More in the video from Joelle Pineau. All of this work is part of FAIR’s continued work towards the goal of achieving advanced machine intelligence A few highlights from what we’re releasing today: • Meta Motivo: A first-of-its-kind behavioral foundation model that controls the movements of a virtual embodied humanoid agent to perform complex tasks. • Meta Video Seal: a state-of-the art comprehensive framework for neural video watermarking. • Meta Explore Theory-of-Mind: A program-guided adversarial data generation for theory of mind reasoning. • Meta Large Concept Models: A fundamentally different training paradigm for language modeling that decouples reasoning from language representation. And much more! We’re excited to share this work with the research community and look forward to seeing how it inspires new innovation across the field. Details and access to everything released by FAIR today ➡️
1:34

Wrapping up the year and coinciding with #NeurIPS2024, today at Meta FAIR we’re releasing a collection of nine new open source AI research artifacts across our work in developing agents, robustness & safety and new architectures. More in the video from Joelle Pineau. All of this work is part of FAIR’s continued work towards the goal of achieving advanced machine intelligence A few highlights from what we’re releasing today: • Meta Motivo: A first-of-its-kind behavioral foundation model that controls the movements of a virtual embodied humanoid agent to perform complex tasks. • Meta Video Seal: a state-of-the art comprehensive framework for neural video watermarking. • Meta Explore Theory-of-Mind: A program-guided adversarial data generation for theory of mind reasoning. • Meta Large Concept Models: A fundamentally different training paradigm for language modeling that decouples reasoning from language representation. And much more! We’re excited to share this work with the research community and look forward to seeing how it inspires new innovation across the field. Details and access to everything released by FAIR today ➡️

AI at Meta

156,123 Aufrufe • vor 1 Jahr

🚀 Introducing HunyuanCustom: An open-source, multimodal-driven architecture for customized video generation, powered by HunyuanVideo-13B. Outperforming existing open-source models, it rivals top closed-source solutions! 🎥 Highlights: ✅Subject Consistency: Maintains identity across single & multi-subject video generation. ✅Multimodal Inputs: Supports text, images, audio, and video for highly controlled outputs. ✅High-Quality Output: Industry-leading detail, motion smoothness, and lighting realism. Key Features: 1️⃣Single-Subject Video: Upload an image + text (e.g., “He’s walking a dog”) to create coherent videos with new actions, outfits, and scenes. 2️⃣Multi-Subject Video: Generate videos with multiple subjects (e.g., a man drinking coffee in a cozy room) from separate image inputs. 3️⃣Audio-Driven Video: Sync audio with visuals for talking or singing in any scene—perfect for digital avatars, virtual customer service, and more. 4️⃣Video-Driven Video: Seamlessly insert or replace subjects into any video for creative enhancements. Dive Deeper: 🌐Project: 📝Technical Details: 💻Code: The single-subject video capability is open-sourced and live on the Hunyuan website, with more features to be released soon! 👉 Try it now: #HunyuanCustom #VideoGeneration #AI #TencentHunyuan
0:40

🚀 Introducing HunyuanCustom: An open-source, multimodal-driven architecture for customized video generation, powered by HunyuanVideo-13B. Outperforming existing open-source models, it rivals top closed-source solutions! 🎥 Highlights: ✅Subject Consistency: Maintains identity across single & multi-subject video generation. ✅Multimodal Inputs: Supports text, images, audio, and video for highly controlled outputs. ✅High-Quality Output: Industry-leading detail, motion smoothness, and lighting realism. Key Features: 1️⃣Single-Subject Video: Upload an image + text (e.g., “He’s walking a dog”) to create coherent videos with new actions, outfits, and scenes. 2️⃣Multi-Subject Video: Generate videos with multiple subjects (e.g., a man drinking coffee in a cozy room) from separate image inputs. 3️⃣Audio-Driven Video: Sync audio with visuals for talking or singing in any scene—perfect for digital avatars, virtual customer service, and more. 4️⃣Video-Driven Video: Seamlessly insert or replace subjects into any video for creative enhancements. Dive Deeper: 🌐Project: 📝Technical Details: 💻Code: The single-subject video capability is open-sourced and live on the Hunyuan website, with more features to be released soon! 👉 Try it now: #HunyuanCustom #VideoGeneration #AI #TencentHunyuan

Tencent HY

191,908 Aufrufe • vor 1 Jahr

We’re excited to announce the release and open-source of HunyuanImage 3.0 — the largest and most powerful open-source text-to-image model to date, with over 80 billion total parameters, of which 13 billion are activated per token during inference.The effect is completely comparable to the industry’s flagship closed-source model.🚀🚀🚀 HunyuanImage 3.0 originates from our internally developed native multimodal large language model, with fine-tuning and post-training focused on text-to-image generation. This unique foundation gives the model a powerful set of capabilities: ✅Reason with world knowledge ✅Understand complex, thousand-word prompts ✅Generate precise text within images Different from traditional DiT architecture image generation models, HunyuanImage 3.0’s MoE architecture uses a Transfusion-based approach to deeply couple Diffusion and LLM training for a single, powerful system. Built on Hunyuan-A13B, HunyuanImage 3.0 was trained on a massive dataset: 5 billion image-text pairs, video frames, interleaved image-text data, and 6 trillion tokens of text corpora. This hybrid training across multimodal generation, understanding, and LLM capabilities allows the model to seamlessly integrate multiple tasks. Whether you're an illustrator, designer, or creator, this is built to slash your workflow from hours to minutes. HunyuanImage 3.0 can generate intricate text, detailed comics, expressive emojis, and lively, engaging illustrations for educational content. The current release focuses solely on text-to-image generation and future updates will include image-to-image, image editing, multi-turn interaction, and more. 👉🏻Try it now: 🔗GitHub: 🤗Hugging Face:
2:09

We’re excited to announce the release and open-source of HunyuanImage 3.0 — the largest and most powerful open-source text-to-image model to date, with over 80 billion total parameters, of which 13 billion are activated per token during inference.The effect is completely comparable to the industry’s flagship closed-source model.🚀🚀🚀 HunyuanImage 3.0 originates from our internally developed native multimodal large language model, with fine-tuning and post-training focused on text-to-image generation. This unique foundation gives the model a powerful set of capabilities: ✅Reason with world knowledge ✅Understand complex, thousand-word prompts ✅Generate precise text within images Different from traditional DiT architecture image generation models, HunyuanImage 3.0’s MoE architecture uses a Transfusion-based approach to deeply couple Diffusion and LLM training for a single, powerful system. Built on Hunyuan-A13B, HunyuanImage 3.0 was trained on a massive dataset: 5 billion image-text pairs, video frames, interleaved image-text data, and 6 trillion tokens of text corpora. This hybrid training across multimodal generation, understanding, and LLM capabilities allows the model to seamlessly integrate multiple tasks. Whether you're an illustrator, designer, or creator, this is built to slash your workflow from hours to minutes. HunyuanImage 3.0 can generate intricate text, detailed comics, expressive emojis, and lively, engaging illustrations for educational content. The current release focuses solely on text-to-image generation and future updates will include image-to-image, image editing, multi-turn interaction, and more. 👉🏻Try it now: 🔗GitHub: 🤗Hugging Face:

Tencent Hy

413,296 Aufrufe • vor 1 Jahr

Introducing Indic Parler-TTS: Open-Source Text-to-Speech for Over a Billion Indic Speakers! 🌏 In collaboration with Hugging Face, we are excited to release Indic Parler-TTS, a state-of-the-art open-source text-to-speech system designed to bring accessible and high-quality speech technology to India’s diverse linguistic community. Supporting 20 of the 22 scheduled Indian languages—and English in various accents (US, British, Indian)—it’s built to serve over a billion speakers and empower companies, developers, researchers, and communities. Why Indic Parler-TTS Stands Out: 1. Open and Accessible: Fully open-source with permissive licensing for unrestricted usage. 2. Wide Language Support: Includes a vast range of Indic languages, with rich diversity in voices. 3. High-Quality Audio: Produces natural, clear, and lifelike speech. 4. Adaptable and Fine-Tunable: Customize it to new languages, accents, or specific applications. 5. State-of-the-Art Performance: Proven through rigorous evaluation. 6. Versatile and Inclusive: Indic Parler-TTS offers 69 unique voices across 18 Indian languages, making it a perfect fit for diverse use cases like audiobooks, virtual assistants, and educational tools. Let's democratize speech technology together and make speech technology more inclusive and accessible for everyone. ▶️ Experience it now: Demo: Model page:
1:06

Introducing Indic Parler-TTS: Open-Source Text-to-Speech for Over a Billion Indic Speakers! 🌏 In collaboration with Hugging Face, we are excited to release Indic Parler-TTS, a state-of-the-art open-source text-to-speech system designed to bring accessible and high-quality speech technology to India’s diverse linguistic community. Supporting 20 of the 22 scheduled Indian languages—and English in various accents (US, British, Indian)—it’s built to serve over a billion speakers and empower companies, developers, researchers, and communities. Why Indic Parler-TTS Stands Out: 1. Open and Accessible: Fully open-source with permissive licensing for unrestricted usage. 2. Wide Language Support: Includes a vast range of Indic languages, with rich diversity in voices. 3. High-Quality Audio: Produces natural, clear, and lifelike speech. 4. Adaptable and Fine-Tunable: Customize it to new languages, accents, or specific applications. 5. State-of-the-Art Performance: Proven through rigorous evaluation. 6. Versatile and Inclusive: Indic Parler-TTS offers 69 unique voices across 18 Indian languages, making it a perfect fit for diverse use cases like audiobooks, virtual assistants, and educational tools. Let's democratize speech technology together and make speech technology more inclusive and accessible for everyone. ▶️ Experience it now: Demo: Model page:

AI4Bharat

28,681 Aufrufe • vor 1 Jahr

Nothing beats open-source! We now have a fully open-source coding agent that can compete with any proprietary solution out there: • 500+ available models • It doesn't train on your code • It doesn't charge a penny for tokens • It's open. You can change if you want. Kilo Code is the #1 app on OpenRouter right now, and the community is releasing new features on GitHub at rocket speed (about 20 releases in the last 30 days). I installed it in Visual Studio Code, but you can install it in pretty much any IDE. I need to spend more time with it, but here is a very interesting design choice: Most coding agents support two modes: an "agentic" mode for writing code and an "ask" mode for talking to the model without making any changes. Kilo Code supports 5 different modes: 1. Architect mode - Research, design, and planning 2. Debug mode - Problem solving and troubleshooting 3. Code mode - To write code. 4. Ask mode - To talk to the model. 5. Orchestrator mode - Combine every agent to solve problems. On paper, I really like this. I'll need more time to try out the specialized agents and see whether niching becomes an advantage. Here is the link to the website: Thanks to the Kilo (acq. by Anaconda) for partnering with me on this post.
6:17

Nothing beats open-source! We now have a fully open-source coding agent that can compete with any proprietary solution out there: • 500+ available models • It doesn't train on your code • It doesn't charge a penny for tokens • It's open. You can change if you want. Kilo Code is the #1 app on OpenRouter right now, and the community is releasing new features on GitHub at rocket speed (about 20 releases in the last 30 days). I installed it in Visual Studio Code, but you can install it in pretty much any IDE. I need to spend more time with it, but here is a very interesting design choice: Most coding agents support two modes: an "agentic" mode for writing code and an "ask" mode for talking to the model without making any changes. Kilo Code supports 5 different modes: 1. Architect mode - Research, design, and planning 2. Debug mode - Problem solving and troubleshooting 3. Code mode - To write code. 4. Ask mode - To talk to the model. 5. Orchestrator mode - Combine every agent to solve problems. On paper, I really like this. I'll need more time to try out the specialized agents and see whether niching becomes an advantage. Here is the link to the website: Thanks to the Kilo (acq. by Anaconda) for partnering with me on this post.

Santiago

73,304 Aufrufe • vor 11 Monaten

✨ I spent the last 48 hours making GPT-4 read the entire Solana validator codebase and write documentation, so doesn't have to. Introducing — an AI-powered chatbot trained on nothing but code that can answer deep technical questions. How it works 👇 But first... A huge shoutout to , Zahid Khawaja, and Sean. Their hard work made prototyping this thing a breeze. Without further ado... Devs like to write code, not documentation. Tribal knowledge is lost when devs move on to other projects, leaving future devs to sort through mountains of code and figure out not just how it works, but why it works that way. This is all about to change. GPT-4's ability to write code is stunning. It seems to understand something fundamental about writing software that previous models just didn't. This comprehension of the principles that drive the design behind a complex system carries over into its ability to document existing codebases in a truly impressive way. With the enlarged context window(s), it's now feasible to feed GPT-4 entire files of code and ask it to write documentation about how the code works. Taking this as a starting point, the process looks something like this: 1. Download repo. 2. Depth-first traversal of repo contents, ignoring things like package-lock and binary files. 3. For each file, feed to GPT-4 and ask it to write documentation in markdown. 4. Save the output in a separate location as [outputRoot]/[inputFilepath][inputFilename].md 5. For each folder, we ask GPT-4 to write a summary of the folder, taking the newly generated documentation for all files in the folder and the summaries from each of its subfolders as context. Write this to the filesystem as markdown. Now we have a filesystem that matches the structure of the input repo, but all files in the tree are markdown documentation of the corresponding code file. From here, we: 1. Load markdown documents into LangChain. 2. Embed all documents via OpenAI embeddings. 3. Store embeddings in Pinecone. When a user sends a query: 1. Embed query. 2. Find k-nearest markdown files. 3. Feed to GPT-4 with a prompt asking to answer the query based on k-nearest markdown documents provided. The craziest part of all this? GPT-4 actually wrote ~30% of the code. The results are pretty good for 2 days of work. There is certainly room for improvement. Some items that are top of mind: 1. TolyGPT will occasionally hallucinate answers. It is especially bad with links to external sources, like GitHub. The base model seems to know a bit about Solana already, and sometimes this creeps in. Fine-tuning the prompt can solve some of this. 2. Context selection is difficult in a codebase this large. For example, sometimes it will pull in details about the Solana SDK when asked about transaction processing. The SDK files can seem relevant depending on the phrasing of the question. It may be worth breaking the documentation into subsystems to limit this. 3. Not all files fit into the 32k token window. As of now, there are 23 (out of ~1,100) files that cannot be documented in their entirety. Some of these files are very important to how Solana works. Final thoughts: 1. GPT-4 is super powerful, and we're going to see a ton of tools that supercharge the entire software development lifecycle. This is not 12 months away. For the people that can afford it, these tools are here now. And they're only getting better. Act accordingly. 2. The price of inference has to come down for this to go mainstream. I spent about $300 prototyping this project, and the final crawl cost about the same. The high cost of GPT-4 will push developers to other, cheaper alternatives with similar performance. This is coming very soon. If you have a large software project and you're interested in something like this for your codebase, fill out this form and we'll be in touch this week. Or just DM me :)
0:51

✨ I spent the last 48 hours making GPT-4 read the entire Solana validator codebase and write documentation, so doesn't have to. Introducing — an AI-powered chatbot trained on nothing but code that can answer deep technical questions. How it works 👇 But first... A huge shoutout to , Zahid Khawaja, and Sean. Their hard work made prototyping this thing a breeze. Without further ado... Devs like to write code, not documentation. Tribal knowledge is lost when devs move on to other projects, leaving future devs to sort through mountains of code and figure out not just how it works, but why it works that way. This is all about to change. GPT-4's ability to write code is stunning. It seems to understand something fundamental about writing software that previous models just didn't. This comprehension of the principles that drive the design behind a complex system carries over into its ability to document existing codebases in a truly impressive way. With the enlarged context window(s), it's now feasible to feed GPT-4 entire files of code and ask it to write documentation about how the code works. Taking this as a starting point, the process looks something like this: 1. Download repo. 2. Depth-first traversal of repo contents, ignoring things like package-lock and binary files. 3. For each file, feed to GPT-4 and ask it to write documentation in markdown. 4. Save the output in a separate location as [outputRoot]/[inputFilepath][inputFilename].md 5. For each folder, we ask GPT-4 to write a summary of the folder, taking the newly generated documentation for all files in the folder and the summaries from each of its subfolders as context. Write this to the filesystem as markdown. Now we have a filesystem that matches the structure of the input repo, but all files in the tree are markdown documentation of the corresponding code file. From here, we: 1. Load markdown documents into LangChain. 2. Embed all documents via OpenAI embeddings. 3. Store embeddings in Pinecone. When a user sends a query: 1. Embed query. 2. Find k-nearest markdown files. 3. Feed to GPT-4 with a prompt asking to answer the query based on k-nearest markdown documents provided. The craziest part of all this? GPT-4 actually wrote ~30% of the code. The results are pretty good for 2 days of work. There is certainly room for improvement. Some items that are top of mind: 1. TolyGPT will occasionally hallucinate answers. It is especially bad with links to external sources, like GitHub. The base model seems to know a bit about Solana already, and sometimes this creeps in. Fine-tuning the prompt can solve some of this. 2. Context selection is difficult in a codebase this large. For example, sometimes it will pull in details about the Solana SDK when asked about transaction processing. The SDK files can seem relevant depending on the phrasing of the question. It may be worth breaking the documentation into subsystems to limit this. 3. Not all files fit into the 32k token window. As of now, there are 23 (out of ~1,100) files that cannot be documented in their entirety. Some of these files are very important to how Solana works. Final thoughts: 1. GPT-4 is super powerful, and we're going to see a ton of tools that supercharge the entire software development lifecycle. This is not 12 months away. For the people that can afford it, these tools are here now. And they're only getting better. Act accordingly. 2. The price of inference has to come down for this to go mainstream. I spent about $300 prototyping this project, and the final crawl cost about the same. The high cost of GPT-4 will push developers to other, cheaper alternatives with similar performance. This is coming very soon. If you have a large software project and you're interested in something like this for your codebase, fill out this form and we'll be in touch this week. Or just DM me :)

Sam Hogan 🇺🇸

374,710 Aufrufe • vor 3 Jahren

Before software engineers even begin writing code, they have to set the stage of the entire development process. This process requires engineers to make complex tradeoffs between requirements, system design, and implementations details. Current IDEs that rely on AI features, like chat and inline coding, can help engineers get the job done quickly on small development tasks. Still, engineers spend much more time on larger projects—even after the initial code is generated—by conducting rigorous testing and creating documentation. This is where today’s AI IDEs can do more to accelerate the development lifecycle—and this is why we built Kiro. Kiro is an AI IDE that helps you go from prototype to production with spec-driven development and agent hooks. From simple to complex tasks, Kiro works alongside you to turn prompts into detailed specs, then into working code, docs, and test so what you build is exactly what you want and ready to share with your team. After a developer builds the code with Kiro, Kiro’s agent hooks help engineers solve challenging problems and automate tasks like generating documentation and unit tests. Kiro brings structure and mature engineering practices to AI coding, so you can go from concept to application while being in the driver’s seat every step of the way. Kiro is free during preview, and supports Mac, Windows, and Linux, and most popular programming languages. We're excited for you to try it out and let us know what you think ➡️
1:12

Before software engineers even begin writing code, they have to set the stage of the entire development process. This process requires engineers to make complex tradeoffs between requirements, system design, and implementations details. Current IDEs that rely on AI features, like chat and inline coding, can help engineers get the job done quickly on small development tasks. Still, engineers spend much more time on larger projects—even after the initial code is generated—by conducting rigorous testing and creating documentation. This is where today’s AI IDEs can do more to accelerate the development lifecycle—and this is why we built Kiro. Kiro is an AI IDE that helps you go from prototype to production with spec-driven development and agent hooks. From simple to complex tasks, Kiro works alongside you to turn prompts into detailed specs, then into working code, docs, and test so what you build is exactly what you want and ready to share with your team. After a developer builds the code with Kiro, Kiro’s agent hooks help engineers solve challenging problems and automate tasks like generating documentation and unit tests. Kiro brings structure and mature engineering practices to AI coding, so you can go from concept to application while being in the driver’s seat every step of the way. Kiro is free during preview, and supports Mac, Windows, and Linux, and most popular programming languages. We're excited for you to try it out and let us know what you think ➡️

Swami Sivasubramanian

154,343 Aufrufe • vor 1 Jahr

"The future of AI is agentic. That includes browsers!" Imagine having an AI agent in your browser that can help you complete complex tasks, answer your questions, and streamline your workflow. Today I'm thrilled to share a sneak peek at Project Mariner, a cutting-edge research collaboration between Chrome and Google DeepMind, exploring the future of agentic AI within the browser! Building on the power of Gemini 2.0, Mariner envisions AI agents seamlessly guiding users through online tasks, streamlining workflows and enriching browsing experiences. Imagine having an intelligent co-pilot in your browser, anticipating your needs and proactively offering assistance. We're in the early stages of experimentation, focusing on core functionalities like understanding user intent, automating actions, and providing personalized recommendations. This prototype leverages Gemini's advanced natural language understanding and reasoning capabilities to interpret user requests, both typed and spoken. Mariner can then interact with web pages, retrieve information, and even perform actions like filling out forms or navigating to specific sites. For example, a user could simply ask "Find me a job near me," and Mariner would understand the request, navigate to a relevant job search site, and tailor the search based on the user's location and preferences. This is just one example of how we're exploring Gemini 2.0's potential to unlock agentic experiences through a series of prototypes, including: 1. Agents with multimodal reasoning: Project Astra, our research prototype exploring the capabilities of a universal AI assistant, is enhanced by Gemini 2.0. 2. Agents that can help you accomplish complex tasks: Project Mariner itself focuses on the future of human-agent interaction within the browser. 3. Agents for developers: Jules is an experimental AI-powered coding agent that integrates directly into a GitHub workflow. 4. Agents applied across domains: We're exploring agents for navigating video games and even applying Gemini 2.0's spatial reasoning to robotics. We believe that integrating AI agents directly into the browser has the potential to revolutionize how we interact with the web. Project Mariner aims to make browsing more intuitive, efficient, and personalized. By understanding user context and proactively offering assistance, Mariner can simplify complex tasks, save users time, and empower them to achieve more online. This aligns perfectly with the vision of Gemini 2.0 to create more helpful and intuitive AI experiences. We’re currently testing Mariner with a small group of trusted users to gather feedback and refine the user experience. We believe that this technology holds immense potential to transform the way we browse and interact with information online.
2:14

"The future of AI is agentic. That includes browsers!" Imagine having an AI agent in your browser that can help you complete complex tasks, answer your questions, and streamline your workflow. Today I'm thrilled to share a sneak peek at Project Mariner, a cutting-edge research collaboration between Chrome and Google DeepMind, exploring the future of agentic AI within the browser! Building on the power of Gemini 2.0, Mariner envisions AI agents seamlessly guiding users through online tasks, streamlining workflows and enriching browsing experiences. Imagine having an intelligent co-pilot in your browser, anticipating your needs and proactively offering assistance. We're in the early stages of experimentation, focusing on core functionalities like understanding user intent, automating actions, and providing personalized recommendations. This prototype leverages Gemini's advanced natural language understanding and reasoning capabilities to interpret user requests, both typed and spoken. Mariner can then interact with web pages, retrieve information, and even perform actions like filling out forms or navigating to specific sites. For example, a user could simply ask "Find me a job near me," and Mariner would understand the request, navigate to a relevant job search site, and tailor the search based on the user's location and preferences. This is just one example of how we're exploring Gemini 2.0's potential to unlock agentic experiences through a series of prototypes, including: 1. Agents with multimodal reasoning: Project Astra, our research prototype exploring the capabilities of a universal AI assistant, is enhanced by Gemini 2.0. 2. Agents that can help you accomplish complex tasks: Project Mariner itself focuses on the future of human-agent interaction within the browser. 3. Agents for developers: Jules is an experimental AI-powered coding agent that integrates directly into a GitHub workflow. 4. Agents applied across domains: We're exploring agents for navigating video games and even applying Gemini 2.0's spatial reasoning to robotics. We believe that integrating AI agents directly into the browser has the potential to revolutionize how we interact with the web. Project Mariner aims to make browsing more intuitive, efficient, and personalized. By understanding user context and proactively offering assistance, Mariner can simplify complex tasks, save users time, and empower them to achieve more online. This aligns perfectly with the vision of Gemini 2.0 to create more helpful and intuitive AI experiences. We’re currently testing Mariner with a small group of trusted users to gather feedback and refine the user experience. We believe that this technology holds immense potential to transform the way we browse and interact with information online.

Addy Osmani

29,518 Aufrufe • vor 1 Jahr

Love OpenClaw but hate the token burn? 💸 Running a 24/7 agent on GPT-4/Claude is overkill. You don't need SOTA reasoning to handle a greeting or a simple lookup. LLMRouter 🩷 OpenClaw The first production-ready, agentic router designed to plug directly into OpenClaw. LLMRouter fully supports Multimodal, Memory-Equipped routing that adapts 100% to your needs—compatible with FREE open-source models. The Logic is Simple:🔹 Simple query → Cheap/Local model 🔹 Complex reasoning → SOTA model (GPT-4/Claude 3.5) 🔹 Multimodal input → Vision/Audio specialized model Why this isn't just a switch: 📉 30–50% drop in inference costs 🧠 Zero loss in response quality 🔓 100% compatible with OpenAI-style APIs 🚀 Deploy in Seconds General Usage: Get the library and serve any model: pip install llmrouter-lib llmrouter serve OpenClaw Native Integration: Want the full agent experience? LLMRouter built a dedicated integration for OpenClaw users: LLMRouter Resources: 🔗 Repo: 📦 PyPI: 🤝 Works with: Route smarter. Train your own. Pay less. More on LLMRouter: Most routers are static if/else. LLMRouter is an intelligent, learning system. 🤖 Agentic & Memory-Aware: Decisions aren't stateless. We use RAG-powered memory to route based on context and history. 👤 Fully Personalized: It learns from your usage patterns via RL feedback loops. 🔬 Research-Grade: Switch between 16+ routing strategies (KNN, SVM, BERT, Graph, RL) with a single flag.
0:44

Love OpenClaw but hate the token burn? 💸 Running a 24/7 agent on GPT-4/Claude is overkill. You don't need SOTA reasoning to handle a greeting or a simple lookup. LLMRouter 🩷 OpenClaw The first production-ready, agentic router designed to plug directly into OpenClaw. LLMRouter fully supports Multimodal, Memory-Equipped routing that adapts 100% to your needs—compatible with FREE open-source models. The Logic is Simple:🔹 Simple query → Cheap/Local model 🔹 Complex reasoning → SOTA model (GPT-4/Claude 3.5) 🔹 Multimodal input → Vision/Audio specialized model Why this isn't just a switch: 📉 30–50% drop in inference costs 🧠 Zero loss in response quality 🔓 100% compatible with OpenAI-style APIs 🚀 Deploy in Seconds General Usage: Get the library and serve any model: pip install llmrouter-lib llmrouter serve OpenClaw Native Integration: Want the full agent experience? LLMRouter built a dedicated integration for OpenClaw users: LLMRouter Resources: 🔗 Repo: 📦 PyPI: 🤝 Works with: Route smarter. Train your own. Pay less. More on LLMRouter: Most routers are static if/else. LLMRouter is an intelligent, learning system. 🤖 Agentic & Memory-Aware: Decisions aren't stateless. We use RAG-powered memory to route based on context and history. 👤 Fully Personalized: It learns from your usage patterns via RL feedback loops. 🔬 Research-Grade: Switch between 16+ routing strategies (KNN, SVM, BERT, Graph, RL) with a single flag.

Jiaxuan You

31,451 Aufrufe • vor 8 Monaten