Video yükleniyor...

Video Yüklenemedi

Bu video yüklenirken bir sorun oluştu. Bu geçici bir ağ sorunundan kaynaklanıyor olabilir veya video kullanılamıyor olabilir.

Ana Sayfaya Dön

LLM inference without internet using Google's LiteRT no ollama or lmstudio just loading and running the AI model in the browser

Melvin Vivas

21,516 subscribers

32,832 görüntüleme • 9 ay önce •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

13 Yorum

Gudjon profil fotoğrafı

Gudjon9 ay önce

~3GB model file. -- this is a development to follow... can you cache this .task file in local storage? How to optimize its distribution? can it be chunked/segmented?

WEB3GISEL profil fotoğrafı

WEB3GISEL9 ay önce

Does it send analytics/data out?

Melvin Vivas profil fotoğrafı

Melvin Vivas9 ay önce

@GiselFlorez no i turned off internet in the demo

Vasilije profil fotoğrafı

Vasilije9 ay önce

On device inference in the browser is big. Edge AI without servers feels inevitable.

Roronoa D. Zoro profil fotoğrafı

Roronoa D. Zoro9 ay önce

What are the minimum requirements for running this?

Melvin Vivas profil fotoğrafı

Melvin Vivas9 ay önce

you can check the litert docs

Meet profil fotoğrafı

Meet9 ay önce

On mobile(android) completely local and no internet

joke lad profil fotoğrafı

joke lad9 ay önce

Future is bright

Vikram profil fotoğrafı

Vikram9 ay önce

So cool! ✨

Sebastian Buzdugan profil fotoğrafı

Sebastian Buzdugan9 ay önce

those webnn demos with gemma are surprisingly smooth

Mitchell Agoma profil fotoğrafı

Mitchell Agoma1 ay önce

Browser-local inference adds a lifecycle worth testing: finish the download, go offline, reload, then repeat after cache eviction. I'd also check a suspended tab and failed model load so offline support is clear to users. My related QA checklist:

Atif Saleem profil fotoğrafı

Atif Saleem9 ay önce

It may not have to send out data but it may need to fetch data from the internet.

🌊 Sorbus profil fotoğrafı

🌊 Sorbus9 ay önce

For CPU inference on android it’s quiet good but on iOS or PC… Not that much.

Benzer Videolar

Run any LLM like Llama 3.2 directly from Hugging Face using Ollama on your local computer (100% free and without internet).

Run any LLM like Llama 3.2 directly from Hugging Face using Ollama on your local computer (100% free and without internet).

Shubham Saboo

33,551 görüntüleme • 2 yıl önce

Build a fully local AI coding agent on Ubuntu using Canonical Workshop and the Qwen3-Coder Inference Snap. In this step-by-step demo, you'll set up an isolated development environment, connect your coding agent to a locally hosted LLM, and run coding tasks without an internet connection. Follow along to recreate the setup on your own machine, using CPU or GPU inference: #Ubuntu #Linux #AI #Agents #Workshop #OpenSource

Build a fully local AI coding agent on Ubuntu using Canonical Workshop and the Qwen3-Coder Inference Snap. In this step-by-step demo, you'll set up an isolated development environment, connect your coding agent to a locally hosted LLM, and run coding tasks without an internet connection. Follow along to recreate the setup on your own machine, using CPU or GPU inference: #Ubuntu #Linux #AI #Agents #Workshop #OpenSource

Ubuntu

42,099 görüntüleme • 7 gün önce

Another example of the multiple TTS parallel pipelines pattern. Here's a voice AI agent that speaks both English and Arabic, using a specific model/voice for each language. These are PlayAI voices. The STT, TTS, and LLM inference is all running on GroqInc. (The LLM is LLama 4 Maverick.)

Another example of the multiple TTS parallel pipelines pattern. Here's a voice AI agent that speaks both English and Arabic, using a specific model/voice for each language. These are PlayAI voices. The STT, TTS, and LLM inference is all running on GroqInc. (The LLM is LLama 4 Maverick.)

kwindla

13,538 görüntüleme • 1 yıl önce

#M5StackNew 🎊 The LLM630 Compute Kit is an #AI large language model (#LLM) inference development kit, powered by the #Axera #AX630C SoC with a 3.2 TOPs NPU, it delivers efficient AI inference for tasks like computer vision (CV) and LLM processing.

#M5StackNew 🎊 The LLM630 Compute Kit is an #AI large language model (#LLM) inference development kit, powered by the #Axera #AX630C SoC with a 3.2 TOPs NPU, it delivers efficient AI inference for tasks like computer vision (CV) and LLM processing.

M5Stack

16,859 görüntüleme • 1 yıl önce

Bring the power of #deepseekR1 into your local VS Code by using the AI Toolkit extension ( Interact with the R1 model in the playground or generate responses in bulk entirely locally. #ollama #DeepSeek

Bring the power of #deepseekR1 into your local VS Code by using the AI Toolkit extension ( Interact with the R1 model in the playground or generate responses in bulk entirely locally. #ollama #DeepSeek

Rong Lu

24,161 görüntüleme • 1 yıl önce

Finally found a meaningful scenario for running LLM in the browser, peering into the internals of LLM to see what's really happening inside

Finally found a meaningful scenario for running LLM in the browser, peering into the internals of LLM to see what's really happening inside

Broooooklyn

46,567 görüntüleme • 5 ay önce

Llama 2 (7B, 13B and 70B) is now available in Lit-GPT! 🚀 Try running inference on the chat model or finetune the foundation model ➡️ #Llama2 #MachineLearning #opensource

Llama 2 (7B, 13B and 70B) is now available in Lit-GPT! 🚀 Try running inference on the chat model or finetune the foundation model ➡️ #Llama2 #MachineLearning #opensource

Lightning AI ⚡️

17,846 görüntüleme • 3 yıl önce

Most AI agents force you into a new browser. Meet Composite, it isn't just a hosted browser session in the cloud, or a completely separate browser. Composite actually does the clicking, typing, and navigation inside your real tabs, in your existing browser. - Uses your existing browser, so there’s no migration, you keep your bookmarks, extensions, workflows - No lock-in. It works with any site, any AI model, you’re in control. - Composite does the clicks, types, and navigation for you. In your tabs. No migration. No lock-in. No learning curve. Download and start now:

Most AI agents force you into a new browser. Meet Composite, it isn't just a hosted browser session in the cloud, or a completely separate browser. Composite actually does the clicking, typing, and navigation inside your real tabs, in your existing browser. - Uses your existing browser, so there’s no migration, you keep your bookmarks, extensions, workflows - No lock-in. It works with any site, any AI model, you’re in control. - Composite does the clicks, types, and navigation for you. In your tabs. No migration. No lock-in. No learning curve. Download and start now:

The AI Colony

48,585 görüntüleme • 1 yıl önce

3️⃣ ways to run Gemma 3. Get a Gemma model up and running fast using Keras, Ollama, and the GenAI SDK in this demo from Developer Relations Engineer Marina Coelho.

3️⃣ ways to run Gemma 3. Get a Gemma model up and running fast using Keras, Ollama, and the GenAI SDK in this demo from Developer Relations Engineer Marina Coelho.

Google AI Developers

25,358 görüntüleme • 1 yıl önce

We just launched Browser Use LLM 2.0 - our fastest and most accurate web agent model yet. Here it is finding the cheapest flight across 3 airports: +46% better than Browser Use LLM 1.0 on travel tasks.

We just launched Browser Use LLM 2.0 - our fastest and most accurate web agent model yet. Here it is finding the cheapest flight across 3 airports: +46% better than Browser Use LLM 1.0 on travel tasks.

Saurav Panda

21,443 görüntüleme • 9 ay önce

Cyberpunk 2077 Running in a browser…still blows my mind. No install, No launcher, Just open the browser and play. Porter omonkey said optimization is on the way, but AI helping transform game players into game makers.

Cyberpunk 2077 Running in a browser…still blows my mind. No install, No launcher, Just open the browser and play. Porter omonkey said optimization is on the way, but AI helping transform game players into game makers.

Md Ismail Šojal 🕷️

16,099 görüntüleme • 18 gün önce

🚨 YOUR GPU IS PROBABLY WASTING MORE THAN YOU THINK. vLLM is built to squeeze far more useful work out of your GPU when serving LLMs. Running an LLM at scale isn’t just about having a powerful GPU. The real problem is how efficiently you use its memory and compute. That’s where vLLM comes in. → High-throughput LLM inference and serving → PagedAttention for smarter KV-cache memory management → Continuous batching to keep GPUs busy → Prefix caching + chunked prefill → OpenAI-compatible API out of the box → Supports a huge range of modern LLM architectures → Quantization support for running models more efficiently And the crazy part? You can start an OpenAI-compatible inference server with: `vllm serve ` So your application can talk to your own model almost like it’s talking to OpenAI. The bigger idea: Don’t just buy more GPUs. Make the GPUs you already have work harder. That’s why vLLM has become such a major project in LLM inference. 🔥 #vLLM #AI #LLM #Inference #GPU #MachineLearning #AIInfrastructure #OpenSource #AIAgents #Developers

🚨 YOUR GPU IS PROBABLY WASTING MORE THAN YOU THINK. vLLM is built to squeeze far more useful work out of your GPU when serving LLMs. Running an LLM at scale isn’t just about having a powerful GPU. The real problem is how efficiently you use its memory and compute. That’s where vLLM comes in. → High-throughput LLM inference and serving → PagedAttention for smarter KV-cache memory management → Continuous batching to keep GPUs busy → Prefix caching + chunked prefill → OpenAI-compatible API out of the box → Supports a huge range of modern LLM architectures → Quantization support for running models more efficiently And the crazy part? You can start an OpenAI-compatible inference server with: `vllm serve ` So your application can talk to your own model almost like it’s talking to OpenAI. The bigger idea: Don’t just buy more GPUs. Make the GPUs you already have work harder. That’s why vLLM has become such a major project in LLM inference. 🔥 #vLLM #AI #LLM #Inference #GPU #MachineLearning #AIInfrastructure #OpenSource #AIAgents #Developers

Vikas gupta

14,138 görüntüleme • 1 ay önce

INTRODUCING Notte Building the agentic internet with the strongest web browser for LLM agents. We transform ANY webpage into structured text, enabling better web understanding and navigation. Plug any LLM to to build your own AI agent

INTRODUCING Notte Building the agentic internet with the strongest web browser for LLM agents. We transform ANY webpage into structured text, enabling better web understanding and navigation. Plug any LLM to to build your own AI agent

Notte

225,232 görüntüleme • 1 yıl önce

Every dApp using AI today calls out to OpenAI or Anthropic. That single API call is a centralised control point. Somnia put LLM inference inside consensus instead, running it across validators. Here's the full conversation which paul.somi had with Ruthy | Surgence Labs 👇🏻

Every dApp using AI today calls out to OpenAI or Anthropic. That single API call is a centralised control point. Somnia put LLM inference inside consensus instead, running it across validators. Here's the full conversation which paul.somi had with Ruthy | Surgence Labs 👇🏻

Surgence Labs

31,036 görüntüleme • 2 ay önce

💪🏻 This is SO COOL I build an app to count dumbbell reps with an AI model in the browser using Tensorflow.js and a pose detection model

💪🏻 This is SO COOL I build an app to count dumbbell reps with an AI model in the browser using Tensorflow.js and a pose detection model

Wes Bos

4,939,583 görüntüleme • 1 yıl önce

Running Llama 3.1 locally as a code assistant in VSCode with ollama ✨ Download Ollama ( ✨ Download CodeGPT ( ✨ Select the Llama 3.1 model (8b, 70b or 405b) ✨ Click download ✨ Enjoy! :)

Running Llama 3.1 locally as a code assistant in VSCode with ollama ✨ Download Ollama ( ✨ Download CodeGPT ( ✨ Select the Llama 3.1 model (8b, 70b or 405b) ✨ Click download ✨ Enjoy! :)

Daniel San

360,032 görüntüleme • 2 yıl önce

naming a product is harder than building one at this point. so i scraped all 6,194 yc companies since 2005 and trained a tiny model to hallucinate fresh ones :) no llm. no backend. just runs in your browser! data using Context.dev

naming a product is harder than building one at this point. so i scraped all 6,194 yc companies since 2005 and trained a tiny model to hallucinate fresh ones :) no llm. no backend. just runs in your browser! data using Context.dev

aadi — e\acc

64,779 görüntüleme • 2 ay önce