正在加载视频...

视频加载失败

加载此视频时出现问题。这可能是由于临时网络问题，或视频可能不可用。

InfiniteHiP Extending Language Model Context Up to 3 Million Tokens on a Single GPU

AK

516,757 subscribers

31,255 次观看 • 1 年前 •via X (Twitter)

科学技术新闻政治教育

Anya Rossi• Live Now

Private livecam show

7 条评论

AK 的头像

AK1 年前

discuss:

Kjael // Skaling Ventures 的头像

Kjael // Skaling Ventures1 年前

Everyone's raving about Sam Altman's one person and 10,000 GPUs vision. 🚀 But remember, solo success isn't new. Pieter Levels crafted million-dollar products alone. Bitcoin, a one-person brainchild, created a trillion in value. WhatsApp? Just 50 people hit a $19B jackpot. What's changed? Tech now makes this more accessible than ever. A staggering 38% of new startups are solo-founded, leveraging AI that can reason, plan, and execute almost everything. Yet, don't confuse accessibility with automation. Your AI's choices mirror yours. Weak in marketing? Automate poor decisions and they multiply rapidly. Misunderstand sales? AI will mimic your mistakes. The harsh truth? AI intensifies the need for human judgment, not diminishes it. Every decision becomes a high-stakes play. Building a one-person empire is feasible—more than ever. But automating success? That's a myth. Want to see how we run lean, profitable micro-SaaS operations? We build in public on Substack >>

Rohan Kulkarni 的头像

Rohan Kulkarni1 年前

which is that tool for monitoring?

Universa 的头像

Universa1 年前

Extending language model context to 3 million tokens on a single GPU is a remarkable achievement. This breakthrough can significantly improve AI systems and enable more complex tasks.

This.is.Amin 的头像

This.is.Amin1 年前

Holly shit

Abhivendra Singh 的头像

Abhivendra Singh1 年前

Extending language model context is a game changer. It’s not just about more tokens; it’s about deeper understanding and more meaningful interactions. This innovation will empower educators and learners alike. Let’s harness this potential to redefine personalized education.

L8NTLABS 的头像

L8NTLABS1 年前

That's a huge leap forward for language models, AK. I've been experimenting with ComfyUI and the possibilities are endless. Extending context up to 3 million tokens on a single GPU will definitely change the game for generative AI.

相关视频

Releasing vui an open source voice mode 300M TTS model Runs on a single consumer gpu / apple sillicon Context aware speech 6 minutes of context

Releasing vui an open source voice mode 300M TTS model Runs on a single consumer gpu / apple sillicon Context aware speech 6 minutes of context

Harry Coultas Blum

11,308 次观看 • 3 个月前

China's Qafind Labs just launched the first Diffusion Language Model (DLM). 🔥 ChatDLM, described as the first Diffusion Language Model (DLM), will be open-sourced soon. - Inference Speed: 2,800 tokens/sec (on A100), which is insanely fast. - Context Window: 131,072 tokens. More details 👇

China's Qafind Labs just launched the first Diffusion Language Model (DLM). 🔥 ChatDLM, described as the first Diffusion Language Model (DLM), will be open-sourced soon. - Inference Speed: 2,800 tokens/sec (on A100), which is insanely fast. - Context Window: 131,072 tokens. More details 👇

AshutoshShrivastava

73,296 次观看 • 1 年前

Andrej Karpathy calls large language models the new computing paradigm: CPU -> LLM bytes -> tokens RAM -> context window this is the large language model OS (LMOS)

Andrej Karpathy calls large language models the new computing paradigm: CPU -> LLM bytes -> tokens RAM -> context window this is the large language model OS (LMOS)

ℏεsam

343,601 次观看 • 1 年前

Kai-Fu Lee says GPU supply constraints are forcing Chinese AI companies to innovate, meaning they can train a frontier model for $3 million contrasted with GPT-5's $1 billion, and deliver inference costs of 10c/million tokens, 1/30th of what an American company charges

Kai-Fu Lee says GPU supply constraints are forcing Chinese AI companies to innovate, meaning they can train a frontier model for $3 million contrasted with GPT-5's $1 billion, and deliver inference costs of 10c/million tokens, 1/30th of what an American company charges

Tsarathustra

218,794 次观看 • 1 年前

Piper Rockelle says children can look up to her as a role model, but they shouldn’t aspire to earn $3 million in a single day unless they’re prepared to put their soul on the line.

Piper Rockelle says children can look up to her as a role model, but they shouldn’t aspire to earn $3 million in a single day unless they’re prepared to put their soul on the line.

Neverclipx

24,835 次观看 • 7 个月前

Introducing DuoAttention: Our new framework slashes both memory and latency for long-context LLMs without sacrificing performance! By applying full KV cache only to critical heads, we achieve: ⚡ 2.55x memory reduction ⚡ 2.18x decoding speedup ⚡ 3.3M tokens on a single A100 GPU

Introducing DuoAttention: Our new framework slashes both memory and latency for long-context LLMs without sacrificing performance! By applying full KV cache only to critical heads, we achieve: ⚡ 2.55x memory reduction ⚡ 2.18x decoding speedup ⚡ 3.3M tokens on a single A100 GPU

Guangxuan Xiao

31,108 次观看 • 1 年前

I trained a small model to do syntax highlighting in the browser with GPU. Meet gpu-lexer from Vercel Labs: Small (27.5KB), fast (runs on WebGPU), and language-agnostic (model guesses the syntax). It is experimental and built for learning!

I trained a small model to do syntax highlighting in the browser with GPU. Meet gpu-lexer from Vercel Labs: Small (27.5KB), fast (runs on WebGPU), and language-agnostic (model guesses the syntax). It is experimental and built for learning!

Shu

793,969 次观看 • 19 天前

We built Curie, colibrì’s own model: 17B parameters, 33 tokens/s on a single CPU core. No GPU. Trained from scratch on a standard laptop. Written in C. Weights on SSD. An engine and a model designed together for the hardware you already own. Early alpha.This is just the beginning

We built Curie, colibrì’s own model: 17B parameters, 33 tokens/s on a single CPU core. No GPU. Trained from scratch on a standard laptop. Written in C. Weights on SSD. An engine and a model designed together for the hardware you already own. Early alpha.This is just the beginning

JustVugg

51,323 次观看 • 16 天前

Introducing 1millioncontext! Visualize what 1 million tokens actually looks like. It also walks through the journey from GPT-3 with 2k tokens to now, where many models are 1M+ context.

Introducing 1millioncontext! Visualize what 1 million tokens actually looks like. It also walks through the journey from GPT-3 with 2k tokens to now, where many models are 1M+ context.

Hassan

18,201 次观看 • 1 个月前

Nearly 3 million $ASTER tokens have just been burned... After announcing an upgraded tokenomic model which introduced a key deflationary element to $ASTER, Aster 🥷 has announced that 2,937,125.53 tokens have now been burned under the new model. The assets have a current market cap of nearly $2 million, the burned tokens coming from the team's allocation.

Nearly 3 million $ASTER tokens have just been burned... After announcing an upgraded tokenomic model which introduced a key deflationary element to $ASTER, Aster 🥷 has announced that 2,937,125.53 tokens have now been burned under the new model. The assets have a current market cap of nearly $2 million, the burned tokens coming from the team's allocation.

BSCN

20,632 次观看 • 3 个月前

CLM: Removing the GPU Memory Barrier for 3D Gaussian Splatting CLM trains a 102-million Gaussian model on the MatrixCity BigCity Aerial Dataset under 4 hours using a single RTX 4090, achieving a PSNR of 25.

CLM: Removing the GPU Memory Barrier for 3D Gaussian Splatting CLM trains a 102-million Gaussian model on the MatrixCity BigCity Aerial Dataset under 4 hours using a single RTX 4090, achieving a PSNR of 25.

Ryohei Sasaki@engineer

34,570 次观看 • 8 个月前

Prime Intellect engineer: "everyone's bragging about a million-token context. here's what they don't tell you. at 256k tokens GPT-5.5 scores 80% on retrieval. push it to a million and it drops to 36%. the model accepts the context, it just can't reason across it. people call it context rot." in a 20-minute talk he explains why bigger context windows won't save your agents. continual learning + training on your own traces + real environments - that's the fix. Watch the talk, then save!

Prime Intellect engineer: "everyone's bragging about a million-token context. here's what they don't tell you. at 256k tokens GPT-5.5 scores 80% on retrieval. push it to a million and it drops to 36%. the model accepts the context, it just can't reason across it. people call it context rot." in a 20-minute talk he explains why bigger context windows won't save your agents. continual learning + training on your own traces + real environments - that's the fix. Watch the talk, then save!

Carnage

749,991 次观看 • 2 个月前

watch gemma 4 12b q8 dancing on a single rtx 3090 at 33 tokens a second average. google dropped this two days ago and it's the kind of thing that quietly moves the floor. a fully multimodal model, text image and audio in one net, 256k context, apache licensed, running entirely on one consumer gpu, no one metering your tokens. what you're watching is the whole loop live: the server streaming tokens top left, the gpu pegged bottom left, the answer landing on the right. all local, all mine. a year ago this needed someone else's datacenter. today it's a card you can buy. open source isn't catching up anymore, it's setting the pace. how fast does yours run?

watch gemma 4 12b q8 dancing on a single rtx 3090 at 33 tokens a second average. google dropped this two days ago and it's the kind of thing that quietly moves the floor. a fully multimodal model, text image and audio in one net, 256k context, apache licensed, running entirely on one consumer gpu, no one metering your tokens. what you're watching is the whole loop live: the server streaming tokens top left, the gpu pegged bottom left, the answer landing on the right. all local, all mine. a year ago this needed someone else's datacenter. today it's a card you can buy. open source isn't catching up anymore, it's setting the pace. how fast does yours run?

Sudo su

36,614 次观看 • 3 个月前

OpenAI Co-founder Andrej Karpathy explains the new computing paradigm: "We're entering a new computing paradigm with large language models acting like CPUs, using tokens instead of bytes, and having a context window instead of RAM. This is the Large Language Model OS (LMOS)"

OpenAI Co-founder Andrej Karpathy explains the new computing paradigm: "We're entering a new computing paradigm with large language models acting like CPUs, using tokens instead of bytes, and having a context window instead of RAM. This is the Large Language Model OS (LMOS)"

Haider.

552,459 次观看 • 2 年前

JUST IN: Google releases Gemini 1.5, a powerful MoE model. It's a huge breakthrough. The model has the longest context window ever seen: 1 million tokens. It can process 1 hour of video, 11 hours of audio, 30,000 lines of code, or 700,000 words in a single prompt. When tested on text, code, image, audio and video evaluations, 1.5 Pro outperforms 1.0 Pro on 87% of the benchmarks used for developing LLMs. You can can sign up in AI Studio to try it out.

JUST IN: Google releases Gemini 1.5, a powerful MoE model. It's a huge breakthrough. The model has the longest context window ever seen: 1 million tokens. It can process 1 hour of video, 11 hours of audio, 30,000 lines of code, or 700,000 words in a single prompt. When tested on text, code, image, audio and video evaluations, 1.5 Pro outperforms 1.0 Pro on 87% of the benchmarks used for developing LLMs. You can can sign up in AI Studio to try it out.

Lior Alexander

83,432 次观看 • 2 年前

the era of agentic AI is here Google just open sourced Gemma 3, you can run it on a single GPU laptop or even phone - agentic AI workflows - 4 Sizes – 1B, 4B, 12B, 27B - up to 128K tokens - speaks 140+ languages - hit 1338 ELO on LMArena try & download for free, links 👇

the era of agentic AI is here Google just open sourced Gemma 3, you can run it on a single GPU laptop or even phone - agentic AI workflows - 4 Sizes – 1B, 4B, 12B, 27B - up to 128K tokens - speaks 140+ languages - hit 1338 ELO on LMArena try & download for free, links 👇

el.cine

199,579 次观看 • 1 年前

PowerInfer - a high-speed inference engine for deploying LLMs locally. Just came across this super interesting project on speeding up inference. It's not MoE but it's a simple approach that exploits the high locality in LLM inference to design a GPU-CPU hybrid inference engine. Hot-activated neurons are preloaded onto the GPU for fast access, while cold-activated neurons (the majority) are computed on the CPU. This approach significantly reduces GPU memory demands and CPU-GPU data transfer. It achieves an average token generation rate of 13.20 tokens/s, with a peak of 29.08 tokens/s, across various LLMs on a single NVIDIA RTX 4090 GPU. It's on only 18% lower than that achieved by a top-tier server-grade A100 GPU. It also significantly outperforms llama.cpp by up to 11.69x while retaining model accuracy. There is a lot more innovation around inference that's coming fast. Really encouraged by the study on sparse computation to enhance the computational efficiency of LLMs. It's now possible to use PowerInfer with Llama 2 and Faclon 40B. Mistral-7B support is coming soon!

PowerInfer - a high-speed inference engine for deploying LLMs locally. Just came across this super interesting project on speeding up inference. It's not MoE but it's a simple approach that exploits the high locality in LLM inference to design a GPU-CPU hybrid inference engine. Hot-activated neurons are preloaded onto the GPU for fast access, while cold-activated neurons (the majority) are computed on the CPU. This approach significantly reduces GPU memory demands and CPU-GPU data transfer. It achieves an average token generation rate of 13.20 tokens/s, with a peak of 29.08 tokens/s, across various LLMs on a single NVIDIA RTX 4090 GPU. It's on only 18% lower than that achieved by a top-tier server-grade A100 GPU. It also significantly outperforms llama.cpp by up to 11.69x while retaining model accuracy. There is a lot more innovation around inference that's coming fast. Really encouraged by the study on sparse computation to enhance the computational efficiency of LLMs. It's now possible to use PowerInfer with Llama 2 and Faclon 40B. Mistral-7B support is coming soon!

elvis

261,622 次观看 • 2 年前

Idefics3-Llama is out! 💥 It's a multimodal model based on Llama 3.1 that accepts arbitrary number of interleaved images with text with a huge context window (10k tokens!) 😍 Link to demo and model in the next one 😏

Idefics3-Llama is out! 💥 It's a multimodal model based on Llama 3.1 that accepts arbitrary number of interleaved images with text with a huge context window (10k tokens!) 😍 Link to demo and model in the next one 😏

merve

28,017 次观看 • 2 年前

$INFQ GPU + CPU + QPU Frontier models are hitting saturation on tokens (data volume) and context windows (memory). $INFQ CTO walks through how they and $NVDA are working together to add a Quantum Processing Unit to the chain as the next inflection point.

$INFQ GPU + CPU + QPU Frontier models are hitting saturation on tokens (data volume) and context windows (memory). $INFQ CTO walks through how they and $NVDA are working together to add a Quantum Processing Unit to the chain as the next inflection point.

YeahDave

57,181 次观看 • 4 个月前