Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

New Preprint📣 "Beyond the Chat💬: Executable▶️ and Verifiable✅ Text-Editing with LLMs" Getting help from LLMs when editing a document can involve lots of manual copy-pasting.✂️📋 What if LLMs could instead suggest one-click edits directly in a text editor for your review? 1/N

Philippe Laban

1,561 subscribers

11,825 views • 2 years ago •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

8 Comments

Philippe Laban2 years ago

To communicate desired edit locations 📍 to the LLM, the user can initiate *local comment conversations*. No need to say “cut some detail from paragraph 8.” 2/N

Philippe Laban2 years ago

The Chat and Comments are reactive, and require user initiation. The user can also define *Markers* 🖍️, which run continuously in the background and suggest edits proactively ✨. Markers are similar to a customizable Grammarly. 3/N

Philippe Laban2 years ago

LLMs make factual errors ❌. In InkSync, we introduce a Warn-Verify-Audit framework to mitigate inaccuracy risks. The user gets *warned*⚠️ when an edit introduces new information, and the system generates search queries 🔍 to help the user *verify* ✅ information accuracy. 4/N

Philippe Laban2 years ago

Once editing is complete, an auditor 🕵️‍♂️ can review the document in an *Audit Interface* that tracks and surfaces all LLM-generated content. This facilitates manual verification of accuracy before the document is sent, published or finalized ✔️. 5/N

Philippe Laban2 years ago

Check our paper ( for details, including two usability studies. InkSync is a collaboration of @PhilippeLaban, @jesse_vig, @MartiHearst, @CaimingXiong & @jasonwu0731 We plan to release a public demo of InkSync. Interested?📋👇

Matt Figdore2 years ago

This is the biggest productivity cheat code right now. Kiss reading documents goodbye. You can get an instant summary of any document with this tool.

TracyYXChen2 years ago

why @Grammarly hasn't implemented this? My guess: the inference cost is eating the profit

Philippe Laban2 years ago

@Grammarly Good question. I think they have some limited LLM features in Grammarly GO, but yes the cost might be prohibitive at scale (advantage of being a researcher!). Btw, we recently released a public demo of InkSync: for anyone who's curious!

Related Videos

Introducing Zed AI, in collaboration with @anthropicAI. Zed AI brings LLMs directly into your editor with an extensible, text-centric approach. We're also piloting @anthropicAI's new Fast Edit mode for Claude 3.5 Sonnet with a small set of Zed users.

Introducing Zed AI, in collaboration with @anthropicAI. Zed AI brings LLMs directly into your editor with an extensible, text-centric approach. We're also piloting @anthropicAI's new Fast Edit mode for Claude 3.5 Sonnet with a small set of Zed users.

Zed

712,520 views • 2 years ago

Cursor for contracts is here. Announcing Spellbook's new AI document editor. Doc editing for lawyers is disjointed: you can edit with AI in chat, but then need to "Open in Word" to make manual tweaks without breaking formatting. Alternatively you can use a Word add-in like Spellbook's, but this only allows you to work on one document at a time. For a long time, Mitch and I talked about getting to a real "Cursor for Contracts" experience. With a lot of blood and sweat, we're finally there. Spellbook's new editor combines the best of AI editing and manual editing in one place—while fully supporting rich Word formatting that lawyers rely on. Ask AI to make edits across an entire document set, then review, accept and tweak all in one window. This sounds like a small thing, but the reduction in friction has totally changed my document editing workflow.

Cursor for contracts is here. Announcing Spellbook's new AI document editor. Doc editing for lawyers is disjointed: you can edit with AI in chat, but then need to "Open in Word" to make manual tweaks without breaking formatting. Alternatively you can use a Word add-in like Spellbook's, but this only allows you to work on one document at a time. For a long time, Mitch and I talked about getting to a real "Cursor for Contracts" experience. With a lot of blood and sweat, we're finally there. Spellbook's new editor combines the best of AI editing and manual editing in one place—while fully supporting rich Word formatting that lawyers rely on. Ask AI to make edits across an entire document set, then review, accept and tweak all in one window. This sounds like a small thing, but the reduction in friction has totally changed my document editing workflow.

Scott Stevenson

57,123 views • 1 month ago

What if LLMs could learn your habits and preferences well enough (across any context!) to anticipate your needs? In a new paper, we present the General User Model (GUM): a model of you built from just your everyday computer use. 🧵

What if LLMs could learn your habits and preferences well enough (across any context!) to anticipate your needs? In a new paper, we present the General User Model (GUM): a model of you built from just your everyday computer use. 🧵

Omar Shaikh

71,472 views • 1 year ago

New short course on Fine-tuning LLMs! Many developers are moving beyond only prompting, to also fine-tuning LLMs - that is, taking a pre-trained model and training it further on your own data, which can deliver superior results inexpensively. In this course, Sharon Zhou, CEO of Lamini (disclosure: I’m a minor shareholder) shows you how to recognize when fine-tuning can be help, and how to train an open-source LLM on your own data. I hope you enjoy the course!

New short course on Fine-tuning LLMs! Many developers are moving beyond only prompting, to also fine-tuning LLMs - that is, taking a pre-trained model and training it further on your own data, which can deliver superior results inexpensively. In this course, Sharon Zhou, CEO of Lamini (disclosure: I’m a minor shareholder) shows you how to recognize when fine-tuning can be help, and how to train an open-source LLM on your own data. I hope you enjoy the course!

Andrew Ng

502,941 views • 3 years ago

Last week we released Meta Chameleon: a new mixed-modal research model from Meta FAIR. Get the models ➡️ The 7B & 34B safety tuned models we’ve released can take any combination of text and images as input and produce text outputs using a new early fusion approach. While some LLMs have separate image and text encoders or decoders, Chameleon is one of the first publicly released approaches using a single unified architecture. We’re releasing Chameleon models under a research license to help democratize access to foundational mixed-modal models & further research on early fusion. Approach & training details in the paper ➡️

Last week we released Meta Chameleon: a new mixed-modal research model from Meta FAIR. Get the models ➡️ The 7B & 34B safety tuned models we’ve released can take any combination of text and images as input and produce text outputs using a new early fusion approach. While some LLMs have separate image and text encoders or decoders, Chameleon is one of the first publicly released approaches using a single unified architecture. We’re releasing Chameleon models under a research license to help democratize access to foundational mixed-modal models & further research on early fusion. Approach & training details in the paper ➡️

AI at Meta

54,428 views • 2 years ago

TurboEdit Instant text-based image editing discuss: We address the challenges of precise image inversion and disentangled image editing in the context of few-step diffusion models. We introduce an encoder based iterative inversion technique. The inversion network is conditioned on the input image and the reconstructed image from the previous step, allowing for correction of the next reconstruction towards the input image. We demonstrate that disentangled controls can be easily achieved in the few-step diffusion model by conditioning on an (automatically generated) detailed text prompt. To manipulate the inverted image, we freeze the noise maps and modify one attribute in the text prompt (either manually or via instruction based editing driven by an LLM), resulting in the generation of a new image similar to the input image with only one attribute changed. It can further control the editing strength and accept instructive text prompt. Our approach facilitates realistic text-guided image edits in real-time, requiring only 8 number of functional evaluations (NFEs) in inversion (one-time cost) and 4 NFEs per edit. Our method is not only fast, but also significantly outperforms state-of-the-art multi-step diffusion editing techniques.

TurboEdit Instant text-based image editing discuss: We address the challenges of precise image inversion and disentangled image editing in the context of few-step diffusion models. We introduce an encoder based iterative inversion technique. The inversion network is conditioned on the input image and the reconstructed image from the previous step, allowing for correction of the next reconstruction towards the input image. We demonstrate that disentangled controls can be easily achieved in the few-step diffusion model by conditioning on an (automatically generated) detailed text prompt. To manipulate the inverted image, we freeze the noise maps and modify one attribute in the text prompt (either manually or via instruction based editing driven by an LLM), resulting in the generation of a new image similar to the input image with only one attribute changed. It can further control the editing strength and accept instructive text prompt. Our approach facilitates realistic text-guided image edits in real-time, requiring only 8 number of functional evaluations (NFEs) in inversion (one-time cost) and 4 NFEs per edit. Our method is not only fast, but also significantly outperforms state-of-the-art multi-step diffusion editing techniques.

AK

16,062 views • 2 years ago

🚀Napkin AI visuals just got better! We’ve launched Elastic Designs to help you get the right visual, faster. Now your visuals: ✅ Scale automatically with your content – during both generation and editing – so they always look polished without manual fixes ✅ Support more elements than ever, so none of your content gets dropped-off ✅ Instantly update with one-click Visual-Text sync when you change your text – no need to regenerate This is a big step toward making visual generation effortless – so you spend less time tweaking visuals and get to the right one faster. #NapkinAI #VisualAI #AITools #ElasticDesigns #NewFeatures

🚀Napkin AI visuals just got better! We’ve launched Elastic Designs to help you get the right visual, faster. Now your visuals: ✅ Scale automatically with your content – during both generation and editing – so they always look polished without manual fixes ✅ Support more elements than ever, so none of your content gets dropped-off ✅ Instantly update with one-click Visual-Text sync when you change your text – no need to regenerate This is a big step toward making visual generation effortless – so you spend less time tweaking visuals and get to the right one faster. #NapkinAI #VisualAI #AITools #ElasticDesigns #NewFeatures

Napkin AI

51,466 views • 1 year ago

A 4-year-old child has seen 50x more information than the biggest LLMs. Yann LeCun is the Chief AI Scientist at Meta. He recently spoke on “The Expanding Universe of Generative Models” panel at the World Economic Forum in Davos. Yann highlighted the idea that a 4-year-old child is way smarter than current cutting-edge large language models (LLMs). “Think about what a child sees through vision. Put a number on how much information a 4-year-old child has seen during their life. It’s 20 Mbps going through the optical nerve for 16,000 wake hours in the first 4 years of life. 3,600 seconds per hour is 10^15 bytes. This is 50x more information than the biggest LLMs we have. A 4-year-old child is way smarter than these models having acquired an enormous amount of knowledge about how the world works.” The real constraint right now is the ability of LLMs to think. Today, LLMs are only capable of System 1 thinking. System 1 vs System 2 thinking was popularised in the book 'Thinking, Fast and Slow' by Daniel Kahneman. System 1 tasks involve quick, instinctive, automatic responses. LLMs struggle with discontinuous tasks that require a creative leap in progress as they imitate human responses. It's hard to go above human response accuracy if LLMs are only trained on humans. Models are building the track in front of them with each word being generated. What could it mean to give language models System 2 thinking? This remains a future development I'm excited about.

A 4-year-old child has seen 50x more information than the biggest LLMs. Yann LeCun is the Chief AI Scientist at Meta. He recently spoke on “The Expanding Universe of Generative Models” panel at the World Economic Forum in Davos. Yann highlighted the idea that a 4-year-old child is way smarter than current cutting-edge large language models (LLMs). “Think about what a child sees through vision. Put a number on how much information a 4-year-old child has seen during their life. It’s 20 Mbps going through the optical nerve for 16,000 wake hours in the first 4 years of life. 3,600 seconds per hour is 10^15 bytes. This is 50x more information than the biggest LLMs we have. A 4-year-old child is way smarter than these models having acquired an enormous amount of knowledge about how the world works.” The real constraint right now is the ability of LLMs to think. Today, LLMs are only capable of System 1 thinking. System 1 vs System 2 thinking was popularised in the book 'Thinking, Fast and Slow' by Daniel Kahneman. System 1 tasks involve quick, instinctive, automatic responses. LLMs struggle with discontinuous tasks that require a creative leap in progress as they imitate human responses. It's hard to go above human response accuracy if LLMs are only trained on humans. Models are building the track in front of them with each word being generated. What could it mean to give language models System 2 thinking? This remains a future development I'm excited about.

Alex Banks

22,985 views • 2 years ago

Standard Intelligence's devansh responds to roon's tweet that "text is the universal interface," and explains why their new foundation model is trained on video: "At some point in the arbitrarily long future, if we only use text models, we could force most things to be text. But I think there are just a lot of things that are much more native when done from a computer-use [perspective]." "GUIs are designed for humans to use. We have this massive long tail of things on the internet that are entirely undoable by LLMs." "For example, when I do ML engineering most of my time is spent doing the grunt work of engineering. It's a lot of looking at graphs, analyzing, and comparing loss curves. You can do this in text, but it's a much larger pain than doing it in the native interface." "There's a reason humans don't interact with a computer purely through text, it would kind of suck."

Standard Intelligence's devansh responds to roon's tweet that "text is the universal interface," and explains why their new foundation model is trained on video: "At some point in the arbitrarily long future, if we only use text models, we could force most things to be text. But I think there are just a lot of things that are much more native when done from a computer-use [perspective]." "GUIs are designed for humans to use. We have this massive long tail of things on the internet that are entirely undoable by LLMs." "For example, when I do ML engineering most of my time is spent doing the grunt work of engineering. It's a lot of looking at graphs, analyzing, and comparing loss curves. You can do this in text, but it's a much larger pain than doing it in the native interface." "There's a reason humans don't interact with a computer purely through text, it would kind of suck."

TBPN

64,699 views • 7 months ago

New course: Transformers in Practice. You'll get a practical view of how transformer-based LLMs work, so you can reason about their behavior, diagnose problems like slow inference, and make smarter decisions about deployment. This course is built in partnership with AMD and taught by Sharon Zhou. You'll see how transformers generate text one token at a time, how the model decides which earlier words matter most when predicting the next one, and how techniques like quantization speed up inference on GPUs. This is not a video-only course; interactive visualizations throughout let you play with these concepts and build intuition that sticks. Skills you'll gain: - Understand why LLMs hallucinate, and RAG and chain-of-thought shape what they generate - Look inside the model to see how attention and layers combine to predict the next token - Diagnose inference bottlenecks and learn the techniques that speed up transformers on GPUs Join and understand what's really happening inside your LLMs:

New course: Transformers in Practice. You'll get a practical view of how transformer-based LLMs work, so you can reason about their behavior, diagnose problems like slow inference, and make smarter decisions about deployment. This course is built in partnership with AMD and taught by Sharon Zhou. You'll see how transformers generate text one token at a time, how the model decides which earlier words matter most when predicting the next one, and how techniques like quantization speed up inference on GPUs. This is not a video-only course; interactive visualizations throughout let you play with these concepts and build intuition that sticks. Skills you'll gain: - Understand why LLMs hallucinate, and RAG and chain-of-thought shape what they generate - Look inside the model to see how attention and layers combine to predict the next token - Diagnose inference bottlenecks and learn the techniques that speed up transformers on GPUs Join and understand what's really happening inside your LLMs:

Andrew Ng

124,677 views • 4 months ago

VTUBERS‼️LOOKING FOR A VIDEO EDITOR⁉️ Look no further as I'm opening my VGen again!🎬✨ If you're interested in getting your clips enhanced and edited with lots and lots of meme by a fellow VTuber who has a lot of experience editing these types of clips, I've got you covered!! ᕙ( •̀ ᗜ •́ )ᕗ Price starts from 10$!! 🫶

VTUBERS‼️LOOKING FOR A VIDEO EDITOR⁉️ Look no further as I'm opening my VGen again!🎬✨ If you're interested in getting your clips enhanced and edited with lots and lots of meme by a fellow VTuber who has a lot of experience editing these types of clips, I've got you covered!! ᕙ( •̀ ᗜ •́ )ᕗ Price starts from 10$!! 🫶

Jeyraident ⌛✨【1.0 Debut → 27th June】

49,947 views • 7 months ago

3D editing is hard: you need to ground an image + instruction and generate a faithful 3D shape in one forward pass -- no test-time optimization. So, we steer pretrained image-to-3D representations to do text-guided 3D edits; no massive 3D edit-pair dataset needed. Key trap: the “no-edit” solution is a nasty local minimum. We fix it with preference optimization, pushing the model to actually edit. Steer3D is the second work that adapts alignment ideas from LLMs to the 3D modality. SAM 3D also used DPO to improve its 3D generations.

3D editing is hard: you need to ground an image + instruction and generate a faithful 3D shape in one forward pass -- no test-time optimization. So, we steer pretrained image-to-3D representations to do text-guided 3D edits; no massive 3D edit-pair dataset needed. Key trap: the “no-edit” solution is a nasty local minimum. We fix it with preference optimization, pushing the model to actually edit. Steer3D is the second work that adapts alignment ideas from LLMs to the 3D modality. SAM 3D also used DPO to improve its 3D generations.

Georgia Gkioxari

116,203 views • 9 months ago

Imagine if language models could tap into the app ecosystem of your iPhone. Would the need for plugins and assistants become obsolete if we simply allowed a model to orchestrate our existing (and many years robust) user interfaces? This demonstrates the extent to which GPT-4V excels as a Generalist Mobile AI Agent – without any fine-tuning or grounding, and merely by integrating with a text model that has JSON mode enabled. I suggest watching this demo for a (maybe) wow factor and the results on iOS 17 using NavAIGuide, a mobile and web navigational agent framework for LLMs:

Imagine if language models could tap into the app ecosystem of your iPhone. Would the need for plugins and assistants become obsolete if we simply allowed a model to orchestrate our existing (and many years robust) user interfaces? This demonstrates the extent to which GPT-4V excels as a Generalist Mobile AI Agent – without any fine-tuning or grounding, and merely by integrating with a text model that has JSON mode enabled. I suggest watching this demo for a (maybe) wow factor and the results on iOS 17 using NavAIGuide, a mobile and web navigational agent framework for LLMs:

Francesco

30,834 views • 2 years ago

For the first time we are fundamentally changing how humans can collaborate with ChatGPT since it launched two years ago. We’re introducing canvas, a new interface for working with ChatGPT on writing and coding projects that go beyond simple chat. Product and model features: 1/ Ask for in-line feedback. With canvas, ChatGPT can better understand the context of what you’re trying to accomplish. You can highlight specific sections to indicate exactly what you want ChatGPT to focus on. Like a copy editor or code reviewer, it can give in-line feedback and suggestions with the entire project in mind. 2/ Directly edit the model's output and select a specific area for targeted editing. You control your creative work on canvas. You can directly edit text or code. 3/ Menu of shortcuts. There’s a menu of shortcuts for you to ask ChatGPT to adjust writing length, debug your code, and quickly perform other useful actions. You can also restore previous versions of your work by using the back button in canvas. 4/ Use search with canvas for research writing! As we are moving towards the new paradigm of reasoning we are fundamentally evolving the chat interface into a more collaborative human-AI interaction. Today you can say “browse / use browsing to find XYZ on the internet and write a report in canvas”

For the first time we are fundamentally changing how humans can collaborate with ChatGPT since it launched two years ago. We’re introducing canvas, a new interface for working with ChatGPT on writing and coding projects that go beyond simple chat. Product and model features: 1/ Ask for in-line feedback. With canvas, ChatGPT can better understand the context of what you’re trying to accomplish. You can highlight specific sections to indicate exactly what you want ChatGPT to focus on. Like a copy editor or code reviewer, it can give in-line feedback and suggestions with the entire project in mind. 2/ Directly edit the model's output and select a specific area for targeted editing. You control your creative work on canvas. You can directly edit text or code. 3/ Menu of shortcuts. There’s a menu of shortcuts for you to ask ChatGPT to adjust writing length, debug your code, and quickly perform other useful actions. You can also restore previous versions of your work by using the back button in canvas. 4/ Use search with canvas for research writing! As we are moving towards the new paradigm of reasoning we are fundamentally evolving the chat interface into a more collaborative human-AI interaction. Today you can say “browse / use browsing to find XYZ on the internet and write a report in canvas”

Karina

845,887 views • 2 years ago

How can LLMs help improve local government services? This is the end of “Please listen closely as our menu options may have changed”: Lizzie Siegle’s team built a superset of San Francisco’s 311 service - which routes all residents through a massive, confusing and slow phone tree - all replaced by a simple natural language phone call with an AI agent as first line of response. voice to text, text to voice, all in a one day hackathon. Para Español, no necesita oprima numero dos.

How can LLMs help improve local government services? This is the end of “Please listen closely as our menu options may have changed”: Lizzie Siegle’s team built a superset of San Francisco’s 311 service - which routes all residents through a massive, confusing and slow phone tree - all replaced by a simple natural language phone call with an AI agent as first line of response. voice to text, text to voice, all in a one day hackathon. Para Español, no necesita oprima numero dos.

swyx

135,247 views • 2 years ago

JUST IN: Google releases Gemini 1.5, a powerful MoE model. It's a huge breakthrough. The model has the longest context window ever seen: 1 million tokens. It can process 1 hour of video, 11 hours of audio, 30,000 lines of code, or 700,000 words in a single prompt. When tested on text, code, image, audio and video evaluations, 1.5 Pro outperforms 1.0 Pro on 87% of the benchmarks used for developing LLMs. You can can sign up in AI Studio to try it out.

JUST IN: Google releases Gemini 1.5, a powerful MoE model. It's a huge breakthrough. The model has the longest context window ever seen: 1 million tokens. It can process 1 hour of video, 11 hours of audio, 30,000 lines of code, or 700,000 words in a single prompt. When tested on text, code, image, audio and video evaluations, 1.5 Pro outperforms 1.0 Pro on 87% of the benchmarks used for developing LLMs. You can can sign up in AI Studio to try it out.

Lior Alexander

83,432 views • 2 years ago

What would "vibe editing" look like. Well, let's take a look shall we. This is Descript, which was first to release a product that uses AI agents to edit your videos. "The Cursor for Video Editing" they say. And it is. I think by the end of 2025 it will feel alot like Cursor for Video Editing (it doesn't yet). I made a prediction a few months ago stating that Descript is best suited to win the Vibe Editing battle because their text based editor is elite. Both Adobe and Capcut have tried to their text based editor, and they both kinda suck. There are other companies throwing their hat in the ring, and I think these companies will need to move FAST and raise a lot of money to compete. Ponder William Wang (Clova) Who else? No matter what, i'm hyped for what it can do for our content team.

What would "vibe editing" look like. Well, let's take a look shall we. This is Descript, which was first to release a product that uses AI agents to edit your videos. "The Cursor for Video Editing" they say. And it is. I think by the end of 2025 it will feel alot like Cursor for Video Editing (it doesn't yet). I made a prediction a few months ago stating that Descript is best suited to win the Vibe Editing battle because their text based editor is elite. Both Adobe and Capcut have tried to their text based editor, and they both kinda suck. There are other companies throwing their hat in the ring, and I think these companies will need to move FAST and raise a lot of money to compete. Ponder William Wang (Clova) Who else? No matter what, i'm hyped for what it can do for our content team.

Riley Brown

29,041 views • 1 year ago

Instead of watching 1 hour of Netflix today, watch this Stanford lecture by ex-GoogleBrain & OpenAI engineers. This is the best explanation of how LLMs like ChatGPT & Claude actually work, and how to unlock 100% of their potential. Worth watching whether you're a senior AI engineer or just taking your first steps in AI. I took the key ideas and turned them into a practical guide for getting 100% out of AI. You can find it below with ready-to-copy prompts and solutions.

Instead of watching 1 hour of Netflix today, watch this Stanford lecture by ex-GoogleBrain & OpenAI engineers. This is the best explanation of how LLMs like ChatGPT & Claude actually work, and how to unlock 100% of their potential. Worth watching whether you're a senior AI engineer or just taking your first steps in AI. I took the key ideas and turned them into a practical guide for getting 100% out of AI. You can find it below with ready-to-copy prompts and solutions.

Dhruv kumar

58,100 views • 27 days ago

Instead of watching 1 hour of Netflix today, watch this Stanford lecture by ex-GoogleBrain & OpenAI engineers. This is the best explanation of how LLMs like ChatGPT & Claude actually work, and how to unlock 100% of their potential. Worth watching whether you're a senior AI engineer or just taking your first steps in AI. I took the key ideas and turned them into a practical guide for getting 100% out of AI. You can find it below with ready-to-copy prompts and solutions.

Instead of watching 1 hour of Netflix today, watch this Stanford lecture by ex-GoogleBrain & OpenAI engineers. This is the best explanation of how LLMs like ChatGPT & Claude actually work, and how to unlock 100% of their potential. Worth watching whether you're a senior AI engineer or just taking your first steps in AI. I took the key ideas and turned them into a practical guide for getting 100% out of AI. You can find it below with ready-to-copy prompts and solutions.

Rohit

72,695 views • 1 month ago