正在加载视频...

视频加载失败

返回首页

My new app is out !! ✨The Common Crawl Pipeline Creator ✨ Create your pipeline easily: ✔Run Text Extraction✂️ ✔Define Language Filters🌐 ✔Customize text quality💯 ✔See Live Results👀 ✔Get Python code 🐍 Based on famous LLM research like Gopher, C4 or FineWeb

lhoestq's profile picture

Quentin Lhoest 🤗

4,772 subscribers

15,001 次观看 • 1 年前 •via X (Twitter)

科学技术教育
LIVE

Anya Rossi• Live Now

Private livecam show

7 条评论

Quentin Lhoest 🤗 的头像
Quentin Lhoest 🤗1 年前

Keep an eye on the data you are filtering out, you don't want to discard interesting quality & diverse data ! You can also check how much data is filtered out at each step

Quentin Lhoest 🤗 的头像
Quentin Lhoest 🤗1 年前

The python code is based on the `datatrove` library that was used to build the FineWeb dataset, which has a text quality so good it makes training LLMs faster !

Quentin Lhoest 🤗 的头像
Quentin Lhoest 🤗1 年前

Try the app by yourself !

Sinclair Wang 的头像
Sinclair Wang1 年前

@qlhoest awesome app!

hannah 的头像
hannah1 年前

@qlhoest this is so cool!

Giocobon 的头像
Giocobon1 年前

@qlhoest Amazing job! What is the aim of the code line "partial(increment_num_warc_samples, num_warc_samples_per_doc=2000 / 1687)" in the app code? I mean, why 2000 / 1687 ?

Quentin Lhoest 🤗 的头像
Quentin Lhoest 🤗1 年前

I load the data from a cache from an intermediate step that has 300+ examples filtered out

相关视频

🧵 WAOC — The Meme of Human Connection A movement. A vibration. A new consciousness born on the blockchain. Today, WAOC steps into the world with one message: We Are One Connection. One planet. One humanity. One shared future. Let me explain what WAOC truly is. ↓ ⸻ 1 — ORIGIN We live in a world where people are divided by borders, beliefs, money and fear. But beneath all of this, there is one truth: **Everything is connected. Every human. Every action. Every vibration.** WAOC was created to remind the world of this connection — through culture, through meme energy, through the language of Web3. ⸻ 2 — WHAT IS WAOC? WAOC is not “just a coin”. WAOC is a consciousness meme — a symbol of peace, unity and awakening. •Born on BSC for trust & stability •Expanded to Solana / for speed, community & light •Powered by a global message: from code → to consciousness, from meme → to movement WAOC is the first meme that carries an actual philosophy. ⸻ 3 — THE VISION Our mission is simple: Unite people through art, culture, blockchain and collective awareness. WAOC will grow through: •Global airdrop events •Community-created art •Meme culture •Consciousness-inspired content •Worldwide movement of “One Connection” This is a meme with purpose. ⸻ 4 — HOW TO JOIN WAOC You don’t join WAOC with money. You join WAOC with connection. ✔ Buy / Hold / Support SOL ( Contract: 82gi7mybA1yHi56FcCC9wvTPzew5hsxP2wdHv4nYpump ✔ Join the global community Telegram: ✔ Become a creator Anyone can create WAOC memes, art, videos, quotes — every creation expands the connection. ⸻ 5 — AIRDROP (LIVE NOW) To welcome the first wave of global supporters: 🎁 5,000,000 WAOC AIRDROP To join: 1️⃣ Retweet 2️⃣ Comment: “I am One Connection” 3️⃣ Add your wallet address This is the easiest way to enter the WAOC movement. ⸻ 6 — THE FUTURE WAOC is building: 🌍 A global unity meme culture ✨ Consciousness-themed digital art 🧠 AI + WAOC experimental projects 🎨 Community-created visuals 🪐 Global peace & connection campaigns 🔥 Multi-chain presence This is only the beginning. ⸻ 7 — FINAL MESSAGE WAOC is more than a symbol. It is a reminder: **When we connect, we rise. When we unite, we evolve. We Are One Connection.** Drop a ❤️ if you feel the vibration. Retweet to spread the awakening. The movement has begun. #WAOC #WeAreOneConnection #Solana #pumpfun #MemeCoin #Airdrop #Consciousness #Unity #CryptoCommunity
1:12

🧵 WAOC — The Meme of Human Connection A movement. A vibration. A new consciousness born on the blockchain. Today, WAOC steps into the world with one message: We Are One Connection. One planet. One humanity. One shared future. Let me explain what WAOC truly is. ↓ ⸻ 1 — ORIGIN We live in a world where people are divided by borders, beliefs, money and fear. But beneath all of this, there is one truth: **Everything is connected. Every human. Every action. Every vibration.** WAOC was created to remind the world of this connection — through culture, through meme energy, through the language of Web3. ⸻ 2 — WHAT IS WAOC? WAOC is not “just a coin”. WAOC is a consciousness meme — a symbol of peace, unity and awakening. •Born on BSC for trust & stability •Expanded to Solana / for speed, community & light •Powered by a global message: from code → to consciousness, from meme → to movement WAOC is the first meme that carries an actual philosophy. ⸻ 3 — THE VISION Our mission is simple: Unite people through art, culture, blockchain and collective awareness. WAOC will grow through: •Global airdrop events •Community-created art •Meme culture •Consciousness-inspired content •Worldwide movement of “One Connection” This is a meme with purpose. ⸻ 4 — HOW TO JOIN WAOC You don’t join WAOC with money. You join WAOC with connection. ✔ Buy / Hold / Support SOL ( Contract: 82gi7mybA1yHi56FcCC9wvTPzew5hsxP2wdHv4nYpump ✔ Join the global community Telegram: ✔ Become a creator Anyone can create WAOC memes, art, videos, quotes — every creation expands the connection. ⸻ 5 — AIRDROP (LIVE NOW) To welcome the first wave of global supporters: 🎁 5,000,000 WAOC AIRDROP To join: 1️⃣ Retweet 2️⃣ Comment: “I am One Connection” 3️⃣ Add your wallet address This is the easiest way to enter the WAOC movement. ⸻ 6 — THE FUTURE WAOC is building: 🌍 A global unity meme culture ✨ Consciousness-themed digital art 🧠 AI + WAOC experimental projects 🎨 Community-created visuals 🪐 Global peace & connection campaigns 🔥 Multi-chain presence This is only the beginning. ⸻ 7 — FINAL MESSAGE WAOC is more than a symbol. It is a reminder: **When we connect, we rise. When we unite, we evolve. We Are One Connection.** Drop a ❤️ if you feel the vibration. Retweet to spread the awakening. The movement has begun. #WAOC #WeAreOneConnection #Solana #pumpfun #MemeCoin #Airdrop #Consciousness #Unity #CryptoCommunity

WAOC Network

100,350 次观看 • 8 个月前

🚨 ALERT AMERICA: HERE IS THE PROOF - ONE OF THE NINE FEDERAL REFUGEE CONTRACTORS ADMITTED THE ENTIRE RESETTLEMENT SCHEME ON CAMERA AMERICA HAS BEEN SOLD OUT! This 2021 video from Episcopal Migration Ministries (EMM), one of the nine federally funded refugee resettlement contractors, is one of the most revealing pieces of footage you will see. They were speaking openly at All Saints Church in Pasadena, assuming no one outside their activist circles was paying attention. But what they admitted is devastating. This wasn’t a sermon or charity. It was a full-blown political strategy briefing, filmed in a church, explaining how the U.S. government and religious NGOs work together to: 📍 dramatically expand resettlement pipelines 📍 pressure Congress to change immigration laws 📍 mobilize churches as pro-immigration activist armies 📍 build permanent support networks for arrivals 📍 redefine “refugee” to include economic & climate migrants 📍 overload the system by design 📍 coordinate lobbying directly with Washington And they said all of it out loud - we are funding our own demise! 1. They admit they are an official U.S. government partner Brother Chris McNabb of EMM states: “EMM is a ministry of the Episcopal Church and one of nine national agencies responsible for resettling refugees in the United States in partnership with the government.” This is not charity. They are federal contractors with a financial quota. More refugees = more money. It’s that simple. This is state-funded demographic policy laundered through churches. 2. They confessed the scale: from 18,000 refugees → to 4,000 Afghans PER WEEK McNabb brags: “In 2020 all nine agencies resettled about 18,000 refugees. We are now resettling 4,000 Afghans a week.” This was a mass importation pipeline, not a humanitarian drip. He openly says they had to “staff up significantly” because the pipeline “exploded overnight.” 3. They admit DHS used “humanitarian parole” to bypass the legal process McNabb explains Afghans were: “Paroled in by the Department of Homeland Security… because they did not have time to wait for the immigration system to operate.” Translation: ➡ Normal vetting was bypassed. ➡ The legal process was circumvented. ➡ Tens of thousands were rushed in without proper screening. Congress still hasn’t adjusted their status, but the arrivals are here permanently. 4. They train congregations to take in asylum seekers and parolees — not just refugees Through the Neighbor to Neighbor program, churches are trained to: 📍 provide physical & emotional support 📍 provide transportation 📍 help navigate legal processes 📍 sponsor asylum seekers & Afghan parolees Teams of 5–10 church members take over the roles official agencies would normally handle. This decentralizes the pipeline and embeds it inside communities. 5. They push churches to DEFUND ICE Rector Mike Kinman proudly declares: “We are strong advocates for the complete defunding of ICE.” So you have: 📍 a federally connected resettlement contractor 📍 working with the State Department 📍 using a church pulpit to demand the abolition of ICE This isn’t ministry. This is open-borders political warfare. 6. They openly plan to expand “refugee” to include climate & economic migrants When asked what happens to migrants who don’t meet asylum criteria, McNabb says: “We will begin seeing more and more climate refugees… more folks who come for economic reasons. We are preparing for that.” They are planning the next wave, economic & climate migrants, long before Congress approves anything. 7. They reveal the Episcopal Church has a lobbying arm on Capitol Hill They talk about OGR - the Office of Government Relations - their official lobbying office in the “God Box” on Capitol Hill. From the stage they instruct congregants to: 📍 call Congress 📍 pressure senators 📍 demand the Afghan Adjustment Act 📍 expand asylum categories 📍 increase refugee admissions 📍 secure more funding for EMM This is organized political lobbying coordinated from the church pews. 8. They admit they operate 11 affiliate offices nationwide, including in red states When asked about IRIS in Los Angeles, McNabb says: “That’s one of our 11 affiliates.” They place arrivals everywhere: Los Angeles, Seattle, Austin, Miami, and beyond. Red states are the main prize. And their “interfaith” branding? A manipulative façade - a left-wing coalition built to push anti-Western, anti-Christian, and anti-Jewish policy through religious cover. 9. They tell congregations to serve undocumented immigrants, too Kinman boasts All Saints is a “sanctuary church” ready to support: 📍 undocumented immigrants 📍 asylum seekers 📍 Afghan parolees 📍 migrants with “various statuses” This goes far beyond refugees — this is a parallel, extra-legal system. 10. They end by admitting the goal is a permanent, embedded migration infrastructure McNabb says this is just: “the beginning of a relationship.” The goal is permanent community integration — meaning once these pipelines enter your town, they never leave. 🚨 WHY THIS MATTERS, AMERICA This is just ONE of the nine federal contractors. In one hour, at one church, they admitted: ✔ their formal partnership with the U.S. government ✔ the massive scale of Afghan importation ✔ bypassing normal processes through parole ✔ pushing to abolish ICE ✔ expansion to climate & economic migrants ✔ their professional lobbying arm ✔ their nationwide network ✔ their plan for permanent demographic restructuring If this is what they say on camera, imagine what they say off camera. Americans deserve to see this video, understand how this system truly works, and decide whether they consent to having their communities transformed, with their tax dollars, without their input. Any politician - Republican or Democrat - who continues funding this racket, this federally backed demographic-engineering pipeline that European leaders openly describe as human-trafficker behavior — WILL be held accountable. If you fund it, defend it, or stay silent while your voters are kept in the dark, Laura Loomer and I will expose you. And WE will fight with everything we have to make sure you never hold office again. No one is going to fight like Laura and me to protect American communities from turning into mini-Islamic states!
23:14

🚨 ALERT AMERICA: HERE IS THE PROOF - ONE OF THE NINE FEDERAL REFUGEE CONTRACTORS ADMITTED THE ENTIRE RESETTLEMENT SCHEME ON CAMERA AMERICA HAS BEEN SOLD OUT! This 2021 video from Episcopal Migration Ministries (EMM), one of the nine federally funded refugee resettlement contractors, is one of the most revealing pieces of footage you will see. They were speaking openly at All Saints Church in Pasadena, assuming no one outside their activist circles was paying attention. But what they admitted is devastating. This wasn’t a sermon or charity. It was a full-blown political strategy briefing, filmed in a church, explaining how the U.S. government and religious NGOs work together to: 📍 dramatically expand resettlement pipelines 📍 pressure Congress to change immigration laws 📍 mobilize churches as pro-immigration activist armies 📍 build permanent support networks for arrivals 📍 redefine “refugee” to include economic & climate migrants 📍 overload the system by design 📍 coordinate lobbying directly with Washington And they said all of it out loud - we are funding our own demise! 1. They admit they are an official U.S. government partner Brother Chris McNabb of EMM states: “EMM is a ministry of the Episcopal Church and one of nine national agencies responsible for resettling refugees in the United States in partnership with the government.” This is not charity. They are federal contractors with a financial quota. More refugees = more money. It’s that simple. This is state-funded demographic policy laundered through churches. 2. They confessed the scale: from 18,000 refugees → to 4,000 Afghans PER WEEK McNabb brags: “In 2020 all nine agencies resettled about 18,000 refugees. We are now resettling 4,000 Afghans a week.” This was a mass importation pipeline, not a humanitarian drip. He openly says they had to “staff up significantly” because the pipeline “exploded overnight.” 3. They admit DHS used “humanitarian parole” to bypass the legal process McNabb explains Afghans were: “Paroled in by the Department of Homeland Security… because they did not have time to wait for the immigration system to operate.” Translation: ➡ Normal vetting was bypassed. ➡ The legal process was circumvented. ➡ Tens of thousands were rushed in without proper screening. Congress still hasn’t adjusted their status, but the arrivals are here permanently. 4. They train congregations to take in asylum seekers and parolees — not just refugees Through the Neighbor to Neighbor program, churches are trained to: 📍 provide physical & emotional support 📍 provide transportation 📍 help navigate legal processes 📍 sponsor asylum seekers & Afghan parolees Teams of 5–10 church members take over the roles official agencies would normally handle. This decentralizes the pipeline and embeds it inside communities. 5. They push churches to DEFUND ICE Rector Mike Kinman proudly declares: “We are strong advocates for the complete defunding of ICE.” So you have: 📍 a federally connected resettlement contractor 📍 working with the State Department 📍 using a church pulpit to demand the abolition of ICE This isn’t ministry. This is open-borders political warfare. 6. They openly plan to expand “refugee” to include climate & economic migrants When asked what happens to migrants who don’t meet asylum criteria, McNabb says: “We will begin seeing more and more climate refugees… more folks who come for economic reasons. We are preparing for that.” They are planning the next wave, economic & climate migrants, long before Congress approves anything. 7. They reveal the Episcopal Church has a lobbying arm on Capitol Hill They talk about OGR - the Office of Government Relations - their official lobbying office in the “God Box” on Capitol Hill. From the stage they instruct congregants to: 📍 call Congress 📍 pressure senators 📍 demand the Afghan Adjustment Act 📍 expand asylum categories 📍 increase refugee admissions 📍 secure more funding for EMM This is organized political lobbying coordinated from the church pews. 8. They admit they operate 11 affiliate offices nationwide, including in red states When asked about IRIS in Los Angeles, McNabb says: “That’s one of our 11 affiliates.” They place arrivals everywhere: Los Angeles, Seattle, Austin, Miami, and beyond. Red states are the main prize. And their “interfaith” branding? A manipulative façade - a left-wing coalition built to push anti-Western, anti-Christian, and anti-Jewish policy through religious cover. 9. They tell congregations to serve undocumented immigrants, too Kinman boasts All Saints is a “sanctuary church” ready to support: 📍 undocumented immigrants 📍 asylum seekers 📍 Afghan parolees 📍 migrants with “various statuses” This goes far beyond refugees — this is a parallel, extra-legal system. 10. They end by admitting the goal is a permanent, embedded migration infrastructure McNabb says this is just: “the beginning of a relationship.” The goal is permanent community integration — meaning once these pipelines enter your town, they never leave. 🚨 WHY THIS MATTERS, AMERICA This is just ONE of the nine federal contractors. In one hour, at one church, they admitted: ✔ their formal partnership with the U.S. government ✔ the massive scale of Afghan importation ✔ bypassing normal processes through parole ✔ pushing to abolish ICE ✔ expansion to climate & economic migrants ✔ their professional lobbying arm ✔ their nationwide network ✔ their plan for permanent demographic restructuring If this is what they say on camera, imagine what they say off camera. Americans deserve to see this video, understand how this system truly works, and decide whether they consent to having their communities transformed, with their tax dollars, without their input. Any politician - Republican or Democrat - who continues funding this racket, this federally backed demographic-engineering pipeline that European leaders openly describe as human-trafficker behavior — WILL be held accountable. If you fund it, defend it, or stay silent while your voters are kept in the dark, Laura Loomer and I will expose you. And WE will fight with everything we have to make sure you never hold office again. No one is going to fight like Laura and me to protect American communities from turning into mini-Islamic states!

Amy Mek

229,978 次观看 • 7 个月前

The keyboard on iOS is the worst I have ever used in comparison to what’s available on Android. When AI voice dictation came out, it quickly became my primary input interface on both desktop and mobile. Though, on mobile it was quite limited to specific apps, cuz in many cases it is not that good or completely missing. Around a month ago, it finally changed when I had a chance to beta test Typeless keyboard for iOS. There, it can be synced between mobile and desktop apps, so I would have access to all my configurations as well as to the dictation history. It comes with a seamless way to switch between typing and dictation and a wave animation (quite an important part), so you can see if it is actually recording what you say. Typeless enables vibe coding in tools like Claude Code or AI Studio from mobile. A custom dictionary saves loads of time cuz you don’t have to correct all your abbreviations that other voice dictation solutions misspell. You can easily access dictated text on your other devices, and it also saves your dictated text from being accidentally lost (a common issue with the dictation feature on other apps). And you can always see how much time you managed to save on the main UI. “Time saved” * “Your hourly rate” is the ROI. AI is money 👀 * Sponsored post
0:25

The keyboard on iOS is the worst I have ever used in comparison to what’s available on Android. When AI voice dictation came out, it quickly became my primary input interface on both desktop and mobile. Though, on mobile it was quite limited to specific apps, cuz in many cases it is not that good or completely missing. Around a month ago, it finally changed when I had a chance to beta test Typeless keyboard for iOS. There, it can be synced between mobile and desktop apps, so I would have access to all my configurations as well as to the dictation history. It comes with a seamless way to switch between typing and dictation and a wave animation (quite an important part), so you can see if it is actually recording what you say. Typeless enables vibe coding in tools like Claude Code or AI Studio from mobile. A custom dictionary saves loads of time cuz you don’t have to correct all your abbreviations that other voice dictation solutions misspell. You can easily access dictated text on your other devices, and it also saves your dictated text from being accidentally lost (a common issue with the dictation feature on other apps). And you can always see how much time you managed to save on the main UI. “Time saved” * “Your hourly rate” is the ROI. AI is money 👀 * Sponsored post

🚨 AI News | TestingCatalog

14,057 次观看 • 6 个月前

Claude Code can now make full videos from your terminal.. Not slideshows. Not text on screen.. Actual motion graphics with animations, transitions, custom photos, and background music. [ SHARED A TUTORIAL BELOW EDITED WITH THIS SETUP IN JUST 5mins ] ▫️Here's the setup: Claude Code + Remotion Remotion is a React based framework that renders video programmatically. You describe what you want in plain English, Claude writes the React components and Remotion renders it into a real MP4. What you can actually do with this: > Generate 9:16 vertical videos for TikTok / Reels / Shorts > Add animated text with viral hooks and safe zones > Pull live web screenshots directly into your scenes using Chrome MCP > Fact-check your content in real time with Perplexity MCP > Drop in your own photos and background music > Edit existing talking-head footage cut bloopers, add captions > Schedule posts to your socials straight from the terminal ▫️How to set it up (takes 5 minutes) : > Make sure you have Node.js installed ( node -v to check ) > Create a new Remotion project: npx create-video@latest Pick the Blank template, enable TailwindCSS, and install the Skills package when prompted. > Install dependencies: cd my-video npm install > Start the preview server: npm run dev > Open Claude Code in the same project folder: cd my-video claude That's it. You can now prompt videos in plain English. If you already have a Remotion project, just add the skill directly: npx skills add remotion-dev/skills This drops a SKILL.md into your project that gives Claude expert knowledge of Remotion.. animations, compositions, captions, assets, 3D content everything. Example prompt you can steal: "Create a 30-second 9:16 vertical video about the top 3 AI tools this week. Use animated text with a hook in the first 2 seconds. Add smooth transitions between scenes. Keep text in the safe zone for TikTok. Use a dark tech aesthetic with blue accent colors." Claude writes all the React code, renders a preview, you tweak with natural language, and export when ready. The crazy part is this whole pipeline is local, free (minus your Claude sub), and you never open a video editor. imo this kills CapCut for anyone making info-style content. You describe the video in English and get back a rendered MP4. try it now.
2:01

Claude Code can now make full videos from your terminal.. Not slideshows. Not text on screen.. Actual motion graphics with animations, transitions, custom photos, and background music. [ SHARED A TUTORIAL BELOW EDITED WITH THIS SETUP IN JUST 5mins ] ▫️Here's the setup: Claude Code + Remotion Remotion is a React based framework that renders video programmatically. You describe what you want in plain English, Claude writes the React components and Remotion renders it into a real MP4. What you can actually do with this: > Generate 9:16 vertical videos for TikTok / Reels / Shorts > Add animated text with viral hooks and safe zones > Pull live web screenshots directly into your scenes using Chrome MCP > Fact-check your content in real time with Perplexity MCP > Drop in your own photos and background music > Edit existing talking-head footage cut bloopers, add captions > Schedule posts to your socials straight from the terminal ▫️How to set it up (takes 5 minutes) : > Make sure you have Node.js installed ( node -v to check ) > Create a new Remotion project: npx create-video@latest Pick the Blank template, enable TailwindCSS, and install the Skills package when prompted. > Install dependencies: cd my-video npm install > Start the preview server: npm run dev > Open Claude Code in the same project folder: cd my-video claude That's it. You can now prompt videos in plain English. If you already have a Remotion project, just add the skill directly: npx skills add remotion-dev/skills This drops a SKILL.md into your project that gives Claude expert knowledge of Remotion.. animations, compositions, captions, assets, 3D content everything. Example prompt you can steal: "Create a 30-second 9:16 vertical video about the top 3 AI tools this week. Use animated text with a hook in the first 2 seconds. Add smooth transitions between scenes. Keep text in the safe zone for TikTok. Use a dark tech aesthetic with blue accent colors." Claude writes all the React code, renders a preview, you tweak with natural language, and export when ready. The crazy part is this whole pipeline is local, free (minus your Claude sub), and you never open a video editor. imo this kills CapCut for anyone making info-style content. You describe the video in English and get back a rendered MP4. try it now.

Axel Bitblaze 🪓

31,250 次观看 • 3 个月前

Web scraping will never be the same. (100% open-source visual search at scale) PixelRAG is a retrieval system that skips HTML parsing completely. Instead of scraping a page into text and embedding chunks, it screenshots the page and retrieves the image. A vision-language model reads the answer straight off the pixels. Why that matters: parsing is where web RAG quietly loses information. - A single HTML-to-text parser can drop 40%+ of a page. - Tables, charts, and layout get flattened or thrown out. - Swapping parsers alone can move accuracy ~10 points on the same docs. PixelRAG indexes the page a person actually sees. The team built a visual index of all of Wikipedia, 30M+ screenshots, and it still beats the strongest text RAG baseline by 18.1% on text-only QA. The repo also ships a Claude Code plugin that gives Claude eyes. It lets Claude screenshot any URL and read the rendered page instead of scraping the DOM. So you can hand it a live page, an arXiv paper, or your local site and ask what it actually looks like. One setup script. No MCP server, no backend. How the pipeline works: - Renders each document (web, PDF, image) to image tiles. - Embeds them with Qwen3-VL-Embedding, LoRA fine-tuned on screenshots. - Builds a FAISS index and serves a search API. A stronger reader model lifts accuracy with no re-indexing, since the index is just pixels. Everything is open-source under Apache-2.0. GitHub repo: Talking about RAG, I recently wrote an article on a new approach that makes retrieval much more efficient by cutting corpus size by 40x, reducing tokens per query by 3x, and improving vector search relevance by 2.3x. The article is quoted below.
0:55

Web scraping will never be the same. (100% open-source visual search at scale) PixelRAG is a retrieval system that skips HTML parsing completely. Instead of scraping a page into text and embedding chunks, it screenshots the page and retrieves the image. A vision-language model reads the answer straight off the pixels. Why that matters: parsing is where web RAG quietly loses information. - A single HTML-to-text parser can drop 40%+ of a page. - Tables, charts, and layout get flattened or thrown out. - Swapping parsers alone can move accuracy ~10 points on the same docs. PixelRAG indexes the page a person actually sees. The team built a visual index of all of Wikipedia, 30M+ screenshots, and it still beats the strongest text RAG baseline by 18.1% on text-only QA. The repo also ships a Claude Code plugin that gives Claude eyes. It lets Claude screenshot any URL and read the rendered page instead of scraping the DOM. So you can hand it a live page, an arXiv paper, or your local site and ask what it actually looks like. One setup script. No MCP server, no backend. How the pipeline works: - Renders each document (web, PDF, image) to image tiles. - Embeds them with Qwen3-VL-Embedding, LoRA fine-tuned on screenshots. - Builds a FAISS index and serves a search API. A stronger reader model lifts accuracy with no re-indexing, since the index is just pixels. Everything is open-source under Apache-2.0. GitHub repo: Talking about RAG, I recently wrote an article on a new approach that makes retrieval much more efficient by cutting corpus size by 40x, reducing tokens per query by 3x, and improving vector search relevance by 2.3x. The article is quoted below.

Akshay 🚀

936,450 次观看 • 24 天前

Crazy... I just found a super cheap way to code with AI Tried GLM 4.5 on Claude Code and I’m really impressed! Prompt: “create an interactive python app where users can track, manage or analyze data” And, It came up with: • Dashboard - a clean interface to see all your data at a glance • Datasets - manage multiple collections of data easily • Data entries - add, edit, and organize your records without hassle • Analysis - get insights and summaries directly from your data What I observed: Everything worked smoothly, GLM 4.5 is definitely one of the best open-source models for coding, I’ll keep using it. Check it out now: And the best part: super affordable coding plans! • Lite - $3/month (promo). ~120 prompts every 5 hours. Works with Claude Code, Roo Code, Cline, Kilo Code, and more. • Pro - $15/month (promo). 5× more usage than Lite. Faster. Includes image/video tools and web search. • Max - $30/month (promo). 4× more usage than Pro. Guaranteed performance. Early access to new features. Bonus - Use my referral link to get an extra 10% off on top of these promo prices! Video demo 👇
0:33

Crazy... I just found a super cheap way to code with AI Tried GLM 4.5 on Claude Code and I’m really impressed! Prompt: “create an interactive python app where users can track, manage or analyze data” And, It came up with: • Dashboard - a clean interface to see all your data at a glance • Datasets - manage multiple collections of data easily • Data entries - add, edit, and organize your records without hassle • Analysis - get insights and summaries directly from your data What I observed: Everything worked smoothly, GLM 4.5 is definitely one of the best open-source models for coding, I’ll keep using it. Check it out now: And the best part: super affordable coding plans! • Lite - $3/month (promo). ~120 prompts every 5 hours. Works with Claude Code, Roo Code, Cline, Kilo Code, and more. • Pro - $15/month (promo). 5× more usage than Lite. Faster. Includes image/video tools and web search. • Max - $30/month (promo). 4× more usage than Pro. Guaranteed performance. Early access to new features. Bonus - Use my referral link to get an extra 10% off on top of these promo prices! Video demo 👇

Chidanand Tripathi

83,854 次观看 • 9 个月前

LLM Wikis are being slept on. I argue that creating knowledge bases with LLMs or coding agents is one of the most valuable applications of AI today. It's about being intentional in building and scaling your intelligence stack. To showcase this, I wanted to share an LLM Wiki I have built over the last couple of months. It's called PaperWiki, and I use it across all my research workflows, along with my research agents. In fact, I also use it to curate papers I share with my communities, newsletter, and on X. The PaperWiki is updated regularly with automations, so I basically have agents on a loop maintaining it. All the entries are ingested from different sources and stored in a vault (Obsidian) and further indexed using qmd. And then further presented via an HTML artifact. So all of it is easily accessible to all my agents and easily searchable through full-text search and rich semantic search. The structure of the wiki has proven significantly useful to start interesting and exciting cutting-edge research projects with my research agents (from building tiny and more efficient gpt/difussion llms to building out SoTA harnesses and memory systems). It turns out that agents love markdown files and can more easily navigate the papers given the rich metadata structure of the wiki. I am just getting started on this, but it's clear to me that we should all be experimenting with LLM Wikis. Here's why: Building LLM knowledge bases gets you into the habit of leveraging AI outputs in all kinds of creative ways. It's the good kind of tokenmaxxing we should all be pushing for. LLM Wikis can be maintained automatically in a loop. I use an automation that updates the wiki every day based on papers I curate. The curation is another automation I run in a loop (with a bit of human in the loop), so I get to build on all my previous knowledge and expertise, and all of it compounds the deeper the integration/layers. One interesting result of this process is that I feel like I can better spot high-quality papers and remove noise more easily. Social media could never solve that. And most paper aggregators use metrics I simply don't trust. I like that agents can help with the noise vs. signal problem. This is important for research. Lots of people consider agents to produce mostly slop. But it doesn't have to be that way. Careful curations, prompts, automations, verifiers, and human-in-the-loop can produce some astonishing results. And you really don't need frontier models for this. I use a combination of frontier models (opus-4.8) and open-weight models (deepseek-v4-flash) to maintain this. An exciting future work (we are working on this DAIR.AI) is to tune specialized models on top of this to allow LLMs to quickly understand cutting-edge research ideas and can better conceptualize research strategies that further accelerate scientific research agents. I plan to open-source a bunch of this work, including the artifact, but this is currently work in progress, and I was excited to share some thoughts as I continue working on it. Sharing more as I go. Stay tuned!
0:27

LLM Wikis are being slept on. I argue that creating knowledge bases with LLMs or coding agents is one of the most valuable applications of AI today. It's about being intentional in building and scaling your intelligence stack. To showcase this, I wanted to share an LLM Wiki I have built over the last couple of months. It's called PaperWiki, and I use it across all my research workflows, along with my research agents. In fact, I also use it to curate papers I share with my communities, newsletter, and on X. The PaperWiki is updated regularly with automations, so I basically have agents on a loop maintaining it. All the entries are ingested from different sources and stored in a vault (Obsidian) and further indexed using qmd. And then further presented via an HTML artifact. So all of it is easily accessible to all my agents and easily searchable through full-text search and rich semantic search. The structure of the wiki has proven significantly useful to start interesting and exciting cutting-edge research projects with my research agents (from building tiny and more efficient gpt/difussion llms to building out SoTA harnesses and memory systems). It turns out that agents love markdown files and can more easily navigate the papers given the rich metadata structure of the wiki. I am just getting started on this, but it's clear to me that we should all be experimenting with LLM Wikis. Here's why: Building LLM knowledge bases gets you into the habit of leveraging AI outputs in all kinds of creative ways. It's the good kind of tokenmaxxing we should all be pushing for. LLM Wikis can be maintained automatically in a loop. I use an automation that updates the wiki every day based on papers I curate. The curation is another automation I run in a loop (with a bit of human in the loop), so I get to build on all my previous knowledge and expertise, and all of it compounds the deeper the integration/layers. One interesting result of this process is that I feel like I can better spot high-quality papers and remove noise more easily. Social media could never solve that. And most paper aggregators use metrics I simply don't trust. I like that agents can help with the noise vs. signal problem. This is important for research. Lots of people consider agents to produce mostly slop. But it doesn't have to be that way. Careful curations, prompts, automations, verifiers, and human-in-the-loop can produce some astonishing results. And you really don't need frontier models for this. I use a combination of frontier models (opus-4.8) and open-weight models (deepseek-v4-flash) to maintain this. An exciting future work (we are working on this DAIR.AI) is to tune specialized models on top of this to allow LLMs to quickly understand cutting-edge research ideas and can better conceptualize research strategies that further accelerate scientific research agents. I plan to open-source a bunch of this work, including the artifact, but this is currently work in progress, and I was excited to share some thoughts as I continue working on it. Sharing more as I go. Stay tuned!

elvis

54,713 次观看 • 12 天前

How I have Claude automate anything I do on my computer just by showing it: I use the free Clips app and hit record to start recording. Then I just do the action - like navigating to Rippling and approving any unapproved time off - and I’ll often just talk out loud as I do it explaining any add’l details like I would when showing a human. The Clips app captures everything on your screen and everything you say, and transforms it into a format Claude can “see and hear”. You'll then get a link that you can easily share with Claude (or humans too). I’ll paste it to Claude and say something like, "automate this for me every day at 8:00 AM." The link has special metadata that agents like Claude can discover. It exposes APIs to see the full transcript with timestamps and grab screenshots of what was on your screen at any of those timestamps. Once it's created the routine, you can go to the Routines tab in Claude Code desktop app. This also works with the CLI, Claude Cowork, Codex etc. You’ll see your new routine there, and you can just hit Run Now to have it start running it. Video attached shows this step by step. The Clips apps is free and open source - I’ll link to it and the source code in the thread.
1:54

How I have Claude automate anything I do on my computer just by showing it: I use the free Clips app and hit record to start recording. Then I just do the action - like navigating to Rippling and approving any unapproved time off - and I’ll often just talk out loud as I do it explaining any add’l details like I would when showing a human. The Clips app captures everything on your screen and everything you say, and transforms it into a format Claude can “see and hear”. You'll then get a link that you can easily share with Claude (or humans too). I’ll paste it to Claude and say something like, "automate this for me every day at 8:00 AM." The link has special metadata that agents like Claude can discover. It exposes APIs to see the full transcript with timestamps and grab screenshots of what was on your screen at any of those timestamps. Once it's created the routine, you can go to the Routines tab in Claude Code desktop app. This also works with the CLI, Claude Cowork, Codex etc. You’ll see your new routine there, and you can just hit Run Now to have it start running it. Video attached shows this step by step. The Clips apps is free and open source - I’ll link to it and the source code in the thread.

Steve (Builder.io)

25,140 次观看 • 15 天前

🚨 RED STATE TRAITORS: WHAT HAVE YOU DONE, UTAH!? Gov. Spencer Cox SOLD OUT Utah, and chose Afghan migrants over his own citizens Utah was supposed to be a conservative fortress — safe, stable, and insulated from the Biden-era mass-migration machine. Instead, Governor Spencer Cox threw open the gates and begged Joe Biden to dump unvetted Afghan migrants into Utah before a single Utahn was asked, warned, or consulted. This wasn’t charity or compassion by your "Conservative" Governor.... This was political ambition and ideological obedience, paid for by your tax dollars. 💥 COX PERSONALLY INVITED AFGHANS INTO UTAH In August 2021, as the Taliban seized control and U.S. intelligence admitted that real vetting was impossible, Spencer Cox wrote Biden a glowing letter: “Utah is eager to help.” “Utah has a long history of welcoming refugees.” “Send them here.” He volunteered YOUR state, YOUR cities, YOUR neighborhoods to absorb unknown, undocumented, unvetted Afghan nationals from a Taliban-controlled country known for jihad, anti-American hatred, and forged identities. Who asked him? Who voted for this? Who benefits? Not the citizens of Utah. 🔥 COX DIDN’T JUST ACCEPT THEM - HE FOUGHT FOR MORE Spencer Cox demanded Afghan migration into Utah. He: ✔ Created the “Afghan Community Fund” A partnership with Zions Bank, the World Trade Center Utah, and the state’s refugee networks to finance mass intake. ✔ Pressured Utahns to offer housing While Utah families were struggling with cost-of-living and housing shortages, Cox begged residents to “step up” and house refugees. ✔ Advocated for permanent residency In 2022, Cox co-signed a letter urging Congress to give Afghan arrivals permanent legal status before we knew who many of them were. ✔ Celebrated their arrival Six months after the airlift, he bragged on TV about 900 Afghan refugees already in Utah and held ceremonies honoring them on the House floor. All while Utahns were dealing with rising crime, homelessness, fentanyl, and inflation. This wasn’t a burden he carried. This was a victory lap. 🕌 AND HE DID ALL THIS WHILE MOSQUES MULTIPLY ACROSS YOUR STATE Utah, the state many believed was sheltered from Islamic political influence, is now seeing: Mosque expansions New Islamic centers Rapid demographic change Aggressive NGO-led migration campaigns And Cox celebrates it, framing it as “compassion” and “Utah values.” Utahns never voted for Islamic migration. Utahns never agreed to demographic transformation. Utahns never asked to bankroll a Taliban-era pipeline. Spencer Cox did it for them, not for you. 💰 BECAUSE REFUGEE RESETTLEMENT IS A BUSINESS. A BIG ONE Behind Cox stands an entire machine: The Utah Department of Workforce Services Asha Parekh - Director of Refugee Services (No wonder she was named Utah Business Woman of the Year and honored with this award in 2021 - REFUGEE RESETTLEMENT IS A BIG BUSINESS) NGO resettlement networks Interfaith groups Corporate donors Left-wing foundations Millions in federal and state funding flow to these networks every time Afghan arrivals land in Utah. Jobs. Grants. Housing contracts. NGO salaries. The migration industry gets rich. and YOU pay for it. Is this charity? No. It is importation for profit and importation for power. Utah keep a close eye on Asha Parekh and Governor Cox - this seems like a racket to me! ⚠️ UTAH, THIS IS YOUR WARNING Your “conservative” governor didn’t protect you. He didn’t defend your values. He didn’t consult your communities. He offered Utah up as a refugee hub. voluntarily. He fought for more Afghans. He celebrated their arrival. He pushed Congress to make them permanent. He poured state resources into resettling them. He honored them on the House floor. And now he’s rumored to be eyeing the presidency. A presidency for whom? Americans or Afghan migrants? Utah, do not forget that the man who is supposed to defend your state is building the very demographic pipeline that will transform it forever. This isn’t leadership or conservatism. This is replacement and betrayal by a so-called conservative!? PROTECT YOUR STATE, UTAH - YOU ARE BEING ISLAMIZED!
2:41

🚨 RED STATE TRAITORS: WHAT HAVE YOU DONE, UTAH!? Gov. Spencer Cox SOLD OUT Utah, and chose Afghan migrants over his own citizens Utah was supposed to be a conservative fortress — safe, stable, and insulated from the Biden-era mass-migration machine. Instead, Governor Spencer Cox threw open the gates and begged Joe Biden to dump unvetted Afghan migrants into Utah before a single Utahn was asked, warned, or consulted. This wasn’t charity or compassion by your "Conservative" Governor.... This was political ambition and ideological obedience, paid for by your tax dollars. 💥 COX PERSONALLY INVITED AFGHANS INTO UTAH In August 2021, as the Taliban seized control and U.S. intelligence admitted that real vetting was impossible, Spencer Cox wrote Biden a glowing letter: “Utah is eager to help.” “Utah has a long history of welcoming refugees.” “Send them here.” He volunteered YOUR state, YOUR cities, YOUR neighborhoods to absorb unknown, undocumented, unvetted Afghan nationals from a Taliban-controlled country known for jihad, anti-American hatred, and forged identities. Who asked him? Who voted for this? Who benefits? Not the citizens of Utah. 🔥 COX DIDN’T JUST ACCEPT THEM - HE FOUGHT FOR MORE Spencer Cox demanded Afghan migration into Utah. He: ✔ Created the “Afghan Community Fund” A partnership with Zions Bank, the World Trade Center Utah, and the state’s refugee networks to finance mass intake. ✔ Pressured Utahns to offer housing While Utah families were struggling with cost-of-living and housing shortages, Cox begged residents to “step up” and house refugees. ✔ Advocated for permanent residency In 2022, Cox co-signed a letter urging Congress to give Afghan arrivals permanent legal status before we knew who many of them were. ✔ Celebrated their arrival Six months after the airlift, he bragged on TV about 900 Afghan refugees already in Utah and held ceremonies honoring them on the House floor. All while Utahns were dealing with rising crime, homelessness, fentanyl, and inflation. This wasn’t a burden he carried. This was a victory lap. 🕌 AND HE DID ALL THIS WHILE MOSQUES MULTIPLY ACROSS YOUR STATE Utah, the state many believed was sheltered from Islamic political influence, is now seeing: Mosque expansions New Islamic centers Rapid demographic change Aggressive NGO-led migration campaigns And Cox celebrates it, framing it as “compassion” and “Utah values.” Utahns never voted for Islamic migration. Utahns never agreed to demographic transformation. Utahns never asked to bankroll a Taliban-era pipeline. Spencer Cox did it for them, not for you. 💰 BECAUSE REFUGEE RESETTLEMENT IS A BUSINESS. A BIG ONE Behind Cox stands an entire machine: The Utah Department of Workforce Services Asha Parekh - Director of Refugee Services (No wonder she was named Utah Business Woman of the Year and honored with this award in 2021 - REFUGEE RESETTLEMENT IS A BIG BUSINESS) NGO resettlement networks Interfaith groups Corporate donors Left-wing foundations Millions in federal and state funding flow to these networks every time Afghan arrivals land in Utah. Jobs. Grants. Housing contracts. NGO salaries. The migration industry gets rich. and YOU pay for it. Is this charity? No. It is importation for profit and importation for power. Utah keep a close eye on Asha Parekh and Governor Cox - this seems like a racket to me! ⚠️ UTAH, THIS IS YOUR WARNING Your “conservative” governor didn’t protect you. He didn’t defend your values. He didn’t consult your communities. He offered Utah up as a refugee hub. voluntarily. He fought for more Afghans. He celebrated their arrival. He pushed Congress to make them permanent. He poured state resources into resettling them. He honored them on the House floor. And now he’s rumored to be eyeing the presidency. A presidency for whom? Americans or Afghan migrants? Utah, do not forget that the man who is supposed to defend your state is building the very demographic pipeline that will transform it forever. This isn’t leadership or conservatism. This is replacement and betrayal by a so-called conservative!? PROTECT YOUR STATE, UTAH - YOU ARE BEING ISLAMIZED!

Amy Mek

348,442 次观看 • 7 个月前

🚨ANNOUNCEMENT! New Partnership + AI Coaching Offer from Anyman Fitness and Spren 🚨 I have a big announcement to make… For the last 18 months I’ve been working quietly behind the scenes on a BIG project… You may have seen me post about the iOS app “Spren” - a groundbreaking app that tests your body fat % using just your iPhone. The app has been 3rd party tested and is as accurate as a DEXA scan. If you’re into fitness, you know how wild that statement is. A DEXA scan is the gold standard for body fat testing (and they cost $200+ to get one). The Spren CEO (Jason Moore) reached out to me in 2023 and he had a proposal. He wanted to create an AI based coaching offer for people who tested their body fat % with Spren and were unhappy with the results. But he didn’t want it to be generic like Chat GPT. He wanted to model the AI after a coach with extensive experience working with the general public. For the last 18 months, we have been working behind the scenes building this offer, and it just got released to the public. The AI has been fully trained on my coaching strategies, teachings, preferences, and mindset/motivational style. Here’s how it works: • Send a text message to 734-499-1101 • If you’re outside of the US, send the text via What’s App • You’ll be sent a link to learn everything the AI can do for you • You get unlimited scans on the Spren iOS body fat testing app ($70 value) • You get a customized nutrition plan for your goals, and “AI me” will guide you on your journey You connect the app to your macro tracker and wearable. Every few weeks, you scan your body fat % again, and “I” will make adjustments as needed for you to make swift and rapid progress. The best part? I don’t take breaks. I’m always here for you. The “style” is all me… You need reminders to get your butt off the couch and go to the gym? I got you. You need done-for-you meal plans that hit your macros/calories exactly? I got you. You need recipe ideas? Workouts when traveling? Exercise substitutions? Supplementation questions? Or you just want to shoot the breeze? (Lol) I got you! I’m here, 24/7, in your back pocket. I’m thrilled to have a shared vision with Spren on this. To learn more about this new, cutting-edge offer, send a text to 734-499-1101 and check it out!
4:20

🚨ANNOUNCEMENT! New Partnership + AI Coaching Offer from Anyman Fitness and Spren 🚨 I have a big announcement to make… For the last 18 months I’ve been working quietly behind the scenes on a BIG project… You may have seen me post about the iOS app “Spren” - a groundbreaking app that tests your body fat % using just your iPhone. The app has been 3rd party tested and is as accurate as a DEXA scan. If you’re into fitness, you know how wild that statement is. A DEXA scan is the gold standard for body fat testing (and they cost $200+ to get one). The Spren CEO (Jason Moore) reached out to me in 2023 and he had a proposal. He wanted to create an AI based coaching offer for people who tested their body fat % with Spren and were unhappy with the results. But he didn’t want it to be generic like Chat GPT. He wanted to model the AI after a coach with extensive experience working with the general public. For the last 18 months, we have been working behind the scenes building this offer, and it just got released to the public. The AI has been fully trained on my coaching strategies, teachings, preferences, and mindset/motivational style. Here’s how it works: • Send a text message to 734-499-1101 • If you’re outside of the US, send the text via What’s App • You’ll be sent a link to learn everything the AI can do for you • You get unlimited scans on the Spren iOS body fat testing app ($70 value) • You get a customized nutrition plan for your goals, and “AI me” will guide you on your journey You connect the app to your macro tracker and wearable. Every few weeks, you scan your body fat % again, and “I” will make adjustments as needed for you to make swift and rapid progress. The best part? I don’t take breaks. I’m always here for you. The “style” is all me… You need reminders to get your butt off the couch and go to the gym? I got you. You need done-for-you meal plans that hit your macros/calories exactly? I got you. You need recipe ideas? Workouts when traveling? Exercise substitutions? Supplementation questions? Or you just want to shoot the breeze? (Lol) I got you! I’m here, 24/7, in your back pocket. I’m thrilled to have a shared vision with Spren on this. To learn more about this new, cutting-edge offer, send a text to 734-499-1101 and check it out!

Jason Helmes

15,819 次观看 • 1 年前

New Course: Post-training of LLMs Learn to post-train and customize an LLM in this short course, taught by Banghua Zhu, Assistant Professor at the University of Washington University of Washington, and co-founder of @NexusflowX. Training an LLM to follow instructions or answer questions has two key stages: pre-training and post-training. In pre-training, it learns to predict the next word or token from large amounts of unlabeled text. In post-training, it learns useful behaviors such as following instructions, tool use, and reasoning. Post-training transforms a general-purpose token predictor—trained on trillions of unlabeled text tokens—into an assistant that follows instructions and performs specific tasks. Because it is much cheaper than pre-training, it is practical for many more teams to incorporate post-training methods into their workflows than pre-training. In this course, you’ll learn three common post-training methods—Supervised Fine-Tuning (SFT), Direct Preference Optimization (DPO), and Online Reinforcement Learning (RL)—and how to use each one effectively. With SFT, you train the model on pairs of input and ideal output responses. With DPO, you provide both a preferred (chosen) and a less preferred (rejected) response and train the model to favor the preferred output. With RL, the model generates an output, receives a reward score based on human or automated feedback, and updates the model to improve performance. You’ll learn the basic concepts, common use cases, and principles for curating high-quality data for effective training. Through hands-on labs, you’ll download a pre-trained model from Hugging Face and post-train it using SFT, DPO, and RL to see how each technique shapes model behavior. In detail, you’ll: - Understand what post-training is, when to use it, and how it differs from pre-training. - Build an SFT pipeline to turn a base model into an instruct model. - Explore how DPO reshapes behavior by minimizing contrastive loss—penalizing poor responses and reinforcing preferred ones. - Implement a DPO pipeline to change the identity of a chat assistant. - Learn online RL methods such as Proximal Policy Optimization (PPO) and Group Relative Policy Optimization (GRPO), and how to design reward functions. - Train a model with GRPO to improve its math capabilities using a verifiable reward. Post-training is one of the most rapidly developing areas of LLM training. Whether you’re building a high-accuracy context-specific assistant, fine-tuning a model's tone, or improving task-specific accuracy, this course will give you experience with the most important techniques shaping how LLMs are post-trained today. Please sign up here:
2:52

New Course: Post-training of LLMs Learn to post-train and customize an LLM in this short course, taught by Banghua Zhu, Assistant Professor at the University of Washington University of Washington, and co-founder of @NexusflowX. Training an LLM to follow instructions or answer questions has two key stages: pre-training and post-training. In pre-training, it learns to predict the next word or token from large amounts of unlabeled text. In post-training, it learns useful behaviors such as following instructions, tool use, and reasoning. Post-training transforms a general-purpose token predictor—trained on trillions of unlabeled text tokens—into an assistant that follows instructions and performs specific tasks. Because it is much cheaper than pre-training, it is practical for many more teams to incorporate post-training methods into their workflows than pre-training. In this course, you’ll learn three common post-training methods—Supervised Fine-Tuning (SFT), Direct Preference Optimization (DPO), and Online Reinforcement Learning (RL)—and how to use each one effectively. With SFT, you train the model on pairs of input and ideal output responses. With DPO, you provide both a preferred (chosen) and a less preferred (rejected) response and train the model to favor the preferred output. With RL, the model generates an output, receives a reward score based on human or automated feedback, and updates the model to improve performance. You’ll learn the basic concepts, common use cases, and principles for curating high-quality data for effective training. Through hands-on labs, you’ll download a pre-trained model from Hugging Face and post-train it using SFT, DPO, and RL to see how each technique shapes model behavior. In detail, you’ll: - Understand what post-training is, when to use it, and how it differs from pre-training. - Build an SFT pipeline to turn a base model into an instruct model. - Explore how DPO reshapes behavior by minimizing contrastive loss—penalizing poor responses and reinforcing preferred ones. - Implement a DPO pipeline to change the identity of a chat assistant. - Learn online RL methods such as Proximal Policy Optimization (PPO) and Group Relative Policy Optimization (GRPO), and how to design reward functions. - Train a model with GRPO to improve its math capabilities using a verifiable reward. Post-training is one of the most rapidly developing areas of LLM training. Whether you’re building a high-accuracy context-specific assistant, fine-tuning a model's tone, or improving task-specific accuracy, this course will give you experience with the most important techniques shaping how LLMs are post-trained today. Please sign up here:

Andrew Ng

125,146 次观看 • 1 年前

This will retire 90% of RAG systems with dignity (and a sad song playlist). Powered by DSPy: If you're still building "text in, text out" chatbots that only perform blind vector and text searches, you're not gonna make it! My team just dropped Elysia, and it's not just an incremental successor to Verba… It's a whole rethink of how we interact with our data using AI. 𝗪𝗵𝗮𝘁 𝗶𝘀 𝗘𝗹𝘆𝗶𝘀𝗮? An open-source platform for building agentic RAG architectures. It learns from your preferences, intelligently categorizes, labels, and searches through your data, and provides complete transparency into its decision-making process. The long & exciting feature list: • 𝗧𝗿𝗮𝗻𝘀𝗽𝗮𝗿𝗲𝗻𝘁 𝗗𝗲𝗰𝗶𝘀𝗶𝗼𝗻-𝗧𝗿𝗲𝗲 𝗔𝗴𝗲𝗻𝘁𝘀: Elysia’s core is a customizable decision tree, and it visualizes its entire reasoning process, showing you why it chooses a specific tool or path. It enables advanced error handling, self-healing from failed queries, and prevents infinite loops. You can also add custom tools and branches to build complex, state-aware workflows. • 𝗗𝗮𝘁𝗮 𝗔𝘄𝗮𝗿𝗲𝗻𝗲𝘀𝘀: Before it even attempts a query, Elysia performs a full analysis of your data collections. This eliminates the blind search problem plaguing most RAG systems and allows for far more complex and accurate query generation. • 𝗗𝘆𝗻𝗮𝗺𝗶𝗰 𝗗𝗮𝘁𝗮 𝗗𝗶𝘀𝗽𝗹𝗮𝘆𝘀: Your RAG pipeline shouldn't be limited to text, right? That’s why Elysia analyzes each query's results and chooses the best way to display them, from tables and charts to product cards and GitHub tickets. It also features a comprehensive data explorer with search, sorting, and filtering capabilities. • 𝗛𝘆𝗽𝗲𝗿-𝗣𝗲𝗿𝘀𝗼𝗻𝗮𝗹𝗶𝘇𝗮𝘁𝗶𝗼𝗻 𝘃𝗶𝗮 𝗙𝗲𝗲𝗱𝗯𝗮𝗰𝗸: It uses your positively-rated queries as few-shot examples to improve future responses. This allows you to use smaller, faster models that perform like larger ones over time, cutting costs without sacrificing quality for most use cases. • 𝗖𝗵𝘂𝗻𝗸-𝗢𝗻-𝗗𝗲𝗺𝗮𝗻𝗱: Elysia chunks documents at query time. It performs initial searches on document-level vectors and only chunks relevant documents on the fly, storing them in a parallel quantized collection with cross references for future use. 𝗧𝗵𝗲 𝗦𝘁𝗮𝗰𝗸 Elysia is built from scratch on Weaviate, using its native features like named vectors, a variety of search types, filters, cross references, quantization, etc. It uses DSPy for LLM interactions and is delivered as a production-ready application via FastAPI, serving a NextJS frontend as static HTML. Also available as a Python package via pip: 𝗽𝗶𝗽 𝗶𝗻𝘀𝘁𝗮𝗹𝗹 𝗲𝗹𝘆𝘀𝗶𝗮-𝗮𝗶 Type: 𝗲𝗹𝘆𝘀𝗶𝗮 𝘀𝘁𝗮𝗿𝘁 Connect your Weaviate cluster and go explore what’s possible.
0:35

This will retire 90% of RAG systems with dignity (and a sad song playlist). Powered by DSPy: If you're still building "text in, text out" chatbots that only perform blind vector and text searches, you're not gonna make it! My team just dropped Elysia, and it's not just an incremental successor to Verba… It's a whole rethink of how we interact with our data using AI. 𝗪𝗵𝗮𝘁 𝗶𝘀 𝗘𝗹𝘆𝗶𝘀𝗮? An open-source platform for building agentic RAG architectures. It learns from your preferences, intelligently categorizes, labels, and searches through your data, and provides complete transparency into its decision-making process. The long & exciting feature list: • 𝗧𝗿𝗮𝗻𝘀𝗽𝗮𝗿𝗲𝗻𝘁 𝗗𝗲𝗰𝗶𝘀𝗶𝗼𝗻-𝗧𝗿𝗲𝗲 𝗔𝗴𝗲𝗻𝘁𝘀: Elysia’s core is a customizable decision tree, and it visualizes its entire reasoning process, showing you why it chooses a specific tool or path. It enables advanced error handling, self-healing from failed queries, and prevents infinite loops. You can also add custom tools and branches to build complex, state-aware workflows. • 𝗗𝗮𝘁𝗮 𝗔𝘄𝗮𝗿𝗲𝗻𝗲𝘀𝘀: Before it even attempts a query, Elysia performs a full analysis of your data collections. This eliminates the blind search problem plaguing most RAG systems and allows for far more complex and accurate query generation. • 𝗗𝘆𝗻𝗮𝗺𝗶𝗰 𝗗𝗮𝘁𝗮 𝗗𝗶𝘀𝗽𝗹𝗮𝘆𝘀: Your RAG pipeline shouldn't be limited to text, right? That’s why Elysia analyzes each query's results and chooses the best way to display them, from tables and charts to product cards and GitHub tickets. It also features a comprehensive data explorer with search, sorting, and filtering capabilities. • 𝗛𝘆𝗽𝗲𝗿-𝗣𝗲𝗿𝘀𝗼𝗻𝗮𝗹𝗶𝘇𝗮𝘁𝗶𝗼𝗻 𝘃𝗶𝗮 𝗙𝗲𝗲𝗱𝗯𝗮𝗰𝗸: It uses your positively-rated queries as few-shot examples to improve future responses. This allows you to use smaller, faster models that perform like larger ones over time, cutting costs without sacrificing quality for most use cases. • 𝗖𝗵𝘂𝗻𝗸-𝗢𝗻-𝗗𝗲𝗺𝗮𝗻𝗱: Elysia chunks documents at query time. It performs initial searches on document-level vectors and only chunks relevant documents on the fly, storing them in a parallel quantized collection with cross references for future use. 𝗧𝗵𝗲 𝗦𝘁𝗮𝗰𝗸 Elysia is built from scratch on Weaviate, using its native features like named vectors, a variety of search types, filters, cross references, quantization, etc. It uses DSPy for LLM interactions and is delivered as a production-ready application via FastAPI, serving a NextJS frontend as static HTML. Also available as a Python package via pip: 𝗽𝗶𝗽 𝗶𝗻𝘀𝘁𝗮𝗹𝗹 𝗲𝗹𝘆𝘀𝗶𝗮-𝗮𝗶 Type: 𝗲𝗹𝘆𝘀𝗶𝗮 𝘀𝘁𝗮𝗿𝘁 Connect your Weaviate cluster and go explore what’s possible.

Philip Vollet

93,598 次观看 • 11 个月前

X's Recommendation Algorithm Analysis ===================================== Used Grok Code Fast to get a quick breakdown of X's recommendation system. What Makes a post Go Viral =========================== tldr: Engagement prediction trumps everything. Post content that generates interactions. Based on the actual algorithm code, posts that rank highest typically have: + High predicted engagement scores (ML models predict likes/reposts/replies) + Strong personalization match (SimClusters similarity to user interests) + Social graph relevance (RealGraph connections to user's network) + Media content (images/videos get engagement multipliers) + Author credibility (follower count, verification, tweepcred score) + Content quality signals (passes spam/NSFW/quality filters) + Timely relevance (freshness factor, trending topics) + Conversation potential (high reply prediction scores) The algorithm uses machine learning models to predict engagement, not simple weighted formulas. Success is measured by actual user interactions, creating a feedback loop that continuously improves ranking predictions. How the Algorithm Actually Works =============================== 1. Candidate Generation (9 sources): - Earlybird (in-network posts) ~50% - UTEG (out-of-network recommendations) - postMixer, Lists, Communities, Content Exploration - Static, Cached, Backfill sources 2. Feature Hydration (~6000 features per post): - User features (interests, behavior, demographics) - post features (text, media, metadata, engagement) - Graph features (SimClusters, RealGraph, social connections) - Real-time signals (current engagement, trending status) 3. Scoring Pipeline (4 models): - Model Scoring (NAVI heavy ranker) - Reranking Pipeline - Heuristic Scoring - Low Signal Scoring 4. Filtering (24 total filters): - 10 Global Filters (age < 48h, deduplication, location, etc.) - 14 Post-Score Filters (Grok safety, language, video duration, etc.) 5. Final Selection & Mixing: - Sort by final scores - Apply diversity rules - Mix with ads, who-to-follow, prompts - Generate timeline Key Prediction Models ==================== The algorithm predicts these engagement types: • PredictedFavoriteScore (likes) • PredictedRetweetScore (reposts) • PredictedReplyScore (replies) • PredictedGoodClickScore (meaningful clicks) • PredictedVideoQualityViewScore (video engagement) • PredictedBookmarkScore (saves) • PredictedShareScore (external shares) • PredictedDwellScore (time spent viewing) • PredictedNegativeFeedbackScore (hides/blocks) Weight System Reality ==================== IMPORTANT: The algorithm does NOT use fixed percentage weights like: ❌ Like Prediction (35%), Repost (28%), etc. ACTUAL SYSTEM: ✅ Weights are learned parameters from ML training ✅ Default values in code are 0.0 (overridden by feature flags) ✅ Weights are personalized per user and constantly A/B tested ✅ Different content types (video vs text) get different treatment ✅ Weights change based on real-time context and user state Example scoring process: 1. ML models predict engagement probabilities 2. Feature flags provide current weight multipliers 3. Personalization adjusts weights for individual user 4. Real-time context modifies final scores 5. Business rules apply quality gates and diversity What Actually Drives Viral Content ================================== Based on code analysis, viral posts typically: 1. Generate High Engagement Predictions: - Models predict high like/repost/reply probability - Content resonates with multiple user communities - Strong early engagement signals 2. Pass All Quality Gates: - Survive 24 different filter stages - Meet safety standards (not spam/NSFW/violent) - Author has good credibility signals 3. Achieve Personalization at Scale: - Match interests across diverse user segments - Trigger SimClusters similarity for many users - Connect through RealGraph social relationships 4. Optimize for Platform Mechanics: - Include media (images/videos perform better) - Post during high-activity periods - Use formats that encourage replies/reposts Key Takeaways ============= ✅ Engagement prediction is everything - the algorithm optimizes for user interactions ✅ Personalization is sophisticated - uses ML embeddings, not simple keyword matching ✅ Quality filtering is extensive - 24 stages prevent low-quality content ✅ Weights are dynamic - constantly optimized through ML and A/B testing ✅ Scale matters - system processes billions of posts daily with <50ms latenc Transparency exists - this analysis is possible because X open-sourced the algorithm The system is designed to surface content users will engage with, creating a feedback loop that rewards creators who understand their audience and produce engaging content. Bottom line: Create content that generates genuine engagement from your target audience. The algorithm will learn and amplify what works.
0:34

X's Recommendation Algorithm Analysis ===================================== Used Grok Code Fast to get a quick breakdown of X's recommendation system. What Makes a post Go Viral =========================== tldr: Engagement prediction trumps everything. Post content that generates interactions. Based on the actual algorithm code, posts that rank highest typically have: + High predicted engagement scores (ML models predict likes/reposts/replies) + Strong personalization match (SimClusters similarity to user interests) + Social graph relevance (RealGraph connections to user's network) + Media content (images/videos get engagement multipliers) + Author credibility (follower count, verification, tweepcred score) + Content quality signals (passes spam/NSFW/quality filters) + Timely relevance (freshness factor, trending topics) + Conversation potential (high reply prediction scores) The algorithm uses machine learning models to predict engagement, not simple weighted formulas. Success is measured by actual user interactions, creating a feedback loop that continuously improves ranking predictions. How the Algorithm Actually Works =============================== 1. Candidate Generation (9 sources): - Earlybird (in-network posts) ~50% - UTEG (out-of-network recommendations) - postMixer, Lists, Communities, Content Exploration - Static, Cached, Backfill sources 2. Feature Hydration (~6000 features per post): - User features (interests, behavior, demographics) - post features (text, media, metadata, engagement) - Graph features (SimClusters, RealGraph, social connections) - Real-time signals (current engagement, trending status) 3. Scoring Pipeline (4 models): - Model Scoring (NAVI heavy ranker) - Reranking Pipeline - Heuristic Scoring - Low Signal Scoring 4. Filtering (24 total filters): - 10 Global Filters (age < 48h, deduplication, location, etc.) - 14 Post-Score Filters (Grok safety, language, video duration, etc.) 5. Final Selection & Mixing: - Sort by final scores - Apply diversity rules - Mix with ads, who-to-follow, prompts - Generate timeline Key Prediction Models ==================== The algorithm predicts these engagement types: • PredictedFavoriteScore (likes) • PredictedRetweetScore (reposts) • PredictedReplyScore (replies) • PredictedGoodClickScore (meaningful clicks) • PredictedVideoQualityViewScore (video engagement) • PredictedBookmarkScore (saves) • PredictedShareScore (external shares) • PredictedDwellScore (time spent viewing) • PredictedNegativeFeedbackScore (hides/blocks) Weight System Reality ==================== IMPORTANT: The algorithm does NOT use fixed percentage weights like: ❌ Like Prediction (35%), Repost (28%), etc. ACTUAL SYSTEM: ✅ Weights are learned parameters from ML training ✅ Default values in code are 0.0 (overridden by feature flags) ✅ Weights are personalized per user and constantly A/B tested ✅ Different content types (video vs text) get different treatment ✅ Weights change based on real-time context and user state Example scoring process: 1. ML models predict engagement probabilities 2. Feature flags provide current weight multipliers 3. Personalization adjusts weights for individual user 4. Real-time context modifies final scores 5. Business rules apply quality gates and diversity What Actually Drives Viral Content ================================== Based on code analysis, viral posts typically: 1. Generate High Engagement Predictions: - Models predict high like/repost/reply probability - Content resonates with multiple user communities - Strong early engagement signals 2. Pass All Quality Gates: - Survive 24 different filter stages - Meet safety standards (not spam/NSFW/violent) - Author has good credibility signals 3. Achieve Personalization at Scale: - Match interests across diverse user segments - Trigger SimClusters similarity for many users - Connect through RealGraph social relationships 4. Optimize for Platform Mechanics: - Include media (images/videos perform better) - Post during high-activity periods - Use formats that encourage replies/reposts Key Takeaways ============= ✅ Engagement prediction is everything - the algorithm optimizes for user interactions ✅ Personalization is sophisticated - uses ML embeddings, not simple keyword matching ✅ Quality filtering is extensive - 24 stages prevent low-quality content ✅ Weights are dynamic - constantly optimized through ML and A/B testing ✅ Scale matters - system processes billions of posts daily with <50ms latenc Transparency exists - this analysis is possible because X open-sourced the algorithm The system is designed to surface content users will engage with, creating a feedback loop that rewards creators who understand their audience and produce engaging content. Bottom line: Create content that generates genuine engagement from your target audience. The algorithm will learn and amplify what works.

tetsuo

307,903 次观看 • 10 个月前

is our AI project to make computing feel more human L A N D E R Here are the 4 best demo videos of the magic of DATA in action. DATA is a personalized assistant who knows and remembers every conversation you have with it accross your iPhone, Mac, iPad, Watch, Texts, Emails, and HomePods. You can talk to DATA right in your AirPods or text it just like a person. DATA can read, write, understand, speak any language, and translate between them. It can help with real work and home life tasks like research, writing, scheduling, reminders, and triage. And it's easily customizable so you can have DATA automatically do whatever you want whenever you want with just a few taps and natural language instructions - no code required. DATA can do just about anything you can do on your phone on your behalf automatically including very advanced things Siri can't, like summarizing, analyzing, and drafting replies or writing documents. It can read web pages, texts or emails you show it, or PDFs of any kind. It can do other real world tasks that require complex analysis and common sense too, like: - figure out where the nearest beach is (even when you're in Colorado) and instantly fetch the current surf report up to the current minute. - summarize and drafting replies to entire email chains - plan out entire work projects or multi-day vacations on your calendar - sketch out ideas for you in picture form or drafting Notion pages with charts and graphs. DATA can also use its own judgement to determine when to run an action or not, even if you've scheduled it, allowing you to make VERY complex automations that require many different inputs to make a decision, like for example: - only opening the blinds on your lunch break if it's sunny out and you're working from home. DATA works natively and easily with Apple HomeKit & other shortcuts. DATA can also take initiative and check in with you throughout the day by voice or text and proactively send messages to you and others on your behalf based on your personal and professional goals, current tasks, and calendar. DATA can integrate with many apps on your phone, and is compatible with multiple large AI language models. I've gotten to make a few demo videos that I think really capture how powerful DATA can be for every day life. Here they are all in one tweet. Make sure your sound is on as you watch them. 1. This is the first demo video I ever made from April 19th, 2023. It walks through all the ways you can interact with and use the DATA shortcuts. Everything from saying "Hey Siri" to tapping on custom apps on your home-screen. 2. The second demo video was made May 5 and is an example use case I made of how commands work - commands allow DATA to actually run actions on your phone like taking pictures and sending messages. This demo shows me taking a picture of an email template, and data drafting an email based on that template. It's gotten much better at realizing when it has just run a command and incorporating that information naturally into the conversation now, especially on GPT-4. 3. This third Commands video, May 12 is a walkthrough of ALL the phone functions that commands allow DATA to do: sending texts and emails, making pictures, seeing pictures, reading things, and scheduling events. Since this video we've added auto-replies to texts and emails, summarizing documents, writing documents, health app data retrieval, web surfing, scheduling alarms, making playlists, and more. 4. This last demo I made today, June 15, shows everything DATA does working in concert to generate a crazy detailed morning briefing with background music - including making a unique playlist and giving a detailed analysis of current events complete with Ski & Surf conditions near me other live information from the internet. So now that you've seen everything DATA can do, what's the coolest feature? What features should we add? What would you use DATA for first?
12:54

is our AI project to make computing feel more human L A N D E R Here are the 4 best demo videos of the magic of DATA in action. DATA is a personalized assistant who knows and remembers every conversation you have with it accross your iPhone, Mac, iPad, Watch, Texts, Emails, and HomePods. You can talk to DATA right in your AirPods or text it just like a person. DATA can read, write, understand, speak any language, and translate between them. It can help with real work and home life tasks like research, writing, scheduling, reminders, and triage. And it's easily customizable so you can have DATA automatically do whatever you want whenever you want with just a few taps and natural language instructions - no code required. DATA can do just about anything you can do on your phone on your behalf automatically including very advanced things Siri can't, like summarizing, analyzing, and drafting replies or writing documents. It can read web pages, texts or emails you show it, or PDFs of any kind. It can do other real world tasks that require complex analysis and common sense too, like: - figure out where the nearest beach is (even when you're in Colorado) and instantly fetch the current surf report up to the current minute. - summarize and drafting replies to entire email chains - plan out entire work projects or multi-day vacations on your calendar - sketch out ideas for you in picture form or drafting Notion pages with charts and graphs. DATA can also use its own judgement to determine when to run an action or not, even if you've scheduled it, allowing you to make VERY complex automations that require many different inputs to make a decision, like for example: - only opening the blinds on your lunch break if it's sunny out and you're working from home. DATA works natively and easily with Apple HomeKit & other shortcuts. DATA can also take initiative and check in with you throughout the day by voice or text and proactively send messages to you and others on your behalf based on your personal and professional goals, current tasks, and calendar. DATA can integrate with many apps on your phone, and is compatible with multiple large AI language models. I've gotten to make a few demo videos that I think really capture how powerful DATA can be for every day life. Here they are all in one tweet. Make sure your sound is on as you watch them. 1. This is the first demo video I ever made from April 19th, 2023. It walks through all the ways you can interact with and use the DATA shortcuts. Everything from saying "Hey Siri" to tapping on custom apps on your home-screen. 2. The second demo video was made May 5 and is an example use case I made of how commands work - commands allow DATA to actually run actions on your phone like taking pictures and sending messages. This demo shows me taking a picture of an email template, and data drafting an email based on that template. It's gotten much better at realizing when it has just run a command and incorporating that information naturally into the conversation now, especially on GPT-4. 3. This third Commands video, May 12 is a walkthrough of ALL the phone functions that commands allow DATA to do: sending texts and emails, making pictures, seeing pictures, reading things, and scheduling events. Since this video we've added auto-replies to texts and emails, summarizing documents, writing documents, health app data retrieval, web surfing, scheduling alarms, making playlists, and more. 4. This last demo I made today, June 15, shows everything DATA does working in concert to generate a crazy detailed morning briefing with background music - including making a unique playlist and giving a detailed analysis of current events complete with Ski & Surf conditions near me other live information from the internet. So now that you've seen everything DATA can do, what's the coolest feature? What features should we add? What would you use DATA for first?

steve

640,114 次观看 • 3 年前

As many of you know, for the last five months I've been working full-time on my next big thing. The challenge was to invent something new and implement it entirely using LLMs for writing code. The first stage of the project is now complete: the web application, which I called is now online and accepting users. You can see a short demo in the video. 100% of the code of the app was generated by LLMs (mostly Gemini and Claude, maybe 10% of ChatGPT). I haven't written a single line of code. The tech stack is TypeScript, React, and Supabase/Postgres which was (and still is) fully new to me. During these five months, I implemented from scratch three versions of the software. It started as a Markdown editor to help me with my book writing and ended up as an AI-assisted reading and self-learning platform. What makes ChapterPal unique is a novel reading experience where the user can use the keyboard keys to reveal or "unreveal" the content and ask questions at any moment. (Mouse wheel, touchpad, smartphone screen, and voice input are also supported.) The LLM receives the entire content of the chapter and tries to answer questions based on the chapter's content, which reduces the chance of hallucination to the minimum. (Though not to 0%, of course, but near it.) This way of content consumption is known as **active reading,** a strategy for engaging with a text to improve comprehension and retention by consciously interacting with the material. The goal is to move beyond passive reading to a deeper understanding of the text and to remember key information more effectively. The registration on ChapterPal is via the waiting list. This is to avoid unexpected load spikes and cloud charges. Usually, it takes less than 24 hours for me to activate a user. Give it a try and let me know what you think. The next stage is finishing the content ingestion pipeline, which will automatically convert high-quality content from sources like HTML, PDF, and LaTeX into Markdown. Obviously, only those pieces whose licenses allow creating copies. ChapterPal has its own collection of textbooks and articles on AI, machine learning, and data science topics. If you don't find a piece of content you would like to read in ChapterPal's collection, a Chrome extension, ChapterPal Uploader, allows you to upload any PDF or HTML page to ChapterPal in one click. The content is only available for you to read to avoid the possibility of copyright infringement. I hope you enjoy using it as much as I enjoy building it.
3:41

As many of you know, for the last five months I've been working full-time on my next big thing. The challenge was to invent something new and implement it entirely using LLMs for writing code. The first stage of the project is now complete: the web application, which I called is now online and accepting users. You can see a short demo in the video. 100% of the code of the app was generated by LLMs (mostly Gemini and Claude, maybe 10% of ChatGPT). I haven't written a single line of code. The tech stack is TypeScript, React, and Supabase/Postgres which was (and still is) fully new to me. During these five months, I implemented from scratch three versions of the software. It started as a Markdown editor to help me with my book writing and ended up as an AI-assisted reading and self-learning platform. What makes ChapterPal unique is a novel reading experience where the user can use the keyboard keys to reveal or "unreveal" the content and ask questions at any moment. (Mouse wheel, touchpad, smartphone screen, and voice input are also supported.) The LLM receives the entire content of the chapter and tries to answer questions based on the chapter's content, which reduces the chance of hallucination to the minimum. (Though not to 0%, of course, but near it.) This way of content consumption is known as **active reading,** a strategy for engaging with a text to improve comprehension and retention by consciously interacting with the material. The goal is to move beyond passive reading to a deeper understanding of the text and to remember key information more effectively. The registration on ChapterPal is via the waiting list. This is to avoid unexpected load spikes and cloud charges. Usually, it takes less than 24 hours for me to activate a user. Give it a try and let me know what you think. The next stage is finishing the content ingestion pipeline, which will automatically convert high-quality content from sources like HTML, PDF, and LaTeX into Markdown. Obviously, only those pieces whose licenses allow creating copies. ChapterPal has its own collection of textbooks and articles on AI, machine learning, and data science topics. If you don't find a piece of content you would like to read in ChapterPal's collection, a Chrome extension, ChapterPal Uploader, allows you to upload any PDF or HTML page to ChapterPal in one click. The content is only available for you to read to avoid the possibility of copyright infringement. I hope you enjoy using it as much as I enjoy building it.

BURKOV

81,180 次观看 • 8 个月前

Introducing Arcana Wallet with Chain Abstraction!🌟 Beta is now live on Chrome Store! Get started👉 You can now unify your USDC, USDT, and ETH balances across Ethereum, Base, Polygon, Arbitrum, and Optimism— and spend it in one click, without bridging. 💡Beyond the core benefits of #ChainAbstraction like unified balances, auto-fund gas fees with stablecoins, and near-instant multi-chain transactions, $XAR Chain Abstraction Protocol does three things extremely well: 🔹EOA Wallet-Based Orchestration: Bring your existing wallet address without locking up funds or depositing into a new account—full self-custody of assets. 🔹Gas Efficiency: Upto 5X lower gas fees compared to smart contract-based chain abstraction 🔹Universal Addresses: Arcana does not create app-specific wallets that require users to deposit tokens. This means your assets remain in a single wallet, accessible across apps —even on apps that do not support chain abstraction. Try spending your unified balances on apps like Uniswap, Aave, Polymarket, Hyperliquid, and Jumper🌐 Arcana Wallet is deployed on the Chain Abstraction Testnet. The Testnet launch marks an important milestone in bringing a chainless experience to users across ecosystems, as we get closer to the launch of Mainnet✨ Arcana’s Chain Abstraction Protocol (powered by $XAR) is the cornerstone of Arcana Wallet's capabilities, driving a new era of multichain usability. Developers will soon be able to leverage our Chain Abstraction SDK, empowering them to build chainless user experiences.
0:55

Introducing Arcana Wallet with Chain Abstraction!🌟 Beta is now live on Chrome Store! Get started👉 You can now unify your USDC, USDT, and ETH balances across Ethereum, Base, Polygon, Arbitrum, and Optimism— and spend it in one click, without bridging. 💡Beyond the core benefits of #ChainAbstraction like unified balances, auto-fund gas fees with stablecoins, and near-instant multi-chain transactions, $XAR Chain Abstraction Protocol does three things extremely well: 🔹EOA Wallet-Based Orchestration: Bring your existing wallet address without locking up funds or depositing into a new account—full self-custody of assets. 🔹Gas Efficiency: Upto 5X lower gas fees compared to smart contract-based chain abstraction 🔹Universal Addresses: Arcana does not create app-specific wallets that require users to deposit tokens. This means your assets remain in a single wallet, accessible across apps —even on apps that do not support chain abstraction. Try spending your unified balances on apps like Uniswap, Aave, Polymarket, Hyperliquid, and Jumper🌐 Arcana Wallet is deployed on the Chain Abstraction Testnet. The Testnet launch marks an important milestone in bringing a chainless experience to users across ecosystems, as we get closer to the launch of Mainnet✨ Arcana’s Chain Abstraction Protocol (powered by $XAR) is the cornerstone of Arcana Wallet's capabilities, driving a new era of multichain usability. Developers will soon be able to leverage our Chain Abstraction SDK, empowering them to build chainless user experiences.

Arcana Network

60,711 次观看 • 1 年前

Traditional chunking: cheap but dumb. ColBERT: smart but expensive. 𝗟𝗮𝘁𝗲 𝗰𝗵𝘂𝗻𝗸𝗶𝗻𝗴: the solution we've been waiting for. Here’s a quick evolution of chunking strategies: → 𝗧𝗿𝗮𝗱𝗶𝘁𝗶𝗼𝗻𝗮𝗹 𝗖𝗵𝘂𝗻𝗸𝗶𝗻𝗴 (the basics we all started with) • Token Chunking - split by token count • Sentence Chunking - split by sentence boundaries • Document-Based Chunking - split by sections/paragraphs → 𝗔𝗱𝘃𝗮𝗻𝗰𝗲𝗱 𝗖𝗵𝘂𝗻𝗸𝗶𝗻𝗴 (when things got sophisticated) • Semantic Chunking - split by meaning • LLM-Based Chunking - let the model decide But each chunking method separates text at defined points, meaning context is lost within the document from one chunk to the next. → 𝗘𝗻𝘁𝗲𝗿 𝗟𝗮𝘁𝗲 𝗖𝗵𝘂𝗻𝗸𝗶𝗻𝗴 (the game changer) Traditional approach: Chunk first → Embed each chunk separately Late chunking approach: Embed the entire document → Then chunk with context preserved 𝗪𝗵𝘆 𝗰𝗵𝗼𝗼𝘀𝗲 𝗹𝗮𝘁𝗲 𝗰𝗵𝘂𝗻𝗸𝗶𝗻𝗴? When you chunk first, each piece loses its contextual relationship to the rest of the document. It's like reading a book by randomly picking paragraphs - you miss the flow. With late chunking, every chunk maintains awareness of its neighbors because the embedding happens at the document level first. Mean pooling is done on segments AFTER the full context is embedded. Jina AI tested and saw significant improvements in retrieval quality - chunks that were previously disconnected now maintain their semantic relationships. As documents get longer and context windows expand, late chunking might just become the new standard for high-quality retrieval systems. 𝗪𝗵𝗮𝘁 𝗱𝗼 𝘆𝗼𝘂 𝗻𝗲𝗲𝗱 𝘁𝗼 𝗺𝗮𝗸𝗲 𝘁𝗵𝗶𝘀 𝘄𝗼𝗿𝗸? No modifications to your retrieval pipeline are needed. 1. Long context embedding models (8192+ tokens) 2. Chunking logic that tracks token spans 3. Less than 30 lines of code to implement All you need is to switch the order at which you chunk and embed. Embed FIRST, then chunk, not the other way around. Dive deeper into late chunking:
1:19

Traditional chunking: cheap but dumb. ColBERT: smart but expensive. 𝗟𝗮𝘁𝗲 𝗰𝗵𝘂𝗻𝗸𝗶𝗻𝗴: the solution we've been waiting for. Here’s a quick evolution of chunking strategies: → 𝗧𝗿𝗮𝗱𝗶𝘁𝗶𝗼𝗻𝗮𝗹 𝗖𝗵𝘂𝗻𝗸𝗶𝗻𝗴 (the basics we all started with) • Token Chunking - split by token count • Sentence Chunking - split by sentence boundaries • Document-Based Chunking - split by sections/paragraphs → 𝗔𝗱𝘃𝗮𝗻𝗰𝗲𝗱 𝗖𝗵𝘂𝗻𝗸𝗶𝗻𝗴 (when things got sophisticated) • Semantic Chunking - split by meaning • LLM-Based Chunking - let the model decide But each chunking method separates text at defined points, meaning context is lost within the document from one chunk to the next. → 𝗘𝗻𝘁𝗲𝗿 𝗟𝗮𝘁𝗲 𝗖𝗵𝘂𝗻𝗸𝗶𝗻𝗴 (the game changer) Traditional approach: Chunk first → Embed each chunk separately Late chunking approach: Embed the entire document → Then chunk with context preserved 𝗪𝗵𝘆 𝗰𝗵𝗼𝗼𝘀𝗲 𝗹𝗮𝘁𝗲 𝗰𝗵𝘂𝗻𝗸𝗶𝗻𝗴? When you chunk first, each piece loses its contextual relationship to the rest of the document. It's like reading a book by randomly picking paragraphs - you miss the flow. With late chunking, every chunk maintains awareness of its neighbors because the embedding happens at the document level first. Mean pooling is done on segments AFTER the full context is embedded. Jina AI tested and saw significant improvements in retrieval quality - chunks that were previously disconnected now maintain their semantic relationships. As documents get longer and context windows expand, late chunking might just become the new standard for high-quality retrieval systems. 𝗪𝗵𝗮𝘁 𝗱𝗼 𝘆𝗼𝘂 𝗻𝗲𝗲𝗱 𝘁𝗼 𝗺𝗮𝗸𝗲 𝘁𝗵𝗶𝘀 𝘄𝗼𝗿𝗸? No modifications to your retrieval pipeline are needed. 1. Long context embedding models (8192+ tokens) 2. Chunking logic that tracks token spans 3. Less than 30 lines of code to implement All you need is to switch the order at which you chunk and embed. Embed FIRST, then chunk, not the other way around. Dive deeper into late chunking:

Femke Plantinga

125,348 次观看 • 11 个月前

Introducing Wikiwise: an open-source Mac app for managing your own Karpathy-style LLM wiki. Set up a new wiki in a few clicks: all you need is Wikiwise + your agent. It's infinitely customizable, just markdown/html under the hood, and one click to share your wiki publicly. Here's how it works: * Install Wikiwise for mac (it's built in Swift so super minimal and performant). In Karpathy's framework, Wikiwise is your IDE. * Start a new Wiki: it generates a new folder on your machine that's scaffolded in the wiki structure Andrej Karpathy describes (index.md, raw folder, wiki folder, CLAUDE.md/AGENTS.md, although it tries to be as un-opinionated as possible). * Then just point your agent (Codex, Claude Code, Cursor, etc) at the folder and tell it what to import -- files on your machine, connect to your Readwise account, or urls from the web. * Your agent creates wthe wiki for you: Your agent will know how to ingest your raw sources (via the AGENTS.md) and will immediately start writing+linking wiki pages for you. * Go crazy on customization! The rendered wiki pages live as static html/css in your folder too so just tell your agent to change stuff, and if you need any more customization Wikiwise is fully open source :) * Ask questions about your research with your agent, ask it to bring in new sources, write new documents, etc. * (optionally) Hit the Publish button to share your wiki with friends/colleagues at a custom URL === I tried to walk the line on a couple constraints with Wikiwise: 1. I wanted it to be easy to spin up new wikis, especially without chaining together a bunch of different apps. It takes me a few minutes to spin up a new wiki on a topic -- I already have five! 2. Infinitely Customizable: one great aspect of building a wiki as Karpathy described is that you can modify any aspect of your wiki with your agent. Every new wiki styling+structure is self-contained in the local folder, which allows you to preserve this. Wikiwise is just an IDE that makes the setup easier and includes a nice un-opinionated starting state. 3. Minimal: Wikiwise is built mostly in Swift, and the DMG you install to download it is only 2.6MB (!) 4. Easy Publishing: my colleague Eleanor Konik has been building her own LLM wikis for months, but has always really struggled to actually share them with her book club. There are tools to do it, but figuring out hosting is always a huge headache. This seemed like an ideal usecase for a tool like Wikiwise to solve. The process of building wikiwise was also pretty interesting -- I "bootstrapped" the app in a way by first building my own wiki based on Karpathy's tweet and other notes I had, and slowly formed the shape of the project in collaboration with my LLM. This was all done in 3 days over the latest Readwise company hackathon we had. Truly an incredible time to be alive. Anyways, curious what you think! Links in next tweet.
3:10

Introducing Wikiwise: an open-source Mac app for managing your own Karpathy-style LLM wiki. Set up a new wiki in a few clicks: all you need is Wikiwise + your agent. It's infinitely customizable, just markdown/html under the hood, and one click to share your wiki publicly. Here's how it works: * Install Wikiwise for mac (it's built in Swift so super minimal and performant). In Karpathy's framework, Wikiwise is your IDE. * Start a new Wiki: it generates a new folder on your machine that's scaffolded in the wiki structure Andrej Karpathy describes (index.md, raw folder, wiki folder, CLAUDE.md/AGENTS.md, although it tries to be as un-opinionated as possible). * Then just point your agent (Codex, Claude Code, Cursor, etc) at the folder and tell it what to import -- files on your machine, connect to your Readwise account, or urls from the web. * Your agent creates wthe wiki for you: Your agent will know how to ingest your raw sources (via the AGENTS.md) and will immediately start writing+linking wiki pages for you. * Go crazy on customization! The rendered wiki pages live as static html/css in your folder too so just tell your agent to change stuff, and if you need any more customization Wikiwise is fully open source :) * Ask questions about your research with your agent, ask it to bring in new sources, write new documents, etc. * (optionally) Hit the Publish button to share your wiki with friends/colleagues at a custom URL === I tried to walk the line on a couple constraints with Wikiwise: 1. I wanted it to be easy to spin up new wikis, especially without chaining together a bunch of different apps. It takes me a few minutes to spin up a new wiki on a topic -- I already have five! 2. Infinitely Customizable: one great aspect of building a wiki as Karpathy described is that you can modify any aspect of your wiki with your agent. Every new wiki styling+structure is self-contained in the local folder, which allows you to preserve this. Wikiwise is just an IDE that makes the setup easier and includes a nice un-opinionated starting state. 3. Minimal: Wikiwise is built mostly in Swift, and the DMG you install to download it is only 2.6MB (!) 4. Easy Publishing: my colleague Eleanor Konik has been building her own LLM wikis for months, but has always really struggled to actually share them with her book club. There are tools to do it, but figuring out hosting is always a huge headache. This seemed like an ideal usecase for a tool like Wikiwise to solve. The process of building wikiwise was also pretty interesting -- I "bootstrapped" the app in a way by first building my own wiki based on Karpathy's tweet and other notes I had, and slowly formed the shape of the project in collaboration with my LLM. This was all done in 3 days over the latest Readwise company hackathon we had. Truly an incredible time to be alive. Anyways, curious what you think! Links in next tweet.

Tristan

95,483 次观看 • 3 个月前

If I lost everything and needed to make $5k-$10k per month with AI starting today here's what I would do: 1) Go to Fiverr and find manual tasks people are charging decent money for. I had one in mind - Keyword Research. I looked at 5-star reviews of Fiverr Pros and found a few profiles that are charing $100-$200+ for this job 2) Go to Manus/o3/Gemini 2.5 (I used Gemini here) and build out a roadmap to create an app using Replit/Bolt. etc. to automate high quality keyword research. BONUS: buy the service from Fiverr, load it into your LLM and ask it to identify where the output is falling short and how it can be improved. 3) Take the roadmap and go to my coding tool. I'm using Replit here. I spent around an hour vibe coding this while Greg is recording podcast episodes here in SF. Demo attached. Get the relevant API keys, do some QA testing, let some folks try it for free and give you feedback. 4) Have Manus, etc. crawl the profiles of top rated Fiverr pros (can also do Upwork, or others) and build a step by step guide to create an optimized profile to attract high quality clients for my service. Use 4o for a profile pic, Claude to write my description. 5) Include photos of my output. Say I have a proprietary custom workflow that allows me to do it faster and higher quality. 1-2 hour turnaround. 6) Keep iterating my workflow/internal tool based on feedback from customers, attack other marketplaces where there is clear demand, partner with agencies, build in public...lots of growth opportunities. 7) Keep productizing. Why not turn it into a micro-saas...validated, clear demand, paying customers. I think this might be the fastest way to financial freedom with AI. Who's doing this?
1:16

If I lost everything and needed to make $5k-$10k per month with AI starting today here's what I would do: 1) Go to Fiverr and find manual tasks people are charging decent money for. I had one in mind - Keyword Research. I looked at 5-star reviews of Fiverr Pros and found a few profiles that are charing $100-$200+ for this job 2) Go to Manus/o3/Gemini 2.5 (I used Gemini here) and build out a roadmap to create an app using Replit/Bolt. etc. to automate high quality keyword research. BONUS: buy the service from Fiverr, load it into your LLM and ask it to identify where the output is falling short and how it can be improved. 3) Take the roadmap and go to my coding tool. I'm using Replit here. I spent around an hour vibe coding this while Greg is recording podcast episodes here in SF. Demo attached. Get the relevant API keys, do some QA testing, let some folks try it for free and give you feedback. 4) Have Manus, etc. crawl the profiles of top rated Fiverr pros (can also do Upwork, or others) and build a step by step guide to create an optimized profile to attract high quality clients for my service. Use 4o for a profile pic, Claude to write my description. 5) Include photos of my output. Say I have a proprietary custom workflow that allows me to do it faster and higher quality. 1-2 hour turnaround. 6) Keep iterating my workflow/internal tool based on feedback from customers, attack other marketplaces where there is clear demand, partner with agencies, build in public...lots of growth opportunities. 7) Keep productizing. Why not turn it into a micro-saas...validated, clear demand, paying customers. I think this might be the fastest way to financial freedom with AI. Who's doing this?

The Boring Marketer

139,400 次观看 • 1 年前

I run my linkedin ads with OpenClaw🦞 for $0/month 😱 here's the system to get real buyers (not bad leads): step 1: read the account → ingests csv exports OR pulls the LinkedIn marketing api directly → campaigns, creatives, lead forms, spend. all of it. → no dashboard. just the raw truth of what's happening in the account. step 2: calls out fake noise → that "efficient" campaign pulling vendors and students? flagged. → high-CTR post pulling people who will never buy? flagged. → everything tied to real crm outcomes, not platform vanity metrics. step 3: grade buyer quality → maps every lead against ICP fit, intent, and sales-readiness → the cheapest lead is usually the worst lead. the kit knows the difference → cross-references crm notes so sales quality compounds every run step 4: find the leaks → good ad + generic form = lost conversation. → surfaces every break between audience, offer, form, crm, and handoff. → tells you which leak is costing the most. not "ctr went up 12%" step 5: score thought leader posts → likes ≠ buyers. ranks posts on ICP fit, buyer pain, trust, organic signal → drafts the sponsored creative. OR a launch packet if the api's gated → no more paying to boost posts that went viral with the wrong audience step 6: draft safe moves → pause, budget update, activate draft. every action is reversible → dry-run first. signed receipt. explicit APPLY confirmation → no freehanding your ad account step 7: compound the memory → every brief writes back to learnings md → which offers create quality. which voices deserve spend. what never to repeat → multi-brand by default. no cross-contamination between projects input: linkedin ads exports (or oauth) + lead data + crm notes output: daily buyer-quality brief + reviewable drafts + brand memory that compounds linkedin ads agencies charge $8-15K/mo to manage B2B accounts. this runs the operator loop for $0. I packaged the entire system as the LinkedIn Ads Kit. 10 claude skills: - linkedin-ads (core operator loop + daily brief) - linkedin-api-connect (oauth, account discovery, api health) - linkedin-ads-apply (safe drafts, dry run, audit trail) - buyer-quality-audit (real buyers vs. fake lead volume) - lead-quality-mapper (fit, intent, sales-readiness per lead) - offer-angle-diagnoser (is the offer strong enough for linkedin) - thought-leader-ad-selector (score posts, draft creative or packet) - form-friction-review (fix lead forms for quality, not volume) - pipeline-brief-writer (executive brief from the findings) - sales-handoff (context so sales knows why the lead converted) also works with OpenClaw🦞, hermes (Nous Research) codex or any agent giving it away free. comment BUYERS + like + follow (must follow so i can DM)
1:05

I run my linkedin ads with OpenClaw🦞 for $0/month 😱 here's the system to get real buyers (not bad leads): step 1: read the account → ingests csv exports OR pulls the LinkedIn marketing api directly → campaigns, creatives, lead forms, spend. all of it. → no dashboard. just the raw truth of what's happening in the account. step 2: calls out fake noise → that "efficient" campaign pulling vendors and students? flagged. → high-CTR post pulling people who will never buy? flagged. → everything tied to real crm outcomes, not platform vanity metrics. step 3: grade buyer quality → maps every lead against ICP fit, intent, and sales-readiness → the cheapest lead is usually the worst lead. the kit knows the difference → cross-references crm notes so sales quality compounds every run step 4: find the leaks → good ad + generic form = lost conversation. → surfaces every break between audience, offer, form, crm, and handoff. → tells you which leak is costing the most. not "ctr went up 12%" step 5: score thought leader posts → likes ≠ buyers. ranks posts on ICP fit, buyer pain, trust, organic signal → drafts the sponsored creative. OR a launch packet if the api's gated → no more paying to boost posts that went viral with the wrong audience step 6: draft safe moves → pause, budget update, activate draft. every action is reversible → dry-run first. signed receipt. explicit APPLY confirmation → no freehanding your ad account step 7: compound the memory → every brief writes back to learnings md → which offers create quality. which voices deserve spend. what never to repeat → multi-brand by default. no cross-contamination between projects input: linkedin ads exports (or oauth) + lead data + crm notes output: daily buyer-quality brief + reviewable drafts + brand memory that compounds linkedin ads agencies charge $8-15K/mo to manage B2B accounts. this runs the operator loop for $0. I packaged the entire system as the LinkedIn Ads Kit. 10 claude skills: - linkedin-ads (core operator loop + daily brief) - linkedin-api-connect (oauth, account discovery, api health) - linkedin-ads-apply (safe drafts, dry run, audit trail) - buyer-quality-audit (real buyers vs. fake lead volume) - lead-quality-mapper (fit, intent, sales-readiness per lead) - offer-angle-diagnoser (is the offer strong enough for linkedin) - thought-leader-ad-selector (score posts, draft creative or packet) - form-friction-review (fix lead forms for quality, not volume) - pipeline-brief-writer (executive brief from the findings) - sales-handoff (context so sales knows why the lead converted) also works with OpenClaw🦞, hermes (Nous Research) codex or any agent giving it away free. comment BUYERS + like + follow (must follow so i can DM)

Matthew Berman

11,080 次观看 • 2 个月前