Introducing webpull instantly pull any website into a directory of clean, searchable markdown files npx webpull

Here's the github

nice terminal dude

hold up i dont care about this software stuff whats that shader on your terminal

Solar flare on warp!

if you dont want to get blocked you should integrate @getcontextdev trust

@getcontextdev in this case docs website are all open (they need to be) so it's fine

what terminal is that?

how does this handle websites with scraper protection or client side rendered websites?

google is in danger

Damn nice!

superfast, best for giving context from docs and if don't want to use web_fetch again and again

cool, i built a similar tool that pipes how a website is looking ( and captures dom as well ) to a frontend agent so it can make further decision on the design of the website it is working on. its called rasterport and its live on npm hehe

this is awesome, will have to look into if we can use this for

Yo good stuff 1. Defuddle link is broken?! 2. This will be amazing for synthesizing skills out of websites. Does it support auth?

insane! does it use defuddle?

yes!

So llms txt?

how is this different than when agents just web search?

what have your use cases been for this?

ask an agent to figure out how to do something!

gotcha so it's kinda like shareable skill files but just any website?

yesss the website is the skill, agent can look through the clean version of the website (run grep etc)

can it go around Cloudflare firewall protected pages 👀

what we keep hitting with webfetch-style tools are SPAs that lazy-render content client-side (most modern docs sites) come back almost empty. is webpull headless-chrome under the hood or fetch+parse?

this is actually something i wanted ngl. i use exa and context7 but this works well when i already know what i need. it is kind of like treating the whole web as a codebase repo. a search api can scrape the internet for me to get an answer, while here i can just ask my agent, especially in a cli and quickly get the info

This is just exactly what I made

w work fr

Let’s goo Dhravya!!

Aha, this can be a fun addition here as well Will add it here

This is brilliant for documentation and competitive research. Being able to convert any site into searchable markdown instantly saves hours of manual copy-paste work. The clean structure makes it perfect for feeding into analysis tools too.

how’s it better/different than firecrawl?

How’s it so fast?🤔

@arlanr worth integrating w @nozomioai ?

Is this a new found interest in Filesystems?

Few people realise how extremely powerful this is

U BEAUTYYY

Ok, but why? Just give your llm a good search/fetch tool

This is amazing!!

Wait, why MD and not the original (html/etc) formats?

because LLMS just get it

They get websites at least equally well tho

But Why?

Oh this is cool