visualizing text splitting & chunking strategies ChunkViz .com I thought I remembered a tool to visualize text chunking, but I couldn't find it, so I built one I didn't realize it would be so visually pleasing to tinker with 4 different splitters featured

Head over to to try it out This is part of a post I'm doing on chunking philosophies. I'll publish on Code:

I have beef though, returned chunks are .trim()'d, so they aren't exactly what the user passes in! It made reconstructing the viz hard 😉

@hwchase17 @langchain Great tool for quick visualization. That recursive paragraph example - makes me think an algorithm that minimizes "orphan" chunks when trying to grab whole paragraphs would be an awesome tool. Otherwise optimal settings hard to gauge.

@hwchase17 @langchain Big time, definitely hard to gauge. That would be cool to have a chunk analyzer that told you how many orphan or random chunks you had

@hwchase17 @langchain You could show quick calculation on text upload - def max_len(txt): ps = txt.split("\n\n") m = 0 for p in ps: m = max(m, len(p)) return m

@hwchase17 @langchain Then show how many "mid splits" occur based on current settings... def mid_split(txt, ml): c, sc = [txt[i:i+ml] for i in range(0, len(txt), ml)], 0 sc = sum(1 for i in c[:-1] if not i.endswith(('\n\n', '\n'))) return sc

@langchain Great tool! 🔥

@langchain Very cool and useful! Thank you!

@hwchase17 @langchain Very good, very useful!

@langchain 🔥

@langchain Thanks Greg! Super helpful!

@langchain Super useful tool. Bookmarked!

@langchain Thanks dude! Very handy tool.

@langchain Nice thanks! What features should be added?

@langchain What about a spell-checker/auto-correcter with use control.. Context is key

@langchain This is great!

@langchain @memdotai mem it

@langchain There's some tools in the You can ask a few experts on their mailing list

@langchain Hey, please help with verctorizing the arabic language. Thanks for the content!

@langchain Thanks Greg

@langchain np - I'm looking for more ideas to provide value with it, lmk if you have any requests.

@langchain Perform knowledge graph analysis of the information in order to determine the optimal chunk size to keep topical corpus of text together.

@langchain do you have an example of this to share? I'm interested the direction

@langchain @memdotai mem it