Data often lie on a low-dimensional manifold embedded in a high-dimensional space. But these manifolds are often highly non-linear, making linear dimensionality reduction methods like PCA insufficient. This has motivated the development of non-linear dimensionality reduction.

I put together this visual interactive blog explaining Isomap, a classic non-linear dimensionality reduction technique. Please check it out:

If you are interested in seeing the code (I haven't cleaned it up much 😅).

Really a nice illustration, How did you make this animation? Manim?

Thanks! This was with d3.js

May I include this in my teaching slides? Very comprehensive!

Thank you! Absolutely!

People using the term “highly non-linear” should be exiled from society.

Yeah there are manifold methods such as UMAP

Nice visualization. Captures how linear methods miss the twists in manifold shape. Did you experiment with other methods like t-SNE or UMAP on this?

Thanks! Perhaps in the future

@karanjagtiani04 Would also like to use a version of tsne and Umap!

@alec_helbling t-SNE and UMAP do catch those complex structures better, especially for high-dimensional data with non-linear relationships.

Non-linear dimensionality reduction??? U mean KPCA 🤣?

Exactly. The universe is an NLDR algorithm where gravity is the learned manifold's geometry and quantum mechanics is the aliasing error from the projection. Observation triggers collapse by de-aliasing the state.

What like kernel PCA ?

Non-linear spectral fit is very good.

Aaahh that was my PhD research and my retarded and insufficient advisor, as well as the commitee made me quit

Neat

** From High-Order Nonlinear Data to Deep Manifold **

@grok how to apply this in ai training?

Mor - Model Object Reduction, is the group of techniques I started using along with lots of projected geometric expansion styles. I will say using a standardized universal 64 bit tensor hash to represent projected data, and having that projected data 'perspective' be a low-dimensional manifold really allows optimal compression, as long as you follow proper physics and geometry rules. So a quick example, 72 spheres pack optimally into a hexagonal polygon. If you were to wrap a 6 point node stream around a toroid, and divide it into 72 sections, you will get a projected 4-sphere that can convert completely to a square. This square can be a matrix that stores the 2D low dimensional array data. It's all about properly back-tracking the steps with the encoders / decoders.

been a long time since I heard about Isomaps!

super cool and much needed for a relatively un-discussed technique ! do you find people use isomap often enough in interpretability work ?

I was hoping to see it uncured onto a line

See the bottom of the blog haha.

This is a model for politics

Nice! When you don't squeeze the spiral into that shape, PCA handles it perfectly. That doesn't happen with a symmetric spiral.

Great blog! I work on data geometry and I definitely learned a few new things from reading your blog.

Thanks! That means a lot

I’m absolutely terrible at maths but this post was written and presented so well I understood it super well!

Looks so good!👍👍

Easier said than done!

Add Spatiotemporal Multidimensional to this. I've made some similar visualizations to this one.

using nonlinear methods is like giving your analysis x-ray vision. most real-world data hides in weird shapes, written by the nonlinear tools.

very nice work

Thank you!

Open Qs: any way to get at ‘latent space’ elegantly? Or does it always need ‘Big Data’, and even w Big Data does it require tool specialized to that latent space (eg transformer in case of Machine Learning)? Eg What specialized tool might be required to see latent space of Big5?

Intuitive and soothing

3.87d maniford in 7d space

you will never stop those of us who are free to manipulate the system! lie in a two-dimensional manifold tonight!

Nicely written! I was gonna ask if you'd do the same for UMap but you already got that covered haha

How does it relate to UMAP?

@grok how to apply in gis - mainly in creating roads from a raster layer