Some math/intuitions behind Lindy (thread)

Mostly based on this amazing paper by @tobyordoxford so check it out for detail

What's Lindy? The Lindy effect claims that for many objects, the longer they have existed, the longer they are likely to keep existing Our best guess of their remaining lifespan depends positively on their current age

To handle Lindy more formally, we borrow some tools from survival analysis, namely the survival function and the hazard function (I show visual intuitions later)

The survival function S(t) is the probability that the object is alive at time t, which in turn is the inverse of F(t), the odds that the object is not alive at t

The hazard function lambda(t) is the probability density of the lifespan ending at t, conditional on its survival until then

The relationship between the two is represented in the following way:

If the hazard rate is constant (lambda(t) = k) survival decays exponentially and the representation simplifies to:

i.e. your survival decays exponentially with time: a memoryless distrib where the time remaining before failure is independent of the time that has elapsed until the observation

Clearly then, a constant hazard rate is not Lindy (your survival is independent of how long you have lived) An increasing hazard is anti-Lindy (your survival negatively depends on age) And decreasing hazard is broadly Lindy (your survival positively depends on age)

DISCRETE INTUITIONS FOR THE ABOVE Assume a game where you lose when you flip a fair coin and it lands on tails In this game you have a constant 0.5 hazard rate Your survival rate is clearly:

(Rewriting this to continuous time and taking the limit yields the exact same constant hazard survival rate representation as above) The intuition is also the same: with a fair coin, you are at the intersection between Lindy and anti-Lindy, your odds of losing are always 0.5

This is a visual representation of the borderline Lindy survival distrib -- exponential decay You are playing with a fair 50/50 coin, your hazard rate is always 50% You're neither Lindy nor anti-Lindy

Now imagine a game where every time you flip, the coin biases against you by 10% You are fragile, your odds of losing grow with age. Now you have a skinny-tailed anti-Lindy survival function:

Now imagine the opposite, the more you play, the more the coin biases in your favor Now you're Lindy, your odds of not losing grow with age: You have a heavy-tailed survival function which declines less than exponentially

So far objects with a decaying hazard = Lindy But such objects in the real world are rare, yet the Lindy effect is prevalent. How can this be? Can we have Lindy survival distribs with flat hazard rates? Yes--

THE EPISTEMIC LINDY Imagine now that we are not sure what the hazard rate is in the game, as with most objects in real life We think the coin can have odds of landing on tails anywhere from 0 to 1, all equally likely --a biased coin (uniform prior over [0, 1])

In this case, each flip will not only be used to inform our current odds of flipping unfavourably, but also to inform our assumption of the actual hazard rate -- the more times we flip favourably, the more likely it is that the hazard rate is favourable

In this case, even with a flat hazard rate [0, 1], our survival rate should be heavy-tailed/Lindy, (even though the coin is fair on average) as the simulation below shows

The math here is the following -- first avg all the possible survival curves

As K or t grow, the exponential term becomes trivial and leaves the below A power law distrib, indeed heavier-tailed/more Lindy than the exponential

Interestingly in many cases, even with an increasing hazard rate, the distrib can be Lindy due to the associated uncertainty with the prior...

(I won't go over that for brevity) Epistemic Lindyness is indeed the kind that explains most real world observations of Lindyness

An object is Lindy because as it lasts longer, we grow increasingly convinced that it's robust, though it's not necessarily becoming naturally more robust

Of course in many famous cases of Lindy, like collectives, they do become more robust as they age due to effects like winnowing (defective members fail, robust ones reproduce) and they satisfy the inherent Lindyness covered in the first section

As @tobyordoxford points out, objects like radioisotopes which have a constant hazard rate are not inherently Lindy, but are epistemically so if the hazard rate is unknown

also outlines an alternative approach of portraying Lindy, which I have briefly illustrated in the first post (with an absorbing barrier) It's beautiful but it has a lot of math so I'll have to cover it another time

in simple terms, my intuition is that this method uses an absorbing barrier to showcase that those brownian motions that survive longer have increasing distance between them and the barrier and are hence healthier

what doesn't kill me makes me stronger... -Nietzsche

@__paleologo @quantymacro @bennpeifert @macrocephalopod @conksresearch I wonder if such approach can be used to find mispricings in random coins/some default swaps..