Nitya

@nityasnotes • 7,106 subscribers

in hiding

Shorts

nityas notes dot com a place where I will keep track of what I’m reading and writing

nityas notes dot com a place where I will keep track of what I’m reading and writing

25,661 Aufrufe

Processing and indexing the web is no small task. Heterogenous content, varying update frequency, compounding derived artifacts, sheer volume. I wrote about our internal data framework exa-d that handles this scaled complexity: exa-d organizes data as typed columns with explicit dependencies. Missing or stale columns signal what needs computation, the system computes only what is necessary and converges deterministically on correct state.

Processing and indexing the web is no small task. Heterogenous content, varying update frequency, compounding derived artifacts, sheer volume. I wrote about our internal data framework exa-d that handles this scaled complexity: exa-d organizes data as typed columns with explicit dependencies. Missing or stale columns signal what needs computation, the system computes only what is necessary and converges deterministically on correct state.

17,345 Aufrufe