Skip navigation
There are many ways to kill performance in a multithreaded program.
0:00 / 26:06

There are many ways to kill performance in a multithreaded program.

Valentin Ignatev

@valigo16K followers

Visit on X
988 likesDownloadOriginal post

53,160 views

There are many ways to kill performance in a multithreaded program. My favorite one is false sharing, because you can get 10x performance just by moving some bytes around. So let's look into it. 00:00 - When multiple threads is not even faster 04:23 - CPU L1 cache primer 09:40 - Getting ALL the performance 15:35 - Looking at disassembler 20:29 - How to find cache line address 22:38 - false sharing in disassembly 25:00 - closing words

21 Comments

Comments from the original post on X (Twitter)

  • Aliaksandr Valialkin's profile picture

    @valyala

    False sharing is the real performance killer on systems with multiple CPU cores. The more CPU cores - the bigger performance penalty is when multiple CPUs access the same cache line. We try hard removing false sharing from hot code paths in VictoriaMetrics products. This helps scaling the performance linearly with the number of available CPU cores. Interesting fact: Go's sync.Pool avoids false sharing -

    10 likes

  • Valentin Ignatev's profile picture

    @valigo

    I've used "cache size" and "cache line size" somewhat lousy here. Every time I say "cache size" assume "cache LINE size" haha. Total size of L1 cache per core is 64KB, but it's irrelevant for the video, so hopefully not too much confusion!

    7 likes

  • حسن | تحت الانشاء's profile picture

    @Hshs_dev

    this guy deserves more, show him some love

    6 likes

  • Valentin Ignatev's profile picture

    @valigo

    Oh wow, didn't know about him! Need to watch the video to see how big overlap is there

  • حسن | تحت الانشاء's profile picture

    @Hshs_dev

    you won't regret it!

    1 like

  • Serhiy's profile picture

    @syshukus

    Extremely quality content!

    3 likes

  • Olex (Solo gamedev Diablo-like) 🇺🇸🇺🇦's profile picture

    @OlexGameDev

    Also when lockless is slower.

    3 likes

  • karakfa's profile picture

    @karakfa

    I'm confused. L1 cache is 32KB. Not 64B!

    1 like

  • Valentin Ignatev's profile picture

    @valigo

    Sorry for the confusion!

  • droug's profile picture

    @entropyreducers

    also an intro to jai in situ. thought it was nice c++ syntax .. jai

    1 like

  • Rudi Servo's profile picture

    @rudiservo

    Awesome video, it reminds me of the Bjarne Stroustrup lecture on why CPU hate jumps.

    1 like

  • PetrosPapa21's profile picture

    @Papa21Petros

    Thank for the video

    1 like

  • Filip Aničić's profile picture

    @anicic_filip

    Wondering if you could answer these

    1 like

  • Valentin Ignatev's profile picture

    @valigo

    First yes, this would also solve this particular example, but I wanted to demonstrate false sharing without super convoluted code. Second - debug symbols are loading, they just aren't as useful because the build is optimized

    1 like

  • Filip Aničić's profile picture

    @anicic_filip

    Ah. Makes sense

    1 like

  • CaptainAmericaTex's profile picture

    @CaptAmericaTx

    Very informative video, thanks! Taking it a step further, the same 64 byte alignment concept would apply when allocating memory in the heap. To my surprise (just checked the docs) malloc only seems to guarantee 16 align. There is a separate posix_memalign that allows 64. 😲

    1 like

  • Circuit Loaf • Aleksandr's profile picture

    @justix_lol

    I was really surprised seeing explicitly cache aligned head/end for an MT queue once, thats how I discovered false sharing issues. Even two simple variables checked from different threads(cores) can lead to significant degradation if false sharing not handled.

    1 like

  • twit2222's profile picture

    @smishy18

    Wow this is great! I’m In an HPC class but ofc we don’t have time to give things this deeply. Where and how can I learn more about this? Watched it end to end and I must now install RAD too lol

    1 like

  • Valentin Ignatev's profile picture

    @valigo

    Idk, I just kinda picked it up out there. Handmade Hero us probably a good start

    1 like

  • Fayyaz Khan's profile picture

    @hey_fayyaz

    learnt a lot from this video alone about multithreading performance. thanks

    1 like