Introducing Firedancer’s latest technical milestone: fd_quic 🔥💃 A high-performance implementation of the QUIC and Solana transaction ingest network protocols. 1/15

The Solana network uses the QUIC protocol for various purposes, including transaction propagation. We are excited to share large improvements in scalability and robustness since it was first introduced to provide protection against spam attacks. 2/15

Firedancer’s custom networking stack is designed ground up to leverage receive-side scaling, a form of hardware-accelerated load balancing. This allows for a shared-nothing parallel architecture: Each CPU core handles a portion of incoming traffic with minimal overhead. 3/15

Receive-side scaling (RSS), as compared to software-based load balancing, improves throughput and simplifies the implementation: No complex global schedulers or async runtimes, no locks, and no atomics. 4/15

The abstractions introduced by sockets-based networking were another limiting factor. Firedancer bypasses the Linux networking stack using XDP/AF_XDP, recent kernel technologies that are now widely available on existing validator hardware. 5/15

The XDP program consists of eBPF bytecode that identifies and then forwards incoming traffic to Firedancer via AF_XDP. AF_XDP is a facility for high-performance transfer of packet data within the validator. 6/15

QUIC’s 151 pages of spec bring considerable complexity. We were unable to find an existing C library that met our licensing, performance, and reliability requirements. So we built our own. 7/15

To achieve our robustness goal, fd_quic brings optimized data structures and algorithms to the table. For example, it performs (almost) zero heap allocations while handling traffic, guaranteeing that the validator won’t run out-of-memory. 8/15

For further color on the implementation, refer to our header files: Last week, we celebrated our months long journey of making QUIC quick with a live demo showing off our 3 design goals: Compatibility, Performance and Scalability 9/15

Compatibility demo: A spam tool built from modules in the Solana Labs validator that sends out many transactions to the Firedancer validator via QUIC. This showed that our QUIC implementation was correct. 10/15

Performance demo: Testing the limits of a single CPU core running fd_quic using many spammers. The result: 5.8 Gbps or 270k TPS. With small transactions, fd_quic hit 1.4 million TPS. Our spammers required 8 CPU cores to generate this rate. 11/15

Scalability demo: Same as the performance demo, but now RSS is enabled on a 25 Gbps network interface, running fd_quic on 4 CPU cores. This demonstrates the linear scaling properties: Each core still handles about the same rate for a near 4x improvement in throughput. 12/15

A Firedancer QUIC server running on just 4 CPU cores on a single server reliably ingested 21.8 Gbps of incoming transactions (about 1.08M TPS with ~1kB txn size) from Solana Labs clients. 13/15

This test was ultimately limited by the network hardware available (a 25 Gbps link). We would love to see if you can push performance further — the documentation is online. Work on many more components is ongoing. Stay tuned! 14/15

Watch the full demo here: 15/15