FreeBSD Foundation - 500% if_bridge Performance Improvement

With FreeBSD Foundation grant, Kristof Provost harnesses new parallel techniques to uncork performance bottleneck

Independent embedded systems developer Kristof Provost certainly knows his way around the networking stack. For the past several years, (since George Neville-Neil approached him at AsiaBSDCon with an offer he couldn’t refuse) he has maintained FreeBSD’s port of the OpenBSD Packet Filter (pf) firewall.

Even before this, if you’ve used IPv6, you’ve benefited from Kristof’s work to clean up fragment handling in the firewall. But it’s his 2018 engagement with Orange (France Telecom) that marks the beginning of this particular story.

With 2019 turnover of €42 bn, Orange is on a mission “to ensure that digital services are well thought-out, made available and used in a more caring, inclusive and sustainable way in all areas of our business.”

FreeBSD plays a role in this mission, serving as the OS for some of Orange’s business gateway devices. Olivier Cochard-Labbé , who at the time worked as a Network Engineer at Orange, discovered a pfsync performance issue and recruited Kristof to come up with a fix. Olivier is a recognized networking expert who founded FreeNAS and BSD Router Project and is a FreeBSD port committer. 

“Olivier set this project up for success very well,” said Kristof in an interview for this blog. “He had extensively researched the issue and provided me with benchmarks and flame graphs that really sped up my work.” After a few weeks of coding and testing, and another few for the commit (work that was spread out across about 6 months), Kristof had a patch that doubled pfsync performance.

In the course of his research, Olivier also found some performance issues in if_bridge, the tool that effectively turns a FreeBSD machine into a switch. The FreeBSD Foundation agreed to fund Kristof’s work on if_bridge through aCommunity Grant.

Kristof provides a detailed technical review of his work in the May/June issue of the FreeBSD Journal. For our purposes here, suffice to say the current if_bridge implementation contends heavily on a single BRIDGE_LOCK mutex. Mutexes are a way to ensure data integrity by only allowing one core to access data at a time. Unfortunately, the if_bridge implementation stops cores from doing the things a switch should do (forwarding packets, let’s say) when ANY core is writing. This limits throughput to about 3.7 million packets per second, regardless of the number of cores in the system.

After iterating through some options, Kristof’s ultimate solution takes advantage of epoch (9) in FreeBSD 13 (CURRENT). Through clever use of concurrency, epoch (9) allows the safe use of protected data structures without acquiring a lock (either a mutex or a read/write lock) at all.

As Kristof explains in his article, “The final result is that we can still only perform one modification of the bridge state (e.g. adding an interface, or learning which interface to use for a given MAC address) at a time, but we can keep processing packets on other CPU cores while we do this. This new if_bridge implementation can forward about 18.6 million packets per second, or a 5x improvement.”

On behalf of all the users of if_bridge, the FreeBSD Foundation wishes to give Kristof a hearty “Thank You” for this amazing improvement. For all the details, be sure to check out Kristof’s article in the May/June issue of FreeBSD Journal .