One commonly-stated problem with PPPoE, especially done on x86-based routers like pfSense and OPNsense is they’re “single-threaded”.
The reason why they’re single-threaded is because of how NICs are designed. These NICs know how to sort IPv4 and IPv6 traffic, but not PPP traffic. combined with both Linux and FreeBSD processing PPPoE in the thread that process packets.
However, using virtualization and bridge interfaces (not PCIe passthrough), you can mitigate this issue and get real multi-threaded PPPoE.
So what do you need?
Some routers work very well. Despite the terrible reputation, FreeBSD-based
routers like OPNsense and pfSense can work with multi-threaded PPPoE if your
WAN uses a paravirtualized NIC like
virtio. VMware and Hyper-V NICs should
also work but I haven’t tested those.
However, if you use FreeBSD, you will need the following in your
If you’re using a Linux-based router distro, like OpenWrt or VyOS, you will need to look into Receive Packet Steering, which does the same thing above on Linux.
Some options to avoid include MikroTik CHR and OpenBSD. Both have single-threaded PPPoE due to their design. On MikroTik, you can’t configure Receive Packet Steering that I’m aware of. OpenBSD seems to not have kernel PPPoE or any packet steering option. If you manage to get RPS with CHR, hit me up at neel AT neelc DOT org.
Virtualize & Bridge
Did I mention you need virtualization? Well, you do. And not only that, you also need to use a bridge interface on the PPPoE WAN.
You may be tempted to use PCIe passthrough, but if you’re dealing with PPPoE, don’t. If you do, you will still have a single-threaded WAN.
While I’m no Linux kernel expert, using a bridge interface means the host OS will not process the packet on a single core, but instead just forward it to the bridge which will then balance it across multiple CPUs.
At least on my host, Rocky Linux 9 with KVM, PPPoE is more-or-less evenly balanced with OPNsense. However, I have not tested ESXi, Windows, or FreeBSD virtualization hosts, but I believe it should be the same also.
As a side unrelated note, in the past I was a big FreeBSD user, but moved on over a year ago outside of OPNsense (but it was more desktop-drive for me).
You must use a paravirtualized NIC. On KVM and bhyve, it’s
ESXi it’s `vmxnet``. Hyper-V and Xen also have their own respective
paravirtualized NICs. Using this allows the guest to have multi-threaded PPPoE.
There are a few things to keep in mind with this setup:
- Host CPU usage will be higher than the guest
- Fortunately, it’s via multiple threads (via
- You should disable IPv4 and IPv6 on the PPPoE WAN interface
- If your ISP uses 6rd and you wish to use it, as I do with CenturyLink in Seattle, DO NOT USE Linux-based firewalls like OpenWrt, CHR or VyOS.
- This is because 6rd on Linux-based routers is SLOW when compared to FreeBSD.
- However, a Linux host OS will be fine.
Also, not related to the technicalities of this article, but around August 2024, I will be moving to NYC, so I will be unable to test PPPoE after that. This is because Verizon FiOS doesn’t use PPPoE, neither does Spectrum or Optimum if FiOS is unavailable.
I have multi-threaded PPPoE on the following system:
- HPE ProLiant ML110 Gen11
- CPU: Intel Xeon Scalable 4410Y (intial), 5412U (current)
- Host RAM: 32GB (initial), 64GB (current)
- Guest RAM: 1.5GB RAM
- Host Cores: 12 (initial), 24 (current)
- Guest Cores: 3
- Host OS: Rocky Linux 9.3 (Linux kernel 5.14.0-362.8.1.el9_3)
- Guest OS: OPNsense 23.07 (FreeBSD 13.2-p1)
- Virtualization: KVM
- Host NIC: Broadcom 57416 OCP3 (dual 10GB)
- two bridges: one WAN and one LAN
- Guest NIC:
You could probably use a much smaller RAM allocation if you just do “typical” stuff with a few connections. I have 1.5GB RAM is because of my high-bandwidth Tor relays which really crunch the NAT state table: I have ~65000 states at the time of writing, and it went up to ~78000.
This is what I get:
Look ma, balanced CPU usage.