Checkpoint IPv6 BGP Configuration Challenges and RouteD Crashes08 May 2019
Recently at work we finished going through all the ARIN buerocracy and aquired our own ASN and IPv4 & IPv6 space. This was aquired so that we’ll have the ability to easily run multiple upstream internet providers, potentially across different geographic locations. Our intention was to then use our Checkpoint 5400 firewalls to peer BGP with our upstream providers. In theory this should have worked fine, the 5400s ship with 8GB of RAM and the full IPv4 + IPv6 route tables need about 4GB on their own.
Checkpoint firewalls support BGP by default; the
routed daemon handles all of the dynamic routing protocols that are running on the firewall. An important thing to note about checkpoint firewalls and the routing architecture therein is at the base level it is the Linux kernel running, and it is the kernel IP forwarding that is forwarding traffic through the box. BGP peering is configured through the Gaia web UI and can also be configured from the CLI.
I started with configuring the IPv6 peering first, as I was still waiting on the final approval from ARIN before they would issue the IPv4 space we had requested. Right off the bat I ran into some odd compatibility issues with IPv6. For starters our upstream ISP initially wanted to use a /127 PTP link address for peering. This practice is fully defined in RFCs and supported on most major “real” network hardware vendors (Cisco, Juniper, etc), but unsupported by checkpoint, which meant I had to request our upstream change to a /126 peering link. The second initial issue is that checkpoint does not support md5 authentication for peering. Again I had to request to our upstream to allow an unauthenticated peering session.
With the initial incompatibilities out of the way I got our session established and started getting routes from our upstream. This is where things got weird. Our route table would quickly climb to just over 60,000 routes, the full IPv6 routing table. Within 60 seconds of the session establishing and the routes being received, the BGP session would die, dropping all of the routes from the table and going back to active state. This would happen continuously every 60-120 seconds. Checking the log of the
routed daemon at
/var/log/routed_messages it was full of error messages similar to this:
Dec 19 14:03:39 ERROR: KRT SEND ADD 2620:1c0:61:: mask ffff:ffff:ffff:: router 2606:xxxx:xxxx::5: Cannot allocate memory which was quite surprising as the current memory utilization was less than 50%, and monitoring with top showed no major variation or increase in memory allocation while the daemon was trying to run. After opening a support case with checkpoint and getting nowhere I was able to surmise that the error messages were actually referring to the kernel’s route table, which is separate from the dynamic route table maintained by
routed, but is still updated with all of the routes from
routed's dynamic table. With that in mind I went searching for Linux kernel issues with the IPv6 route table, and right away I was able to find other users encountering similar issues with the inability to handle any significant number of IPv6 routes with the Linux kernel. I was then able to confirm this by running
routed on the firewall and waiting for the route table to fill up. I then tried to manually add an IPv6 route to the kernel:
[Expert@gateway:0]# ip -6 route add 2001:db8::1 via 2606:1234::1 this command exits with the error
RTNETLINK answers: Cannot allocate memory. So it is the kernel that cannot allocate memory not the
routed daemon. One more google search and I found the kernel parameter
net.ipv6.route.max_size which determines that max size of the route table in bytes. Notice that it is a unique parameter for both protocol versions and can have different values. Checking what the default was set to, it was a measly 4096 kilobytes!! In comparision,
net.ipv4.route.max_size is set to 4194304 kilobytes. Thankfully checkpoint allows root access to their firewalls, so it’s an easy fix to up the IPv6 limit to match the IPv4 limit. After running
sysctl -w net.ipv6.route.max_size = 4194304
routed immediately stabilised and the established session stayed online.
It’s clear that checkpoint QA never actually tested their IPv6 functionality to the extent that IPv4 is tested, as this would easily have been caught, and the value that the IPv4 limit is set to is non-standard. So at one point they new they needed to accomodate large routing tables, but they never made the same accomodations for IPv6.