-
Notifications
You must be signed in to change notification settings - Fork 229
Network
The Gatekeeper network configuration allows the user to specify parameters that are relevant to the interfaces and network in general. Since the network configuration is used by each functional block, some of its state serves as a sort of global configuration for Gatekeeper.
The network configuration allows users to fully specify the Gatekeeper interfaces. Gatekeeper interfaces are composed of one or more component interfaces (or ports). Each Gatekeeper interface is assigned at most one IPv4 address and at most one IPv6 address.
Gatekeeper servers have a front and a back interface. The front interface announces routes via BGP to peers in the vantage point, and accepts incoming traffic on behalf of the protected destination. It then forwards traffic through the back interface, either to a Grantor server, to a gateway to be delivered to another network, or to a neighbor in the same network.
Grantor servers have only a front interface. They accept packets from Gatekeeper servers on the front interface, and for granted packets, they transmit them back through the front interface to the ultimate destination. When run as a Grantor server, any configuration done for the back interface is ignored.
The network configuration allows users to specify other options on Gatekeeper interfaces, such as a VLAN tag, the MTU, and a bonding mode (when more than one port is used to compose a single Gatekeeper interface).
All static configuration variables can be configured in lua/cps.lua.
These variables are likely to change from deployment-to-deployment based on the operator's preferences.
log_level
The log level for network-related logs. Can be set to any one of the following values: RTE_LOG_EMERG, RTE_LOG_ALERT, RTE_LOG_CRIT, RTE_LOG_ERR, RTE_LOG_WARNING, RTE_LOG_NOTICE, RTE_LOG_INFO, RTE_LOG_DEBUG.
Since we typically use RTE_LOG_ERROR as the most severe log condition, we recommend not to set this value below RTE_LOG_ERROR.
user
The user that will run Gatekeeper after it boots. If this parameter is left undefined (i.e. nil), Gatekeeper will run as the user it is called, which is typically the root user.
front_ports & back_ports
A comma-separated string array of interface names to compose the front and back Gatekeeper interfaces, respectively.
Each interface name in the array is used as an index into the interface map specified by if_map.lua, which maps interface names to PCI addresses for use in DPDK. Therefore, to use an interface name in these arrays, it needs to have a mapping in if_map.lua.
When more than one port is specified for an interface, the ports are bonded together. See the bonding mode parameters below for more information.
front_ips & back_ips
A comma-separated string array of IP addresses for the front and back interfaces, respectively.
Each Gatekeeper interface should have at least one IP address. It can have at most one IPv4 address and at most one IPv6 address.
front_bonding_mode & back_bonding_mode
The bonding mode for the front and back ports (when applicable), respectively.
When more than one port is specified in front_ports or back_ports, the ports are bonded together using the DPDK bonding library. When ports are bonded together, a bonding mode is specified to determine how the ports function together. For example, round robin mode provides load balancing across all component ports.
Can be set to any one of the following values: BONDING_MODE_ROUND_ROBIN,BONDING_MODE_ACTIVE_BACKUP, BONDING_MODE_BALANCE, BONDING_MODE_BROADCAST, BONDING_MODE_8023AD, BONDING_MODE_TLB, BONDING_MODE_ALB.
- front_vlan_tag & back_vlan_tag
- front_vlan_insert & back_vlan_insert
Each interface can have a 12-bit VLAN tag {front,back}_vlan_tag applied to all egress traffic if {front,back}_vlan_insert is set to true. If set to false, no tag is applied.
front_mtu & back_mtu
The MTU of the front and back interface, respectively.
Since Gatekeeper encapsulates packets sent to Grantor with a 20 byte outer IP header, the back interface's MTU should be set higher than the front interface if possible.
It is not crucial to change these variables, and they only need to be changed to fine tune the performance of Gatekeeper. Otherwise, the default values are likely fine.
- front_arp_cache_timeout_sec & back_arp_cache_timeout_sec
- front_nd_cache_timeout_sec & back_nd_cache_timeout_sec
The ARP and ND (IPv6 Neighbor Discovery Protocol) caches hold entries that map IP addresses to Ethernet addresses. These caches are occasionally scanned and their entries are marked as stale if their resolution has passed a certain timeout value. A reasonable value is 7200 seconds (2 hours).
- front_num_rx_desc & back_num_rx_desc
- front_num_tx_desc & back_num_tx_desc
A larger queue size can mitigate bursty behavior, but can also increase pressure on the memory caches and can lead to lower performance.
Gatekeeper servers are expected to transmit much fewer packets than they receive, while Grantor servers are expected to transmit about as many packets as they receive.
These variables likely only need to be changed under extreme circumstances or for deployment-specific reasons.
guarantee_random_entropy
In Linux, using /dev/random may require waiting for the result as it uses the so-called entropy pool, where random data may not immediately be available. In contrast, /dev/urandom returns as many bytes as the user requested and thus it is sometimes less random than /dev/random.
The flags parameter in getrandom() will alter the behavior of the call. In the case where flags == 0, getrandom() will block until the /dev/urandom pool has been initialized. Alternatively, the GRND_RANDOM flag bit can be used to switch to the /dev/random pool, subject to the entropy requirements of that pool.
guarantee_random_entropy is used to decide if flag GRND_RANDOM should be passed to calls of getrandom(2). If set to 0, GRND_RANDOM will not be used. Otherwise, it will be. This is relevant for production environments to guarantee entropy while machines are booting up.
num_attempts_link_get; This option was removed in Gatekeeper version 1.2.
The number of attempts to wait for each interface to come up.
Gatekeeper waits for each interface to come up before continuing by calling rte_eth_link_get() and checking its status. This is especially useful for bonded ports where the slaves must be activated after starting the bonded device. The slaves are activated on a timer, so it may take a few seconds. Once the link comes up, the device is ready for full speed RX/TX. Gatekeeper checks whether a link comes up every 1 second.
A reasonable value is 5.
front_ipv6_default_hop_limits & back_ipv6_default_hop_limits
The maximum lifetime of a packet in terms of the number of hops, specified by the "Hop Limit" field in IPv6. It is decremented by 1 by each node that forwards the packet. The packet is discarded if the Hop Limit field is decremented to zero. A reasonable value is 255.
This value is used for packets encapsulated by Gatekeeper in an IPv6 header and sent to Grantor. It is also used for IPv6 policy notification packets that are sent from Grantor to Gatekeeper.
rotate_log_interval_sec
The interval at which the Gatekeeper log is rotated (in seconds). A reasonable value is 3600 (1 hour).
- front_ipv4_hw_udp_cksum & back_ipv4_hw_udp_cksum
- front_ipv6_hw_udp_cksum & back_ipv6_hw_udp_cksum
If hardware support is available for UDP checksumming, these fields can be set to true to offload the checksum computation. Typically, a NIC will support both IPv4 and IPv6, but we have found at least one driver or NIC with issues supporting IPv6 (Amazon's ENA). Therefore, IPv4 and IPv6 UDP checksumming are configured separately.
If these parameters are true, but NICs do not support offloading UDP checksumming, Gatekeeper will issue warnings in the log and will fall back to computing the checksum in software. Thus, unless a NIC shows problems with checksum computation, it is safe to set all these parameters to true.
UDP checksumming is only used by the GT block (in Grantor servers), which only uses a front interface. Therefore, setting the UDP checksumming options for the back interface will currently have no effect.
front_ipv4_hw_cksum & back_ipv4_hw_cksum
Whether the front and back interfaces should use IPv4 checksumming in hardware.
If hardware support is available for IPv4 checksumming, these fields can be set to true to offload the checksum computation.
If these parameters are true, but NICs do not support offloading IPv4 checksumming, Gatekeeper will issue warnings in the log and will fall back to computing the checksum in software. Thus, unless a NIC shows problems with checksum computation, it is safe to set all these parameters to true.
front_alternative_rss_hash & back_alternative_rss_hash; These options were introduced in Gatekeeper version 1.1.
Some NICs do not support the RSS hash functions ETH_RSS_IPV4 amd ETH_RSS_IPV6 (i.e. RSS hash for IPv4 or IPv6 non-fragmented packets). But they may support the hash functions ETH_RSS_NONFRAG_IPV4_TCP, ETH_RSS_NONFRAG_IPV4_UDP, ETH_RSS_NONFRAG_IPV6_TCP, and ETH_RSS_NONFRAG_IPV6_UDP, and setting the input set of the hash these hash functions. Enabling the parameter below, Gatekeeper will try the alternative RSS hash.
If the interface is bonded, all ports in the bond must either need this parameter disabled or enabled.
Currently, this parameter only works for PMD i40e (i.e. DPDK's NIC driver).
Technical details are available at pull request #625.