-
Notifications
You must be signed in to change notification settings - Fork 229
Functional Block: CPS
The Control Plane Services (CPS) block has the responsibility of handling all control plane operations for Gatekeeper and Grantor servers. This includes passing any control plane protocol messages to routing daemons running on the host, and accepting routing tables updates from the routing daemons.
Although Gatekeeper servers' main role is only in the data plane, they still need to peer in vantage points and with internal routers of the AS in order to become a node in the incoming and outgoing network paths.
Instead of implementing individual control plane protocols like BGP, OSPF, and IS-IS in Gatekeeper, we enable network operators to run control plane daemons on the host alongside Gatekeeper. This is a win-win scenario, as Gatekeeper avoids the trouble of supporting these protocols directly and network operators do not need to learn another tool to manage their routes. More information about routing daemon companions is available at Routing Daemon Configuration.
The control plane component accomplishes its goal by steering packets from control plane protocols received on the front and back interfaces and sending them to Kernel NIC Interfaces (KNIs) running on Gatekeeper. Routing daemons receive the control plane packets on KNIs, update and compute new routes, and send them back to Gatekeeper through the KNI as well.
The CPS block is allocated one lcore in Gatekeeper.
All static configuration variables can be configured in lua/cps.lua.
These variables are likely to change from deployment-to-deployment based on the operator's preferences.
log_level
The log level for the CPS block. Can be set to any one of the following values: RTE_LOG_EMERG, RTE_LOG_ALERT, RTE_LOG_CRIT, RTE_LOG_ERR, RTE_LOG_WARNING, RTE_LOG_NOTICE, RTE_LOG_INFO, RTE_LOG_DEBUG.
Since we typically use RTE_LOG_ERROR as the most severe log condition, we recommend not to set this value below RTE_LOG_ERROR.
It is not crucial to change these variables, and they only need to be changed to fine tune the performance of Gatekeeper. Otherwise, the default values are likely fine.
mailbox_max_entries_exp
The log (base 2) of the maximum size of the CPS mailbox. For example, if the variable is set to 7, then room for 2^7 = 128 entries will be made in the mailbox.
Also used to determine how many entries will actually be available for use in the mailbox, which for efficiency reasons is one less than the maximum size of the mailbox (127 in the example above).
mailbox_mem_cache_size
Number of mailbox entries to keep in the cache for more efficient use of the mailbox. Set to 0 to disable the cache of the memory pool for the mailbox.
mailbox_burst_size
Maximum number of entries to receive in a burst every time the mailbox is checked.
log_ratelimit_interval_ms
The interval at which logs are rate limited (in milliseconds). For a given interval, only log_ratelimit_burst log entries are permitted. The count of entries is reset for each new interval.
log_ratelimit_burst
The number of entries per interval allowed to be logged. When the number of log entries exceeds this limit in a given interval, the entries will be dropped.
front_max_pkt_burst & back_max_pkt_burst
Maximum number of packets received in each burst on the front and back interfaces, respectively.
The CPS block only receives bursts of packets directly from the interfaces when ntuple filters are available in hardware. Otherwise, packets relevant to the CPS block are received through the CPS mailbox and these variables are not used.
arp_max_entries_exp
This parameter sets the maximum number of parallel ARP address resolutions that users of the KNI interfaces, usually the Linux kernel, can have. This parameter is important, for example, in an Internet exchange at which a large number of peers are constantly being probed.
This parameter is the exponent in the following expression: 2^n - 1. Thus, if this parameter is set to 10, the maximum number of parallel ARP requests is 2^10 - 1 = 1023.
nd_max_entries_exp
This parameter sets the maximum number of parallel ND address resolutions that users of the KNI interfaces, usually the Linux kernel, can have. This parameter is important, for example, in an Internet exchange at which a large number of peers are constantly being probed.
This parameter is the exponent in the following expression: 2^n - 1. Thus, if this parameter is set to 10, the maximum number of parallel ND requests is 2^10 - 1 = 1023.
These variables likely only need to be changed under extreme circumstances or for deployment-specific reasons.
kni_queue_size
This parameter was introduced in version 1.2.
The maximum number of packets in receiving and transmitting queues of a KNI interface. The recommended number is 1024.
num_attempts_kni_link_set
This parameter is available in Gatekeeper versions 1.0 and 1.1. It was removed in version 1.2.
The number of attempts to try to set the KNIs up or down. Because doing so requires a callback through the kernel, and because bringing these interfaces up and down may take time, this can sometimes take multiple attempts. A reasonable value is 5.
max_rt_update_pkts
Maximum number of Netlink route update packets from the routing daemon to serve in a batch. In normal operation there won't often be many route updates at once. A reasonable value is 8.
scan_interval_sec
The time in seconds between scans of outstanding IP address resolution requests. We keep track of which IP address resolution requests the KNI has made, and occasionally scan this list to remove outstanding requests that were not answered. A reasonable value is 5.
kni_kmod_path
This parameter is available in Gatekeeper versions 1.0 and 1.1. It was removed in version 1.2.
The KNI mechanism in DPDK requires the support of a kernel module. If not specified (default), the system automatically searches for the kernel module in /lib/modules/.
If specified, this variable defines where in the filesystem the rte_kni.ko kernel module file is found. It should be an absolute (not relative) path, such as "/home/user/gatekeeper/dependencies/dpdk/build/kmod/rte_kni.ko". Specifying a particular "rte_kni.ko" file in this way could be useful while working to port Gatekeeper to a newer version of DPDK or patching the file rte_kni.ko.
Instead of implementing its own routing protocols, Gatekeeper makes use of standalone routing daemons. One such option is the BIRD Internet Routing Daemon. We maintain a modified version of BIRD 2.0 that is packaged with Gatekeeper (gatekeeper-bird_*_amd64.deb), which allows the daemon to be configured to interact with userspace routing tables (in Gatekeeper) instead of kernel routing tables via Netlink socket port IDs. Note that other routing daemons could be used, but they must have a similar way of interacting with a userspace client.
By default, Gatekeeper listens for routing updates on port 0x6A7E, though this can be changed in the configuration of the CPS block variable nl_pid.
If using the BIRD utility packaged with Gatekeeper, the BIRD configuration files should reference the userspace port that Gatekeeper is using to listen for routing table updates. This port should be mentioned in two places:
- In the device configuration:
protocol device {
...
port 0x6A7E;
...
}
This configures BIRD to interact with a userspace process listening on a socket identified by the given port ID, instead of scanning network interfaces via the kernel.
- In the kernel configuration:
protocol kernel {
...
port 0x6A7E;
...
}
This configures BIRD to interact with a userspace process listening on a socket identified by the given port ID, instead of scanning and sending route updates to the kernel.