-
Notifications
You must be signed in to change notification settings - Fork 229
Setup on EC2
Gatekeeper can be run on Amazon Elastic Compute Cloud (EC2).
To create an EC2 instance that can run Gatekeeper, follow these steps from within the EC2 dashboard:
1. Click on "Instances" in the sidebar and then the "Launch Instance" button to create a new instance. Choose an Ubuntu image. Select an instance type that supports the Elastic Network Adapter (ENA). A list of instance types that support the ENA is available here; for example you may choose the m4.16xlarge instance type. Follow any other steps necessary to create the instance.
2. Start the instance.
3. Add two additional interfaces to the instance. From within the EC2 dashboard, choose "Network Interfaces" from the sidebar. Click "Create Network Interface," and choose the same zone (e.g. us-east-1d) as the default interface already attached to the instance. Choose the default security group. Repeat this process so that you have created two new interfaces. Attach the interfaces to the instance by selecting them and then clicking "Attach."
4. SSH into the instance. To get the connection information, go back to the Instances page and click the instance. Then click the "Connect" button. Use the SSH command in a shell to SSH into the instance.
5. Once using a shell within the instance, follow the steps to obtain the necessary dependencies and source code for Gatekeeper. They can be found in the README of the Gatekeeper repository. These steps will have you install dependencies, set up shell variables, setup the Gatekeeper environment, compile Gatekeeper, rebind network devices (see information below about this), and initialize hugepages.
6. We only want to bind two of our interfaces to the DPDK driver. We want to keep the one that is connected to the Internet attached via the ENA driver. Use ifconfig to tell which interface is being used for Internet connectivity (which interface has an IP address), and then bind the other two to the DPDK driver. For example, on one instance, these two interfaces were ens4 and ens5, which had PCI identifiers 00:04.0 and 00:05.0, respectively. Therefore, the commands to bind these were:
sudo dependencies/dpdk/tools/dpdk-devbind.py --bind=igb_uio 00:04.0 sudo dependencies/dpdk/tools/dpdk-devbind.py --bind=igb_uio 00:05.0
7. When initializing hugepages, it seems sufficient to use 2 MB pages. Your mileage may vary between different instance types, but you can try this first:
sudo -i echo 1024 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
mkdir /mnt/huge mount -t hugetlbfs nodev /mnt/huge
The mount point can be made permanent across reboots by adding the following line to the /etc/fstab file:
nodev /mnt/huge hugetlbfs defaults 0 0
8. Run Gatekeeper from the command line:
./build/gatekeeper
After creating an instance using the steps above, if you stop the instance and want to restart it, there are additional steps that you need to take.
1. If you start the machine with the two additional interfaces still attached, the machine will lose Internet connectivity. Therefore, before starting the instance, go to the Interfaces page of the EC2 dashboard and detach both of the extra interfaces. Then, start the instance, and then go back to the Interfaces page to re-add the interfaces.
2. The igb_uio module that is built by the DPDK code does not seem to be automatically loaded, even though the Gatekeeper setup script adds it to be automatically loaded. On restart, load it:
sudo insmod ${RTE_SDK}/build/kmod/igb_uio.ko
This allows us to bind the extra interfaces to the DPDK drivers, which might look like:
sudo dependencies/dpdk/tools/dpdk-devbind.py --bind=igb_uio 00:04.0 sudo dependencies/dpdk/tools/dpdk-devbind.py --bind=igb_uio 00:05.0
3. The number of available hugepages doesn't seem to be persistent, even though the hugepages mount is still in place. So we need to alter the number of available hugepages:
sudo -i echo 1024 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
TODO