diff --git a/content/docs/latest/setup/storage/lvm.md b/content/docs/latest/setup/storage/lvm.md new file mode 100644 index 00000000..5f988eb5 --- /dev/null +++ b/content/docs/latest/setup/storage/lvm.md @@ -0,0 +1,333 @@ +--- +title: Use LVM on Flatcar Container Linux +linktitle: Using LVM +description: How to use LVM to extend a logical volume across multiple disks +weight: 30 +aliases: +- ../../os/lvm +- ../../clusters/management/lvm +--- + +LVM - Logical Volume Management - allows you to create logical volumes, for example to use multiple physical disks as +one volume. This allows you to make the full use of all attached disks. + +Flatcar Linux has built-in support for LVM. +This guide covers creation of logical volumes using LVM and how to use them. + + +## Creating LVM + +There are two main ways to do this: create everything manually or use an Ignition config. We will first cover +the manual way to get a better grip of what is happening, then we will cover the Ignition way. + +### Manual + +You can find all volumes using the `lsblk` command. For example: + +```shell + +# lsblk +NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS +loop3 7:3 0 68.3M 1 loop +loop4 7:4 0 39.3M 1 loop +loop5 7:5 0 4K 1 loop +sda 8:0 0 223.6G 0 disk +|-sda1 8:1 0 128M 0 part +|-sda2 8:2 0 2M 0 part +|-sda3 8:3 0 1G 0 part +| `-usr 254:0 0 1016M 1 crypt /usr +|-sda4 8:4 0 1G 0 part +|-sda6 8:6 0 128M 0 part /oem +|-sda7 8:7 0 64M 0 part +`-sda9 8:9 0 221.3G 0 part / +sdb 8:16 0 447.1G 0 disk +sdc 8:32 0 447.1G 0 disk +sdd 8:48 0 223.6G 0 disk + +``` + +Now we know that we have `/dev/sda`, `/dev/sdb`, `/dev/sdc`, `/dev/sdd` available. However, we cannot use `/dev/sda` in +this scenario, but we can work with the others. + + +```shell +# pvcreate /dev/sdb /dev/sdc /dev/sdd + WARNING: Failed to connect to lvmetad. Falling back to device scanning. + Physical volume "/dev/sdb" successfully created. + Physical volume "/dev/sdc" successfully created. + Physical volume "/dev/sdd" successfully created. +``` + +You can verify that everything worked with the following commands: + +```shell +# pvs + PV VG Fmt Attr PSize PFree + /dev/sdb lvm2 --- 447.13g 447.13g + /dev/sdc lvm2 --- 447.13g 447.13g + /dev/sdd lvm2 --- 223.57g 223.57g + +# pvdisplay + "/dev/sdd" is a new physical volume of "223.57 GiB" + --- NEW Physical volume --- + PV Name /dev/sdd + VG Name + PV Size 223.57 GiB + Allocatable NO + PE Size 0 + Total PE 0 + Free PE 0 + Allocated PE 0 + PV UUID x0N0k2-5c6j-HlHZ-vuAX-s82V-Yx8v-Pi3rWa + + "/dev/sdc" is a new physical volume of "447.13 GiB" + --- NEW Physical volume --- + PV Name /dev/sdc + VG Name + PV Size 447.13 GiB + Allocatable NO + PE Size 0 + Total PE 0 + Free PE 0 + Allocated PE 0 + PV UUID dRa71o-rYk9-gJKC-bJdC-tJVV-yarW-LHTwPu + + "/dev/sdb" is a new physical volume of "447.13 GiB" + --- NEW Physical volume --- + PV Name /dev/sdb + VG Name + PV Size 447.13 GiB + Allocatable NO + PE Size 0 + Total PE 0 + Free PE 0 + Allocated PE 0 + PV UUID vu98O9-4UDD-TTGK-PvsY-g4bN-FeL4-lcqLy7 + +``` + +As you can see, you do not yet have a virtual group. You use the `vgcreate` command to create one and add your PVs. + +You need to specify the name of the group and the volumes you want to add to it like so: + +```shell +# vgcreate base-layer /dev/sdb /dev/sdc /dev/sdd + Volume group "base-layer" successfully created + +``` + + +Now you can go ahead and create a logical volume. We recommend to name it according to its purpose as in this example: + +```shell +# lvcreate -n vol_docker -l 100%FREE base-layer + Logical volume "vol_docker" created. +``` + +You can verify that everything worked well by issuing the following command: + +```shell +# lvdisplay + --- Logical volume --- + LV Path /dev/base-layer/vol_docker + LV Name vol_docker + VG Name base-layer + LV UUID d0ne0u-zBZQ-29f5-rkd9-XnZv-0vhE-rGmLvA + LV Write Access read/write + LV Creation host, time rrackow-test, 2024-09-18 06:42:09 +0000 + LV Status available + # open 0 + LV Size 1.09 TiB + Current LE 286163 + Segments 3 + Allocation inherit + Read ahead sectors auto + - currently set to 256 + Block device 254:1 +``` + +As you can see we now have a total size the sum of the individual disks. + +Next we need to use `mkfs` to create an `ext4` filesystem: + +```shell +# mkfs.ext4 /dev/base-layer/vol_docker +mke2fs 1.47.0 (5-Feb-2023) +Discarding device blocks: done +Creating filesystem with 293030912 4k blocks and 73261056 inodes +Filesystem UUID: eed7e226-87f8-40e0-a49b-21eae4ef9620 +Superblock backups stored on blocks: + 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, + 4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968, + 102400000, 214990848 + +Allocating group tables: done +Writing inode tables: done +Creating journal (262144 blocks): done +Writing superblocks and filesystem accounting information: done +``` + + +Now you can for example mount the volume for use with docker, by mounting it to `/var/lib/docker` like so: + +```shell +# mkdir /var/lib/docker +# mount /dev/base-layer/vol_docker /var/lib/docker +``` + +### Ignition + + +In your Ignition config you will need two units: one to create the volume group and one to mount the volume. +Additionally, you will also need a script that executes all the required commands. + +We will start with the script. It basically packages everything from the manual part into a script like so: + +```Bash +#!/bin/bash +set -euo pipefail + + +# Function to find all disks +find_volumes(){ + lsblk -d -o NAME,TYPE | awk '$2 == "disk" {print "/dev/" $1}' +} + +disks=$(find_volumes) + +# Create Physical Volumes +pvcreate "${disks}" + +# Create Volume Group +vgcreate vg-root "${disks}" + +# Create Logical Volume for data +lvcreate -n vol_root -l 100%FREE vg-root + +# Format the data volume with ext4 filesystem +mkfs.ext4 /dev/vg-root/vol_root +``` + +As you can see, we use a function to list **all** available disks. If you do not want to use all disks, you need to adjust +the script accordingly. If you want to mount to a different place, e.g. `/var/lib/docker` instead of `/`, you need to adjust this +bit as well. + +The next step is to create the unit file that executes the script: + +```Bash +[Unit] +Description=LVM Setup +ConditionFirstBoot=yes +Before=local-fs-pre.target +[Service] +Type=oneshot +Restart=on-failure +RemainAfterExit=yes +ExecStart=/etc/systemd/multi-user.target/lvm.sh #This is the name and path of the file above +[Install] +WantedBy=multi-user.target +``` + +However, we also need to mount the volume we created: + +```Bash +[Unit] +Description=LVM Mount +[Mount] +What=/dev/vg-root/vol_root +Where=/ +Type=ext4 +Options=defaults +[Install] +WantedBy=local-fs.target +``` + +Now we need to put it all together into a butane yaml: + +```yaml +variant: flatcar +version: 1.0.0 +systemd: + units: + - name: lvm-setup.service + enabled: true + contents: | + [Unit] + Description=LVM Setup + ConditionFirstBoot=yes + DefaultDependencies=no + Before=local-fs-pre.target + [Service] + Type=oneshot + Restart=on-failure + RemainAfterExit=yes + ExecStart=/opt/lvm.sh + [Install] + WantedBy=multi-user.target + - name: var-lib-docker.mount + enabled: true + contents: | + [Unit] + Description=Mount LVM to docker dir + After=lvm-setup.service + [Mount] + What=/dev/vg-docker/vol_docker + Where=/var/lib/docker + Type=ext4 + Options=defaults + [Install] + WantedBy=local-fs.target + - name: docker.service + dropins: + - name: 10-wait-docker.conf + contents: | + [Unit] + After=var-lib-docker.mount + Requires=var-lib-docker.mount +storage: + files: + - path: /opt/lvm.sh + mode: 0744 + contents: + inline: | + #!/bin/bash + set -euo pipefail + + + # Function to find all disks + find_volumes(){ + lsblk -d -o NAME,TYPE | awk '$2 == "disk" {print "/dev/" $1}' + } + + disks=$(find_volumes) + + # Create Physical Volumes + pvcreate "${disks}" + + # Create Volume Group + vgcreate vg-root "${disks}" + + # Create Logical Volume for data + lvcreate -n vol_root -l 100%FREE vg-root + + # Format the data volume with ext4 filesystem + mkfs.ext4 /dev/vg-root/vol_root +``` + + +As mentioned before, we need to still transpile from a butane yaml to an Ignition config like so: + +```Bash +$ docker run --rm -i quay.io/coreos/butane:latest < lvm.yaml > ignition.json +``` + +You can verify your config with the following: + +```Bash +$ cat ignition.json +{"ignition":{"version":"3.3.0"},"storage":{"files":[{"path":"/etc/systemd/system/multi-user.target.wants/lvm.sh","contents":{"compression":"gzip","source":"data:;base64,H4sIAAAAAAAC/3ySQY/TPhTE7+9TzN/Nn20lQrYrbqsiIdTdC6AVQkicKjd5bqy4doidLFW33x05rtosB47Om/mN38Sz/4qttsVW+po8B+TcO7S6ZSW1IaIZHnpbBu0sgoPStoI0BpX2jad43AzO9Hv288WRAOO3pkFeIXf4+vHL+u33n09rvEA+N7jJ7rBaQUSvwLHttA0QRcVDIZAtTzegE1HZsQy8aQc/XyAix6hVNp+GLQjovdwa3qSxEESAct0oh7bIxsE9KkcAoBXaIbGRH9IUdx/eLPGCXcct8l8Qvi9L9l71xhyQxJW4R6jZjpC/Q7PpMTHFKFQ6XtxZjrfisnZ4JY17zvDIIaJhtA9w6sw+V/sqKJtfW1lE66e0yFN98LqUBj9SLXRZMTtOAaeJJ0nx2Lm+pWF31g+7vHJlw90/nJ/dbhKWypZBkrn0ajE4szmDcoPl7e3/D9/W6yt+fFCu28u0evQj/VM861CDf4f3UNqwP/jAe9o3yr8bP47v5MIprkH0JwAA//+D2TD6wwIAAA=="},"mode":484}]},"systemd":{"units":[{"contents":"[Unit]\nDescription=LVM Setup\nConditionFirstBoot=yes\nBefore=local-fs-pre.target\n[Service]\nType=oneshot\nRestart=on-failure\nRemainAfterExit=yes\nExecStart=/etc/systemd/system/multi-user.target.wants/lvm.sh\n[Install]\nWantedBy=multi-user.target\n","enabled":true,"name":"lvm-setup.service"},{"contents":"[Unit]\nDescription=Mount LVM to docker dir\n[Mount]\nWhat=/dev/vg-docker/vol_docker\nWhere=/var/lib/docker\nType=ext4\nOptions=defaults\n[Install]\nAfter=lvm-setup.service\nWantedBy=local-fs.target\n","enabled":true,"name":"var-lib-docker.mount"},{"dropins":[{"contents":"[Unit]\nAfter=var-lib-docker.mount\nRequires=var-lib-docker.mount\n","name":"10-wait-docker.conf"}],"name":"docker.service"}]}} +``` + +Add this Ignition config to your cloud provider of choice now as user-data and create the given instance. + +[Butane Config Transpiler]: https://www.flatcar.org/docs/latest/provisioning/config-transpiler/