GCE : Flatcar Linux Stable > TroubleShooting Guide.

[decyphertek-io]

Following the commands suggested in these docs are not sufficient , https://www.flatcar.org/docs/latest/installing/cloud/gcp/ , OS Login is not working. I have OS-Login setup and works fine on all other systems. I even added OS Login to the Gcloud CLI command and the config.ign . I have no idea how to SSH in using your docs. ( Note: this works fine in AWS and Azure, not GCP )

Gcloud CLI:

gcloud compute instances create flatcar \
    --image-project kinvolk-public \
    --image-family flatcar-stable \
    --zone us-east1-b \
    --machine-type e2-medium \
    --metadata-from-file user-data=config.ign \
    --metadata enable-os-login=true \
    --network netname \
    --subnet netname-subnet \
    --service-account [email protected] \
    --scopes=https://www.googleapis.com/auth/cloud-platform \
    --tags=flatcar \
    --boot-disk-size 20GB \
    --boot-disk-type pd-standard \
    --boot-disk-device-name flatcar

Butane.bu ( Converted to IGN via butane ):

version: 1.0.0
variant: flatcar
storage:
  files:
    - path: /etc/flatcar/update.conf
      contents:
        inline: |
          SERVER=https://public.update.flatcar-linux.net/v1/update/
          GROUP=stable
          REBOOT_STRATEGY=off

    - path: /opt/.startup.sh
      contents:
        inline: |
          #!/bin/bash
          # Set hostname
          hostnamectl set-hostname decyphertek
          
          # Update Flatcar
          sudo systemctl daemon-reload
          sudo systemctl restart update-engine
          sudo systemctl restart locksmithd
          sudo update_engine_client -update

          # Install Docker and Vim
          sudo rpm-ostree install docker vim

          # Install Docker Compose
          sudo curl -SL https://github.com/docker/compose/releases/download/v2.29.1/docker-compose-linux-x86_64 -o /opt/bin/docker-compose
          sudo chmod +x /opt/bin/docker-compose
          /opt/bin/docker-compose --version

          # Create .docker directory
          mkdir -p /root/.docker

    - path: /etc/systemd/system/startup.service
      contents:
        inline: |
          [Unit]
          Description=Startup Script

          [Service]
          User=root
          Type=oneshot
          ExecStart=/opt/.startup.sh

          [Install]
          WantedBy=multi-user.target

systemd:
  units:
    - name: startup.service
      enabled: true
    - name: oem-gce-enable-oslogin.service
      enabled: true
      mask: false

Install Butane on Debian 12:

# Install Butane
sudo apt update && sudo apt install -y curl
curl -LO https://github.com/coreos/butane/releases/download/v0.22.0/butane-x86_64-unknown-linux-gnu
sudo chmod +x butane-x86_64-unknown-linux-gnu
sudo mv butane-x86_64-unknown-linux-gnu /usr/local/bin/butane

# Convert Butane configuration to Ignition
butane butane.bu -o config.ign

GCP SSH Error:

Connection via Cloud Identity-Aware Proxy Failed
Code: 4003
Reason: failed to connect to backend

Connection to VM is refused.
Please ensure that:
- VM has a firewall rule that allows TCP ingress traffic from the IP range 35.235.240.0/20, port: 22
- SSH daemon on target VM is up and running

You may be able to connect without using the Cloud Identity-Aware Proxy.

Note:

It does work when i launch it from the marketplace . They have a disclaimer and not sure why the glcoud cli version doesnt have that? When launching a GCP Compute instance , I only see Fedora CoreOS , they do not support OS Login. Also, when i launch it from the marketplace , it dont see a way to add the config.ign ? Can I run this directly from terminal, I cant find that anywhere in the docs, like reading hieroglyphs. A lot of info, nothing like a logical progression of how to get it working . Quick start guide for those who dont want to spend a weekend reading tech docs only to find you just need to run a few commands.

[tormath1]

Hi,

I just gave a try to a Stable instance on GCP and it works fine:

$ gcloud compute instances list
NAME                           ZONE           MACHINE_TYPE    PREEMPTIBLE  INTERNAL_IP  EXTERNAL_IP     STATUS
tormath1-793cef4c4e3b3f9f1273  us-central1-a  t2d-standard-1               1.2.3.4   1.2.3.4  RUNNING
$ gcloud compute ssh --zone us-central1-a core@tormath1-793cef4c4e3b3f9f1273
Updating instance ssh metadata...⠛Updated
Updating instance ssh metadata...done.
Waiting for SSH key to propagate.
Flatcar Container Linux by Kinvolk stable 3975.2.2 for Google Compute Engine
core@tormath1-793cef4c4e3b3f9f1273 ~ $

Regarding your parameters:

--subnet netname-subnet \
--network netname \

Are you sure those are reachable from the outside? Note that default network has ingress permissions for SSH which is not the case for custom networks.

Can you try again with --troubleshoot options?

     --troubleshoot
        If you can't connect to a virtual machine (VM) instance using SSH, you
        can investigate the problem using the --troubleshoot flag:

            $ gcloud compute ssh VM_NAME --zone=ZONE \
                --troubleshoot [--tunnel-through-iap]

        The troubleshoot flag runs tests and returns recommendations for four
        types of issues:
        ◆ VM status
        ◆ Network connectivity
        ◆ User permissions
        ◆ Virtual Private Cloud (VPC) settings
        ◆ VM boot

A side note: this won't work on Flatcar, this is a Fedora CoreOS command - docker and vim are by default available on Flatcar:

          # Install Docker and Vim
          sudo rpm-ostree install docker vim

[decyphertek-io]

Thank you for the quick response. I tested again without adding the config.ign and it works. The reason why it is failing is probably due to the rpm-ostree command and a faulty ign . The approach I am using to make the ignition is for the most part correct? I would think that to make it user friendly , Flatcar Linux would automatically convert the yaml config into the right format , would make adoption easier. Why not just include Butane on Flatcar and when setting the metadata , Flatcar would convert automatically. As well as making some common sense quick docs. I also noticed that Flatcar Linux isnt completely immutable. I am able to make changes to the system configs outside of the ignition that are persistent. The /opt/ directory is an example, so in theory if someone gets access they can just launch a binary from the opt directory , modify systemctl services , and crontab as well as update and upgrade strategies. While this makes it easier for users, its is only partially immutable. Why not just make the use of ignition easier and then actually make it immutable? I noticed Google Container OS is more in line with the concept and I was not able to change anything.

[decyphertek-io]

Not sure why an open source distro requires a license? AWS & Azure , I am able to sell VMs on Flatcar Linux. Google appears to be a different story, maybe they are promoting their own Container OS and CoreOS ( Which doestn support OS Login ) ?

{
  "taskReports": [
    {
      "taskType": "CHECK_LICENSE_INFO",
      "createTime": "2024-10-23T16:25:26.741Z",
      "taskExecutionStatus": "ISSUES_FOUND",
      "digest": "One or more licenses are unhealthy.",
      "errorMessage": "Licenses: projects/kinvolk-public/global/licenses/flatcar-container-linux found in the deployment manifest. These licenses are not owned by this Listing and should not be attached to any GCE disks in the deployment"
    }
  ]
}

[tormath1]

Hello @decyphertek-io,

The Ignition approach is almost correct, you could even leverage Ignition / Butane more to remove this startup script and the startup unit:

• docker-compose can be downloaded via Ignition (note: you can use systemd-sysext image for docker-compose: https://github.com/flatcar/sysext-bakery/releases/tag/latest)
• /root/.docker can be created via Ignition
• This section is not required as, by default, update-engine and locksmithd are already running:

          # Update Flatcar
          sudo systemctl daemon-reload
          sudo systemctl restart update-engine
          sudo systemctl restart locksmithd
          sudo update_engine_client -update

• As said, vim and docker are already installed on Flatcar

Regarding the immutability, that's a design choice. Flatcar is an image based OS, all the Flatcar components are delivered via /usr (binaries, libraries, configuration) - this partition will be mounted as read-only with dm-verity protection to ensure integrity of Flatcar base system. The rest (/ without /usr) is read-write, it mainly allows you to keep your data across reboots or Flatcar updates (e.g containers, volumes, etc.) - more on this here: https://www.flatcar.org/docs/latest/reference/developer-guides/sdk-disk-partitions/#mounted-filesystems.

The /opt/ directory is an example, so in theory if someone gets access they can just launch a binary from the opt directory , modify systemctl services , and crontab as well as update and upgrade strategies.

In theory yes, but in practice if someone has already this kind of access, I would say you have bigger problems - to mitigate this, you can use enforced SELinux for example (to prevent /opt binaries to execute stuffs for example).

For the license, I will investigate - but even Open Source software have licenses.

Thanks for your questions!

[tormath1]

@decyphertek-io I can't reproduce the license warning you got by following the documentation - what did you run to get there?

[decyphertek-io]

Google Solution:

"Thank you for your response, As noticed that the error is due to you have not configured the licensed image.Could you please configure the license image as per below link step-6.https://cloud.google.com/marketplace/docs/partners/vm/build-vm-image

gcloud compute images create CREATE_IMAGE_NAME \ 
--project PUBLIC_PROJECT_NAME \ 
--source-disk projects/DEV_PROJECT_NAME/zones/SOURCE_DISK_ZONE/disks/SOURCE_DISK_NAME \ 
--licenses LICENSE_NAME \ 
--description ADD_DESCRIPTION

Once the license image gets created, please use the licensed image during the validation of the deployment package."

Question:

• How would I find the right Flatcar Linux License to add?

[decyphertek-io]

Flatcar Linux CIS Benchmark:

• How would SE Linux be installed on Flatcar Linux?
• How I would make it immutable after I setup: ( System processes, Volumes, DB, & trusted user exceptions. )

sudo chattr +i -R /  

• ACL:

sudo setfacl -m u:username:rwx directoryname

References:

• https://github.com/flatcar/Flatcar/blob/main/CIS/README.md