Merge pull request #394 from juliuskoskela/bpmp-virt

Boot and power management virtualization for Nvidia Jetson
tiiuae · Jan 7, 2024 · 9305441 · 9305441
2 parents 6a69188 + da167c6
commit 9305441
Show file tree

Hide file tree

Showing 17 changed files with 1,989 additions and 1 deletion.
diff --git a/docs/src/SUMMARY.md b/docs/src/SUMMARY.md
@@ -39,6 +39,7 @@
         - [NVIDIA Jetson AGX Orin: UART Passthrough](technologies/nvidia_agx_pt_uart.md)
         - [NVIDIA Jetson AGX Orin: PCIe Passthrough](technologies/nvidia_agx_pt_pcie.md)
         - [Generic x86: PCIe Passthrough on crosvm](technologies/x86_pcie_crosvm.md)
+    - [Platform Bus Virtualization - NVIDIA BPMP](technologies/nvidia_virtualization_bpmp.md)
     - [Hypervisor Options](technologies/hypervisor_options.md)
 
 # Build System and Supply Chain

diff --git a/docs/src/technologies/nvidia_virtualization_bpmp.md b/docs/src/technologies/nvidia_virtualization_bpmp.md
@@ -0,0 +1,105 @@
+<!--
+    Copyright 2022-2023 TII (SSRC) and the Ghaf contributors
+    SPDX-License-Identifier: CC-BY-SA-4.0
+-->
+
+# NVIDIA Jetson AGX Orin: Boot and Power Management Processor Virtualization
+
+## Introduction
+
+Boot and Power Management Processor (BPMP) virtualization on the NVIDIA Jetson AGX Orin involves enabling virtual machines (VMs) to access specific BPMP resources. This capability is crucial for passing through platform devices where control over resets and clocks configurations is required.
+
+## Architectural Overview
+
+- **Resource Access**: BPMP virtualization allows VMs to access and manage resources such as device clocks and resets.
+- **Foundation for Device Virtualization**: This setup lays the groundwork for future virtualization of more complex devices like GPUs.
+- **Module Introduction**: A new `virtualization` module is introduced, divided into `common` and `host` modules, with a plan to add a `guest` module for NixOS-based guests.
+- **Device Tree Configurations**: Modifications are made via patching to support virtualization features.
+- **Compatibility**: The current implementation supports a Ghaf host with an Ubuntu guest.
+
+## Use Cases
+
+The current implementation includes a host configuration for UARTA
+passthrough as a test case, demonstrating the practical application of
+BPMP virtualization. Current implementation still requires a manuall
+built Ubuntu guest. Work continues to integrate `microvm.nix` declared
+guest that supports NVIDIA BPMP virtualization with the UARTA
+passthrough demo. In general, this is work is important for future
+NVIDIA Jetson platform bus GPU passthrough. With this feature, it is
+possible to virtualize NVIDIA Jetson integrated GPU connected to
+platform bus.
+
+## Instructions for Using BPMP Virtualization Options on NVIDIA Jetson AGX Orin
+
+1. Enable NVIDIA BPMP virtualization on Ghaf host for a NVIDIA
+Jetson-target using the following configuration options:
+
+
+```nix
+  hardware.nvidia = {
+    virtualization.enable = true;
+    passthroughs.uarta.enable = true;
+};
+```
+Please note that these options are integrated to [NVIDIA Jetson Orin
+  targets](https://github.com/tiiuae/ghaf/blob/main/targets/nvidia-jetson-orin/default.nix)
+  but disabled by default until the implementation is finished.
+
+2. Build the target and boot with the image. You can write the image
+to an SSD for testing with a recent NVIDIA UEFI FW.
+
+## Testing
+
+### Host
+
+1. Check for `bpmp-host` device.
+
+```
+[ghaf@ghaf-host:~]$ ls /dev | grep bpmp-host
+bpmp-host
+```
+
+2. Check that `vfio-platform` binding is successful.
+
+```
+ghaf@ghaf-host:~]$ ls -l /sys/bus/platform/drivers/vfio-platform/3100000.serial
+lrwxrwxrwx 1 root root 0 Dec  8 08:26 /sys/bus/platform/drivers/vfio-platform/3100000.serial -> ../../../../devices/platform/3100000.serial
+```
+
+### Guest for UARTA Test
+
+1. Build guest kernel according to instructions at [https://github.com/jpruiz84/bpmp-virt](https://github.com/jpruiz84/bpmp-virt) and use the following script to start the vm (IMG is the kernel image and FS the rootfs).
+
+```
+IMG=$1
+FS=$2
+
+qemu-system-aarch64 \
+    -nographic \
+    -machine virt,accel=kvm \
+    -cpu host \
+    -m 1G \
+    -no-reboot \
+    -kernel $IMG \
+    -drive file=$FS,if=virtio,format=qcow2 \
+    -net user,hostfwd=tcp::2222-:22 -net nic \
+    -device vfio-platform,host=3100000.serial \
+    -dtb virt.dtb \
+    -append "rootwait root=/dev/vda console=ttyAMA0"
+```
+
+2. With UARTA connected as instructed in [https://github.com/jpruiz84/bpmp-virt](https://github.com/jpruiz84/bpmp-virt), start minicom on the working PC:
+
+```
+minicom -b 9600 -D /dev/ttyUSB0
+```
+
+3. Test UARTA by echoing a string to the correct tty in the vm:
+
+```
+echo 123 > /dev/ttyTHS0
+```
+
+## Related Topics
+
+- [NVIDIA Jetson AGX Orin: UART Passthrough](https://tiiuae.github.io/ghaf/technologies/nvidia_agx_pt_uart.html#nvidia-jetson-agx-orin-uart-passthrough)
diff --git a/hydrajobs.nix b/hydrajobs.nix
@@ -1,6 +1,19 @@
 # Copyright 2022-2024 TII (SSRC) and the Ghaf contributors
 # SPDX-License-Identifier: Apache-2.0
-{self, ...}: {
+{self, ...}: let
+  mkBpmpEnabled = cfg: let
+    bpmpEnableModule = {lib, ...}: {
+      ghaf.hardware.nvidia = {
+        virtualization.enable = lib.mkForce true;
+        virtualization.host.bpmp.enable = lib.mkForce true;
+        passthroughs.host.uarta.enable = lib.mkForce true;
+      };
+    };
+    newCfg = cfg.extendModules {modules = [bpmpEnableModule];};
+    package = newCfg.config.system.build.${newCfg.config.formatAttr};
+  in
+    package;
+in {
   flake.hydraJobs = {
     generic-x86_64-debug.x86_64-linux = self.packages.x86_64-linux.generic-x86_64-debug;
     lenovo-x1-carbon-gen11-debug.x86_64-linux = self.packages.x86_64-linux.lenovo-x1-carbon-gen11-debug;
@@ -19,5 +32,11 @@
     # Build also cross-compiled images without demo apps
     nvidia-jetson-orin-agx-debug-nodemoapps-from-x86_64.x86_64-linux = self.packages.x86_64-linux.nvidia-jetson-orin-agx-debug-nodemoapps-from-x86_64;
     nvidia-jetson-orin-nx-debug-nodemoapps-from-x86_64.x86_64-linux = self.packages.x86_64-linux.nvidia-jetson-orin-nx-debug-nodemoapps-from-x86_64;
+
+    # BPMP virt enabled versions
+    nvidia-jetson-orin-agx-debug-bpmp.aarch64-linux = mkBpmpEnabled self.nixosConfigurations.nvidia-jetson-orin-agx-debug;
+    nvidia-jetson-orin-nx-debug-bpmp.aarch64-linux = mkBpmpEnabled self.nixosConfigurations.nvidia-jetson-orin-nx-debug;
+    nvidia-jetson-orin-agx-debug-bpmp-from-x86_64.x86_64-linux = mkBpmpEnabled self.nixosConfigurations.nvidia-jetson-orin-agx-debug-from-x86_64;
+    nvidia-jetson-orin-nx-debug-bpmp-from-x86_64.x86_64-linux = mkBpmpEnabled self.nixosConfigurations.nvidia-jetson-orin-nx-debug-from-x86_64;
   };
 }
diff --git a/modules/hardware/nvidia-jetson-orin/default.nix b/modules/hardware/nvidia-jetson-orin/default.nix
@@ -13,5 +13,6 @@
     ./nx-netvm-ethernet-pci-passthrough.nix
 
     ./ota-utils-fix.nix
+    ./virtualization
   ];
 }
diff --git a/modules/hardware/nvidia-jetson-orin/virtualization/common/bpmp-virt-common/default.nix b/modules/hardware/nvidia-jetson-orin/virtualization/common/bpmp-virt-common/default.nix
@@ -0,0 +1,63 @@
+# Copyright 2022-2023 TII (SSRC) and the Ghaf contributors
+# SPDX-License-Identifier: Apache-2.0
+{
+  lib,
+  config,
+  ...
+}: let
+  cfg = config.ghaf.hardware.nvidia.virtualization;
+in {
+  options.ghaf.hardware.nvidia.virtualization.enable = lib.mkOption {
+    type = lib.types.bool;
+    default = false;
+    description = ''
+      Enable virtualization support for NVIDIA Orin
+
+      This option is an implementation level detail and is toggled automatically
+      by modules that need it. Manually enabling this option is not recommended in
+      release builds.
+    '';
+  };
+
+  config = lib.mkIf cfg.enable {
+    boot.kernelPatches = [
+      {
+        name = "Added Configurations to Support Vda";
+        patch = null;
+        extraStructuredConfig = with lib.kernel; {
+          PCI_STUB = lib.mkDefault yes;
+          VFIO = lib.mkDefault yes;
+          VIRTIO_PCI = lib.mkDefault yes;
+          VIRTIO_MMIO = lib.mkDefault yes;
+          HOTPLUG_PCI = lib.mkDefault yes;
+          PCI_DEBUG = lib.mkDefault yes;
+          PCI_HOST_GENERIC = lib.mkDefault yes;
+          VFIO_IOMMU_TYPE1 = lib.mkDefault yes;
+          HOTPLUG_PCI_ACPI = lib.mkDefault yes;
+          PCI_HOST_COMMON = lib.mkDefault yes;
+          VFIO_PLATFORM = lib.mkDefault yes;
+          TEGRA_BPMP_GUEST_PROXY = lib.mkDefault no;
+          TEGRA_BPMP_HOST_PROXY = lib.mkDefault no;
+        };
+      }
+      {
+        name = "Vfio_platform Reset Required False";
+        patch = ./patches/0002-vfio_platform-reset-required-false.patch;
+      }
+      {
+        name = "Bpmp Support Virtualization";
+        patch = ./patches/0003-bpmp-support-bpmp-virt.patch;
+      }
+      {
+        name = "Bpmp Virt Drivers";
+        patch = ./patches/0004-bpmp-virt-drivers.patch;
+      }
+      {
+        name = "Bpmp Overlay";
+        patch = ./patches/0005-bpmp-overlay.patch;
+      }
+    ];
+
+    boot.kernelParams = ["vfio_iommu_type1.allow_unsafe_interrupts=1"];
+  };
+}
diff --git a/...tualization/common/bpmp-virt-common/patches/0002-vfio_platform-reset-required-false.patch b/...tualization/common/bpmp-virt-common/patches/0002-vfio_platform-reset-required-false.patch
@@ -0,0 +1,24 @@
+From 03c7ecfd16e46838ab70c43a18990ef3dd35d08c Mon Sep 17 00:00:00 2001
+From: Juan Pablo Ruiz <[email protected]>
+Date: Thu, 4 May 2023 12:19:37 +0400
+Subject: [PATCH 2/3] vfio_platform: reset required false
+
+---
+ drivers/vfio/platform/vfio_platform.c | 2 +-
+ 1 file changed, 1 insertion(+), 1 deletion(-)
+
+diff --git a/drivers/vfio/platform/vfio_platform.c b/drivers/vfio/platform/vfio_platform.c
+index 1e2769010089..3eabe37f400d 100644
+--- a/drivers/vfio/platform/vfio_platform.c
++++ b/drivers/vfio/platform/vfio_platform.c
+@@ -15,7 +15,7 @@
+ #define DRIVER_AUTHOR   "Antonios Motakis <[email protected]>"
+ #define DRIVER_DESC     "VFIO for platform devices - User Level meta-driver"
+
+-static bool reset_required = true;
++static bool reset_required = false;
+ module_param(reset_required, bool, 0444);
+ MODULE_PARM_DESC(reset_required, "override reset requirement (default: 1)");
+
+-- 
+2.25.1
diff --git a/...son-orin/virtualization/common/bpmp-virt-common/patches/0003-bpmp-support-bpmp-virt.patch b/...son-orin/virtualization/common/bpmp-virt-common/patches/0003-bpmp-support-bpmp-virt.patch
@@ -0,0 +1,153 @@
+diff --git a/drivers/firmware/tegra/bpmp-tegra186.c b/drivers/firmware/tegra/bpmp-tegra186.c
+index f1a7ffa1b8dc..4d34a916f4ec 100644
+--- a/drivers/firmware/tegra/bpmp-tegra186.c
++++ b/drivers/firmware/tegra/bpmp-tegra186.c
+@@ -30,6 +30,9 @@ struct tegra186_bpmp {
+ 	} mbox;
+ };
+
++extern uint64_t bpmp_vpa;
++int tegra_bpmp_guest_init(void);
++
+ static inline struct tegra_bpmp *
+ mbox_client_to_bpmp(struct mbox_client *client)
+ {
+@@ -177,7 +180,17 @@ static int tegra186_bpmp_init(struct tegra_bpmp *bpmp)
+ 	bpmp->priv = priv;
+ 	priv->parent = bpmp;
+
+-	priv->tx.pool = of_gen_pool_get(bpmp->dev->of_node, "shmem", 0);
++#ifdef CONFIG_TEGRA_BPMP_GUEST_PROXY
++	// If virtual-pa node is defined, it means that we are using a virtual BPMP
++	// then we have to initialize the bpmp-guest
++	err = of_property_read_u64(bpmp->dev->of_node, "virtual-pa", &bpmp_vpa);
++	if(!err){
++		printk("BPMP virtual-pa: 0x%llX", bpmp_vpa);	
++		return tegra_bpmp_guest_init();
++	}
++#endif
++ 
++ 	priv->tx.pool = of_gen_pool_get(bpmp->dev->of_node, "shmem", 0);
+ 	if (!priv->tx.pool) {
+ 		dev_err(bpmp->dev, "TX shmem pool not found\n");
+ 		return -EPROBE_DEFER;
+@@ -267,6 +280,11 @@ static void tegra186_bpmp_deinit(struct tegra_bpmp *bpmp)
+ 	struct tegra186_bpmp *priv = bpmp->priv;
+ 	unsigned int i;
+
++	// If using BPMP guest proxy, do no deinit the module
++	if(bpmp_vpa){
++		return;
++	}
++
+ 	mbox_free_channel(priv->mbox.channel);
+
+ 	for (i = 0; i < bpmp->threaded.count; i++)
+diff --git a/drivers/firmware/tegra/bpmp.c b/drivers/firmware/tegra/bpmp.c
+index c7e39db14dac..802c2f0f7cf6 100644
+--- a/drivers/firmware/tegra/bpmp.c
++++ b/drivers/firmware/tegra/bpmp.c
+@@ -40,6 +40,18 @@ channel_to_ops(struct tegra_bpmp_channel *channel)
+ 	return bpmp->soc->ops;
+ }
+
++struct tegra_bpmp *tegra_bpmp_host_device = NULL;
++EXPORT_SYMBOL_GPL(tegra_bpmp_host_device);
++
++int (*tegra_bpmp_transfer_redirect)(struct tegra_bpmp *bpmp,
++			struct tegra_bpmp_message *msg) = NULL;
++int tegra_bpmp_outloud = 0;
++uint64_t bpmp_vpa = 0;
++
++EXPORT_SYMBOL_GPL(tegra_bpmp_transfer_redirect);
++EXPORT_SYMBOL_GPL(tegra_bpmp_outloud);
++EXPORT_SYMBOL_GPL(bpmp_vpa);
++
+ struct tegra_bpmp *tegra_bpmp_get(struct device *dev)
+ {
+ 	struct platform_device *pdev;
+@@ -65,6 +77,7 @@ struct tegra_bpmp *tegra_bpmp_get(struct device *dev)
+
+ put:
+ 	of_node_put(np);
++	tegra_bpmp_host_device = bpmp;
+ 	return bpmp;
+ }
+ EXPORT_SYMBOL_GPL(tegra_bpmp_get);
+@@ -315,6 +328,30 @@ static ssize_t tegra_bpmp_channel_write(struct tegra_bpmp_channel *channel,
+ 	return __tegra_bpmp_channel_write(channel, mrq, flags, data, size);
+ }
+
++int _tegra_bpmp_transfer(struct tegra_bpmp *bpmp,
++	struct tegra_bpmp_message *msg)
++{
++	int err = 0;
++
++	// vadikas -- redirect request to virtio module
++	// the tegra_bpmp_transfer_redirect code is in bpmp-virt overlay
++	if (tegra_bpmp_outloud){
++	        printk("tegra_bpmp_transfer_redirect tx: %x tx.size= %ld \n", 
++				msg->mrq, msg->tx.size);
++	        print_hex_dump(KERN_INFO, "tegra_bpmp_transfer_redirect tx:",
++				DUMP_PREFIX_NONE, 16, 1, msg->tx.data, msg->tx.size, false);
++	}
++	err = (*tegra_bpmp_transfer_redirect)(bpmp, msg);
++	
++	if (tegra_bpmp_outloud){
++	        printk("tegra_bpmp_transfer_redirect rx: err=%d\n msg->rx.ret=%d", 
++				err, msg->rx.ret);
++	        print_hex_dump(KERN_INFO, "tegra_bpmp_transfer_redirect rx:" ,
++				DUMP_PREFIX_NONE, 16, 1, msg->rx.data, msg->rx.size, false);
++	}
++	return err; 
++}
++
+ int tegra_bpmp_transfer_atomic(struct tegra_bpmp *bpmp,
+ 			       struct tegra_bpmp_message *msg)
+ {
+@@ -331,6 +368,10 @@ int tegra_bpmp_transfer_atomic(struct tegra_bpmp *bpmp,
+
+ 	spin_lock(&bpmp->atomic_tx_lock);
+
++	// vadikas -- redirect request to virtio module
++	if (tegra_bpmp_transfer_redirect)
++		return _tegra_bpmp_transfer(bpmp, msg);
++
+ 	err = tegra_bpmp_channel_write(channel, msg->mrq, MSG_ACK,
+ 				       msg->tx.data, msg->tx.size);
+ 	if (err < 0) {
+@@ -366,8 +407,17 @@ int tegra_bpmp_transfer(struct tegra_bpmp *bpmp,
+ 	if (!tegra_bpmp_message_valid(msg))
+ 		return -EINVAL;
+
++	if (tegra_bpmp_transfer_redirect)
++		return _tegra_bpmp_transfer(bpmp, msg);
++
+ 	channel = tegra_bpmp_write_threaded(bpmp, msg->mrq, msg->tx.data,
+ 					    msg->tx.size);
++	
++	if (tegra_bpmp_outloud){
++	    printk("tegra_bpmp_transfer tx: %x tx.size= %ld \n", msg->mrq, msg->tx.size);
++	    print_hex_dump(KERN_INFO, "tegra_bpmp_transfer tx:" ,DUMP_PREFIX_NONE, 16, 1, msg->tx.data, msg->tx.size, false);
++	}
++
+ 	if (IS_ERR(channel))
+ 		return PTR_ERR(channel);
+
+@@ -381,8 +431,15 @@ int tegra_bpmp_transfer(struct tegra_bpmp *bpmp,
+ 	if (err == 0)
+ 		return -ETIMEDOUT;
+
+-	return tegra_bpmp_channel_read(channel, msg->rx.data, msg->rx.size,
++	err = tegra_bpmp_channel_read(channel, msg->rx.data, msg->rx.size,
+ 				       &msg->rx.ret);
++
++	if(tegra_bpmp_outloud){
++ 	    printk("tegra_bpmp_transfer rx: err=%d\n msg->rx.ret=%d", err, msg->rx.ret);
++ 	    print_hex_dump(KERN_INFO,"tegra_bpmp_transfer rx:" ,DUMP_PREFIX_NONE, 16, 1, msg->rx.data, msg->rx.size, false);
++ 	}
++
++	return err;
+ }
+ EXPORT_SYMBOL_GPL(tegra_bpmp_transfer);
+