nixos/proxmox-image: Disable O_DIRECT to fix assert

Context summary:
'vma create' can't otherwise write to tmpfs such as /dev/shm.
This is important when used from non-nixos machines which may
have /build as tmpfs.

VMA is Proxmox's virtual machine image format that wraps QEMU images,
augmenting these with proxmox-specific configuration file.
proxmox-image.nix uses the VMA tool to create vma image files.
The VMA tool exists as a patchset ontop of QEMU.

VMA writes its output with open() and O_DIRECT flag.
O_DIRECT does not work on Linux tmpfs [1]. Thus:
$ vma create ~/output.vma ...  # works, assuming home isn't tmpfs.
$ vma create /dev/shm/output.vma ...  # fails since /dev/shm is tmpfs
Failure results in assert(*errp == NULL).

O_DIRECT is a cache performance hint.
But it currently blocks our usage of nixos-generate -f proxmox from
Non-NixOS hosts and Docker.

The patch here simply removes O_DIRECT:
vma-writer.c later performs memalign due to O_DIRECT, but this is
safe to do with or without O_DIRECT.
Ideally, this should be fixed in upstream Proxmox: Perhaps by falling
back to open without O_DIRECT.

Another attempt to fix this SIGABRT is [2], which writes the vma file
directory to $out/ folder -- however that may still be tmpfs mounted
which it is in our case.

[1] https://lore.kernel.org/lkml/45A29EC2.8020502@tmr.com/t/
[2] https://github.com/NixOS/nixpkgs/pull/224282

1 files changed, 16 insertions, 0 deletions

diff --git a/nixos/modules/virtualisation/proxmox-image.nix b/nixos/modules/virtualisation/proxmox-image.nix
index c66a4f178ec73..82b33a341799e 100644
--- a/nixos/modules/virtualisation/proxmox-image.nix
+++ b/nixos/modules/virtualisation/proxmox-image.nix
@@ -193,6 +193,7 @@ with lib;
             sha256 = "sha256-9rN1x5UfcoQCeYsLqrsthkeMpT1Eztvvq74cRr9G+Dk=";
           };
           patches = [
+            # Proxmox' VMA tool is published as a particular patch upon QEMU
             (pkgs.fetchpatch {
               url =
                 let
@@ -201,6 +202,21 @@ with lib;
                 in "https://git.proxmox.com/?p=pve-qemu.git;a=blob_plain;hb=${rev};f=${path}";
               hash = "sha256-2Dz+ceTwrcyYYxi76RtyY3v15/2pwGcDhFuoZWlgbjc=";
             })
+
+            # Proxmox' VMA tool uses O_DIRECT which fails on tmpfs
+            # Filed to upstream issue tracker: https://bugzilla.proxmox.com/show_bug.cgi?id=4710
+            (pkgs.writeText "inline.patch" ''
+                --- a/vma-writer.c   2023-05-01 15:11:13.361341177 +0200
+                +++ b/vma-writer.c   2023-05-01 15:10:51.785293129 +0200
+                @@ -306,7 +306,7 @@
+                             /* try to use O_NONBLOCK */
+                             fcntl(vmaw->fd, F_SETFL, fcntl(vmaw->fd, F_GETFL)|O_NONBLOCK);
+                         } else  {
+                -            oflags = O_NONBLOCK|O_DIRECT|O_WRONLY|O_EXCL;
+                +            oflags = O_NONBLOCK|O_WRONLY|O_EXCL;
+                             vmaw->fd = qemu_create(filename, oflags, 0644, errp);
+                         }
+            '')
           ];
 
           buildInputs = super.buildInputs ++ [ pkgs.libuuid ];