nixos/garage: provide multiple versions to provide an upgrade path when using NixOS service

- Add mention to release notes 23.05
- Introduce Garage v0.8
- Protect against unexpected upgrade with stateVersion
- Test matrix over 0.7 × 0.8

2 files changed, 148 insertions, 4 deletions

diff --git a/nixos/modules/services/web-servers/garage-doc.xml b/nixos/modules/services/web-servers/garage-doc.xml
new file mode 100644
index 0000000000000..16f6fde94b5a8
--- /dev/null
+++ b/nixos/modules/services/web-servers/garage-doc.xml
@@ -0,0 +1,139 @@
+<chapter xmlns="http://docbook.org/ns/docbook"
+         xmlns:xlink="http://www.w3.org/1999/xlink"
+         xmlns:xi="http://www.w3.org/2001/XInclude"
+         version="5.0"
+         xml:id="module-services-garage">
+ <title>Garage</title>
+ <para>
+  <link xlink:href="https://garagehq.deuxfleurs.fr/">Garage</link>
+  is an open-source, self-hostable S3 store, simpler than MinIO, for geodistributed stores.
+  The server setup can be automated using
+  <link linkend="opt-services.garage.enable">services.garage</link>. A
+   client configured to your local Garage instance is available in
+   the global environment as <literal>garage-manage</literal>.
+ </para>
+ <para>
+  The current default by NixOS is <package>garage_0_8</package> which is also the latest
+  major version available.
+ </para>
+ <section xml:id="module-services-garage-upgrade-scenarios">
+  <title>General considerations on upgrades</title>
+
+  <para>
+    Garage provides a cookbook documentation on how to upgrade:
+   <link xlink:href="https://garagehq.deuxfleurs.fr/documentation/cookbook/upgrading/">https://garagehq.deuxfleurs.fr/documentation/cookbook/upgrading/</link>
+  </para>
+
+ <warning>
+   <para>Garage has two types of upgrades: patch-level upgrades and minor/major version upgrades.</para>
+
+   <para>In all cases, you should read the changelog and ideally test the upgrade on a staging cluster.</para>
+
+   <para>Checking the health of your cluster can be achieved using <literal>garage-manage repair</literal>.</para>
+  </warning>
+
+
+ <warning>
+   <para>Until 1.0 is released, patch-level upgrades are considered as minor version upgrades.
+   Minor version upgrades are considered as major version upgrades.
+    i.e. 0.6 to 0.7 is a major version upgrade.</para>
+ </warning>
+
+ <itemizedlist>
+  <listitem>
+   <formalpara>
+    <title>Straightforward upgrades (patch-level upgrades)</title>
+    <para>
+     Upgrades must be performed one by one, i.e. for each node, stop it, upgrade it : change <link linkend="opt-system.stateVersion">stateVersion</link> or <link linkend="opt-services.garage.package">services.garage.package</link>, restart it if it was not already by switching.
+    </para>
+   </formalpara>
+  </listitem>
+
+  <listitem>
+   <formalpara>
+    <title>Multiple version upgrades</title>
+    <para>
+     Garage do not provide any guarantee on moving more than one major-version forward.
+     E.g., if you're on <literal>0.7</literal>, you cannot upgrade to <literal>0.9</literal>.
+     You need to upgrade to <literal>0.8</literal> first.
+
+     As long as <link linkend="opt-system.stateVersion">stateVersion</link> is declared properly,
+     this is enforced automatically. The module will issue a warning to remind the user to upgrade to latest
+     Garage <emphasis>after</emphasis> that deploy.
+   </para>
+  </formalpara>
+ </listitem>
+</itemizedlist>
+</section>
+
+<section xml:id="module-services-garage-advanced-upgrades">
+ <title>Advanced upgrades (minor/major version upgrades)</title>
+ <para>Here are some baseline instructions to handle advanced upgrades in Garage, when in doubt, please refer to upstream instructions.</para>
+
+ <itemizedlist>
+   <listitem><para>Disable API and web access to Garage.</para></listitem>
+   <listitem><para>Perform <literal>garage-manage repair --all-nodes --yes tables</literal> and <literal>garage-manage repair --all-nodes --yes blocks</literal>.</para></listitem>
+   <listitem><para>Verify the resulting logs and check that data is synced properly between all nodes.
+    If you have time, do additional checks (<literal>scrub</literal>, <literal>block_refs</literal>, etc.).</para></listitem>
+   <listitem><para>Check if queues are empty by <literal>garage-manage stats</literal> or through monitoring tools.</para></listitem>
+   <listitem><para>Run <literal>systemctl stop garage</literal> to stop the actual Garage version.</para></listitem>
+   <listitem><para>Backup the metadata folder of ALL your nodes, e.g. for a metadata directory (the default one) in <literal>/var/lib/garage/meta</literal>,
+    you can run <literal>pushd /var/lib/garage; tar -acf meta-v0.7.tar.zst meta/; popd</literal>.</para></listitem>
+   <listitem><para>Run the offline migration: <literal>nix-shell -p garage_0_8 --run "garage offline-repair --yes"</literal>, this can take some time depending on how many objects are stored in your cluster.</para></listitem>
+   <listitem><para>Bump Garage version in your NixOS configuration, either by changing <link linkend="opt-system.stateVersion">stateVersion</link> or bumping <link linkend="opt-services.garage.package">services.garage.package</link>, this should restart Garage automatically.</para></listitem>
+   <listitem><para>Perform <literal>garage-manage repair --all-nodes --yes tables</literal> and <literal>garage-manage repair --all-nodes --yes blocks</literal>.</para></listitem>
+   <listitem><para>Wait for a full table sync to run.</para></listitem>
+ </itemizedlist>
+
+ <para>
+   Your upgraded cluster should be in a working state, re-enable API and web access.
+ </para>
+</section>
+
+<section xml:id="module-services-garage-maintainer-info">
+  <title>Maintainer information</title>
+
+  <para>
+   As stated in the previous paragraph, we must provide a clean upgrade-path for Garage
+   since it cannot move more than one major version forward on a single upgrade. This chapter
+   adds some notes how Garage updates should be rolled out in the future.
+
+   This is inspired from how Nextcloud does it.
+  </para>
+
+  <para>
+   While patch-level updates are no problem and can be done directly in the
+   package-expression (and should be backported to supported stable branches after that),
+   major-releases should be added in a new attribute (e.g. Garage <literal>v0.8.0</literal>
+   should be available in <literal>nixpkgs</literal> as <literal>pkgs.garage_0_8_0</literal>).
+   To provide simple upgrade paths it's generally useful to backport those as well to stable
+   branches. As long as the package-default isn't altered, this won't break existing setups.
+   After that, the versioning-warning in the <literal>garage</literal>-module should be
+   updated to make sure that the
+   <link linkend="opt-services.garage.package">package</link>-option selects the latest version
+   on fresh setups.
+  </para>
+
+  <para>
+   If major-releases will be abandoned by upstream, we should check first if those are needed
+   in NixOS for a safe upgrade-path before removing those. In that case we shold keep those
+   packages, but mark them as insecure in an expression like this (in
+   <literal>&lt;nixpkgs/pkgs/tools/filesystem/garage/default.nix&gt;</literal>):
+<programlisting>/* ... */
+{
+  garage_0_7_3 = generic {
+    version = "0.7.3";
+    sha256 = "0000000000000000000000000000000000000000000000000000";
+    eol = true;
+  };
+}</programlisting>
+  </para>
+
+  <para>
+   Ideally we should make sure that it's possible to jump two NixOS versions forward:
+   i.e. the warnings and the logic in the module should guard a user to upgrade from a
+   Garage on e.g. 22.11 to a Garage on 23.11.
+  </para>
+ </section>
+
+</chapter>
diff --git a/nixos/modules/services/web-servers/garage.nix b/nixos/modules/services/web-servers/garage.nix
index 76ab273483eb4..d66bcd7315082 100644
--- a/nixos/modules/services/web-servers/garage.nix
+++ b/nixos/modules/services/web-servers/garage.nix
@@ -8,7 +8,10 @@ let
   configFile = toml.generate "garage.toml" cfg.settings;
 in
 {
-  meta.maintainers = [ maintainers.raitobezarius ];
+  meta = {
+    doc = ./garage-doc.xml;
+    maintainers = with pkgs.lib.maintainers; [ raitobezarius ];
+  };
 
   options.services.garage = {
     enable = mkEnableOption (lib.mdDoc "Garage Object Storage (S3 compatible)");
@@ -56,10 +59,12 @@ in
     };
 
     package = mkOption {
-      default = pkgs.garage;
-      defaultText = literalExpression "pkgs.garage";
+      # TODO: when 23.05 is released and if Garage 0.9 is the default, put a stateVersion check.
+      default = if versionAtLeast stateVersion "23.05" then pkgs.garage_0_8_0
+                else pkgs.garage_0_7;
+      defaultText = literalExpression "pkgs.garage_0_7";
       type = types.package;
-      description = lib.mdDoc "Garage package to use.";
+      description = lib.mdDoc "Garage package to use, if you are upgrading from a major version, please read NixOS and Garage release notes for upgrade instructions.";
     };
   };