nixos/foundationdb: convert manual chapter to MD

3 files changed, 667 insertions, 376 deletions

diff --git a/nixos/modules/services/databases/foundationdb.md b/nixos/modules/services/databases/foundationdb.md
new file mode 100644
index 0000000000000..f852c6888d841
--- /dev/null
+++ b/nixos/modules/services/databases/foundationdb.md
@@ -0,0 +1,309 @@
+# FoundationDB {#module-services-foundationdb}
+
+*Source:* {file}`modules/services/databases/foundationdb.nix`
+
+*Upstream documentation:* <https://apple.github.io/foundationdb/>
+
+*Maintainer:* Austin Seipp
+
+*Available version(s):* 5.1.x, 5.2.x, 6.0.x
+
+FoundationDB (or "FDB") is an open source, distributed, transactional
+key-value store.
+
+## Configuring and basic setup {#module-services-foundationdb-configuring}
+
+To enable FoundationDB, add the following to your
+{file}`configuration.nix`:
+```
+services.foundationdb.enable = true;
+services.foundationdb.package = pkgs.foundationdb52; # FoundationDB 5.2.x
+```
+
+The {option}`services.foundationdb.package` option is required, and
+must always be specified. Due to the fact FoundationDB network protocols and
+on-disk storage formats may change between (major) versions, and upgrades
+must be explicitly handled by the user, you must always manually specify
+this yourself so that the NixOS module will use the proper version. Note
+that minor, bugfix releases are always compatible.
+
+After running {command}`nixos-rebuild`, you can verify whether
+FoundationDB is running by executing {command}`fdbcli` (which is
+added to {option}`environment.systemPackages`):
+```ShellSession
+$ sudo -u foundationdb fdbcli
+Using cluster file `/etc/foundationdb/fdb.cluster'.
+
+The database is available.
+
+Welcome to the fdbcli. For help, type `help'.
+fdb> status
+
+Using cluster file `/etc/foundationdb/fdb.cluster'.
+
+Configuration:
+  Redundancy mode        - single
+  Storage engine         - memory
+  Coordinators           - 1
+
+Cluster:
+  FoundationDB processes - 1
+  Machines               - 1
+  Memory availability    - 5.4 GB per process on machine with least available
+  Fault Tolerance        - 0 machines
+  Server time            - 04/20/18 15:21:14
+
+...
+
+fdb>
+```
+
+You can also write programs using the available client libraries. For
+example, the following Python program can be run in order to grab the
+cluster status, as a quick example. (This example uses
+{command}`nix-shell` shebang support to automatically supply the
+necessary Python modules).
+```ShellSession
+a@link> cat fdb-status.py
+#! /usr/bin/env nix-shell
+#! nix-shell -i python -p python pythonPackages.foundationdb52
+
+import fdb
+import json
+
+def main():
+    fdb.api_version(520)
+    db = fdb.open()
+
+    @fdb.transactional
+    def get_status(tr):
+        return str(tr['\xff\xff/status/json'])
+
+    obj = json.loads(get_status(db))
+    print('FoundationDB available: %s' % obj['client']['database_status']['available'])
+
+if __name__ == "__main__":
+    main()
+a@link> chmod +x fdb-status.py
+a@link> ./fdb-status.py
+FoundationDB available: True
+a@link>
+```
+
+FoundationDB is run under the {command}`foundationdb` user and group
+by default, but this may be changed in the NixOS configuration. The systemd
+unit {command}`foundationdb.service` controls the
+{command}`fdbmonitor` process.
+
+By default, the NixOS module for FoundationDB creates a single SSD-storage
+based database for development and basic usage. This storage engine is
+designed for SSDs and will perform poorly on HDDs; however it can handle far
+more data than the alternative "memory" engine and is a better default
+choice for most deployments. (Note that you can change the storage backend
+on-the-fly for a given FoundationDB cluster using
+{command}`fdbcli`.)
+
+Furthermore, only 1 server process and 1 backup agent are started in the
+default configuration. See below for more on scaling to increase this.
+
+FoundationDB stores all data for all server processes under
+{file}`/var/lib/foundationdb`. You can override this using
+{option}`services.foundationdb.dataDir`, e.g.
+```
+services.foundationdb.dataDir = "/data/fdb";
+```
+
+Similarly, logs are stored under {file}`/var/log/foundationdb`
+by default, and there is a corresponding
+{option}`services.foundationdb.logDir` as well.
+
+## Scaling processes and backup agents {#module-services-foundationdb-scaling}
+
+Scaling the number of server processes is quite easy; simply specify
+{option}`services.foundationdb.serverProcesses` to be the number of
+FoundationDB worker processes that should be started on the machine.
+
+FoundationDB worker processes typically require 4GB of RAM per-process at
+minimum for good performance, so this option is set to 1 by default since
+the maximum amount of RAM is unknown. You're advised to abide by this
+restriction, so pick a number of processes so that each has 4GB or more.
+
+A similar option exists in order to scale backup agent processes,
+{option}`services.foundationdb.backupProcesses`. Backup agents are
+not as performance/RAM sensitive, so feel free to experiment with the number
+of available backup processes.
+
+## Clustering {#module-services-foundationdb-clustering}
+
+FoundationDB on NixOS works similarly to other Linux systems, so this
+section will be brief. Please refer to the full FoundationDB documentation
+for more on clustering.
+
+FoundationDB organizes clusters using a set of
+*coordinators*, which are just specially-designated
+worker processes. By default, every installation of FoundationDB on NixOS
+will start as its own individual cluster, with a single coordinator: the
+first worker process on {command}`localhost`.
+
+Coordinators are specified globally using the
+{command}`/etc/foundationdb/fdb.cluster` file, which all servers and
+client applications will use to find and join coordinators. Note that this
+file *can not* be managed by NixOS so easily:
+FoundationDB is designed so that it will rewrite the file at runtime for all
+clients and nodes when cluster coordinators change, with clients
+transparently handling this without intervention. It is fundamentally a
+mutable file, and you should not try to manage it in any way in NixOS.
+
+When dealing with a cluster, there are two main things you want to do:
+
+  - Add a node to the cluster for storage/compute.
+  - Promote an ordinary worker to a coordinator.
+
+A node must already be a member of the cluster in order to properly be
+promoted to a coordinator, so you must always add it first if you wish to
+promote it.
+
+To add a machine to a FoundationDB cluster:
+
+  - Choose one of the servers to start as the initial coordinator.
+  - Copy the {command}`/etc/foundationdb/fdb.cluster` file from this
+    server to all the other servers. Restart FoundationDB on all of these
+    other servers, so they join the cluster.
+  - All of these servers are now connected and working together in the
+    cluster, under the chosen coordinator.
+
+At this point, you can add as many nodes as you want by just repeating the
+above steps. By default there will still be a single coordinator: you can
+use {command}`fdbcli` to change this and add new coordinators.
+
+As a convenience, FoundationDB can automatically assign coordinators based
+on the redundancy mode you wish to achieve for the cluster. Once all the
+nodes have been joined, simply set the replication policy, and then issue
+the {command}`coordinators auto` command
+
+For example, assuming we have 3 nodes available, we can enable double
+redundancy mode, then auto-select coordinators. For double redundancy, 3
+coordinators is ideal: therefore FoundationDB will make
+*every* node a coordinator automatically:
+
+```ShellSession
+fdbcli> configure double ssd
+fdbcli> coordinators auto
+```
+
+This will transparently update all the servers within seconds, and
+appropriately rewrite the {command}`fdb.cluster` file, as well as
+informing all client processes to do the same.
+
+## Client connectivity {#module-services-foundationdb-connectivity}
+
+By default, all clients must use the current {command}`fdb.cluster`
+file to access a given FoundationDB cluster. This file is located by default
+in {command}`/etc/foundationdb/fdb.cluster` on all machines with the
+FoundationDB service enabled, so you may copy the active one from your
+cluster to a new node in order to connect, if it is not part of the cluster.
+
+## Client authorization and TLS {#module-services-foundationdb-authorization}
+
+By default, any user who can connect to a FoundationDB process with the
+correct cluster configuration can access anything. FoundationDB uses a
+pluggable design to transport security, and out of the box it supports a
+LibreSSL-based plugin for TLS support. This plugin not only does in-flight
+encryption, but also performs client authorization based on the given
+endpoint's certificate chain. For example, a FoundationDB server may be
+configured to only accept client connections over TLS, where the client TLS
+certificate is from organization *Acme Co* in the
+*Research and Development* unit.
+
+Configuring TLS with FoundationDB is done using the
+{option}`services.foundationdb.tls` options in order to control the
+peer verification string, as well as the certificate and its private key.
+
+Note that the certificate and its private key must be accessible to the
+FoundationDB user account that the server runs under. These files are also
+NOT managed by NixOS, as putting them into the store may reveal private
+information.
+
+After you have a key and certificate file in place, it is not enough to
+simply set the NixOS module options -- you must also configure the
+{command}`fdb.cluster` file to specify that a given set of
+coordinators use TLS. This is as simple as adding the suffix
+{command}`:tls` to your cluster coordinator configuration, after the
+port number. For example, assuming you have a coordinator on localhost with
+the default configuration, simply specifying:
+
+```
+XXXXXX:XXXXXX@127.0.0.1:4500:tls
+```
+
+will configure all clients and server processes to use TLS from now on.
+
+## Backups and Disaster Recovery {#module-services-foundationdb-disaster-recovery}
+
+The usual rules for doing FoundationDB backups apply on NixOS as written in
+the FoundationDB manual. However, one important difference is the security
+profile for NixOS: by default, the {command}`foundationdb` systemd
+unit uses *Linux namespaces* to restrict write access to
+the system, except for the log directory, data directory, and the
+{command}`/etc/foundationdb/` directory. This is enforced by default
+and cannot be disabled.
+
+However, a side effect of this is that the {command}`fdbbackup`
+command doesn't work properly for local filesystem backups: FoundationDB
+uses a server process alongside the database processes to perform backups
+and copy the backups to the filesystem. As a result, this process is put
+under the restricted namespaces above: the backup process can only write to
+a limited number of paths.
+
+In order to allow flexible backup locations on local disks, the FoundationDB
+NixOS module supports a
+{option}`services.foundationdb.extraReadWritePaths` option. This
+option takes a list of paths, and adds them to the systemd unit, allowing
+the processes inside the service to write (and read) the specified
+directories.
+
+For example, to create backups in {command}`/opt/fdb-backups`, first
+set up the paths in the module options:
+
+```
+services.foundationdb.extraReadWritePaths = [ "/opt/fdb-backups" ];
+```
+
+Restart the FoundationDB service, and it will now be able to write to this
+directory (even if it does not yet exist.) Note: this path
+*must* exist before restarting the unit. Otherwise,
+systemd will not include it in the private FoundationDB namespace (and it
+will not add it dynamically at runtime).
+
+You can now perform a backup:
+
+```ShellSession
+$ sudo -u foundationdb fdbbackup start  -t default -d file:///opt/fdb-backups
+$ sudo -u foundationdb fdbbackup status -t default
+```
+
+## Known limitations {#module-services-foundationdb-limitations}
+
+The FoundationDB setup for NixOS should currently be considered beta.
+FoundationDB is not new software, but the NixOS compilation and integration
+has only undergone fairly basic testing of all the available functionality.
+
+  - There is no way to specify individual parameters for individual
+    {command}`fdbserver` processes. Currently, all server processes
+    inherit all the global {command}`fdbmonitor` settings.
+  - Ruby bindings are not currently installed.
+  - Go bindings are not currently installed.
+
+## Options {#module-services-foundationdb-options}
+
+NixOS's FoundationDB module allows you to configure all of the most relevant
+configuration options for {command}`fdbmonitor`, matching it quite
+closely. A complete list of options for the FoundationDB module may be found
+[here](#opt-services.foundationdb.enable). You should
+also read the FoundationDB documentation as well.
+
+## Full documentation {#module-services-foundationdb-full-docs}
+
+FoundationDB is a complex piece of software, and requires careful
+administration to properly use. Full documentation for administration can be
+found here: <https://apple.github.io/foundationdb/>.
diff --git a/nixos/modules/services/databases/foundationdb.nix b/nixos/modules/services/databases/foundationdb.nix
index 16d539b661eb4..fdfe5a28f31af 100644
--- a/nixos/modules/services/databases/foundationdb.nix
+++ b/nixos/modules/services/databases/foundationdb.nix
@@ -424,6 +424,8 @@ in
     };
   };
 
+  # Don't edit the docbook xml directly, edit the md and generate it:
+  # `pandoc foundationdb.md -t docbook --top-level-division=chapter --extract-media=media -f markdown-smart --lua-filter ../../../../doc/build-aux/pandoc-filters/myst-reader/roles.lua --lua-filter ../../../../doc/build-aux/pandoc-filters/docbook-writer/rst-roles.lua > foundationdb.xml`
   meta.doc         = ./foundationdb.xml;
   meta.maintainers = with lib.maintainers; [ thoughtpolice ];
 }
diff --git a/nixos/modules/services/databases/foundationdb.xml b/nixos/modules/services/databases/foundationdb.xml
index b0b1ebeab45f6..ae7a6dae955ed 100644
--- a/nixos/modules/services/databases/foundationdb.xml
+++ b/nixos/modules/services/databases/foundationdb.xml
@@ -1,60 +1,56 @@
-<chapter xmlns="http://docbook.org/ns/docbook"
-         xmlns:xlink="http://www.w3.org/1999/xlink"
-         xmlns:xi="http://www.w3.org/2001/XInclude"
-         version="5.0"
-         xml:id="module-services-foundationdb">
- <title>FoundationDB</title>
- <para>
-  <emphasis>Source:</emphasis>
-  <filename>modules/services/databases/foundationdb.nix</filename>
- </para>
- <para>
-  <emphasis>Upstream documentation:</emphasis>
-  <link xlink:href="https://apple.github.io/foundationdb/"/>
- </para>
- <para>
-  <emphasis>Maintainer:</emphasis> Austin Seipp
- </para>
- <para>
-  <emphasis>Available version(s):</emphasis> 5.1.x, 5.2.x, 6.0.x
- </para>
- <para>
-  FoundationDB (or "FDB") is an open source, distributed, transactional
-  key-value store.
- </para>
- <section xml:id="module-services-foundationdb-configuring">
-  <title>Configuring and basic setup</title>
-
+<chapter xmlns="http://docbook.org/ns/docbook" xmlns:xlink="http://www.w3.org/1999/xlink" xml:id="module-services-foundationdb">
+  <title>FoundationDB</title>
   <para>
-   To enable FoundationDB, add the following to your
-   <filename>configuration.nix</filename>:
-<programlisting>
-services.foundationdb.enable = true;
-services.foundationdb.package = pkgs.foundationdb52; # FoundationDB 5.2.x
-</programlisting>
+    <emphasis>Source:</emphasis>
+    <filename>modules/services/databases/foundationdb.nix</filename>
   </para>
-
   <para>
-   The <option>services.foundationdb.package</option> option is required, and
-   must always be specified. Due to the fact FoundationDB network protocols and
-   on-disk storage formats may change between (major) versions, and upgrades
-   must be explicitly handled by the user, you must always manually specify
-   this yourself so that the NixOS module will use the proper version. Note
-   that minor, bugfix releases are always compatible.
+    <emphasis>Upstream documentation:</emphasis>
+    <link xlink:href="https://apple.github.io/foundationdb/" role="uri">https://apple.github.io/foundationdb/</link>
   </para>
-
   <para>
-   After running <command>nixos-rebuild</command>, you can verify whether
-   FoundationDB is running by executing <command>fdbcli</command> (which is
-   added to <option>environment.systemPackages</option>):
-<screen>
-<prompt>$ </prompt>sudo -u foundationdb fdbcli
+    <emphasis>Maintainer:</emphasis> Austin Seipp
+  </para>
+  <para>
+    <emphasis>Available version(s):</emphasis> 5.1.x, 5.2.x, 6.0.x
+  </para>
+  <para>
+    FoundationDB (or &quot;FDB&quot;) is an open source, distributed,
+    transactional key-value store.
+  </para>
+  <section xml:id="module-services-foundationdb-configuring">
+    <title>Configuring and basic setup</title>
+    <para>
+      To enable FoundationDB, add the following to your
+      <filename>configuration.nix</filename>:
+    </para>
+    <programlisting>
+services.foundationdb.enable = true;
+services.foundationdb.package = pkgs.foundationdb52; # FoundationDB 5.2.x
+</programlisting>
+    <para>
+      The <option>services.foundationdb.package</option> option is
+      required, and must always be specified. Due to the fact
+      FoundationDB network protocols and on-disk storage formats may
+      change between (major) versions, and upgrades must be explicitly
+      handled by the user, you must always manually specify this
+      yourself so that the NixOS module will use the proper version.
+      Note that minor, bugfix releases are always compatible.
+    </para>
+    <para>
+      After running <command>nixos-rebuild</command>, you can verify
+      whether FoundationDB is running by executing
+      <command>fdbcli</command> (which is added to
+      <option>environment.systemPackages</option>):
+    </para>
+    <programlisting>
+$ sudo -u foundationdb fdbcli
 Using cluster file `/etc/foundationdb/fdb.cluster'.
 
 The database is available.
 
 Welcome to the fdbcli. For help, type `help'.
-<prompt>fdb> </prompt>status
+fdb&gt; status
 
 Using cluster file `/etc/foundationdb/fdb.cluster'.
 
@@ -72,18 +68,17 @@ Cluster:
 
 ...
 
-<prompt>fdb></prompt>
-</screen>
-  </para>
-
-  <para>
-   You can also write programs using the available client libraries. For
-   example, the following Python program can be run in order to grab the
-   cluster status, as a quick example. (This example uses
-   <command>nix-shell</command> shebang support to automatically supply the
-   necessary Python modules).
-<screen>
-<prompt>a@link> </prompt>cat fdb-status.py
+fdb&gt;
+</programlisting>
+    <para>
+      You can also write programs using the available client libraries.
+      For example, the following Python program can be run in order to
+      grab the cluster status, as a quick example. (This example uses
+      <command>nix-shell</command> shebang support to automatically
+      supply the necessary Python modules).
+    </para>
+    <programlisting>
+a@link&gt; cat fdb-status.py
 #! /usr/bin/env nix-shell
 #! nix-shell -i python -p python pythonPackages.foundationdb52
 
@@ -101,343 +96,328 @@ def main():
     obj = json.loads(get_status(db))
     print('FoundationDB available: %s' % obj['client']['database_status']['available'])
 
-if __name__ == "__main__":
+if __name__ == &quot;__main__&quot;:
     main()
-<prompt>a@link> </prompt>chmod +x fdb-status.py
-<prompt>a@link> </prompt>./fdb-status.py
+a@link&gt; chmod +x fdb-status.py
+a@link&gt; ./fdb-status.py
 FoundationDB available: True
-<prompt>a@link></prompt>
-</screen>
-  </para>
-
-  <para>
-   FoundationDB is run under the <command>foundationdb</command> user and group
-   by default, but this may be changed in the NixOS configuration. The systemd
-   unit <command>foundationdb.service</command> controls the
-   <command>fdbmonitor</command> process.
-  </para>
-
-  <para>
-   By default, the NixOS module for FoundationDB creates a single SSD-storage
-   based database for development and basic usage. This storage engine is
-   designed for SSDs and will perform poorly on HDDs; however it can handle far
-   more data than the alternative "memory" engine and is a better default
-   choice for most deployments. (Note that you can change the storage backend
-   on-the-fly for a given FoundationDB cluster using
-   <command>fdbcli</command>.)
-  </para>
-
-  <para>
-   Furthermore, only 1 server process and 1 backup agent are started in the
-   default configuration. See below for more on scaling to increase this.
-  </para>
-
-  <para>
-   FoundationDB stores all data for all server processes under
-   <filename>/var/lib/foundationdb</filename>. You can override this using
-   <option>services.foundationdb.dataDir</option>, e.g.
-<programlisting>
-services.foundationdb.dataDir = "/data/fdb";
+a@link&gt;
 </programlisting>
-  </para>
-
-  <para>
-   Similarly, logs are stored under <filename>/var/log/foundationdb</filename>
-   by default, and there is a corresponding
-   <option>services.foundationdb.logDir</option> as well.
-  </para>
- </section>
- <section xml:id="module-services-foundationdb-scaling">
-  <title>Scaling processes and backup agents</title>
-
-  <para>
-   Scaling the number of server processes is quite easy; simply specify
-   <option>services.foundationdb.serverProcesses</option> to be the number of
-   FoundationDB worker processes that should be started on the machine.
-  </para>
-
-  <para>
-   FoundationDB worker processes typically require 4GB of RAM per-process at
-   minimum for good performance, so this option is set to 1 by default since
-   the maximum amount of RAM is unknown. You're advised to abide by this
-   restriction, so pick a number of processes so that each has 4GB or more.
-  </para>
-
-  <para>
-   A similar option exists in order to scale backup agent processes,
-   <option>services.foundationdb.backupProcesses</option>. Backup agents are
-   not as performance/RAM sensitive, so feel free to experiment with the number
-   of available backup processes.
-  </para>
- </section>
- <section xml:id="module-services-foundationdb-clustering">
-  <title>Clustering</title>
-
-  <para>
-   FoundationDB on NixOS works similarly to other Linux systems, so this
-   section will be brief. Please refer to the full FoundationDB documentation
-   for more on clustering.
-  </para>
-
-  <para>
-   FoundationDB organizes clusters using a set of
-   <emphasis>coordinators</emphasis>, which are just specially-designated
-   worker processes. By default, every installation of FoundationDB on NixOS
-   will start as its own individual cluster, with a single coordinator: the
-   first worker process on <command>localhost</command>.
-  </para>
-
-  <para>
-   Coordinators are specified globally using the
-   <command>/etc/foundationdb/fdb.cluster</command> file, which all servers and
-   client applications will use to find and join coordinators. Note that this
-   file <emphasis>can not</emphasis> be managed by NixOS so easily:
-   FoundationDB is designed so that it will rewrite the file at runtime for all
-   clients and nodes when cluster coordinators change, with clients
-   transparently handling this without intervention. It is fundamentally a
-   mutable file, and you should not try to manage it in any way in NixOS.
-  </para>
-
-  <para>
-   When dealing with a cluster, there are two main things you want to do:
-  </para>
-
-  <itemizedlist>
-   <listitem>
     <para>
-     Add a node to the cluster for storage/compute.
+      FoundationDB is run under the <command>foundationdb</command> user
+      and group by default, but this may be changed in the NixOS
+      configuration. The systemd unit
+      <command>foundationdb.service</command> controls the
+      <command>fdbmonitor</command> process.
     </para>
-   </listitem>
-   <listitem>
     <para>
-     Promote an ordinary worker to a coordinator.
+      By default, the NixOS module for FoundationDB creates a single
+      SSD-storage based database for development and basic usage. This
+      storage engine is designed for SSDs and will perform poorly on
+      HDDs; however it can handle far more data than the alternative
+      &quot;memory&quot; engine and is a better default choice for most
+      deployments. (Note that you can change the storage backend
+      on-the-fly for a given FoundationDB cluster using
+      <command>fdbcli</command>.)
     </para>
-   </listitem>
-  </itemizedlist>
-
-  <para>
-   A node must already be a member of the cluster in order to properly be
-   promoted to a coordinator, so you must always add it first if you wish to
-   promote it.
-  </para>
-
-  <para>
-   To add a machine to a FoundationDB cluster:
-  </para>
-
-  <itemizedlist>
-   <listitem>
     <para>
-     Choose one of the servers to start as the initial coordinator.
+      Furthermore, only 1 server process and 1 backup agent are started
+      in the default configuration. See below for more on scaling to
+      increase this.
     </para>
-   </listitem>
-   <listitem>
     <para>
-     Copy the <command>/etc/foundationdb/fdb.cluster</command> file from this
-     server to all the other servers. Restart FoundationDB on all of these
-     other servers, so they join the cluster.
+      FoundationDB stores all data for all server processes under
+      <filename>/var/lib/foundationdb</filename>. You can override this
+      using <option>services.foundationdb.dataDir</option>, e.g.
     </para>
-   </listitem>
-   <listitem>
+    <programlisting>
+services.foundationdb.dataDir = &quot;/data/fdb&quot;;
+</programlisting>
     <para>
-     All of these servers are now connected and working together in the
-     cluster, under the chosen coordinator.
+      Similarly, logs are stored under
+      <filename>/var/log/foundationdb</filename> by default, and there
+      is a corresponding <option>services.foundationdb.logDir</option>
+      as well.
     </para>
-   </listitem>
-  </itemizedlist>
-
-  <para>
-   At this point, you can add as many nodes as you want by just repeating the
-   above steps. By default there will still be a single coordinator: you can
-   use <command>fdbcli</command> to change this and add new coordinators.
-  </para>
-
-  <para>
-   As a convenience, FoundationDB can automatically assign coordinators based
-   on the redundancy mode you wish to achieve for the cluster. Once all the
-   nodes have been joined, simply set the replication policy, and then issue
-   the <command>coordinators auto</command> command
-  </para>
-
-  <para>
-   For example, assuming we have 3 nodes available, we can enable double
-   redundancy mode, then auto-select coordinators. For double redundancy, 3
-   coordinators is ideal: therefore FoundationDB will make
-   <emphasis>every</emphasis> node a coordinator automatically:
-  </para>
-
-<screen>
-<prompt>fdbcli> </prompt>configure double ssd
-<prompt>fdbcli> </prompt>coordinators auto
-</screen>
-
-  <para>
-   This will transparently update all the servers within seconds, and
-   appropriately rewrite the <command>fdb.cluster</command> file, as well as
-   informing all client processes to do the same.
-  </para>
- </section>
- <section xml:id="module-services-foundationdb-connectivity">
-  <title>Client connectivity</title>
-
-  <para>
-   By default, all clients must use the current <command>fdb.cluster</command>
-   file to access a given FoundationDB cluster. This file is located by default
-   in <command>/etc/foundationdb/fdb.cluster</command> on all machines with the
-   FoundationDB service enabled, so you may copy the active one from your
-   cluster to a new node in order to connect, if it is not part of the cluster.
-  </para>
- </section>
- <section xml:id="module-services-foundationdb-authorization">
-  <title>Client authorization and TLS</title>
-
-  <para>
-   By default, any user who can connect to a FoundationDB process with the
-   correct cluster configuration can access anything. FoundationDB uses a
-   pluggable design to transport security, and out of the box it supports a
-   LibreSSL-based plugin for TLS support. This plugin not only does in-flight
-   encryption, but also performs client authorization based on the given
-   endpoint's certificate chain. For example, a FoundationDB server may be
-   configured to only accept client connections over TLS, where the client TLS
-   certificate is from organization <emphasis>Acme Co</emphasis> in the
-   <emphasis>Research and Development</emphasis> unit.
-  </para>
-
-  <para>
-   Configuring TLS with FoundationDB is done using the
-   <option>services.foundationdb.tls</option> options in order to control the
-   peer verification string, as well as the certificate and its private key.
-  </para>
-
-  <para>
-   Note that the certificate and its private key must be accessible to the
-   FoundationDB user account that the server runs under. These files are also
-   NOT managed by NixOS, as putting them into the store may reveal private
-   information.
-  </para>
-
-  <para>
-   After you have a key and certificate file in place, it is not enough to
-   simply set the NixOS module options -- you must also configure the
-   <command>fdb.cluster</command> file to specify that a given set of
-   coordinators use TLS. This is as simple as adding the suffix
-   <command>:tls</command> to your cluster coordinator configuration, after the
-   port number. For example, assuming you have a coordinator on localhost with
-   the default configuration, simply specifying:
-  </para>
-
-<programlisting>
+  </section>
+  <section xml:id="module-services-foundationdb-scaling">
+    <title>Scaling processes and backup agents</title>
+    <para>
+      Scaling the number of server processes is quite easy; simply
+      specify <option>services.foundationdb.serverProcesses</option> to
+      be the number of FoundationDB worker processes that should be
+      started on the machine.
+    </para>
+    <para>
+      FoundationDB worker processes typically require 4GB of RAM
+      per-process at minimum for good performance, so this option is set
+      to 1 by default since the maximum amount of RAM is unknown. You're
+      advised to abide by this restriction, so pick a number of
+      processes so that each has 4GB or more.
+    </para>
+    <para>
+      A similar option exists in order to scale backup agent processes,
+      <option>services.foundationdb.backupProcesses</option>. Backup
+      agents are not as performance/RAM sensitive, so feel free to
+      experiment with the number of available backup processes.
+    </para>
+  </section>
+  <section xml:id="module-services-foundationdb-clustering">
+    <title>Clustering</title>
+    <para>
+      FoundationDB on NixOS works similarly to other Linux systems, so
+      this section will be brief. Please refer to the full FoundationDB
+      documentation for more on clustering.
+    </para>
+    <para>
+      FoundationDB organizes clusters using a set of
+      <emphasis>coordinators</emphasis>, which are just
+      specially-designated worker processes. By default, every
+      installation of FoundationDB on NixOS will start as its own
+      individual cluster, with a single coordinator: the first worker
+      process on <command>localhost</command>.
+    </para>
+    <para>
+      Coordinators are specified globally using the
+      <command>/etc/foundationdb/fdb.cluster</command> file, which all
+      servers and client applications will use to find and join
+      coordinators. Note that this file <emphasis>can not</emphasis> be
+      managed by NixOS so easily: FoundationDB is designed so that it
+      will rewrite the file at runtime for all clients and nodes when
+      cluster coordinators change, with clients transparently handling
+      this without intervention. It is fundamentally a mutable file, and
+      you should not try to manage it in any way in NixOS.
+    </para>
+    <para>
+      When dealing with a cluster, there are two main things you want to
+      do:
+    </para>
+    <itemizedlist spacing="compact">
+      <listitem>
+        <para>
+          Add a node to the cluster for storage/compute.
+        </para>
+      </listitem>
+      <listitem>
+        <para>
+          Promote an ordinary worker to a coordinator.
+        </para>
+      </listitem>
+    </itemizedlist>
+    <para>
+      A node must already be a member of the cluster in order to
+      properly be promoted to a coordinator, so you must always add it
+      first if you wish to promote it.
+    </para>
+    <para>
+      To add a machine to a FoundationDB cluster:
+    </para>
+    <itemizedlist spacing="compact">
+      <listitem>
+        <para>
+          Choose one of the servers to start as the initial coordinator.
+        </para>
+      </listitem>
+      <listitem>
+        <para>
+          Copy the <command>/etc/foundationdb/fdb.cluster</command> file
+          from this server to all the other servers. Restart
+          FoundationDB on all of these other servers, so they join the
+          cluster.
+        </para>
+      </listitem>
+      <listitem>
+        <para>
+          All of these servers are now connected and working together in
+          the cluster, under the chosen coordinator.
+        </para>
+      </listitem>
+    </itemizedlist>
+    <para>
+      At this point, you can add as many nodes as you want by just
+      repeating the above steps. By default there will still be a single
+      coordinator: you can use <command>fdbcli</command> to change this
+      and add new coordinators.
+    </para>
+    <para>
+      As a convenience, FoundationDB can automatically assign
+      coordinators based on the redundancy mode you wish to achieve for
+      the cluster. Once all the nodes have been joined, simply set the
+      replication policy, and then issue the
+      <command>coordinators auto</command> command
+    </para>
+    <para>
+      For example, assuming we have 3 nodes available, we can enable
+      double redundancy mode, then auto-select coordinators. For double
+      redundancy, 3 coordinators is ideal: therefore FoundationDB will
+      make <emphasis>every</emphasis> node a coordinator automatically:
+    </para>
+    <programlisting>
+fdbcli&gt; configure double ssd
+fdbcli&gt; coordinators auto
+</programlisting>
+    <para>
+      This will transparently update all the servers within seconds, and
+      appropriately rewrite the <command>fdb.cluster</command> file, as
+      well as informing all client processes to do the same.
+    </para>
+  </section>
+  <section xml:id="module-services-foundationdb-connectivity">
+    <title>Client connectivity</title>
+    <para>
+      By default, all clients must use the current
+      <command>fdb.cluster</command> file to access a given FoundationDB
+      cluster. This file is located by default in
+      <command>/etc/foundationdb/fdb.cluster</command> on all machines
+      with the FoundationDB service enabled, so you may copy the active
+      one from your cluster to a new node in order to connect, if it is
+      not part of the cluster.
+    </para>
+  </section>
+  <section xml:id="module-services-foundationdb-authorization">
+    <title>Client authorization and TLS</title>
+    <para>
+      By default, any user who can connect to a FoundationDB process
+      with the correct cluster configuration can access anything.
+      FoundationDB uses a pluggable design to transport security, and
+      out of the box it supports a LibreSSL-based plugin for TLS
+      support. This plugin not only does in-flight encryption, but also
+      performs client authorization based on the given endpoint's
+      certificate chain. For example, a FoundationDB server may be
+      configured to only accept client connections over TLS, where the
+      client TLS certificate is from organization <emphasis>Acme
+      Co</emphasis> in the <emphasis>Research and Development</emphasis>
+      unit.
+    </para>
+    <para>
+      Configuring TLS with FoundationDB is done using the
+      <option>services.foundationdb.tls</option> options in order to
+      control the peer verification string, as well as the certificate
+      and its private key.
+    </para>
+    <para>
+      Note that the certificate and its private key must be accessible
+      to the FoundationDB user account that the server runs under. These
+      files are also NOT managed by NixOS, as putting them into the
+      store may reveal private information.
+    </para>
+    <para>
+      After you have a key and certificate file in place, it is not
+      enough to simply set the NixOS module options -- you must also
+      configure the <command>fdb.cluster</command> file to specify that
+      a given set of coordinators use TLS. This is as simple as adding
+      the suffix <command>:tls</command> to your cluster coordinator
+      configuration, after the port number. For example, assuming you
+      have a coordinator on localhost with the default configuration,
+      simply specifying:
+    </para>
+    <programlisting>
 XXXXXX:XXXXXX@127.0.0.1:4500:tls
 </programlisting>
-
-  <para>
-   will configure all clients and server processes to use TLS from now on.
-  </para>
- </section>
- <section xml:id="module-services-foundationdb-disaster-recovery">
-  <title>Backups and Disaster Recovery</title>
-
-  <para>
-   The usual rules for doing FoundationDB backups apply on NixOS as written in
-   the FoundationDB manual. However, one important difference is the security
-   profile for NixOS: by default, the <command>foundationdb</command> systemd
-   unit uses <emphasis>Linux namespaces</emphasis> to restrict write access to
-   the system, except for the log directory, data directory, and the
-   <command>/etc/foundationdb/</command> directory. This is enforced by default
-   and cannot be disabled.
-  </para>
-
-  <para>
-   However, a side effect of this is that the <command>fdbbackup</command>
-   command doesn't work properly for local filesystem backups: FoundationDB
-   uses a server process alongside the database processes to perform backups
-   and copy the backups to the filesystem. As a result, this process is put
-   under the restricted namespaces above: the backup process can only write to
-   a limited number of paths.
-  </para>
-
-  <para>
-   In order to allow flexible backup locations on local disks, the FoundationDB
-   NixOS module supports a
-   <option>services.foundationdb.extraReadWritePaths</option> option. This
-   option takes a list of paths, and adds them to the systemd unit, allowing
-   the processes inside the service to write (and read) the specified
-   directories.
-  </para>
-
-  <para>
-   For example, to create backups in <command>/opt/fdb-backups</command>, first
-   set up the paths in the module options:
-  </para>
-
-<programlisting>
-services.foundationdb.extraReadWritePaths = [ "/opt/fdb-backups" ];
+    <para>
+      will configure all clients and server processes to use TLS from
+      now on.
+    </para>
+  </section>
+  <section xml:id="module-services-foundationdb-disaster-recovery">
+    <title>Backups and Disaster Recovery</title>
+    <para>
+      The usual rules for doing FoundationDB backups apply on NixOS as
+      written in the FoundationDB manual. However, one important
+      difference is the security profile for NixOS: by default, the
+      <command>foundationdb</command> systemd unit uses <emphasis>Linux
+      namespaces</emphasis> to restrict write access to the system,
+      except for the log directory, data directory, and the
+      <command>/etc/foundationdb/</command> directory. This is enforced
+      by default and cannot be disabled.
+    </para>
+    <para>
+      However, a side effect of this is that the
+      <command>fdbbackup</command> command doesn't work properly for
+      local filesystem backups: FoundationDB uses a server process
+      alongside the database processes to perform backups and copy the
+      backups to the filesystem. As a result, this process is put under
+      the restricted namespaces above: the backup process can only write
+      to a limited number of paths.
+    </para>
+    <para>
+      In order to allow flexible backup locations on local disks, the
+      FoundationDB NixOS module supports a
+      <option>services.foundationdb.extraReadWritePaths</option> option.
+      This option takes a list of paths, and adds them to the systemd
+      unit, allowing the processes inside the service to write (and
+      read) the specified directories.
+    </para>
+    <para>
+      For example, to create backups in
+      <command>/opt/fdb-backups</command>, first set up the paths in the
+      module options:
+    </para>
+    <programlisting>
+services.foundationdb.extraReadWritePaths = [ &quot;/opt/fdb-backups&quot; ];
 </programlisting>
-
-  <para>
-   Restart the FoundationDB service, and it will now be able to write to this
-   directory (even if it does not yet exist.) Note: this path
-   <emphasis>must</emphasis> exist before restarting the unit. Otherwise,
-   systemd will not include it in the private FoundationDB namespace (and it
-   will not add it dynamically at runtime).
-  </para>
-
-  <para>
-   You can now perform a backup:
-  </para>
-
-<screen>
-<prompt>$ </prompt>sudo -u foundationdb fdbbackup start  -t default -d file:///opt/fdb-backups
-<prompt>$ </prompt>sudo -u foundationdb fdbbackup status -t default
-</screen>
- </section>
- <section xml:id="module-services-foundationdb-limitations">
-  <title>Known limitations</title>
-
-  <para>
-   The FoundationDB setup for NixOS should currently be considered beta.
-   FoundationDB is not new software, but the NixOS compilation and integration
-   has only undergone fairly basic testing of all the available functionality.
-  </para>
-
-  <itemizedlist>
-   <listitem>
     <para>
-     There is no way to specify individual parameters for individual
-     <command>fdbserver</command> processes. Currently, all server processes
-     inherit all the global <command>fdbmonitor</command> settings.
+      Restart the FoundationDB service, and it will now be able to write
+      to this directory (even if it does not yet exist.) Note: this path
+      <emphasis>must</emphasis> exist before restarting the unit.
+      Otherwise, systemd will not include it in the private FoundationDB
+      namespace (and it will not add it dynamically at runtime).
     </para>
-   </listitem>
-   <listitem>
     <para>
-     Ruby bindings are not currently installed.
+      You can now perform a backup:
     </para>
-   </listitem>
-   <listitem>
+    <programlisting>
+$ sudo -u foundationdb fdbbackup start  -t default -d file:///opt/fdb-backups
+$ sudo -u foundationdb fdbbackup status -t default
+</programlisting>
+  </section>
+  <section xml:id="module-services-foundationdb-limitations">
+    <title>Known limitations</title>
     <para>
-     Go bindings are not currently installed.
+      The FoundationDB setup for NixOS should currently be considered
+      beta. FoundationDB is not new software, but the NixOS compilation
+      and integration has only undergone fairly basic testing of all the
+      available functionality.
     </para>
-   </listitem>
-  </itemizedlist>
- </section>
- <section xml:id="module-services-foundationdb-options">
-  <title>Options</title>
-
-  <para>
-   NixOS's FoundationDB module allows you to configure all of the most relevant
-   configuration options for <command>fdbmonitor</command>, matching it quite
-   closely. A complete list of options for the FoundationDB module may be found
-   <link linkend="opt-services.foundationdb.enable">here</link>. You should
-   also read the FoundationDB documentation as well.
-  </para>
- </section>
- <section xml:id="module-services-foundationdb-full-docs">
-  <title>Full documentation</title>
-
-  <para>
-   FoundationDB is a complex piece of software, and requires careful
-   administration to properly use. Full documentation for administration can be
-   found here: <link xlink:href="https://apple.github.io/foundationdb/"/>.
-  </para>
- </section>
+    <itemizedlist spacing="compact">
+      <listitem>
+        <para>
+          There is no way to specify individual parameters for
+          individual <command>fdbserver</command> processes. Currently,
+          all server processes inherit all the global
+          <command>fdbmonitor</command> settings.
+        </para>
+      </listitem>
+      <listitem>
+        <para>
+          Ruby bindings are not currently installed.
+        </para>
+      </listitem>
+      <listitem>
+        <para>
+          Go bindings are not currently installed.
+        </para>
+      </listitem>
+    </itemizedlist>
+  </section>
+  <section xml:id="module-services-foundationdb-options">
+    <title>Options</title>
+    <para>
+      NixOS's FoundationDB module allows you to configure all of the
+      most relevant configuration options for
+      <command>fdbmonitor</command>, matching it quite closely. A
+      complete list of options for the FoundationDB module may be found
+      <link linkend="opt-services.foundationdb.enable">here</link>. You
+      should also read the FoundationDB documentation as well.
+    </para>
+  </section>
+  <section xml:id="module-services-foundationdb-full-docs">
+    <title>Full documentation</title>
+    <para>
+      FoundationDB is a complex piece of software, and requires careful
+      administration to properly use. Full documentation for
+      administration can be found here:
+      <link xlink:href="https://apple.github.io/foundationdb/" role="uri">https://apple.github.io/foundationdb/</link>.
+    </para>
+  </section>
 </chapter>