Merge remote-tracking branch 'upstream/master' into doc-trivial-builders-writetext

7 files changed, 200 insertions, 51 deletions

diff --git a/doc/README.md b/doc/README.md
index 616409beaaf52..43eb39c02ab18 100644
--- a/doc/README.md
+++ b/doc/README.md
@@ -106,6 +106,19 @@ The following are supported:
 - [`note`](https://tdg.docbook.org/tdg/5.0/note.html)
 - [`tip`](https://tdg.docbook.org/tdg/5.0/tip.html)
 - [`warning`](https://tdg.docbook.org/tdg/5.0/warning.html)
+- [`example`](https://tdg.docbook.org/tdg/5.0/example.html)
+
+Example admonitions require a title to work.
+If you don't provide one, the manual won't be built.
+
+```markdown
+::: {.example #ex-showing-an-example}
+
+# Title for this example
+
+Text for the example.
+:::
+```
 
 #### [Definition lists](https://github.com/jgm/commonmark-hs/blob/master/commonmark-extensions/test/definition_lists.md)
 
@@ -139,3 +152,54 @@ watermelon
     Closes #216321.
 
 - If the commit contains more than just documentation changes, follow the commit message format relevant for the rest of the changes.
+
+## Documentation conventions
+
+In an effort to keep the Nixpkgs manual in a consistent style, please follow the conventions below, unless they prevent you from properly documenting something.
+In that case, please open an issue about the particular documentation convention and tag it with a "needs: documentation" label.
+
+- Put each sentence in its own line.
+  This makes reviewing documentation much easier, since GitHub's review system is based on lines.
+
+- Use the admonitions syntax for any callouts and examples (see [section above](#admonitions)).
+
+- If you provide an example involving Nix code, make your example into a fully-working package (something that can be passed to `pkgs.callPackage`).
+  This will help others quickly test that the example works, and will also make it easier if we start automatically testing all example code to make sure it works.
+  For example, instead of providing something like:
+
+  ```
+  pkgs.dockerTools.buildLayeredImage {
+    name = "hello";
+    contents = [ pkgs.hello ];
+  }
+  ```
+
+  Provide something like:
+
+  ```
+  { dockerTools, hello }:
+  dockerTools.buildLayeredImage {
+    name = "hello";
+    contents = [ hello ];
+  }
+  ```
+
+- Use [definition lists](#definition-lists) to document function arguments, and the attributes of such arguments. For example:
+
+  ```markdown
+  # pkgs.coolFunction
+
+  Description of what `coolFunction` does.
+  `coolFunction` expects a single argument which should be an attribute set, with the following possible attributes:
+
+  `name`
+
+  : The name of the resulting image.
+
+  `tag` _optional_
+
+  : Tag of the generated image.
+
+      _Default value:_ the output path's hash.
+
+  ```
diff --git a/doc/build-helpers/images/binarycache.section.md b/doc/build-helpers/images/binarycache.section.md
index 62e47dad7c66f..9946603c958e4 100644
--- a/doc/build-helpers/images/binarycache.section.md
+++ b/doc/build-helpers/images/binarycache.section.md
@@ -1,49 +1,58 @@
 # pkgs.mkBinaryCache {#sec-pkgs-binary-cache}
 
-`pkgs.mkBinaryCache` is a function for creating Nix flat-file binary caches. Such a cache exists as a directory on disk, and can be used as a Nix substituter by passing `--substituter file:///path/to/cache` to Nix commands.
+`pkgs.mkBinaryCache` is a function for creating Nix flat-file binary caches.
+Such a cache exists as a directory on disk, and can be used as a Nix substituter by passing `--substituter file:///path/to/cache` to Nix commands.
 
-Nix packages are most commonly shared between machines using [HTTP, SSH, or S3](https://nixos.org/manual/nix/stable/package-management/sharing-packages.html), but a flat-file binary cache can still be useful in some situations. For example, you can copy it directly to another machine, or make it available on a network file system. It can also be a convenient way to make some Nix packages available inside a container via bind-mounting.
+Nix packages are most commonly shared between machines using [HTTP, SSH, or S3](https://nixos.org/manual/nix/stable/package-management/sharing-packages.html), but a flat-file binary cache can still be useful in some situations.
+For example, you can copy it directly to another machine, or make it available on a network file system.
+It can also be a convenient way to make some Nix packages available inside a container via bind-mounting.
 
-Note that this function is meant for advanced use-cases. The more idiomatic way to work with flat-file binary caches is via the [nix-copy-closure](https://nixos.org/manual/nix/stable/command-ref/nix-copy-closure.html) command. You may also want to consider [dockerTools](#sec-pkgs-dockerTools) for your containerization needs.
+`mkBinaryCache` expects an argument with the `rootPaths` attribute.
+`rootPaths` must be a list of derivations.
+The transitive closure of these derivations' outputs will be copied into the cache.
 
-## Example {#sec-pkgs-binary-cache-example}
+::: {.note}
+This function is meant for advanced use cases.
+The more idiomatic way to work with flat-file binary caches is via the [nix-copy-closure](https://nixos.org/manual/nix/stable/command-ref/nix-copy-closure.html) command.
+You may also want to consider [dockerTools](#sec-pkgs-dockerTools) for your containerization needs.
+:::
+
+[]{#sec-pkgs-binary-cache-example}
+:::{.example #ex-mkbinarycache-copying-package-closure}
+
+# Copying a package and its closure to another machine with `mkBinaryCache`
 
 The following derivation will construct a flat-file binary cache containing the closure of `hello`.
 
 ```nix
+{ mkBinaryCache, hello }:
 mkBinaryCache {
   rootPaths = [hello];
 }
 ```
 
-- `rootPaths` specifies a list of root derivations. The transitive closure of these derivations' outputs will be copied into the cache.
-
-Here's an example of building and using the cache.
-
-Build the cache on one machine, `host1`:
-
-```shellSession
-nix-build -E 'with import <nixpkgs> {}; mkBinaryCache { rootPaths = [hello]; }'
-```
+Build the cache on a machine.
+Note that the command still builds the exact nix package above, but adds some boilerplate to build it directly from an expression.
 
 ```shellSession
-/nix/store/cc0562q828rnjqjyfj23d5q162gb424g-binary-cache
+$ nix-build -E 'let pkgs = import <nixpkgs> {}; in pkgs.callPackage ({ mkBinaryCache, hello }: mkBinaryCache { rootPaths = [hello]; }) {}'
+/nix/store/azf7xay5xxdnia4h9fyjiv59wsjdxl0g-binary-cache
 ```
 
-Copy the resulting directory to the other machine, `host2`:
+Copy the resulting directory to another machine, which we'll call `host2`:
 
 ```shellSession
-scp result host2:/tmp/hello-cache
+$ scp result host2:/tmp/hello-cache
 ```
 
-Substitute the derivation using the flat-file binary cache on the other machine, `host2`:
+At this point, the cache can be used as a substituter when building derivations on `host2`:
+
 ```shellSession
-nix-build -A hello '<nixpkgs>' \
+$ nix-build -A hello '<nixpkgs>' \
   --option require-sigs false \
   --option trusted-substituters file:///tmp/hello-cache \
   --option substituters file:///tmp/hello-cache
+/nix/store/zhl06z4lrfrkw5rp0hnjjfrgsclzvxpm-hello-2.12.1
 ```
 
-```shellSession
-/nix/store/gl5a41azbpsadfkfmbilh9yk40dh5dl0-hello-2.12.1
-```
+:::
diff --git a/doc/build-helpers/special/checkpoint-build.section.md b/doc/build-helpers/special/checkpoint-build.section.md
index 5f01e699b9477..f60afe801ed4c 100644
--- a/doc/build-helpers/special/checkpoint-build.section.md
+++ b/doc/build-helpers/special/checkpoint-build.section.md
@@ -2,35 +2,38 @@
 
 `pkgs.checkpointBuildTools` provides a way to build derivations incrementally. It consists of two functions to make checkpoint builds using Nix possible.
 
-For hermeticity, Nix derivations do not allow any state to carry over between builds, making a transparent incremental build within a derivation impossible.
+For hermeticity, Nix derivations do not allow any state to be carried over between builds, making a transparent incremental build within a derivation impossible.
 
 However, we can tell Nix explicitly what the previous build state was, by representing that previous state as a derivation output. This allows the passed build state to be used for an incremental build.
 
 To change a normal derivation to a checkpoint based build, these steps must be taken:
-  - apply `prepareCheckpointBuild` on the desired derivation
-    e.g.:
+  - apply `prepareCheckpointBuild` on the desired derivation, e.g.
 ```nix
 checkpointArtifacts = (pkgs.checkpointBuildTools.prepareCheckpointBuild pkgs.virtualbox);
 ```
-  - change something you want in the sources of the package. (e.g. using a source override)
+  - change something you want in the sources of the package, e.g. use a source override:
 ```nix
 changedVBox = pkgs.virtualbox.overrideAttrs (old: {
   src = path/to/vbox/sources;
-}
+});
 ```
-  - use `mkCheckpointedBuild changedVBox buildOutput`
+  - use `mkCheckpointBuild changedVBox checkpointArtifacts`
   - enjoy shorter build times
 
 ## Example {#sec-checkpoint-build-example}
 ```nix
-{ pkgs ? import <nixpkgs> {} }: with (pkgs) checkpointBuildTools;
+{ pkgs ? import <nixpkgs> {} }:
 let
-  helloCheckpoint = checkpointBuildTools.prepareCheckpointBuild pkgs.hello;
+  inherit (pkgs.checkpointBuildTools)
+    prepareCheckpointBuild
+    mkCheckpointBuild
+    ;
+  helloCheckpoint = prepareCheckpointBuild pkgs.hello;
   changedHello = pkgs.hello.overrideAttrs (_: {
     doCheck = false;
     patchPhase = ''
       sed -i 's/Hello, world!/Hello, Nix!/g' src/hello.c
     '';
   });
-in checkpointBuildTools.mkCheckpointBuild changedHello helloCheckpoint
+in mkCheckpointBuild changedHello helloCheckpoint
 ```
diff --git a/doc/languages-frameworks/dart.section.md b/doc/languages-frameworks/dart.section.md
index 9da43714a164d..fca87fa70e4ec 100644
--- a/doc/languages-frameworks/dart.section.md
+++ b/doc/languages-frameworks/dart.section.md
@@ -4,22 +4,33 @@
 
 The function `buildDartApplication` builds Dart applications managed with pub.
 
-It fetches its Dart dependencies automatically through `fetchDartDeps`, and (through a series of hooks) builds and installs the executables specified in the pubspec file. The hooks can be used in other derivations, if needed. The phases can also be overridden to do something different from installing binaries.
+It fetches its Dart dependencies automatically through `pub2nix`, and (through a series of hooks) builds and installs the executables specified in the pubspec file. The hooks can be used in other derivations, if needed. The phases can also be overridden to do something different from installing binaries.
 
 If you are packaging a Flutter desktop application, use [`buildFlutterApplication`](#ssec-dart-flutter) instead.
 
-`vendorHash`: is the hash of the output of the dependency fetcher derivation. To obtain it, set it to `lib.fakeHash` (or omit it) and run the build ([more details here](#sec-source-hashes)).
+`pubspecLock` is the parsed pubspec.lock file. pub2nix uses this to download required packages.
+This can be converted to JSON from YAML with something like `yq . pubspec.lock`, and then read by Nix.
 
-If the upstream source is missing a `pubspec.lock` file, you'll have to vendor one and specify it using `pubspecLockFile`. If it is needed, one will be generated for you and printed when attempting to build the derivation.
+Alternatively, `autoPubspecLock` can be used instead, and set to a path to a regular `pubspec.lock` file. This relies on import-from-derivation, and is not permitted in Nixpkgs, but can be useful at other times.
 
-The `depsListFile` must always be provided when packaging in Nixpkgs. It will be generated and printed if the derivation is attempted to be built without one. Alternatively, `autoDepsList` may be set to `true` only when outside of Nixpkgs, as it relies on import-from-derivation.
+::: {.warning}
+When using `autoPubspecLock` with a local source directory, make sure to use a
+concatenation operator (e.g. `autoPubspecLock = src + "/pubspec.lock";`), and
+not string interpolation.
+
+String interpolation will copy your entire source directory to the Nix store and
+use its store path, meaning that unrelated changes to your source tree will
+cause the generated `pubspec.lock` derivation to rebuild!
+:::
+
+If the package has Git package dependencies, the hashes must be provided in the `gitHashes` set. If a hash is missing, an error message prompting you to add it will be shown.
 
 The `dart` commands run can be overridden through `pubGetScript` and `dartCompileCommand`, you can also add flags using `dartCompileFlags` or `dartJitFlags`.
 
 Dart supports multiple [outputs types](https://dart.dev/tools/dart-compile#types-of-output), you can choose between them using `dartOutputType` (defaults to `exe`). If you want to override the binaries path or the source path they come from, you can use `dartEntryPoints`. Outputs that require a runtime will automatically be wrapped with the relevant runtime (`dartaotruntime` for `aot-snapshot`, `dart run` for `jit-snapshot` and `kernel`, `node` for `js`), this can be overridden through `dartRuntimeCommand`.
 
 ```nix
-{ buildDartApplication, fetchFromGitHub }:
+{ lib, buildDartApplication, fetchFromGitHub }:
 
 buildDartApplication rec {
   pname = "dart-sass";
@@ -32,12 +43,53 @@ buildDartApplication rec {
     hash = "sha256-U6enz8yJcc4Wf8m54eYIAnVg/jsGi247Wy8lp1r1wg4=";
   };
 
-  pubspecLockFile = ./pubspec.lock;
-  depsListFile = ./deps.json;
-  vendorHash = "sha256-Atm7zfnDambN/BmmUf4BG0yUz/y6xWzf0reDw3Ad41s=";
+  pubspecLock = lib.importJSON ./pubspec.lock.json;
 }
 ```
 
+### Patching dependencies {#ssec-dart-applications-patching-dependencies}
+
+Some Dart packages require patches or build environment changes. Package derivations can be customised with the `customSourceBuilders` argument.
+
+A collection of such customisations can be found in Nixpkgs, in the `development/compilers/dart/package-source-builders` directory.
+
+This allows fixes for packages to be shared between all applications that use them. It is strongly recommended to add to this collection instead of including fixes in your application derivation itself.
+
+### Running executables from dev_dependencies {#ssec-dart-applications-build-tools}
+
+Many Dart applications require executables from the `dev_dependencies` section in `pubspec.yaml` to be run before building them.
+
+This can be done in `preBuild`, in one of two ways:
+
+1. Packaging the tool with `buildDartApplication`, adding it to Nixpkgs, and running it like any other application
+2. Running the tool from the package cache
+
+Of these methods, the first is recommended when using a tool that does not need
+to be of a specific version.
+
+For the second method, the `packageRun` function from the `dartConfigHook` can be used.
+This is an alternative to `dart run` that does not rely on Pub.
+
+e.g., for `build_runner`:
+
+```bash
+packageRun build_runner build
+```
+
+Do _not_ use `dart run <package_name>`, as this will attempt to download dependencies with Pub.
+
+### Usage with nix-shell {#ssec-dart-applications-nix-shell}
+
+As `buildDartApplication` provides dependencies instead of `pub get`, Dart needs to be explicitly told where to find them.
+
+Run the following commands in the source directory to configure Dart appropriately.
+Do not use `pub` after doing so; it will download the dependencies itself and overwrite these changes.
+
+```bash
+cp --no-preserve=all "$pubspecLockFilePath" pubspec.lock
+mkdir -p .dart_tool && cp --no-preserve=all "$packageConfig" .dart_tool/package_config.json
+```
+
 ## Flutter applications {#ssec-dart-flutter}
 
 The function `buildFlutterApplication` builds Flutter applications.
@@ -59,8 +111,10 @@ flutter.buildFlutterApplication {
     fetchSubmodules = true;
   };
 
-  pubspecLockFile = ./pubspec.lock;
-  depsListFile = ./deps.json;
-  vendorHash = "sha256-cdMO+tr6kYiN5xKXa+uTMAcFf2C75F3wVPrn21G4QPQ=";
+  pubspecLock = lib.importJSON ./pubspec.lock.json;
 }
 ```
+
+### Usage with nix-shell {#ssec-dart-flutter-nix-shell}
+
+See the [Dart documentation](#ssec-dart-applications-nix-shell) for nix-shell instructions.
diff --git a/doc/languages-frameworks/python.section.md b/doc/languages-frameworks/python.section.md
index c921756bc5113..0849aacdf1668 100644
--- a/doc/languages-frameworks/python.section.md
+++ b/doc/languages-frameworks/python.section.md
@@ -299,14 +299,13 @@ python3Packages.buildPythonApplication rec {
     hash  = "sha256-Pe229rT0aHwA98s+nTHQMEFKZPo/yw6sot8MivFDvAw=";
   };
 
-  nativeBuildInputs = [
-    python3Packages.setuptools
-    python3Packages.wheel
+  nativeBuildInputs = with python3Packages; [
+    setuptools
   ];
 
-  propagatedBuildInputs = [
-    python3Packages.tornado
-    python3Packages.python-daemon
+  propagatedBuildInputs = with python3Packages; [
+    tornado
+    python-daemon
   ];
 
   meta = with lib; {
diff --git a/doc/languages-frameworks/ruby.section.md b/doc/languages-frameworks/ruby.section.md
index 920c84eee689c..9527395de58f3 100644
--- a/doc/languages-frameworks/ruby.section.md
+++ b/doc/languages-frameworks/ruby.section.md
@@ -2,13 +2,13 @@
 
 ## Using Ruby {#using-ruby}
 
-Several versions of Ruby interpreters are available on Nix, as well as over 250 gems and many applications written in Ruby. The attribute `ruby` refers to the default Ruby interpreter, which is currently MRI 2.6. It's also possible to refer to specific versions, e.g. `ruby_2_y`, `jruby`, or `mruby`.
+Several versions of Ruby interpreters are available on Nix, as well as over 250 gems and many applications written in Ruby. The attribute `ruby` refers to the default Ruby interpreter, which is currently MRI 3.1. It's also possible to refer to specific versions, e.g. `ruby_3_y`, `jruby`, or `mruby`.
 
 In the Nixpkgs tree, Ruby packages can be found throughout, depending on what they do, and are called from the main package set. Ruby gems, however are separate sets, and there's one default set for each interpreter (currently MRI only).
 
 There are two main approaches for using Ruby with gems. One is to use a specifically locked `Gemfile` for an application that has very strict dependencies. The other is to depend on the common gems, which we'll explain further down, and rely on them being updated regularly.
 
-The interpreters have common attributes, namely `gems`, and `withPackages`. So you can refer to `ruby.gems.nokogiri`, or `ruby_2_7.gems.nokogiri` to get the Nokogiri gem already compiled and ready to use.
+The interpreters have common attributes, namely `gems`, and `withPackages`. So you can refer to `ruby.gems.nokogiri`, or `ruby_3_2.gems.nokogiri` to get the Nokogiri gem already compiled and ready to use.
 
 Since not all gems have executables like `nokogiri`, it's usually more convenient to use the `withPackages` function like this: `ruby.withPackages (p: with p; [ nokogiri ])`. This will also make sure that the Ruby in your environment will be able to find the gem and it can be used in your Ruby code (for example via `ruby` or `irb` executables) via `require "nokogiri"` as usual.
 
@@ -33,7 +33,7 @@ Again, it's possible to launch the interpreter from the shell. The Ruby interpre
 #### Load Ruby environment from `.nix` expression {#load-ruby-environment-from-.nix-expression}
 
 As explained [in the `nix-shell` section](https://nixos.org/manual/nix/stable/command-ref/nix-shell) of the Nix manual, `nix-shell` can also load an expression from a `.nix` file.
-Say we want to have Ruby 2.6, `nokogori`, and `pry`. Consider a `shell.nix` file with:
+Say we want to have Ruby, `nokogori`, and `pry`. Consider a `shell.nix` file with:
 
 ```nix
 with import <nixpkgs> {};
@@ -114,7 +114,7 @@ With this file in your directory, you can run `nix-shell` to build and use the g
 
 The `bundlerEnv` is a wrapper over all the gems in your gemset. This means that all the `/lib` and `/bin` directories will be available, and the executables of all gems (even of indirect dependencies) will end up in your `$PATH`. The `wrappedRuby` provides you with all executables that come with Ruby itself, but wrapped so they can easily find the gems in your gemset.
 
-One common issue that you might have is that you have Ruby 2.6, but also `bundler` in your gemset. That leads to a conflict for `/bin/bundle` and `/bin/bundler`. You can resolve this by wrapping either your Ruby or your gems in a `lowPrio` call. So in order to give the `bundler` from your gemset priority, it would be used like this:
+One common issue that you might have is that you have Ruby, but also `bundler` in your gemset. That leads to a conflict for `/bin/bundle` and `/bin/bundler`. You can resolve this by wrapping either your Ruby or your gems in a `lowPrio` call. So in order to give the `bundler` from your gemset priority, it would be used like this:
 
 ```nix
 # ...
diff --git a/doc/languages-frameworks/texlive.section.md b/doc/languages-frameworks/texlive.section.md
index 8b1ed92a450c1..01b59f6f34a93 100644
--- a/doc/languages-frameworks/texlive.section.md
+++ b/doc/languages-frameworks/texlive.section.md
@@ -208,3 +208,23 @@ EOF
   cp test.pdf $out
 ''
 ```
+
+## LuaLaTeX font cache {#sec-language-texlive-lualatex-font-cache}
+
+The font cache for LuaLaTeX is written to `$HOME`.
+Therefore, it is necessary to set `$HOME` to a writable path, e.g. [before using LuaLaTeX in nix derivations](https://github.com/NixOS/nixpkgs/issues/180639):
+```nix
+runCommandNoCC "lualatex-hello-world" {
+  buildInputs = [ texliveFull ];
+} ''
+  mkdir $out
+  echo '\documentclass{article} \begin{document} Hello world \end{document}' > main.tex
+  env HOME=$(mktemp -d) lualatex  -interaction=nonstopmode -output-format=pdf -output-directory=$out ./main.tex
+''
+```
+
+Additionally, [the cache of a user can diverge from the nix store](https://github.com/NixOS/nixpkgs/issues/278718).
+To resolve font issues that might follow, the cache can be removed by the user:
+```ShellSession
+luaotfload-tool --cache=erase --flush-lookups --force
+```