diff options
Diffstat (limited to 'doc')
| -rw-r--r-- | doc/README.md | 1 | ||||
| -rw-r--r-- | doc/build-helpers.md | 28 | ||||
| -rw-r--r-- | doc/build-helpers/fetchers.chapter.md (renamed from doc/builders/fetchers.chapter.md) | 46 | ||||
| -rw-r--r-- | doc/build-helpers/images.md (renamed from doc/builders/images.md) | 0 | ||||
| -rw-r--r-- | doc/build-helpers/images/appimagetools.section.md (renamed from doc/builders/images/appimagetools.section.md) | 0 | ||||
| -rw-r--r-- | doc/build-helpers/images/binarycache.section.md (renamed from doc/builders/images/binarycache.section.md) | 0 | ||||
| -rw-r--r-- | doc/build-helpers/images/dockertools.section.md (renamed from doc/builders/images/dockertools.section.md) | 2 | ||||
| -rw-r--r-- | doc/build-helpers/images/makediskimage.section.md (renamed from doc/builders/images/makediskimage.section.md) | 0 | ||||
| -rw-r--r-- | doc/build-helpers/images/ocitools.section.md (renamed from doc/builders/images/ocitools.section.md) | 0 | ||||
| -rw-r--r-- | doc/build-helpers/images/portableservice.section.md (renamed from doc/builders/images/portableservice.section.md) | 0 | ||||
| -rw-r--r-- | doc/build-helpers/images/snaptools.section.md (renamed from doc/builders/images/snaptools.section.md) | 0 | ||||
| -rw-r--r-- | doc/build-helpers/special.md (renamed from doc/builders/special.md) | 5 | ||||
| -rw-r--r-- | doc/build-helpers/special/fhs-environments.section.md (renamed from doc/builders/special/fhs-environments.section.md) | 0 | ||||
| -rw-r--r-- | doc/build-helpers/special/makesetuphook.section.md (renamed from doc/builders/special/makesetuphook.section.md) | 2 | ||||
| -rw-r--r-- | doc/build-helpers/special/mkshell.section.md (renamed from doc/builders/special/mkshell.section.md) | 0 | ||||
| -rw-r--r-- | doc/build-helpers/special/vm-tools.section.md (renamed from doc/builders/special/vm-tools.section.md) | 0 | ||||
| -rw-r--r-- | doc/build-helpers/testers.chapter.md (renamed from doc/builders/testers.chapter.md) | 0 | ||||
| -rw-r--r-- | doc/build-helpers/trivial-build-helpers.chapter.md (renamed from doc/builders/trivial-builders.chapter.md) | 23 | ||||
| -rw-r--r-- | doc/builders.md | 12 | ||||
| -rw-r--r-- | doc/default.nix | 1 | ||||
| -rw-r--r-- | doc/functions/fileset.section.md | 3 | ||||
| -rw-r--r-- | doc/hooks/autopatchelf.section.md | 2 | ||||
| -rw-r--r-- | doc/hooks/bmake.section.md | 7 | ||||
| -rw-r--r-- | doc/hooks/index.md | 1 | ||||
| -rw-r--r-- | doc/hooks/mpi-check-hook.section.md | 2 | ||||
| -rw-r--r-- | doc/languages-frameworks/agda.section.md | 4 | ||||
| -rw-r--r-- | doc/languages-frameworks/beam.section.md | 22 | ||||
| -rw-r--r-- | doc/languages-frameworks/dart.section.md | 9 | ||||
| -rw-r--r-- | doc/languages-frameworks/dhall.section.md | 11 | ||||
| -rw-r--r-- | doc/languages-frameworks/dotnet.section.md | 2 | ||||
| -rw-r--r-- | doc/languages-frameworks/emscripten.section.md | 297 | ||||
| -rw-r--r-- | doc/languages-frameworks/go.section.md | 2 | ||||
| -rw-r--r-- | doc/languages-frameworks/haskell.section.md | 10 | ||||
| -rw-r--r-- | doc/languages-frameworks/javascript.section.md | 21 | ||||
| -rw-r--r-- | doc/languages-frameworks/lisp.section.md | 2 | ||||
| -rw-r--r-- | doc/languages-frameworks/lua.section.md | 6 | ||||
| -rw-r--r-- | doc/languages-frameworks/maven.section.md | 2 | ||||
| -rw-r--r-- | doc/languages-frameworks/php.section.md | 8 | ||||
| -rw-r--r-- | doc/languages-frameworks/python.section.md | 202 | ||||
| -rw-r--r-- | doc/languages-frameworks/ruby.section.md | 4 | ||||
| -rw-r--r-- | doc/languages-frameworks/rust.section.md | 2 | ||||
| -rw-r--r-- | doc/languages-frameworks/swift.section.md | 4 | ||||
| -rw-r--r-- | doc/languages-frameworks/texlive.section.md | 58 | ||||
| -rw-r--r-- | doc/languages-frameworks/vim.section.md | 13 | ||||
| -rw-r--r-- | doc/manual.md.in | 2 | ||||
| -rw-r--r-- | doc/packages/cataclysm-dda.section.md (renamed from doc/builders/packages/cataclysm-dda.section.md) | 0 | ||||
| -rw-r--r-- | doc/packages/citrix.section.md (renamed from doc/builders/packages/citrix.section.md) | 0 | ||||
| -rw-r--r-- | doc/packages/darwin-builder.section.md (renamed from doc/builders/special/darwin-builder.section.md) | 30 | ||||
| -rw-r--r-- | doc/packages/dlib.section.md (renamed from doc/builders/packages/dlib.section.md) | 0 | ||||
| -rw-r--r-- | doc/packages/eclipse.section.md (renamed from doc/builders/packages/eclipse.section.md) | 0 | ||||
| -rw-r--r-- | doc/packages/elm.section.md (renamed from doc/builders/packages/elm.section.md) | 0 | ||||
| -rw-r--r-- | doc/packages/emacs.section.md (renamed from doc/builders/packages/emacs.section.md) | 0 | ||||
| -rw-r--r-- | doc/packages/etc-files.section.md (renamed from doc/builders/packages/etc-files.section.md) | 0 | ||||
| -rw-r--r-- | doc/packages/firefox.section.md (renamed from doc/builders/packages/firefox.section.md) | 0 | ||||
| -rw-r--r-- | doc/packages/fish.section.md (renamed from doc/builders/packages/fish.section.md) | 0 | ||||
| -rw-r--r-- | doc/packages/fuse.section.md (renamed from doc/builders/packages/fuse.section.md) | 0 | ||||
| -rw-r--r-- | doc/packages/ibus.section.md (renamed from doc/builders/packages/ibus.section.md) | 0 | ||||
| -rw-r--r-- | doc/packages/index.md (renamed from doc/builders/packages/index.md) | 1 | ||||
| -rw-r--r-- | doc/packages/kakoune.section.md (renamed from doc/builders/packages/kakoune.section.md) | 0 | ||||
| -rw-r--r-- | doc/packages/linux.section.md (renamed from doc/builders/packages/linux.section.md) | 0 | ||||
| -rw-r--r-- | doc/packages/locales.section.md (renamed from doc/builders/packages/locales.section.md) | 0 | ||||
| -rw-r--r-- | doc/packages/nginx.section.md (renamed from doc/builders/packages/nginx.section.md) | 0 | ||||
| -rw-r--r-- | doc/packages/opengl.section.md (renamed from doc/builders/packages/opengl.section.md) | 0 | ||||
| -rw-r--r-- | doc/packages/shell-helpers.section.md (renamed from doc/builders/packages/shell-helpers.section.md) | 0 | ||||
| -rw-r--r-- | doc/packages/steam.section.md (renamed from doc/builders/packages/steam.section.md) | 0 | ||||
| -rw-r--r-- | doc/packages/urxvt.section.md (renamed from doc/builders/packages/urxvt.section.md) | 2 | ||||
| -rw-r--r-- | doc/packages/weechat.section.md (renamed from doc/builders/packages/weechat.section.md) | 0 | ||||
| -rw-r--r-- | doc/packages/xorg.section.md (renamed from doc/builders/packages/xorg.section.md) | 0 | ||||
| -rw-r--r-- | doc/stdenv/multiple-output.chapter.md | 44 | ||||
| -rw-r--r-- | doc/stdenv/stdenv.chapter.md | 112 | ||||
| -rw-r--r-- | doc/using-nixpkgs.md | 1 | ||||
| -rw-r--r-- | doc/using/overlays.chapter.md | 2 | ||||
| -rw-r--r-- | doc/using/platform-support.chapter.md | 18 |
73 files changed, 635 insertions, 391 deletions
diff --git a/doc/README.md b/doc/README.md index 03df6ad611388..3f9aff1a38a6d 100644 --- a/doc/README.md +++ b/doc/README.md @@ -3,6 +3,7 @@ This directory houses the sources files for the Nixpkgs manual. You can find the [rendered documentation for Nixpkgs `unstable` on nixos.org](https://nixos.org/manual/nixpkgs/unstable/). +The rendering tool is [nixos-render-docs](../pkgs/tools/nix/nixos-render-docs/src/nixos_render_docs), sometimes abbreviated `nrd`. [Docs for Nixpkgs stable](https://nixos.org/manual/nixpkgs/stable/) are also available. diff --git a/doc/build-helpers.md b/doc/build-helpers.md new file mode 100644 index 0000000000000..06737e1667602 --- /dev/null +++ b/doc/build-helpers.md @@ -0,0 +1,28 @@ +# Build helpers {#part-builders} + +A build helper is a function that produces derivations. + +:::{.warning} +This is not to be confused with the [`builder` argument of the Nix `derivation` primitive](https://nixos.org/manual/nix/unstable/language/derivations.html), which refers to the executable that produces the build result, or [remote builder](https://nixos.org/manual/nix/stable/advanced-topics/distributed-builds.html), which refers to a remote machine that could run such an executable. +::: + +Such a function is usually designed to abstract over a typical workflow for a given programming language or framework. +This allows declaring a build recipe by setting a limited number of options relevant to the particular use case instead of using the `derivation` function directly. + +[`stdenv.mkDerivation`](#part-stdenv) is the most widely used build helper, and serves as a basis for many others. +In addition, it offers various options to customize parts of the builds. + +There is no uniform interface for build helpers. +[Trivial build helpers](#chap-trivial-builders) and [fetchers](#chap-pkgs-fetchers) have various input types for convenience. +[Language- or framework-specific build helpers](#chap-language-support) usually follow the style of `stdenv.mkDerivation`, which accepts an attribute set or a fixed-point function taking an attribute set. + +```{=include=} chapters +build-helpers/fetchers.chapter.md +build-helpers/trivial-build-helpers.chapter.md +build-helpers/testers.chapter.md +build-helpers/special.md +build-helpers/images.md +hooks/index.md +languages-frameworks/index.md +packages/index.md +``` diff --git a/doc/builders/fetchers.chapter.md b/doc/build-helpers/fetchers.chapter.md index 75a261db8dcd6..7bd1bbd6de029 100644 --- a/doc/builders/fetchers.chapter.md +++ b/doc/build-helpers/fetchers.chapter.md @@ -1,12 +1,27 @@ # Fetchers {#chap-pkgs-fetchers} Building software with Nix often requires downloading source code and other files from the internet. -`nixpkgs` provides *fetchers* for different protocols and services. Fetchers are functions that simplify downloading files. +To this end, Nixpkgs provides *fetchers*: functions to obtain remote sources via various protocols and services. -## Caveats {#chap-pkgs-fetchers-caveats} +Nixpkgs fetchers differ from built-in fetchers such as [`builtins.fetchTarball`](https://nixos.org/manual/nix/stable/language/builtins.html#builtins-fetchTarball): +- A built-in fetcher will download and cache files at evaluation time and produce a [store path](https://nixos.org/manual/nix/stable/glossary#gloss-store-path). + A Nixpkgs fetcher will create a ([fixed-output](https://nixos.org/manual/nix/stable/glossary#gloss-fixed-output-derivation)) [derivation](https://nixos.org/manual/nix/stable/language/derivations), and files are downloaded at build time. +- Built-in fetchers will invalidate their cache after [`tarball-ttl`](https://nixos.org/manual/nix/stable/command-ref/conf-file#conf-tarball-ttl) expires, and will require network activity to check if the cache entry is up to date. + Nixpkgs fetchers only re-download if the specified hash changes or the store object is not otherwise available. +- Built-in fetchers do not use [substituters](https://nixos.org/manual/nix/stable/command-ref/conf-file#conf-substituters). + Derivations produced by Nixpkgs fetchers will use any configured binary cache transparently. + +This significantly reduces the time needed to evaluate the entirety of Nixpkgs, and allows [Hydra](https://nixos.org/hydra) to retain and re-distribute sources used by Nixpkgs in the [public binary cache](https://cache.nixos.org). +For these reasons, built-in fetchers are not allowed in Nixpkgs source code. + +The following table shows an overview of the differences: + +| Fetchers | Download | Output | Cache | Re-download when | +|-|-|-|-|-| +| `builtins.fetch*` | evaluation time | store path | `/nix/store`, `~/.cache/nix` | `tarball-ttl` expires, cache miss in `~/.cache/nix`, output store object not in local store | +| `pkgs.fetch*` | build time | derivation | `/nix/store`, substituters | output store object not available | -Fetchers create [fixed output derivations](https://nixos.org/manual/nix/stable/#fixed-output-drvs) from downloaded files. -Nix can reuse the downloaded files via the hash of the resulting derivation. +## Caveats {#chap-pkgs-fetchers-caveats} The fact that the hash belongs to the Nix derivation output and not the file itself can lead to confusion. For example, consider the following fetcher: @@ -243,3 +258,26 @@ or *** ``` +## `fetchtorrent` {#fetchtorrent} + +`fetchtorrent` expects two arguments. `url` which can either be a Magnet URI (Magnet Link) such as `magnet:?xt=urn:btih:dd8255ecdc7ca55fb0bbf81323d87062db1f6d1c` or an HTTP URL pointing to a `.torrent` file. It can also take a `config` argument which will craft a `settings.json` configuration file and give it to `transmission`, the underlying program that is performing the fetch. The available config options for `transmission` can be found [here](https://github.com/transmission/transmission/blob/main/docs/Editing-Configuration-Files.md#options) + +``` +{ fetchtorrent }: + +fetchtorrent { + config = { peer-limit-global = 100; }; + url = "magnet:?xt=urn:btih:dd8255ecdc7ca55fb0bbf81323d87062db1f6d1c"; + sha256 = ""; +} +``` + +### Parameters {#fetchtorrent-parameters} + +- `url`: Magnet URI (Magnet Link) such as `magnet:?xt=urn:btih:dd8255ecdc7ca55fb0bbf81323d87062db1f6d1c` or an HTTP URL pointing to a `.torrent` file. + +- `backend`: Which bittorrent program to use. Default: `"transmission"`. Valid values are `"rqbit"` or `"transmission"`. These are the two most suitable torrent clients for fetching in a fixed-output derivation at the time of writing, as they can be easily exited after usage. `rqbit` is written in Rust and has a smaller closure size than `transmission`, and the performance and peer discovery properties differs between these clients, requiring experimentation to decide upon which is the best. + +- `config`: When using `transmission` as the `backend`, a json configuration can + be supplied to transmission. Refer to the [upstream documentation](https://github.com/transmission/transmission/blob/main/docs/Editing-Configuration-Files.md) for information on how to configure. + diff --git a/doc/builders/images.md b/doc/build-helpers/images.md index 5596784bfa487..5596784bfa487 100644 --- a/doc/builders/images.md +++ b/doc/build-helpers/images.md diff --git a/doc/builders/images/appimagetools.section.md b/doc/build-helpers/images/appimagetools.section.md index 0c72315a26e88..0c72315a26e88 100644 --- a/doc/builders/images/appimagetools.section.md +++ b/doc/build-helpers/images/appimagetools.section.md diff --git a/doc/builders/images/binarycache.section.md b/doc/build-helpers/images/binarycache.section.md index 62e47dad7c66f..62e47dad7c66f 100644 --- a/doc/builders/images/binarycache.section.md +++ b/doc/build-helpers/images/binarycache.section.md diff --git a/doc/builders/images/dockertools.section.md b/doc/build-helpers/images/dockertools.section.md index 3ac4f224b5def..42d6e297f529c 100644 --- a/doc/builders/images/dockertools.section.md +++ b/doc/build-helpers/images/dockertools.section.md @@ -275,7 +275,7 @@ pullImage { `nix-prefetch-docker` command can be used to get required image parameters: ```ShellSession -$ nix run nixpkgs.nix-prefetch-docker -c nix-prefetch-docker --image-name mysql --image-tag 5 +$ nix run nixpkgs#nix-prefetch-docker -- --image-name mysql --image-tag 5 ``` Since a given `imageName` may transparently refer to a manifest list of images which support multiple architectures and/or operating systems, you can supply the `--os` and `--arch` arguments to specify exactly which image you want. By default it will match the OS and architecture of the host the command is run on. diff --git a/doc/builders/images/makediskimage.section.md b/doc/build-helpers/images/makediskimage.section.md index e50479c4e83e4..e50479c4e83e4 100644 --- a/doc/builders/images/makediskimage.section.md +++ b/doc/build-helpers/images/makediskimage.section.md diff --git a/doc/builders/images/ocitools.section.md b/doc/build-helpers/images/ocitools.section.md index c35f65bce0073..c35f65bce0073 100644 --- a/doc/builders/images/ocitools.section.md +++ b/doc/build-helpers/images/ocitools.section.md diff --git a/doc/builders/images/portableservice.section.md b/doc/build-helpers/images/portableservice.section.md index 5400928b158f8..5400928b158f8 100644 --- a/doc/builders/images/portableservice.section.md +++ b/doc/build-helpers/images/portableservice.section.md diff --git a/doc/builders/images/snaptools.section.md b/doc/build-helpers/images/snaptools.section.md index 259fa1b061808..259fa1b061808 100644 --- a/doc/builders/images/snaptools.section.md +++ b/doc/build-helpers/images/snaptools.section.md diff --git a/doc/builders/special.md b/doc/build-helpers/special.md index 6d07fa87f3f3e..f88648207fdcd 100644 --- a/doc/builders/special.md +++ b/doc/build-helpers/special.md @@ -1,11 +1,10 @@ -# Special builders {#chap-special} +# Special build helpers {#chap-special} -This chapter describes several special builders. +This chapter describes several special build helpers. ```{=include=} sections special/fhs-environments.section.md special/makesetuphook.section.md special/mkshell.section.md -special/darwin-builder.section.md special/vm-tools.section.md ``` diff --git a/doc/builders/special/fhs-environments.section.md b/doc/build-helpers/special/fhs-environments.section.md index 8145fbd730f7e..8145fbd730f7e 100644 --- a/doc/builders/special/fhs-environments.section.md +++ b/doc/build-helpers/special/fhs-environments.section.md diff --git a/doc/builders/special/makesetuphook.section.md b/doc/build-helpers/special/makesetuphook.section.md index eb042412137b8..e83164b7eb701 100644 --- a/doc/builders/special/makesetuphook.section.md +++ b/doc/build-helpers/special/makesetuphook.section.md @@ -1,6 +1,6 @@ # pkgs.makeSetupHook {#sec-pkgs.makeSetupHook} -`pkgs.makeSetupHook` is a builder that produces hooks that go in to `nativeBuildInputs` +`pkgs.makeSetupHook` is a build helper that produces hooks that go in to `nativeBuildInputs` ## Usage {#sec-pkgs.makeSetupHook-usage} diff --git a/doc/builders/special/mkshell.section.md b/doc/build-helpers/special/mkshell.section.md index 96d43535955fa..96d43535955fa 100644 --- a/doc/builders/special/mkshell.section.md +++ b/doc/build-helpers/special/mkshell.section.md diff --git a/doc/builders/special/vm-tools.section.md b/doc/build-helpers/special/vm-tools.section.md index 8feab04902d8f..8feab04902d8f 100644 --- a/doc/builders/special/vm-tools.section.md +++ b/doc/build-helpers/special/vm-tools.section.md diff --git a/doc/builders/testers.chapter.md b/doc/build-helpers/testers.chapter.md index b2a581c3dd8d9..b2a581c3dd8d9 100644 --- a/doc/builders/testers.chapter.md +++ b/doc/build-helpers/testers.chapter.md diff --git a/doc/builders/trivial-builders.chapter.md b/doc/build-helpers/trivial-build-helpers.chapter.md index c05511785bf55..a0cda86a66071 100644 --- a/doc/builders/trivial-builders.chapter.md +++ b/doc/build-helpers/trivial-build-helpers.chapter.md @@ -1,12 +1,28 @@ -# Trivial builders {#chap-trivial-builders} +# Trivial build helpers {#chap-trivial-builders} Nixpkgs provides a couple of functions that help with building derivations. The most important one, `stdenv.mkDerivation`, has already been documented above. The following functions wrap `stdenv.mkDerivation`, making it easier to use in certain cases. ## `runCommand` {#trivial-builder-runCommand} -This takes three arguments, `name`, `env`, and `buildCommand`. `name` is just the name that Nix will append to the store path in the same way that `stdenv.mkDerivation` uses its `name` attribute. `env` is an attribute set specifying environment variables that will be set for this derivation. These attributes are then passed to the wrapped `stdenv.mkDerivation`. `buildCommand` specifies the commands that will be run to create this derivation. Note that you will need to create `$out` for Nix to register the command as successful. +`runCommand :: String -> AttrSet -> String -> Derivation` -An example of using `runCommand` is provided below. +`runCommand name drvAttrs buildCommand` returns a derivation that is built by running the specified shell commands. + +`name :: String` +: The name that Nix will append to the store path in the same way that `stdenv.mkDerivation` uses its `name` attribute. + +`drvAttr :: AttrSet` +: Attributes to pass to the underlying call to [`stdenv.mkDerivation`](#chap-stdenv). + +`buildCommand :: String` +: Shell commands to run in the derivation builder. + + ::: {.note} + You have to create a file or directory `$out` for Nix to be able to run the builder successfully. + ::: + +::: {.example #ex-runcommand-simple} +# Invocation of `runCommand` ```nix (import <nixpkgs> {}).runCommand "my-example" {} '' @@ -28,6 +44,7 @@ An example of using `runCommand` is provided below. date '' ``` +::: ## `runCommandCC` {#trivial-builder-runCommandCC} diff --git a/doc/builders.md b/doc/builders.md deleted file mode 100644 index 2e959422405b2..0000000000000 --- a/doc/builders.md +++ /dev/null @@ -1,12 +0,0 @@ -# Builders {#part-builders} - -```{=include=} chapters -builders/fetchers.chapter.md -builders/trivial-builders.chapter.md -builders/testers.chapter.md -builders/special.md -builders/images.md -hooks/index.md -languages-frameworks/index.md -builders/packages/index.md -``` diff --git a/doc/default.nix b/doc/default.nix index 18e12c1a8aca1..61bbd2ba8dc82 100644 --- a/doc/default.nix +++ b/doc/default.nix @@ -23,6 +23,7 @@ let { name = "sources"; description = "source filtering functions"; } { name = "cli"; description = "command-line serialization functions"; } { name = "gvariant"; description = "GVariant formatted string serialization functions"; } + { name = "customisation"; description = "Functions to customise (derivation-related) functions, derivatons, or attribute sets"; } ]; }; diff --git a/doc/functions/fileset.section.md b/doc/functions/fileset.section.md index b24ebe26cc3b7..c42337feaba4f 100644 --- a/doc/functions/fileset.section.md +++ b/doc/functions/fileset.section.md @@ -6,11 +6,8 @@ The [`lib.fileset`](#sec-functions-library-fileset) library allows you to work w A file set is a mathematical set of local files that can be added to the Nix store for use in Nix derivations. File sets are easy and safe to use, providing obvious and composable semantics with good error messages to prevent mistakes. -These sections apply to the entire library. See the [function reference](#sec-functions-library-fileset) for function-specific documentation. -The file set library is currently very limited but is being expanded to include more functions over time. - ## Implicit coercion from paths to file sets {#sec-fileset-path-coercion} All functions accepting file sets as arguments can also accept [paths](https://nixos.org/manual/nix/stable/language/values.html#type-path) as arguments. diff --git a/doc/hooks/autopatchelf.section.md b/doc/hooks/autopatchelf.section.md index 008a90d46140c..995204b902199 100644 --- a/doc/hooks/autopatchelf.section.md +++ b/doc/hooks/autopatchelf.section.md @@ -6,6 +6,6 @@ You can also specify a `runtimeDependencies` variable which lists dependencies t In certain situations you may want to run the main command (`autoPatchelf`) of the setup hook on a file or a set of directories instead of unconditionally patching all outputs. This can be done by setting the `dontAutoPatchelf` environment variable to a non-empty value. -By default `autoPatchelf` will fail as soon as any ELF file requires a dependency which cannot be resolved via the given build inputs. In some situations you might prefer to just leave missing dependencies unpatched and continue to patch the rest. This can be achieved by setting the `autoPatchelfIgnoreMissingDeps` environment variable to a non-empty value. `autoPatchelfIgnoreMissingDeps` can be set to a list like `autoPatchelfIgnoreMissingDeps = [ "libcuda.so.1" "libcudart.so.1" ];` or to simply `[ "*" ]` to ignore all missing dependencies. +By default `autoPatchelf` will fail as soon as any ELF file requires a dependency which cannot be resolved via the given build inputs. In some situations you might prefer to just leave missing dependencies unpatched and continue to patch the rest. This can be achieved by setting the `autoPatchelfIgnoreMissingDeps` environment variable to a non-empty value. `autoPatchelfIgnoreMissingDeps` can be set to a list like `autoPatchelfIgnoreMissingDeps = [ "libcuda.so.1" "libcudart.so.1" ];` or to `[ "*" ]` to ignore all missing dependencies. The `autoPatchelf` command also recognizes a `--no-recurse` command line flag, which prevents it from recursing into subdirectories. diff --git a/doc/hooks/bmake.section.md b/doc/hooks/bmake.section.md new file mode 100644 index 0000000000000..6b40ac13e8b95 --- /dev/null +++ b/doc/hooks/bmake.section.md @@ -0,0 +1,7 @@ +# bmake {#bmake-hook} + +[bmake](https://www.crufty.net/help/sjg/bmake.html) is the portable variant of +NetBSD make utility. + +In Nixpkgs, `bmake` comes with a hook that overrides the default build, check, +install and dist phases. diff --git a/doc/hooks/index.md b/doc/hooks/index.md index 8100e91c8b481..363d627e52582 100644 --- a/doc/hooks/index.md +++ b/doc/hooks/index.md @@ -8,6 +8,7 @@ The stdenv built-in hooks are documented in [](#ssec-setup-hooks). autoconf.section.md automake.section.md autopatchelf.section.md +bmake.section.md breakpoint.section.md cmake.section.md gdk-pixbuf.section.md diff --git a/doc/hooks/mpi-check-hook.section.md b/doc/hooks/mpi-check-hook.section.md index e3fb5c40dadaa..586ee2cc7c2dc 100644 --- a/doc/hooks/mpi-check-hook.section.md +++ b/doc/hooks/mpi-check-hook.section.md @@ -3,7 +3,7 @@ This hook can be used to setup a check phase that requires running a MPI application. It detects the -used present MPI implementaion type and exports +used present MPI implementation type and exports the neceesary environment variables to use `mpirun` and `mpiexec` in a Nix sandbox. diff --git a/doc/languages-frameworks/agda.section.md b/doc/languages-frameworks/agda.section.md index ff3d70ef0c626..cb1f12eec234a 100644 --- a/doc/languages-frameworks/agda.section.md +++ b/doc/languages-frameworks/agda.section.md @@ -146,7 +146,7 @@ agdaPackages.mkDerivation { ### Building Agda packages {#building-agda-packages} -The default build phase for `agdaPackages.mkDerivation` simply runs `agda` on the `Everything.agda` file. +The default build phase for `agdaPackages.mkDerivation` runs `agda` on the `Everything.agda` file. If something else is needed to build the package (e.g. `make`) then the `buildPhase` should be overridden. Additionally, a `preBuild` or `configurePhase` can be used if there are steps that need to be done prior to checking the `Everything.agda` file. `agda` and the Agda libraries contained in `buildInputs` are made available during the build phase. @@ -250,7 +250,7 @@ Usually, the maintainers will answer within a week or two with a new release. Bumping the version of that reverse dependency should be a further commit on your PR. In the rare case that a new release is not to be expected within an acceptable time, -simply mark the broken package as broken by setting `meta.broken = true;`. +mark the broken package as broken by setting `meta.broken = true;`. This will exclude it from the build test. It can be added later when it is fixed, and does not hinder the advancement of the whole package set in the meantime. diff --git a/doc/languages-frameworks/beam.section.md b/doc/languages-frameworks/beam.section.md index 5e85723085c12..1e83d4b93c7c5 100644 --- a/doc/languages-frameworks/beam.section.md +++ b/doc/languages-frameworks/beam.section.md @@ -44,11 +44,29 @@ There is also a `buildMix` helper, whose behavior is closer to that of `buildErl ## How to Install BEAM Packages {#how-to-install-beam-packages} -BEAM builders are not registered at the top level, simply because they are not relevant to the vast majority of Nix users. To install any of those builders into your profile, refer to them by their attribute path `beamPackages.rebar3`: +BEAM builders are not registered at the top level, because they are not relevant to the vast majority of Nix users. +To use any of those builders into your environment, refer to them by their attribute path under `beamPackages`, e.g. `beamPackages.rebar3`: + +::: {.example #ex-beam-ephemeral-shell} +# Ephemeral shell ```ShellSession -$ nix-env -f "<nixpkgs>" -iA beamPackages.rebar3 +$ nix-shell -p beamPackages.rebar3 +``` +::: + +::: {.example #ex-beam-declarative-shell} +# Declarative shell + +```nix +let + pkgs = import <nixpkgs> { config = {}; overlays = []; }; +in +pkgs.mkShell { + packages = [ pkgs.beamPackages.rebar3 ]; +} ``` +::: ## Packaging BEAM Applications {#packaging-beam-applications} diff --git a/doc/languages-frameworks/dart.section.md b/doc/languages-frameworks/dart.section.md index b00327b78eb29..9da43714a164d 100644 --- a/doc/languages-frameworks/dart.section.md +++ b/doc/languages-frameworks/dart.section.md @@ -8,10 +8,12 @@ It fetches its Dart dependencies automatically through `fetchDartDeps`, and (thr If you are packaging a Flutter desktop application, use [`buildFlutterApplication`](#ssec-dart-flutter) instead. -`vendorHash`: is the hash of the output of the dependency fetcher derivation. To obtain it, simply set it to `lib.fakeHash` (or omit it) and run the build ([more details here](#sec-source-hashes)). +`vendorHash`: is the hash of the output of the dependency fetcher derivation. To obtain it, set it to `lib.fakeHash` (or omit it) and run the build ([more details here](#sec-source-hashes)). If the upstream source is missing a `pubspec.lock` file, you'll have to vendor one and specify it using `pubspecLockFile`. If it is needed, one will be generated for you and printed when attempting to build the derivation. +The `depsListFile` must always be provided when packaging in Nixpkgs. It will be generated and printed if the derivation is attempted to be built without one. Alternatively, `autoDepsList` may be set to `true` only when outside of Nixpkgs, as it relies on import-from-derivation. + The `dart` commands run can be overridden through `pubGetScript` and `dartCompileCommand`, you can also add flags using `dartCompileFlags` or `dartJitFlags`. Dart supports multiple [outputs types](https://dart.dev/tools/dart-compile#types-of-output), you can choose between them using `dartOutputType` (defaults to `exe`). If you want to override the binaries path or the source path they come from, you can use `dartEntryPoints`. Outputs that require a runtime will automatically be wrapped with the relevant runtime (`dartaotruntime` for `aot-snapshot`, `dart run` for `jit-snapshot` and `kernel`, `node` for `js`), this can be overridden through `dartRuntimeCommand`. @@ -31,6 +33,7 @@ buildDartApplication rec { }; pubspecLockFile = ./pubspec.lock; + depsListFile = ./deps.json; vendorHash = "sha256-Atm7zfnDambN/BmmUf4BG0yUz/y6xWzf0reDw3Ad41s="; } ``` @@ -39,9 +42,7 @@ buildDartApplication rec { The function `buildFlutterApplication` builds Flutter applications. -The deps.json file must always be provided when packaging in Nixpkgs. It will be generated and printed if the derivation is attempted to be built without one. Alternatively, `autoDepsList` may be set to `true` when outside of Nixpkgs, as it relies on import-from-derivation. - -A `pubspec.lock` file must be available. See the [Dart documentation](#ssec-dart-applications) for more details. +See the [Dart documentation](#ssec-dart-applications) for more details on required files and arguments. ```nix { flutter, fetchFromGitHub }: diff --git a/doc/languages-frameworks/dhall.section.md b/doc/languages-frameworks/dhall.section.md index 1a209dbc06801..7322a61687dde 100644 --- a/doc/languages-frameworks/dhall.section.md +++ b/doc/languages-frameworks/dhall.section.md @@ -303,11 +303,8 @@ You can use the `dhall-to-nixpkgs` command-line utility to automate packaging Dhall code. For example: ```ShellSession -$ nix-env --install --attr haskellPackages.dhall-nixpkgs - -$ nix-env --install --attr nix-prefetch-git # Used by dhall-to-nixpkgs - -$ dhall-to-nixpkgs github https://github.com/Gabriella439/dhall-semver.git +$ nix-shell -p haskellPackages.dhall-nixpkgs nix-prefetch-git +[nix-shell]$ dhall-to-nixpkgs github https://github.com/Gabriella439/dhall-semver.git { buildDhallGitHubPackage, Prelude }: buildDhallGitHubPackage { name = "dhall-semver"; @@ -325,6 +322,10 @@ $ dhall-to-nixpkgs github https://github.com/Gabriella439/dhall-semver.git } ``` +:::{.note} +`nix-prefetch-git` has to be in `$PATH` for `dhall-to-nixpkgs` to work. +::: + The utility takes care of automatically detecting remote imports and converting them to package dependencies. You can also use the utility on local Dhall directories, too: diff --git a/doc/languages-frameworks/dotnet.section.md b/doc/languages-frameworks/dotnet.section.md index 39e7416182691..9ba0fef2a27bb 100644 --- a/doc/languages-frameworks/dotnet.section.md +++ b/doc/languages-frameworks/dotnet.section.md @@ -161,7 +161,7 @@ in buildDotnetModule rec { They can be installed either as a global tool for the entire system, or as a local tool specific to project. The local installation is the easiest and works on NixOS in the same way as on other Linux distributions. -[See dotnet documention](https://learn.microsoft.com/en-us/dotnet/core/tools/global-tools#install-a-local-tool) to learn more. +[See dotnet documentation](https://learn.microsoft.com/en-us/dotnet/core/tools/global-tools#install-a-local-tool) to learn more. [The global installation method](https://learn.microsoft.com/en-us/dotnet/core/tools/global-tools#install-a-global-tool) should also work most of the time. You have to remember to update the `PATH` diff --git a/doc/languages-frameworks/emscripten.section.md b/doc/languages-frameworks/emscripten.section.md index 5f93dd5ff3159..20d358f2e9e3f 100644 --- a/doc/languages-frameworks/emscripten.section.md +++ b/doc/languages-frameworks/emscripten.section.md @@ -2,168 +2,159 @@ [Emscripten](https://github.com/kripken/emscripten): An LLVM-to-JavaScript Compiler -This section of the manual covers how to use `emscripten` in nixpkgs. +If you want to work with `emcc`, `emconfigure` and `emmake` as you are used to from Ubuntu and similar distributions, -Minimal requirements: - -* nix -* nixpkgs - -Modes of use of `emscripten`: - -* **Imperative usage** (on the command line): - - If you want to work with `emcc`, `emconfigure` and `emmake` as you are used to from Ubuntu and similar distributions you can use these commands: - - * `nix-env -f "<nixpkgs>" -iA emscripten` - * `nix-shell -p emscripten` - -* **Declarative usage**: - - This mode is far more power full since this makes use of `nix` for dependency management of emscripten libraries and targets by using the `mkDerivation` which is implemented by `pkgs.emscriptenStdenv` and `pkgs.buildEmscriptenPackage`. The source for the packages is in `pkgs/top-level/emscripten-packages.nix` and the abstraction behind it in `pkgs/development/em-modules/generic/default.nix`. From the root of the nixpkgs repository: - * build and install all packages: - * `nix-env -iA emscriptenPackages` - - * dev-shell for zlib implementation hacking: - * `nix-shell -A emscriptenPackages.zlib` - -## Imperative usage {#imperative-usage} +```console +nix-shell -p emscripten +``` A few things to note: * `export EMCC_DEBUG=2` is nice for debugging -* `~/.emscripten`, the build artifact cache sometimes creates issues and needs to be removed from time to time +* The build artifact cache in `~/.emscripten` sometimes creates issues and needs to be removed from time to time -## Declarative usage {#declarative-usage} +## Examples {#declarative-usage} Let's see two different examples from `pkgs/top-level/emscripten-packages.nix`: * `pkgs.zlib.override` * `pkgs.buildEmscriptenPackage` -Both are interesting concepts. - -A special requirement of the `pkgs.buildEmscriptenPackage` is the `doCheck = true` is a default meaning that each emscriptenPackage requires a `checkPhase` implemented. - -* Use `export EMCC_DEBUG=2` from within a emscriptenPackage's `phase` to get more detailed debug output what is going wrong. -* ~/.emscripten cache is requiring us to set `HOME=$TMPDIR` in individual phases. This makes compilation slower but also makes it more deterministic. - -### Usage 1: pkgs.zlib.override {#usage-1-pkgs.zlib.override} - -This example uses `zlib` from nixpkgs but instead of compiling **C** to **ELF** it compiles **C** to **JS** since we were using `pkgs.zlib.override` and changed stdenv to `pkgs.emscriptenStdenv`. A few adaptions and hacks were set in place to make it working. One advantage is that when `pkgs.zlib` is updated, it will automatically update this package as well. However, this can also be the downside... - -See the `zlib` example: - - zlib = (pkgs.zlib.override { - stdenv = pkgs.emscriptenStdenv; - }).overrideAttrs - (old: rec { - buildInputs = old.buildInputs ++ [ pkg-config ]; - # we need to reset this setting! - env = (old.env or { }) // { NIX_CFLAGS_COMPILE = ""; }; - configurePhase = '' - # FIXME: Some tests require writing at $HOME - HOME=$TMPDIR - runHook preConfigure - - #export EMCC_DEBUG=2 - emconfigure ./configure --prefix=$out --shared - - runHook postConfigure - ''; - dontStrip = true; - outputs = [ "out" ]; - buildPhase = '' - emmake make - ''; - installPhase = '' - emmake make install - ''; - checkPhase = '' - echo "================= testing zlib using node =================" - - echo "Compiling a custom test" - set -x - emcc -O2 -s EMULATE_FUNCTION_POINTER_CASTS=1 test/example.c -DZ_SOLO \ - libz.so.${old.version} -I . -o example.js - - echo "Using node to execute the test" - ${pkgs.nodejs}/bin/node ./example.js - - set +x - if [ $? -ne 0 ]; then - echo "test failed for some reason" - exit 1; - else - echo "it seems to work! very good." - fi - echo "================= /testing zlib using node =================" - ''; - - postPatch = pkgs.lib.optionalString pkgs.stdenv.isDarwin '' - substituteInPlace configure \ - --replace '/usr/bin/libtool' 'ar' \ - --replace 'AR="libtool"' 'AR="ar"' \ - --replace 'ARFLAGS="-o"' 'ARFLAGS="-r"' - ''; - }); - -### Usage 2: pkgs.buildEmscriptenPackage {#usage-2-pkgs.buildemscriptenpackage} - -This `xmlmirror` example features a emscriptenPackage which is defined completely from this context and no `pkgs.zlib.override` is used. - - xmlmirror = pkgs.buildEmscriptenPackage rec { - name = "xmlmirror"; - - buildInputs = [ pkg-config autoconf automake libtool gnumake libxml2 nodejs openjdk json_c ]; - nativeBuildInputs = [ pkg-config zlib ]; - - src = pkgs.fetchgit { - url = "https://gitlab.com/odfplugfest/xmlmirror.git"; - rev = "4fd7e86f7c9526b8f4c1733e5c8b45175860a8fd"; - hash = "sha256-i+QgY+5PYVg5pwhzcDnkfXAznBg3e8sWH2jZtixuWsk="; - }; - - configurePhase = '' - rm -f fastXmlLint.js* - # a fix for ERROR:root:For asm.js, TOTAL_MEMORY must be a multiple of 16MB, was 234217728 - # https://gitlab.com/odfplugfest/xmlmirror/issues/8 - sed -e "s/TOTAL_MEMORY=234217728/TOTAL_MEMORY=268435456/g" -i Makefile.emEnv - # https://github.com/kripken/emscripten/issues/6344 - # https://gitlab.com/odfplugfest/xmlmirror/issues/9 - sed -e "s/\$(JSONC_LDFLAGS) \$(ZLIB_LDFLAGS) \$(LIBXML20_LDFLAGS)/\$(JSONC_LDFLAGS) \$(LIBXML20_LDFLAGS) \$(ZLIB_LDFLAGS) /g" -i Makefile.emEnv - # https://gitlab.com/odfplugfest/xmlmirror/issues/11 - sed -e "s/-o fastXmlLint.js/-s EXTRA_EXPORTED_RUNTIME_METHODS='[\"ccall\", \"cwrap\"]' -o fastXmlLint.js/g" -i Makefile.emEnv - ''; - - buildPhase = '' - HOME=$TMPDIR - make -f Makefile.emEnv - ''; - - outputs = [ "out" "doc" ]; - - installPhase = '' - mkdir -p $out/share - mkdir -p $doc/share/${name} - - cp Demo* $out/share - cp -R codemirror-5.12 $out/share - cp fastXmlLint.js* $out/share - cp *.xsd $out/share - cp *.js $out/share - cp *.xhtml $out/share - cp *.html $out/share - cp *.json $out/share - cp *.rng $out/share - cp README.md $doc/share/${name} - ''; - checkPhase = '' - - ''; - }; - -### Declarative debugging {#declarative-debugging} +A special requirement of the `pkgs.buildEmscriptenPackage` is the `doCheck = true`. +This means each Emscripten package requires that a [`checkPhase`](#ssec-check-phase) is implemented. + +* Use `export EMCC_DEBUG=2` from within a phase to get more detailed debug output what is going wrong. +* The cache at `~/.emscripten` requires to set `HOME=$TMPDIR` in individual phases. + This makes compilation slower but also more deterministic. + +::: {.example #usage-1-pkgs.zlib.override} + +# Using `pkgs.zlib.override {}` + +This example uses `zlib` from Nixpkgs, but instead of compiling **C** to **ELF** it compiles **C** to **JavaScript** since we were using `pkgs.zlib.override` and changed `stdenv` to `pkgs.emscriptenStdenv`. + +A few adaptions and hacks were put in place to make it work. +One advantage is that when `pkgs.zlib` is updated, it will automatically update this package as well. + + +```nix +(pkgs.zlib.override { + stdenv = pkgs.emscriptenStdenv; +}).overrideAttrs +(old: rec { + buildInputs = old.buildInputs ++ [ pkg-config ]; + # we need to reset this setting! + env = (old.env or { }) // { NIX_CFLAGS_COMPILE = ""; }; + configurePhase = '' + # FIXME: Some tests require writing at $HOME + HOME=$TMPDIR + runHook preConfigure + + #export EMCC_DEBUG=2 + emconfigure ./configure --prefix=$out --shared + + runHook postConfigure + ''; + dontStrip = true; + outputs = [ "out" ]; + buildPhase = '' + emmake make + ''; + installPhase = '' + emmake make install + ''; + checkPhase = '' + echo "================= testing zlib using node =================" + + echo "Compiling a custom test" + set -x + emcc -O2 -s EMULATE_FUNCTION_POINTER_CASTS=1 test/example.c -DZ_SOLO \ + libz.so.${old.version} -I . -o example.js + + echo "Using node to execute the test" + ${pkgs.nodejs}/bin/node ./example.js + + set +x + if [ $? -ne 0 ]; then + echo "test failed for some reason" + exit 1; + else + echo "it seems to work! very good." + fi + echo "================= /testing zlib using node =================" + ''; + + postPatch = pkgs.lib.optionalString pkgs.stdenv.isDarwin '' + substituteInPlace configure \ + --replace '/usr/bin/libtool' 'ar' \ + --replace 'AR="libtool"' 'AR="ar"' \ + --replace 'ARFLAGS="-o"' 'ARFLAGS="-r"' + ''; +}) +``` + +:::{.example #usage-2-pkgs.buildemscriptenpackage} + +# Using `pkgs.buildEmscriptenPackage {}` + +This `xmlmirror` example features an Emscripten package that is defined completely from this context and no `pkgs.zlib.override` is used. + +```nix +pkgs.buildEmscriptenPackage rec { + name = "xmlmirror"; + + buildInputs = [ pkg-config autoconf automake libtool gnumake libxml2 nodejs openjdk json_c ]; + nativeBuildInputs = [ pkg-config zlib ]; + + src = pkgs.fetchgit { + url = "https://gitlab.com/odfplugfest/xmlmirror.git"; + rev = "4fd7e86f7c9526b8f4c1733e5c8b45175860a8fd"; + hash = "sha256-i+QgY+5PYVg5pwhzcDnkfXAznBg3e8sWH2jZtixuWsk="; + }; + + configurePhase = '' + rm -f fastXmlLint.js* + # a fix for ERROR:root:For asm.js, TOTAL_MEMORY must be a multiple of 16MB, was 234217728 + # https://gitlab.com/odfplugfest/xmlmirror/issues/8 + sed -e "s/TOTAL_MEMORY=234217728/TOTAL_MEMORY=268435456/g" -i Makefile.emEnv + # https://github.com/kripken/emscripten/issues/6344 + # https://gitlab.com/odfplugfest/xmlmirror/issues/9 + sed -e "s/\$(JSONC_LDFLAGS) \$(ZLIB_LDFLAGS) \$(LIBXML20_LDFLAGS)/\$(JSONC_LDFLAGS) \$(LIBXML20_LDFLAGS) \$(ZLIB_LDFLAGS) /g" -i Makefile.emEnv + # https://gitlab.com/odfplugfest/xmlmirror/issues/11 + sed -e "s/-o fastXmlLint.js/-s EXTRA_EXPORTED_RUNTIME_METHODS='[\"ccall\", \"cwrap\"]' -o fastXmlLint.js/g" -i Makefile.emEnv + ''; + + buildPhase = '' + HOME=$TMPDIR + make -f Makefile.emEnv + ''; + + outputs = [ "out" "doc" ]; + + installPhase = '' + mkdir -p $out/share + mkdir -p $doc/share/${name} + + cp Demo* $out/share + cp -R codemirror-5.12 $out/share + cp fastXmlLint.js* $out/share + cp *.xsd $out/share + cp *.js $out/share + cp *.xhtml $out/share + cp *.html $out/share + cp *.json $out/share + cp *.rng $out/share + cp README.md $doc/share/${name} + ''; + checkPhase = '' + + ''; +} +``` + +::: + +## Debugging {#declarative-debugging} Use `nix-shell -I nixpkgs=/some/dir/nixpkgs -A emscriptenPackages.libz` and from there you can go trough the individual steps. This makes it easy to build a good `unit test` or list the files of the project. @@ -174,9 +165,3 @@ Use `nix-shell -I nixpkgs=/some/dir/nixpkgs -A emscriptenPackages.libz` and from 5. `configurePhase` 6. `buildPhase` 7. ... happy hacking... - -## Summary {#summary} - -Using this toolchain makes it easy to leverage `nix` from NixOS, MacOSX or even Windows (WSL+ubuntu+nix). This toolchain is reproducible, behaves like the rest of the packages from nixpkgs and contains a set of well working examples to learn and adapt from. - -If in trouble, ask the maintainers. diff --git a/doc/languages-frameworks/go.section.md b/doc/languages-frameworks/go.section.md index 7fd38a7d21c54..59856b92c9edf 100644 --- a/doc/languages-frameworks/go.section.md +++ b/doc/languages-frameworks/go.section.md @@ -18,7 +18,7 @@ In the following is an example expression using `buildGoModule`, the following a To avoid updating this field when dependencies change, run `go mod vendor` in your source repo and set `vendorHash = null;` - To obtain the actual hash, set `vendorHash = lib.fakeSha256;` and run the build ([more details here](#sec-source-hashes)). + To obtain the actual hash, set `vendorHash = lib.fakeHash;` and run the build ([more details here](#sec-source-hashes)). - `proxyVendor`: Fetches (go mod download) and proxies the vendor directory. This is useful if your code depends on c code and go mod tidy does not include the needed sources to build or if any dependency has case-insensitive conflicts which will produce platform-dependent `vendorHash` checksums. - `modPostBuild`: Shell commands to run after the build of the goModules executes `go mod vendor`, and before calculating fixed output derivation's `vendorHash` (or `vendorSha256`). Note that if you change this attribute, you need to update `vendorHash` (or `vendorSha256`) attribute. diff --git a/doc/languages-frameworks/haskell.section.md b/doc/languages-frameworks/haskell.section.md index 60972331840a1..b0b5f5c3bb2f1 100644 --- a/doc/languages-frameworks/haskell.section.md +++ b/doc/languages-frameworks/haskell.section.md @@ -177,7 +177,7 @@ exactly one version. Those versions need to satisfy all the version constraints given in the `.cabal` file of your package and all its dependencies. The [Haskell builder in nixpkgs](#haskell-mkderivation) does no such thing. -It will simply take as input packages with names off the desired dependencies +It will take as input packages with names off the desired dependencies and just check whether they fulfill the version bounds and fail if they don’t (by default, see `jailbreak` to circumvent this). @@ -221,7 +221,7 @@ Sadly we currently don’t have tooling for this. For this you might be interested in the alternative [haskell.nix] framework, which, be warned, is completely incompatible with packages from `haskellPackages`. -<!-- TODO(@maralorn) Link to package set generation docs in the contributers guide below. --> +<!-- TODO(@maralorn) Link to package set generation docs in the contributors guide below. --> ## `haskellPackages.mkDerivation` {#haskell-mkderivation} @@ -780,7 +780,7 @@ there instead. The top level `pkgs.haskell-language-server` attribute is just a convenience wrapper to make it possible to install HLS for multiple GHC versions at the same time. If you know, that you only use one GHC version, e.g., in a project -specific `nix-shell` you can simply use +specific `nix-shell` you can use `pkgs.haskellPackages.haskell-language-server` or `pkgs.haskell.packages.*.haskell-language-server` from the package set you use. @@ -1029,7 +1029,7 @@ ugly, and we may want to deprecate them at some point. --> `disableCabalFlag flag drv` : Makes sure that the Cabal flag `flag` is disabled in Cabal's configure step. -`appendBuildflags list drv` +`appendBuildFlags list drv` : Adds the strings in `list` to the `buildFlags` argument for `drv`. <!-- TODO(@sternenseemann): removeConfigureFlag --> @@ -1192,7 +1192,7 @@ with GHC), it is recommended to use overlays for Nixpkgs to change them. Since the interrelated parts, i.e. the package set and GHC, are connected via the Nixpkgs fixpoint, we need to modify them both in a way that preserves their connection (or else we'd have to wire it up again manually). This is -achieved by changing GHC and the package set in seperate overlays to prevent +achieved by changing GHC and the package set in separate overlays to prevent the package set from pulling in GHC from `prev`. The result is two overlays like the ones shown below. Adjustable parts are diff --git a/doc/languages-frameworks/javascript.section.md b/doc/languages-frameworks/javascript.section.md index 0a2099b0a6b12..79cb09572503b 100644 --- a/doc/languages-frameworks/javascript.section.md +++ b/doc/languages-frameworks/javascript.section.md @@ -161,6 +161,8 @@ git config --global url."https://github.com/".insteadOf git://github.com/ `buildNpmPackage` allows you to package npm-based projects in Nixpkgs without the use of an auto-generated dependencies file (as used in [node2nix](#javascript-node2nix)). It works by utilizing npm's cache functionality -- creating a reproducible cache that contains the dependencies of a project, and pointing npm to it. +Here's an example: + ```nix { lib, buildNpmPackage, fetchFromGitHub }: @@ -191,6 +193,8 @@ buildNpmPackage rec { } ``` +In the default `installPhase` set by `buildNpmPackage`, it uses `npm pack --json --dry-run` to decide what files to install in `$out/lib/node_modules/$name/`, where `$name` is the `name` string defined in the package's `package.json`. Additionally, the `bin` and `man` keys in the source's `package.json` are used to decide what binaries and manpages are supposed to be installed. If these are not defined, `npm pack` may miss some files, and no binaries will be produced. + #### Arguments {#javascript-buildNpmPackage-arguments} * `npmDepsHash`: The output hash of the dependencies for this project. Can be calculated in advance with [`prefetch-npm-deps`](#javascript-buildNpmPackage-prefetch-npm-deps). @@ -204,10 +208,12 @@ buildNpmPackage rec { * `npmBuildFlags`: Flags to pass to `npm run ${npmBuildScript}`. * `npmPackFlags`: Flags to pass to `npm pack`. * `npmPruneFlags`: Flags to pass to `npm prune`. Defaults to the value of `npmInstallFlags`. +* `makeWrapperArgs`: Flags to pass to `makeWrapper`, added to executable calling the generated `.js` with `node` as an interpreter. These scripts are defined in `package.json`. +* `nodejs`: The `nodejs` package to build against, using the corresponding `npm` shipped with that version of `node`. Defaults to `pkgs.nodejs`. #### prefetch-npm-deps {#javascript-buildNpmPackage-prefetch-npm-deps} -`prefetch-npm-deps` can calculate the hash of the dependencies of an npm project ahead of time. +`prefetch-npm-deps` is a Nixpkgs package that calculates the hash of the dependencies of an npm project ahead of time. ```console $ ls @@ -217,6 +223,19 @@ $ prefetch-npm-deps package-lock.json sha256-AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA= ``` +#### fetchNpmDeps {#javascript-buildNpmPackage-fetchNpmDeps} + +`fetchNpmDeps` is a Nix function that requires the following mandatory arguments: + +- `src`: A directory / tarball with `package-lock.json` file +- `hash`: The output hash of the node dependencies defined in `package-lock.json`. + +It returns a derivation with all `package-lock.json` dependencies downloaded into `$out/`, usable as an npm cache. + +### corepack {#javascript-corepack} + +This package puts the corepack wrappers for pnpm and yarn in your PATH, and they will honor the `packageManager` setting in the `package.json`. + ### node2nix {#javascript-node2nix} #### Preparation {#javascript-node2nix-preparation} diff --git a/doc/languages-frameworks/lisp.section.md b/doc/languages-frameworks/lisp.section.md index 8712c34120645..fe7f2ef80a2e8 100644 --- a/doc/languages-frameworks/lisp.section.md +++ b/doc/languages-frameworks/lisp.section.md @@ -268,7 +268,7 @@ getting an environment variable for `ext:getenv`. This will load the ### Loading systems {#lisp-loading-systems} -There, you can simply use `asdf:load-system`. This works by setting the right +There, you can use `asdf:load-system`. This works by setting the right values for the `CL_SOURCE_REGISTRY`/`ASDF_OUTPUT_TRANSLATIONS` environment variables, so that systems are found in the Nix store and pre-compiled FASLs are loaded. diff --git a/doc/languages-frameworks/lua.section.md b/doc/languages-frameworks/lua.section.md index c5049326a776f..310ea88a86d55 100644 --- a/doc/languages-frameworks/lua.section.md +++ b/doc/languages-frameworks/lua.section.md @@ -134,11 +134,11 @@ The site proposes two types of packages, the `rockspec` and the `src.rock` Luarocks-based packages are generated in [pkgs/development/lua-modules/generated-packages.nix](https://github.com/NixOS/nixpkgs/tree/master/pkgs/development/lua-modules/generated-packages.nix) from the whitelist maintainers/scripts/luarocks-packages.csv and updated by running -the script -[maintainers/scripts/update-luarocks-packages](https://github.com/NixOS/nixpkgs/tree/master/maintainers/scripts/update-luarocks-packages): +the package `luarocks-packages-updater`: ```sh -./maintainers/scripts/update-luarocks-packages update + +nix-shell -p luarocks-packages-updater --run luarocks-packages-updater ``` [luarocks2nix](https://github.com/nix-community/luarocks) is a tool capable of generating nix derivations from both rockspec and src.rock (and favors the src.rock). diff --git a/doc/languages-frameworks/maven.section.md b/doc/languages-frameworks/maven.section.md index 7e287a097c7e0..b86733a758984 100644 --- a/doc/languages-frameworks/maven.section.md +++ b/doc/languages-frameworks/maven.section.md @@ -53,7 +53,7 @@ After setting `maven.buildMavenPackage`, we then do standard Java `.jar` install Maven defines default versions for its core plugins, e.g. `maven-compiler-plugin`. If your project does not override these versions, an upgrade of Maven will change the version of the used plugins, and therefore the derivation and hash. -When `maven` is upgraded, `mvnHash` for the derivation must be updated as well: otherwise, the project will simply be built on the derivation of old plugins, and fail because the requested plugins are missing. +When `maven` is upgraded, `mvnHash` for the derivation must be updated as well: otherwise, the project will be built on the derivation of old plugins, and fail because the requested plugins are missing. This clearly prevents automatic upgrades of Maven: a manual effort must be made throughout nixpkgs by any maintainer wishing to push the upgrades. diff --git a/doc/languages-frameworks/php.section.md b/doc/languages-frameworks/php.section.md index 2ca55aef1eff9..154d8174f9aaf 100644 --- a/doc/languages-frameworks/php.section.md +++ b/doc/languages-frameworks/php.section.md @@ -58,7 +58,7 @@ php.withExtensions ({ enabled, all }: ++ [ all.imagick ]) ``` -To build your list of extensions from the ground up, you can simply +To build your list of extensions from the ground up, you can ignore `enabled`: ```nix @@ -140,7 +140,7 @@ Example of building `composer` with additional extensions: ### Overriding PHP packages {#ssec-php-user-guide-overriding-packages} `php-packages.nix` form a scope, allowing us to override the packages defined -within. For example, to apply a patch to a `mysqlnd` extension, you can simply +within. For example, to apply a patch to a `mysqlnd` extension, you can pass an overlay-style function to `php`’s `packageOverrides` argument: ```nix @@ -191,7 +191,7 @@ using the `bin` attribute in `composer.json`, these binaries will be automatically linked and made accessible in the derivation. In this context, "binaries" refer to PHP scripts that are intended to be executable. -To use the helper effectively, simply add the `vendorHash` attribute, which +To use the helper effectively, add the `vendorHash` attribute, which enables the wrapper to handle the heavy lifting. Internally, the helper operates in three stages: @@ -200,7 +200,7 @@ Internally, the helper operates in three stages: composer repository on the filesystem containing dependencies specified in `composer.json`. This process uses the function `php.mkComposerRepository` which in turn uses the - `php.composerHooks.composerRepositoryHook` hook. Internaly this function uses + `php.composerHooks.composerRepositoryHook` hook. Internally this function uses a custom [Composer plugin](https://github.com/nix-community/composer-local-repo-plugin) to generate the repository. diff --git a/doc/languages-frameworks/python.section.md b/doc/languages-frameworks/python.section.md index 5291fc300faaf..19d4496eef516 100644 --- a/doc/languages-frameworks/python.section.md +++ b/doc/languages-frameworks/python.section.md @@ -9,9 +9,10 @@ | python27 | python2, python | CPython 2.7 | | python38 | | CPython 3.8 | | python39 | | CPython 3.9 | -| python310 | python3 | CPython 3.10 | -| python311 | | CPython 3.11 | +| python310 | | CPython 3.10 | +| python311 | python3 | CPython 3.11 | | python312 | | CPython 3.12 | +| python313 | | CPython 3.13 | | pypy27 | pypy2, pypy | PyPy2.7 | | pypy39 | pypy3 | PyPy 3.9 | @@ -32,8 +33,8 @@ Each interpreter has the following attributes: - `libPrefix`. Name of the folder in `${python}/lib/` for corresponding interpreter. - `interpreter`. Alias for `${python}/bin/${executable}`. -- `buildEnv`. Function to build python interpreter environments with extra packages bundled together. See section *python.buildEnv function* for usage and documentation. -- `withPackages`. Simpler interface to `buildEnv`. See section *python.withPackages function* for usage and documentation. +- `buildEnv`. Function to build python interpreter environments with extra packages bundled together. See [](#python.buildenv-function) for usage and documentation. +- `withPackages`. Simpler interface to `buildEnv`. See [](#python.withpackages-function) for usage and documentation. - `sitePackages`. Alias for `lib/${libPrefix}/site-packages`. - `executable`. Name of the interpreter executable, e.g. `python3.10`. - `pkgs`. Set of Python packages for that specific interpreter. The package set can be modified by overriding the interpreter and passing `packageOverrides`. @@ -41,8 +42,8 @@ Each interpreter has the following attributes: ### Building packages and applications {#building-packages-and-applications} Python libraries and applications that use `setuptools` or -`distutils` are typically built with respectively the `buildPythonPackage` and -`buildPythonApplication` functions. These two functions also support installing a `wheel`. +`distutils` are typically built with respectively the [`buildPythonPackage`](#buildpythonpackage-function) and +[`buildPythonApplication`](#buildpythonapplication-function) functions. These two functions also support installing a `wheel`. All Python packages reside in `pkgs/top-level/python-packages.nix` and all applications elsewhere. In case a package is used as both a library and an @@ -63,12 +64,14 @@ sets are * `pkgs.python39Packages` * `pkgs.python310Packages` * `pkgs.python311Packages` +* `pkgs.python312Packages` +* `pkgs.python313Packages` * `pkgs.pypyPackages` and the aliases * `pkgs.python2Packages` pointing to `pkgs.python27Packages` -* `pkgs.python3Packages` pointing to `pkgs.python310Packages` +* `pkgs.python3Packages` pointing to `pkgs.python311Packages` * `pkgs.pythonPackages` pointing to `pkgs.python2Packages` #### `buildPythonPackage` function {#buildpythonpackage-function} @@ -141,23 +144,23 @@ buildPythonPackage rec { The `buildPythonPackage` mainly does four things: -* In the `buildPhase`, it calls `${python.pythonForBuild.interpreter} setup.py bdist_wheel` to +* In the [`buildPhase`](#build-phase), it calls `${python.pythonOnBuildForHost.interpreter} setup.py bdist_wheel` to build a wheel binary zipfile. -* In the `installPhase`, it installs the wheel file using `pip install *.whl`. -* In the `postFixup` phase, the `wrapPythonPrograms` bash function is called to +* In the [`installPhase`](#ssec-install-phase), it installs the wheel file using `pip install *.whl`. +* In the [`postFixup`](#var-stdenv-postFixup) phase, the `wrapPythonPrograms` bash function is called to wrap all programs in the `$out/bin/*` directory to include `$PATH` environment variable and add dependent libraries to script's `sys.path`. -* In the `installCheck` phase, `${python.interpreter} setup.py test` is run. +* In the [`installCheck`](#ssec-installCheck-phase) phase, `${python.interpreter} setup.py test` is run. -By default tests are run because `doCheck = true`. Test dependencies, like -e.g. the test runner, should be added to `nativeCheckInputs`. +By default tests are run because [`doCheck = true`](#var-stdenv-doCheck). Test dependencies, like +e.g. the test runner, should be added to [`nativeCheckInputs`](#var-stdenv-nativeCheckInputs). By default `meta.platforms` is set to the same value as the interpreter unless overridden otherwise. ##### `buildPythonPackage` parameters {#buildpythonpackage-parameters} -All parameters from `stdenv.mkDerivation` function are still supported. The +All parameters from [`stdenv.mkDerivation`](#sec-using-stdenv) function are still supported. The following are specific to `buildPythonPackage`: * `catchConflicts ? true`: If `true`, abort package build if a package name @@ -177,8 +180,8 @@ following are specific to `buildPythonPackage`: format. When unset, the legacy `setuptools` hooks are used for backwards compatibility. * `makeWrapperArgs ? []`: A list of strings. Arguments to be passed to - `makeWrapper`, which wraps generated binaries. By default, the arguments to - `makeWrapper` set `PATH` and `PYTHONPATH` environment variables before calling + [`makeWrapper`](#fun-makeWrapper), which wraps generated binaries. By default, the arguments to + [`makeWrapper`](#fun-makeWrapper) set `PATH` and `PYTHONPATH` environment variables before calling the binary. Additional arguments here can allow a developer to set environment variables which will be available when the binary is run. For example, `makeWrapperArgs = ["--set FOO BAR" "--set BAZ QUX"]`. @@ -190,7 +193,7 @@ following are specific to `buildPythonPackage`: * `pipBuildFlags ? []`: A list of strings. Arguments to be passed to `pip wheel`. * `pypaBuildFlags ? []`: A list of strings. Arguments to be passed to `python -m build --wheel`. * `pythonPath ? []`: List of packages to be added into `$PYTHONPATH`. Packages - in `pythonPath` are not propagated (contrary to `propagatedBuildInputs`). + in `pythonPath` are not propagated (contrary to [`propagatedBuildInputs`](#var-stdenv-propagatedBuildInputs)). * `preShellHook`: Hook to execute commands before `shellHook`. * `postShellHook`: Hook to execute commands after `shellHook`. * `removeBinByteCode ? true`: Remove bytecode from `/bin`. Bytecode is only @@ -198,7 +201,7 @@ following are specific to `buildPythonPackage`: * `setupPyGlobalFlags ? []`: List of flags passed to `setup.py` command. * `setupPyBuildFlags ? []`: List of flags passed to `setup.py build_ext` command. -The `stdenv.mkDerivation` function accepts various parameters for describing +The [`stdenv.mkDerivation`](#sec-using-stdenv) function accepts various parameters for describing build inputs (see "Specifying dependencies"). The following are of special interest for Python packages, either because these are primarily used, or because their behaviour is different: @@ -208,8 +211,8 @@ because their behaviour is different: * `buildInputs ? []`: Build and/or run-time dependencies that need to be compiled for the host machine. Typically non-Python libraries which are being linked. -* `nativeCheckInputs ? []`: Dependencies needed for running the `checkPhase`. These - are added to `nativeBuildInputs` when `doCheck = true`. Items listed in +* `nativeCheckInputs ? []`: Dependencies needed for running the [`checkPhase`](#ssec-check-phase). These + are added to [`nativeBuildInputs`](#var-stdenv-nativeBuildInputs) when [`doCheck = true`](#var-stdenv-doCheck). Items listed in `tests_require` go here. * `propagatedBuildInputs ? []`: Aside from propagating dependencies, `buildPythonPackage` also injects code into and wraps executables with the @@ -261,23 +264,23 @@ python3MyBlas = pkgs.python3.override { ``` This is particularly useful for numpy and scipy users who want to gain speed with other blas implementations. -Note that using simply `scipy = super.scipy.override { blas = super.pkgs.mkl; };` will likely result in +Note that using `scipy = super.scipy.override { blas = super.pkgs.mkl; };` will likely result in compilation issues, because scipy dependencies need to use the same blas implementation as well. #### `buildPythonApplication` function {#buildpythonapplication-function} -The `buildPythonApplication` function is practically the same as -`buildPythonPackage`. The main purpose of this function is to build a Python +The [`buildPythonApplication`](#buildpythonapplication-function) function is practically the same as +[`buildPythonPackage`](#buildpythonpackage-function). The main purpose of this function is to build a Python package where one is interested only in the executables, and not importable -modules. For that reason, when adding this package to a `python.buildEnv`, the +modules. For that reason, when adding this package to a [`python.buildEnv`](#python.buildenv-function), the modules won't be made available. -Another difference is that `buildPythonPackage` by default prefixes the names of +Another difference is that [`buildPythonPackage`](#buildpythonpackage-function) by default prefixes the names of the packages with the version of the interpreter. Because this is irrelevant for applications, the prefix is omitted. -When packaging a Python application with `buildPythonApplication`, it should be -called with `callPackage` and passed `python` or `pythonPackages` (possibly +When packaging a Python application with [`buildPythonApplication`](#buildpythonapplication-function), it should be +called with `callPackage` and passed `python3` or `python3Packages` (possibly specifying an interpreter version), like this: ```nix @@ -319,7 +322,7 @@ luigi = callPackage ../applications/networking/cluster/luigi { }; ``` Since the package is an application, a consumer doesn't need to care about -Python versions or modules, which is why they don't go in `pythonPackages`. +Python versions or modules, which is why they don't go in `python3Packages`. #### `toPythonApplication` function {#topythonapplication-function} @@ -329,19 +332,19 @@ package is used as both. In this case the package is added as a library to duplication the `toPythonApplication` can be used to convert a library to an application. -The Nix expression shall use `buildPythonPackage` and be called from +The Nix expression shall use [`buildPythonPackage`](#buildpythonpackage-function) and be called from `python-packages.nix`. A reference shall be created from `all-packages.nix` to the attribute in `python-packages.nix`, and the `toPythonApplication` shall be applied to the reference: ```nix -youtube-dl = with pythonPackages; toPythonApplication youtube-dl; +youtube-dl = with python3Packages; toPythonApplication youtube-dl; ``` #### `toPythonModule` function {#topythonmodule-function} In some cases, such as bindings, a package is created using -`stdenv.mkDerivation` and added as attribute in `all-packages.nix`. The Python +[`stdenv.mkDerivation`](#sec-using-stdenv) and added as attribute in `all-packages.nix`. The Python bindings should be made available from `python-packages.nix`. The `toPythonModule` function takes a derivation and makes certain Python-specific modifications. @@ -364,8 +367,8 @@ Saving the following as `default.nix` ```nix with import <nixpkgs> {}; -python.buildEnv.override { - extraLibs = [ pythonPackages.pyramid ]; +python3.buildEnv.override { + extraLibs = [ python3Packages.pyramid ]; ignoreCollisions = true; } ``` @@ -407,9 +410,9 @@ specified packages in its path. #### `python.withPackages` function {#python.withpackages-function} -The `python.withPackages` function provides a simpler interface to the `python.buildEnv` functionality. +The [`python.withPackages`](#python.withpackages-function) function provides a simpler interface to the [`python.buildEnv`](#python.buildenv-function) functionality. It takes a function as an argument that is passed the set of python packages and returns the list -of the packages to be included in the environment. Using the `withPackages` function, the previous +of the packages to be included in the environment. Using the [`withPackages`](#python.withpackages-function) function, the previous example for the Pyramid Web Framework environment can be written like this: ```nix @@ -418,7 +421,7 @@ with import <nixpkgs> {}; python.withPackages (ps: [ ps.pyramid ]) ``` -`withPackages` passes the correct package set for the specific interpreter +[`withPackages`](#python.withpackages-function) passes the correct package set for the specific interpreter version as an argument to the function. In the above example, `ps` equals `pythonPackages`. But you can also easily switch to using python3: @@ -430,7 +433,7 @@ python3.withPackages (ps: [ ps.pyramid ]) Now, `ps` is set to `python3Packages`, matching the version of the interpreter. -As `python.withPackages` simply uses `python.buildEnv` under the hood, it also +As [`python.withPackages`](#python.withpackages-function) uses [`python.buildEnv`](#python.buildenv-function) under the hood, it also supports the `env` attribute. The `shell.nix` file from the previous section can thus be also written like this: @@ -443,23 +446,22 @@ with import <nixpkgs> {}; ])).env ``` -In contrast to `python.buildEnv`, `python.withPackages` does not support the +In contrast to [`python.buildEnv`](#python.buildenv-function), [`python.withPackages`](#python.withpackages-function) does not support the more advanced options such as `ignoreCollisions = true` or `postBuild`. If you -need them, you have to use `python.buildEnv`. +need them, you have to use [`python.buildEnv`](#python.buildenv-function). Python 2 namespace packages may provide `__init__.py` that collide. In that case -`python.buildEnv` should be used with `ignoreCollisions = true`. +[`python.buildEnv`](#python.buildenv-function) should be used with `ignoreCollisions = true`. #### Setup hooks {#setup-hooks} The following are setup hooks specifically for Python packages. Most of these -are used in `buildPythonPackage`. +are used in [`buildPythonPackage`](#buildpythonpackage-function). - `eggUnpackhook` to move an egg to the correct folder so it can be installed with the `eggInstallHook` - `eggBuildHook` to skip building for eggs. - `eggInstallHook` to install eggs. -- `flitBuildHook` to build a wheel using `flit`. - `pipBuildHook` to build a wheel using `pip` and PEP 517. Note a build system (e.g. `setuptools` or `flit`) should still be added as `nativeBuildInput`. - `pypaBuildHook` to build a wheel using @@ -486,7 +488,7 @@ are used in `buildPythonPackage`. ### Development mode {#development-mode} Development or editable mode is supported. To develop Python packages -`buildPythonPackage` has additional logic inside `shellPhase` to run `pip +[`buildPythonPackage`](#buildpythonpackage-function) has additional logic inside `shellPhase` to run `pip install -e . --prefix $TMPDIR/`for the package. Warning: `shellPhase` is executed only if `setup.py` exists. @@ -496,9 +498,9 @@ Given a `default.nix`: ```nix with import <nixpkgs> {}; -pythonPackages.buildPythonPackage { +python3Packages.buildPythonPackage { name = "myproject"; - buildInputs = with pythonPackages; [ pyramid ]; + buildInputs = with python3Packages; [ pyramid ]; src = ./.; } @@ -510,7 +512,7 @@ the package would be built with `nix-build`. Shortcut to setup environments with C headers/libraries and Python packages: ```shell -nix-shell -p pythonPackages.pyramid zlib libjpeg git +nix-shell -p python3Packages.pyramid zlib libjpeg git ``` ::: {.note} @@ -525,7 +527,7 @@ There is a boolean value `lib.inNixShell` set to `true` if nix-shell is invoked. Several versions of the Python interpreter are available on Nix, as well as a high amount of packages. The attribute `python3` refers to the default -interpreter, which is currently CPython 3.10. The attribute `python` refers to +interpreter, which is currently CPython 3.11. The attribute `python` refers to CPython 2.7 for backwards-compatibility. It is also possible to refer to specific versions, e.g. `python311` refers to CPython 3.11, and `pypy` refers to the default PyPy interpreter. @@ -543,7 +545,7 @@ however, are in separate sets, with one set per interpreter version. The interpreters have several common attributes. One of these attributes is `pkgs`, which is a package set of Python libraries for this specific interpreter. E.g., the `toolz` package corresponding to the default interpreter -is `python.pkgs.toolz`, and the CPython 3.11 version is `python311.pkgs.toolz`. +is `python3.pkgs.toolz`, and the CPython 3.11 version is `python311.pkgs.toolz`. The main package set contains aliases to these package sets, e.g. `pythonPackages` refers to `python.pkgs` and `python311Packages` to `python311.pkgs`. @@ -567,7 +569,7 @@ without impacting other applications or polluting your user environment. But Python libraries you would like to use for development cannot be installed, at least not individually, because they won't be able to find each other resulting in import errors. Instead, it is possible to create an environment -with `python.buildEnv` or `python.withPackages` where the interpreter and other +with [`python.buildEnv`](#python.buildenv-function) or [`python.withPackages`](#python.withpackages-function) where the interpreter and other executables are wrapped to be able to find each other and all of the modules. In the following examples we will start by creating a simple, ad-hoc environment @@ -680,7 +682,7 @@ b = np.array([3,4]) print(f"The dot product of {a} and {b} is: {np.dot(a, b)}") ``` -Then we simply execute it, without requiring any environment setup at all! +Then we execute it, without requiring any environment setup at all! ```sh $ ./foo.py @@ -747,8 +749,8 @@ What's happening here? imports the `<nixpkgs>` function, `{}` calls it and the `with` statement brings all attributes of `nixpkgs` in the local scope. These attributes form the main package set. -2. Then we create a Python 3.11 environment with the `withPackages` function, as before. -3. The `withPackages` function expects us to provide a function as an argument +2. Then we create a Python 3.11 environment with the [`withPackages`](#python.withpackages-function) function, as before. +3. The [`withPackages`](#python.withpackages-function) function expects us to provide a function as an argument that takes the set of all Python packages and returns a list of packages to include in the environment. Here, we select the packages `numpy` and `toolz` from the package set. @@ -859,7 +861,7 @@ we will look at how you can use development mode with your code. #### Python library packages in Nixpkgs {#python-library-packages-in-nixpkgs} With Nix all packages are built by functions. The main function in Nix for -building Python libraries is `buildPythonPackage`. Let's see how we can build the +building Python libraries is [`buildPythonPackage`](#buildpythonpackage-function). Let's see how we can build the `toolz` package. ```nix @@ -904,13 +906,13 @@ buildPythonPackage rec { } ``` -What happens here? The function `buildPythonPackage` is called and as argument +What happens here? The function [`buildPythonPackage`](#buildpythonpackage-function) is called and as argument it accepts a set. In this case the set is a recursive set, `rec`. One of the arguments is the name of the package, which consists of a basename (generally following the name on PyPi) and a version. Another argument, `src` specifies the source, which in this case is fetched from PyPI using the helper function `fetchPypi`. The argument `doCheck` is used to set whether tests should be run -when building the package. Since there are no tests, we rely on `pythonImportsCheck` +when building the package. Since there are no tests, we rely on [`pythonImportsCheck`](#using-pythonimportscheck) to test whether the package can be imported. Furthermore, we specify some meta information. The output of the function is a derivation. @@ -969,7 +971,7 @@ for which Python version we want to build a package. So, what did we do here? Well, we took the Nix expression that we used earlier to build a Python environment, and said that we wanted to include our own version of `toolz`, named `my_toolz`. To introduce our own package in the scope -of `withPackages` we used a `let` expression. You can see that we used +of [`withPackages`](#python.withpackages-function) we used a `let` expression. You can see that we used `ps.numpy` to select numpy from the nixpkgs package set (`ps`). We did not take `toolz` from the Nixpkgs package set this time, but instead took our own version that we introduced with the `let` expression. @@ -977,14 +979,14 @@ that we introduced with the `let` expression. #### Handling dependencies {#handling-dependencies} Our example, `toolz`, does not have any dependencies on other Python packages or -system libraries. According to the manual, `buildPythonPackage` uses the -arguments `buildInputs` and `propagatedBuildInputs` to specify dependencies. If +system libraries. According to the manual, [`buildPythonPackage`](#buildpythonpackage-function) uses the +arguments [`buildInputs`](#var-stdenv-buildInputs) and [`propagatedBuildInputs`](#var-stdenv-propagatedBuildInputs) to specify dependencies. If something is exclusively a build-time dependency, then the dependency should be -included in `buildInputs`, but if it is (also) a runtime dependency, then it -should be added to `propagatedBuildInputs`. Test dependencies are considered -build-time dependencies and passed to `nativeCheckInputs`. +included in [`buildInputs`](#var-stdenv-buildInputs), but if it is (also) a runtime dependency, then it +should be added to [`propagatedBuildInputs`](#var-stdenv-propagatedBuildInputs). Test dependencies are considered +build-time dependencies and passed to [`nativeCheckInputs`](#var-stdenv-nativeCheckInputs). -The following example shows which arguments are given to `buildPythonPackage` in +The following example shows which arguments are given to [`buildPythonPackage`](#buildpythonpackage-function) in order to build [`datashape`](https://github.com/blaze/datashape). ```nix @@ -1038,14 +1040,14 @@ buildPythonPackage rec { ``` We can see several runtime dependencies, `numpy`, `multipledispatch`, and -`python-dateutil`. Furthermore, we have `nativeCheckInputs` with `pytest`. -`pytest` is a test runner and is only used during the `checkPhase` and is -therefore not added to `propagatedBuildInputs`. +`python-dateutil`. Furthermore, we have [`nativeCheckInputs`](#var-stdenv-nativeCheckInputs) with `pytest`. +`pytest` is a test runner and is only used during the [`checkPhase`](#ssec-check-phase) and is +therefore not added to [`propagatedBuildInputs`](#var-stdenv-propagatedBuildInputs). In the previous case we had only dependencies on other Python packages to consider. Occasionally you have also system libraries to consider. E.g., `lxml` provides Python bindings to `libxml2` and `libxslt`. These libraries are only required -when building the bindings and are therefore added as `buildInputs`. +when building the bindings and are therefore added as [`buildInputs`](#var-stdenv-buildInputs). ```nix { lib @@ -1093,7 +1095,7 @@ files of the dependencies are. This is not always the case. The example below shows bindings to The Fastest Fourier Transform in the West, commonly known as FFTW. On Nix we have separate packages of FFTW for the different types of floats (`"single"`, `"double"`, `"long-double"`). The -bindings need all three types, and therefore we add all three as `buildInputs`. +bindings need all three types, and therefore we add all three as [`buildInputs`](#var-stdenv-buildInputs). The bindings don't expect to find each of them in a different folder, and therefore we have to set `LDFLAGS` and `CFLAGS`. @@ -1158,7 +1160,7 @@ buildPythonPackage rec { } ``` -Note also the line `doCheck = false;`, we explicitly disabled running the test-suite. +Note also the line [`doCheck = false;`](#var-stdenv-doCheck), we explicitly disabled running the test-suite. #### Testing Python Packages {#testing-python-packages} @@ -1167,10 +1169,10 @@ helps to avoid situations where the package was able to build and install, but is not usable at runtime. Currently, all packages will use the `test` command provided by the setup.py (i.e. `python setup.py test`). However, this is currently deprecated https://github.com/pypa/setuptools/pull/1878 -and your package should provide its own checkPhase. +and your package should provide its own [`checkPhase`](#ssec-check-phase). ::: {.note} -The `checkPhase` for python maps to the `installCheckPhase` on a +The [`checkPhase`](#ssec-check-phase) for python maps to the `installCheckPhase` on a normal derivation. This is due to many python packages not behaving well to the pre-installed version of the package. Version info, and natively compiled extensions generally only exist in the install directory, and @@ -1235,7 +1237,7 @@ been removed, in this case, it's recommended to use `pytestCheckHook`. #### Using pytestCheckHook {#using-pytestcheckhook} `pytestCheckHook` is a convenient hook which will substitute the setuptools -`test` command for a `checkPhase` which runs `pytest`. This is also beneficial +`test` command for a [`checkPhase`](#ssec-check-phase) which runs `pytest`. This is also beneficial when a package may need many items disabled to run the test suite. Using the example above, the analogous `pytestCheckHook` usage would be: @@ -1280,14 +1282,14 @@ for example: ``` Trying to concatenate the related strings to disable tests in a regular -`checkPhase` would be much harder to read. This also enables us to comment on +[`checkPhase`](#ssec-check-phase) would be much harder to read. This also enables us to comment on why specific tests are disabled. #### Using pythonImportsCheck {#using-pythonimportscheck} Although unit tests are highly preferred to validate correctness of a package, not all packages have test suites that can be run easily, and some have none at all. -To help ensure the package still works, `pythonImportsCheck` can attempt to import +To help ensure the package still works, [`pythonImportsCheck`](#using-pythonimportscheck) can attempt to import the listed modules. ``` @@ -1306,7 +1308,7 @@ roughly translates to: ''; ``` -However, this is done in its own phase, and not dependent on whether `doCheck = true;`. +However, this is done in its own phase, and not dependent on whether [`doCheck = true;`](#var-stdenv-doCheck). This can also be useful in verifying that the package doesn't assume commonly present packages (e.g. `setuptools`). @@ -1378,11 +1380,11 @@ instead of a dev dependency). Keep in mind that while the examples above are done with `requirements.txt`, `pythonRelaxDepsHook` works by modifying the resulting wheel file, so it should -work with any of the existing [hooks](#setup-hooks). +work with any of the [existing hooks](#setup-hooks). #### Using unittestCheckHook {#using-unittestcheckhook} -`unittestCheckHook` is a hook which will substitute the setuptools `test` command for a `checkPhase` which runs `python -m unittest discover`: +`unittestCheckHook` is a hook which will substitute the setuptools `test` command for a [`checkPhase`](#ssec-check-phase) which runs `python -m unittest discover`: ``` nativeCheckInputs = [ @@ -1452,15 +1454,15 @@ mode is also available. Let's see how you can use it. In the previous Nix expression the source was fetched from a url. We can also refer to a local source instead using `src = ./path/to/source/tree;` -If we create a `shell.nix` file which calls `buildPythonPackage`, and if `src` +If we create a `shell.nix` file which calls [`buildPythonPackage`](#buildpythonpackage-function), and if `src` is a local source, and if the local source has a `setup.py`, then development mode is activated. In the following example, we create a simple environment that has a Python 3.11 version of our package in it, as well as its dependencies and other packages we -like to have in the environment, all specified with `propagatedBuildInputs`. +like to have in the environment, all specified with [`propagatedBuildInputs`](#var-stdenv-propagatedBuildInputs). Indeed, we can just add any package we like to have in our environment to -`propagatedBuildInputs`. +[`propagatedBuildInputs`](#var-stdenv-propagatedBuildInputs). ```nix with import <nixpkgs> {}; @@ -1494,7 +1496,7 @@ own packages. The important functions here are `import` and `callPackage`. ### Including a derivation using `callPackage` {#including-a-derivation-using-callpackage} -Earlier we created a Python environment using `withPackages`, and included the +Earlier we created a Python environment using [`withPackages`](#python.withpackages-function), and included the `toolz` package via a `let` expression. Let's split the package definition from the environment definition. @@ -1533,7 +1535,7 @@ buildPythonPackage rec { } ``` -It takes an argument `buildPythonPackage`. We now call this function using +It takes an argument [`buildPythonPackage`](#buildpythonpackage-function). We now call this function using `callPackage` in the definition of our environment ```nix @@ -1552,10 +1554,10 @@ Packages.buildPythonPackage; ``` Important to remember is that the Python version for which the package is made -depends on the `python` derivation that is passed to `buildPythonPackage`. Nix +depends on the `python` derivation that is passed to [`buildPythonPackage`](#buildpythonpackage-function). Nix tries to automatically pass arguments when possible, which is why generally you don't explicitly define which `python` derivation should be used. In the above -example we use `buildPythonPackage` that is part of the set `python3Packages`, +example we use [`buildPythonPackage`](#buildpythonpackage-function) that is part of the set `python3Packages`, and in this case the `python3` interpreter is automatically used. ## FAQ {#faq} @@ -1682,7 +1684,7 @@ of such package using the feature is `pkgs/tools/X11/xpra/default.nix`. As workaround install it as an extra `preInstall` step: ```shell -${python.pythonForBuild.interpreter} setup.py install_data --install-dir=$out --root=$out +${python.pythonOnBuildForHost.interpreter} setup.py install_data --install-dir=$out --root=$out sed -i '/ = data\_files/d' setup.py ``` @@ -1698,7 +1700,7 @@ Python, guarantees the right versions of the interpreter and libraries or packages are available. There is therefore no need to maintain a global `site-packages`. If you want to create a Python environment for development, then the recommended -method is to use `nix-shell`, either with or without the `python.buildEnv` +method is to use `nix-shell`, either with or without the [`python.buildEnv`](#python.buildenv-function) function. ### How to consume Python modules using pip in a virtual environment like I am used to on other Operating Systems? {#how-to-consume-python-modules-using-pip-in-a-virtual-environment-like-i-am-used-to-on-other-operating-systems} @@ -1711,7 +1713,7 @@ This is an example of a `default.nix` for a `nix-shell`, which allows to consume a virtual environment created by `venv`, and install Python modules through `pip` the traditional way. -Create this `default.nix` file, together with a `requirements.txt` and simply +Create this `default.nix` file, together with a `requirements.txt` and execute `nix-shell`. ```nix @@ -1835,7 +1837,7 @@ If you need to change a package's attribute(s) from `configuration.nix` you coul }; ``` -`pythonPackages.twisted` is now globally overridden. +`python3Packages.twisted` is now globally overridden. All packages and also all NixOS services that reference `twisted` (such as `services.buildbot-worker`) now use the new definition. Note that `python-super` refers to the old package set and `python-self` @@ -1845,7 +1847,7 @@ To modify only a Python package set instead of a whole Python derivation, use this snippet: ```nix - myPythonPackages = pythonPackages.override { + myPythonPackages = python3Packages.override { overrides = self: super: { twisted = ...; }; @@ -1875,7 +1877,7 @@ self: super: { ### How to override a Python package for all Python versions using extensions? {#how-to-override-a-python-package-for-all-python-versions-using-extensions} -The following overlay overrides the call to `buildPythonPackage` for the +The following overlay overrides the call to [`buildPythonPackage`](#buildpythonpackage-function) for the `foo` package for all interpreters by appending a Python extension to the `pythonPackagesExtensions` list of extensions. @@ -1902,9 +1904,9 @@ configure alternatives](#sec-overlays-alternatives-blas-lapack)". In a `setup.py` or `setup.cfg` it is common to declare dependencies: -* `setup_requires` corresponds to `nativeBuildInputs` -* `install_requires` corresponds to `propagatedBuildInputs` -* `tests_require` corresponds to `nativeCheckInputs` +* `setup_requires` corresponds to [`nativeBuildInputs`](#var-stdenv-nativeBuildInputs) +* `install_requires` corresponds to [`propagatedBuildInputs`](#var-stdenv-propagatedBuildInputs) +* `tests_require` corresponds to [`nativeCheckInputs`](#var-stdenv-nativeCheckInputs) ### How to enable interpreter optimizations? {#optimizations} @@ -1951,7 +1953,7 @@ collisions. ### How to contribute a Python package to nixpkgs? {#tools} -Packages inside nixpkgs must use the `buildPythonPackage` or `buildPythonApplication` function directly, +Packages inside nixpkgs must use the [`buildPythonPackage`](#buildpythonpackage-function) or [`buildPythonApplication`](#buildpythonapplication-function) function directly, because we can only provide security support for non-vendored dependencies. We recommend [nix-init](https://github.com/nix-community/nix-init) for creating new python packages within nixpkgs, @@ -1965,7 +1967,7 @@ has security implications and is relevant for those using Python in a `nix-shell`. When the environment variable `DETERMINISTIC_BUILD` is set, all bytecode will -have timestamp 1. The `buildPythonPackage` function sets `DETERMINISTIC_BUILD=1` +have timestamp 1. The [`buildPythonPackage`](#buildpythonpackage-function) function sets `DETERMINISTIC_BUILD=1` and [PYTHONHASHSEED=0](https://docs.python.org/3.11/using/cmdline.html#envvar-PYTHONHASHSEED). Both are also exported in `nix-shell`. @@ -1975,12 +1977,12 @@ It is recommended to test packages as part of the build process. Source distributions (`sdist`) often include test files, but not always. By default the command `python setup.py test` is run as part of the -`checkPhase`, but often it is necessary to pass a custom `checkPhase`. An +[`checkPhase`](#ssec-check-phase), but often it is necessary to pass a custom [`checkPhase`](#ssec-check-phase). An example of such a situation is when `py.test` is used. #### Common issues {#common-issues} -* Non-working tests can often be deselected. By default `buildPythonPackage` +* Non-working tests can often be deselected. By default [`buildPythonPackage`](#buildpythonpackage-function) runs `python setup.py test`. which is deprecated. Most Python modules however do follow the standard test protocol where the pytest runner can be used instead. `pytest` supports the `-k` and `--ignore` parameters to ignore test @@ -2015,17 +2017,19 @@ example of such a situation is when `py.test` is used. The following rules are desired to be respected: * Python libraries are called from `python-packages.nix` and packaged with - `buildPythonPackage`. The expression of a library should be in + [`buildPythonPackage`](#buildpythonpackage-function). The expression of a library should be in `pkgs/development/python-modules/<name>/default.nix`. * Python applications live outside of `python-packages.nix` and are packaged - with `buildPythonApplication`. + with [`buildPythonApplication`](#buildpythonapplication-function). * Make sure libraries build for all Python interpreters. * By default we enable tests. Make sure the tests are found and, in the case of libraries, are passing for all interpreters. If certain tests fail they can be disabled individually. Try to avoid disabling the tests altogether. In any case, when you disable tests, leave a comment explaining why. * Commit names of Python libraries should reflect that they are Python - libraries, so write for example `pythonPackages.numpy: 1.11 -> 1.12`. + libraries, so write for example `python311Packages.numpy: 1.11 -> 1.12`. + It is highly recommended to specify the current default version to enable + automatic build by ofborg. * Attribute names in `python-packages.nix` as well as `pname`s should match the library's name on PyPI, but be normalized according to [PEP 0503](https://www.python.org/dev/peps/pep-0503/#normalized-names). This means diff --git a/doc/languages-frameworks/ruby.section.md b/doc/languages-frameworks/ruby.section.md index d3b896686c067..920c84eee689c 100644 --- a/doc/languages-frameworks/ruby.section.md +++ b/doc/languages-frameworks/ruby.section.md @@ -94,7 +94,7 @@ $ bundle lock $ bundix ``` -If you already have a `Gemfile.lock`, you can simply run `bundix` and it will work the same. +If you already have a `Gemfile.lock`, you can run `bundix` and it will work the same. To update the gems in your `Gemfile.lock`, you may use the `bundix -l` flag, which will create a new `Gemfile.lock` in case the `Gemfile` has a more recent time of modification. @@ -251,7 +251,7 @@ source 'https://rubygems.org' do end ``` -If you want to package a specific version, you can use the standard Gemfile syntax for that, e.g. `gem 'mdl', '0.5.0'`, but if you want the latest stable version anyway, it's easier to update by simply running the `bundle lock` and `bundix` steps again. +If you want to package a specific version, you can use the standard Gemfile syntax for that, e.g. `gem 'mdl', '0.5.0'`, but if you want the latest stable version anyway, it's easier to update by running the `bundle lock` and `bundix` steps again. Now you can also make a `default.nix` that looks like this: diff --git a/doc/languages-frameworks/rust.section.md b/doc/languages-frameworks/rust.section.md index 67e23cc74d4e4..3bd8e1c765120 100644 --- a/doc/languages-frameworks/rust.section.md +++ b/doc/languages-frameworks/rust.section.md @@ -817,7 +817,7 @@ $ cargo test ## Using community maintained Rust toolchains {#using-community-maintained-rust-toolchains} ::: {.note} -Note: The following projects cannot be used within nixpkgs since [IFD](#ssec-import-from-derivation) is disallowed. +The following projects cannot be used within Nixpkgs since [Import From Derivation](https://nixos.org/manual/nix/unstable/language/import-from-derivation) (IFD) is disallowed in Nixpkgs. To package things that require Rust nightly, `RUSTC_BOOTSTRAP = true;` can sometimes be used as a hack. ::: diff --git a/doc/languages-frameworks/swift.section.md b/doc/languages-frameworks/swift.section.md index 1cc452cc9b9bf..213d444f499fa 100644 --- a/doc/languages-frameworks/swift.section.md +++ b/doc/languages-frameworks/swift.section.md @@ -32,7 +32,7 @@ look for the following directories: (If not targeting macOS, replace `macosx` with the Xcode platform name.) - On other platforms: `lib/swift/linux/x86_64` (Where `linux` and `x86_64` are from lowercase `uname -sm`.) -- For convenience, Nixpkgs also adds simply `lib/swift` to the search path. +- For convenience, Nixpkgs also adds `lib/swift` to the search path. This can save a bit of work packaging Swift modules, because many Nix builds will produce output for just one target any way. @@ -123,7 +123,7 @@ swiftpmFlags = [ "--disable-dead-strip" ]; The default `buildPhase` already passes `-j` for parallel building. -If these two customization options are insufficient, simply provide your own +If these two customization options are insufficient, provide your own `buildPhase` that invokes `swift build`. ### Running tests {#ssec-swiftpm-running-tests} diff --git a/doc/languages-frameworks/texlive.section.md b/doc/languages-frameworks/texlive.section.md index a4c81daa54bbb..2ba846dc492d9 100644 --- a/doc/languages-frameworks/texlive.section.md +++ b/doc/languages-frameworks/texlive.section.md @@ -2,6 +2,46 @@ Since release 15.09 there is a new TeX Live packaging that lives entirely under attribute `texlive`. +## User's guide (experimental new interface) {#sec-language-texlive-user-guide-experimental} + +Release 23.11 ships with a new interface that will eventually replace `texlive.combine`. + +- For basic usage, use some of the prebuilt environments available at the top level, such as `texliveBasic`, `texliveSmall`. For the full list of prebuilt environments, inspect `texlive.schemes`. + +- Packages cannot be used directly but must be assembled in an environment. To create or add packages to an environment, use + ```nix + texliveSmall.withPackages (ps: with ps; [ collection-langkorean algorithms cm-super ]) + ``` + The function `withPackages` can be called multiple times to add more packages. + + - **Note.** Within Nixpkgs, packages should only use prebuilt environments as inputs, such as `texliveSmall` or `texliveInfraOnly`, and should not depend directly on `texlive`. Further dependencies should be added by calling `withPackages`. This is to ensure that there is a consistent and simple way to override the inputs. + +- `texlive.withPackages` uses the same logic as `buildEnv`. Only parts of a package are installed in an environment: its 'runtime' files (`tex` output), binaries (`out` output), and support files (`tlpkg` output). Moreover, man and info pages are assembled into separate `man` and `info` outputs. To add only the TeX files of a package, or its documentation (`texdoc` output), just specify the outputs: + ```nix + texlive.withPackages (ps: with ps; [ + texdoc # recommended package to navigate the documentation + perlPackages.LaTeXML.tex # tex files of LaTeXML, omit binaries + cm-super + cm-super.texdoc # documentation of cm-super + ]) + ``` + +- All packages distributed by TeX Live, which contains most of CTAN, are available and can be found under `texlive.pkgs`: + ```ShellSession + $ nix repl + nix-repl> :l <nixpkgs> + nix-repl> texlive.pkgs.[TAB] + ``` + Note that the packages in `texlive.pkgs` are only provided for search purposes and must not be used directly. + +- **Experimental and subject to change without notice:** to add the documentation for all packages in the environment, use + ```nix + texliveSmall.__overrideTeXConfig { withDocs = true; } + ``` + This can be applied before or after calling `withPackages`. + + The function currently support the parameters `withDocs`, `withSources`, and `requireTeXPackages`. + ## User's guide {#sec-language-texlive-user-guide} - For basic usage just pull `texlive.combined.scheme-basic` for an environment with basic LaTeX support. @@ -38,6 +78,24 @@ Since release 15.09 there is a new TeX Live packaging that lives entirely under - Note that the wrapper assumes that the result has a chance to be useful. For example, the core executables should be present, as well as some core data files. The supported way of ensuring this is by including some scheme, for example `scheme-basic`, into the combination. +- TeX Live packages are also available under `texlive.pkgs` as derivations with outputs `out`, `tex`, `texdoc`, `texsource`, `tlpkg`, `man`, `info`. They cannot be installed outside of `texlive.combine` but are available for other uses. To repackage a font, for instance, use + + ```nix + stdenvNoCC.mkDerivation rec { + src = texlive.pkgs.iwona; + + inherit (src) pname version; + + installPhase = '' + runHook preInstall + install -Dm644 fonts/opentype/nowacki/iwona/*.otf -t $out/share/fonts/opentype + runHook postInstall + ''; + } + ``` + + See `biber`, `iwona` for complete examples. + ## Custom packages {#sec-language-texlive-custom-packages} You may find that you need to use an external TeX package. A derivation for such package has to provide the contents of the "texmf" directory in its output and provide the appropriate `tlType` attribute (one of `"run"`, `"bin"`, `"doc"`, `"source"`). Dependencies on other TeX packages can be listed in the attribute `tlDeps`. diff --git a/doc/languages-frameworks/vim.section.md b/doc/languages-frameworks/vim.section.md index bf0d663179b92..1f3727f552c80 100644 --- a/doc/languages-frameworks/vim.section.md +++ b/doc/languages-frameworks/vim.section.md @@ -134,7 +134,7 @@ If one of your favourite plugins isn't packaged, you can package it yourself: { config, pkgs, ... }: let - easygrep = pkgs.vimUtils.buildVimPluginFrom2Nix { + easygrep = pkgs.vimUtils.buildVimPlugin { name = "vim-easygrep"; src = pkgs.fetchFromGitHub { owner = "dkprice"; @@ -212,9 +212,9 @@ Note: this is not possible anymore for Neovim. ## Adding new plugins to nixpkgs {#adding-new-plugins-to-nixpkgs} -Nix expressions for Vim plugins are stored in [pkgs/applications/editors/vim/plugins](https://github.com/NixOS/nixpkgs/tree/master/pkgs/applications/editors/vim/plugins). For the vast majority of plugins, Nix expressions are automatically generated by running [`./update.py`](https://github.com/NixOS/nixpkgs/blob/master/pkgs/applications/editors/vim/plugins/update.py). This creates a [generated.nix](https://github.com/NixOS/nixpkgs/blob/master/pkgs/applications/editors/vim/plugins/generated.nix) file based on the plugins listed in [vim-plugin-names](https://github.com/NixOS/nixpkgs/blob/master/pkgs/applications/editors/vim/plugins/vim-plugin-names). +Nix expressions for Vim plugins are stored in [pkgs/applications/editors/vim/plugins](https://github.com/NixOS/nixpkgs/tree/master/pkgs/applications/editors/vim/plugins). For the vast majority of plugins, Nix expressions are automatically generated by running [`nix-shell -p vimPluginsUpdater --run vim-plugins-updater`](https://github.com/NixOS/nixpkgs/blob/master/pkgs/applications/editors/vim/plugins/updater.nix). This creates a [generated.nix](https://github.com/NixOS/nixpkgs/blob/master/pkgs/applications/editors/vim/plugins/generated.nix) file based on the plugins listed in [vim-plugin-names](https://github.com/NixOS/nixpkgs/blob/master/pkgs/applications/editors/vim/plugins/vim-plugin-names). -After running `./update.py`, if nvim-treesitter received an update, also run [`nvim-treesitter/update.py`](https://github.com/NixOS/nixpkgs/blob/master/pkgs/applications/editors/vim/plugins/update.py) to update the tree sitter grammars for `nvim-treesitter`. +After running the updater, if nvim-treesitter received an update, also run [`nvim-treesitter/update.py`](https://github.com/NixOS/nixpkgs/blob/master/pkgs/applications/editors/vim/plugins/update.py) to update the tree sitter grammars for `nvim-treesitter`. Some plugins require overrides in order to function properly. Overrides are placed in [overrides.nix](https://github.com/NixOS/nixpkgs/blob/master/pkgs/applications/editors/vim/plugins/overrides.nix). Overrides are most often required when a plugin requires some dependencies, or extra steps are required during the build process. For example `deoplete-fish` requires both `deoplete-nvim` and `vim-fish`, and so the following override was added: @@ -241,7 +241,8 @@ GITHUB_API_TOKEN=my_token ./pkgs/applications/editors/vim/plugins/update.py Alternatively, set the number of processes to a lower count to avoid rate-limiting. ```sh -./pkgs/applications/editors/vim/plugins/update.py --proc 1 + +nix-shell -p vimPluginsUpdater --run 'vim-plugins-updater --proc 1' ``` ## How to maintain an out-of-tree overlay of vim plugins ? {#vim-out-of-tree-overlays} @@ -250,7 +251,7 @@ You can use the updater script to generate basic packages out of a custom vim plugin list: ``` -pkgs/applications/editors/vim/plugins/update.py -i vim-plugin-names -o generated.nix --no-commit +nix-shell -p vimPluginsUpdater --run vim-plugins-updater -i vim-plugin-names -o generated.nix --no-commit ``` with the contents of `vim-plugin-names` being for example: @@ -264,7 +265,7 @@ You can then reference the generated vim plugins via: ```nix myVimPlugins = pkgs.vimPlugins.extend ( - (pkgs.callPackage generated.nix {}) + (pkgs.callPackage ./generated.nix {}) ); ``` diff --git a/doc/manual.md.in b/doc/manual.md.in index 6b8d351380f95..52971ff526c28 100644 --- a/doc/manual.md.in +++ b/doc/manual.md.in @@ -9,7 +9,7 @@ preface.chapter.md using-nixpkgs.md lib.md stdenv.md -builders.md +build-helpers.md development.md contributing.md ``` diff --git a/doc/builders/packages/cataclysm-dda.section.md b/doc/packages/cataclysm-dda.section.md index f401e9b9efa53..f401e9b9efa53 100644 --- a/doc/builders/packages/cataclysm-dda.section.md +++ b/doc/packages/cataclysm-dda.section.md diff --git a/doc/builders/packages/citrix.section.md b/doc/packages/citrix.section.md index bcf0924249bce..bcf0924249bce 100644 --- a/doc/builders/packages/citrix.section.md +++ b/doc/packages/citrix.section.md diff --git a/doc/builders/special/darwin-builder.section.md b/doc/packages/darwin-builder.section.md index 13d01a0e3af83..89c2445667dcc 100644 --- a/doc/builders/special/darwin-builder.section.md +++ b/doc/packages/darwin-builder.section.md @@ -1,10 +1,10 @@ # darwin.linux-builder {#sec-darwin-builder} -`darwin.linux-builder` provides a way to bootstrap a Linux builder on a macOS machine. +`darwin.linux-builder` provides a way to bootstrap a Linux remote builder on a macOS machine. This requires macOS version 12.4 or later. -The builder runs on host port 31022 by default. +The remote builder runs on host port 31022 by default. You can change it by overriding `virtualisation.darwin-builder.hostPort`. See the [example](#sec-darwin-builder-example-flake). @@ -15,7 +15,7 @@ words, your `/etc/nix/nix.conf` should have something like: extra-trusted-users = <your username goes here> ``` -To launch the builder, run the following flake: +To launch the remote builder, run the following flake: ```ShellSession $ nix run nixpkgs#darwin.linux-builder @@ -57,7 +57,7 @@ builders = ssh-ng://builder@linux-builder ${ARCH}-linux /etc/nix/builder_ed25519 builders-use-substitutes = true ``` -To allow Nix to connect to a builder not running on port 22, you will also need to create a new file at `/etc/ssh/ssh_config.d/100-linux-builder.conf`: +To allow Nix to connect to a remote builder not running on port 22, you will also need to create a new file at `/etc/ssh/ssh_config.d/100-linux-builder.conf`: ``` Host linux-builder @@ -130,11 +130,11 @@ $ sudo launchctl kickstart -k system/org.nixos.nix-daemon } ``` -## Reconfiguring the builder {#sec-darwin-builder-reconfiguring} +## Reconfiguring the remote builder {#sec-darwin-builder-reconfiguring} -Initially you should not change the builder configuration else you will not be -able to use the binary cache. However, after you have the builder running locally -you may use it to build a modified builder with additional storage or memory. +Initially you should not change the remote builder configuration else you will not be +able to use the binary cache. However, after you have the remote builder running locally +you may use it to build a modified remote builder with additional storage or memory. To do this, you just need to set the `virtualisation.darwin-builder.*` parameters as in the example below and rebuild. @@ -157,3 +157,17 @@ in the example below and rebuild. You may make any other changes to your VM in this attribute set. For example, you could enable Docker or X11 forwarding to your Darwin host. +## Troubleshooting the generated configuration {#sec-darwin-builder-troubleshoot} + +The `linux-builder` package exposes the attributes `nixosConfig` and `nixosOptions` that allow you to inspect the generated NixOS configuration in the `nix repl`. For example: + +``` +$ nix repl --file ~/src/nixpkgs --argstr system aarch64-darwin + +nix-repl> darwin.linux-builder.nixosConfig.nix.package +«derivation /nix/store/...-nix-2.17.0.drv» + +nix-repl> :p darwin.linux-builder.nixosOptions.virtualisation.memorySize.definitionsWithLocations +[ { file = "/home/user/src/nixpkgs/nixos/modules/profiles/macos-builder.nix"; value = 3072; } ] + +``` diff --git a/doc/builders/packages/dlib.section.md b/doc/packages/dlib.section.md index bd5b1a20a4d46..bd5b1a20a4d46 100644 --- a/doc/builders/packages/dlib.section.md +++ b/doc/packages/dlib.section.md diff --git a/doc/builders/packages/eclipse.section.md b/doc/packages/eclipse.section.md index e19510e131a09..e19510e131a09 100644 --- a/doc/builders/packages/eclipse.section.md +++ b/doc/packages/eclipse.section.md diff --git a/doc/builders/packages/elm.section.md b/doc/packages/elm.section.md index 063dd73d9de43..063dd73d9de43 100644 --- a/doc/builders/packages/elm.section.md +++ b/doc/packages/elm.section.md diff --git a/doc/builders/packages/emacs.section.md b/doc/packages/emacs.section.md index c50c7815537dc..c50c7815537dc 100644 --- a/doc/builders/packages/emacs.section.md +++ b/doc/packages/emacs.section.md diff --git a/doc/builders/packages/etc-files.section.md b/doc/packages/etc-files.section.md index 94a769ed33555..94a769ed33555 100644 --- a/doc/builders/packages/etc-files.section.md +++ b/doc/packages/etc-files.section.md diff --git a/doc/builders/packages/firefox.section.md b/doc/packages/firefox.section.md index 46bc0457a3dc2..46bc0457a3dc2 100644 --- a/doc/builders/packages/firefox.section.md +++ b/doc/packages/firefox.section.md diff --git a/doc/builders/packages/fish.section.md b/doc/packages/fish.section.md index 85b57acd1090f..85b57acd1090f 100644 --- a/doc/builders/packages/fish.section.md +++ b/doc/packages/fish.section.md diff --git a/doc/builders/packages/fuse.section.md b/doc/packages/fuse.section.md index 6deea6b5626ed..6deea6b5626ed 100644 --- a/doc/builders/packages/fuse.section.md +++ b/doc/packages/fuse.section.md diff --git a/doc/builders/packages/ibus.section.md b/doc/packages/ibus.section.md index 817e55d56f1f9..817e55d56f1f9 100644 --- a/doc/builders/packages/ibus.section.md +++ b/doc/packages/ibus.section.md diff --git a/doc/builders/packages/index.md b/doc/packages/index.md index 1f44357024064..1f45018ffc4a0 100644 --- a/doc/builders/packages/index.md +++ b/doc/packages/index.md @@ -4,6 +4,7 @@ This chapter contains information about how to use and maintain the Nix expressi ```{=include=} sections citrix.section.md +darwin-builder.section.md dlib.section.md eclipse.section.md elm.section.md diff --git a/doc/builders/packages/kakoune.section.md b/doc/packages/kakoune.section.md index 8e054777a7578..8e054777a7578 100644 --- a/doc/builders/packages/kakoune.section.md +++ b/doc/packages/kakoune.section.md diff --git a/doc/builders/packages/linux.section.md b/doc/packages/linux.section.md index b64da85791a0d..b64da85791a0d 100644 --- a/doc/builders/packages/linux.section.md +++ b/doc/packages/linux.section.md diff --git a/doc/builders/packages/locales.section.md b/doc/packages/locales.section.md index 3a983f13a396e..3a983f13a396e 100644 --- a/doc/builders/packages/locales.section.md +++ b/doc/packages/locales.section.md diff --git a/doc/builders/packages/nginx.section.md b/doc/packages/nginx.section.md index 0704b534e5f72..0704b534e5f72 100644 --- a/doc/builders/packages/nginx.section.md +++ b/doc/packages/nginx.section.md diff --git a/doc/builders/packages/opengl.section.md b/doc/packages/opengl.section.md index f4d282267a079..f4d282267a079 100644 --- a/doc/builders/packages/opengl.section.md +++ b/doc/packages/opengl.section.md diff --git a/doc/builders/packages/shell-helpers.section.md b/doc/packages/shell-helpers.section.md index e7c2b0abebfca..e7c2b0abebfca 100644 --- a/doc/builders/packages/shell-helpers.section.md +++ b/doc/packages/shell-helpers.section.md diff --git a/doc/builders/packages/steam.section.md b/doc/packages/steam.section.md index 25728aa52aef0..25728aa52aef0 100644 --- a/doc/builders/packages/steam.section.md +++ b/doc/packages/steam.section.md diff --git a/doc/builders/packages/urxvt.section.md b/doc/packages/urxvt.section.md index 507feaa6fd861..7aff0997dd2b4 100644 --- a/doc/builders/packages/urxvt.section.md +++ b/doc/packages/urxvt.section.md @@ -34,7 +34,7 @@ $ nix repl map (p: p.name) pkgs.rxvt-unicode.plugins ``` -Alternatively, if your shell is bash or zsh and have completion enabled, simply type `nixpkgs.rxvt-unicode.plugins.<tab>`. +Alternatively, if your shell is bash or zsh and have completion enabled, type `nixpkgs.rxvt-unicode.plugins.<tab>`. In addition to `plugins` the options `extraDeps` and `perlDeps` can be used to install extra packages. `extraDeps` can be used, for example, to provide `xsel` (a clipboard manager) to the clipboard plugin, without installing it globally: diff --git a/doc/builders/packages/weechat.section.md b/doc/packages/weechat.section.md index 755b6e6ad1ea4..755b6e6ad1ea4 100644 --- a/doc/builders/packages/weechat.section.md +++ b/doc/packages/weechat.section.md diff --git a/doc/builders/packages/xorg.section.md b/doc/packages/xorg.section.md index ae885f9234677..ae885f9234677 100644 --- a/doc/builders/packages/xorg.section.md +++ b/doc/packages/xorg.section.md diff --git a/doc/stdenv/multiple-output.chapter.md b/doc/stdenv/multiple-output.chapter.md index c19d497ab61e2..1ee063c0c2f49 100644 --- a/doc/stdenv/multiple-output.chapter.md +++ b/doc/stdenv/multiple-output.chapter.md @@ -1,7 +1,5 @@ # Multiple-output packages {#chap-multiple-output} -## Introduction {#sec-multiple-outputs-introduction} - The Nix language allows a derivation to produce multiple outputs, which is similar to what is utilized by other Linux distribution packaging systems. The outputs reside in separate Nix store paths, so they can be mostly handled independently of each other, including passing to build inputs, garbage collection or binary substitution. The exception is that building from source always produces all the outputs. The main motivation is to save disk space by reducing runtime closure sizes; consequently also sizes of substituted binaries get reduced. Splitting can be used to have more granular runtime dependencies, for example the typical reduction is to split away development-only files, as those are typically not needed during runtime. As a result, closure sizes of many packages can get reduced to a half or even much less. @@ -10,44 +8,12 @@ The main motivation is to save disk space by reducing runtime closure sizes; con The reduction effects could be instead achieved by building the parts in completely separate derivations. That would often additionally reduce build-time closures, but it tends to be much harder to write such derivations, as build systems typically assume all parts are being built at once. This compromise approach of single source package producing multiple binary packages is also utilized often by rpm and deb. ::: -A number of attributes can be used to work with a derivation with multiple outputs. The attribute `outputs` is a list of strings, which are the names of the outputs. For each of these names, an identically named attribute is created, corresponding to that output. The attribute `meta.outputsToInstall` is used to determine the default set of outputs to install when using the derivation name unqualified. - -## Installing a split package {#sec-multiple-outputs-installing} - -When installing a package with multiple outputs, the package’s `meta.outputsToInstall` attribute determines which outputs are actually installed. `meta.outputsToInstall` is a list whose [default installs binaries and the associated man pages](https://github.com/NixOS/nixpkgs/blob/f1680774340d5443a1409c3421ced84ac1163ba9/pkgs/stdenv/generic/make-derivation.nix#L310-L320). The following sections describe ways to install different outputs. - -### Selecting outputs to install via NixOS {#sec-multiple-outputs-installing-nixos} - -NixOS provides two ways to select the outputs to install for packages listed in `environment.systemPackages`: - -- The configuration option `environment.extraOutputsToInstall` is appended to each package’s `meta.outputsToInstall` attribute to determine the outputs to install. It can for example be used to install `info` documentation or debug symbols for all packages. - -- The outputs can be listed as packages in `environment.systemPackages`. For example, the `"out"` and `"info"` outputs for the `coreutils` package can be installed by including `coreutils` and `coreutils.info` in `environment.systemPackages`. - -### Selecting outputs to install via `nix-env` {#sec-multiple-outputs-installing-nix-env} - -`nix-env` lacks an easy way to select the outputs to install. When installing a package, `nix-env` always installs the outputs listed in `meta.outputsToInstall`, even when the user explicitly selects an output. +A number of attributes can be used to work with a derivation with multiple outputs. +The attribute `outputs` is a list of strings, which are the names of the outputs. +For each of these names, an identically named attribute is created, corresponding to that output. -::: {.warning} -`nix-env` silently disregards the outputs selected by the user, and instead installs the outputs from `meta.outputsToInstall`. For example, - -```ShellSession -$ nix-env -iA nixpkgs.coreutils.info -``` - -installs the `"out"` output (`coreutils.meta.outputsToInstall` is `[ "out" ]`) instead of the requested `"info"`. -::: - -The only recourse to select an output with `nix-env` is to override the package’s `meta.outputsToInstall`, using the functions described in [](#chap-overrides). For example, the following overlay adds the `"info"` output for the `coreutils` package: - -```nix -self: super: -{ - coreutils = super.coreutils.overrideAttrs (oldAttrs: { - meta = oldAttrs.meta // { outputsToInstall = oldAttrs.meta.outputsToInstall or [ "out" ] ++ [ "info" ]; }; - }); -} -``` +The attribute `meta.outputsToInstall` is used to determine the [default set of outputs to install](https://github.com/NixOS/nixpkgs/blob/08c3198f1c6fd89a09f8f0ea09b425028a34de3e/pkgs/stdenv/generic/check-meta.nix#L411-L426) when using the derivation name unqualified: +`bin`, or `out`, or the first specified output; as well as `man` if that is specified. ## Using a split package {#sec-multiple-outputs-using-split-packages} diff --git a/doc/stdenv/stdenv.chapter.md b/doc/stdenv/stdenv.chapter.md index 15cb081a04e22..26c43bd9e943c 100644 --- a/doc/stdenv/stdenv.chapter.md +++ b/doc/stdenv/stdenv.chapter.md @@ -101,25 +101,62 @@ genericBuild ### Building a `stdenv` package in `nix-shell` {#sec-building-stdenv-package-in-nix-shell} -To build a `stdenv` package in a [`nix-shell`](https://nixos.org/manual/nix/unstable/command-ref/nix-shell.html), use +To build a `stdenv` package in a [`nix-shell`](https://nixos.org/manual/nix/unstable/command-ref/nix-shell.html), enter a shell, find the [phases](#sec-stdenv-phases) you wish to build, then invoke `genericBuild` manually: + +Go to an empty directory, invoke `nix-shell` with the desired package, and from inside the shell, set the output variables to a writable directory: ```bash +cd "$(mktemp -d)" nix-shell '<nixpkgs>' -A some_package -eval "${unpackPhase:-unpackPhase}" -cd $sourceRoot -eval "${patchPhase:-patchPhase}" -eval "${configurePhase:-configurePhase}" -eval "${buildPhase:-buildPhase}" +export out=$(pwd)/out +``` + +Next, invoke the desired parts of the build. +First, run the phases that generate a working copy of the sources, which will change directory to the sources for you: + +```bash +phases="${prePhases[*]:-} unpackPhase patchPhase" genericBuild +``` + +Then, run more phases up until the failure is reached. +For example, if the failure is in the build phase, the following phases would be required: + +```bash +phases="${preConfigurePhases[*]:-} configurePhase ${preBuildPhases[*]:-} buildPhase" genericBuild +``` + +Re-run a single phase as many times as necessary to examine the failure like so: + +```bash +phases="buildPhase" genericBuild ``` To modify a [phase](#sec-stdenv-phases), first print it with ```bash +echo "$buildPhase" +``` + +Or, if that is empty, for instance, if it is using a function: + +```bash type buildPhase ``` then change it in a text editor, and paste it back to the terminal. +::: {.note} +This method may have some inconsistencies in environment variables and behaviour compared to a normal build within the [Nix build sandbox](https://nixos.org/manual/nix/unstable/language/derivations#builder-execution). +The following is a non-exhaustive list of such differences: + +- `TMP`, `TMPDIR`, and similar variables likely point to non-empty directories that the build might conflict with files in. +- Output store paths are not writable, so the variables for outputs need to be overridden to writable paths. +- Other environment variables may be inconsistent with a `nix-build` either due to `nix-shell`'s initialization script or due to the use of `nix-shell` without the `--pure` option. + +If the build fails differently inside the shell than in the sandbox, consider using [`breakpointHook`](#breakpointhook) and invoking `nix-build` instead. +The [`--keep-failed`](https://nixos.org/manual/nix/unstable/command-ref/conf-file#opt--keep-failed) option for `nix-build` may also be useful to examine the build directory of a failed build. +::: + ## Tools provided by `stdenv` {#sec-tools-of-stdenv} The standard environment provides the following packages: @@ -282,7 +319,7 @@ let f(h, h + 1, i) = i + (if i <= 0 then h else h) let f(h, h + 1, i) = i + h ``` -This is where “sum-like” comes in from above: We can just sum all of the host offsets to get the host offset of the transitive dependency. The target offset is the transitive dependency is simply the host offset + 1, just as it was with the dependencies composed to make this transitive one; it can be ignored as it doesn’t add any new information. +This is where “sum-like” comes in from above: We can just sum all of the host offsets to get the host offset of the transitive dependency. The target offset is the transitive dependency is the host offset + 1, just as it was with the dependencies composed to make this transitive one; it can be ignored as it doesn’t add any new information. Because of the bounds checks, the uncommon cases are `h = t` and `h + 2 = t`. In the former case, the motivation for `mapOffset` is that since its host and target platforms are the same, no transitive dependency of it should be able to “discover” an offset greater than its reduced target offsets. `mapOffset` effectively “squashes” all its transitive dependencies’ offsets so that none will ever be greater than the target offset of the original `h = t` package. In the other case, `h + 1` is skipped over between the host and target offsets. Instead of squashing the offsets, we need to “rip” them apart so no transitive dependencies’ offset is that one. @@ -491,7 +528,7 @@ If the returned array contains exactly one object (e.g. `[{}]`), all values are ``` ::: -### Recursive attributes in `mkDerivation` {#mkderivation-recursive-attributes} +### Fixed-point arguments of `mkDerivation` {#mkderivation-recursive-attributes} If you pass a function to `mkDerivation`, it will receive as its argument the final arguments, including the overrides when reinvoked via `overrideAttrs`. For example: @@ -612,7 +649,7 @@ Zip files are unpacked using `unzip`. However, `unzip` is not in the standard en #### Directories in the Nix store {#directories-in-the-nix-store} -These are simply copied to the current directory. The hash part of the file name is stripped, e.g. `/nix/store/1wydxgby13cz...-my-sources` would be copied to `my-sources`. +These are copied to the current directory. The hash part of the file name is stripped, e.g. `/nix/store/1wydxgby13cz...-my-sources` would be copied to `my-sources`. Additional file types can be supported by setting the `unpackCmd` variable (see below). @@ -751,7 +788,7 @@ Hook executed at the end of the configure phase. ### The build phase {#build-phase} -The build phase is responsible for actually building the package (e.g. compiling it). The default `buildPhase` simply calls `make` if a file named `Makefile`, `makefile` or `GNUmakefile` exists in the current directory (or the `makefile` is explicitly set); otherwise it does nothing. +The build phase is responsible for actually building the package (e.g. compiling it). The default `buildPhase` calls `make` if a file named `Makefile`, `makefile` or `GNUmakefile` exists in the current directory (or the `makefile` is explicitly set); otherwise it does nothing. #### Variables controlling the build phase {#variables-controlling-the-build-phase} @@ -991,13 +1028,56 @@ Hook executed at the end of the fixup phase. If set to `true`, the standard environment will enable debug information in C/C++ builds. After installation, the debug information will be separated from the executables and stored in the output named `debug`. (This output is enabled automatically; you don’t need to set the `outputs` attribute explicitly.) To be precise, the debug information is stored in `debug/lib/debug/.build-id/XX/YYYY…`, where \<XXYYYY…\> is the \<build ID\> of the binary — a SHA-1 hash of the contents of the binary. Debuggers like GDB use the build ID to look up the separated debug information. -For example, with GDB, you can add +:::{.example #ex-gdb-debug-symbols-socat} -``` -set debug-file-directory ~/.nix-profile/lib/debug +# Enable debug symbols for use with GDB + +To make GDB find debug information for the `socat` package and its dependencies, you can use the following `shell.nix`: + +```nix +let + pkgs = import ./. { + config = {}; + overlays = [ + (final: prev: { + ncurses = prev.ncurses.overrideAttrs { separateDebugInfo = true; }; + readline = prev.readline.overrideAttrs { separateDebugInfo = true; }; + }) + ]; + }; + + myDebugInfoDirs = pkgs.symlinkJoin { + name = "myDebugInfoDirs"; + paths = with pkgs; [ + glibc.debug + ncurses.debug + openssl.debug + readline.debug + ]; + }; +in + pkgs.mkShell { + + NIX_DEBUG_INFO_DIRS = "${pkgs.lib.getLib myDebugInfoDirs}/lib/debug"; + + packages = [ + pkgs.gdb + pkgs.socat + ]; + + shellHook = '' + ${pkgs.lib.getBin pkgs.gdb}/bin/gdb ${pkgs.lib.getBin pkgs.socat}/bin/socat + ''; + } ``` -to `~/.gdbinit`. GDB will then be able to find debug information installed via `nix-env -i`. +This setup works as follows: +- Add [`overlays`](#chap-overlays) to the package set, since debug symbols are disabled for `ncurses` and `readline` by default. +- Create a derivation to combine all required debug symbols under one path with [`symlinkJoin`](#trivial-builder-symlinkJoin). +- Set the environment variable `NIX_DEBUG_INFO_DIRS` in the shell. Nixpkgs patches `gdb` to use it for looking up debug symbols. +- Run `gdb` on the `socat` binary on shell startup in the [`shellHook`](#sec-pkgs-mkShell). Here we use [`lib.getBin`](#function-library-lib.attrsets.getBin) to ensure that the correct derivation output is selected rather than the default one. + +::: ### The installCheck phase {#ssec-installCheck-phase} @@ -1237,7 +1317,7 @@ Nix itself considers a build-time dependency as merely something that should pre In order to alleviate this burden, the setup hook mechanism was written, where any package can include a shell script that \[by convention rather than enforcement by Nix\], any downstream reverse-dependency will source as part of its build process. That allows the downstream dependency to merely specify its dependencies, and lets those dependencies effectively initialize themselves. No boilerplate mirroring the list of dependencies is needed. -The setup hook mechanism is a bit of a sledgehammer though: a powerful feature with a broad and indiscriminate area of effect. The combination of its power and implicit use may be expedient, but isn’t without costs. Nix itself is unchanged, but the spirit of added dependencies being effect-free is violated even if the latter isn’t. For example, if a derivation path is mentioned more than once, Nix itself doesn’t care and simply makes sure the dependency derivation is already built just the same—depending is just needing something to exist, and needing is idempotent. However, a dependency specified twice will have its setup hook run twice, and that could easily change the build environment (though a well-written setup hook will therefore strive to be idempotent so this is in fact not observable). More broadly, setup hooks are anti-modular in that multiple dependencies, whether the same or different, should not interfere and yet their setup hooks may well do so. +The setup hook mechanism is a bit of a sledgehammer though: a powerful feature with a broad and indiscriminate area of effect. The combination of its power and implicit use may be expedient, but isn’t without costs. Nix itself is unchanged, but the spirit of added dependencies being effect-free is violated even if the latter isn’t. For example, if a derivation path is mentioned more than once, Nix itself doesn’t care and makes sure the dependency derivation is already built just the same—depending is just needing something to exist, and needing is idempotent. However, a dependency specified twice will have its setup hook run twice, and that could easily change the build environment (though a well-written setup hook will therefore strive to be idempotent so this is in fact not observable). More broadly, setup hooks are anti-modular in that multiple dependencies, whether the same or different, should not interfere and yet their setup hooks may well do so. The most typical use of the setup hook is actually to add other hooks which are then run (i.e. after all the setup hooks) on each dependency. For example, the C compiler wrapper’s setup hook feeds itself flags for each dependency that contains relevant libraries and headers. This is done by defining a bash function, and appending its name to one of `envBuildBuildHooks`, `envBuildHostHooks`, `envBuildTargetHooks`, `envHostHostHooks`, `envHostTargetHooks`, or `envTargetTargetHooks`. These 6 bash variables correspond to the 6 sorts of dependencies by platform (there’s 12 total but we ignore the propagated/non-propagated axis). @@ -1478,7 +1558,7 @@ This flag can break dynamic shared object loading. For instance, the module syst #### `bindnow` {#bindnow} -Adds the `-z bindnow` linker option. During program load, all dynamic symbols are resolved, allowing for the complete GOT to be marked read-only (due to `relro`). This prevents GOT overwrite attacks. For very large applications, this can incur some performance loss during initial load while symbols are resolved, but this shouldn’t be an issue for daemons. +Adds the `-z now` linker option. During program load, all dynamic symbols are resolved, allowing for the complete GOT to be marked read-only (due to `relro`). This prevents GOT overwrite attacks. For very large applications, this can incur some performance loss during initial load while symbols are resolved, but this shouldn’t be an issue for daemons. This flag can break dynamic shared object loading. For instance, the module systems of Xorg and PHP are incompatible with this flag. Programs incompatible with this flag often fail at runtime due to missing symbols, like: diff --git a/doc/using-nixpkgs.md b/doc/using-nixpkgs.md index bb222ae384fa6..f850b2e83c28d 100644 --- a/doc/using-nixpkgs.md +++ b/doc/using-nixpkgs.md @@ -1,6 +1,7 @@ # Using Nixpkgs {#part-using} ```{=include=} chapters +using/platform-support.chapter.md using/configuration.chapter.md using/overlays.chapter.md using/overrides.chapter.md diff --git a/doc/using/overlays.chapter.md b/doc/using/overlays.chapter.md index 6ee52215a4e18..1e965e5f0c741 100644 --- a/doc/using/overlays.chapter.md +++ b/doc/using/overlays.chapter.md @@ -156,7 +156,7 @@ All programs that are built with [MPI](https://en.wikipedia.org/wiki/Message_Pas - [MVAPICH](https://mvapich.cse.ohio-state.edu/), attribute name `mvapich` -To provide MPI enabled applications that use `MPICH`, instead of the default `Open MPI`, simply use the following overlay: +To provide MPI enabled applications that use `MPICH`, instead of the default `Open MPI`, use the following overlay: ```nix self: super: diff --git a/doc/using/platform-support.chapter.md b/doc/using/platform-support.chapter.md new file mode 100644 index 0000000000000..3f91b3d5d9808 --- /dev/null +++ b/doc/using/platform-support.chapter.md @@ -0,0 +1,18 @@ +# Platform Support {#chap-platform-support} + +Packages receive varying degrees of support, both in terms of maintainer attention and available computation resources for continuous integration (CI). + +Below is the list of the best supported platforms: + +- `x86_64-linux`: Highest level of support. +- `aarch64-linux`: Well supported, with most packages building successfully in CI. +- `aarch64-darwin`: Receives better support than `x86_64-darwin`. +- `x86_64-darwin`: Receives some support. + +There are many other platforms with varying levels of support. +The provisional platform list in [Appendix A] of [RFC046], while not up to date, can be used as guidance. + +A more formal definition of the platform support tiers is provided in [RFC046], but has not been fully implemented yet. + +[RFC046]: https://github.com/NixOS/rfcs/blob/master/rfcs/0046-platform-support-tiers.md +[Appendix A]: https://github.com/NixOS/rfcs/blob/master/rfcs/0046-platform-support-tiers.md#appendix-a-non-normative-description-of-platforms-in-november-2019 |