From b0fce27ce22c1c2b5c1198feee92bec708fa3a1d Mon Sep 17 00:00:00 2001 From: fricklerhandwerk Date: Wed, 28 Apr 2021 13:34:16 +0200 Subject: docs: expand explanation of patchShebangs hook - clarify motivation and mechanism - explain usage - add interlinks - add links to sources to enable research based on https://discourse.nixos.org/t/what-is-the-patchshebangs-command-in-nix-build-expressions/12656 --- doc/stdenv/stdenv.chapter.md | 42 +++++++++++++++++++++++++++++++++++------- 1 file changed, 35 insertions(+), 7 deletions(-) (limited to 'doc') diff --git a/doc/stdenv/stdenv.chapter.md b/doc/stdenv/stdenv.chapter.md index 40f295b178bbd..a73b70a34ac96 100644 --- a/doc/stdenv/stdenv.chapter.md +++ b/doc/stdenv/stdenv.chapter.md @@ -77,7 +77,7 @@ where the builder can do anything it wants, but typically starts with source $stdenv/setup ``` -to let `stdenv` set up the environment (e.g., process the `buildInputs`). If you want, you can still use `stdenv`’s generic builder: +to let `stdenv` set up the environment (e.g. by resetting `PATH` and populating it from `buildInputs`). If you want, you can still use `stdenv`’s generic builder: ```bash source $stdenv/setup @@ -644,12 +644,12 @@ Hook executed at the end of the install phase. ### The fixup phase {#ssec-fixup-phase} -The fixup phase performs some (Nix-specific) post-processing actions on the files installed under `$out` by the install phase. The default `fixupPhase` does the following: +The fixup phase performs (Nix-specific) post-processing actions on the files installed under `$out` by the install phase. The default `fixupPhase` does the following: - It moves the `man/`, `doc/` and `info/` subdirectories of `$out` to `share/`. - It strips libraries and executables of debug information. - On Linux, it applies the `patchelf` command to ELF executables and libraries to remove unused directories from the `RPATH` in order to prevent unnecessary runtime dependencies. -- It rewrites the interpreter paths of shell scripts to paths found in `PATH`. E.g., `/usr/bin/perl` will be rewritten to `/nix/store/some-perl/bin/perl` found in `PATH`. +- It rewrites the interpreter paths of shell scripts to paths found in `PATH`. E.g., `/usr/bin/perl` will be rewritten to `/nix/store/some-perl/bin/perl` found in `PATH`. See [](#patch-shebangs.sh) for details. #### Variables controlling the fixup phase {#variables-controlling-the-fixup-phase} @@ -695,7 +695,7 @@ If set, the `patchelf` command is not used to remove unnecessary `RPATH` entries ##### `dontPatchShebangs` {#var-stdenv-dontPatchShebangs} -If set, scripts starting with `#!` do not have their interpreter paths rewritten to paths in the Nix store. +If set, scripts starting with `#!` do not have their interpreter paths rewritten to paths in the Nix store. See [](#patch-shebangs.sh) on how patching shebangs works. ##### `dontPruneLibtoolFiles` {#var-stdenv-dontPruneLibtoolFiles} @@ -929,7 +929,7 @@ addEnvHooks "$hostOffset" myBashFunction The *existence* of setups hooks has long been documented and packages inside Nixpkgs are free to use this mechanism. Other packages, however, should not rely on these mechanisms not changing between Nixpkgs versions. Because of the existing issues with this system, there’s little benefit from mandating it be stable for any period of time. -First, let’s cover some setup hooks that are part of Nixpkgs default stdenv. This means that they are run for every package built using `stdenv.mkDerivation`. Some of these are platform specific, so they may run on Linux but not Darwin or vice-versa. +First, let’s cover some setup hooks that are part of Nixpkgs default `stdenv`. This means that they are run for every package built using `stdenv.mkDerivation` or when using a custom builder that has `source $stdenv/setup`. Some of these are platform specific, so they may run on Linux but not Darwin or vice-versa. ### `move-docs.sh` {#move-docs.sh} @@ -945,7 +945,35 @@ This runs the strip command on installed binaries and libraries. This removes un ### `patch-shebangs.sh` {#patch-shebangs.sh} -This setup hook patches installed scripts to use the full path to the shebang interpreter. A shebang interpreter is the first commented line of a script telling the operating system which program will run the script (e.g `#!/bin/bash`). In Nix, we want an exact path to that interpreter to be used. This often replaces `/bin/sh` with a path in the Nix store. +This setup hook patches installed scripts to use Nix store paths to their shebang interpreter as found in the build environment's `PATH`. The [shebang](https://en.wikipedia.org/wiki/Shebang_(Unix)) line tells a Unix-like operating system what interpreter to use to execute the script's contents. + +::: note +The [generic builder](https://github.com/NixOS/nixpkgs/blob/6ba632c2a442082f353bf2d7028fda11a888d099/pkgs/stdenv/generic/builder.sh) populates `PATH` from inputs of the derivation. +::: + +`#!/bin/sh` will be rewritten to `#!/nix/store/-some-bash/bin/sh`. +`#!/usr/bin/env` gets special treatment: `#!/usr/bin/env python` is rewritten to `/nix/store//bin/python`. Interpreters that are already in the store are left untouched. + +::: note +A script file must be marked as executable, otherwise it will not be +considered. [Trivial Builders](#chap-trivial-builders) do this automatically where appropriate. +::: + +This mechanism ensures that the interpreter for a given script is always found and is exactly the one specified by the build. + +It can be disabled by setting [`dontPatchShebangs`](#var-stdenv-dontPatchShebangs): + +```nix +stdenv.mkDerivation { + # ... + dontPatchShebangs = true; + # ... +} +``` + +The file [`patch-shebangs.sh`](https://github.com/NixOS/nixpkgs/blob/ca156a66b75999153d746f57a11a19222fd5cdfb/pkgs/build-support/setup-hooks/patch-shebangs.sh) defines the [`patchShebangs`](https://github.com/NixOS/nixpkgs/blob/ca156a66b75999153d746f57a11a19222fd5cdfb/pkgs/build-support/setup-hooks/patch-shebangs.sh#L24) function. It is used to implement [`patchShebangsAuto`](https://github.com/NixOS/nixpkgs/blob/ca156a66b75999153d746f57a11a19222fd5cdfb/pkgs/build-support/setup-hooks/patch-shebangs.sh#L107), the [setup hook](#ssec-setup-hooks) that is registered to run during the [fixup phase](#ssec-fixup-phase) by default. + +If you need to run `patchShebangs` at build time, it must be called explicitly within [one of the build phases](#sec-stdenv-phases). ### `audit-tmpdir.sh` {#audit-tmpdir.sh} @@ -1262,7 +1290,7 @@ If the libraries lack `-fPIE`, you will get the error `recompile with -fPIE`. [^footnote-stdenv-ignored-build-platform]: The build platform is ignored because it is a mere implementation detail of the package satisfying the dependency: As a general programming principle, dependencies are always *specified* as interfaces, not concrete implementation. [^footnote-stdenv-native-dependencies-in-path]: Currently, this means for native builds all dependencies are put on the `PATH`. But in the future that may not be the case for sake of matching cross: the platforms would be assumed to be unique for native and cross builds alike, so only the `depsBuild*` and `nativeBuildInputs` would be added to the `PATH`. -[^footnote-stdenv-propagated-dependencies]: Nix itself already takes a package’s transitive dependencies into account, but this propagation ensures nixpkgs-specific infrastructure like setup hooks (mentioned above) also are run as if the propagated dependency. +[^footnote-stdenv-propagated-dependencies]: Nix itself already takes a package’s transitive dependencies into account, but this propagation ensures nixpkgs-specific infrastructure like [setup hooks](#ssec-setup-hooks) also are run as if it were a propagated dependency. [^footnote-stdenv-find-inputs-location]: The `findInputs` function, currently residing in `pkgs/stdenv/generic/setup.sh`, implements the propagation logic. [^footnote-stdenv-sys-lib-search-path]: It clears the `sys_lib_*search_path` variables in the Libtool script to prevent Libtool from using libraries in `/usr/lib` and such. [^footnote-stdenv-build-time-guessing-impurity]: Eventually these will be passed building natively as well, to improve determinism: build-time guessing, as is done today, is a risk of impurity. -- cgit 1.4.1 From e3883d2ce0e23f07d0ffe90da37d08e65f1e534e Mon Sep 17 00:00:00 2001 From: fricklerhandwerk <6599296+fricklerhandwerk@users.noreply.github.com> Date: Fri, 13 Aug 2021 17:14:21 +0200 Subject: docs: clarify note on existing store paths --- doc/stdenv/stdenv.chapter.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) (limited to 'doc') diff --git a/doc/stdenv/stdenv.chapter.md b/doc/stdenv/stdenv.chapter.md index a73b70a34ac96..23503235a105d 100644 --- a/doc/stdenv/stdenv.chapter.md +++ b/doc/stdenv/stdenv.chapter.md @@ -952,7 +952,7 @@ The [generic builder](https://github.com/NixOS/nixpkgs/blob/6ba632c2a442082f353b ::: `#!/bin/sh` will be rewritten to `#!/nix/store/-some-bash/bin/sh`. -`#!/usr/bin/env` gets special treatment: `#!/usr/bin/env python` is rewritten to `/nix/store//bin/python`. Interpreters that are already in the store are left untouched. +`#!/usr/bin/env` gets special treatment: `#!/usr/bin/env python` is rewritten to `/nix/store//bin/python`. Interpreter paths that point to a valid Nix store location are not changed. ::: note A script file must be marked as executable, otherwise it will not be -- cgit 1.4.1 From b4d9d682c8cdfdfc8919f3140578971dcb64bd25 Mon Sep 17 00:00:00 2001 From: fricklerhandwerk Date: Mon, 21 Mar 2022 11:34:55 +0100 Subject: docs: clean up and update links to source code --- doc/stdenv/stdenv.chapter.md | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) (limited to 'doc') diff --git a/doc/stdenv/stdenv.chapter.md b/doc/stdenv/stdenv.chapter.md index 23503235a105d..3ac58cc7c0439 100644 --- a/doc/stdenv/stdenv.chapter.md +++ b/doc/stdenv/stdenv.chapter.md @@ -948,9 +948,11 @@ This runs the strip command on installed binaries and libraries. This removes un This setup hook patches installed scripts to use Nix store paths to their shebang interpreter as found in the build environment's `PATH`. The [shebang](https://en.wikipedia.org/wiki/Shebang_(Unix)) line tells a Unix-like operating system what interpreter to use to execute the script's contents. ::: note -The [generic builder](https://github.com/NixOS/nixpkgs/blob/6ba632c2a442082f353bf2d7028fda11a888d099/pkgs/stdenv/generic/builder.sh) populates `PATH` from inputs of the derivation. +The [generic builder][generic-builder] populates `PATH` from inputs of the derivation. ::: +[generic-builder]: https://github.com/NixOS/nixpkgs/blob/19d4f7dc485f74109bd66ef74231285ff797a823/pkgs/stdenv/generic/builder.sh + `#!/bin/sh` will be rewritten to `#!/nix/store/-some-bash/bin/sh`. `#!/usr/bin/env` gets special treatment: `#!/usr/bin/env python` is rewritten to `/nix/store//bin/python`. Interpreter paths that point to a valid Nix store location are not changed. @@ -971,10 +973,14 @@ stdenv.mkDerivation { } ``` -The file [`patch-shebangs.sh`](https://github.com/NixOS/nixpkgs/blob/ca156a66b75999153d746f57a11a19222fd5cdfb/pkgs/build-support/setup-hooks/patch-shebangs.sh) defines the [`patchShebangs`](https://github.com/NixOS/nixpkgs/blob/ca156a66b75999153d746f57a11a19222fd5cdfb/pkgs/build-support/setup-hooks/patch-shebangs.sh#L24) function. It is used to implement [`patchShebangsAuto`](https://github.com/NixOS/nixpkgs/blob/ca156a66b75999153d746f57a11a19222fd5cdfb/pkgs/build-support/setup-hooks/patch-shebangs.sh#L107), the [setup hook](#ssec-setup-hooks) that is registered to run during the [fixup phase](#ssec-fixup-phase) by default. +The file [`patch-shebangs.sh`][patch-shebangs.sh] defines the [`patchShebangs`][patchShebangs] function. It is used to implement [`patchShebangsAuto`][patchShebangsAuto], the [setup hook](#ssec-setup-hooks) that is registered to run during the [fixup phase](#ssec-fixup-phase) by default. If you need to run `patchShebangs` at build time, it must be called explicitly within [one of the build phases](#sec-stdenv-phases). +[patch-shebangs.sh]: https://github.com/NixOS/nixpkgs/blob/19d4f7dc485f74109bd66ef74231285ff797a823/pkgs/build-support/setup-hooks/patch-shebangs.sh +[patchShebangs]: https://github.com/NixOS/nixpkgs/blob/19d4f7dc485f74109bd66ef74231285ff797a823/pkgs/build-support/setup-hooks/patch-shebangs.sh#L24-L105 +[patchShebangsAuto]: https://github.com/NixOS/nixpkgs/blob/19d4f7dc485f74109bd66ef74231285ff797a823/pkgs/build-support/setup-hooks/patch-shebangs.sh#L107-L119 + ### `audit-tmpdir.sh` {#audit-tmpdir.sh} This verifies that no references are left from the install binaries to the directory used to build those binaries. This ensures that the binaries do not need things outside the Nix store. This is currently supported in Linux only. -- cgit 1.4.1 From 9a2ed653704baabceb6c5a74603d1814583426be Mon Sep 17 00:00:00 2001 From: Valentin Gagarin Date: Wed, 20 Apr 2022 21:21:31 +0200 Subject: fix wording, remove too much specificity Co-authored-by: Robert Hensing --- doc/stdenv/stdenv.chapter.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) (limited to 'doc') diff --git a/doc/stdenv/stdenv.chapter.md b/doc/stdenv/stdenv.chapter.md index 3ac58cc7c0439..a713bfab03b84 100644 --- a/doc/stdenv/stdenv.chapter.md +++ b/doc/stdenv/stdenv.chapter.md @@ -945,7 +945,7 @@ This runs the strip command on installed binaries and libraries. This removes un ### `patch-shebangs.sh` {#patch-shebangs.sh} -This setup hook patches installed scripts to use Nix store paths to their shebang interpreter as found in the build environment's `PATH`. The [shebang](https://en.wikipedia.org/wiki/Shebang_(Unix)) line tells a Unix-like operating system what interpreter to use to execute the script's contents. +This setup hook patches installed scripts to add Nix store paths to their shebang interpreter as found in the build environment. The [shebang](https://en.wikipedia.org/wiki/Shebang_(Unix)) line tells a Unix-like operating system what interpreter to use to execute the script's contents. ::: note The [generic builder][generic-builder] populates `PATH` from inputs of the derivation. -- cgit 1.4.1 From 311d322febdc3e931b0a22b018121ad055dca0ce Mon Sep 17 00:00:00 2001 From: Valentin Gagarin Date: Wed, 20 Apr 2022 21:49:58 +0200 Subject: docs: sync `patchShebangs` comments with manual this is not an actual sync, but rather the manual taking the leading role. right now it does not make sense to actually change `patch-shebangs.sh` as that would cause a rebuild of the entire universe. we should figure out how to keep them aligned with minimal effort both in terms of maintenance as well as navigation for readers. --- doc/stdenv/stdenv.chapter.md | 33 +++++++++++++++++++++++++++++++-- 1 file changed, 31 insertions(+), 2 deletions(-) (limited to 'doc') diff --git a/doc/stdenv/stdenv.chapter.md b/doc/stdenv/stdenv.chapter.md index a713bfab03b84..3539dd2b18468 100644 --- a/doc/stdenv/stdenv.chapter.md +++ b/doc/stdenv/stdenv.chapter.md @@ -945,7 +945,7 @@ This runs the strip command on installed binaries and libraries. This removes un ### `patch-shebangs.sh` {#patch-shebangs.sh} -This setup hook patches installed scripts to add Nix store paths to their shebang interpreter as found in the build environment. The [shebang](https://en.wikipedia.org/wiki/Shebang_(Unix)) line tells a Unix-like operating system what interpreter to use to execute the script's contents. +This setup hook patches installed scripts to add Nix store paths to their shebang interpreter as found in the build environment. The [shebang](https://en.wikipedia.org/wiki/Shebang_(Unix)) line tells a Unix-like operating system which interpreter to use to execute the script's contents. ::: note The [generic builder][generic-builder] populates `PATH` from inputs of the derivation. @@ -953,8 +953,37 @@ The [generic builder][generic-builder] populates `PATH` from inputs of the deriv [generic-builder]: https://github.com/NixOS/nixpkgs/blob/19d4f7dc485f74109bd66ef74231285ff797a823/pkgs/stdenv/generic/builder.sh +#### Invocation + +Multiple paths can be specified. + +``` +patchShebangs [--build | --host] PATH... +``` + +#### Flags + +`--build` +: Look up commands available at build time + +`--host` +: Look up commands available at run time + +#### Examples + +```sh +patchShebangs --host /nix/store/-hello-1.0/bin +``` + +```sh +patchShebangs --build configure +``` + `#!/bin/sh` will be rewritten to `#!/nix/store/-some-bash/bin/sh`. -`#!/usr/bin/env` gets special treatment: `#!/usr/bin/env python` is rewritten to `/nix/store//bin/python`. Interpreter paths that point to a valid Nix store location are not changed. + +`#!/usr/bin/env` gets special treatment: `#!/usr/bin/env python` is rewritten to `/nix/store//bin/python`. + +Interpreter paths that point to a valid Nix store location are not changed. ::: note A script file must be marked as executable, otherwise it will not be -- cgit 1.4.1 From f70073d72d44717d18200595873a09f7f7ccfb98 Mon Sep 17 00:00:00 2001 From: Valentin Gagarin Date: Wed, 8 Jun 2022 11:43:32 +0200 Subject: remove specifics on where build inputs come from in `PATH` Co-authored-by: Robert Hensing --- doc/stdenv/stdenv.chapter.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) (limited to 'doc') diff --git a/doc/stdenv/stdenv.chapter.md b/doc/stdenv/stdenv.chapter.md index 3539dd2b18468..e860c39a6e0a4 100644 --- a/doc/stdenv/stdenv.chapter.md +++ b/doc/stdenv/stdenv.chapter.md @@ -77,7 +77,7 @@ where the builder can do anything it wants, but typically starts with source $stdenv/setup ``` -to let `stdenv` set up the environment (e.g. by resetting `PATH` and populating it from `buildInputs`). If you want, you can still use `stdenv`’s generic builder: +to let `stdenv` set up the environment (e.g. by resetting `PATH` and populating it from build inputs). If you want, you can still use `stdenv`’s generic builder: ```bash source $stdenv/setup -- cgit 1.4.1 From e132e6be3c7ea5d05dd8e817b6165c3b11eb7e7b Mon Sep 17 00:00:00 2001 From: Valentin Gagarin Date: Thu, 9 Jun 2022 13:43:21 +0200 Subject: fix heading level --- doc/stdenv/stdenv.chapter.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) (limited to 'doc') diff --git a/doc/stdenv/stdenv.chapter.md b/doc/stdenv/stdenv.chapter.md index e860c39a6e0a4..dbe0a589e4a44 100644 --- a/doc/stdenv/stdenv.chapter.md +++ b/doc/stdenv/stdenv.chapter.md @@ -961,7 +961,7 @@ Multiple paths can be specified. patchShebangs [--build | --host] PATH... ``` -#### Flags +##### Flags `--build` : Look up commands available at build time @@ -969,7 +969,7 @@ patchShebangs [--build | --host] PATH... `--host` : Look up commands available at run time -#### Examples +##### Examples ```sh patchShebangs --host /nix/store/-hello-1.0/bin -- cgit 1.4.1 From c3ea8c4dd909d081ff6aae007f1872b367dacc14 Mon Sep 17 00:00:00 2001 From: Valentin Gagarin Date: Fri, 10 Jun 2022 11:43:11 +0200 Subject: do not mention trivial builders --- doc/stdenv/stdenv.chapter.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) (limited to 'doc') diff --git a/doc/stdenv/stdenv.chapter.md b/doc/stdenv/stdenv.chapter.md index dbe0a589e4a44..7ca9ee3ff76db 100644 --- a/doc/stdenv/stdenv.chapter.md +++ b/doc/stdenv/stdenv.chapter.md @@ -987,7 +987,7 @@ Interpreter paths that point to a valid Nix store location are not changed. ::: note A script file must be marked as executable, otherwise it will not be -considered. [Trivial Builders](#chap-trivial-builders) do this automatically where appropriate. +considered. ::: This mechanism ensures that the interpreter for a given script is always found and is exactly the one specified by the build. -- cgit 1.4.1 From b7b86c4f548d8c127902406c7c51d6922c2cff81 Mon Sep 17 00:00:00 2001 From: Valentin Gagarin Date: Wed, 13 Jul 2022 10:19:23 +0100 Subject: add stable anchor Co-authored-by: Jan Tojnar --- doc/stdenv/stdenv.chapter.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) (limited to 'doc') diff --git a/doc/stdenv/stdenv.chapter.md b/doc/stdenv/stdenv.chapter.md index 7ca9ee3ff76db..4597f9d162785 100644 --- a/doc/stdenv/stdenv.chapter.md +++ b/doc/stdenv/stdenv.chapter.md @@ -953,7 +953,7 @@ The [generic builder][generic-builder] populates `PATH` from inputs of the deriv [generic-builder]: https://github.com/NixOS/nixpkgs/blob/19d4f7dc485f74109bd66ef74231285ff797a823/pkgs/stdenv/generic/builder.sh -#### Invocation +#### Invocation {#patch-shebangs.sh-invocation} Multiple paths can be specified. -- cgit 1.4.1