about summary refs log tree commit diff
path: root/pkgs/test
diff options
context:
space:
mode:
Diffstat (limited to 'pkgs/test')
-rw-r--r--pkgs/test/nixpkgs-check-by-name/README.md23
-rw-r--r--pkgs/test/nixpkgs-check-by-name/default.nix17
-rw-r--r--pkgs/test/nixpkgs-check-by-name/scripts/README.md19
-rwxr-xr-xpkgs/test/nixpkgs-check-by-name/scripts/fetch-pinned-tool.sh30
-rw-r--r--pkgs/test/nixpkgs-check-by-name/scripts/pinned-tool.json4
-rwxr-xr-xpkgs/test/nixpkgs-check-by-name/scripts/run-local.sh18
-rwxr-xr-xpkgs/test/nixpkgs-check-by-name/scripts/update-pinned-tool.sh19
-rw-r--r--pkgs/test/nixpkgs-check-by-name/src/eval.rs68
-rw-r--r--pkgs/test/nixpkgs-check-by-name/src/main.rs42
-rw-r--r--pkgs/test/nixpkgs-check-by-name/src/ratchet.rs99
-rw-r--r--pkgs/test/nixpkgs-check-by-name/src/references.rs4
-rw-r--r--pkgs/test/nixpkgs-check-by-name/tests/manual-definition/all-packages.nix10
-rw-r--r--pkgs/test/nixpkgs-check-by-name/tests/manual-definition/base/all-packages.nix9
-rw-r--r--pkgs/test/nixpkgs-check-by-name/tests/manual-definition/base/default.nix (renamed from pkgs/test/nixpkgs-check-by-name/tests/ref-parse-failure/default.nix)0
-rw-r--r--pkgs/test/nixpkgs-check-by-name/tests/manual-definition/base/pkgs/by-name/README.md0
-rw-r--r--pkgs/test/nixpkgs-check-by-name/tests/manual-definition/base/some-pkg.nix (renamed from pkgs/test/nixpkgs-check-by-name/tests/no-eval/pkgs/by-name/fo/foo/package.nix)0
-rw-r--r--pkgs/test/nixpkgs-check-by-name/tests/manual-definition/default.nix1
-rw-r--r--pkgs/test/nixpkgs-check-by-name/tests/manual-definition/expected2
-rw-r--r--pkgs/test/nixpkgs-check-by-name/tests/manual-definition/pkgs/by-name/no/noEval/package.nix (renamed from pkgs/test/nixpkgs-check-by-name/tests/ref-parse-failure/pkgs/by-name/aa/aa/package.nix)0
-rw-r--r--pkgs/test/nixpkgs-check-by-name/tests/manual-definition/pkgs/by-name/on/onlyMove/package.nix1
-rw-r--r--pkgs/test/nixpkgs-check-by-name/tests/multiple-failures/default.nix1
-rw-r--r--pkgs/test/nixpkgs-check-by-name/tests/multiple-failures/expected13
l---------pkgs/test/nixpkgs-check-by-name/tests/multiple-failures/pkgs/by-name/A/fo@/foo1
l---------pkgs/test/nixpkgs-check-by-name/tests/multiple-failures/pkgs/by-name/A/fo@/package.nix1
-rw-r--r--pkgs/test/nixpkgs-check-by-name/tests/multiple-failures/pkgs/by-name/aa0
-rw-r--r--pkgs/test/nixpkgs-check-by-name/tests/multiple-failures/pkgs/by-name/ba/bar0
-rw-r--r--pkgs/test/nixpkgs-check-by-name/tests/multiple-failures/pkgs/by-name/ba/baz/package.nix/default.nix1
-rw-r--r--pkgs/test/nixpkgs-check-by-name/tests/multiple-failures/pkgs/by-name/ba/foo/package.nix6
-rw-r--r--pkgs/test/nixpkgs-check-by-name/tests/multiple-failures/pkgs/by-name/fo/foo/.git-keep0
-rw-r--r--pkgs/test/nixpkgs-check-by-name/tests/multiple-failures/someDrv.nix1
-rw-r--r--pkgs/test/nixpkgs-check-by-name/tests/no-eval/all-packages.nix2
-rw-r--r--pkgs/test/nixpkgs-check-by-name/tests/no-eval/base/all-packages.nix3
-rw-r--r--pkgs/test/nixpkgs-check-by-name/tests/no-eval/base/default.nix1
-rw-r--r--pkgs/test/nixpkgs-check-by-name/tests/no-eval/base/pkgs/by-name/README.md0
-rw-r--r--pkgs/test/nixpkgs-check-by-name/tests/no-eval/pkgs/by-name/README.md0
-rw-r--r--pkgs/test/nixpkgs-check-by-name/tests/override-empty-arg/base/pkgs/by-name/README.md1
-rw-r--r--pkgs/test/nixpkgs-check-by-name/tests/ref-parse-failure/expected1
-rw-r--r--pkgs/test/nixpkgs-check-by-name/tests/ref-parse-failure/pkgs/by-name/aa/aa/invalid.nix1
-rw-r--r--pkgs/test/nixpkgs-check-by-name/tests/symlink-invalid/expected2
l---------pkgs/test/nixpkgs-check-by-name/tests/symlink-invalid/pkgs/by-name/fo/foo/foo1
l---------pkgs/test/nixpkgs-check-by-name/tests/symlink-invalid/pkgs/by-name/fo/foo/foo.nix1
41 files changed, 231 insertions, 172 deletions
diff --git a/pkgs/test/nixpkgs-check-by-name/README.md b/pkgs/test/nixpkgs-check-by-name/README.md
index d779529c7baf3..1aa2569784161 100644
--- a/pkgs/test/nixpkgs-check-by-name/README.md
+++ b/pkgs/test/nixpkgs-check-by-name/README.md
@@ -1,9 +1,13 @@
 # Nixpkgs pkgs/by-name checker
 
 This directory implements a program to check the [validity](#validity-checks) of the `pkgs/by-name` Nixpkgs directory.
-It is being used by [this GitHub Actions workflow](../../../.github/workflows/check-by-name.yml).
 This is part of the implementation of [RFC 140](https://github.com/NixOS/rfcs/pull/140).
 
+A [pinned version](./scripts/pinned-tool.json) of this tool is used by [this GitHub Actions workflow](../../../.github/workflows/check-by-name.yml).
+See [./scripts](./scripts/README.md#update-pinned-toolsh) for how to update the pinned version.
+
+The source of the tool being right inside Nixpkgs allows any Nixpkgs committer to make updates to it.
+
 ## Interface
 
 The interface of the tool is shown with `--help`:
@@ -69,7 +73,7 @@ Tests are declared in [`./tests`](./tests) as subdirectories imitating Nixpkgs w
 - `default.nix`:
   Always contains
   ```nix
-  import ../mock-nixpkgs.nix { root = ./.; }
+  import <test-nixpkgs> { root = ./.; }
   ```
   which makes
   ```
@@ -96,18 +100,3 @@ Tests are declared in [`./tests`](./tests) as subdirectories imitating Nixpkgs w
 - `expected` (optional):
   A file containing the expected standard output.
   The default is expecting an empty standard output.
-
-## Hydra builds
-
-This program will always be available pre-built for `x86_64-linux` on the `nixos-unstable` channel and `nixos-XX.YY` channels.
-This is ensured by including it in the `tested` jobset description in [`nixos/release-combined.nix`](../../../nixos/release-combined.nix).
-
-This allows CI for PRs to development branches `master` and `release-XX.YY` to fetch the pre-built program from the corresponding channel and use that to check the PR. This has the following benefits:
-- It allows CI to check all PRs, even if they would break the CI tooling.
-- It makes the CI check very fast, since no Nix builds need to be done, even for mass rebuilds.
-- It improves security, since we don't have to build potentially untrusted code from PRs.
-  The tool only needs a very minimal Nix evaluation at runtime, which can work with [readonly-mode](https://nixos.org/manual/nix/stable/command-ref/opt-common.html#opt-readonly-mode) and [restrict-eval](https://nixos.org/manual/nix/stable/command-ref/conf-file.html#conf-restrict-eval).
-- It allows anybody to make updates to the tooling and for those updates to be automatically used by CI without needing a separate release mechanism.
-
-The tradeoff is that there's a delay between updates to the tool and those updates being used by CI.
-This needs to be considered when updating the [API](#api).
diff --git a/pkgs/test/nixpkgs-check-by-name/default.nix b/pkgs/test/nixpkgs-check-by-name/default.nix
index f2a6473ad202e..8836da1f403f1 100644
--- a/pkgs/test/nixpkgs-check-by-name/default.nix
+++ b/pkgs/test/nixpkgs-check-by-name/default.nix
@@ -11,6 +11,7 @@
 let
   runtimeExprPath = ./src/eval.nix;
   nixpkgsLibPath = ../../../lib;
+  testNixpkgsPath = ./tests/mock-nixpkgs.nix;
 
   # Needed to make Nix evaluation work inside nix builds
   initNix = ''
@@ -26,10 +27,20 @@ let
     nix-store --init
   '';
 
+  fs = lib.fileset;
+
   package =
     rustPlatform.buildRustPackage {
       name = "nixpkgs-check-by-name";
-      src = lib.cleanSource ./.;
+      src = fs.toSource {
+        root = ./.;
+        fileset = fs.unions [
+          ./Cargo.lock
+          ./Cargo.toml
+          ./src
+          ./tests
+        ];
+      };
       cargoLock.lockFile = ./Cargo.lock;
       nativeBuildInputs = [
         nix
@@ -38,7 +49,7 @@ let
         makeWrapper
       ];
       env.NIX_CHECK_BY_NAME_EXPR_PATH = "${runtimeExprPath}";
-      env.NIXPKGS_LIB_PATH = "${nixpkgsLibPath}";
+      env.NIX_PATH = "test-nixpkgs=${testNixpkgsPath}:test-nixpkgs/lib=${nixpkgsLibPath}";
       preCheck = initNix;
       postCheck = ''
         cargo fmt --check
@@ -50,7 +61,7 @@ let
       '';
       passthru.shell = mkShell {
         env.NIX_CHECK_BY_NAME_EXPR_PATH = toString runtimeExprPath;
-        env.NIXPKGS_LIB_PATH = toString nixpkgsLibPath;
+        env.NIX_PATH = "test-nixpkgs=${toString testNixpkgsPath}:test-nixpkgs/lib=${toString nixpkgsLibPath}";
         inputsFrom = [ package ];
       };
 
diff --git a/pkgs/test/nixpkgs-check-by-name/scripts/README.md b/pkgs/test/nixpkgs-check-by-name/scripts/README.md
index cb520f4728d38..ccd4108ea288d 100644
--- a/pkgs/test/nixpkgs-check-by-name/scripts/README.md
+++ b/pkgs/test/nixpkgs-check-by-name/scripts/README.md
@@ -19,15 +19,20 @@ Arguments:
 ## `./update-pinned-tool.sh`
 
 Updates the pinned CI tool in [`./pinned-tool.json`](./pinned-tool.json) to the
-[latest version from the `nixos-unstable` channel](https://hydra.nixos.org/job/nixos/trunk-combined/nixpkgs.tests.nixpkgs-check-by-name.x86_64-linux)
+[latest version from the `nixos-unstable` channel](https://hydra.nixos.org/job/nixos/trunk-combined/nixpkgs.tests.nixpkgs-check-by-name.x86_64-linux).
 
-This script is called manually once the CI tooling needs to be updated.
+This script needs to be called manually when the CI tooling needs to be updated.
 
-## `./fetch-pinned-tool.sh OUTPUT_PATH`
+The `pinned-tool.json` file gets populated with both:
+- The `/nix/store` path for `x86_64-linux`, such that CI doesn't have to evaluate Nixpkgs and can directly fetch it from the cache instead.
+- The Nixpkgs revision, such that the `./run-local.sh` script can be used to run the checks locally on any system.
 
-Fetches the pinned tooling specified in [`./pinned-tool.json`](./pinned-tool.json).
+To ensure that the tool is always pre-built for `x86_64-linux` in the `nixos-unstable` channel,
+it's included in the `tested` jobset description in [`nixos/release-combined.nix`](../../../nixos/release-combined.nix).
 
-This script is used both by [`./run-local.sh`](#run-local-sh-base-branch-repository) and CI.
+Why not just build the tooling right from the PRs Nixpkgs version?
+- Because it allows CI to check all PRs, even if they would break the CI tooling.
+- Because it makes the CI check very fast, since no Nix builds need to be done, even for mass rebuilds.
+- Because it improves security, since we don't have to build potentially untrusted code from PRs.
+  The tool only needs a very minimal Nix evaluation at runtime, which can work with [readonly-mode](https://nixos.org/manual/nix/stable/command-ref/opt-common.html#opt-readonly-mode) and [restrict-eval](https://nixos.org/manual/nix/stable/command-ref/conf-file.html#conf-restrict-eval).
 
-Arguments:
-- `OUTPUT_PATH`: The output symlink path for the tool
diff --git a/pkgs/test/nixpkgs-check-by-name/scripts/fetch-pinned-tool.sh b/pkgs/test/nixpkgs-check-by-name/scripts/fetch-pinned-tool.sh
deleted file mode 100755
index 2e52275cab874..0000000000000
--- a/pkgs/test/nixpkgs-check-by-name/scripts/fetch-pinned-tool.sh
+++ /dev/null
@@ -1,30 +0,0 @@
-#!/usr/bin/env bash
-# Try to not use nix-shell here to avoid fetching Nixpkgs,
-# especially since this is used in CI
-# The only dependency is `jq`, which in CI is implicitly available
-# And when run from ./run-local.sh is provided by that parent script
-
-set -o pipefail -o errexit -o nounset
-
-trace() { echo >&2 "$@"; }
-
-SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
-
-pin_file=$SCRIPT_DIR/pinned-tool.json
-
-if (( $# < 1 )); then
-    trace "Usage: $0 fetch OUTPUT_PATH"
-    trace "OUTPUT_PATH: The output symlink path for the tool"
-    exit 1
-fi
-output=$1
-
-trace "Reading $pin_file.. "
-rev=$(jq -r .rev "$SCRIPT_DIR"/pinned-tool.json)
-trace -e "Git revision is \e[34m$rev\e[0m"
-path=$(jq -r .path "$SCRIPT_DIR"/pinned-tool.json)
-trace "Tooling path is $path"
-
-trace -n "Fetching the prebuilt version of nixpkgs-check-by-name.. "
-nix-store --add-root "$output" -r "$path" >/dev/null
-realpath "$output"
diff --git a/pkgs/test/nixpkgs-check-by-name/scripts/pinned-tool.json b/pkgs/test/nixpkgs-check-by-name/scripts/pinned-tool.json
index bbcbc57e86b28..c4dff2c6554a5 100644
--- a/pkgs/test/nixpkgs-check-by-name/scripts/pinned-tool.json
+++ b/pkgs/test/nixpkgs-check-by-name/scripts/pinned-tool.json
@@ -1,4 +1,4 @@
 {
-  "rev": "9b19f5e77dd906cb52dade0b7bd280339d2a1f3d",
-  "path": "/nix/store/qlls5ca8q88qpyygg9ddi60gl1nmvpij-nixpkgs-check-by-name"
+  "rev": "842d9d80cfd4560648c785f8a4e6f3b096790e19",
+  "ci-path": "/nix/store/8habk3j25bs2a34zn5q5p17b9dl3fywg-nixpkgs-check-by-name"
 }
diff --git a/pkgs/test/nixpkgs-check-by-name/scripts/run-local.sh b/pkgs/test/nixpkgs-check-by-name/scripts/run-local.sh
index b464515b37f6e..1dd52293cc7e6 100755
--- a/pkgs/test/nixpkgs-check-by-name/scripts/run-local.sh
+++ b/pkgs/test/nixpkgs-check-by-name/scripts/run-local.sh
@@ -1,5 +1,5 @@
 #!/usr/bin/env nix-shell
-#!nix-shell -i bash -p jq
+#!nix-shell -i bash -p jq -I nixpkgs=../../../..
 
 set -o pipefail -o errexit -o nounset
 
@@ -14,6 +14,7 @@ cleanup() {
 
     [[ -e "$tmp/base" ]] && git worktree remove --force "$tmp/base"
     [[ -e "$tmp/merged" ]] && git worktree remove --force "$tmp/merged"
+    [[ -e "$tmp/tool-nixpkgs" ]] && git worktree remove --force "$tmp/tool-nixpkgs"
 
     rm -rf "$tmp"
 
@@ -62,7 +63,20 @@ trace -n "Merging base branch into the HEAD commit in $tmp/merged.. "
 git -C "$tmp/merged" merge -q --no-edit "$baseSha"
 trace -e "\e[34m$(git -C "$tmp/merged" rev-parse HEAD)\e[0m"
 
-"$tmp/merged/pkgs/test/nixpkgs-check-by-name/scripts/fetch-pinned-tool.sh" "$tmp/tool"
+trace -n "Reading pinned nixpkgs-check-by-name revision from pinned-tool.json.. "
+toolSha=$(jq -r .rev "$tmp/merged/pkgs/test/nixpkgs-check-by-name/scripts/pinned-tool.json")
+trace -e "\e[34m$toolSha\e[0m"
+
+trace -n "Creating Git worktree for the nixpkgs-check-by-name revision in $tmp/tool-nixpkgs.. "
+git worktree add -q "$tmp/tool-nixpkgs" "$toolSha"
+trace "Done"
+
+trace "Building/fetching nixpkgs-check-by-name.."
+nix-build -o "$tmp/tool" "$tmp/tool-nixpkgs" \
+    -A tests.nixpkgs-check-by-name \
+    --arg config '{}' \
+    --arg overlays '[]' \
+    -j 0
 
 trace "Running nixpkgs-check-by-name.."
 "$tmp/tool/bin/nixpkgs-check-by-name" --base "$tmp/base" "$tmp/merged"
diff --git a/pkgs/test/nixpkgs-check-by-name/scripts/update-pinned-tool.sh b/pkgs/test/nixpkgs-check-by-name/scripts/update-pinned-tool.sh
index 3e44cf35b0d27..b2bc629e86604 100755
--- a/pkgs/test/nixpkgs-check-by-name/scripts/update-pinned-tool.sh
+++ b/pkgs/test/nixpkgs-check-by-name/scripts/update-pinned-tool.sh
@@ -1,5 +1,5 @@
 #!/usr/bin/env nix-shell
-#!nix-shell -i bash -p jq
+#!nix-shell -i bash -p jq -I nixpkgs=../../../..
 
 set -o pipefail -o errexit -o nounset
 
@@ -7,6 +7,9 @@ trace() { echo >&2 "$@"; }
 
 SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
 
+# Determined by `runs-on: ubuntu-latest` in .github/workflows/check-by-name.yml
+CI_SYSTEM=x86_64-linux
+
 channel=nixos-unstable
 pin_file=$SCRIPT_DIR/pinned-tool.json
 
@@ -19,13 +22,19 @@ trace "$nixpkgs"
 rev=$(<"$nixpkgs/.git-revision")
 trace -e "Git revision of channel $channel is \e[34m$rev\e[0m"
 
-
-trace -n "Fetching the prebuilt version of nixpkgs-check-by-name.. "
-path=$(nix-build --no-out-link "$nixpkgs" -A tests.nixpkgs-check-by-name -j 0 | tee /dev/stderr)
+trace -n "Fetching the prebuilt version of nixpkgs-check-by-name for $CI_SYSTEM.. "
+# This is the architecture used by CI, we want to prefetch the exact path to avoid having to evaluate Nixpkgs
+ci_path=$(nix-build --no-out-link "$nixpkgs" \
+    -A tests.nixpkgs-check-by-name \
+    --arg config '{}' \
+    --argstr system "$CI_SYSTEM" \
+    --arg overlays '[]' \
+    -j 0 \
+    | tee /dev/stderr)
 
 trace "Updating $pin_file"
 jq -n \
     --arg rev "$rev" \
-    --arg path "$path" \
+    --arg ci-path "$ci_path" \
     '$ARGS.named' \
     > "$pin_file"
diff --git a/pkgs/test/nixpkgs-check-by-name/src/eval.rs b/pkgs/test/nixpkgs-check-by-name/src/eval.rs
index c4dad8257cf98..dd30cb9045e53 100644
--- a/pkgs/test/nixpkgs-check-by-name/src/eval.rs
+++ b/pkgs/test/nixpkgs-check-by-name/src/eval.rs
@@ -2,8 +2,6 @@ use crate::nixpkgs_problem::NixpkgsProblem;
 use crate::ratchet;
 use crate::structure;
 use crate::validation::{self, Validation::Success};
-use std::collections::HashMap;
-use std::ffi::OsString;
 use std::path::Path;
 
 use anyhow::Context;
@@ -73,7 +71,7 @@ enum CallPackageVariant {
 pub fn check_values(
     nixpkgs_path: &Path,
     package_names: Vec<String>,
-    eval_nix_path: &HashMap<String, PathBuf>,
+    keep_nix_path: bool,
 ) -> validation::Result<ratchet::Nixpkgs> {
     // Write the list of packages we need to check into a temporary JSON file.
     // This can then get read by the Nix evaluation.
@@ -99,8 +97,6 @@ pub fn check_values(
     command
         // Inherit stderr so that error messages always get shown
         .stderr(process::Stdio::inherit())
-        // Clear NIX_PATH to be sure it doesn't influence the result
-        .env_remove("NIX_PATH")
         .args([
             "--eval",
             "--json",
@@ -121,15 +117,12 @@ pub fn check_values(
         .arg("-I")
         .arg(nixpkgs_path);
 
-    // Also add extra paths that need to be accessible
-    for (name, path) in eval_nix_path {
-        command.arg("-I");
-        let mut name_value = OsString::new();
-        name_value.push(name);
-        name_value.push("=");
-        name_value.push(path);
-        command.arg(name_value);
+    // Clear NIX_PATH to be sure it doesn't influence the result
+    // But not when requested to keep it, used so that the tests can pass extra Nix files
+    if !keep_nix_path {
+        command.env_remove("NIX_PATH");
     }
+
     command.args(["-I", &expr_path]);
     command.arg(expr_path);
 
@@ -166,8 +159,8 @@ pub fn check_values(
                     let uses_by_name = match attribute_info {
                         // In these cases the package doesn't qualify for being in pkgs/by-name,
                         // so the UsesByName ratchet is already as tight as it can be
-                        NonAttributeSet => Success(Tight),
-                        NonCallPackage => Success(Tight),
+                        NonAttributeSet => Success(NonApplicable),
+                        NonCallPackage => Success(NonApplicable),
                         // This is the case when the `pkgs/by-name`-internal _internalCallByNamePackageFile
                         // is used for a package outside `pkgs/by-name`
                         CallPackage(CallPackageInfo {
@@ -183,14 +176,14 @@ pub fn check_values(
                             // In the future we could kind of abuse this behavior to have better
                             // enforcement of conditional aliases, but for now we just need to not
                             // give an error.
-                            Success(Tight)
+                            Success(NonApplicable)
                         }
                         // Only derivations can be in pkgs/by-name,
                         // so this attribute doesn't qualify
                         CallPackage(CallPackageInfo {
                             is_derivation: false,
                             ..
-                        }) => Success(Tight),
+                        }) => Success(NonApplicable),
 
                         // The case of an attribute that qualifies:
                         // - Uses callPackage
@@ -198,30 +191,35 @@ pub fn check_values(
                         CallPackage(CallPackageInfo {
                             is_derivation: true,
                             call_package_variant: Manual { path, empty_arg },
-                        }) => Success(Loose(ratchet::UsesByName {
+                        }) => Success(Loose(ratchet::CouldUseByName {
                             call_package_path: path,
                             empty_arg,
                         })),
                     };
                     uses_by_name.map(|x| ratchet::Package {
-                        empty_non_auto_called: Tight,
+                        manual_definition: Tight,
                         uses_by_name: x,
                     })
                 }
                 NonByName(EvalFailure) => {
-                    // This is a bit of an odd case: We don't even _know_ whether this attribute
-                    // would qualify for using pkgs/by-name. We can either:
-                    // - Assume it's not using pkgs/by-name, which has the problem that if a
-                    //   package evaluation gets broken temporarily, the fix can remove it from
-                    //   pkgs/by-name again
-                    // - Assume it's using pkgs/by-name already, which has the problem that if a
-                    //   package evaluation gets broken temporarily, fixing it requires a move to
-                    //   pkgs/by-name
-                    // We choose the latter, since we want to move towards pkgs/by-name, not away
-                    // from it
+                    // We don't know anything about this attribute really
                     Success(ratchet::Package {
-                        empty_non_auto_called: Tight,
-                        uses_by_name: Tight,
+                        // We'll assume that we can't remove any manual definitions, which has the
+                        // minimal drawback that if there was a manual definition that could've
+                        // been removed, fixing the package requires removing the definition, no
+                        // big deal, that's a minor edit.
+                        manual_definition: Tight,
+
+                        // Regarding whether this attribute could `pkgs/by-name`, we don't really
+                        // know, so return NonApplicable, which has the effect that if a
+                        // package evaluation gets broken temporarily, the fix can remove it from
+                        // pkgs/by-name again. For now this isn't our problem, but in the future we
+                        // might have another check to enforce that evaluation must not be broken.
+                        // The alternative of assuming that it's using `pkgs/by-name` already
+                        // has the problem that if a package evaluation gets broken temporarily,
+                        // fixing it requires a move to pkgs/by-name, which could happen more
+                        // often and isn't really justified.
+                        uses_by_name: NonApplicable,
                     })
                 }
                 ByName(Missing) => NixpkgsProblem::UndefinedAttr {
@@ -255,7 +253,7 @@ pub fn check_values(
 
                     check_result.and(match &call_package_variant {
                         Auto => Success(ratchet::Package {
-                            empty_non_auto_called: Tight,
+                            manual_definition: Tight,
                             uses_by_name: Tight,
                         }),
                         Manual { path, empty_arg } => {
@@ -268,11 +266,7 @@ pub fn check_values(
                             if correct_file {
                                 Success(ratchet::Package {
                                     // Empty arguments for non-auto-called packages are not allowed anymore.
-                                    empty_non_auto_called: if *empty_arg {
-                                        Loose(ratchet::EmptyNonAutoCalled)
-                                    } else {
-                                        Tight
-                                    },
+                                    manual_definition: if *empty_arg { Loose(()) } else { Tight },
                                     uses_by_name: Tight,
                                 })
                             } else {
diff --git a/pkgs/test/nixpkgs-check-by-name/src/main.rs b/pkgs/test/nixpkgs-check-by-name/src/main.rs
index 273ebca1643ed..8179ec8ded74b 100644
--- a/pkgs/test/nixpkgs-check-by-name/src/main.rs
+++ b/pkgs/test/nixpkgs-check-by-name/src/main.rs
@@ -12,7 +12,6 @@ use crate::validation::Validation::Success;
 use anyhow::Context;
 use clap::Parser;
 use colored::Colorize;
-use std::collections::HashMap;
 use std::io;
 use std::path::{Path, PathBuf};
 use std::process::ExitCode;
@@ -45,12 +44,7 @@ pub struct Args {
 
 fn main() -> ExitCode {
     let args = Args::parse();
-    match process(
-        &args.base,
-        &args.nixpkgs,
-        &HashMap::new(),
-        &mut io::stderr(),
-    ) {
+    match process(&args.base, &args.nixpkgs, false, &mut io::stderr()) {
         Ok(true) => {
             eprintln!("{}", "Validated successfully".green());
             ExitCode::SUCCESS
@@ -71,9 +65,9 @@ fn main() -> ExitCode {
 /// # Arguments
 /// - `base_nixpkgs`: Path to the base Nixpkgs to run ratchet checks against.
 /// - `main_nixpkgs`: Path to the main Nixpkgs to check.
-/// - `eval_accessible_paths`:
-///   Extra paths that need to be accessible to evaluate Nixpkgs using `restrict-eval`.
-///   This is used to allow the tests to access the mock-nixpkgs.nix file
+/// - `keep_nix_path`: Whether the value of the NIX_PATH environment variable should be kept for
+/// the evaluation stage, allowing its contents to be accessed.
+///   This is used to allow the tests to access e.g. the mock-nixpkgs.nix file
 /// - `error_writer`: An `io::Write` value to write validation errors to, if any.
 ///
 /// # Return value
@@ -83,15 +77,15 @@ fn main() -> ExitCode {
 pub fn process<W: io::Write>(
     base_nixpkgs: &Path,
     main_nixpkgs: &Path,
-    eval_nix_path: &HashMap<String, PathBuf>,
+    keep_nix_path: bool,
     error_writer: &mut W,
 ) -> anyhow::Result<bool> {
     // Check the main Nixpkgs first
-    let main_result = check_nixpkgs(main_nixpkgs, eval_nix_path, error_writer)?;
+    let main_result = check_nixpkgs(main_nixpkgs, keep_nix_path, error_writer)?;
     let check_result = main_result.result_map(|nixpkgs_version| {
         // If the main Nixpkgs doesn't have any problems, run the ratchet checks against the base
         // Nixpkgs
-        check_nixpkgs(base_nixpkgs, eval_nix_path, error_writer)?.result_map(
+        check_nixpkgs(base_nixpkgs, keep_nix_path, error_writer)?.result_map(
             |base_nixpkgs_version| {
                 Ok(ratchet::Nixpkgs::compare(
                     base_nixpkgs_version,
@@ -119,7 +113,7 @@ pub fn process<W: io::Write>(
 /// ratchet check against another result.
 pub fn check_nixpkgs<W: io::Write>(
     nixpkgs_path: &Path,
-    eval_nix_path: &HashMap<String, PathBuf>,
+    keep_nix_path: bool,
     error_writer: &mut W,
 ) -> validation::Result<ratchet::Nixpkgs> {
     Ok({
@@ -140,7 +134,7 @@ pub fn check_nixpkgs<W: io::Write>(
         } else {
             check_structure(&nixpkgs_path)?.result_map(|package_names|
                 // Only if we could successfully parse the structure, we do the evaluation checks
-                eval::check_values(&nixpkgs_path, package_names, eval_nix_path))?
+                eval::check_values(&nixpkgs_path, package_names, keep_nix_path))?
         }
     })
 }
@@ -150,10 +144,8 @@ mod tests {
     use crate::process;
     use crate::utils;
     use anyhow::Context;
-    use std::collections::HashMap;
     use std::fs;
     use std::path::Path;
-    use std::path::PathBuf;
     use tempfile::{tempdir_in, TempDir};
 
     #[test]
@@ -234,20 +226,6 @@ mod tests {
     }
 
     fn test_nixpkgs(name: &str, path: &Path, expected_errors: &str) -> anyhow::Result<()> {
-        let eval_nix_path = HashMap::from([
-            (
-                "test-nixpkgs".to_string(),
-                PathBuf::from("tests/mock-nixpkgs.nix"),
-            ),
-            (
-                "test-nixpkgs/lib".to_string(),
-                PathBuf::from(
-                    std::env::var("NIXPKGS_LIB_PATH")
-                        .with_context(|| "Could not get environment variable NIXPKGS_LIB_PATH")?,
-                ),
-            ),
-        ]);
-
         let base_path = path.join("base");
         let base_nixpkgs = if base_path.exists() {
             base_path.as_path()
@@ -258,7 +236,7 @@ mod tests {
         // We don't want coloring to mess up the tests
         let writer = temp_env::with_var("NO_COLOR", Some("1"), || -> anyhow::Result<_> {
             let mut writer = vec![];
-            process(base_nixpkgs, &path, &eval_nix_path, &mut writer)
+            process(base_nixpkgs, &path, true, &mut writer)
                 .with_context(|| format!("Failed test case {name}"))?;
             Ok(writer)
         })?;
diff --git a/pkgs/test/nixpkgs-check-by-name/src/ratchet.rs b/pkgs/test/nixpkgs-check-by-name/src/ratchet.rs
index f8c129626cc00..10ecc01d3580c 100644
--- a/pkgs/test/nixpkgs-check-by-name/src/ratchet.rs
+++ b/pkgs/test/nixpkgs-check-by-name/src/ratchet.rs
@@ -33,7 +33,7 @@ impl Nixpkgs {
 /// The ratchet value for a top-level package
 pub struct Package {
     /// The ratchet value for the check for non-auto-called empty arguments
-    pub empty_non_auto_called: RatchetState<EmptyNonAutoCalled>,
+    pub manual_definition: RatchetState<ManualDefinition>,
 
     /// The ratchet value for the check for new packages using pkgs/by-name
     pub uses_by_name: RatchetState<UsesByName>,
@@ -43,10 +43,10 @@ impl Package {
     /// Validates the ratchet checks for a top-level package
     pub fn compare(name: &str, optional_from: Option<&Self>, to: &Self) -> Validation<()> {
         validation::sequence_([
-            RatchetState::<EmptyNonAutoCalled>::compare(
+            RatchetState::<ManualDefinition>::compare(
                 name,
-                optional_from.map(|x| &x.empty_non_auto_called),
-                &to.empty_non_auto_called,
+                optional_from.map(|x| &x.manual_definition),
+                &to.manual_definition,
             ),
             RatchetState::<UsesByName>::compare(
                 name,
@@ -58,55 +58,82 @@ impl Package {
 }
 
 /// The ratchet state of a generic ratchet check.
-pub enum RatchetState<Context> {
+pub enum RatchetState<Ratchet: ToNixpkgsProblem> {
     /// The ratchet is loose, it can be tightened more.
     /// In other words, this is the legacy state we're trying to move away from.
     /// Introducing new instances is not allowed but previous instances will continue to be allowed.
     /// The `Context` is context for error messages in case a new instance of this state is
     /// introduced
-    Loose(Context),
+    Loose(Ratchet::ToContext),
     /// The ratchet is tight, it can't be tightened any further.
     /// This is either because we already use the latest state, or because the ratchet isn't
     /// relevant.
     Tight,
+    /// This ratchet can't be applied.
+    /// State transitions from/to NonApplicable are always allowed
+    NonApplicable,
 }
 
 /// A trait that can convert an attribute-specific error context into a NixpkgsProblem
 pub trait ToNixpkgsProblem {
+    /// Context relating to the Nixpkgs that is being transitioned _to_
+    type ToContext;
+
     /// How to convert an attribute-specific error context into a NixpkgsProblem
-    fn to_nixpkgs_problem(name: &str, context: &Self, existed_before: bool) -> NixpkgsProblem;
+    fn to_nixpkgs_problem(
+        name: &str,
+        optional_from: Option<()>,
+        to: &Self::ToContext,
+    ) -> NixpkgsProblem;
 }
 
 impl<Context: ToNixpkgsProblem> RatchetState<Context> {
     /// Compare the previous ratchet state of an attribute to the new state.
     /// The previous state may be `None` in case the attribute is new.
     fn compare(name: &str, optional_from: Option<&Self>, to: &Self) -> Validation<()> {
-        // If we don't have a previous state, enforce a tight ratchet
-        let from = optional_from.unwrap_or(&RatchetState::Tight);
-        match (from, to) {
-            // Always okay to keep it tight or tighten the ratchet
-            (_, RatchetState::Tight) => Success(()),
-
-            // Grandfathering policy for a loose ratchet
-            (RatchetState::Loose { .. }, RatchetState::Loose { .. }) => Success(()),
-
+        match (optional_from, to) {
             // Loosening a ratchet is now allowed
-            (RatchetState::Tight, RatchetState::Loose(context)) => {
-                Context::to_nixpkgs_problem(name, context, optional_from.is_some()).into()
+            (Some(RatchetState::Tight), RatchetState::Loose(loose_context)) => {
+                Context::to_nixpkgs_problem(name, Some(()), loose_context).into()
             }
+
+            // Introducing a loose ratchet is also not allowed
+            (None, RatchetState::Loose(loose_context)) => {
+                Context::to_nixpkgs_problem(name, None, loose_context).into()
+            }
+
+            // Everything else is allowed, including:
+            // - Loose -> Loose (grandfathering policy for a loose ratchet)
+            // - -> Tight (always okay to keep or make the ratchet tight)
+            // - Anything involving NotApplicable, where we can't really make any good calls
+            _ => Success(()),
         }
     }
 }
 
-/// The ratchet value of an attribute
-/// for the non-auto-called empty argument check of a single.
+/// The ratchet to check whether a top-level attribute has/needs
+/// a manual definition, e.g. in all-packages.nix.
+///
+/// This ratchet is only tight for attributes that:
+/// - Are not defined in `pkgs/by-name`, and rely on a manual definition
+/// - Are defined in `pkgs/by-name` without any manual definition,
+///   (no custom argument overrides)
+/// - Are defined with `pkgs/by-name` with a manual definition that can't be removed
+///   because it provides custom argument overrides
 ///
-/// This checks that packages defined in `pkgs/by-name` cannot be overridden
-/// with an empty second argument like `callPackage ... { }`.
-pub struct EmptyNonAutoCalled;
+/// In comparison, this ratchet is loose for attributes that:
+/// - Are defined in `pkgs/by-name` with a manual definition
+///   that doesn't have any custom argument overrides
+pub enum ManualDefinition {}
+
+impl ToNixpkgsProblem for ManualDefinition {
+    type ToContext = ();
 
-impl ToNixpkgsProblem for EmptyNonAutoCalled {
-    fn to_nixpkgs_problem(name: &str, _context: &Self, _existed_before: bool) -> NixpkgsProblem {
+    fn to_nixpkgs_problem(
+        name: &str,
+        _optional_from: Option<()>,
+        _to: &Self::ToContext,
+    ) -> NixpkgsProblem {
         NixpkgsProblem::WrongCallPackage {
             relative_package_file: structure::relative_file_for_package(name),
             package_name: name.to_owned(),
@@ -119,8 +146,10 @@ impl ToNixpkgsProblem for EmptyNonAutoCalled {
 ///
 /// This checks that all new package defined using callPackage must be defined via pkgs/by-name
 /// It also checks that once a package uses pkgs/by-name, it can't switch back to all-packages.nix
+pub enum UsesByName {}
+
 #[derive(Clone)]
-pub struct UsesByName {
+pub struct CouldUseByName {
     /// The first callPackage argument, used for better errors
     pub call_package_path: Option<PathBuf>,
     /// Whether the second callPackage argument is empty, used for better errors
@@ -128,18 +157,24 @@ pub struct UsesByName {
 }
 
 impl ToNixpkgsProblem for UsesByName {
-    fn to_nixpkgs_problem(name: &str, a: &Self, existed_before: bool) -> NixpkgsProblem {
-        if existed_before {
+    type ToContext = CouldUseByName;
+
+    fn to_nixpkgs_problem(
+        name: &str,
+        optional_from: Option<()>,
+        to: &Self::ToContext,
+    ) -> NixpkgsProblem {
+        if let Some(()) = optional_from {
             NixpkgsProblem::MovedOutOfByName {
                 package_name: name.to_owned(),
-                call_package_path: a.call_package_path.clone(),
-                empty_arg: a.empty_arg,
+                call_package_path: to.call_package_path.clone(),
+                empty_arg: to.empty_arg,
             }
         } else {
             NixpkgsProblem::NewPackageNotUsingByName {
                 package_name: name.to_owned(),
-                call_package_path: a.call_package_path.clone(),
-                empty_arg: a.empty_arg,
+                call_package_path: to.call_package_path.clone(),
+                empty_arg: to.empty_arg,
             }
         }
     }
diff --git a/pkgs/test/nixpkgs-check-by-name/src/references.rs b/pkgs/test/nixpkgs-check-by-name/src/references.rs
index 3b3b05419780a..ce7403afb32d6 100644
--- a/pkgs/test/nixpkgs-check-by-name/src/references.rs
+++ b/pkgs/test/nixpkgs-check-by-name/src/references.rs
@@ -106,6 +106,10 @@ fn check_nix_file(
 
     let root = Root::parse(&contents);
     if let Some(error) = root.errors().first() {
+        // NOTE: There's now another Nixpkgs CI check to make sure all changed Nix files parse
+        // correctly, though that uses mainline Nix instead of rnix, so it doesn't give the same
+        // errors. In the future we should unify these two checks, ideally moving the other CI
+        // check into this tool as well and checking for both mainline Nix and rnix.
         return Ok(NixpkgsProblem::CouldNotParseNix {
             relative_package_dir: relative_package_dir.to_path_buf(),
             subpath: subpath.to_path_buf(),
diff --git a/pkgs/test/nixpkgs-check-by-name/tests/manual-definition/all-packages.nix b/pkgs/test/nixpkgs-check-by-name/tests/manual-definition/all-packages.nix
new file mode 100644
index 0000000000000..07b2caaab4e71
--- /dev/null
+++ b/pkgs/test/nixpkgs-check-by-name/tests/manual-definition/all-packages.nix
@@ -0,0 +1,10 @@
+self: super: {
+  nonAttributeSet = self.callPackage ({ someDrv }: someDrv) { };
+  nonCallPackage = self.callPackage ({ someDrv }: someDrv) { };
+  internalCallByName = self.callPackage ({ someDrv }: someDrv) { };
+  nonDerivation = self.callPackage ({ someDrv }: someDrv) { };
+
+  onlyMove = self.callPackage ./pkgs/by-name/on/onlyMove/package.nix { };
+
+  noEval = self.callPackage ./pkgs/by-name/no/noEval/package.nix { };
+}
diff --git a/pkgs/test/nixpkgs-check-by-name/tests/manual-definition/base/all-packages.nix b/pkgs/test/nixpkgs-check-by-name/tests/manual-definition/base/all-packages.nix
new file mode 100644
index 0000000000000..75efb5952e7a6
--- /dev/null
+++ b/pkgs/test/nixpkgs-check-by-name/tests/manual-definition/base/all-packages.nix
@@ -0,0 +1,9 @@
+self: super: {
+  nonAttributeSet = null;
+  nonCallPackage = self.someDrv;
+  internalCallByName = self._internalCallByNamePackageFile ./some-pkg.nix;
+  nonDerivation = self.callPackage ({ }: { }) { };
+
+  onlyMove = self.callPackage ({ someDrv }: someDrv) { };
+  noEval = throw "foo";
+}
diff --git a/pkgs/test/nixpkgs-check-by-name/tests/ref-parse-failure/default.nix b/pkgs/test/nixpkgs-check-by-name/tests/manual-definition/base/default.nix
index 861260cdca4b2..861260cdca4b2 100644
--- a/pkgs/test/nixpkgs-check-by-name/tests/ref-parse-failure/default.nix
+++ b/pkgs/test/nixpkgs-check-by-name/tests/manual-definition/base/default.nix
diff --git a/pkgs/test/nixpkgs-check-by-name/tests/manual-definition/base/pkgs/by-name/README.md b/pkgs/test/nixpkgs-check-by-name/tests/manual-definition/base/pkgs/by-name/README.md
new file mode 100644
index 0000000000000..e69de29bb2d1d
--- /dev/null
+++ b/pkgs/test/nixpkgs-check-by-name/tests/manual-definition/base/pkgs/by-name/README.md
diff --git a/pkgs/test/nixpkgs-check-by-name/tests/no-eval/pkgs/by-name/fo/foo/package.nix b/pkgs/test/nixpkgs-check-by-name/tests/manual-definition/base/some-pkg.nix
index a1b92efbbadb9..a1b92efbbadb9 100644
--- a/pkgs/test/nixpkgs-check-by-name/tests/no-eval/pkgs/by-name/fo/foo/package.nix
+++ b/pkgs/test/nixpkgs-check-by-name/tests/manual-definition/base/some-pkg.nix
diff --git a/pkgs/test/nixpkgs-check-by-name/tests/manual-definition/default.nix b/pkgs/test/nixpkgs-check-by-name/tests/manual-definition/default.nix
new file mode 100644
index 0000000000000..861260cdca4b2
--- /dev/null
+++ b/pkgs/test/nixpkgs-check-by-name/tests/manual-definition/default.nix
@@ -0,0 +1 @@
+import <test-nixpkgs> { root = ./.; }
diff --git a/pkgs/test/nixpkgs-check-by-name/tests/manual-definition/expected b/pkgs/test/nixpkgs-check-by-name/tests/manual-definition/expected
new file mode 100644
index 0000000000000..29d33f7dbdc00
--- /dev/null
+++ b/pkgs/test/nixpkgs-check-by-name/tests/manual-definition/expected
@@ -0,0 +1,2 @@
+pkgs.noEval: This attribute is manually defined (most likely in pkgs/top-level/all-packages.nix), which is only allowed if the definition is of the form `pkgs.callPackage pkgs/by-name/no/noEval/package.nix { ... }` with a non-empty second argument.
+pkgs.onlyMove: This attribute is manually defined (most likely in pkgs/top-level/all-packages.nix), which is only allowed if the definition is of the form `pkgs.callPackage pkgs/by-name/on/onlyMove/package.nix { ... }` with a non-empty second argument.
diff --git a/pkgs/test/nixpkgs-check-by-name/tests/ref-parse-failure/pkgs/by-name/aa/aa/package.nix b/pkgs/test/nixpkgs-check-by-name/tests/manual-definition/pkgs/by-name/no/noEval/package.nix
index a1b92efbbadb9..a1b92efbbadb9 100644
--- a/pkgs/test/nixpkgs-check-by-name/tests/ref-parse-failure/pkgs/by-name/aa/aa/package.nix
+++ b/pkgs/test/nixpkgs-check-by-name/tests/manual-definition/pkgs/by-name/no/noEval/package.nix
diff --git a/pkgs/test/nixpkgs-check-by-name/tests/manual-definition/pkgs/by-name/on/onlyMove/package.nix b/pkgs/test/nixpkgs-check-by-name/tests/manual-definition/pkgs/by-name/on/onlyMove/package.nix
new file mode 100644
index 0000000000000..a1b92efbbadb9
--- /dev/null
+++ b/pkgs/test/nixpkgs-check-by-name/tests/manual-definition/pkgs/by-name/on/onlyMove/package.nix
@@ -0,0 +1 @@
+{ someDrv }: someDrv
diff --git a/pkgs/test/nixpkgs-check-by-name/tests/multiple-failures/default.nix b/pkgs/test/nixpkgs-check-by-name/tests/multiple-failures/default.nix
new file mode 100644
index 0000000000000..861260cdca4b2
--- /dev/null
+++ b/pkgs/test/nixpkgs-check-by-name/tests/multiple-failures/default.nix
@@ -0,0 +1 @@
+import <test-nixpkgs> { root = ./.; }
diff --git a/pkgs/test/nixpkgs-check-by-name/tests/multiple-failures/expected b/pkgs/test/nixpkgs-check-by-name/tests/multiple-failures/expected
new file mode 100644
index 0000000000000..ff5d18556ef03
--- /dev/null
+++ b/pkgs/test/nixpkgs-check-by-name/tests/multiple-failures/expected
@@ -0,0 +1,13 @@
+pkgs/by-name/A: Invalid directory name "A", must be at most 2 ASCII characters consisting of a-z, 0-9, "-" or "_".
+pkgs/by-name/A/fo@: Invalid package directory name "fo@", must be ASCII characters consisting of a-z, A-Z, 0-9, "-" or "_".
+pkgs/by-name/A/fo@: Path foo is a symlink which cannot be resolved: No such file or directory (os error 2).
+pkgs/by-name/A/fo@: Path package.nix is a symlink pointing to a path outside the directory of that package.
+pkgs/by-name/aa: This is a file, but it should be a directory.
+pkgs/by-name/ba/bar: This path is a file, but it should be a directory.
+pkgs/by-name/ba/baz: "package.nix" must be a file.
+pkgs/by-name/ba/foo: Incorrect directory location, should be pkgs/by-name/fo/foo instead.
+pkgs/by-name/ba/foo: File package.nix at line 2 contains the path expression "/bar" which cannot be resolved: No such file or directory (os error 2).
+pkgs/by-name/ba/foo: File package.nix at line 3 contains the path expression "../." which may point outside the directory of that package.
+pkgs/by-name/ba/foo: File package.nix at line 4 contains the nix search path expression "<nixpkgs>" which may point outside the directory of that package.
+pkgs/by-name/ba/foo: File package.nix at line 5 contains the path expression "./${"test"}", which is not yet supported and may point outside the directory of that package.
+pkgs/by-name/fo/foo: Missing required "package.nix" file.
diff --git a/pkgs/test/nixpkgs-check-by-name/tests/multiple-failures/pkgs/by-name/A/fo@/foo b/pkgs/test/nixpkgs-check-by-name/tests/multiple-failures/pkgs/by-name/A/fo@/foo
new file mode 120000
index 0000000000000..c86c3f3551ecf
--- /dev/null
+++ b/pkgs/test/nixpkgs-check-by-name/tests/multiple-failures/pkgs/by-name/A/fo@/foo
@@ -0,0 +1 @@
+none
\ No newline at end of file
diff --git a/pkgs/test/nixpkgs-check-by-name/tests/multiple-failures/pkgs/by-name/A/fo@/package.nix b/pkgs/test/nixpkgs-check-by-name/tests/multiple-failures/pkgs/by-name/A/fo@/package.nix
new file mode 120000
index 0000000000000..f079163d158a2
--- /dev/null
+++ b/pkgs/test/nixpkgs-check-by-name/tests/multiple-failures/pkgs/by-name/A/fo@/package.nix
@@ -0,0 +1 @@
+../../../../someDrv.nix
\ No newline at end of file
diff --git a/pkgs/test/nixpkgs-check-by-name/tests/multiple-failures/pkgs/by-name/aa b/pkgs/test/nixpkgs-check-by-name/tests/multiple-failures/pkgs/by-name/aa
new file mode 100644
index 0000000000000..e69de29bb2d1d
--- /dev/null
+++ b/pkgs/test/nixpkgs-check-by-name/tests/multiple-failures/pkgs/by-name/aa
diff --git a/pkgs/test/nixpkgs-check-by-name/tests/multiple-failures/pkgs/by-name/ba/bar b/pkgs/test/nixpkgs-check-by-name/tests/multiple-failures/pkgs/by-name/ba/bar
new file mode 100644
index 0000000000000..e69de29bb2d1d
--- /dev/null
+++ b/pkgs/test/nixpkgs-check-by-name/tests/multiple-failures/pkgs/by-name/ba/bar
diff --git a/pkgs/test/nixpkgs-check-by-name/tests/multiple-failures/pkgs/by-name/ba/baz/package.nix/default.nix b/pkgs/test/nixpkgs-check-by-name/tests/multiple-failures/pkgs/by-name/ba/baz/package.nix/default.nix
new file mode 100644
index 0000000000000..a1b92efbbadb9
--- /dev/null
+++ b/pkgs/test/nixpkgs-check-by-name/tests/multiple-failures/pkgs/by-name/ba/baz/package.nix/default.nix
@@ -0,0 +1 @@
+{ someDrv }: someDrv
diff --git a/pkgs/test/nixpkgs-check-by-name/tests/multiple-failures/pkgs/by-name/ba/foo/package.nix b/pkgs/test/nixpkgs-check-by-name/tests/multiple-failures/pkgs/by-name/ba/foo/package.nix
new file mode 100644
index 0000000000000..31b4742f5914f
--- /dev/null
+++ b/pkgs/test/nixpkgs-check-by-name/tests/multiple-failures/pkgs/by-name/ba/foo/package.nix
@@ -0,0 +1,6 @@
+{ someDrv }: someDrv // {
+  escapeAbsolute = /bar;
+  escapeRelative = ../.;
+  nixPath = <nixpkgs>;
+  pathWithSubexpr = ./${"test"};
+}
diff --git a/pkgs/test/nixpkgs-check-by-name/tests/multiple-failures/pkgs/by-name/fo/foo/.git-keep b/pkgs/test/nixpkgs-check-by-name/tests/multiple-failures/pkgs/by-name/fo/foo/.git-keep
new file mode 100644
index 0000000000000..e69de29bb2d1d
--- /dev/null
+++ b/pkgs/test/nixpkgs-check-by-name/tests/multiple-failures/pkgs/by-name/fo/foo/.git-keep
diff --git a/pkgs/test/nixpkgs-check-by-name/tests/multiple-failures/someDrv.nix b/pkgs/test/nixpkgs-check-by-name/tests/multiple-failures/someDrv.nix
new file mode 100644
index 0000000000000..a1b92efbbadb9
--- /dev/null
+++ b/pkgs/test/nixpkgs-check-by-name/tests/multiple-failures/someDrv.nix
@@ -0,0 +1 @@
+{ someDrv }: someDrv
diff --git a/pkgs/test/nixpkgs-check-by-name/tests/no-eval/all-packages.nix b/pkgs/test/nixpkgs-check-by-name/tests/no-eval/all-packages.nix
index e2831c2d542e6..38762c6de1cc9 100644
--- a/pkgs/test/nixpkgs-check-by-name/tests/no-eval/all-packages.nix
+++ b/pkgs/test/nixpkgs-check-by-name/tests/no-eval/all-packages.nix
@@ -1,3 +1,5 @@
 self: super: {
   iDontEval = throw "I don't eval";
+
+  futureEval = self.callPackage ({ someDrv }: someDrv) { };
 }
diff --git a/pkgs/test/nixpkgs-check-by-name/tests/no-eval/base/all-packages.nix b/pkgs/test/nixpkgs-check-by-name/tests/no-eval/base/all-packages.nix
new file mode 100644
index 0000000000000..ac763b454eb08
--- /dev/null
+++ b/pkgs/test/nixpkgs-check-by-name/tests/no-eval/base/all-packages.nix
@@ -0,0 +1,3 @@
+self: super: {
+  futureEval = throw "foo";
+}
diff --git a/pkgs/test/nixpkgs-check-by-name/tests/no-eval/base/default.nix b/pkgs/test/nixpkgs-check-by-name/tests/no-eval/base/default.nix
new file mode 100644
index 0000000000000..861260cdca4b2
--- /dev/null
+++ b/pkgs/test/nixpkgs-check-by-name/tests/no-eval/base/default.nix
@@ -0,0 +1 @@
+import <test-nixpkgs> { root = ./.; }
diff --git a/pkgs/test/nixpkgs-check-by-name/tests/no-eval/base/pkgs/by-name/README.md b/pkgs/test/nixpkgs-check-by-name/tests/no-eval/base/pkgs/by-name/README.md
new file mode 100644
index 0000000000000..e69de29bb2d1d
--- /dev/null
+++ b/pkgs/test/nixpkgs-check-by-name/tests/no-eval/base/pkgs/by-name/README.md
diff --git a/pkgs/test/nixpkgs-check-by-name/tests/no-eval/pkgs/by-name/README.md b/pkgs/test/nixpkgs-check-by-name/tests/no-eval/pkgs/by-name/README.md
new file mode 100644
index 0000000000000..e69de29bb2d1d
--- /dev/null
+++ b/pkgs/test/nixpkgs-check-by-name/tests/no-eval/pkgs/by-name/README.md
diff --git a/pkgs/test/nixpkgs-check-by-name/tests/override-empty-arg/base/pkgs/by-name/README.md b/pkgs/test/nixpkgs-check-by-name/tests/override-empty-arg/base/pkgs/by-name/README.md
index b0d2b34e338a8..e69de29bb2d1d 100644
--- a/pkgs/test/nixpkgs-check-by-name/tests/override-empty-arg/base/pkgs/by-name/README.md
+++ b/pkgs/test/nixpkgs-check-by-name/tests/override-empty-arg/base/pkgs/by-name/README.md
@@ -1 +0,0 @@
-(this is just here so the directory can get tracked by git)
diff --git a/pkgs/test/nixpkgs-check-by-name/tests/ref-parse-failure/expected b/pkgs/test/nixpkgs-check-by-name/tests/ref-parse-failure/expected
deleted file mode 100644
index 281aba0092367..0000000000000
--- a/pkgs/test/nixpkgs-check-by-name/tests/ref-parse-failure/expected
+++ /dev/null
@@ -1 +0,0 @@
-pkgs/by-name/aa/aa: File invalid.nix could not be parsed by rnix: unexpected token at 28..29
diff --git a/pkgs/test/nixpkgs-check-by-name/tests/ref-parse-failure/pkgs/by-name/aa/aa/invalid.nix b/pkgs/test/nixpkgs-check-by-name/tests/ref-parse-failure/pkgs/by-name/aa/aa/invalid.nix
deleted file mode 100644
index ee6b002a529be..0000000000000
--- a/pkgs/test/nixpkgs-check-by-name/tests/ref-parse-failure/pkgs/by-name/aa/aa/invalid.nix
+++ /dev/null
@@ -1 +0,0 @@
-this is not a valid nix file!
diff --git a/pkgs/test/nixpkgs-check-by-name/tests/symlink-invalid/expected b/pkgs/test/nixpkgs-check-by-name/tests/symlink-invalid/expected
index f622f3e7fd6dd..c1e7a28205a77 100644
--- a/pkgs/test/nixpkgs-check-by-name/tests/symlink-invalid/expected
+++ b/pkgs/test/nixpkgs-check-by-name/tests/symlink-invalid/expected
@@ -1 +1 @@
-pkgs/by-name/fo/foo: Path foo.nix is a symlink which cannot be resolved: No such file or directory (os error 2).
+pkgs/by-name/fo/foo: Path foo is a symlink which cannot be resolved: No such file or directory (os error 2).
diff --git a/pkgs/test/nixpkgs-check-by-name/tests/symlink-invalid/pkgs/by-name/fo/foo/foo b/pkgs/test/nixpkgs-check-by-name/tests/symlink-invalid/pkgs/by-name/fo/foo/foo
new file mode 120000
index 0000000000000..c86c3f3551ecf
--- /dev/null
+++ b/pkgs/test/nixpkgs-check-by-name/tests/symlink-invalid/pkgs/by-name/fo/foo/foo
@@ -0,0 +1 @@
+none
\ No newline at end of file
diff --git a/pkgs/test/nixpkgs-check-by-name/tests/symlink-invalid/pkgs/by-name/fo/foo/foo.nix b/pkgs/test/nixpkgs-check-by-name/tests/symlink-invalid/pkgs/by-name/fo/foo/foo.nix
deleted file mode 120000
index 49cd425a8cdba..0000000000000
--- a/pkgs/test/nixpkgs-check-by-name/tests/symlink-invalid/pkgs/by-name/fo/foo/foo.nix
+++ /dev/null
@@ -1 +0,0 @@
-none.nix
\ No newline at end of file