From b4acd9772975d549f491d48debb679cf4ec78133 Mon Sep 17 00:00:00 2001 From: Frederik Rietdijk Date: Sat, 5 Jan 2019 10:26:57 +0100 Subject: buildPython*: enable strictDeps --- doc/languages-frameworks/python.section.md | 19 +++++++++++++++++-- 1 file changed, 17 insertions(+), 2 deletions(-) (limited to 'doc') diff --git a/doc/languages-frameworks/python.section.md b/doc/languages-frameworks/python.section.md index acd2bf769b0dc..4cb275f5f3ee4 100644 --- a/doc/languages-frameworks/python.section.md +++ b/doc/languages-frameworks/python.section.md @@ -602,10 +602,9 @@ as the interpreter unless overridden otherwise. All parameters from `stdenv.mkDerivation` function are still supported. The following are specific to `buildPythonPackage`: * `catchConflicts ? true`: If `true`, abort package build if a package name appears more than once in dependency tree. Default is `true`. -* `checkInputs ? []`: Dependencies needed for running the `checkPhase`. These are added to `buildInputs` when `doCheck = true`. * `disabled` ? false: If `true`, package is not build for the particular Python interpreter version. * `dontWrapPythonPrograms ? false`: Skip wrapping of python programs. -* `installFlags ? []`: A list of strings. Arguments to be passed to `pip install`. To pass options to `python setup.py install`, use `--install-option`. E.g., `installFlags=["--install-option='--cpp_implementation'"]. +* `installFlags ? []`: A list of strings. Arguments to be passed to `pip install`. To pass options to `python setup.py install`, use `--install-option`. E.g., `installFlags=["--install-option='--cpp_implementation'"]`. * `format ? "setuptools"`: Format of the source. Valid options are `"setuptools"`, `"flit"`, `"wheel"`, and `"other"`. `"setuptools"` is for when the source has a `setup.py` and `setuptools` is used to build a wheel, `flit`, in case `flit` should be used to build a wheel, and `wheel` in case a wheel is provided. Use `other` when a custom `buildPhase` and/or `installPhase` is needed. * `makeWrapperArgs ? []`: A list of strings. Arguments to be passed to `makeWrapper`, which wraps generated binaries. By default, the arguments to `makeWrapper` set `PATH` and `PYTHONPATH` environment variables before calling the binary. Additional arguments here can allow a developer to set environment variables which will be available when the binary is run. For example, `makeWrapperArgs = ["--set FOO BAR" "--set BAZ QUX"]`. * `namePrefix`: Prepends text to `${name}` parameter. In case of libraries, this defaults to `"python3.5-"` for Python 3.5, etc., and in case of applications to `""`. @@ -615,6 +614,14 @@ All parameters from `stdenv.mkDerivation` function are still supported. The foll * `removeBinByteCode ? true`: Remove bytecode from `/bin`. Bytecode is only created when the filenames end with `.py`. * `setupPyBuildFlags ? []`: List of flags passed to `setup.py build_ext` command. +The `stdenv.mkDerivation` function accepts various parameters for describing build inputs (see "Specifying dependencies"). The following are of special +interest for Python packages, either because these are primarily used, or because their behaviour is different: + +* `nativeBuildInputs ? []`: Build-time only dependencies. Typically executables as well as the items listed in `setup_requires`. +* `buildInputs ? []`: Build and/or run-time dependencies that need to be be compiled for the host machine. Typically non-Python libraries which are being linked. +* `checkInputs ? []`: Dependencies needed for running the `checkPhase`. These are added to `nativeBuildInputs` when `doCheck = true`. Items listed in `tests_require` go here. +* `propagatedBuildInputs ? []`: Aside from propagating dependencies, `buildPythonPackage` also injects code into and wraps executables with the paths included in this list. Items listed in `install_requires` go here. + ##### Overriding Python packages The `buildPythonPackage` function has a `overridePythonAttrs` method that @@ -1123,6 +1130,14 @@ LLVM implementation. To use that one instead, Intel recommends users set it with Note that `mkl` is only available on `x86_64-{linux,darwin}` platforms; moreover, Hydra is not building and distributing pre-compiled binaries using it. +### What inputs do `setup_requires`, `install_requires` and `tests_require` map to? + +In a `setup.py` or `setup.cfg` it is common to declare dependencies: + +* `setup_requires` corresponds to `nativeBuildInputs` +* `install_requires` corresponds to `propagatedBuildInputs` +* `tests_require` corresponds to `checkInputs` + ## Contributing ### Contributing guidelines -- cgit 1.4.1 From e7b4608d029dec2d14758837c0378fcf4746f3be Mon Sep 17 00:00:00 2001 From: Frederik Rietdijk Date: Sun, 17 Feb 2019 20:33:30 +0100 Subject: buildPythonPackage: initial support for PEP 517 --- doc/languages-frameworks/python.section.md | 2 +- .../python/build-python-package-pyproject.nix | 53 ++++++++++++++++++++++ .../interpreters/python/build-python-package.nix | 4 +- 3 files changed, 57 insertions(+), 2 deletions(-) create mode 100644 pkgs/development/interpreters/python/build-python-package-pyproject.nix (limited to 'doc') diff --git a/doc/languages-frameworks/python.section.md b/doc/languages-frameworks/python.section.md index 4cb275f5f3ee4..36a87c8380faf 100644 --- a/doc/languages-frameworks/python.section.md +++ b/doc/languages-frameworks/python.section.md @@ -605,7 +605,7 @@ All parameters from `stdenv.mkDerivation` function are still supported. The foll * `disabled` ? false: If `true`, package is not build for the particular Python interpreter version. * `dontWrapPythonPrograms ? false`: Skip wrapping of python programs. * `installFlags ? []`: A list of strings. Arguments to be passed to `pip install`. To pass options to `python setup.py install`, use `--install-option`. E.g., `installFlags=["--install-option='--cpp_implementation'"]`. -* `format ? "setuptools"`: Format of the source. Valid options are `"setuptools"`, `"flit"`, `"wheel"`, and `"other"`. `"setuptools"` is for when the source has a `setup.py` and `setuptools` is used to build a wheel, `flit`, in case `flit` should be used to build a wheel, and `wheel` in case a wheel is provided. Use `other` when a custom `buildPhase` and/or `installPhase` is needed. +* `format ? "setuptools"`: Format of the source. Valid options are `"setuptools"`, `"pyproject"`, `"flit"`, `"wheel"`, and `"other"`. `"setuptools"` is for when the source has a `setup.py` and `setuptools` is used to build a wheel, `flit`, in case `flit` should be used to build a wheel, and `wheel` in case a wheel is provided. Use `other` when a custom `buildPhase` and/or `installPhase` is needed. * `makeWrapperArgs ? []`: A list of strings. Arguments to be passed to `makeWrapper`, which wraps generated binaries. By default, the arguments to `makeWrapper` set `PATH` and `PYTHONPATH` environment variables before calling the binary. Additional arguments here can allow a developer to set environment variables which will be available when the binary is run. For example, `makeWrapperArgs = ["--set FOO BAR" "--set BAZ QUX"]`. * `namePrefix`: Prepends text to `${name}` parameter. In case of libraries, this defaults to `"python3.5-"` for Python 3.5, etc., and in case of applications to `""`. * `pythonPath ? []`: List of packages to be added into `$PYTHONPATH`. Packages in `pythonPath` are not propagated (contrary to `propagatedBuildInputs`). diff --git a/pkgs/development/interpreters/python/build-python-package-pyproject.nix b/pkgs/development/interpreters/python/build-python-package-pyproject.nix new file mode 100644 index 0000000000000..86c450fcf92c8 --- /dev/null +++ b/pkgs/development/interpreters/python/build-python-package-pyproject.nix @@ -0,0 +1,53 @@ +# This function provides specific bits for building a setuptools-based Python package. + +{ lib +, python +}: + +{ +# passed to "python setup.py build_ext" +# https://github.com/pypa/pip/issues/881 +# Rename to `buildOptions` because it is not setuptools specific? + setupPyBuildFlags ? [] +# Execute before shell hook +, preShellHook ? "" +# Execute after shell hook +, postShellHook ? "" +, ... } @ attrs: + +let + options = lib.concatMapStringsSep " " (option: "--global-option ${option}") setupPyBuildFlags; +in attrs // { + buildPhase = attrs.buildPhase or '' + runHook preBuild + mkdir -p dist + echo "Creating a wheel..." + ${python.pythonForBuild.interpreter} -m pip wheel --no-index --no-deps --no-clean --no-build-isolation --wheel-dir dist ${options} . + echo "Finished creating a wheel..." + runHook postBuild + ''; + + installCheckPhase = '' + runHook preCheck + echo "No checkPhase defined. Either provide a checkPhase or disable tests in case tests are not available."; exit 1 + runHook postCheck + ''; + + # With Python it's a common idiom to run the tests + # after the software has been installed. + doCheck = attrs.doCheck or true; + + shellHook = attrs.shellHook or '' + ${preShellHook} + # Long-term setup.py should be dropped. + if [ -e pyproject.toml ]; then + tmp_path=$(mktemp -d) + export PATH="$tmp_path/bin:$PATH" + export PYTHONPATH="$tmp_path/${python.pythonForBuild.sitePackages}:$PYTHONPATH" + mkdir -p $tmp_path/${python.pythonForBuild.sitePackages} + ${python.pythonForBuild.pkgs.bootstrapped-pip}/bin/pip install -e . --prefix $tmp_path >&2 + fi + ${postShellHook} + ''; + +} \ No newline at end of file diff --git a/pkgs/development/interpreters/python/build-python-package.nix b/pkgs/development/interpreters/python/build-python-package.nix index b664cf0b14f46..98322312f7f3a 100644 --- a/pkgs/development/interpreters/python/build-python-package.nix +++ b/pkgs/development/interpreters/python/build-python-package.nix @@ -17,6 +17,7 @@ let setuptools-specific = import ./build-python-package-setuptools.nix { inherit lib python; }; + pyproject-specific = import ./build-python-package-pyproject.nix { inherit lib python; }; flit-specific = import ./build-python-package-flit.nix { inherit python flit; }; wheel-specific = import ./build-python-package-wheel.nix { }; common = import ./build-python-package-common.nix { inherit python; }; @@ -37,7 +38,8 @@ format ? "setuptools" let formatspecific = - if format == "setuptools" then common (setuptools-specific attrs) + if format == "pyproject" then common (pyproject-specific attrs) + else if format == "setuptools" then common (setuptools-specific attrs) else if format == "flit" then common (flit-specific attrs) else if format == "wheel" then common (wheel-specific attrs) else if format == "other" then {} -- cgit 1.4.1 From 1fccd255953b6025a4d0342c2cf5641e5b6b6ced Mon Sep 17 00:00:00 2001 From: Frederik Rietdijk Date: Fri, 15 Feb 2019 13:30:59 +0100 Subject: buildPythonPackage: always export `LANG=C.UTF-8` --- doc/languages-frameworks/python.section.md | 1 - nixos/doc/manual/release-notes/rl-1903.xml | 8 +++++++- pkgs/development/interpreters/python/mk-python-derivation.nix | 2 ++ 3 files changed, 9 insertions(+), 2 deletions(-) (limited to 'doc') diff --git a/doc/languages-frameworks/python.section.md b/doc/languages-frameworks/python.section.md index 36a87c8380faf..1c9f9fdebe427 100644 --- a/doc/languages-frameworks/python.section.md +++ b/doc/languages-frameworks/python.section.md @@ -881,7 +881,6 @@ example of such a situation is when `py.test` is used. ''; } ``` -- Unicode issues can typically be fixed by including `glibcLocales` in `buildInputs` and exporting `LC_ALL=en_US.utf-8`. - Tests that attempt to access `$HOME` can be fixed by using the following work-around before running tests (e.g. `preCheck`): `export HOME=$(mktemp -d)` ## FAQ diff --git a/nixos/doc/manual/release-notes/rl-1903.xml b/nixos/doc/manual/release-notes/rl-1903.xml index cba4c08708e9e..0785784f25acc 100644 --- a/nixos/doc/manual/release-notes/rl-1903.xml +++ b/nixos/doc/manual/release-notes/rl-1903.xml @@ -158,12 +158,18 @@ - The buildPythonPackage now sets strictDeps = true + The buildPythonPackage function now sets strictDeps = true to help distinguish between native and non-native dependencies in order to improve cross-compilation compatibility. Note however that this may break user expressions. + + + The buildPythonPackage function now sets LANG = C.UTF-8 + to enable Unicode support. The glibcLocales package is no longer needed as a build input. + + The Syncthing state and configuration data has been moved from diff --git a/pkgs/development/interpreters/python/mk-python-derivation.nix b/pkgs/development/interpreters/python/mk-python-derivation.nix index 4142cb922f7f0..b7477c9b7b7d9 100644 --- a/pkgs/development/interpreters/python/mk-python-derivation.nix +++ b/pkgs/development/interpreters/python/mk-python-derivation.nix @@ -90,6 +90,8 @@ let self = toPythonModule (python.stdenv.mkDerivation (builtins.removeAttrs attr # Enabled to detect some (native)BuildInputs mistakes strictDeps = true; + LANG = "${if python.stdenv.isDarwin then "en_US" else "C"}.UTF-8"; + # Python packages don't have a checkPhase, only an installCheckPhase doCheck = false; doInstallCheck = doCheck; -- cgit 1.4.1