No material changes to docs, but trying to sanitize them for consistent
readability prior to looking at #75837.
- Use `*` for lists instead of `-`. I have no opinion one way or the other, but
the latter was only used in 1-2 places.
- Pad the code blocks with whitespace.
- Wrap to 80 characters, except for a few 1-liners that were only slightly over.
Executing `nix-shell` gives you again a Nix shell from which you can run Python.
What's happening here?
1. We begin with importing the Nix Packages collections. `import <nixpkgs>` imports the `<nixpkgs>` function, `{}` calls it and the `with` statement brings all attributes of `nixpkgs` in the local scope. These attributes form the main package set.
1. We begin with importing the Nix Packages collections. `import <nixpkgs>`
imports the `<nixpkgs>` function, `{}` calls it and the `with` statement
brings all attributes of `nixpkgs` in the local scope. These attributes form
the main package set.
2. Then we create a Python 3.5 environment with the `withPackages` function.
3. The `withPackages` function expects us to provide a function as an argument that takes the set of all python packages and returns a list of packages to include in the environment. Here, we select the packages `numpy` and `toolz` from the package set.
3. The `withPackages` function expects us to provide a function as an argument
that takes the set of all python packages and returns a list of packages to
include in the environment. Here, we select the packages `numpy` and `toolz`
from the package set.
To combine this with `mkShell` you can:
@ -166,20 +186,23 @@ in mkShell {
A convenient option with `nix-shell` is the `--run`
option, with which you can execute a command in the `nix-shell`. We can
@ -404,17 +428,18 @@ instead of installing the package this command creates a special link to the pro
That way, you can run updated code without having to reinstall after each and every change you make.
Development mode is also available. Let's see how you can use it.
In the previous Nix expression the source was fetched from an url. We can also refer to a local source instead using
`src = ./path/to/source/tree;`
In the previous Nix expression the source was fetched from an url. We can also
refer to a local source instead using `src = ./path/to/source/tree;`
If we create a `shell.nix` file which calls `buildPythonPackage`, and if `src`
is a local source, and if the local source has a `setup.py`, then development
mode is activated.
In the following example we create a simple environment that
has a Python 3.5 version of our package in it, as well as its dependencies and
other packages we like to have in the environment, all specified with `propagatedBuildInputs`.
Indeed, we can just add any package we like to have in our environment to `propagatedBuildInputs`.
In the following example we create a simple environment that has a Python 3.5
version of our package in it, as well as its dependencies and other packages we
like to have in the environment, all specified with `propagatedBuildInputs`.
Indeed, we can just add any package we like to have in our environment to
`propagatedBuildInputs`.
```nix
with import <nixpkgs> {};
@ -427,7 +452,8 @@ buildPythonPackage rec {
}
```
It is important to note that due to how development mode is implemented on Nix it is not possible to have multiple packages simultaneously in development mode.
It is important to note that due to how development mode is implemented on Nix
it is not possible to have multiple packages simultaneously in development mode.
### Organising your packages
@ -497,13 +523,13 @@ and in this case the `python35` interpreter is automatically used.
### Interpreters
Versions 2.7, 3.5, 3.6, 3.7 and 3.8 of the CPython interpreter are available as
respectively `python27`, `python35`, `python36`, `python37` and `python38`. The aliases
`python2` and `python3` correspond to respectively `python27` and
respectively `python27`, `python35`, `python36`, `python37` and `python38`. The
aliases `python2` and `python3` correspond to respectively `python27` and
`python37`. The default interpreter, `python`, maps to `python2`. The PyPy
interpreters compatible with Python 2.7 and 3 are available as `pypy27` and
`pypy3`, with aliases `pypy2` mapping to `pypy27` and `pypy` mapping to
`pypy2`. The Nix expressions for the interpreters can be
found in `pkgs/development/interpreters/python`.
`pypy3`, with aliases `pypy2` mapping to `pypy27` and `pypy` mapping to`pypy2`.
The Nix expressions for the interpreters can be found in
`pkgs/development/interpreters/python`.
All packages depending on any Python interpreter get appended
`out/{python.sitePackages}` to `$PYTHONPATH` if such directory
@ -532,9 +558,10 @@ Python libraries and applications that use `setuptools` or
`buildPythonApplication` functions. These two functions also support installing a `wheel`.
All Python packages reside in `pkgs/top-level/python-packages.nix` and all
applications elsewhere. In case a package is used as both a library and an application,
then the package should be in `pkgs/top-level/python-packages.nix` since only those packages are made
available for all interpreter versions. The preferred location for library expressions is in
applications elsewhere. In case a package is used as both a library and an
application, then the package should be in `pkgs/top-level/python-packages.nix`
since only those packages are made available for all interpreter versions. The
preferred location for library expressions is in
`pkgs/development/python-modules`. It is important that these packages are
called from `pkgs/top-level/python-packages.nix` and not elsewhere, to guarantee
the right version of the package is built.
@ -562,6 +589,7 @@ The `buildPythonPackage` function is implemented in
@ -608,38 +636,67 @@ as the interpreter unless overridden otherwise.
##### `buildPythonPackage` parameters
All parameters from `stdenv.mkDerivation` function are still supported. The following are specific to `buildPythonPackage`:
All parameters from `stdenv.mkDerivation` function are still supported. The
following are specific to `buildPythonPackage`:
* `catchConflicts ? true`: If `true`, abort package build if a package name appears more than once in dependency tree. Default is `true`.
* `disabled` ? false: If `true`, package is not built for the particular Python interpreter version.
* `catchConflicts ? true`: If `true`, abort package build if a package name
appears more than once in dependency tree. Default is `true`.
* `disabled` ? false: If `true`, package is not built for the particular Python
interpreter version.
* `dontWrapPythonPrograms ? false`: Skip wrapping of python programs.
* `permitUserSite ? false`: Skip setting the `PYTHONNOUSERSITE` environment variable in wrapped programs.
* `installFlags ? []`: A list of strings. Arguments to be passed to `pip install`. To pass options to `python setup.py install`, use `--install-option`. E.g., `installFlags=["--install-option='--cpp_implementation'"]`.
* `format ? "setuptools"`: Format of the source. Valid options are `"setuptools"`, `"pyproject"`, `"flit"`, `"wheel"`, and `"other"`. `"setuptools"` is for when the source has a `setup.py` and `setuptools` is used to build a wheel, `flit`, in case `flit` should be used to build a wheel, and `wheel` in case a wheel is provided. Use `other` when a custom `buildPhase` and/or `installPhase` is needed.
* `makeWrapperArgs ? []`: A list of strings. Arguments to be passed to `makeWrapper`, which wraps generated binaries. By default, the arguments to `makeWrapper` set `PATH` and `PYTHONPATH` environment variables before calling the binary. Additional arguments here can allow a developer to set environment variables which will be available when the binary is run. For example, `makeWrapperArgs = ["--set FOO BAR" "--set BAZ QUX"]`.
* `namePrefix`: Prepends text to `${name}` parameter. In case of libraries, this defaults to `"python3.5-"` for Python 3.5, etc., and in case of applications to `""`.
* `pythonPath ? []`: List of packages to be added into `$PYTHONPATH`. Packages in `pythonPath` are not propagated (contrary to `propagatedBuildInputs`).
* `permitUserSite ? false`: Skip setting the `PYTHONNOUSERSITE` environment
variable in wrapped programs.
* `installFlags ? []`: A list of strings. Arguments to be passed to `pip
install`. To pass options to `python setup.py install`, use
* `namePrefix`: Prepends text to `${name}` parameter. In case of libraries, this
defaults to `"python3.5-"` for Python 3.5, etc., and in case of applications
to `""`.
* `pythonPath ? []`: List of packages to be added into `$PYTHONPATH`. Packages
in `pythonPath` are not propagated (contrary to `propagatedBuildInputs`).
* `preShellHook`: Hook to execute commands before `shellHook`.
* `postShellHook`: Hook to execute commands after `shellHook`.
* `removeBinByteCode ? true`: Remove bytecode from `/bin`. Bytecode is only created when the filenames end with `.py`.
* `removeBinByteCode ? true`: Remove bytecode from `/bin`. Bytecode is only
created when the filenames end with `.py`.
* `setupPyGlobalFlags ? []`: List of flags passed to `setup.py` command.
* `setupPyBuildFlags ? []`: List of flags passed to `setup.py build_ext` command.
The `stdenv.mkDerivation` function accepts various parameters for describing build inputs (see "Specifying dependencies"). The following are of special
interest for Python packages, either because these are primarily used, or because their behaviour is different:
* `nativeBuildInputs ? []`: Build-time only dependencies. Typically executables as well as the items listed in `setup_requires`.
* `buildInputs ? []`: Build and/or run-time dependencies that need to be be compiled for the host machine. Typically non-Python libraries which are being linked.
* `checkInputs ? []`: Dependencies needed for running the `checkPhase`. These are added to `nativeBuildInputs` when `doCheck = true`. Items listed in `tests_require` go here.
* `propagatedBuildInputs ? []`: Aside from propagating dependencies, `buildPythonPackage` also injects code into and wraps executables with the paths included in this list. Items listed in `install_requires` go here.
The `stdenv.mkDerivation` function accepts various parameters for describing
build inputs (see "Specifying dependencies"). The following are of special
interest for Python packages, either because these are primarily used, or
because their behaviour is different:
* `nativeBuildInputs ? []`: Build-time only dependencies. Typically executables
as well as the items listed in `setup_requires`.
* `buildInputs ? []`: Build and/or run-time dependencies that need to be be
compiled for the host machine. Typically non-Python libraries which are being
linked.
* `checkInputs ? []`: Dependencies needed for running the `checkPhase`. These
are added to `nativeBuildInputs` when `doCheck = true`. Items listed in
`tests_require` go here.
* `propagatedBuildInputs ? []`: Aside from propagating dependencies,
`buildPythonPackage` also injects code into and wraps executables with the
paths included in this list. Items listed in `install_requires` go here.
##### Overriding Python packages
The `buildPythonPackage` function has a `overridePythonAttrs` method that
can be used to override the package. In the following example we create an
environment where we have the `blaze` package using an older version of `pandas`.
We override first the Python interpreter and pass
`packageOverrides` which contains the overrides for packages in the package set.
The `buildPythonPackage` function has a `overridePythonAttrs` method that can be
used to override the package. In the following example we create an environment
where we have the `blaze` package using an older version of `pandas`. We
override first the Python interpreter and pass `packageOverrides` which contains
the overrides for packages in the package set.
```nix
with import <nixpkgs> {};
@ -725,15 +782,18 @@ youtube-dl = with pythonPackages; toPythonApplication youtube-dl;
#### `toPythonModule` function
In some cases, such as bindings, a package is created using
`stdenv.mkDerivation` and added as attribute in `all-packages.nix`.
The Python bindings should be made available from `python-packages.nix`.
The `toPythonModule` function takes a derivation and makes certain Python-specific modifications.
`stdenv.mkDerivation` and added as attribute in `all-packages.nix`. The Python
bindings should be made available from `python-packages.nix`. The
`toPythonModule` function takes a derivation and makes certain Python-specific
modifications.
```nix
opencv = toPythonModule (pkgs.opencv.override {
enablePython = true;
pythonPackages = self;
});
```
Do pay attention to passing in the right Python version!
#### `python.buildEnv` function
@ -741,6 +801,7 @@ Do pay attention to passing in the right Python version!
Python environments can be created using the low-level `pkgs.buildEnv` function.
This example shows how to create an environment that has the Pyramid Web Framework.
In contrast to `python.buildEnv`, `python.withPackages` does not support the more advanced options
such as `ignoreCollisions = true` or `postBuild`. If you need them, you have to use `python.buildEnv`.
In contrast to `python.buildEnv`, `python.withPackages` does not support the
more advanced options such as `ignoreCollisions = true` or `postBuild`. If you
need them, you have to use `python.buildEnv`.
Python 2 namespace packages may provide `__init__.py` that collide. In that case`python.buildEnv`
should be used with `ignoreCollisions = true`.
Python 2 namespace packages may provide `__init__.py` that collide. In that case
`python.buildEnv`should be used with `ignoreCollisions = true`.
#### Setup hooks
The following are setup hooks specifically for Python packages. Most of these are
used in `buildPythonPackage`.
The following are setup hooks specifically for Python packages. Most of these
are used in `buildPythonPackage`.
- `eggUnpackhook` to move an egg to the correct folder so it can be installed with the `eggInstallHook`
- `eggUnpackhook` to move an egg to the correct folder so it can be installed
with the `eggInstallHook`
- `eggBuildHook` to skip building for eggs.
- `eggInstallHook` to install eggs.
- `flitBuildHook` to build a wheel using `flit`.
- `pipBuildHook` to build a wheel using `pip` and PEP 517. Note a build system (e.g. `setuptools` or `flit`) should still be added as `nativeBuildInput`.
- `pipBuildHook` to build a wheel using `pip` and PEP 517. Note a build system
(e.g. `setuptools` or `flit`) should still be added as `nativeBuildInput`.
- `pipInstallHook` to install wheels.
- `pytestCheckHook` to run tests with `pytest`.
- `pythonCatchConflictsHook` to check whether a Python package is not already existing.
@ -833,8 +904,10 @@ used in `buildPythonPackage`.
- `pythonRemoveBinBytecode` to remove bytecode from the `/bin` folder.
- `setuptoolsBuildHook` to build a wheel using `setuptools`.
- `setuptoolsCheckHook` to run tests with `python setup.py test`.
- `venvShellHook` to source a Python 3 `venv` at the `venvDir` location. A `venv` is created if it does not yet exist.
- `wheelUnpackHook` to move a wheel to the correct folder so it can be installed with the `pipInstallHook`.
- `venvShellHook` to source a Python 3 `venv` at the `venvDir` location. A
`venv` is created if it does not yet exist.
- `wheelUnpackHook` to move a wheel to the correct folder so it can be installed
@ -872,19 +945,22 @@ Note: There is a boolean value `lib.inNixShell` set to `true` if nix-shell is in
Packages inside nixpkgs are written by hand. However many tools exist in
community to help save time. No tool is preferred at the moment.
- [pypi2nix](https://github.com/nix-community/pypi2nix): Generate Nix expressions for your Python project. Note that [sharing derivations from pypi2nix with nixpkgs is possible but not encouraged](https://github.com/nix-community/pypi2nix/issues/222#issuecomment-443497376).
When the environment variable `DETERMINISTIC_BUILD` is set, all bytecode will
have timestamp 1. The `buildPythonPackage` function sets `DETERMINISTIC_BUILD=1`
and [PYTHONHASHSEED=0](https://docs.python.org/3.5/using/cmdline.html#envvar-PYTHONHASHSEED).
Both are also exported in `nix-shell`.
@ -899,9 +975,10 @@ example of such a situation is when `py.test` is used.
#### Common issues
- Non-working tests can often be deselected. By default `buildPythonPackage` runs `python setup.py test`.
Most python modules follows the standard test protocol where the pytest runner can be used instead.
`py.test` supports a `-k` parameter to ignore test methods or classes:
* Non-working tests can often be deselected. By default `buildPythonPackage`
runs `python setup.py test`. Most python modules follows the standard test
protocol where the pytest runner can be used instead. `py.test` supports a
`-k` parameter to ignore test methods or classes:
```nix
buildPythonPackage {
@ -913,7 +990,8 @@ example of such a situation is when `py.test` is used.
'';
}
```
- Tests that attempt to access `$HOME` can be fixed by using the following work-around before running tests (e.g. `preCheck`): `export HOME=$(mktemp -d)`
* Tests that attempt to access `$HOME` can be fixed by using the following
work-around before running tests (e.g. `preCheck`): `export HOME=$(mktemp -d)`
## FAQ
@ -925,8 +1003,9 @@ should also be done when packaging `A`.
### How to override a Python package?
We can override the interpreter and pass `packageOverrides`.
In the following example we rename the `pandas` package and build it.
We can override the interpreter and pass `packageOverrides`. In the following
example we rename the `pandas` package and build it.
```nix
with import <nixpkgs> {};
@ -939,14 +1018,16 @@ with import <nixpkgs> {};
in python.withPackages(ps: [ps.pandas])).env
```
Using `nix-build` on this expression will build an environment that contains the
package `pandas` but with the new name `foo`.
All packages in the package set will use the renamed package.
A typical use case is to switch to another version of a certain package.
For example, in the Nixpkgs repository we have multiple versions of `django` and `scipy`.
In the following example we use a different version of `scipy` and create an environment that uses it.
All packages in the Python package set will now use the updated `scipy` version.
All packages in the package set will use the renamed package. A typical use case
is to switch to another version of a certain package. For example, in the
Nixpkgs repository we have multiple versions of `django` and `scipy`. In the
following example we use a different version of `scipy` and create an
environment that uses it. All packages in the Python package set will now use
the updated `scipy` version.
```nix
with import <nixpkgs> {};
@ -958,10 +1039,13 @@ with import <nixpkgs> {};
in (pkgs.python35.override {inherit packageOverrides;}).withPackages (ps: [ps.blaze])
).env
```
The requested package `blaze` depends on `pandas` which itself depends on `scipy`.
If you want the whole of Nixpkgs to use your modifications, then you can use `overlays`
as explained in this manual. In the following example we build a `inkscape` using a different version of `numpy`.
If you want the whole of Nixpkgs to use your modifications, then you can use
`overlays` as explained in this manual. In the following example we build a
`inkscape` using a different version of `numpy`.
```nix
let
pkgs = import <nixpkgs> {};
@ -982,19 +1066,28 @@ Executing `python setup.py bdist_wheel` in a `nix-shell `fails with
ValueError: ZIP does not support timestamps before 1980
```
This is because files from the Nix store (which have a timestamp of the UNIX epoch of January 1, 1970) are included in the .ZIP, but .ZIP archives follow the DOS convention of counting timestamps from 1980.
This is because files from the Nix store (which have a timestamp of the UNIX
epoch of January 1, 1970) are included in the .ZIP, but .ZIP archives follow the
DOS convention of counting timestamps from 1980.
The command `bdist_wheel` reads the `SOURCE_DATE_EPOCH` environment variable, which `nix-shell` sets to 1. Unsetting this variable or giving it a value corresponding to 1980 or later enables building wheels.
The command `bdist_wheel` reads the `SOURCE_DATE_EPOCH` environment variable,
which `nix-shell` sets to 1. Unsetting this variable or giving it a value
corresponding to 1980 or later enables building wheels.
### How to consume python modules using pip in a virtual environment like I am used to on other Operating Systems?
While this approach is not very idiomatic from Nix perspective, it can still be useful when dealing with pre-existing
projects or in situations where it's not feasible or desired to write derivations for all required dependencies.
While this approach is not very idiomatic from Nix perspective, it can still be
useful when dealing with pre-existing projects or in situations where it's not
feasible or desired to write derivations for all required dependencies.
This is an example of a `default.nix` for a `nix-shell`, which allows to consume a virtual environment created by `venv`,
and install python modules through `pip` the traditional way.
This is an example of a `default.nix` for a `nix-shell`, which allows to consume
a virtual environment created by `venv`, and install python modules through
`pip` the traditional way.
Create this `default.nix` file, together with a `requirements.txt` and simply execute `nix-shell`.
Create this `default.nix` file, together with a `requirements.txt` and simply
execute `nix-shell`.
```nix
with import <nixpkgs> { };
@ -1082,8 +1183,9 @@ in pkgs.mkShell rec {
}
```
In case the supplied venvShellHook is insufficient, or when python 2 support is needed,
you can define your own shell hook and adapt to your needs like in the following example:
In case the supplied venvShellHook is insufficient, or when python 2 support is
needed, you can define your own shell hook and adapt to your needs like in the
following example:
```nix
with import <nixpkgs> { };
@ -1152,11 +1254,11 @@ If you need to change a package's attribute(s) from `configuration.nix` you coul
```
`pythonPackages.zerobin` is now globally overridden. All packages and also the
`zerobin` NixOS service use the new definition.
Note that `python-super` refers to the old package set and `python-self`
to the new, overridden version.
`zerobin` NixOS service use the new definition. Note that `python-super` refers
to the old package set and `python-self` to the new, overridden version.
To modify only a Python package set instead of a whole Python derivation, use this snippet:
To modify only a Python package set instead of a whole Python derivation, use
this snippet:
```nix
myPythonPackages = pythonPackages.override {
@ -1188,11 +1290,12 @@ self: super: {
### How to use Intel's MKL with numpy and scipy?
A `site.cfg` is created that configures BLAS based on the `blas` parameter
of the `numpy` derivation. By passing in `mkl`, `numpy` and packages depending
on `numpy` will be built with `mkl`.
A `site.cfg` is created that configures BLAS based on the `blas` parameter of
the `numpy` derivation. By passing in `mkl`, `numpy` and packages depending on
`numpy` will be built with `mkl`.
The following is an overlay that configures `numpy` to use `mkl`:
```nix
self: super: {
python37 = super.python37.override {
@ -1228,10 +1331,21 @@ In a `setup.py` or `setup.cfg` it is common to declare dependencies:
Following rules are desired to be respected:
* Python libraries are called from `python-packages.nix` and packaged with `buildPythonPackage`. The expression of a library should be in `pkgs/development/python-modules/<name>/default.nix`. Libraries in `pkgs/top-level/python-packages.nix` are sorted quasi-alphabetically to avoid merge conflicts.
* Python applications live outside of `python-packages.nix` and are packaged with `buildPythonApplication`.
* Python libraries are called from `python-packages.nix` and packaged with
`buildPythonPackage`. The expression of a library should be in
`pkgs/development/python-modules/<name>/default.nix`. Libraries in
`pkgs/top-level/python-packages.nix` are sorted quasi-alphabetically to avoid
merge conflicts.
* Python applications live outside of `python-packages.nix` and are packaged
with `buildPythonApplication`.
* Make sure libraries build for all Python interpreters.
* By default we enable tests. Make sure the tests are found and, in the case of libraries, are passing for all interpreters. If certain tests fail they can be disabled individually. Try to avoid disabling the tests altogether. In any case, when you disable tests, leave a comment explaining why.
* Commit names of Python libraries should reflect that they are Python libraries, so write for example `pythonPackages.numpy: 1.11 -> 1.12`.
* Attribute names in `python-packages.nix` should be normalized according to [PEP 0503](https://www.python.org/dev/peps/pep-0503/#normalized-names).
This means that characters should be converted to lowercase and `.` and `_` should be replaced by a single `-` (foo-bar-baz instead of Foo__Bar.baz )
* By default we enable tests. Make sure the tests are found and, in the case of
libraries, are passing for all interpreters. If certain tests fail they can be
disabled individually. Try to avoid disabling the tests altogether. In any
case, when you disable tests, leave a comment explaining why.
* Commit names of Python libraries should reflect that they are Python
libraries, so write for example `pythonPackages.numpy: 1.11 -> 1.12`.
* Attribute names in `python-packages.nix` should be normalized according to
[PEP 0503](https://www.python.org/dev/peps/pep-0503/#normalized-names). This
means that characters should be converted to lowercase and `.` and `_` should
be replaced by a single `-` (foo-bar-baz instead of Foo__Bar.baz )
Benjamin Hipple
4 years ago