Based on some feedback in #87094 and discussion with @fridh, this re-organizes
the onboarding tutorial in the Nixpkgs manual's python section, so that we start
with the simplest, most ad-hoc examples and work our way up. This progresses
from:
1. How to create an temporary python env at the cmdline, then
2. How to create a specific python env for a single script, then
3. How to create a specific python env for a project in a shell.nix, then
4. How to install a specific python env globally on the system or in a user profile.
Additionally, I've tried to standardize on some of the "best practice" ways of
doing things:
1. Instead of saying that this command style is "supported but strongly not
discouraged", I've just deleted it to avoid confusion.
Bad: nix-shell -p python38Packages.numpy python38Packages.toolz
Good: nix-shell -p 'python38.withPackages(ps: with ps; [ numpy toolz ])'
2. In the portion where we show how to add stuff to the user's
`XDG_CONFIG_HOME`, use overlays instead of `config.nix`. The former can do
everything the latter can do, but is also much more generic and powerful,
because it can compose with other files, compose with other envs, compose
with overlays that do things like swap whether tensorflow and pytorch are
built openblas/mkl/cuda stacks, and so on. The user is eventually going to
see the overlay, so to avoid confusion let's standardize on it.
wip/yesman
Benjamin Hipple4 years agocommitted byFrederik Rietdijk
For the sake of completeness, here's how to install the environment system-wide
on NixOS.
import numpy
```nix
{ # ...
print(numpy.__version__)
environment.systemPackages = with pkgs; [
(python38.withPackages(ps: with ps; [ numpy toolz ]))
];
}
```
### Developing with Python
Now that you know how to get a working Python environment with Nix, it is time
to go forward and start actually developing with Python. We will first have a
look at how Python packages are packaged on Nix. Then, we will look at how you
can use development mode with your code.
Above, we were mostly just focused on use cases and what to do to get started
creating working Python environments in nix.
#### Packaging a library
Now that you know the basics to be up and running, it is time to take a step
back and take a deeper look at at how Python packages are packaged on Nix. Then,
we will look at how you can use development mode with your code.
#### Python library packages in Nixpkgs
With Nix all packages are built by functions. The main function in Nix for
building Python libraries is `buildPythonPackage`. Let's see how we can build the
@ -262,6 +379,7 @@ An expression for `toolz` can be found in the Nixpkgs repository. As explained
in the introduction of this Python section, a derivation of `toolz` is available
for each interpreter version, e.g. `python38.pkgs.toolz` refers to the `toolz`
derivation corresponding to the CPython 3.8 interpreter.
The above example works when you're directly working on
`pkgs/top-level/python-packages.nix` in the Nixpkgs repository. Often though,
you will want to test a Nix expression outside of the Nixpkgs tree.
@ -312,7 +430,7 @@ Our example, `toolz`, does not have any dependencies on other Python packages or
system libraries. According to the manual, `buildPythonPackage` uses the
arguments `buildInputs` and `propagatedBuildInputs` to specify dependencies. If
something is exclusively a build-time dependency, then the dependency should be
included as a `buildInput`, but if it is (also) a runtime dependency, then it
included in `buildInputs`, but if it is (also) a runtime dependency, then it
should be added to `propagatedBuildInputs`. Test dependencies are considered
build-time dependencies and passed to `checkInputs`.
@ -423,10 +541,11 @@ Note also the line `doCheck = false;`, we explicitly disabled running the test-s
#### Develop local package
As a Python developer you're likely aware of [development mode](http://setuptools.readthedocs.io/en/latest/setuptools.html#development-mode) (`python setup.py develop`);
instead of installing the package this command creates a special link to the project code.
That way, you can run updated code without having to reinstall after each and every change you make.
Development mode is also available. Let's see how you can use it.
As a Python developer you're likely aware of [development mode](http://setuptools.readthedocs.io/en/latest/setuptools.html#development-mode)
(`python setup.py develop`); instead of installing the package this command
creates a special link to the project code. That way, you can run updated code
without having to reinstall after each and every change you make. Development
mode is also available. Let's see how you can use it.
In the previous Nix expression the source was fetched from an url. We can also
refer to a local source instead using `src = ./path/to/source/tree;`
@ -455,7 +574,6 @@ buildPythonPackage rec {
It is important to note that due to how development mode is implemented on Nix
it is not possible to have multiple packages simultaneously in development mode.
### Organising your packages
So far we discussed how you can use Python on Nix, and how you can develop with
@ -481,11 +599,11 @@ We first create a function that builds `toolz` in `~/path/to/toolz/release.nix`
We now call this function using `callPackage` in the definition of our environment
It takes an argument `buildPythonPackage`. We now call this function using
`callPackage` in the definition of our environment
```nix
with import <nixpkgs> {};
@ -548,7 +666,7 @@ Each interpreter has the following attributes:
- `buildEnv`. Function to build python interpreter environments with extra packages bundled together. See section *python.buildEnv function* for usage and documentation.
- `withPackages`. Simpler interface to `buildEnv`. See section *python.withPackages function* for usage and documentation.
- `sitePackages`. Alias for `lib/${libPrefix}/site-packages`.
- `executable`. Name of the interpreter executable, e.g. `python3.7`.
- `executable`. Name of the interpreter executable, e.g. `python3.8`.
- `pkgs`. Set of Python packages for that specific interpreter. The package set can be modified by overriding the interpreter and passing `packageOverrides`.
### Building packages and applications
@ -643,7 +761,7 @@ following are specific to `buildPythonPackage`:
appears more than once in dependency tree. Default is `true`.
* `disabled` ? false: If `true`, package is not built for the particular Python
interpreter version.
* `dontWrapPythonPrograms ? false`: Skip wrapping of python programs.
* `dontWrapPythonPrograms ? false`: Skip wrapping of Python programs.
* `permitUserSite ? false`: Skip setting the `PYTHONNOUSERSITE` environment
variable in wrapped programs.
* `installFlags ? []`: A list of strings. Arguments to be passed to `pip
@ -730,7 +848,7 @@ Another difference is that `buildPythonPackage` by default prefixes the names of
the packages with the version of the interpreter. Because this is irrelevant for
applications, the prefix is omitted.
When packaging a python application with `buildPythonApplication`, it should be
When packaging a Python application with `buildPythonApplication`, it should be
called with `callPackage` and passed `python` or `pythonPackages` (possibly
specifying an interpreter version), like this:
@ -761,7 +879,7 @@ luigi = callPackage ../applications/networking/cluster/luigi { };
```
Since the package is an application, a consumer doesn't need to care about
python versions or modules, which is why they don't go in `pythonPackages`.
Python versions or modules, which is why they don't go in `pythonPackages`.
#### `toPythonApplication` function
@ -875,7 +993,7 @@ thus be also written like this: