- [ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside [nixos/tests](https://github.com/NixOS/nixpkgs/blob/master/nixos/tests))
- [ ] Tested compilation of all pkgs that depend on this change using `nix-shell -p nixpkgs-review --run "nixpkgs-review wip"`
- [ ] Tested execution of all binary files (usually in `./result/bin/`)
Bot-based backport to `${target_branch}`, triggered by a label in#${pull_number}.
*[]Before merging, ensure that this backport complies with the [Criteria for Backporting](https://github.com/NixOS/nixpkgs/blob/master/CONTRIBUTING.md#criteria-for-backporting-changes).
*Even as a non-commiter, if you find that it does not comply, leave a comment.
@ -59,8 +59,19 @@ Follow these steps to backport a change into a release branch in compliance with
5. Push to GitHub and open a backport pull request. Make sure to select the release branch (e.g. `release-20.09`) as the target branch of the pull request, and link to the pull request in which the original change was comitted to `master`. The pull request title should be the commit title with the release version as prefix, e.g. `[20.09]`.
6. When the backport pull request is merged and you have the necessary privileges you can also replace the label `9.needs: port to stable` with `8.has: port to stable` on the original pull request. This way maintainers can keep track of missing backports easier.
## Criteria for Backporting changes
Anything that does not cause user or downstream dependency regressions can be backported. This includes:
- New Packages / Modules
- Security / Patch updates
- Version updates which include new functionality (but no breaking changes)
- Services which require a client to be up-to-date regardless. (E.g. `spotify`, `steam`, or `discord`)
- Security critical applications (E.g. `firefox`)
## Generating 21.11 Release Notes
(This section also applies to backporting 21.05 release notes: substitute "rl-2111" for "rl-2105".)
Documentation in nixpkgs is transitioning to a markdown-centric workflow. Release notes now require a translation step to convert from markdown to a compatible docbook document.
@ -20,59 +20,58 @@ The main difference between `fetchurl` and `fetchzip` is in how they store the c
`fetchpatch` works very similarly to `fetchurl` with the same arguments expected. It expects patch files as a source and performs normalization on them before computing the checksum. For example it will remove comments or other unstable parts that are sometimes added by version control systems and can change over time.
Other fetcher functions allow you to add source code directly from a VCS such as subversion or git. These are mostly straightforward nambes based on the name of the command used with the VCS system. Because they give you a working repository, they act most like `fetchzip`.
## `fetchsvn`
## `fetchsvn` {#fetchsvn}
Used with Subversion. Expects `url` to a Subversion directory, `rev`, and `sha256`.
## `fetchgit`
## `fetchgit` {#fetchgit}
Used with Git. Expects `url` to a Git repo, `rev`, and `sha256`. `rev` in this case can be full the git commit id (SHA1 hash) or a tag name like `refs/tags/v1.0`.
Additionally the following optional arguments can be given: `fetchSubmodules = true` makes `fetchgit` also fetch the submodules of a repository. If `deepClone` is set to true, the entire repository is cloned as opposing to just creating a shallow clone. `deepClone = true` also implies `leaveDotGit = true` which means that the `.git` directory of the clone won't be removed after checkout.
## `fetchfossil`
## `fetchfossil` {#fetchfossil}
Used with Fossil. Expects `url` to a Fossil archive, `rev`, and `sha256`.
## `fetchcvs`
## `fetchcvs` {#fetchcvs}
Used with CVS. Expects `cvsRoot`, `tag`, and `sha256`.
## `fetchhg`
## `fetchhg` {#fetchhg}
Used with Mercurial. Expects `url`, `rev`, and `sha256`.
A number of fetcher functions wrap part of `fetchurl` and `fetchzip`. They are mainly convenience functions intended for commonly used destinations of source code in Nixpkgs. These wrapper fetchers are listed below.
## `fetchFromGitHub`
## `fetchFromGitHub` {#fetchfromgithub}
`fetchFromGitHub` expects four arguments. `owner` is a string corresponding to the GitHub user or organization that controls this repository. `repo` corresponds to the name of the software repository. These are located at the top of every GitHub HTML page as `owner`/`repo`. `rev` corresponds to the Git commit hash or tag (e.g `v1.0`) that will be downloaded from Git. Finally, `sha256` corresponds to the hash of the extracted directory. Again, other hash algorithms are also available but `sha256` is currently preferred.
`fetchFromGitHub` uses `fetchzip` to download the source archive generated by GitHub for the specified revision. If `leaveDotGit`, `deepClone` or `fetchSubmodules` are set to `true`, `fetchFromGitHub` will use `fetchgit` instead. Refer to its section for documentation of these options.
## `fetchFromGitLab`
## `fetchFromGitLab` {#fetchfromgitlab}
This is used with GitLab repositories. The arguments expected are very similar to fetchFromGitHub above.
## `fetchFromGitiles`
## `fetchFromGitiles` {#fetchfromgitiles}
This is used with Gitiles repositories. The arguments expected are similar to fetchgit.
## `fetchFromBitbucket`
## `fetchFromBitbucket` {#fetchfrombitbucket}
This is used with BitBucket repositories. The arguments expected are very similar to fetchFromGitHub above.
## `fetchFromSavannah`
## `fetchFromSavannah` {#fetchfromsavannah}
This is used with Savannah repositories. The arguments expected are very similar to fetchFromGitHub above.
## `fetchFromRepoOrCz`
## `fetchFromRepoOrCz` {#fetchfromrepoorcz}
This is used with repo.or.cz repositories. The arguments expected are very similar to fetchFromGitHub above.
## `fetchFromSourcehut`
## `fetchFromSourcehut` {#fetchfromsourcehut}
This is used with sourcehut repositories. The arguments expected are very similar to fetchFromGitHub above. Don't forget the tilde (~) in front of the user name!
`pkgs.appimageTools` is a set of functions for extracting and wrapping [AppImage](https://appimage.org/) files. They are meant to be used if traditional packaging from source is infeasible, or it would take too long. To quickly run an AppImage file, `pkgs.appimage-run` can be used as well.
::: warning
::: {.warning}
The `appimageTools` API is unstable and may be subject to backwards-incompatible changes in the future.
`pkgs.dockerTools` is a set of functions for creating and manipulating Docker images according to the [Docker Image Specification v1.2.0](https://github.com/moby/moby/blob/master/image/spec/v1.2.md#docker-image-specification-v120). Docker itself is not used to perform any of the operations done by these functions.
`pkgs.dockerTools` is a set of functions for creating and manipulating Docker images according to the [Docker Image Specification v1.2.0](https://github.com/moby/moby/blob/master/image/spec/v1.2.md#docker-image-specification-v120). Docker itself is not used to perform any of the operations done by these functions.
## buildImage {#ssec-pkgs-dockerTools-buildImage}
@ -52,7 +52,7 @@ The above example will build a Docker image `redis/latest` from the given base i
> **_NOTE:_** Using this parameter requires the `kvm` device to be available.
- `config` is used to specify the configuration of the containers that will be started off the built image in Docker. The available options are listed in the [Docker Image Specification v1.2.0](https://github.com/moby/moby/blob/master/image/spec/v1.2.md#image-json-field-descriptions).
- `config` is used to specify the configuration of the containers that will be started off the built image in Docker. The available options are listed in the [Docker Image Specification v1.2.0](https://github.com/moby/moby/blob/master/image/spec/v1.2.md#image-json-field-descriptions).
After the new layer has been created, its closure (to which `contents`, `config` and `runAsRoot` contribute) will be copied in the layer itself. Only new dependencies that are not already in the existing layers will be copied.
@ -14,7 +14,7 @@ Currently, `makeSnap` does not support creating GUI stubs.
The following expression packages GNU Hello as a Snapcraft snap.
```{#ex-snapTools-buildSnap-hello .nix}
```{#ex-snapTools-buildSnap-hello .nix}
let
inherit (import <nixpkgs> { }) snapTools hello;
in snapTools.makeSnap {
@ -35,7 +35,7 @@ in snapTools.makeSnap {
Graphical programs require many more integrations with the host. This example uses Firefox as an example, because it is one of the most complicated programs we could package.
## Build wrapped Firefox with extensions and policies
## Build wrapped Firefox with extensions and policies {#build-wrapped-firefox-with-extensions-and-policies}
The `wrapFirefox` function allows to pass policies, preferences and extension that are available to firefox. With the help of `fetchFirefoxAddon` this allows build a firefox version that already comes with addons pre-installed:
@ -4,11 +4,11 @@ OpenGL support varies depending on which hardware is used and which drivers are
Broadly, we support both GL vendors: Mesa and NVIDIA.
## NixOS Desktop
## NixOS Desktop {#nixos-desktop}
The NixOS desktop or other non-headless configurations are the primary target for OpenGL libraries and applications. The current solution for discovering which drivers are available is based on [libglvnd](https://gitlab.freedesktop.org/glvnd/libglvnd). `libglvnd` performs "vendor-neutral dispatch", trying a variety of techniques to find the system's GL implementation. In practice, this will be either via standard GLX for X11 users or EGL for Wayland users, and supporting either NVIDIA or Mesa extensions.
## Nix on GNU/Linux
## Nix on GNU/Linux {#nix-on-gnulinux}
If you are using a non-NixOS GNU/Linux/X11 desktop with free software video drivers, consider launching OpenGL-dependent programs from Nixpkgs with Nixpkgs versions of `libglvnd` and `mesa.drivers` in `LD_LIBRARY_PATH`. For Mesa drivers, the Linux kernel version doesn't have to match nixpkgs.
The Nix expressions for the X.org packages reside in `pkgs/servers/x11/xorg/default.nix`. This file is automatically generated from lists of tarballs in an X.org release. As such it should not be modified directly; rather, you should modify the lists, the generator script or the file `pkgs/servers/x11/xorg/overrides.nix`, in which you can override or add to the derivations produced by the generator.
## Katamari Tarballs
## Katamari Tarballs {#katamari-tarballs}
X.org upstream releases used to include [katamari](https://en.wiktionary.org/wiki/%E3%81%8B%E3%81%9F%E3%81%BE%E3%82%8A) releases, which included a holistic recommended version for each tarball, up until 7.7. To create a list of tarballs in a katamari release:
The upstream release process for [X11R7.8](https://x.org/wiki/Releases/7.8/) does not include a planned katamari. Instead, each component of X.org is released as its own tarball. We maintain `pkgs/servers/x11/xorg/tarballs.list` as a list of tarballs for each individual package. This list includes X.org core libraries and protocol descriptions, extra newer X11 interface libraries, like `xorg.libxcb`, and classic utilities which are largely unused but still available if needed, like `xorg.imake`.
For each of the tarballs in the `.list` files, the script downloads it, unpacks it, and searches its `configure.ac` and `*.pc.in` files for dependencies. This information is used to generate `default.nix`. The generator caches downloaded tarballs between runs. Pay close attention to the `NOT FOUND: $NAME` messages at the end of the run, since they may indicate missing dependencies. (Some might be optional dependencies, however.)
## Overriding the Generator
## Overriding the Generator {#overriding-the-generator}
If the expression for a package requires derivation attributes that the generator cannot figure out automatically (say, `patches` or a `postInstall` hook), you should modify `pkgs/servers/x11/xorg/overrides.nix`.
@ -37,7 +37,7 @@ This works just like `runCommand`. The only difference is that it also provides
Variant of `runCommand` that forces the derivation to be built locally, it is not substituted. This is intended for very cheap commands (<1sexecutiontime).Itsavesonthenetworkroundripandcanspeedupabuild.
::: note
::: {.note}
This sets [`allowSubstitutes` to `false`](https://nixos.org/nix/manual/#adv-attr-allowSubstitutes), so only use `runCommandLocal` if you are certain the user will always have a builder for the `system` of the derivation. This should be true for most trivial use cases (e.g. just copying some files to a different location or adding symlinks), because there the `system` is usually the same as `builtins.currentSystem`.
- Do not use tab characters, i.e. configure your editor to use soft tabs. For instance, use `(setq-default indent-tabs-mode nil)` in Emacs. Everybody has different tab settings so it’s asking for trouble.
- Use `lowerCamelCase` for variable names, not `UpperCamelCase`. Note, this rule does not apply to package attribute names, which instead follow the rules in <xreflinkend="sec-package-naming"/>.
- Use `lowerCamelCase` for variable names, not `UpperCamelCase`. Note, this rule does not apply to package attribute names, which instead follow the rules in [](#sec-package-naming).
- Function calls with attribute set arguments are written as
@ -209,7 +209,7 @@ There are a few naming guidelines:
- Dashes in the package name _should_ be preserved in new variable names, rather than converted to underscores or camel cased — e.g., `http-parser` instead of `http_parser` or `httpParser`. The hyphenated style is preferred in all three package names.
- If there are multiple versions of a package, this _should_ be reflected in the variable names in `all-packages.nix`, e.g. `json-c-0-9` and `json-c-0-11`. If there is an obvious “default” version, make an attribute like `json-c = json-c-0-9;`. See also <xreflinkend="sec-versioning"/>
- If there are multiple versions of a package, this _should_ be reflected in the variable names in `all-packages.nix`, e.g. `json-c-0-9` and `json-c-0-11`. If there is an obvious “default” version, make an attribute like `json-c = json-c-0-9;`. See also [](#sec-versioning)
## File naming and organisation {#sec-organisation}
@ -462,9 +462,9 @@ Preferred source hash type is sha256. There are several ways to get it.
For package updates it is enough to change one symbol to make hash fake. For new packages, you can use `lib.fakeSha256`, `lib.fakeSha512` or any other fake hash.
This is last resort method when reconstructing source URL is non-trivial and `nix-prefetch-url -A` isn't applicable (for example, [one of `kodi` dependencies](https://github.com/NixOS/nixpkgs/blob/d2ab091dd308b99e4912b805a5eb088dd536adb9/pkgs/applications/video/kodi/default.nix#L73")). The easiest way then would be replace hash with a fake one and rebuild. Nix build will fail and error message will contain desired hash.
This is last resort method when reconstructing source URL is non-trivial and `nix-prefetch-url -A` isn’t applicable (for example, [one of `kodi` dependencies](https://github.com/NixOS/nixpkgs/blob/d2ab091dd308b99e4912b805a5eb088dd536adb9/pkgs/applications/video/kodi/default.nix#L73)). The easiest way then would be replace hash with a fake one and rebuild. Nix build will fail and error message will contain desired hash.
::: warning
::: {.warning}
This method has security problems. Check below for details.
# Contributing to this documentation {#chap-contributing}
The DocBook sources of the Nixpkgs manual are in the [doc](https://github.com/NixOS/nixpkgs/tree/master/doc) subdirectory of the Nixpkgs repository.
The sources of the Nixpkgs manual are in the [doc](https://github.com/NixOS/nixpkgs/tree/master/doc) subdirectory of the Nixpkgs repository. The manual is still partially written in DocBook but it is progressively being converted to [Markdown](#sec-contributing-markup).
You can quickly check your edits with `make`:
@ -22,3 +22,78 @@ $ nix-shell
```
If the build succeeds, the manual will be in `./result/share/doc/nixpkgs/manual.html`.
## Syntax {#sec-contributing-markup}
As per [RFC 0072](https://github.com/NixOS/rfcs/pull/72), all new documentation content should be written in [CommonMark](https://commonmark.org/) Markdown dialect.
Additionally, the following syntax extensions are currently used:
- []{#ssec-contributing-markup-anchors}
Explicitly defined **anchors** on headings, to allow linking to sections. These should be always used, to ensure the anchors can be linked even when the heading text changes, and to prevent conflicts between [automatically assigned identifiers](https://github.com/jgm/commonmark-hs/blob/master/commonmark-extensions/test/auto_identifiers.md).
It uses the widely compatible [header attributes](https://github.com/jgm/commonmark-hs/blob/master/commonmark-extensions/test/attributes.md) syntax:
```markdown
## Syntax {#sec-contributing-markup}
```
- []{#ssec-contributing-markup-anchors-inline}
**Inline anchors**, which allow linking arbitrary place in the text (e.g. individual list items, sentences…).
They are defined using a hybrid of the link syntax with the attributes syntax known from headings, called [bracketed spans](https://github.com/jgm/commonmark-hs/blob/master/commonmark-extensions/test/bracketed_spans.md):
```markdown
- []{#ssec-gnome-hooks-glib} `glib` setup hook will populate `GSETTINGS_SCHEMAS_PATH` and then `wrapGAppsHook` will prepend it to `XDG_DATA_DIRS`.
```
- []{#ssec-contributing-markup-automatic-links}
If you **omit a link text** for a link pointing to a section, the text will be substituted automatically. For example, `[](#chap-contributing)` will result in [](#chap-contributing).
This syntax is taken from [MyST](https://myst-parser.readthedocs.io/en/latest/using/syntax.html#targets-and-cross-referencing).
- []{#ssec-contributing-markup-admonitions}
**Admonitions**, set off from the text to bring attention to something.
It uses pandoc’s [fenced `div`s syntax](https://github.com/jgm/commonmark-hs/blob/master/commonmark-extensions/test/fenced_divs.md):
[**Definition lists**](https://github.com/jgm/commonmark-hs/blob/master/commonmark-extensions/test/definition_lists.md), for defining a group of terms:
```markdown
pear
: green or yellow bulbous fruit
watermelon
: green fruit with red flesh
```
which renders as
> pear
> : green or yellow bulbous fruit
>
> watermelon
> : green fruit with red flesh
For contributing to the legacy parts, please see [DocBook: The Definitive Guide](https://tdg.docbook.org/) or the [DocBook rocks! primer](https://web.archive.org/web/20200816233747/https://docbook.rocks/).
2. Find a good place in the Nixpkgs tree to add the Nix expression for your package. For instance, a library package typically goes into `pkgs/development/libraries/pkgname`, while a web browser goes into `pkgs/applications/networking/browsers/pkgname`. See <xreflinkend="sec-organisation"/> for some hints on the tree organisation. Create a directory for your package, e.g.
2. Find a good place in the Nixpkgs tree to add the Nix expression for your package. For instance, a library package typically goes into `pkgs/development/libraries/pkgname`, while a web browser goes into `pkgs/applications/networking/browsers/pkgname`. See [](#sec-organisation) for some hints on the tree organisation. Create a directory for your package, e.g.
The following section is a draft, and the policy for reviewing is still being discussed in issues such as [#11166](https://github.com/NixOS/nixpkgs/issues/11166) and [#20836](https://github.com/NixOS/nixpkgs/issues/20836).
:::
@ -35,15 +35,18 @@ Reviewing process:
- Building the package locally.
- pull requests are often targeted to the master or staging branch, and building the pull request locally when it is submitted can trigger many source builds.
- It is possible to rebase the changes on nixos-unstable or nixpkgs-unstable for easier review by running the following commands from a nixpkgs clone.
```ShellSession
$ git fetch origin nixos-unstable
$ git fetch origin pull/PRNUMBER/head
$ git rebase --onto nixos-unstable BASEBRANCH FETCH_HEAD
```
- The first command fetches the nixos-unstable branch.
- The second command fetches the pull request changes, `PRNUMBER` is the number at the end of the pull request title and `BASEBRANCH` the base branch of the pull request.
- The third command rebases the pull request changes to the nixos-unstable branch.
- The [nixpkgs-review](https://github.com/Mic92/nixpkgs-review) tool can be used to review a pull request content in a single command. `PRNUMBER` should be replaced by the number at the end of the pull request title. You can also provide the full github pull request url.
@ -71,6 +71,7 @@ Security fixes are submitted in the same way as other changes and thus the same
- If a new version fixing the vulnerability has been released, update the package;
- If the security fix comes in the form of a patch and a CVE is available, then add the patch to the Nixpkgs tree, and apply it to the package.
The name of the patch should be the CVE identifier, so e.g. `CVE-2019-13636.patch`; If a patch is fetched the name needs to be set as well, e.g.:
```nix
(fetchpatch {
name = "CVE-2019-11068.patch";
@ -89,7 +90,7 @@ There is currently no policy when to remove a package.
Before removing a package, one should try to find a new maintainer or fix smaller issues first.
### Steps to remove a package from Nixpkgs
### Steps to remove a package from Nixpkgs {#steps-to-remove-a-package-from-nixpkgs}
We use jbidwatcher as an example for a discontinued project here.
@ -100,6 +101,7 @@ We use jbidwatcher as an example for a discontinued project here.
1. Add an alias for the package name in `pkgs/top-level/aliases.nix` (There is also `pkgs/misc/vim-plugins/aliases.nix`. Package sets typically do not have aliases, so we can't add them there.)
For example in this case:
```
jbidwatcher = throw "jbidwatcher was discontinued in march 2021"; # added 2021-03-15
```
@ -191,7 +193,7 @@ It’s important to test any executables generated by a build when you change or
The last checkbox is fits [CONTRIBUTING.md](https://github.com/NixOS/nixpkgs/blob/master/.github/CONTRIBUTING.md). The contributing document has detailed information on standards the Nix community has for commit messages, reviews, licensing of contributions you make to the project, etc\... Everyone should read and understand the standards the community has for contributing before submitting a pull request.
The last checkbox is fits [CONTRIBUTING.md](https://github.com/NixOS/nixpkgs/blob/master/CONTRIBUTING.md). The contributing document has detailed information on standards the Nix community has for commit messages, reviews, licensing of contributions you make to the project, etc\... Everyone should read and understand the standards the community has for contributing before submitting a pull request.
@ -267,3 +269,14 @@ Other examples of reasons are:
- Previously the build would fail due to, e.g., `getaddrinfo` not being defined
- The previous download links were all broken
- Crash when starting on some X11 systems
#### Acceptable backport criteria
The stable branch does have some changes which cannot be backported. Most notable are breaking changes. The desire is to have stable users be uninterrupted when updating packages.
However, many changes are able to be backported, including:
- New Packages / Modules
- Security / Patch updates
- Version updates which include new functionality (but no breaking changes)
- Services which require a client to be up-to-date regardless. (E.g. `spotify`, `steam`, or `discord`)
Nix is a unityped, dynamic language, this means every value can potentially appear anywhere. Since it is also non-strict, evaluation order and what ultimately is evaluated might surprise you. Therefore it is important to be able to debug nix expressions.
In the `lib/debug.nix` file you will find a number of functions that help (pretty-)printing values while evaluation is running. You can even specify how deep these values should be printed recursively, and transform them on the fly. Please consult the docstrings in `lib/debug.nix` for usage information.
Nix is a unityped, dynamic language, this means every value can potentially appear anywhere. Since it is also non-strict, evaluation order and what ultimately is evaluated might surprise you. Therefore it is important to be able to debug nix expressions.
</para>
<para>
In the <literal>lib/debug.nix</literal> file you will find a number of functions that help (pretty-)printing values while evaluation is runnnig. You can even specify how deep these values should be printed recursively, and transform them on the fly. Please consult the docstrings in <literal>lib/debug.nix</literal> for usage information.
Generators are functions that create file formats from nix data structures, e.g. for configuration files. There are generators available for: `INI`, `JSON` and `YAML`
All generators follow a similar call interface: `generatorName configFunctions data`, where `configFunctions` is an attrset of user-defined functions that format nested parts of the content. They each have common defaults, so often they do not need to be set manually. An example is `mkSectionName ? (name: libStr.escape [ "[" "]" ] name)` from the `INI` generator. It receives the name of a section and sanitizes it. The default `mkSectionName` escapes `[` and `]` with a backslash.
Generators can be fine-tuned to produce exactly the file format required by your application/service. One example is an INI-file format which uses `: ` as separator, the strings `"yes"`/`"no"` as boolean values and requires all string values to be quoted:
```nix
with lib;
let
customToINI = generators.toINI {
# specifies how to format a key/value pair
mkKeyValue = generators.mkKeyValueDefault {
# specifies the generated string for a subset of nix values
mkValueString = v:
if v == true then ''"yes"''
else if v == false then ''"no"''
else if isString v then ''"${v}"''
# and delegats all other values to the default generator
else generators.mkValueStringDefault {} v;
} ":";
};
# the INI file can now be given as plain old nix values
in customToINI {
main = {
pushinfo = true;
autopush = false;
host = "localhost";
port = 42;
};
mergetool = {
merge = "diff3";
};
}
```
This will produce the following INI file as nix string:
```INI
[main]
autopush:"no"
host:"localhost"
port:42
pushinfo:"yes"
str\:ange:"very::strange"
[mergetool]
merge:"diff3"
```
::: {.note}
Nix store paths can be converted to strings by enclosing a derivation attribute like so: `"${drv}"`.
:::
Detailed documentation for each generator can be found in `lib/generators.nix`.
Generators are functions that create file formats from nix data structures, e.g. for configuration files. There are generators available for: <literal>INI</literal>, <literal>JSON</literal> and <literal>YAML</literal>
</para>
<para>
All generators follow a similar call interface: <code>generatorName configFunctions data</code>, where <literal>configFunctions</literal> is an attrset of user-defined functions that format nested parts of the content. They each have common defaults, so often they do not need to be set manually. An example is <code>mkSectionName ? (name: libStr.escape [ "[" "]" ] name)</code> from the <literal>INI</literal> generator. It receives the name of a section and sanitizes it. The default <literal>mkSectionName</literal> escapes <literal>[</literal> and <literal>]</literal> with a backslash.
</para>
<para>
Generators can be fine-tuned to produce exactly the file format required by your application/service. One example is an INI-file format which uses <literal>: </literal> as separator, the strings <literal>"yes"</literal>/<literal>"no"</literal> as boolean values and requires all string values to be quoted:
</para>
<programlisting>
with lib;
let
customToINI = generators.toINI {
# specifies how to format a key/value pair
mkKeyValue = generators.mkKeyValueDefault {
# specifies the generated string for a subset of nix values
mkValueString = v:
if v == true then ''"yes"''
else if v == false then ''"no"''
else if isString v then ''"${v}"''
# and delegats all other values to the default generator
else generators.mkValueStringDefault {} v;
} ":";
};
# the INI file can now be given as plain old nix values
in customToINI {
main = {
pushinfo = true;
autopush = false;
host = "localhost";
port = 42;
};
mergetool = {
merge = "diff3";
};
}
</programlisting>
<para>
This will produce the following INI file as nix string:
</para>
<programlisting>
[main]
autopush:"no"
host:"localhost"
port:42
pushinfo:"yes"
str\:ange:"very::strange"
[mergetool]
merge:"diff3"
</programlisting>
<note>
<para>
Nix store paths can be converted to strings by enclosing a derivation attribute like so: <code>"${drv}"</code>.
</para>
</note>
<para>
Detailed documentation for each generator can be found in <literal>lib/generators.nix</literal>.
Merge sets of attributes and combine each attribute value in to a list. Similar to <xreflinkend="function-library-lib.attrsets.zipAttrsWith"/> where the merge function returns a list of all values.
Merge sets of attributes and combine each attribute value in to a list. Similar to [](#function-library-lib.attrsets.zipAttrsWith) where the merge function returns a list of all values.
`pkgs.nix-gitignore` is a function that acts similarly to `builtins.filterSource` but also allows filtering with the help of the gitignore format.
## Usage {#sec-pkgs-nix-gitignore-usage}
`pkgs.nix-gitignore` exports a number of functions, but you\'ll most likely need either `gitignoreSource` or `gitignoreSourcePure`. As their first argument, they both accept either 1. a file with gitignore lines or 2. a string with gitignore lines, or 3. a list of either of the two. They will be concatenated into a single big string.
Those filter functions accept the same arguments the `builtins.filterSource` function would pass to its filters, thus `fn: gitignoreFilterSourcePure fn ""` should be extensionally equivalent to `filterSource`. The file is blacklisted if it\'s blacklisted by either your filter or the gitignoreFilter.
If you want to make your own filter from scratch, you may use
<function>pkgs.nix-gitignore</function> is a function that acts similarly to <literal>builtins.filterSource</literal> but also allows filtering with the help of the gitignore format.
</para>
<sectionxml:id="sec-pkgs-nix-gitignore-usage">
<title>Usage</title>
<para>
<literal>pkgs.nix-gitignore</literal> exports a number of functions, but you'll most likely need either <literal>gitignoreSource</literal> or <literal>gitignoreSourcePure</literal>. As their first argument, they both accept either 1. a file with gitignore lines or 2. a string with gitignore lines, or 3. a list of either of the two. They will be concatenated into a single big string.
Those filter functions accept the same arguments the <literal>builtins.filterSource</literal> function would pass to its filters, thus <literal>fn: gitignoreFilterSourcePure fn ""</literal> should be extensionally equivalent to <literal>filterSource</literal>. The file is blacklisted iff it's blacklisted by either your filter or the gitignoreFilter.
</para>
<para>
If you want to make your own filter from scratch, you may use
`prefer-remote-fetch` is an overlay that download sources on remote builder. This is useful when the evaluating machine has a slow upload while the builder can fetch faster directly from the source. To use it, put the following snippet as a new overlay:
```nix
self: super:
(super.prefer-remote-fetch self super)
```
A full configuration example for that sets the overlay up for your own account, could look like this
<function>prefer-remote-fetch</function> is an overlay that download sources on remote builder. This is useful when the evaluating machine has a slow upload while the builder can fetch faster directly from the source. To use it, put the following snippet as a new overlay:
<programlisting>
self: super:
(super.prefer-remote-fetch self super)
</programlisting>
A full configuration example for that sets the overlay up for your own account, could look like this
Agda is available as the [agda](https://search.nixos.org/packages?channel=unstable&show=agda&from=0&size=30&sort=relevance&query=agda)
package.
@ -43,6 +43,7 @@ agda.withPackages (p: [
```
You can also reference a GitHub repository
```nix
agda.withPackages (p: [
(p.standard-library.overrideAttrs (oldAttrs: {
@ -59,6 +60,7 @@ agda.withPackages (p: [
If you want to use a library not added to Nixpkgs, you can add a
dependency to a local library by calling `agdaPackages.mkDerivation`.
```nix
agda.withPackages (p: [
(p.mkDerivation {
@ -92,20 +94,21 @@ See [Building Agda Packages](#building-agda-packages) for more information on `m
Agda will not by default use these libraries. To tell Agda to use a library we have some options:
* Call `agda` with the library flag:
```ShellSession
$ agda -l standard-library -i . MyFile.agda
```
```ShellSession
$ agda -l standard-library -i . MyFile.agda
```
* Write a `my-library.agda-lib` file for the project you are working on which may look like:
```
name: my-library
include: .
depend: standard-library
```
```
name: my-library
include: .
depend: standard-library
```
* Create the file `~/.agda/defaults` and add any libraries you want to use by default.
More information can be found in the [official Agda documentation on library management](https://agda.readthedocs.io/en/v2.6.1/tools/package-system.html).
## Compiling Agda
## Compiling Agda {#compiling-agda}
Agda modules can be compiled using the GHC backend with the `--compile` flag. A version of `ghc` with `ieee754` is made available to the Agda program via the `--with-compiler` flag.
This can be overridden by a different version of `ghc` as follows:
@ -116,7 +119,8 @@ agda.withPackages {
}
```
## Writing Agda packages
## Writing Agda packages {#writing-agda-packages}
To write a nix derivation for an Agda library, first check that the library has a `*.agda-lib` file.
A derivation can then be written using `agdaPackages.mkDerivation`. This has similar arguments to `stdenv.mkDerivation` with the following additions:
@ -140,19 +144,21 @@ agdaPackages.mkDerivation {
}
```
### Building Agda packages
### Building Agda packages {#building-agda-packages}
The default build phase for `agdaPackages.mkDerivation` simply runs `agda` on the `Everything.agda` file.
If something else is needed to build the package (e.g. `make`) then the `buildPhase` should be overridden.
Additionally, a `preBuild` or `configurePhase` can be used if there are steps that need to be done prior to checking the `Everything.agda` file.
`agda` and the Agda libraries contained in `buildInputs` are made available during the build phase.
The default install phase copies Agda source files, Agda interface files (`*.agdai`) and `*.agda-lib` files to the output directory.
This can be overridden.
By default, Agda sources are files ending on `.agda`, or literate Agda files ending on `.lagda`, `.lagda.tex`, `.lagda.org`, `.lagda.md`, `.lagda.rst`. The list of recognised Agda source extensions can be extended by setting the `extraExtensions` config variable.
## Adding Agda packages to Nixpkgs
## Adding Agda packages to Nixpkgs {#adding-agda-packages-to-nixpkgs}
To add an Agda package to `nixpkgs`, the derivation should be written to `pkgs/development/libraries/agda/${library-name}/` and an entry should be added to `pkgs/top-level/agda-packages.nix`. Here it is called in a scope with access to all other Agda libraries, so the top line of the `default.nix` can look like:
@ -182,6 +188,7 @@ mkDerivation {
'';
}
```
This library has a file called `.agda-lib`, and so we give an empty string to `libraryFile` as nothing precedes `.agda-lib` in the filename. This file contains `name: IAL-1.3`, and so we let `libraryName = "IAL-1.3"`. This library does not use an `Everything.agda` file and instead has a Makefile, so there is no need to set `everythingFile` and we set a custom `buildPhase`.
When writing an Agda package it is essential to make sure that no `.agda-lib` file gets added to the store as a single file (for example by using `writeText`). This causes Agda to think that the nix store is a Agda library and it will attempt to write to it whenever it typechecks something. See [https://github.com/agda/agda/issues/4613](https://github.com/agda/agda/issues/4613).
In this document and related Nix expressions, we use the term, _BEAM_, to describe the environment. BEAM is the name of the Erlang Virtual Machine and, as far as we're concerned, from a packaging perspective, all languages that run on the BEAM are interchangeable. That which varies, like the build system, is transparent to users of any given BEAM package, so we make no distinction.
## Available versions and deprecations schedule
## Available versions and deprecations schedule {#available-versions-and-deprecations-schedule}
### Elixir
### Elixir {#elixir}
nixpkgs follows the [official elixir deprecation schedule](https://hexdocs.pm/elixir/compatibility-and-deprecations.html) and keeps the last 5 released versions of Elixir available.
@ -68,7 +68,7 @@ Erlang.mk functions similarly to Rebar3, except we use `buildErlangMk` instead o
`mixRelease` is used to make a release in the mix sense. Dependencies will need to be fetched with `fetchMixDeps` and passed to it.
#### mixRelease - Elixir Phoenix example
#### mixRelease - Elixir Phoenix example {#mixrelease---elixir-phoenix-example}
Here is how your `default.nix` file would look.
@ -148,7 +148,7 @@ Setup will require the following steps:
- you can now `nix-build .`
- To run the release, set the `RELEASE_TMP` environment variable to a directory that your program has write access to. It will be used to store the BEAM settings.
#### Example of creating a service for an Elixir - Phoenix project
#### Example of creating a service for an Elixir - Phoenix project {#example-of-creating-a-service-for-an-elixir---phoenix-project}
In order to create a service with your release, you could add a `service.nix`
The Coq derivation is overridable through the `coq.override overrides`, where overrides is an attribute set which contains the arguments to override. We recommend overriding either of the following
@ -8,7 +8,7 @@ The Coq derivation is overridable through the `coq.override overrides`, where ov
* `customOCamlPackage` (optional, defaults to `null`, which lets Coq choose a version automatically), which can be set to any of the ocaml packages attribute of `ocaml-ng` (such as `ocaml-ng.ocamlPackages_4_10` which is the default for Coq 8.11 for example).
* `coq-version` (optional, defaults to the short version e.g. "8.10"), is a version number of the form "x.y" that indicates which Coq's version build behavior to mimic when using a source which is not a release. E.g. `coq.override { version = "d370a9d1328a4e1cdb9d02ee032f605a9d94ec7a"; coq-version = "8.10"; }`.
The recommended way of defining a derivation for a Coq library, is to use the `coqPackages.mkCoqDerivation` function, which is essentially a specialization of `mkDerivation` taking into account most of the specifics of Coq libraries. The following attributes are supported:
## Building a Crystal package {#building-a-crystal-package}
This section uses [Mint](https://github.com/mint-lang/mint) as an example for how to build a Crystal package.
If the Crystal project has any dependencies, the first step is to get a `shards.nix` file encoding those. Get a copy of the project and go to its root directory such that its `shard.lock` file is in the current directory, then run `crystal2nix` in it
## Local Development Workflow {#local-development-workflow}
For local development, it's recommended to use nix-shell to create a dotnet environment:
@ -16,7 +16,7 @@ mkShell {
}
```
### Using many sdks in a workflow
### Using many sdks in a workflow {#using-many-sdks-in-a-workflow}
It's very likely that more than one sdk will be needed on a given project. Dotnet provides several different frameworks (E.g dotnetcore, aspnetcore, etc.) as well as many versions for a given framework. Normally, dotnet is able to fetch a framework and install it relative to the executable. However, this would mean writing to the nix store in nixpkgs, which is read-only. To support the many-sdk use case, one can compose an environment using `dotnetCorePackages.combinePackages`:
@ -37,7 +37,7 @@ mkShell {
This will produce a dotnet installation that has the dotnet 3.1, 3.0, and 2.1 sdk. The first sdk listed will have it's cli utility present in the resulting environment. Example info output:
## dotnet-sdk vs dotnetCorePackages.sdk {#dotnet-sdk-vs-dotnetcorepackages.sdk}
The `dotnetCorePackages.sdk_X_Y` is preferred over the old dotnet-sdk as both major and minor version are very important for a dotnet environment. If a given minor version isn't present (or was changed), then this will likely break your ability to build a project.
## dotnetCorePackages.sdk vs dotnetCorePackages.net vs dotnetCorePackages.netcore vs dotnetCorePackages.aspnetcore
## dotnetCorePackages.sdk vs dotnetCorePackages.net vs dotnetCorePackages.netcore vs dotnetCorePackages.aspnetcore {#dotnetcorepackages.sdk-vs-dotnetcorepackages.net-vs-dotnetcorepackages.netcore-vs-dotnetcorepackages.aspnetcore}
The `dotnetCorePackages.sdk` contains both a runtime and the full sdk of a given version. The `net`, `netcore` and `aspnetcore` packages are meant to serve as minimal runtimes to deploy alongside already built applications. For runtime versions >= .NET 5 `net` is used while `netcore` is used for older .NET Core runtime version.
## Packaging a Dotnet Application
## Packaging a Dotnet Application {#packaging-a-dotnet-application}
Ideally, we would like to build against the sdk, then only have the dotnet runtime available in the runtime closure.
This example uses `zlib` from nixpkgs but instead of compiling **C** to **ELF** it compiles **C** to **JS** since we were using `pkgs.zlib.override` and changed stdenv to `pkgs.emscriptenStdenv`. A few adaptions and hacks were set in place to make it working. One advantage is that when `pkgs.zlib` is updated, it will automatically update this package as well. However, this can also be the downside...
Use `nix-shell -I nixpkgs=/some/dir/nixpkgs -A emscriptenPackages.libz` and from there you can go trough the individual steps. This makes it easy to build a good `unit test` or list the files of the project.
@ -177,7 +175,7 @@ Use `nix-shell -I nixpkgs=/some/dir/nixpkgs -A emscriptenPackages.libz` and from
6. `buildPhase`
7. ... happy hacking...
## Summary
## Summary {#summary}
Using this toolchain makes it easy to leverage `nix` from NixOS, MacOSX or even Windows (WSL+ubuntu+nix). This toolchain is reproducible, behaves like the rest of the packages from nixpkgs and contains a set of well working examples to learn and adapt from.
@ -8,12 +8,30 @@ Programs in the GNOME universe are written in various languages but they all use
[GSettings](https://developer.gnome.org/gio/stable/GSettings.html) API is often used for storing settings. GSettings schemas are required, to know the type and other metadata of the stored values. GLib looks for `glib-2.0/schemas/gschemas.compiled` files inside the directories of `XDG_DATA_DIRS`.
On Linux, GSettings API is implemented using [dconf](https://wiki.gnome.org/Projects/dconf) backend. You will need to add `dconf` GIO module to `GIO_EXTRA_MODULES` variable, otherwise the `memory` backend will be used and the saved settings will not be persistent.
On Linux, GSettings API is implemented using [dconf](https://wiki.gnome.org/Projects/dconf) backend. You will need to add `dconf`[GIO module](#ssec-gnome-gio-modules) to `GIO_EXTRA_MODULES` variable, otherwise the `memory` backend will be used and the saved settings will not be persistent.
Last you will need the dconf database D-Bus service itself. You can enable it using `programs.dconf.enable`.
Some applications will also require `gsettings-desktop-schemas` for things like reading proxy configuration or user interface customization. This dependency is often not mentioned by upstream, you should grep for `org.gnome.desktop` and `org.gnome.system` to see if the schemas are needed.
### GIO modules {#ssec-gnome-gio-modules}
GLib’s [GIO](https://developer.gnome.org/gio/stable/ch01.html) library supports several [extension points](https://developer.gnome.org/gio/stable/extending-gio.html). Notably, they allow:
The modules are typically installed to `lib/gio/modules/` directory of a package and you need to add them to `GIO_EXTRA_MODULES` if you need any of those features.
In particular, we recommend:
* adding `dconf.lib` for any software on Linux that reads [GSettings](#ssec-gnome-settings) (even transitivily through e.g. GTK’s file manager)
* adding `glib-networking` for any software that accesses network using GIO or libsoup – glib-networking contains a module that implements TLS support and loads system-wide proxy settings
To allow software to use various virtual file systems, `gvfs` package can be also added. But that is usually an optional feature so we typically use `gvfs` from the system (e.g. installed globally using NixOS module).
GTK applications typically use [GdkPixbuf](https://developer.gnome.org/gdk-pixbuf/stable/) to load images. But `gdk-pixbuf` package only supports basic bitmap formats like JPEG, PNG or TIFF, requiring to use third-party loader modules for other formats. This is especially painful since GTK itself includes SVG icons, which cannot be rendered without a loader provided by `librsvg`.
@ -84,7 +102,7 @@ For convenience, it also adds `dconf.lib` for a GIO module implementing a GSetti
- []{#ssec-gnome-hooks-gobject-introspection} `gobject-introspection` setup hook populates `GI_TYPELIB_PATH` variable with `lib/girepository-1.0` directories of dependencies, which is then added to wrapper by `wrapGAppsHook`. It also adds `share` directories of dependencies to `XDG_DATA_DIRS`, which is intended to promote GIR files but it also [pollutes the closures](https://github.com/NixOS/nixpkgs/issues/32790) of packages using `wrapGAppsHook`.
::: warning
::: {.warning}
The setup hook [currently](https://github.com/NixOS/nixpkgs/issues/56943) does not work in expressions with `strictDeps` enabled, like Python packages. In those cases, you will need to disable it with `strictDeps = false;`.
The function `buildGoPackage` builds legacy Go programs, not supporting Go modules.
### Example for `buildGoPackage`
### Example for `buildGoPackage` {#example-for-buildgopackage}
In the following is an example expression using buildGoPackage, the following arguments are of special significance to the function:
@ -140,4 +140,4 @@ Removes the pre-existing vendor directory. This should only be used if the depen
### `subPackages` {#var-go-subPackages}
Limits the builder from building child packages that have not been listed. If <varname>subPackages</varname> is not specified, all child packages will be built.
Limits the builder from building child packages that have not been listed. If `subPackages` is not specified, all child packages will be built.
### Installation without packages {#installation-without-packages}
You can install `hy` via nix-env or by adding it to `configuration.nix` by reffering to it as a `hy` attribute. This kind of installation adds `hy` to your environment and it succesfully works with `python3`.
::: {.caution}
Packages that are installed with your python derivation, are not accesible by `hy` this way.
:::
### Installation with packages {#installation-with-packages}
Creating `hy` derivation with custom `python` packages is really simple and similar to the way that python does it. Attribute `hy` provides function `withPackages` that creates custom `hy` derivation with specified packages.
For example if you want to create shell with `matplotlib` and `numpy`, you can do it like so:
There is also the `buildLuaPackage` function that can be used when lua modules
are not packaged for luarocks. You can see a few examples at `pkgs/top-level/lua-packages.nix`.
## Lua Reference
## Lua Reference {#lua-reference}
### Lua interpreters
### Lua interpreters {#lua-interpreters}
Versions 5.1, 5.2 and 5.3 of the lua interpreter are available as
respectively `lua5_1`, `lua5_2` and `lua5_3`. Luajit is available too.
The Nix expressions for the interpreters can be found in `pkgs/development/interpreters/lua-5`.
#### Attributes on lua interpreters packages
#### Attributes on lua interpreters packages {#attributes-on-lua-interpreters-packages}
Each interpreter has the following attributes:
@ -164,8 +174,7 @@ Each interpreter has the following attributes:
- `withPackages`. Simpler interface to `buildEnv`.
- `pkgs`. Set of Lua packages for that specific interpreter. The package set can be modified by overriding the interpreter and passing `packageOverrides`.
#### `buildLuarocksPackage` function
#### `buildLuarocksPackage` function {#buildluarockspackage-function}
The `buildLuarocksPackage` function is implemented in `pkgs/development/interpreters/lua-5/build-lua-package.nix`
The following is an example:
@ -205,16 +214,17 @@ install the package
By default `meta.platforms` is set to the same value as the interpreter unless overridden otherwise.
#### `buildLuaApplication` function
#### `buildLuaApplication` function {#buildluaapplication-function}
The `buildLuaApplication` function is practically the same as `buildLuaPackage`.
The difference is that `buildLuaPackage` by default prefixes the names of the packages with the version of the interpreter.
Because with an application we're not interested in multiple version the prefix is dropped.
#### lua.withPackages function
#### lua.withPackages function {#lua.withpackages-function}
The `lua.withPackages` takes a function as an argument that is passed the set of lua packages and returns the list of packages to be included in the environment.
Using the `withPackages` function, the previous example for the luafilesystem environment can be written like this:
`withPackages` passes the correct package set for the specific interpreter version as an argument to the function. In the above example, `ps` equals `luaPackages`.
> ⚠️ This pattern is the simplest but may cause unnecessary rebuilds due to the output hash changing.
@ -163,7 +165,7 @@ The build will fail, and tell you the expected `outputHash` to place. When you'v
If your package uses _SNAPSHOT_ dependencies or _version ranges_; there is a strong likelihood that over-time your output hash will change since the resolved dependencies may change. Hence this method is less recommended then using `buildMaven`.
## Building a JAR
## Building a JAR {#building-a-jar}
Regardless of which strategy is chosen above, the step to build the derivation is the same.
@ -201,7 +203,7 @@ in stdenv.mkDerivation rec {
2 directories, 1 file
```
## Runnable JAR
## Runnable JAR {#runnable-jar}
The previous example builds a `jar` file but that's not a file one can run.
@ -213,7 +215,7 @@ We will use the same repository we built above (either _double invocation_ or _b
The following two methods are more suited to Nix then building an [UberJar](https://imagej.net/Uber-JAR) which may be the more traditional approach.
### CLASSPATH
### CLASSPATH {#classpath}
> This is ideal if you are providing a derivation for _nixpkgs_ and don't want to patch the project's `pom.xml`.
@ -252,11 +254,12 @@ in stdenv.mkDerivation rec {
}
```
### MANIFEST file via Maven Plugin
### MANIFEST file via Maven Plugin {#manifest-file-via-maven-plugin}
> This is ideal if you are the project owner and want to change your `pom.xml` to set the CLASSPATH within it.
Augment the `pom.xml` to create a JAR with the following manifest:
@ -4,60 +4,83 @@ OCaml libraries should be installed in `$(out)/lib/ocaml/${ocaml.version}/site-l
Given that most of the OCaml ecosystem is now built with dune, nixpkgs includes a convenience build support function called `buildDunePackage` that will build an OCaml package using dune, OCaml and findlib and any additional dependencies provided as `buildInputs` or `propagatedBuildInputs`.
Here is a simple package example. It defines an (optional) attribute `minimumOCamlVersion` that will be used to throw a descriptive evaluation error if building with an older OCaml is attempted. It uses the `fetchFromGitHub` fetcher to get its source. It sets the `doCheck` (optional) attribute to `true` which means that tests will be run with `dune runtest -p angstrom` after the build (`dune build -p angstrom`) is complete. It uses `alcotest` as a build input (because it is needed to run the tests) and `bigstringaf` and `result` as propagated build inputs (thus they will also be available to libraries depending on this library). The library will be installed using the `angstrom.install` file that dune generates.
Here is a simple package example.
- It defines an (optional) attribute `minimalOCamlVersion` that will be used to
throw a descriptive evaluation error if building with an older OCaml is
attempted.
- It uses the `fetchFromGitHub` fetcher to get its source.
- `useDune2 = true` ensures that the latest version of Dune is used for the
build (this may become the default value in a future release).
- It sets the optional `doCheck` attribute such that tests will be run with
`dune runtest -p angstrom` after the build (`dune build -p angstrom`) is
complete, but only if the Ocaml version is at at least `"4.05"`.
- It uses the package `ocaml-syntax-shims` as a build input, `alcotest` and
`ppx_let` as check inputs (because they are needed to run the tests), and
`bigstringaf` and `result` as propagated build inputs (thus they will also be
available to libraries depending on this library).
- The library will be installed using the `angstrom.install` file that dune
description = "OCaml parser combinators built for speed and memory efficiency";
license = licenses.bsd3;
maintainers = with maintainers; [ sternenseemann ];
license = lib.licenses.bsd3;
maintainers = with lib.maintainers; [ sternenseemann ];
};
}
```
Here is a second example, this time using a source archive generated with `dune-release`. It is a good idea to use this archive when it is available as it will usually contain substituted variables such as a `%%VERSION%%` field. This library does not depend on any other OCaml library and no tests are run after building it.
It is highly encouraged to have testing as part of the package build. This
helps to avoid situations where the package was able to build and install,
@ -559,10 +560,11 @@ thus can cause issues when a test suite asserts on that behavior.
as many tests should be enabled as possible. Failing tests can still be
a good indication that the package is not in a valid state.
#### Using pytest
#### Using pytest {#using-pytest}
Pytest is the most common test runner for python repositories. A trivial
test run would be:
```
checkInputs = [ pytest ];
checkPhase = "pytest";
@ -572,6 +574,7 @@ However, many repositories' test suites do not translate well to nix's build
sandbox, and will generally need many tests to be disabled.
To filter tests using pytest, one can do the following:
```
checkInputs = [ pytest ];
# avoid tests which need additional data or touch network
@ -587,19 +590,20 @@ easier than having to create a new package.
`-k` is used to define a predicate for test names. In this example, we are
filtering out tests which contain `download` or `update` in their test case name.
Only one `-k` argument is allows, and thus a long predicate should be concatenated
with "\" and wrapped to the next line.
Only one `-k` argument is allowed, and thus a long predicate should be concatenated
with “\\” and wrapped to the next line.
*NOTE:* In pytest==6.0.1, the use of "\" to continue a line (e.g. `-k 'not download \'`) has
*NOTE:* In pytest==6.0.1, the use of “\\” to continue a line (e.g. `-k 'not download \'`) has
been removed, in this case, it's recommended to use `pytestCheckHook`.
#### Using pytestCheckHook
#### Using pytestCheckHook {#using-pytestcheckhook}
`pytestCheckHook` is a convenient hook which will substitute the setuptools
`test` command for a checkPhase which runs `pytest`. This is also beneficial
when a package may need many items disabled to run the test suite.
Using the example above, the analagous pytestCheckHook usage would be:
```
checkInputs = [ pytestCheckHook ];
@ -637,7 +641,7 @@ Trying to concatenate the related strings to disable tests in a regular checkPha
would be much harder to read. This also enables us to comment on why specific tests
are disabled.
#### Using pythonImportsCheck
#### Using pythonImportsCheck {#using-pythonimportscheck}
Although unit tests are highly prefered to validate correctness of a package, not
all packages have test suites that can be ran easily, and some have none at all.
@ -659,7 +663,7 @@ However, this is done in it's own phase, and not dependent on whether `doCheck =
This can also be useful in verifying that the package doesn't assume commonly
present packages (e.g. `setuptools`)
### Develop local package
### Develop local package {#develop-local-package}
As a Python developer you're likely aware of [development mode](http://setuptools.readthedocs.io/en/latest/setuptools.html#development-mode)
(`python setup.py develop`); instead of installing the package this command
@ -694,7 +698,7 @@ buildPythonPackage rec {
It is important to note that due to how development mode is implemented on Nix
it is not possible to have multiple packages simultaneously in development mode.
### Organising your packages
### Organising your packages {#organising-your-packages}
So far we discussed how you can use Python on Nix, and how you can develop with
it. We've looked at how you write expressions to package Python packages, and we
@ -706,7 +710,7 @@ like to be able to use in different projects. In order to minimise unnecessary
duplication we now look at how you can maintain a repository with your
own packages. The important functions here are `import` and `callPackage`.
### Including a derivation using `callPackage`
### Including a derivation using `callPackage` {#including-a-derivation-using-callpackage}
Earlier we created a Python environment using `withPackages`, and included the
`toolz` package via a `let` expression.
@ -756,9 +760,9 @@ don't explicitly define which `python` derivation should be used. In the above
example we use `buildPythonPackage` that is part of the set `python38Packages`,
and in this case the `python38` interpreter is automatically used.
## Reference
## Reference {#reference}
### Interpreters
### Interpreters {#interpreters}
Versions 2.7, 3.6, 3.7, 3.8 and 3.9 of the CPython interpreter are available as
respectively `python27`, `python36`, `python37`, `python38` and `python39`. The
@ -773,11 +777,11 @@ All packages depending on any Python interpreter get appended
`out/{python.sitePackages}` to `$PYTHONPATH` if such directory
exists.
#### Missing `tkinter` module standard library
#### Missing `tkinter` module standard library {#missing-tkinter-module-standard-library}
To reduce closure size the `Tkinter`/`tkinter` is available as a separate package, `pythonPackages.tkinter`.
#### Attributes on interpreters packages
#### Attributes on interpreters packages {#attributes-on-interpreters-packages}
Each interpreter has the following attributes:
@ -789,7 +793,7 @@ Each interpreter has the following attributes:
- `executable`. Name of the interpreter executable, e.g. `python3.8`.
- `pkgs`. Set of Python packages for that specific interpreter. The package set can be modified by overriding the interpreter and passing `packageOverrides`.
### Optimizations
### Optimizations {#optimizations}
The Python interpreters are by default not build with optimizations enabled, because
the builds are in that case not reproducible. To enable optimizations, override the
@ -806,7 +810,7 @@ let
in mypython
```
### Building packages and applications
### Building packages and applications {#building-packages-and-applications}
Python libraries and applications that use `setuptools` or
`distutils` are typically built with respectively the `buildPythonPackage` and
@ -838,7 +842,7 @@ and the aliases
* `pkgs.python3Packages` pointing to `pkgs.python38Packages`
* `pkgs.pythonPackages` pointing to `pkgs.python2Packages`
#### `buildPythonPackage` function
#### `buildPythonPackage` function {#buildpythonpackage-function}
The `buildPythonPackage` function is implemented in
### Rationale of non-existent global site-packages
### Rationale of non-existent global site-packages {#rationale-of-non-existent-global-site-packages}
On most operating systems a global `site-packages` is maintained. This however
becomes problematic if you want to run multiple Python versions or have multiple
@ -1384,7 +1391,7 @@ If you want to create a Python environment for development, then the recommended
method is to use `nix-shell`, either with or without the `python.buildEnv`
function.
### How to consume Python modules using pip in a virtual environment like I am used to on other Operating Systems?
### How to consume Python modules using pip in a virtual environment like I am used to on other Operating Systems? {#how-to-consume-python-modules-using-pip-in-a-virtual-environment-like-i-am-used-to-on-other-operating-systems}
While this approach is not very idiomatic from Nix perspective, it can still be
useful when dealing with pre-existing projects or in situations where it's not
@ -1497,7 +1504,7 @@ is executed it will attempt to download the Python modules listed in
requirements.txt. However these will be cached locally within the `virtualenv`
folder and not downloaded again.
### How to override a Python package from `configuration.nix`?
### How to override a Python package from `configuration.nix`? {#how-to-override-a-python-package-from-configuration.nix}
If you need to change a package's attribute(s) from `configuration.nix` you could do:
@ -1535,7 +1542,7 @@ this snippet:
}
```
### How to override a Python package using overlays?
### How to override a Python package using overlays? {#how-to-override-a-python-package-using-overlays}
Use the following overlay template:
@ -1556,12 +1563,12 @@ self: super: {
}
```
### How to use Intel's MKL with numpy and scipy?
### How to use Intel’s MKL with numpy and scipy? {#how-to-use-intels-mkl-with-numpy-and-scipy}
MKL can be configured using an overlay. See the section "[Using overlays to
### What inputs do `setup_requires`, `install_requires` and `tests_require` map to?
### What inputs do `setup_requires`, `install_requires` and `tests_require` map to? {#what-inputs-do-setup_requires-install_requires-and-tests_require-map-to}
In a `setup.py` or `setup.cfg` it is common to declare dependencies:
@ -1569,9 +1576,9 @@ In a `setup.py` or `setup.cfg` it is common to declare dependencies:
* `install_requires` corresponds to `propagatedBuildInputs`
Several versions of Ruby interpreters are available on Nix, as well as over 250 gems and many applications written in Ruby. The attribute `ruby` refers to the default Ruby interpreter, which is currently MRI 2.6. It's also possible to refer to specific versions, e.g. `ruby_2_y`, `jruby`, or `mruby`.
@ -12,7 +12,7 @@ The interpreters have common attributes, namely `gems`, and `withPackages`. So y
Since not all gems have executables like `nokogiri`, it's usually more convenient to use the `withPackages` function like this: `ruby.withPackages (p: with p; [ nokogiri ])`. This will also make sure that the Ruby in your environment will be able to find the gem and it can be used in your Ruby code (for example via `ruby` or `irb` executables) via `require "nokogiri"` as usual.
### Temporary Ruby environment with `nix-shell`
### Temporary Ruby environment with `nix-shell` {#temporary-ruby-environment-with-nix-shell}
Rather than having a single Ruby environment shared by all Ruby development projects on a system, Nix allows you to create separate environments per project. `nix-shell` gives you the possibility to temporarily load another environment akin to a combined `chruby` or `rvm` and `bundle exec`.
Again, it's possible to launch the interpreter from the shell. The Ruby interpreter has the attribute `gems` which contains all Ruby gems for that specific interpreter.
#### Load Ruby environment from `.nix` expression
#### Load Ruby environment from `.nix` expression {#load-ruby-environment-from-.nix-expression}
As explained in the Nix manual, `nix-shell` can also load an expression from a `.nix` file. Say we want to have Ruby 2.6, `nokogori`, and `pry`. Consider a `shell.nix` file with:
@ -45,7 +45,7 @@ What's happening here?
2. Then we create a Ruby environment with the `withPackages` function.
3. The `withPackages` function expects us to provide a function as an argument that takes the set of all ruby gems and returns a list of packages to include in the environment. Here, we select the packages `nokogiri` and `pry` from the package set.
#### Execute command with `--run`
#### Execute command with `--run` {#execute-command-with---run}
A convenient flag for `nix-shell` is `--run`. It executes a command in the `nix-shell`. We can e.g. directly open a `pry` REPL:
@ -65,7 +65,7 @@ Or run a script using this environment:
#### Using `nix-shell` as shebang {#using-nix-shell-as-shebang}
In fact, for the last case, there is a more convenient method. You can add a [shebang](<https://en.wikipedia.org/wiki/Shebang_(Unix)>) to your script specifying which dependencies `nix-shell` needs. With the following shebang, you can just execute `./example.rb`, and it will run with all dependencies.
@ -80,9 +80,9 @@ body = RestClient.get('http://example.com').body
puts Nokogiri::HTML(body).at('h1').text
```
## Developing with Ruby
## Developing with Ruby {#developing-with-ruby}
### Using an existing Gemfile
### Using an existing Gemfile {#using-an-existing-gemfile}
In most cases, you'll already have a `Gemfile.lock` listing all your dependencies. This can be used to generate a `gemset.nix` which is used to fetch the gems and combine them into a single environment. The reason why you need to have a separate file for this, is that Nix requires you to have a checksum for each input to your build. Since the `Gemfile.lock` that `bundler` generates doesn't provide us with checksums, we have to first download each gem, calculate its SHA256, and store it in this separate file.
@ -120,7 +120,7 @@ One common issue that you might have is that you have Ruby 2.6, but also `bundle
Of course for this use-case one could also use overlays since the configuration for `pg` depends on the `postgresql` alias, but for demonstration purposes this has to suffice.
### Adding a gem to the default gemset
### Adding a gem to the default gemset {#adding-a-gem-to-the-default-gemset}
Now that you know how to get a working Ruby environment with Nix, it's time to go forward and start actually developing with Ruby. We will first have a look at how Ruby gems are packaged on Nix. Then, we will look at how you can use development mode with your code.
@ -215,7 +215,7 @@ To test that it works, you can then try using the gem with:
NIX_PATH=nixpkgs=$PWD nix-shell -p "ruby.withPackages (ps: with ps; [ name-of-your-gem ])"
A common task is to add a ruby executable to nixpkgs, popular examples would be `chef`, `jekyll`, or `sass`. A good way to do that is to use the `bundlerApp` function, that allows you to make a package that only exposes the listed executables, otherwise the package may cause conflicts through common paths like `bin/rake` or `bin/bundler` that aren't meant to be used.
@ -243,7 +243,7 @@ bundlerApp {
All that's left to do is to generate the corresponding `Gemfile.lock` and `gemset.nix` as described above in the `Using an existing Gemfile` section.
#### Packaging executables that require wrapping
#### Packaging executables that require wrapping {#packaging-executables-that-require-wrapping}
Sometimes your app will depend on other executables at runtime, and tries to find it through the `PATH` environment variable.
- For basic usage just pull `texlive.combined.scheme-basic` for an environment with basic LaTeX support.
- It typically won't work to use separately installed packages together. Instead, you can build a custom set of packages like this:
```nix
@ -14,6 +15,7 @@ Since release 15.09 there is a new TeX Live packaging that lives entirely under
```
- There are all the schemes, collections and a few thousand packages, as defined upstream (perhaps with tiny differences).
- By default you only get executables and files needed during runtime, and a little documentation for the core packages. To change that, you need to add `pkgFilter` function to `combine`.
@ -12,7 +12,7 @@ At the moment we support three different methods for managing plugins:
- Pathogen
- vim-plug
## Custom configuration
## Custom configuration {#custom-configuration}
Adding custom .vimrc lines can be done using the following code:
@ -56,7 +56,7 @@ neovim-qt.override {
}
```
## Managing plugins with Vim packages
## Managing plugins with Vim packages {#managing-plugins-with-vim-packages}
To store you plugins in Vim packages (the native Vim plugin manager, see `:help packages`) the following example can be used:
@ -116,7 +116,7 @@ The resulting package can be added to `packageOverrides` in `~/.nixpkgs/config.n
After that you can install your special grafted `myVim` or `myNeovim` packages.
### What if your favourite Vim plugin isn't already packaged?
### What if your favourite Vim plugin isn’t already packaged? {#what-if-your-favourite-vim-plugin-isnt-already-packaged}
If one of your favourite plugins isn't packaged, you can package it yourself:
@ -154,7 +154,7 @@ in
}
```
## Managing plugins with vim-plug
## Managing plugins with vim-plug {#managing-plugins-with-vim-plug}
To use [vim-plug](https://github.com/junegunn/vim-plug) to manage your Vim
plugins the following example can be used:
@ -183,14 +183,14 @@ neovim.override {
}
```
## Managing plugins with VAM
## Managing plugins with VAM {#managing-plugins-with-vam}
### Handling dependencies of Vim plugins
### Handling dependencies of Vim plugins {#handling-dependencies-of-vim-plugins}
VAM introduced .json files supporting dependencies without versioning
assuming that "using latest version" is ok most of the time.
### Example
### Example {#example}
First create a vim-scripts file having one plugin name per line. Example:
@ -280,7 +280,7 @@ Sample output2:
]
```
## Adding new plugins to nixpkgs
## Adding new plugins to nixpkgs {#adding-new-plugins-to-nixpkgs}
Nix expressions for Vim plugins are stored in [pkgs/misc/vim-plugins](/pkgs/misc/vim-plugins). For the vast majority of plugins, Nix expressions are automatically generated by running [`./update.py`](/pkgs/misc/vim-plugins/update.py). This creates a [generated.nix](/pkgs/misc/vim-plugins/generated.nix) file based on the plugins listed in [vim-plugin-names](/pkgs/misc/vim-plugins/vim-plugin-names). Plugins are listed in alphabetical order in `vim-plugin-names` using the format `[github username]/[repository]`. For example https://github.com/scrooloose/nerdtree becomes `scrooloose/nerdtree`.
@ -298,7 +298,7 @@ To add a new plugin, run `./update.py --add "[owner]/[name]"`. **NOTE**: This sc
Finally, there are some plugins that are also packaged in nodePackages because they have Javascript-related build steps, such as running webpack. Those plugins are not listed in `vim-plugin-names` or managed by `update.py` at all, and are included separately in `overrides.nix`. Currently, all these plugins are related to the `coc.nvim` ecosystem of Language Server Protocol integration with vim/neovim.
## Updating plugins in nixpkgs
## Updating plugins in nixpkgs {#updating-plugins-in-nixpkgs}
Run the update script with a GitHub API token that has at least `public_repo` access. Running the script without the token is likely to result in rate-limiting (429 errors). For steps on creating an API token, please refer to [GitHub's token documentation](https://docs.github.com/en/free-pro-team@latest/github/authenticating-to-github/creating-a-personal-access-token).
@ -312,7 +312,7 @@ Alternatively, set the number of processes to a lower count to avoid rate-limiti
./pkgs/misc/vim-plugins/update.py --proc 1
```
## Important repositories
## Important repositories {#important-repositories}
- [vim-pi](https://bitbucket.org/vimcommunity/vim-pi) is a plugin repository
from VAM plugin manager meant to be used by others as well used by
This chapter will be organized in three parts. First, it will describe the basics of how to package software in a way that supports cross-compilation. Second, it will describe how to use Nixpkgs when cross-compiling. Third, it will describe the internal infrastructure supporting cross-compilation.
## Packaging in a cross-friendly manner {#sec-cross-packaging}
@ -65,8 +64,8 @@ The exact schema these fields follow is a bit ill-defined due to a long and conv
### Theory of dependency categorization {#ssec-cross-dependency-categorization}
::: note
This is a rather philosophical description that isn't very Nixpkgs-specific. For an overview of all the relevant attributes given to `mkDerivation`, see <xreflinkend="ssec-stdenv-dependencies"/>. For a description of how everything is implemented, see <xreflinkend="ssec-cross-dependency-implementation"/>.
::: {.note}
This is a rather philosophical description that isn't very Nixpkgs-specific. For an overview of all the relevant attributes given to `mkDerivation`, see [](#ssec-stdenv-dependencies). For a description of how everything is implemented, see [](#ssec-cross-dependency-implementation).
:::
In this section we explore the relationship between both runtime and build-time dependencies and the 3 Autoconf platforms.
@ -81,10 +80,10 @@ Finally, if the depending package is a compiler or other machine-code-producing
Putting this all together, that means we have dependencies in the form "host → target", in at most the following six combinations:
#### Possible dependency types {#possible-dependency-types}
@ -113,15 +112,18 @@ On less powerful machines, it can be inconvenient to cross-compile a package onl
$ nix-build '<nixpkgs>' -A pkgsCross.raspberryPi.hello
```
#### What if my package's build system needs to build a C program to be run under the build environment? {#cross-qa-build-c-program-in-build-environment}
#### What if my package’s build system needs to build a C program to be run under the build environment? {#cross-qa-build-c-program-in-build-environment}
Add the following to your `mkDerivation` invocation.
```nix
depsBuildBuild = [ buildPackages.stdenv.cc ];
```
#### My package's testsuite needs to run host platform code. {#cross-testsuite-runs-host-code}
#### My package’s testsuite needs to run host platform code. {#cross-testsuite-runs-host-code}
Add the following to your `mkDerivation` invocation.
@ -134,7 +136,7 @@ Nixpkgs can be instantiated with `localSystem` alone, in which case there is no
$ nix-build '<nixpkgs>' --arg crossSystem '(import <nixpkgs/lib>).systems.examples.fooBarBaz' -A whatever
```
::: note
::: {.note}
Eventually we would like to make these platform examples an unnecessary convenience so that
```ShellSession
@ -146,7 +148,7 @@ works in the vast majority of cases. The problem today is dependencies on other
While one is free to pass both parameters in full, there's a lot of logic to fill in missing fields. As discussed in the previous section, only one of `system`, `config`, and `parsed` is needed to infer the other two. Additionally, `libc` will be inferred from `parse`. Finally, `localSystem.system` is also _impurely_ inferred based on the platform evaluation occurs. This means it is often not necessary to pass `localSystem` at all, as in the command-line example in the previous paragraph.
::: note
::: {.note}
Many sources (manual, wiki, etc) probably mention passing `system`, `platform`, along with the optional `crossSystem` to Nixpkgs: `import <nixpkgs> { system = ..; platform = ..; crossSystem = ..; }`. Passing those two instead of `localSystem` is still supported for compatibility, but is discouraged. Indeed, much of the inference we do for these parameters is motivated by compatibility as much as convenience.
:::
@ -156,7 +158,7 @@ One would think that `localSystem` and `crossSystem` overlap horribly with the t
### Implementation of dependencies {#ssec-cross-dependency-implementation}
The categories of dependencies developed in <xreflinkend="ssec-cross-dependency-categorization"/> are specified as lists of derivations given to `mkDerivation`, as documented in <xreflinkend="ssec-stdenv-dependencies"/>. In short, each list of dependencies for "host → target" of "foo → bar" is called `depsFooBar`, with exceptions for backwards compatibility that `depsBuildHost` is instead called `nativeBuildInputs` and `depsHostTarget` is instead called `buildInputs`. Nixpkgs is now structured so that each `depsFooBar` is automatically taken from `pkgsFooBar`. (These `pkgsFooBar`s are quite new, so there is no special case for `nativeBuildInputs` and `buildInputs`.) For example, `pkgsBuildHost.gcc` should be used at build-time, while `pkgsHostTarget.gcc` should be used at run-time.
The categories of dependencies developed in [](#ssec-cross-dependency-categorization) are specified as lists of derivations given to `mkDerivation`, as documented in [](#ssec-stdenv-dependencies). In short, each list of dependencies for "host → target" of "foo → bar" is called `depsFooBar`, with exceptions for backwards compatibility that `depsBuildHost` is instead called `nativeBuildInputs` and `depsHostTarget` is instead called `buildInputs`. Nixpkgs is now structured so that each `depsFooBar` is automatically taken from `pkgsFooBar`. (These `pkgsFooBar`s are quite new, so there is no special case for `nativeBuildInputs` and `buildInputs`.) For example, `pkgsBuildHost.gcc` should be used at build-time, while `pkgsHostTarget.gcc` should be used at run-time.
Now, for most of Nixpkgs's history, there were no `pkgsFooBar` attributes, and most packages have not been refactored to use it explicitly. Prior to those, there were just `buildPackages`, `pkgs`, and `targetPackages`. Those are now redefined as aliases to `pkgsBuildHost`, `pkgsHostTarget`, and `pkgsTargetTarget`. It is acceptable, even recommended, to use them for libraries to show that the host platform is irrelevant.
@ -178,7 +180,7 @@ While there are many package sets, and thus many edges, the stages can also be a
In each stage, `pkgsBuildHost` refers to the previous stage, `pkgsBuildBuild` refers to the one before that, and `pkgsHostTarget` refers to the current one, and `pkgsTargetTarget` refers to the next one. When there is no previous or next stage, they instead refer to the current stage. Note how all the invariants regarding the mapping between dependency and depending packages' build host and target platforms are preserved. `pkgsBuildTarget` and `pkgsHostHost` are more complex in that the stage fitting the requirements isn't always a fixed chain of "prevs" and "nexts" away (modulo the "saturating" self-references at the ends). We just special case each instead. All the primary edges are implemented is in `pkgs/stdenv/booter.nix`, and secondarily aliases in `pkgs/top-level/stage.nix`.
::: note
::: {.note}
The native stages are bootstrapped in legacy ways that predate the current cross implementation. This is why the bootstrapping stages leading up to the final stages are ignored in the previous paragraph.
:::
@ -186,6 +188,7 @@ If one looks at the 3 platform triples, one can see that they overlap such that
```
(native, native, native, foreign, foreign)
```
If one imagines the saturating self references at the end being replaced with infinite stages, and then overlays those platform triples, one ends up with the infinite tuple:
@ -193,8 +196,8 @@ If one imagines the saturating self references at the end being replaced with in
One can then imagine any sequence of platforms such that there are bootstrap stages with their 3 platforms determined by "sliding a window" that is the 3 tuple through the sequence. This was the original model for bootstrapping. Without a target platform (assume a better world where all compilers are multi-target and all standard libraries are built in their own derivation), this is sufficient. Conversely if one wishes to cross compile "faster", with a "Canadian Cross" bootstrapping stage where `build != host != target`, more bootstrapping stages are needed since no sliding window provides the pesky `pkgsBuildTarget` package set since it skips the Canadian cross stage's "host".
::: note
It is much better to refer to `buildPackages` than `targetPackages`, or more broadly package sets that do not mention "target". There are three reasons for this.
::: {.note}
It is much better to refer to `buildPackages` than `targetPackages`, or more broadly package sets that do not mention “target”. There are three reasons for this.
First, it is because bootstrapping stages do not have a unique `targetPackages`. For example a `(x86-linux, x86-linux, arm-linux)` and `(x86-linux, x86-linux, x86-windows)` package set both have a `(x86-linux, x86-linux, x86-linux)` package set. Because there is no canonical `targetPackages` for such a native (`build == host == target`) package set, we set their `targetPackages`
@ -203,6 +206,6 @@ Second, it is because this is a frequent source of hard-to-follow "infinite recu
Thirdly, it is because everything target-mentioning only exists to accommodate compilers with lousy build systems that insist on the compiler itself and standard library being built together. Of course that is bad because bigger derivations means longer rebuilds. It is also problematic because it tends to make the standard libraries less like other libraries than they could be, complicating code and build systems alike. Because of the other problems, and because of these innate disadvantages, compilers ought to be packaged another way where possible.
:::
::: note
If one explores Nixpkgs, they will see derivations with names like `gccCross`. Such `*Cross` derivations is a holdover from before we properly distinguished between the host and target platforms—the derivation with "Cross" in the name covered the `build = host != target` case, while the other covered the `host = target`, with build platform the same or not based on whether one was using its `.nativeDrv` or `.crossDrv`. This ugliness will disappear soon.
::: {.note}
If one explores Nixpkgs, they will see derivations with names like `gccCross`. Such `*Cross` derivations is a holdover from before we properly distinguished between the host and target platforms—the derivation with “Cross” in the name covered the `build = host != target` case, while the other covered the `host = target`, with build platform the same or not based on whether one was using its `.nativeDrv` or `.crossDrv`. This ugliness will disappear soon.
@ -130,7 +130,7 @@ Attribute Set `lib.platforms` defines [various common lists](https://github.com/
### `tests` {#var-meta-tests}
::: warning
::: {.warning}
This attribute is special in that it is not actually under the `meta` attribute set but rather under the `passthru` attribute set. This is due to how `meta` attributes work, and the fact that they are supposed to contain only metadata, not derivations.
:::
@ -175,20 +175,20 @@ The `meta.license` attribute should preferably contain a value from `lib.license
Although it’s typically better to indicate the specific license, a few generic options are available:
Unfree package that can be redistributed in binary form. That is, it’s legal to redistribute the *output* of the derivation. This means that the package can be included in the Nixpkgs channel.
Sometimes proprietary software can only be redistributed unmodified. Make sure the builder doesn’t actually modify the original binaries; otherwise we’re breaking the license. For instance, the NVIDIA X11 drivers can be redistributed unmodified, but our builder applies `patchelf` to make them work. Thus, its license is `"unfree"` and it cannot be included in the Nixpkgs channel.
Unfree package that cannot be redistributed. You can build it yourself, but you cannot redistribute the output of the derivation. Thus it cannot be included in the Nixpkgs channel.
This package supplies unfree, redistributable firmware. This is a separate value from `unfree-redistributable` because not everybody cares whether firmware is free.
@ -6,7 +6,7 @@ The Nix language allows a derivation to produce multiple outputs, which is simil
The main motivation is to save disk space by reducing runtime closure sizes; consequently also sizes of substituted binaries get reduced. Splitting can be used to have more granular runtime dependencies, for example the typical reduction is to split away development-only files, as those are typically not needed during runtime. As a result, closure sizes of many packages can get reduced to a half or even much less.
::: note
::: {.note}
The reduction effects could be instead achieved by building the parts in completely separate derivations. That would often additionally reduce build-time closures, but it tends to be much harder to write such derivations, as build systems typically assume all parts are being built at once. This compromise approach of single source package producing multiple binary packages is also utilized often by rpm and deb.
:::
@ -28,7 +28,7 @@ NixOS provides two ways to select the outputs to install for packages listed in
`nix-env` lacks an easy way to select the outputs to install. When installing a package, `nix-env` always installs the outputs listed in `meta.outputsToInstall`, even when the user explicitly selects an output.
::: warning
::: {.warning}
`nix-env` silenty disregards the outputs selected by the user, and instead installs the outputs from `meta.outputsToInstall`. For example,
installs the `"out"` output (`coreutils.meta.outputsToInstall` is `[ "out" ]`) instead of the requested `"info"`.
:::
The only recourse to select an output with `nix-env` is to override the package’s `meta.outputsToInstall`, using the functions described in <xreflinkend="chap-overrides"/>. For example, the following overlay adds the `"info"` output for the `coreutils` package:
The only recourse to select an output with `nix-env` is to override the package’s `meta.outputsToInstall`, using the functions described in [](#chap-overrides). For example, the following overlay adds the `"info"` output for the `coreutils` package:
```nix
self: super:
@ -53,7 +53,7 @@ self: super:
In the Nix language the individual outputs can be reached explicitly as attributes, e.g. `coreutils.info`, but the typical case is just using packages as build inputs.
When a multiple-output derivation gets into a build input of another derivation, the `dev` output is added if it exists, otherwise the first output is added. In addition to that, `propagatedBuildOutputs` of that package which by default contain `$outputBin` and `$outputLib` are also added. (See <xreflinkend="multiple-output-file-type-groups"/>.)
When a multiple-output derivation gets into a build input of another derivation, the `dev` output is added if it exists, otherwise the first output is added. In addition to that, `propagatedBuildOutputs` of that package which by default contain `$outputBin` and `$outputLib` are also added. (See [](#multiple-output-file-type-groups).)
In some cases it may be desirable to combine different outputs under a single store path. A function `symlinkJoin` can be used to do this. (Note that it may negate some closure size benefits of using a multiple-output package.)
Often such a single line is enough. For each output an equally named environment variable is passed to the builder and contains the path in nix store for that output. Typically you also want to have the main `out` output, as it catches any files that didn’t get elsewhere.
::: note
There is a special handling of the `debug` output, described at <xreflinkend="stdenv-separateDebugInfo"/>.
::: {.note}
There is a special handling of the `debug` output, described at [](#stdenv-separateDebugInfo).
@ -85,35 +85,35 @@ The reason for why `glibc` deviates from the convention is because referencing a
The support code currently recognizes some particular kinds of outputs and either instructs the build system of the package to put files into their desired outputs or it moves the files during the fixup phase. Each group of file types has an `outputFoo` variable specifying the output name where they should go. If that variable isn’t defined by the derivation writer, it is guessed – a default output name is defined, falling back to other possibilities if the output isn’t defined.
#### `$outputDev`
#### `$outputDev` {#outputdev}
is for development-only files. These include C(++) headers (`include/`), pkg-config (`lib/pkgconfig/`), cmake (`lib/cmake/`) and aclocal files (`share/aclocal/`). They go to `dev` or `out` by default.
#### `$outputBin`
#### `$outputBin` {#outputbin}
is meant for user-facing binaries, typically residing in `bin/`. They go to `bin` or `out` by default.
#### `$outputLib`
#### `$outputLib` {#outputlib}
is meant for libraries, typically residing in `lib/` and `libexec/`. They go to `lib` or `out` by default.
#### `$outputDoc`
#### `$outputDoc` {#outputdoc}
is for user documentation, typically residing in `share/doc/`. It goes to `doc` or `out` by default.
#### `$outputDevdoc`
#### `$outputDevdoc` {#outputdevdoc}
is for _developer_ documentation. Currently we count gtk-doc and devhelp books, typically residing in `share/gtk-doc/` and `share/devhelp/`, in there. It goes to `devdoc` or is removed (!) by default. This is because e.g. gtk-doc tends to be rather large and completely unused by nixpkgs users.
#### `$outputMan`
#### `$outputMan` {#outputman}
is for man pages (except for section 3), typically residing in `share/man/man[0-9]/`. They go to `man` or `$outputBin` by default.
#### `$outputDevman`
#### `$outputDevman` {#outputdevman}
is for section 3 man pages, typically residing in `share/man/man[0-9]/`. They go to `devman` or `$outputMan` by default.
#### `$outputInfo`
#### `$outputInfo` {#outputinfo}
is for info pages, typically residing in `share/info/`. They go to `info` or `$outputBin` by default.
This attribute ensures that the `bin` subdirectories of these packages appear in the `PATH` environment variable during the build, that their `include` subdirectories are searched by the C compiler, and so on. (See <xreflinkend="ssec-setup-hooks"/> for details.)
This attribute ensures that the `bin` subdirectories of these packages appear in the `PATH` environment variable during the build, that their `include` subdirectories are searched by the C compiler, and so on. (See [](#ssec-setup-hooks) for details.)
Often it is necessary to override or modify some aspect of the build. To make this easier, the standard environment breaks the package build into a number of *phases*, all of which can be overridden or modified individually: unpacking the sources, applying patches, configuring, building, and installing. (There are some others; see <xreflinkend="sec-stdenv-phases"/>.) For instance, a package that doesn’t supply a makefile but instead has to be compiled "manually" could be handled like this:
Often it is necessary to override or modify some aspect of the build. To make this easier, the standard environment breaks the package build into a number of *phases*, all of which can be overridden or modified individually: unpacking the sources, applying patches, configuring, building, and installing. (There are some others; see [](#sec-stdenv-phases).) For instance, a package that doesn’t supply a makefile but instead has to be compiled "manually" could be handled like this:
```nix
stdenv.mkDerivation {
@ -59,7 +59,7 @@ stdenv.mkDerivation {
(Note the use of `''`-style string literals, which are very convenient for large multi-line script fragments because they don’t need escaping of `"` and `\`, and because indentation is intelligently removed.)
There are many other attributes to customise the build. These are listed in <xreflinkend="ssec-stdenv-attributes"/>.
There are many other attributes to customise the build. These are listed in [](#ssec-stdenv-attributes).
While the standard environment provides a generic builder, you can still supply your own build script:
@ -116,9 +116,9 @@ On Linux, `stdenv` also includes the `patchelf` utility.
As described in the Nix manual, almost any `*.drv` store path in a derivation’s attribute set will induce a dependency on that derivation. `mkDerivation`, however, takes a few attributes intended to, between them, include all the dependencies of a package. This is done both for structure and consistency, but also so that certain other setup can take place. For example, certain dependencies need their bin directories added to the `PATH`. That is built-in, but other setup is done via a pluggable mechanism that works in conjunction with these dependency attributes. See <xreflinkend="ssec-setup-hooks"/> for details.
As described in the Nix manual, almost any `*.drv` store path in a derivation’s attribute set will induce a dependency on that derivation. `mkDerivation`, however, takes a few attributes intended to, between them, include all the dependencies of a package. This is done both for structure and consistency, but also so that certain other setup can take place. For example, certain dependencies need their bin directories added to the `PATH`. That is built-in, but other setup is done via a pluggable mechanism that works in conjunction with these dependency attributes. See [](#ssec-setup-hooks) for details.
Dependencies can be broken down along three axes: their host and target platforms relative to the new derivation’s, and whether they are propagated. The platform distinctions are motivated by cross compilation; see <xreflinkend="chap-cross"/> for exactly what each platform means. [^footnote-stdenv-ignored-build-platform] But even if one is not cross compiling, the platforms imply whether or not the dependency is needed at run-time or build-time, a concept that makes perfect sense outside of cross compilation. By default, the run-time/build-time distinction is just a hint for mental clarity, but with `strictDeps` set it is mostly enforced even in the native case.
Dependencies can be broken down along three axes: their host and target platforms relative to the new derivation’s, and whether they are propagated. The platform distinctions are motivated by cross compilation; see [](#chap-cross) for exactly what each platform means. [^footnote-stdenv-ignored-build-platform] But even if one is not cross compiling, the platforms imply whether or not the dependency is needed at run-time or build-time, a concept that makes perfect sense outside of cross compilation. By default, the run-time/build-time distinction is just a hint for mental clarity, but with `strictDeps` set it is mostly enforced even in the native case.
The extension of `PATH` with dependencies, alluded to above, proceeds according to the relative platforms alone. The process is carried out only for dependencies whose host platform matches the new derivation’s build platform i.e. dependencies which run on the platform where the new derivation will be built. [^footnote-stdenv-native-dependencies-in-path] For each dependency \<dep\> of those dependencies, `dep/bin`, if present, is added to the `PATH` environment variable.
@ -175,7 +175,8 @@ Because of the bounds checks, the uncommon cases are `h = t` and `h + 2 = t`. In
Overall, the unifying theme here is that propagation shouldn’t be introducing transitive dependencies involving platforms the depending package is unaware of. \[One can imagine the dependending package asking for dependencies with the platforms it knows about; other platforms it doesn’t know how to ask for. The platform description in that scenario is a kind of unforagable capability.\] The offset bounds checking and definition of `mapOffset` together ensure that this is the case. Discovering a new offset is discovering a new platform, and since those platforms weren’t in the derivation “spec” of the needing package, they cannot be relevant. From a capability perspective, we can imagine that the host and target platforms of a package are the capabilities a package requires, and the depending package must provide the capability to the dependency.
A list of dependencies whose host and target platforms are the new derivation’s build platform. This means a `-1` host and `-1` target offset from the new derivation’s platforms. These are programs and libraries used at build time that produce programs and libraries also used at build time. If the dependency doesn’t care about the target platform (i.e. isn’t a compiler or similar tool), put it in `nativeBuildInputs` instead. The most common use of this `buildPackages.stdenv.cc`, the default C compiler for this role. That example crops up more than one might think in old commonly used C libraries.
@ -236,13 +237,13 @@ The propagated equivalent of `depsTargetTarget`. This is prefixed for the same r
A natural number indicating how much information to log. If set to 1 or higher, `stdenv` will print moderate debugging information during the build. In particular, the `gcc` and `ld` wrapper scripts will print out the complete command line passed to the wrapped tools. If set to 6 or higher, the `stdenv` setup script will be run with `set -x` tracing. If set to 7 or higher, the `gcc` and `ld` wrapper scripts will also be run with `set -x` tracing.
The script will be run with `UPDATE_NIX_ATTR_PATH` environment variable set to the attribute path it is supposed to update.
::: note
::: {.note}
The script will be usually run from the root of the Nixpkgs repository but you should not rely on that. Also note that the update scripts will be run in parallel by default; you should avoid running `git commit` or any other commands that cannot handle that.
:::
@ -314,7 +315,7 @@ Each phase can be overridden in its entirety either by setting the environment v
There are a number of variables that control what phases are executed and in what order:
#### Variables affecting phase control
#### Variables affecting phase control {#variables-affecting-phase-control}
##### `phases` {#var-stdenv-phases}
@ -354,21 +355,22 @@ Additional phases executed after any of the default phases.
The unpack phase is responsible for unpacking the source code of the package. The default implementation of `unpackPhase` unpacks the source files listed in the `src` environment variable to the current directory. It supports the following files by default:
#### Tar files
#### Tar files {#tar-files}
These can optionally be compressed using `gzip` (`.tar.gz`, `.tgz` or `.tar.Z`), `bzip2` (`.tar.bz2`, `.tbz2` or `.tbz`) or `xz` (`.tar.xz`, `.tar.lzma` or `.txz`).
#### Zip files
#### Zip files {#zip-files}
Zip files are unpacked using `unzip`. However, `unzip` is not in the standard environment, so you should add it to `nativeBuildInputs` yourself.
#### Directories in the Nix store
#### Directories in the Nix store {#directories-in-the-nix-store}
These are simply copied to the current directory. The hash part of the file name is stripped, e.g. `/nix/store/1wydxgby13cz...-my-sources` would be copied to `my-sources`.
Additional file types can be supported by setting the `unpackCmd` variable (see below).
#### Variables controlling the unpack phase
#### Variables controlling the unpack phase {#variables-controlling-the-unpack-phase}
##### `srcs` / `src` {#var-stdenv-src}
The list of source files or directories to be unpacked or copied. One of these must be set.
@ -405,7 +407,7 @@ The unpack phase evaluates the string `$unpackCmd` for any unrecognised file. Th
The patch phase applies the list of patches defined in the `patches` variable.
#### Variables controlling the patch phase
#### Variables controlling the patch phase {#variables-controlling-the-patch-phase}
##### `dontPatch` {#var-stdenv-dontPatch}
@ -431,7 +433,7 @@ Hook executed at the end of the patch phase.
The configure phase prepares the source tree for building. The default `configurePhase` runs `./configure` (typically an Autoconf-generated script) if it exists.
#### Variables controlling the configure phase
#### Variables controlling the configure phase {#variables-controlling-the-configure-phase}
By default, the flag `--disable-dependency-tracking` is added to the configure flags to speed up Automake-based builds. If this is undesirable, set this variable to true.
@ -473,7 +481,7 @@ By default, the configure phase applies some special hackery to all files called
By default, when the configure script has `--enable-static`, the option `--disable-static` is added to the configure flags.
If this is undesirable, set this variable to true.
If this is undesirable, set this variable to true. It is automatically set to true when building statically, for example through `pkgsStatic`.
@ -491,7 +499,7 @@ Hook executed at the end of the configure phase.
The build phase is responsible for actually building the package (e.g. compiling it). The default `buildPhase` simply calls `make` if a file named `Makefile`, `makefile` or `GNUmakefile` exists in the current directory (or the `makefile` is explicitly set); otherwise it does nothing.
#### Variables controlling the build phase
#### Variables controlling the build phase {#variables-controlling-the-build-phase}
##### `dontBuild` {#var-stdenv-dontBuild}
@ -509,7 +517,7 @@ A list of strings passed as additional flags to `make`. These flags are also use
makeFlags = [ "PREFIX=$(out)" ];
```
::: note
::: {.note}
The flags are quoted in bash, but environment variables can be specified by using the make syntax.
:::
@ -545,7 +553,7 @@ Before and after running `make`, the hooks `preBuild` and `postBuild` are called
The check phase checks whether the package was built correctly by running its test suite. The default `checkPhase` calls `make check`, but only if the `doCheck` variable is enabled.
#### Variables controlling the check phase
#### Variables controlling the check phase {#variables-controlling-the-check-phase}
##### `doCheck` {#var-stdenv-doCheck}
@ -557,7 +565,7 @@ doCheck = true;
in the derivation to enable checks. The exception is cross compilation. Cross compiled builds never run tests, no matter how `doCheck` is set, as the newly-built program won’t run on the platform used to build it.
See the [build phase](#var-stdenv-makeFlags) for details.
@ -585,13 +593,13 @@ Hook executed at the end of the check phase.
The install phase is responsible for installing the package in the Nix store under `out`. The default `installPhase` creates the directory `$out` and calls `make install`.
#### Variables controlling the install phase
#### Variables controlling the install phase {#variables-controlling-the-install-phase}
See the [build phase](#var-stdenv-makeFlags) for details.
@ -624,7 +632,7 @@ The fixup phase performs some (Nix-specific) post-processing actions on the file
- On Linux, it applies the `patchelf` command to ELF executables and libraries to remove unused directories from the `RPATH` in order to prevent unnecessary runtime dependencies.
- It rewrites the interpreter paths of shell scripts to paths found in `PATH`. E.g., `/usr/bin/perl` will be rewritten to `/nix/store/some-perl/bin/perl` found in `PATH`.
#### Variables controlling the fixup phase
#### Variables controlling the fixup phase {#variables-controlling-the-fixup-phase}
##### `dontFixup` {#var-stdenv-dontFixup}
@ -706,7 +714,7 @@ to `~/.gdbinit`. GDB will then be able to find debug information installed via `
The installCheck phase checks whether the package was installed correctly by running its test suite against the installed directories. The default `installCheck` calls `make installcheck`.
#### Variables controlling the installCheck phase
#### Variables controlling the installCheck phase {#variables-controlling-the-installcheck-phase}
@ -742,7 +750,7 @@ Hook executed at the end of the installCheck phase.
The distribution phase is intended to produce a source distribution of the package. The default `distPhase` first calls `make dist`, then it copies the resulting source tarballs to `$out/tarballs/`. This phase is only executed if the attribute `doDist` is set.
#### Variables controlling the distribution phase
#### Variables controlling the distribution phase {#variables-controlling-the-distribution-phase}
##### `distTarget` {#var-stdenv-distTarget}
@ -879,7 +887,7 @@ The most typical use of the setup hook is actually to add other hooks which are
Packages adding a hook should not hard code a specific hook, but rather choose a variable *relative* to how they are included. Returning to the C compiler wrapper example, if the wrapper itself is an `n` dependency, then it only wants to accumulate flags from `n + 1` dependencies, as only those ones match the compiler’s target platform. The `hostOffset` variable is defined with the current dependency’s host offset `targetOffset` with its target offset, before its setup hook is sourced. Additionally, since most environment hooks don’t care about the target platform, that means the setup hook can append to the right bash array by doing something like
```{.bash}
```bash
addEnvHooks "$hostOffset" myBashFunction
```
@ -887,47 +895,47 @@ The *existence* of setups hooks has long been documented and packages inside Nix
First, let’s cover some setup hooks that are part of Nixpkgs default stdenv. This means that they are run for every package built using `stdenv.mkDerivation`. Some of these are platform specific, so they may run on Linux but not Darwin or vice-versa.
### `move-docs.sh`
### `move-docs.sh` {#move-docs.sh}
This setup hook moves any installed documentation to the `/share` subdirectory directory. This includes the man, doc and info directories. This is needed for legacy programs that do not know how to use the `share` subdirectory.
This setup hook compresses any man pages that have been installed. The compression is done using the gzip program. This helps to reduce the installed size of packages.
### `strip.sh`
### `strip.sh` {#strip.sh}
This runs the strip command on installed binaries and libraries. This removes unnecessary information like debug symbols when they are not needed. This also helps to reduce the installed size of packages.
### `patch-shebangs.sh`
### `patch-shebangs.sh` {#patch-shebangs.sh}
This setup hook patches installed scripts to use the full path to the shebang interpreter. A shebang interpreter is the first commented line of a script telling the operating system which program will run the script (e.g `#!/bin/bash`). In Nix, we want an exact path to that interpreter to be used. This often replaces `/bin/sh` with a path in the Nix store.
### `audit-tmpdir.sh`
### `audit-tmpdir.sh` {#audit-tmpdir.sh}
This verifies that no references are left from the install binaries to the directory used to build those binaries. This ensures that the binaries do not need things outside the Nix store. This is currently supported in Linux only.
### `multiple-outputs.sh`
### `multiple-outputs.sh` {#multiple-outputs.sh}
This setup hook adds configure flags that tell packages to install files into any one of the proper outputs listed in `outputs`. This behavior can be turned off by setting `setOutputFlags` to false in the derivation environment. See <xreflinkend="chap-multiple-output"/> for more information.
This setup hook adds configure flags that tell packages to install files into any one of the proper outputs listed in `outputs`. This behavior can be turned off by setting `setOutputFlags` to false in the derivation environment. See [](#chap-multiple-output) for more information.
### `move-sbin.sh`
### `move-sbin.sh` {#move-sbin.sh}
This setup hook moves any binaries installed in the `sbin/` subdirectory into `bin/`. In addition, a link is provided from `sbin/` to `bin/` for compatibility.
### `move-lib64.sh`
### `move-lib64.sh` {#move-lib64.sh}
This setup hook moves any libraries installed in the `lib64/` subdirectory into `lib/`. In addition, a link is provided from `lib64/` to `lib/` for compatibility.
This setup hook moves any systemd user units installed in the `lib/` subdirectory into `share/`. In addition, a link is provided from `share/` to `lib/` for compatibility. This is needed for systemd to find user services when installed into the user profile.
This sets `SOURCE_DATE_EPOCH` to the modification time of the most recent file.
### Bintools Wrapper
### Bintools Wrapper {#bintools-wrapper}
The Bintools Wrapper wraps the binary utilities for a bunch of miscellaneous purposes. These are GNU Binutils when targetting Linux, and a mix of cctools and GNU binutils for Darwin. \[The “Bintools” name is supposed to be a compromise between “Binutils” and “cctools” not denoting any specific implementation.\] Specifically, the underlying bintools package, and a C standard library (glibc or Darwin’s libSystem, just for the dynamic loader) are all fed in, and dependency finding, hardening (see below), and purity checks for each are handled by the Bintools Wrapper. Packages typically depend on CC Wrapper, which in turn (at run time) depends on the Bintools Wrapper.
@ -937,7 +945,7 @@ A final task of the setup hook is defining a number of standard environment vari
A problem with this final task is that the Bintools Wrapper is honest and defines `LD` as `ld`. Most packages, however, firstly use the C compiler for linking, secondly use `LD` anyways, defining it as the C compiler, and thirdly, only so define `LD` when it is undefined as a fallback. This triple-threat means Bintools Wrapper will break those packages, as LD is already defined as the actual linker which the package won’t override yet doesn’t want to use. The workaround is to define, just for the problematic package, `LD` as the C compiler. A good way to do this would be `preConfigure = "LD=$CC"`.
### CC Wrapper
### CC Wrapper {#cc-wrapper}
The CC Wrapper wraps a C toolchain for a bunch of miscellaneous purposes. Specifically, a C compiler (GCC or Clang), wrapped binary tools, and a C standard library (glibc or Darwin’s libSystem, just for the dynamic loader) are all fed in, and dependency finding, hardening (see below), and purity checks for each are handled by the CC Wrapper. Packages typically depend on the CC Wrapper, which in turn (at run-time) depends on the Bintools Wrapper.
@ -971,11 +979,11 @@ The `autoreconfHook` derivation adds `autoreconfPhase`, which runs autoreconf, l
Adds every file named `catalog.xml` found under the `xml/dtd` and `xml/xsl` subdirectories of each build input to the `XML_CATALOG_FILES` environment variable.
### teTeX / TeX Live
### teTeX / TeX Live {#tetex-tex-live}
Adds the `share/texmf-nix` subdirectory of each build input to the `TEXINPUTS` environment variable.
### Qt 4
### Qt 4 {#qt-4}
Sets the `QTDIR` environment variable to Qt’s path.
@ -983,13 +991,13 @@ Sets the `QTDIR` environment variable to Qt’s path.
Exports `GDK_PIXBUF_MODULE_FILE` environment variable to the builder. Add librsvg package to `buildInputs` to get svg support. See also the [setup hook description in GNOME platform docs](#ssec-gnome-hooks-gdk-pixbuf).
### GHC
### GHC {#ghc}
Creates a temporary package database and registers every Haskell build input in it (TODO: how?).
### GNOME platform
### GNOME platform {#gnome-platform}
Hooks related to GNOME platform and related libraries like GLib, GTK and GStreamer are described in <xreflinkend="sec-language-gnome"/>.
Hooks related to GNOME platform and related libraries like GLib, GTK and GStreamer are described in [](#sec-language-gnome).
@ -1003,7 +1011,7 @@ By default `autoPatchelf` will fail as soon as any ELF file requires a dependenc
The `autoPatchelf` command also recognizes a `--no-recurse` command line flag, which prevents it from recursing into subdirectories.
### breakpointHook
### breakpointHook {#breakpointhook}
This hook will make a build pause instead of stopping when a failure happens. It prevents nix from cleaning up the build environment immediately and allows the user to attach to a build environment using the `cntr` command. Upon build error it will print instructions on how to use `cntr`, which can be used to enter the environment for debugging. Installing cntr and running the command will provide shell access to the build sandbox of failed build. At `/var/lib/cntr` the sandboxed filesystem is mounted. All commands and files of the system are still accessible within the shell. To execute commands from the sandbox use the cntr exec subcommand. `cntr` is only supported on Linux-based platforms. To use it first add `cntr` to your `environment.systemPackages` on NixOS or alternatively to the root user on non-NixOS systems. Then in the package that is supposed to be inspected, add `breakpointHook` to `nativeBuildInputs`.
When a build failure happens there will be an instruction printed that shows how to attach with `cntr` to the build sandbox.
::: note
::: title
::: {.note}
::: {.title}
Caution with remote builds
:::
This won’t work with remote builds as the build environment is on a different machine and can’t be accessed by `cntr`. Remote builds can be turned off by setting `--option builders ''` for `nix-build` or `--builders ''` for `nix build`.
:::
### installShellFiles
### installShellFiles {#installshellfiles}
This hook helps with installing manpages and shell completion files. It exposes 2 shell functions `installManPage` and `installShellCompletion` that can be used from your `postInstall` hook.
@ -1047,61 +1055,61 @@ postInstall = ''
'';
```
### libiconv, libintl
### libiconv, libintl {#libiconv-libintl}
A few libraries automatically add to `NIX_LDFLAGS` their library, making their symbols automatically available to the linker. This includes libiconv and libintl (gettext). This is done to provide compatibility between GNU Linux, where libiconv and libintl are bundled in, and other systems where that might not be the case. Sometimes, this behavior is not desired. To disable this behavior, set `dontAddExtraLibs`.
### validatePkgConfig
### validatePkgConfig {#validatepkgconfig}
The `validatePkgConfig` hook validates all pkg-config (`.pc`) files in a package. This helps catching some common errors in pkg-config files, such as undefined variables.
### cmake
### cmake {#cmake}
Overrides the default configure phase to run the CMake command. By default, we use the Make generator of CMake. In addition, dependencies are added automatically to CMAKE_PREFIX_PATH so that packages are correctly detected by CMake. Some additional flags are passed in to give similar behavior to configure-based packages. You can disable this hook’s behavior by setting configurePhase to a custom value, or by setting dontUseCmakeConfigure. cmakeFlags controls flags passed only to CMake. By default, parallel building is enabled as CMake supports parallel building almost everywhere. When Ninja is also in use, CMake will detect that and use the ninja generator.
### xcbuildHook
### xcbuildHook {#xcbuildhook}
Overrides the build and install phases to run the "xcbuild" command. This hook is needed when a project only comes with build files for the XCode build system. You can disable this behavior by setting buildPhase and configurePhase to a custom value. xcbuildFlags controls flags passed only to xcbuild.
### Meson
### Meson {#meson}
Overrides the configure phase to run meson to generate Ninja files. To run these files, you should accompany Meson with ninja. By default, `enableParallelBuilding` is enabled as Meson supports parallel building almost everywhere.
Which [`--buildtype`](https://mesonbuild.com/Builtin-options.html#core-options) to pass to Meson. We default to `plain`.
##### `mesonAutoFeatures`
##### `mesonAutoFeatures` {#mesonautofeatures}
What value to set [`-Dauto_features=`](https://mesonbuild.com/Builtin-options.html#core-options) to. We default to `enabled`.
##### `mesonWrapMode`
##### `mesonWrapMode` {#mesonwrapmode}
What value to set [`-Dwrap_mode=`](https://mesonbuild.com/Builtin-options.html#core-options) to. We default to `nodownload` as we disallow network access.
Overrides the build, install, and check phase to run ninja instead of make. You can disable this behavior with the `dontUseNinjaBuild`, `dontUseNinjaInstall`, and `dontUseNinjaCheck`, respectively. Parallel building is enabled by default in Ninja.
### unzip
### unzip {#unzip}
This setup hook will allow you to unzip .zip files specified in `$src`. There are many similar packages like `unrar`, `undmg`, etc.
### wafHook
### wafHook {#wafhook}
Overrides the configure, build, and install phases. This will run the “waf” script used by many projects. If `wafPath` (default `./waf`) doesn’t exist, it will copy the version of waf available in Nixpkgs. `wafFlags` can be used to pass flags to the waf script.
### scons
### scons {#scons}
Overrides the build, install, and check phases. This uses the scons build system as a replacement for make. scons does not provide a configure phase, so everything is managed at build and install time.
@ -1119,7 +1127,7 @@ Both parameters take a list of flags as strings. The special `"all"` flag can be
The following flags are enabled by default and might require disabling with `hardeningDisable` if the program to package is incompatible.
### `format`
### `format` {#format}
Adds the `-Wformat -Wformat-security -Werror=format-security` compiler options. At present, this warns about calls to `printf` and `scanf` functions where the format string is not a string literal and there are no format arguments, as in `printf(foo);`. This may be a security hole if the format string came from untrusted input and contains `%n`.
@ -1132,7 +1140,7 @@ This needs to be turned off or fixed for errors similar to:
cc1plus: some warnings being treated as errors
```
### `stackprotector`
### `stackprotector` {#stackprotector}
Adds the `-fstack-protector-strong --param ssp-buffer-size=4` compiler options. This adds safety checks against stack overwrites rendering many potential code injection attacks into aborting situations. In the best case this turns code injection vulnerabilities into denial of service or into non-issues (depending on the application).
@ -1143,7 +1151,7 @@ bin/blib.a(bios_console.o): In function `bios_handle_cup':
/tmp/nix-build-ipxe-20141124-5cbdc41.drv-0/ipxe-5cbdc41/src/arch/i386/firmware/pcbios/bios_console.c:86: undefined reference to `__stack_chk_fail'
```
### `fortify`
### `fortify` {#fortify}
Adds the `-O2 -D_FORTIFY_SOURCE=2` compiler options. During code generation the compiler knows a great deal of information about buffer sizes (where possible), and attempts to replace insecure unlimited length buffer function calls with length-limited ones. This is especially useful for old, crufty code. Additionally, format strings in writable memory that contain `%n` are blocked. If an application depends on such a format string, it will need to be worked around.
@ -1164,7 +1172,7 @@ installwatch.c:3751:5: error: conflicting types for '__open_2'
fcntl2.h:50:4: error: call to '__open_missing_mode' declared with attribute error: open with O_CREAT or O_TMPFILE in second argument needs 3 arguments
```
### `pic`
### `pic` {#pic}
Adds the `-fPIC` compiler options. This options adds support for position independent code in shared libraries and thus making ASLR possible.
ccbLfRgg.s:33: Error: missing or invalid displacement expression `private_key_len@GOTOFF'
```
### `strictoverflow`
### `strictoverflow` {#strictoverflow}
Signed integer overflow is undefined behaviour according to the C standard. If it happens, it is an error in the program as it should check for overflow before it can happen, not afterwards. GCC provides built-in functions to perform arithmetic with overflow checking, which are correct and faster than any custom implementation. As a workaround, the option `-fno-strict-overflow` makes gcc behave as if signed integer overflows were defined.
This flag should not trigger any build or runtime errors.
### `relro`
### `relro` {#relro}
Adds the `-z relro` linker option. During program load, several ELF memory sections need to be written to by the linker, but can be turned read-only before turning over control to the program. This prevents some GOT (and .dtors) overwrite attacks, but at least the part of the GOT used by the dynamic linker (.got.plt) is still vulnerable.
This flag can break dynamic shared object loading. For instance, the module systems of Xorg and OpenCV are incompatible with this flag. In almost all cases the `bindnow` flag must also be disabled and incompatible programs typically fail with similar errors at runtime.
### `bindnow`
### `bindnow` {#bindnow}
Adds the `-z bindnow` linker option. During program load, all dynamic symbols are resolved, allowing for the complete GOT to be marked read-only (due to `relro`). This prevents GOT overwrite attacks. For very large applications, this can incur some performance loss during initial load while symbols are resolved, but this shouldn’t be an issue for daemons.
The following flags are disabled by default and should be enabled with `hardeningEnable` for packages that take untrusted input like network services.
### `pie`
### `pie` {#pie}
Adds the `-fPIE` compiler and `-pie` linker options. Position Independent Executables are needed to take advantage of Address Space Layout Randomization, supported by modern kernel versions. While ASLR can already be enforced for data areas in the stack and heap (brk and mmap), the code areas must be compiled as position-independent. Shared libraries already do this with the `pic` flag, so they gain ASLR automatically, but binary .text regions need to be build with `pie` to gain ASLR. When this happens, ROP attacks are much harder since there are no static locations to bounce off of during a memory corruption attack.
@ -63,7 +63,7 @@ The second argument (`super`) corresponds to the result of the evaluation of the
The value returned by this function should be a set similar to `pkgs/top-level/all-packages.nix`, containing overridden and/or new packages.
Overlays are similar to other methods for customizing Nixpkgs, in particular the `packageOverrides` attribute described in <xreflinkend="sec-modify-via-packageOverrides"/>. Indeed, `packageOverrides` acts as an overlay with only the `super` argument. It is therefore appropriate for basic use, but overlays are more powerful and easier to distribute.
Overlays are similar to other methods for customizing Nixpkgs, in particular the `packageOverrides` attribute described in [](#sec-modify-via-packageOverrides). Indeed, `packageOverrides` acts as an overlay with only the `super` argument. It is therefore appropriate for basic use, but overlays are more powerful and easier to distribute.
## Using overlays to configure alternatives {#sec-overlays-alternatives}
@ -48,17 +48,17 @@ In the above example, the `separateDebugInfo` attribute is overridden to be true
The argument `oldAttrs` is conventionally used to refer to the attr set originally passed to `stdenv.mkDerivation`.
::: note
Note that `separateDebugInfo` is processed only by the `stdenv.mkDerivation` function, not the generated, raw Nix derivation. Thus, using `overrideDerivation` will not work in this case, as it overrides only the attributes of the final derivation. It is for this reason that `overrideAttrs` should be preferred in (almost) all cases to `overrideDerivation`, i.e. to allow using `stdenv.mkDerivation` to process input arguments, as well as the fact that it is easier to use (you can use the same attribute names you see in your Nix code, instead of the ones generated (e.g. `buildInputs` vs `nativeBuildInputs`), and it involves less typing).
::: {.note}
Note that `separateDebugInfo` is processed only by the `stdenv.mkDerivation` function, not the generated, raw Nix derivation. Thus, using `overrideDerivation` will not work in this case, as it overrides only the attributes of the final derivation. It is for this reason that `overrideAttrs` should be preferred in (almost) all cases to `overrideDerivation`, i.e. to allow using `stdenv.mkDerivation` to process input arguments, as well as the fact that it is easier to use (you can use the same attribute names you see in your Nix code, instead of the ones generated (e.g. `buildInputs` vs `nativeBuildInputs`), and it involves less typing).
You should prefer `overrideAttrs` in almost all cases, see its documentation for the reasons why. `overrideDerivation` is not deprecated and will continue to work, but is less nice to use and does not have as many abilities as `overrideAttrs`.
:::
::: warning
::: {.warning}
Do not use this function in Nixpkgs as it evaluates a Derivation before modifying it, which breaks package abstraction and removes error-checking of function arguments. In addition, this evaluation-per-function application incurs a performance penalty, which can become a problem if many overrides are used. It is only intended for ad-hoc customisation, such as in `~/.config/nixpkgs/config.nix`.
:::
@ -81,8 +81,8 @@ In the above example, the `name`, `src`, and `patches` of the derivation will be
The argument `oldAttrs` is used to refer to the attribute set of the original derivation.
::: note
A package's attributes are evaluated *before* being modified by the `overrideDerivation` function. For example, the `name` attribute reference in `url = "mirror://gnu/hello/${name}.tar.gz";` is filled-in *before* the `overrideDerivation` function modifies the attribute set. This means that overriding the `name` attribute, in this example, *will not* change the value of the `url` attribute. Instead, we need to override both the `name`*and*`url` attributes.
::: {.note}
A package's attributes are evaluated *before* being modified by the `overrideDerivation` function. For example, the `name` attribute reference in `url = "mirror://gnu/hello/${name}.tar.gz";` is filled-in *before* the `overrideDerivation` function modifies the attribute set. This means that overriding the `name` attribute, in this example, *will not* change the value of the `url` attribute. Instead, we need to override both the `name`*and*`url` attributes.
If NixOS fails to boot, there are a number of kernel command line parameters that may help you to identify or fix the issue. You can add these parameters in the GRUB boot menu by pressing “e” to modify the selected boot entry and editing the line starting with `linux`. The following are some useful kernel command line parameters that are recognised by the NixOS boot scripts or by systemd:
`boot.shell_on_fail`
: Allows the user to start a root shell if something goes wrong in stage 1 of the boot process (the initial ramdisk). This is disabled by default because there is no authentication for the root shell.
`boot.debug1`
: Start an interactive shell in stage 1 before anything useful has been done. That is, no modules have been loaded and no file systems have been mounted, except for `/proc` and `/sys`.
`boot.debug1devices`
: Like `boot.debug1`, but runs stage1 until kernel modules are loaded and device nodes are created. This may help with e.g. making the keyboard work.
`boot.debug1mounts`
: Like `boot.debug1` or `boot.debug1devices`, but runs stage1 until all filesystems that are mounted during initrd are mounted (see [neededForBoot](#opt-fileSystems._name_.neededForBoot)). As a motivating example, this could be useful if you've forgotten to set [neededForBoot](options.html#opt-fileSystems._name_.neededForBoot) on a file system.
`boot.trace`
: Print every shell command executed by the stage 1 and 2 boot scripts.
`single`
: Boot into rescue mode (a.k.a. single user mode). This will cause systemd to start nothing but the unit `rescue.target`, which runs `sulogin` to prompt for the root password and start a root login shell. Exiting the shell causes the system to continue with the normal boot process.
: Make systemd very verbose and send log messages to the console instead of the journal. For more parameters recognised by systemd, see systemd(1).
Notice that for `boot.shell_on_fail`, `boot.debug1`, `boot.debug1devices`, and `boot.debug1mounts`, if you did **not** select "start the new shell as pid 1", and you `exit` from the new shell, boot will proceed normally from the point where it failed, as if you'd chosen "ignore the error and continue".
If no login prompts or X11 login screens appear (e.g. due to hanging dependencies), you can press Alt+ArrowUp. If you’re lucky, this will start rescue mode (described above). (Also note that since most units have a 90-second timeout before systemd gives up on them, the `agetty` login prompts should appear eventually unless something is very wrong.)
If you find yourself repeating yourself over and over, it’s time to abstract. Take, for instance, this Apache HTTP Server configuration:
```nix
{
services.httpd.virtualHosts =
{ "blog.example.org" = {
documentRoot = "/webroot/blog.example.org";
adminAddr = "alice@example.org";
forceSSL = true;
enableACME = true;
enablePHP = true;
};
"wiki.example.org" = {
documentRoot = "/webroot/wiki.example.org";
adminAddr = "alice@example.org";
forceSSL = true;
enableACME = true;
enablePHP = true;
};
};
}
```
It defines two virtual hosts with nearly identical configuration; the only difference is the document root directories. To prevent this duplication, we can use a `let`:
The `let commonConfig = ...` defines a variable named `commonConfig`. The `//` operator merges two attribute sets, so the configuration of the second virtual host is the set `commonConfig` extended with the document root option.
You can write a `let` wherever an expression is allowed. Thus, you also could have written:
```nix
{
services.httpd.virtualHosts =
let commonConfig = ...; in
{ "blog.example.org" = (commonConfig // { ... })
"wiki.example.org" = (commonConfig // { ... })
};
}
```
but not `{ let commonConfig = ...; in ...; }` since attributes (as opposed to attribute values) are not expressions.
**Functions** provide another method of abstraction. For instance, suppose that we want to generate lots of different virtual hosts, all with identical configuration except for the document root. This can be done as follows:
Here, `makeVirtualHost` is a function that takes a single argument `webroot` and returns the configuration for a virtual host. That function is then called for several names to produce the list of virtual host configurations.
When configuration problems are detectable in a module, it is a good idea to write an assertion or warning. Doing so provides clear feedback to the user and prevents errors after the build.
Although Nix has the `abort` and `builtins.trace` [functions](https://nixos.org/nix/manual/#ssec-builtins) to perform such tasks, they are not ideally suited for NixOS modules. Instead of these functions, you can declare your warnings and assertions using the NixOS module system.
## Warnings {#sec-assertions-warnings}
This is an example of using `warnings`.
```nix
{ config, lib, ... }:
{
config = lib.mkIf config.services.foo.enable {
warnings =
if config.services.foo.bar
then [ ''You have enabled the bar feature of the foo service.
This is known to cause some specific problems in certain situations.
'' ]
else [];
}
}
```
## Assertions {#sec-assertions-assetions}
This example, extracted from the [`syslogd` module](https://github.com/NixOS/nixpkgs/blob/release-17.09/nixos/modules/services/logging/syslogd.nix) shows how to use `assertions`. Since there can only be one active syslog daemon at a time, an assertion is useful to prevent such a broken system from being built.