commit
9838b6a794
GPG Key ID:
F972AEEA2887D547
@ -0,0 +1,256 @@ |
||||
# Javascript {#language-javascript} |
||||
|
||||
## Introduction {#javascript-introduction} |
||||
|
||||
This contains instructions on how to package javascript applications. For instructions on how to add a cli package from npm please consult the #node.js section |
||||
|
||||
The various tools available will be listed in the [tools-overview](#javascript-tools-overview). Some general principles for packaging will follow. Finally some tool specific instructions will be given. |
||||
|
||||
## Getting unstuck / finding code examples |
||||
|
||||
If you find you are lacking inspiration for packing javascript applications, the links below might prove useful. |
||||
Searching online for prior art can be helpful if you are running into solved problems. |
||||
|
||||
### Github |
||||
|
||||
- Searching Nix files for `mkYarnPackage`: <https://github.com/search?q=mkYarnPackage+language%3ANix&type=code> |
||||
|
||||
- Searching just `flake.nix` files for `mkYarnPackage`: <https://github.com/search?q=mkYarnPackage+filename%3Aflake.nix&type=code> |
||||
|
||||
### Gitlab |
||||
|
||||
- Searching Nix files for `mkYarnPackage`: <https://gitlab.com/search?scope=blobs&search=mkYarnPackage+extension%3Anix> |
||||
|
||||
- Searching just `flake.nix` files for `mkYarnPackage`: <https://gitlab.com/search?scope=blobs&search=mkYarnPackage+filename%3Aflake.nix> |
||||
|
||||
## Tools overview {#javascript-tools-overview} |
||||
|
||||
## General principles {#javascript-general-principles} |
||||
|
||||
The following principles are given in order of importance with potential exceptions. |
||||
|
||||
### Try to use the same node version used upstream {#javascript-upstream-node-version} |
||||
|
||||
It is often not documented which node version is used upstream, but if it is, try to use the same version when packaging. |
||||
|
||||
This can be a problem if upstream is using the latest and greatest and you are trying to use an earlier version of node. Some cryptic errors regarding V8 may appear. |
||||
|
||||
An exception to this: |
||||
|
||||
### Try to respect the package manager originally used by upstream (and use the upstream lock file) {#javascript-upstream-package-manager} |
||||
|
||||
A lock file (package-lock.json, yarn.lock...) is supposed to make reproducible installations of node_modules for each tool. |
||||
|
||||
Guidelines of package managers, recommend to commit those lock files to the repos. If a particular lock file is present, it is a strong indication of which package manager is used upstream. |
||||
|
||||
It's better to try to use a nix tool that understand the lock file. Using a different tool might give you hard to understand error because different packages have been installed. An example of problems that could arise can be found [here](https://github.com/NixOS/nixpkgs/pull/126629). Upstream uses npm, but this is an attempt to package it with yarn2nix (that uses yarn.lock) |
||||
|
||||
Using a different tool forces to commit a lock file to the repository. Those files are fairly large, so when packaging for nixpkgs, this approach does not scale well. |
||||
|
||||
Exceptions to this rule are: |
||||
|
||||
- when you encounter one of the bugs from a nix tool. In each of the tool specific instructions, known problems will be detailed. If you have a problem with a particular tool, then it's best to try another tool, even if this means you will have to recreate a lock file and commit it to nixpkgs. In general yarn2nix has less known problems and so a simple search in nixpkgs will reveal many yarn.lock files committed |
||||
- Some lock files contain particular version of a package that has been pulled off npm for some reason. In that case, you can recreate upstream lock (by removing the original and `npm install`, `yarn`, ...) and commit this to nixpkgs. |
||||
- The only tool that supports workspaces (a feature of npm that helps manage sub-directories with different package.json from a single top level package.json) is yarn2nix. If upstream has workspaces you should try yarn2nix. |
||||
|
||||
### Try to use upstream package.json {#javascript-upstream-package-json} |
||||
|
||||
Exceptions to this rule are |
||||
|
||||
- Sometimes the upstream repo assumes some dependencies be installed globally. In that case you can add them manually to the upstream package.json (`yarn add xxx` or `npm install xxx`, ...). Dependencies that are installed locally can be executed with `npx` for cli tools. (e.g. `npx postcss ...`, this is how you can call those dependencies in the phases). |
||||
- Sometimes there is a version conflict between some dependency requirements. In that case you can fix a version (by removing the `^`). |
||||
- Sometimes the script defined in the package.json does not work as is. Some scripts for example use cli tools that might not be available, or cd in directory with a different package.json (for workspaces notably). In that case, it's perfectly fine to look at what the particular script is doing and break this down in the phases. In the build script you can see `build:*` calling in turns several other build scripts like `build:ui` or `build:server`. If one of those fails, you can try to separate those into: |
||||
|
||||
```Shell |
||||
yarn build:ui |
||||
yarn build:server |
||||
# OR |
||||
npm run build:ui |
||||
npm run build:server |
||||
``` |
||||
|
||||
when you need to override a package.json. It's nice to use the one from the upstream src and do some explicit override. Here is an example. |
||||
|
||||
```nix |
||||
patchedPackageJSON = final.runCommand "package.json" { } '' |
||||
${jq}/bin/jq '.version = "0.4.0" | |
||||
.devDependencies."@jsdoc/cli" = "^0.2.5" |
||||
${sonar-src}/package.json > $out |
||||
''; |
||||
``` |
||||
|
||||
you will still need to commit the modified version of the lock files, but at least the overrides are explicit for everyone to see. |
||||
|
||||
### Using node_modules directly {#javascript-using-node_modules} |
||||
|
||||
each tool has an abstraction to just build the node_modules (dependencies) directory. you can always use the stdenv.mkDerivation with the node_modules to build the package (symlink the node_modules directory and then use the package build command). the node_modules abstraction can be also used to build some web framework frontends. For an example of this see how [plausible](https://github.com/NixOS/nixpkgs/blob/master/pkgs/servers/web-apps/plausible/default.nix) is built. mkYarnModules to make the derivation containing node_modules. Then when building the frontend you can just symlink the node_modules directory |
||||
|
||||
## javascript packages inside nixpkgs {#javascript-packages-nixpkgs} |
||||
|
||||
The `pkgs/development/node-packages` folder contains a generated collection of |
||||
[NPM packages](https://npmjs.com/) that can be installed with the Nix package |
||||
manager. |
||||
|
||||
As a rule of thumb, the package set should only provide _end user_ software |
||||
packages, such as command-line utilities. Libraries should only be added to the |
||||
package set if there is a non-NPM package that requires it. |
||||
|
||||
When it is desired to use NPM libraries in a development project, use the |
||||
`node2nix` generator directly on the `package.json` configuration file of the |
||||
project. |
||||
|
||||
The package set provides support for the official stable Node.js versions. |
||||
The latest stable LTS release in `nodePackages`, as well as the latest stable |
||||
Current release in `nodePackages_latest`. |
||||
|
||||
If your package uses native addons, you need to examine what kind of native |
||||
build system it uses. Here are some examples: |
||||
|
||||
- `node-gyp` |
||||
- `node-gyp-builder` |
||||
- `node-pre-gyp` |
||||
|
||||
After you have identified the correct system, you need to override your package |
||||
expression while adding in build system as a build input. For example, `dat` |
||||
requires `node-gyp-build`, so [we override](https://github.com/NixOS/nixpkgs/blob/32f5e5da4a1b3f0595527f5195ac3a91451e9b56/pkgs/development/node-packages/default.nix#L37-L40) its expression in [`default.nix`](https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/node-packages/default.nix): |
||||
|
||||
```nix |
||||
dat = super.dat.override { |
||||
buildInputs = [ self.node-gyp-build pkgs.libtool pkgs.autoconf pkgs.automake ]; |
||||
meta.broken = since "12"; |
||||
}; |
||||
``` |
||||
|
||||
To add a package from NPM to nixpkgs: |
||||
|
||||
1. Modify `pkgs/development/node-packages/node-packages.json` to add, update |
||||
or remove package entries to have it included in `nodePackages` and |
||||
`nodePackages_latest`. |
||||
2. Run the script: `cd pkgs/development/node-packages && ./generate.sh`. |
||||
3. Build your new package to test your changes: |
||||
`cd /path/to/nixpkgs && nix-build -A nodePackages.<new-or-updated-package>`. |
||||
To build against the latest stable Current Node.js version (e.g. 14.x): |
||||
`nix-build -A nodePackages_latest.<new-or-updated-package>` |
||||
4. Add and commit all modified and generated files. |
||||
|
||||
For more information about the generation process, consult the |
||||
[README.md](https://github.com/svanderburg/node2nix) file of the `node2nix` |
||||
tool. |
||||
|
||||
## Tool specific instructions {#javascript-tool-specific} |
||||
|
||||
### node2nix {#javascript-node2nix} |
||||
|
||||
#### Preparation {#javascript-node2nix-preparation} |
||||
|
||||
you will need to generate a nix expression for the dependencies |
||||
|
||||
- don't forget the `-l package-lock.json` if there is a lock file |
||||
- Most probably you will need the `--development` to include the `devDependencies` |
||||
|
||||
so the command will most likely be |
||||
`node2nix --development -l package-lock.json` |
||||
|
||||
[link to the doc in the repo](https://github.com/svanderburg/node2nix) |
||||
|
||||
#### Pitfalls {#javascript-node2nix-pitfalls} |
||||
|
||||
- if upstream package.json does not have a "version" attribute, node2nix will crash. You will need to add it like shown in [the package.json section](#javascript-upstream-package-json) |
||||
- node2nix has some [bugs](https://github.com/svanderburg/node2nix/issues/238). related to working with lock files from npm distributed with nodejs-16_x |
||||
- node2nix does not like missing packages from npm. If you see something like `Cannot resolve version: vue-loader-v16@undefined` then you might want to try another tool. The package might have been pulled off of npm. |
||||
|
||||
### yarn2nix {#javascript-yarn2nix} |
||||
|
||||
#### Preparation {#javascript-yarn2nix-preparation} |
||||
|
||||
you will need at least a yarn.lock and yarn.nix file |
||||
|
||||
- generate a yarn.lock in upstream if it is not already there |
||||
- `yarn2nix > yarn.nix` will generate the dependencies in a nix format |
||||
|
||||
#### mkYarnPackage {#javascript-yarn2nix-mkYarnPackage} |
||||
|
||||
this will by default try to generate a binary. For package only generating static assets (Svelte, Vue, React...), you will need to explicitly override the build step with your instructions. It's important to use the `--offline` flag. For example if you script is `"build": "something"` in package.json use |
||||
|
||||
```nix |
||||
buildPhase = '' |
||||
yarn build --offline |
||||
''; |
||||
``` |
||||
|
||||
The dist phase is also trying to build a binary, the only way to override it is with |
||||
|
||||
```nix |
||||
distPhase = "true"; |
||||
``` |
||||
|
||||
the configure phase can sometimes fail because it tries to be too clever. |
||||
One common override is |
||||
|
||||
```nix |
||||
configurePhase = "ln -s $node_modules node_modules"; |
||||
``` |
||||
|
||||
#### mkYarnModules {#javascript-yarn2nix-mkYarnModules} |
||||
|
||||
this will generate a derivation including the node_modules. If you have to build a derivation for an integrated web framework (rails, phoenix..), this is probably the easiest way. [Plausible](https://github.com/NixOS/nixpkgs/blob/master/pkgs/servers/web-apps/plausible/default.nix#L39) offers a good example of how to do this. |
||||
|
||||
#### Overriding dependency behavior |
||||
|
||||
In the `mkYarnPackage` record the property `pkgConfig` can be used to override packages when you encounter problems building. |
||||
|
||||
For instance, say your package is throwing errors when trying to invoke node-sass: `ENOENT: no such file or directory, scandir '/build/source/node_modules/node-sass/vendor'` |
||||
|
||||
To fix this we will specify different versions of build inputs to use, as well as some post install steps to get the software built the way we want: |
||||
|
||||
```nix |
||||
mkYarnPackage rec { |
||||
pkgConfig = { |
||||
node-sass = { |
||||
buildInputs = with final;[ python libsass pkg-config ]; |
||||
postInstall = '' |
||||
LIBSASS_EXT=auto yarn --offline run build |
||||
rm build/config.gypi |
||||
''; |
||||
}; |
||||
}; |
||||
} |
||||
``` |
||||
|
||||
#### Pitfalls {#javascript-yarn2nix-pitfalls} |
||||
|
||||
- if version is missing from upstream package.json, yarn will silently install nothing. In that case, you will need to override package.json as shown in the [package.json section](#javascript-upstream-package-json) |
||||
|
||||
- having trouble with node-gyp? Try adding these lines to the `yarnPreBuild` steps: |
||||
|
||||
```nix |
||||
yarnPreBuild = '' |
||||
mkdir -p $HOME/.node-gyp/${nodejs.version} |
||||
echo 9 > $HOME/.node-gyp/${nodejs.version}/installVersion |
||||
ln -sfv ${nodejs}/include $HOME/.node-gyp/${nodejs.version} |
||||
export npm_config_nodedir=${nodejs} |
||||
''; |
||||
``` |
||||
|
||||
- The `echo 9` steps comes from this answer: <https://stackoverflow.com/a/49139496> |
||||
- Exporting the headers in `npm_config_nodedir` comes from this issue: <https://github.com/nodejs/node-gyp/issues/1191#issuecomment-301243919> |
||||
|
||||
## Outside of nixpkgs {#javascript-outside-nixpkgs} |
||||
|
||||
There are some other options available that can't be used inside nixpkgs. Those other options are written in nix. Importing them in nixpkgs will require moving the source code into nixpkgs. Using [Import From Derivation](https://nixos.wiki/wiki/Import_From_Derivation) is not allowed in hydra at present. If you are packaging something outside nixpkgs, those can be considered |
||||
|
||||
### npmlock2nix {#javascript-npmlock2nix} |
||||
|
||||
[npmlock2nix](https://github.com/nix-community/npmlock2nix) aims at building node_modules without code generation. It hasn't reached v1 yet, the api might be subject to change. |
||||
|
||||
#### Pitfalls {#javascript-npmlock2nix-pitfalls} |
||||
|
||||
- there are some [problems with npm v7](https://github.com/tweag/npmlock2nix/issues/45). |
||||
|
||||
### nix-npm-buildpackage {#javascript-nix-npm-buildpackage} |
||||
|
||||
[nix-npm-buildpackage](https://github.com/serokell/nix-npm-buildpackage) aims at building node_modules without code generation. It hasn't reached v1 yet, the api might change. It supports both package-lock.json and yarn.lock. |
||||
|
||||
#### Pitfalls {#javascript-nix-npm-buildpackage-pitfalls} |
||||
|
||||
- there are some [problems with npm v7](https://github.com/serokell/nix-npm-buildpackage/issues/33). |
||||
@ -1,51 +0,0 @@ |
||||
# Node.js {#node.js} |
||||
|
||||
The `pkgs/development/node-packages` folder contains a generated collection of |
||||
[NPM packages](https://npmjs.com/) that can be installed with the Nix package |
||||
manager. |
||||
|
||||
As a rule of thumb, the package set should only provide *end user* software |
||||
packages, such as command-line utilities. Libraries should only be added to the |
||||
package set if there is a non-NPM package that requires it. |
||||
|
||||
When it is desired to use NPM libraries in a development project, use the |
||||
`node2nix` generator directly on the `package.json` configuration file of the |
||||
project. |
||||
|
||||
The package set provides support for the official stable Node.js versions. |
||||
The latest stable LTS release in `nodePackages`, as well as the latest stable |
||||
Current release in `nodePackages_latest`. |
||||
|
||||
If your package uses native addons, you need to examine what kind of native |
||||
build system it uses. Here are some examples: |
||||
|
||||
* `node-gyp` |
||||
* `node-gyp-builder` |
||||
* `node-pre-gyp` |
||||
|
||||
After you have identified the correct system, you need to override your package |
||||
expression while adding in build system as a build input. For example, `dat` |
||||
requires `node-gyp-build`, so [we override](https://github.com/NixOS/nixpkgs/blob/32f5e5da4a1b3f0595527f5195ac3a91451e9b56/pkgs/development/node-packages/default.nix#L37-L40) its expression in [`default.nix`](https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/node-packages/default.nix): |
||||
|
||||
```nix |
||||
dat = super.dat.override { |
||||
buildInputs = [ self.node-gyp-build pkgs.libtool pkgs.autoconf pkgs.automake ]; |
||||
meta.broken = since "12"; |
||||
}; |
||||
``` |
||||
|
||||
To add a package from NPM to nixpkgs: |
||||
|
||||
1. Modify `pkgs/development/node-packages/node-packages.json` to add, update |
||||
or remove package entries to have it included in `nodePackages` and |
||||
`nodePackages_latest`. |
||||
2. Run the script: `(cd pkgs/development/node-packages && ./generate.sh)`. |
||||
3. Build your new package to test your changes: |
||||
`cd /path/to/nixpkgs && nix-build -A nodePackages.<new-or-updated-package>`. |
||||
To build against the latest stable Current Node.js version (e.g. 14.x): |
||||
`nix-build -A nodePackages_latest.<new-or-updated-package>` |
||||
4. Add and commit all modified and generated files. |
||||
|
||||
For more information about the generation process, consult the |
||||
[README.md](https://github.com/svanderburg/node2nix) file of the `node2nix` |
||||
tool. |
||||
@ -1,5 +1,5 @@ |
||||
{ |
||||
description = "Library of low-level helper functions for nix expressions."; |
||||
|
||||
outputs = { self }: { lib = import ./lib; }; |
||||
outputs = { self }: { lib = import ./.; }; |
||||
} |
||||
|
||||
@ -0,0 +1,21 @@ |
||||
#! /usr/bin/env nix-shell |
||||
#! nix-shell -i bash -p nix curl gnused -I nixpkgs=. |
||||
|
||||
# On Hackage every package description shows a category "Distributions" which |
||||
# lists a "NixOS" version. |
||||
# This script uploads a csv to hackage which will update the displayed versions |
||||
# based on the current versions in nixpkgs. This happens with a simple http |
||||
# request. |
||||
|
||||
# For authorization you just need to have any valid hackage account. This |
||||
# script uses the `username` and `password-command` field from your |
||||
# ~/.cabal/config file. |
||||
|
||||
# e.g. username: maralorn |
||||
# password-command: pass hackage.haskell.org (this can be any command, but not an arbitrary shell expression. Like cabal we only read the first output line and ignore the rest.) |
||||
# Those fields are specified under `upload` on the `cabal` man page. |
||||
|
||||
package_list="$(nix-build -A haskell.package-list)/nixos-hackage-packages.csv" |
||||
username=$(grep "^username:" ~/.cabal/config | sed "s/^username: //") |
||||
password_command=$(grep "^password-command:" ~/.cabal/config | sed "s/^password-command: //") |
||||
curl -u "$username:$($password_command | head -n1)" --digest -H "Content-type: text/csv" -T "$package_list" http://hackage.haskell.org/distro/NixOS/packages.csv |
||||
|
@ -0,0 +1,6 @@ |
||||
# Linking NixOS tests to packages {#sec-linking-nixos-tests-to-packages} |
||||
|
||||
You can link NixOS module tests to the packages that they exercised, |
||||
so that the tests can be run automatically during code review when the package gets changed. |
||||
This is |
||||
[described in the nixpkgs manual](https://nixos.org/manual/nixpkgs/stable/#ssec-nixos-tests-linking). |
||||
@ -1,33 +1,72 @@ |
||||
<chapter xmlns="http://docbook.org/ns/docbook" xmlns:xlink="http://www.w3.org/1999/xlink" xml:id="sec-building-cd"> |
||||
<title>Building Your Own NixOS CD</title> |
||||
<chapter xmlns="http://docbook.org/ns/docbook" xmlns:xlink="http://www.w3.org/1999/xlink" xml:id="sec-building-image"> |
||||
<title>Building a NixOS (Live) ISO</title> |
||||
<para> |
||||
Building a NixOS CD is as easy as configuring your own computer. The |
||||
idea is to use another module which will replace your |
||||
<literal>configuration.nix</literal> to configure the system that |
||||
would be installed on the CD. |
||||
Default live installer configurations are available inside |
||||
<literal>nixos/modules/installer/cd-dvd</literal>. For building |
||||
other system images, |
||||
<link xlink:href="https://github.com/nix-community/nixos-generators">nixos-generators</link> |
||||
is a good place to start looking at. |
||||
</para> |
||||
<para> |
||||
Default CD/DVD configurations are available inside |
||||
<literal>nixos/modules/installer/cd-dvd</literal> |
||||
You have two options: |
||||
</para> |
||||
<programlisting> |
||||
<itemizedlist spacing="compact"> |
||||
<listitem> |
||||
<para> |
||||
Use any of those default configurations as is |
||||
</para> |
||||
</listitem> |
||||
<listitem> |
||||
<para> |
||||
Combine them with (any of) your host config(s) |
||||
</para> |
||||
</listitem> |
||||
</itemizedlist> |
||||
<para> |
||||
System images, such as the live installer ones, know how to enforce |
||||
configuration settings on wich they immediately depend in order to |
||||
work correctly. |
||||
</para> |
||||
<para> |
||||
However, if you are confident, you can opt to override those |
||||
enforced values with <literal>mkForce</literal>. |
||||
</para> |
||||
<section xml:id="sec-building-image-instructions"> |
||||
<title>Practical Instructions</title> |
||||
<programlisting> |
||||
$ git clone https://github.com/NixOS/nixpkgs.git |
||||
$ cd nixpkgs/nixos |
||||
$ nix-build -A config.system.build.isoImage -I nixos-config=modules/installer/cd-dvd/installation-cd-minimal.nix default.nix |
||||
</programlisting> |
||||
<para> |
||||
Before burning your CD/DVD, you can check the content of the image |
||||
by mounting anywhere like suggested by the following command: |
||||
</para> |
||||
<programlisting> |
||||
# mount -o loop -t iso9660 ./result/iso/cd.iso /mnt/iso</screen> |
||||
<para> |
||||
To check the content of an ISO image, mount it like so: |
||||
</para> |
||||
<programlisting> |
||||
# mount -o loop -t iso9660 ./result/iso/cd.iso /mnt/iso |
||||
</programlisting> |
||||
<para> |
||||
If you want to customize your NixOS CD in more detail, or generate |
||||
other kinds of images, you might want to check out |
||||
<link xlink:href="https://github.com/nix-community/nixos-generators">nixos-generators</link>. |
||||
This can also be a good starting point when you want to use Nix to |
||||
build a <quote>minimal</quote> image that doesn’t include a NixOS |
||||
installation. |
||||
</para> |
||||
</section> |
||||
<section xml:id="sec-building-image-tech-notes"> |
||||
<title>Technical Notes</title> |
||||
<para> |
||||
The config value enforcement is implemented via |
||||
<literal>mkImageMediaOverride = mkOverride 60;</literal> and |
||||
therefore primes over simple value assignments, but also yields to |
||||
<literal>mkForce</literal>. |
||||
</para> |
||||
<para> |
||||
This property allows image designers to implement in semantically |
||||
correct ways those configuration values upon which the correct |
||||
functioning of the image depends. |
||||
</para> |
||||
<para> |
||||
For example, the iso base image overrides those file systems which |
||||
it needs at a minimum for correct functioning, while the installer |
||||
base image overrides the entire file system layout because there |
||||
can’t be any other guarantees on a live medium than those given by |
||||
the live medium itself. The latter is especially true befor |
||||
formatting the target block device(s). On the other hand, the |
||||
netboot iso only overrides its minimum dependencies since netboot |
||||
images are always made-to-target. |
||||
</para> |
||||
</section> |
||||
</chapter> |
||||
|
||||
@ -0,0 +1,10 @@ |
||||
<section xmlns="http://docbook.org/ns/docbook" xmlns:xlink="http://www.w3.org/1999/xlink" xml:id="sec-linking-nixos-tests-to-packages"> |
||||
<title>Linking NixOS tests to packages</title> |
||||
<para> |
||||
You can link NixOS module tests to the packages that they exercised, |
||||
so that the tests can be run automatically during code review when |
||||
the package gets changed. This is |
||||
<link xlink:href="https://nixos.org/manual/nixpkgs/stable/#ssec-nixos-tests-linking">described |
||||
in the nixpkgs manual</link>. |
||||
</para> |
||||
</section> |
||||
@ -0,0 +1,333 @@ |
||||
# Note: This is a private API, internal to NixOS. Its interface is subject |
||||
# to change without notice. |
||||
# |
||||
# The result of this builder is two disk images: |
||||
# |
||||
# * `boot` - a small disk formatted with FAT to be used for /boot. FAT is |
||||
# chosen to support EFI. |
||||
# * `root` - a larger disk with a zpool taking the entire disk. |
||||
# |
||||
# This two-disk approach is taken to satisfy ZFS's requirements for |
||||
# autoexpand. |
||||
# |
||||
# # Why doesn't autoexpand work with ZFS in a partition? |
||||
# |
||||
# When ZFS owns the whole disk doesn’t really use a partition: it has |
||||
# a marker partition at the start and a marker partition at the end of |
||||
# the disk. |
||||
# |
||||
# If ZFS is constrained to a partition, ZFS leaves expanding the partition |
||||
# up to the user. Obviously, the user may not choose to do so. |
||||
# |
||||
# Once the user expands the partition, calling zpool online -e expands the |
||||
# vdev to use the whole partition. It doesn’t happen automatically |
||||
# presumably because zed doesn’t get an event saying it’s partition grew, |
||||
# whereas it can and does get an event saying the whole disk it is on is |
||||
# now larger. |
||||
{ lib |
||||
, pkgs |
||||
, # The NixOS configuration to be installed onto the disk image. |
||||
config |
||||
|
||||
, # size of the FAT boot disk, in megabytes. |
||||
bootSize ? 1024 |
||||
|
||||
, # The size of the root disk, in megabytes. |
||||
rootSize ? 2048 |
||||
|
||||
, # The name of the ZFS pool |
||||
rootPoolName ? "tank" |
||||
|
||||
, # zpool properties |
||||
rootPoolProperties ? { |
||||
autoexpand = "on"; |
||||
} |
||||
, # pool-wide filesystem properties |
||||
rootPoolFilesystemProperties ? { |
||||
acltype = "posixacl"; |
||||
atime = "off"; |
||||
compression = "on"; |
||||
mountpoint = "legacy"; |
||||
xattr = "sa"; |
||||
} |
||||
|
||||
, # datasets, with per-attribute options: |
||||
# mount: (optional) mount point in the VM |
||||
# properties: (optional) ZFS properties on the dataset, like filesystemProperties |
||||
# Notes: |
||||
# 1. datasets will be created from shorter to longer names as a simple topo-sort |
||||
# 2. you should define a root's dataset's mount for `/` |
||||
datasets ? { } |
||||
|
||||
, # The files and directories to be placed in the target file system. |
||||
# This is a list of attribute sets {source, target} where `source' |
||||
# is the file system object (regular file or directory) to be |
||||
# grafted in the file system at path `target'. |
||||
contents ? [] |
||||
|
||||
, # The initial NixOS configuration file to be copied to |
||||
# /etc/nixos/configuration.nix. This configuration will be embedded |
||||
# inside a configuration which includes the described ZFS fileSystems. |
||||
configFile ? null |
||||
|
||||
, # Shell code executed after the VM has finished. |
||||
postVM ? "" |
||||
|
||||
, name ? "nixos-disk-image" |
||||
|
||||
, # Disk image format, one of qcow2, qcow2-compressed, vdi, vpc, raw. |
||||
format ? "raw" |
||||
|
||||
, # Include a copy of Nixpkgs in the disk image |
||||
includeChannel ? true |
||||
}: |
||||
let |
||||
formatOpt = if format == "qcow2-compressed" then "qcow2" else format; |
||||
|
||||
compress = lib.optionalString (format == "qcow2-compressed") "-c"; |
||||
|
||||
filenameSuffix = "." + { |
||||
qcow2 = "qcow2"; |
||||
vdi = "vdi"; |
||||
vpc = "vhd"; |
||||
raw = "img"; |
||||
}.${formatOpt} or formatOpt; |
||||
bootFilename = "nixos.boot${filenameSuffix}"; |
||||
rootFilename = "nixos.root${filenameSuffix}"; |
||||
|
||||
# FIXME: merge with channel.nix / make-channel.nix. |
||||
channelSources = |
||||
let |
||||
nixpkgs = lib.cleanSource pkgs.path; |
||||
in |
||||
pkgs.runCommand "nixos-${config.system.nixos.version}" {} '' |
||||
mkdir -p $out |
||||
cp -prd ${nixpkgs.outPath} $out/nixos |
||||
chmod -R u+w $out/nixos |
||||
if [ ! -e $out/nixos/nixpkgs ]; then |
||||
ln -s . $out/nixos/nixpkgs |
||||
fi |
||||
rm -rf $out/nixos/.git |
||||
echo -n ${config.system.nixos.versionSuffix} > $out/nixos/.version-suffix |
||||
''; |
||||
|
||||
closureInfo = pkgs.closureInfo { |
||||
rootPaths = [ config.system.build.toplevel ] |
||||
++ (lib.optional includeChannel channelSources); |
||||
}; |
||||
|
||||
modulesTree = pkgs.aggregateModules |
||||
(with config.boot.kernelPackages; [ kernel zfs ]); |
||||
|
||||
tools = lib.makeBinPath ( |
||||
with pkgs; [ |
||||
config.system.build.nixos-enter |
||||
config.system.build.nixos-install |
||||
dosfstools |
||||
e2fsprogs |
||||
gptfdisk |
||||
nix |
||||
parted |
||||
utillinux |
||||
zfs |
||||
] |
||||
); |
||||
|
||||
hasDefinedMount = disk: ((disk.mount or null) != null); |
||||
|
||||
stringifyProperties = prefix: properties: lib.concatStringsSep " \\\n" ( |
||||
lib.mapAttrsToList |
||||
( |
||||
property: value: "${prefix} ${lib.escapeShellArg property}=${lib.escapeShellArg value}" |
||||
) |
||||
properties |
||||
); |
||||
|
||||
featuresToProperties = features: |
||||
lib.listToAttrs |
||||
(builtins.map (feature: { |
||||
name = "feature@${feature}"; |
||||
value = "enabled"; |
||||
}) features); |
||||
|
||||
createDatasets = |
||||
let |
||||
datasetlist = lib.mapAttrsToList lib.nameValuePair datasets; |
||||
sorted = lib.sort (left: right: (lib.stringLength left.name) < (lib.stringLength right.name)) datasetlist; |
||||
cmd = { name, value }: |
||||
let |
||||
properties = stringifyProperties "-o" (value.properties or {}); |
||||
in |
||||
"zfs create -p ${properties} ${name}"; |
||||
in |
||||
lib.concatMapStringsSep "\n" cmd sorted; |
||||
|
||||
mountDatasets = |
||||
let |
||||
datasetlist = lib.mapAttrsToList lib.nameValuePair datasets; |
||||
mounts = lib.filter ({ value, ... }: hasDefinedMount value) datasetlist; |
||||
sorted = lib.sort (left: right: (lib.stringLength left.value.mount) < (lib.stringLength right.value.mount)) mounts; |
||||
cmd = { name, value }: |
||||
'' |
||||
mkdir -p /mnt${lib.escapeShellArg value.mount} |
||||
mount -t zfs ${name} /mnt${lib.escapeShellArg value.mount} |
||||
''; |
||||
in |
||||
lib.concatMapStringsSep "\n" cmd sorted; |
||||
|
||||
unmountDatasets = |
||||
let |
||||
datasetlist = lib.mapAttrsToList lib.nameValuePair datasets; |
||||
mounts = lib.filter ({ value, ... }: hasDefinedMount value) datasetlist; |
||||
sorted = lib.sort (left: right: (lib.stringLength left.value.mount) > (lib.stringLength right.value.mount)) mounts; |
||||
cmd = { name, value }: |
||||
'' |
||||
umount /mnt${lib.escapeShellArg value.mount} |
||||
''; |
||||
in |
||||
lib.concatMapStringsSep "\n" cmd sorted; |
||||
|
||||
|
||||
fileSystemsCfgFile = |
||||
let |
||||
mountable = lib.filterAttrs (_: value: hasDefinedMount value) datasets; |
||||
in |
||||
pkgs.runCommand "filesystem-config.nix" { |
||||
buildInputs = with pkgs; [ jq nixpkgs-fmt ]; |
||||
filesystems = builtins.toJSON { |
||||
fileSystems = lib.mapAttrs' |
||||
( |
||||
dataset: attrs: |
||||
{ |
||||
name = attrs.mount; |
||||
value = { |
||||
fsType = "zfs"; |
||||
device = "${dataset}"; |
||||
}; |
||||
} |
||||
) |
||||
mountable; |
||||
}; |
||||
passAsFile = [ "filesystems" ]; |
||||
} '' |
||||
( |
||||
echo "builtins.fromJSON '''" |
||||
jq . < "$filesystemsPath" |
||||
echo "'''" |
||||
) > $out |
||||
|
||||
nixpkgs-fmt $out |
||||
''; |
||||
|
||||
mergedConfig = |
||||
if configFile == null |
||||
then fileSystemsCfgFile |
||||
else |
||||
pkgs.runCommand "configuration.nix" { |
||||
buildInputs = with pkgs; [ nixpkgs-fmt ]; |
||||
} |
||||
'' |
||||
( |
||||
echo '{ imports = [' |
||||
printf "(%s)\n" "$(cat ${fileSystemsCfgFile})"; |
||||
printf "(%s)\n" "$(cat ${configFile})"; |
||||
echo ']; }' |
||||
) > $out |
||||
|
||||
nixpkgs-fmt $out |
||||
''; |
||||
|
||||
image = ( |
||||
pkgs.vmTools.override { |
||||
rootModules = |
||||
[ "zfs" "9p" "9pnet_virtio" "virtio_pci" "virtio_blk" ] ++ |
||||
(pkgs.lib.optional (pkgs.stdenv.isi686 || pkgs.stdenv.isx86_64) "rtc_cmos"); |
||||
kernel = modulesTree; |
||||
} |
||||
).runInLinuxVM ( |
||||
pkgs.runCommand name |
||||
{ |
||||
QEMU_OPTS = "-drive file=$bootDiskImage,if=virtio,cache=unsafe,werror=report" |
||||
+ " -drive file=$rootDiskImage,if=virtio,cache=unsafe,werror=report"; |
||||
preVM = '' |
||||
PATH=$PATH:${pkgs.qemu_kvm}/bin |
||||
mkdir $out |
||||
bootDiskImage=boot.raw |
||||
qemu-img create -f raw $bootDiskImage ${toString bootSize}M |
||||
|
||||
rootDiskImage=root.raw |
||||
qemu-img create -f raw $rootDiskImage ${toString rootSize}M |
||||
''; |
||||
|
||||
postVM = '' |
||||
${if formatOpt == "raw" then '' |
||||
mv $bootDiskImage $out/${bootFilename} |
||||
mv $rootDiskImage $out/${rootFilename} |
||||
'' else '' |
||||
${pkgs.qemu}/bin/qemu-img convert -f raw -O ${formatOpt} ${compress} $bootDiskImage $out/${bootFilename} |
||||
${pkgs.qemu}/bin/qemu-img convert -f raw -O ${formatOpt} ${compress} $rootDiskImage $out/${rootFilename} |
||||
''} |
||||
bootDiskImage=$out/${bootFilename} |
||||
rootDiskImage=$out/${rootFilename} |
||||
set -x |
||||
${postVM} |
||||
''; |
||||
} '' |
||||
export PATH=${tools}:$PATH |
||||
set -x |
||||
|
||||
cp -sv /dev/vda /dev/sda |
||||
cp -sv /dev/vda /dev/xvda |
||||
|
||||
parted --script /dev/vda -- \ |
||||
mklabel gpt \ |
||||
mkpart no-fs 1MiB 2MiB \ |
||||
set 1 bios_grub on \ |
||||
align-check optimal 1 \ |
||||
mkpart ESP fat32 2MiB -1MiB \ |
||||
align-check optimal 2 \ |
||||
print |
||||
|
||||
sfdisk --dump /dev/vda |
||||
|
||||
|
||||
zpool create \ |
||||
${stringifyProperties " -o" rootPoolProperties} \ |
||||
${stringifyProperties " -O" rootPoolFilesystemProperties} \ |
||||
${rootPoolName} /dev/vdb |
||||
parted --script /dev/vdb -- print |
||||
|
||||
${createDatasets} |
||||
${mountDatasets} |
||||
|
||||
mkdir -p /mnt/boot |
||||
mkfs.vfat -n ESP /dev/vda2 |
||||
mount /dev/vda2 /mnt/boot |
||||
|
||||
mount |
||||
|
||||
# Install a configuration.nix |
||||
mkdir -p /mnt/etc/nixos |
||||
# `cat` so it is mutable on the fs |
||||
cat ${mergedConfig} > /mnt/etc/nixos/configuration.nix |
||||
|
||||
export NIX_STATE_DIR=$TMPDIR/state |
||||
nix-store --load-db < ${closureInfo}/registration |
||||
|
||||
nixos-install \ |
||||
--root /mnt \ |
||||
--no-root-passwd \ |
||||
--system ${config.system.build.toplevel} \ |
||||
--substituters "" \ |
||||
${lib.optionalString includeChannel ''--channel ${channelSources}''} |
||||
|
||||
df -h |
||||
|
||||
umount /mnt/boot |
||||
${unmountDatasets} |
||||
|
||||
zpool export ${rootPoolName} |
||||
'' |
||||
); |
||||
in |
||||
image |
||||
@ -0,0 +1,12 @@ |
||||
{ |
||||
imports = [ ./amazon-image.nix ]; |
||||
ec2.zfs = { |
||||
enable = true; |
||||
datasets = { |
||||
"tank/system/root".mount = "/"; |
||||
"tank/system/var".mount = "/var"; |
||||
"tank/local/nix".mount = "/nix"; |
||||
"tank/user/home".mount = "/home"; |
||||
}; |
||||
}; |
||||
} |
||||
@ -1,4 +0,0 @@ |
||||
ATTRS{idVendor}=="16c0", ATTRS{idProduct}=="04[789B]?", ENV{ID_MM_DEVICE_IGNORE}="1" |
||||
ATTRS{idVendor}=="16c0", ATTRS{idProduct}=="04[789A]?", ENV{MTP_NO_PROBE}="1" |
||||
SUBSYSTEMS=="usb", ATTRS{idVendor}=="16c0", ATTRS{idProduct}=="04[789ABCD]?", GROUP+="plugdev" |
||||
KERNEL=="ttyACM*", ATTRS{idVendor}=="16c0", ATTRS{idProduct}=="04[789B]?", GROUP+="plugdev" |
||||
@ -0,0 +1,18 @@ |
||||
# UDEV Rules for OnlyKey, https://docs.crp.to/linux.html |
||||
ATTRS{idVendor}=="1d50", ATTRS{idProduct}=="60fc", ENV{ID_MM_DEVICE_IGNORE}="1" |
||||
ATTRS{idVendor}=="1d50", ATTRS{idProduct}=="60fc", ENV{MTP_NO_PROBE}="1" |
||||
SUBSYSTEMS=="usb", ATTRS{idVendor}=="1d50", ATTRS{idProduct}=="60fc", MODE:="0666" |
||||
KERNEL=="ttyACM*", ATTRS{idVendor}=="1d50", ATTRS{idProduct}=="60fc", MODE:="0666" |
||||
|
||||
|
||||
# The udev rules were updated upstream without an explanation as you can |
||||
# see in [this comment][commit]. Assuming that hey have changed the |
||||
# idVendor/idProduct, I've kept the old values. |
||||
# TODO: Contact them upstream. |
||||
# |
||||
# [commit]: https://github.com/trustcrypto/trustcrypto.github.io/commit/0bcf928adaea559e75efa02ebd1040f0a15f611d |
||||
# |
||||
ATTRS{idVendor}=="16c0", ATTRS{idProduct}=="04[789B]?", ENV{ID_MM_DEVICE_IGNORE}="1" |
||||
ATTRS{idVendor}=="16c0", ATTRS{idProduct}=="04[789A]?", ENV{MTP_NO_PROBE}="1" |
||||
SUBSYSTEMS=="usb", ATTRS{idVendor}=="16c0", ATTRS{idProduct}=="04[789ABCD]?", GROUP+="plugdev" |
||||
KERNEL=="ttyACM*", ATTRS{idVendor}=="16c0", ATTRS{idProduct}=="04[789B]?", GROUP+="plugdev" |
||||
@ -1,7 +1,7 @@ |
||||
{ |
||||
x86_64-linux = "/nix/store/qsgz2hhn6mzlzp53a7pwf9z2pq3l5z6h-nix-2.3.14"; |
||||
i686-linux = "/nix/store/1yw40bj04lykisw2jilq06lir3k9ga4a-nix-2.3.14"; |
||||
aarch64-linux = "/nix/store/32yzwmynmjxfrkb6y6l55liaqdrgkj4a-nix-2.3.14"; |
||||
x86_64-darwin = "/nix/store/06j0vi2d13w4l0p3jsigq7lk4x6gkycj-nix-2.3.14"; |
||||
aarch64-darwin = "/nix/store/77wi7vpbrghw5rgws25w30bwb8yggnk9-nix-2.3.14"; |
||||
x86_64-linux = "/nix/store/jhbxh1jwjc3hjhzs9y2hifdn0rmnfwaj-nix-2.3.15"; |
||||
i686-linux = "/nix/store/9pspwnkdrgzma1l4xlv7arhwa56y16di-nix-2.3.15"; |
||||
aarch64-linux = "/nix/store/72aqi5g7f4fhgvgafbcqwcpqjgnczj48-nix-2.3.15"; |
||||
x86_64-darwin = "/nix/store/6p6qwp73dgfkqhynmxrzbx1lcfgfpqal-nix-2.3.15"; |
||||
aarch64-darwin = "/nix/store/dmq2vksdhssgfl822shd0ky3x5x0klh4-nix-2.3.15"; |
||||
} |
||||
|
||||
@ -0,0 +1,25 @@ |
||||
{ config, lib, pkgs, ... }: |
||||
|
||||
with lib; |
||||
|
||||
let |
||||
cfg = config.programs.calls; |
||||
in { |
||||
options = { |
||||
programs.calls = { |
||||
enable = mkEnableOption '' |
||||
Whether to enable GNOME calls: a phone dialer and call handler. |
||||
''; |
||||
}; |
||||
}; |
||||
|
||||
config = mkIf cfg.enable { |
||||
environment.systemPackages = [ |
||||
pkgs.calls |
||||
]; |
||||
|
||||
services.dbus.packages = [ |
||||
pkgs.callaudiod |
||||
]; |
||||
}; |
||||
} |
||||
Some files were not shown because too many files have changed in this diff Show More
Loading…
Reference in new issue