parent
357d2c8f60
commit
163976c296
@ -0,0 +1,301 @@ |
||||
# Writing Tests {#sec-writing-nixos-tests} |
||||
|
||||
A NixOS test is a Nix expression that has the following structure: |
||||
|
||||
```nix |
||||
import ./make-test-python.nix { |
||||
|
||||
# Either the configuration of a single machine: |
||||
machine = |
||||
{ config, pkgs, ... }: |
||||
{ configuration… |
||||
}; |
||||
|
||||
# Or a set of machines: |
||||
nodes = |
||||
{ machine1 = |
||||
{ config, pkgs, ... }: { … }; |
||||
machine2 = |
||||
{ config, pkgs, ... }: { … }; |
||||
… |
||||
}; |
||||
|
||||
testScript = |
||||
'' |
||||
Python code… |
||||
''; |
||||
} |
||||
``` |
||||
|
||||
The attribute `testScript` is a bit of Python code that executes the |
||||
test (described below). During the test, it will start one or more |
||||
virtual machines, the configuration of which is described by the |
||||
attribute `machine` (if you need only one machine in your test) or by |
||||
the attribute `nodes` (if you need multiple machines). For instance, |
||||
[`login.nix`](https://github.com/NixOS/nixpkgs/blob/master/nixos/tests/login.nix) |
||||
only needs a single machine to test whether users can log in |
||||
on the virtual console, whether device ownership is correctly maintained |
||||
when switching between consoles, and so on. On the other hand, |
||||
[`nfs/simple.nix`](https://github.com/NixOS/nixpkgs/blob/master/nixos/tests/nfs/simple.nix), |
||||
which tests NFS client and server functionality in the |
||||
Linux kernel (including whether locks are maintained across server |
||||
crashes), requires three machines: a server and two clients. |
||||
|
||||
There are a few special NixOS configuration options for test VMs: |
||||
|
||||
`virtualisation.memorySize` |
||||
|
||||
: The memory of the VM in megabytes. |
||||
|
||||
`virtualisation.vlans` |
||||
|
||||
: The virtual networks to which the VM is connected. See |
||||
[`nat.nix`](https://github.com/NixOS/nixpkgs/blob/master/nixos/tests/nat.nix) |
||||
for an example. |
||||
|
||||
`virtualisation.writableStore` |
||||
|
||||
: By default, the Nix store in the VM is not writable. If you enable |
||||
this option, a writable union file system is mounted on top of the |
||||
Nix store to make it appear writable. This is necessary for tests |
||||
that run Nix operations that modify the store. |
||||
|
||||
For more options, see the module |
||||
[`qemu-vm.nix`](https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/virtualisation/qemu-vm.nix). |
||||
|
||||
The test script is a sequence of Python statements that perform various |
||||
actions, such as starting VMs, executing commands in the VMs, and so on. |
||||
Each virtual machine is represented as an object stored in the variable |
||||
`name` if this is also the identifier of the machine in the declarative |
||||
config. If you didn\'t specify multiple machines using the `nodes` |
||||
attribute, it is just `machine`. The following example starts the |
||||
machine, waits until it has finished booting, then executes a command |
||||
and checks that the output is more-or-less correct: |
||||
|
||||
```py |
||||
machine.start() |
||||
machine.wait_for_unit("default.target") |
||||
if not "Linux" in machine.succeed("uname"): |
||||
raise Exception("Wrong OS") |
||||
``` |
||||
|
||||
The first line is actually unnecessary; machines are implicitly started |
||||
when you first execute an action on them (such as `wait_for_unit` or |
||||
`succeed`). If you have multiple machines, you can speed up the test by |
||||
starting them in parallel: |
||||
|
||||
```py |
||||
start_all() |
||||
``` |
||||
|
||||
The following methods are available on machine objects: |
||||
|
||||
`start` |
||||
|
||||
: Start the virtual machine. This method is asynchronous --- it does |
||||
not wait for the machine to finish booting. |
||||
|
||||
`shutdown` |
||||
|
||||
: Shut down the machine, waiting for the VM to exit. |
||||
|
||||
`crash` |
||||
|
||||
: Simulate a sudden power failure, by telling the VM to exit |
||||
immediately. |
||||
|
||||
`block` |
||||
|
||||
: Simulate unplugging the Ethernet cable that connects the machine to |
||||
the other machines. |
||||
|
||||
`unblock` |
||||
|
||||
: Undo the effect of `block`. |
||||
|
||||
`screenshot` |
||||
|
||||
: Take a picture of the display of the virtual machine, in PNG format. |
||||
The screenshot is linked from the HTML log. |
||||
|
||||
`get_screen_text_variants` |
||||
|
||||
: Return a list of different interpretations of what is currently |
||||
visible on the machine\'s screen using optical character |
||||
recognition. The number and order of the interpretations is not |
||||
specified and is subject to change, but if no exception is raised at |
||||
least one will be returned. |
||||
|
||||
::: {.note} |
||||
This requires passing `enableOCR` to the test attribute set. |
||||
::: |
||||
|
||||
`get_screen_text` |
||||
|
||||
: Return a textual representation of what is currently visible on the |
||||
machine\'s screen using optical character recognition. |
||||
|
||||
::: {.note} |
||||
This requires passing `enableOCR` to the test attribute set. |
||||
::: |
||||
|
||||
`send_monitor_command` |
||||
|
||||
: Send a command to the QEMU monitor. This is rarely used, but allows |
||||
doing stuff such as attaching virtual USB disks to a running |
||||
machine. |
||||
|
||||
`send_key` |
||||
|
||||
: Simulate pressing keys on the virtual keyboard, e.g., |
||||
`send_key("ctrl-alt-delete")`. |
||||
|
||||
`send_chars` |
||||
|
||||
: Simulate typing a sequence of characters on the virtual keyboard, |
||||
e.g., `send_chars("foobar\n")` will type the string `foobar` |
||||
followed by the Enter key. |
||||
|
||||
`execute` |
||||
|
||||
: Execute a shell command, returning a list `(status, stdout)`. |
||||
|
||||
`succeed` |
||||
|
||||
: Execute a shell command, raising an exception if the exit status is |
||||
not zero, otherwise returning the standard output. Commands are run |
||||
with `set -euo pipefail` set: |
||||
|
||||
- If several commands are separated by `;` and one fails, the |
||||
command as a whole will fail. |
||||
|
||||
- For pipelines, the last non-zero exit status will be returned |
||||
(if there is one, zero will be returned otherwise). |
||||
|
||||
- Dereferencing unset variables fail the command. |
||||
|
||||
`fail` |
||||
|
||||
: Like `succeed`, but raising an exception if the command returns a zero |
||||
status. |
||||
|
||||
`wait_until_succeeds` |
||||
|
||||
: Repeat a shell command with 1-second intervals until it succeeds. |
||||
|
||||
`wait_until_fails` |
||||
|
||||
: Repeat a shell command with 1-second intervals until it fails. |
||||
|
||||
`wait_for_unit` |
||||
|
||||
: Wait until the specified systemd unit has reached the "active" |
||||
state. |
||||
|
||||
`wait_for_file` |
||||
|
||||
: Wait until the specified file exists. |
||||
|
||||
`wait_for_open_port` |
||||
|
||||
: Wait until a process is listening on the given TCP port (on |
||||
`localhost`, at least). |
||||
|
||||
`wait_for_closed_port` |
||||
|
||||
: Wait until nobody is listening on the given TCP port. |
||||
|
||||
`wait_for_x` |
||||
|
||||
: Wait until the X11 server is accepting connections. |
||||
|
||||
`wait_for_text` |
||||
|
||||
: Wait until the supplied regular expressions matches the textual |
||||
contents of the screen by using optical character recognition (see |
||||
`get_screen_text` and `get_screen_text_variants`). |
||||
|
||||
::: {.note} |
||||
This requires passing `enableOCR` to the test attribute set. |
||||
::: |
||||
|
||||
`wait_for_console_text` |
||||
|
||||
: Wait until the supplied regular expressions match a line of the |
||||
serial console output. This method is useful when OCR is not |
||||
possibile or accurate enough. |
||||
|
||||
`wait_for_window` |
||||
|
||||
: Wait until an X11 window has appeared whose name matches the given |
||||
regular expression, e.g., `wait_for_window("Terminal")`. |
||||
|
||||
`copy_from_host` |
||||
|
||||
: Copies a file from host to machine, e.g., |
||||
`copy_from_host("myfile", "/etc/my/important/file")`. |
||||
|
||||
The first argument is the file on the host. The file needs to be |
||||
accessible while building the nix derivation. The second argument is |
||||
the location of the file on the machine. |
||||
|
||||
`systemctl` |
||||
|
||||
: Runs `systemctl` commands with optional support for |
||||
`systemctl --user` |
||||
|
||||
```py |
||||
machine.systemctl("list-jobs --no-pager") # runs `systemctl list-jobs --no-pager` |
||||
machine.systemctl("list-jobs --no-pager", "any-user") # spawns a shell for `any-user` and runs `systemctl --user list-jobs --no-pager` |
||||
``` |
||||
|
||||
`shell_interact` |
||||
|
||||
: Allows you to directly interact with the guest shell. This should |
||||
only be used during test development, not in production tests. |
||||
Killing the interactive session with `Ctrl-d` or `Ctrl-c` also ends |
||||
the guest session. |
||||
|
||||
To test user units declared by `systemd.user.services` the optional |
||||
`user` argument can be used: |
||||
|
||||
```py |
||||
machine.start() |
||||
machine.wait_for_x() |
||||
machine.wait_for_unit("xautolock.service", "x-session-user") |
||||
``` |
||||
|
||||
This applies to `systemctl`, `get_unit_info`, `wait_for_unit`, |
||||
`start_job` and `stop_job`. |
||||
|
||||
For faster dev cycles it\'s also possible to disable the code-linters |
||||
(this shouldn\'t be commited though): |
||||
|
||||
```nix |
||||
import ./make-test-python.nix { |
||||
skipLint = true; |
||||
machine = |
||||
{ config, pkgs, ... }: |
||||
{ configuration… |
||||
}; |
||||
|
||||
testScript = |
||||
'' |
||||
Python code… |
||||
''; |
||||
} |
||||
``` |
||||
|
||||
This will produce a Nix warning at evaluation time. To fully disable the |
||||
linter, wrap the test script in comment directives to disable the Black |
||||
linter directly (again, don\'t commit this within the Nixpkgs |
||||
repository): |
||||
|
||||
```nix |
||||
testScript = |
||||
'' |
||||
# fmt: off |
||||
Python code… |
||||
# fmt: on |
||||
''; |
||||
``` |
@ -1,517 +0,0 @@ |
||||
<section xmlns="http://docbook.org/ns/docbook" |
||||
xmlns:xlink="http://www.w3.org/1999/xlink" |
||||
xmlns:xi="http://www.w3.org/2001/XInclude" |
||||
version="5.0" |
||||
xml:id="sec-writing-nixos-tests"> |
||||
<title>Writing Tests</title> |
||||
|
||||
<para> |
||||
A NixOS test is a Nix expression that has the following structure: |
||||
<programlisting> |
||||
import ./make-test-python.nix { |
||||
|
||||
# Either the configuration of a single machine: |
||||
machine = |
||||
{ config, pkgs, ... }: |
||||
{ <replaceable>configuration…</replaceable> |
||||
}; |
||||
|
||||
# Or a set of machines: |
||||
nodes = |
||||
{ <replaceable>machine1</replaceable> = |
||||
{ config, pkgs, ... }: { <replaceable>…</replaceable> }; |
||||
<replaceable>machine2</replaceable> = |
||||
{ config, pkgs, ... }: { <replaceable>…</replaceable> }; |
||||
… |
||||
}; |
||||
|
||||
testScript = |
||||
'' |
||||
<replaceable>Python code…</replaceable> |
||||
''; |
||||
} |
||||
</programlisting> |
||||
The attribute <literal>testScript</literal> is a bit of Python code that |
||||
executes the test (described below). During the test, it will start one or |
||||
more virtual machines, the configuration of which is described by the |
||||
attribute <literal>machine</literal> (if you need only one machine in your |
||||
test) or by the attribute <literal>nodes</literal> (if you need multiple |
||||
machines). For instance, |
||||
<filename |
||||
xlink:href="https://github.com/NixOS/nixpkgs/blob/master/nixos/tests/login.nix">login.nix</filename> |
||||
only needs a single machine to test whether users can log in on the virtual |
||||
console, whether device ownership is correctly maintained when switching |
||||
between consoles, and so on. On the other hand, |
||||
<filename |
||||
xlink:href="https://github.com/NixOS/nixpkgs/blob/master/nixos/tests/nfs/simple.nix">nfs/simple.nix</filename>, |
||||
which tests NFS client and server functionality in the Linux kernel |
||||
(including whether locks are maintained across server crashes), requires |
||||
three machines: a server and two clients. |
||||
</para> |
||||
|
||||
<para> |
||||
There are a few special NixOS configuration options for test VMs: |
||||
<!-- FIXME: would be nice to generate this automatically. --> |
||||
<variablelist> |
||||
<varlistentry> |
||||
<term> |
||||
<option>virtualisation.memorySize</option> |
||||
</term> |
||||
<listitem> |
||||
<para> |
||||
The memory of the VM in megabytes. |
||||
</para> |
||||
</listitem> |
||||
</varlistentry> |
||||
<varlistentry> |
||||
<term> |
||||
<option>virtualisation.vlans</option> |
||||
</term> |
||||
<listitem> |
||||
<para> |
||||
The virtual networks to which the VM is connected. See |
||||
<filename |
||||
xlink:href="https://github.com/NixOS/nixpkgs/blob/master/nixos/tests/nat.nix">nat.nix</filename> |
||||
for an example. |
||||
</para> |
||||
</listitem> |
||||
</varlistentry> |
||||
<varlistentry> |
||||
<term> |
||||
<option>virtualisation.writableStore</option> |
||||
</term> |
||||
<listitem> |
||||
<para> |
||||
By default, the Nix store in the VM is not writable. If you enable this |
||||
option, a writable union file system is mounted on top of the Nix store |
||||
to make it appear writable. This is necessary for tests that run Nix |
||||
operations that modify the store. |
||||
</para> |
||||
</listitem> |
||||
</varlistentry> |
||||
</variablelist> |
||||
For more options, see the module |
||||
<filename |
||||
xlink:href="https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/virtualisation/qemu-vm.nix">qemu-vm.nix</filename>. |
||||
</para> |
||||
|
||||
<para> |
||||
The test script is a sequence of Python statements that perform various |
||||
actions, such as starting VMs, executing commands in the VMs, and so on. Each |
||||
virtual machine is represented as an object stored in the variable |
||||
<literal><replaceable>name</replaceable></literal> if this is also the |
||||
identifier of the machine in the declarative config. |
||||
If you didn't specify multiple machines using the <literal>nodes</literal> |
||||
attribute, it is just <literal>machine</literal>. |
||||
The following example starts the machine, waits until it has finished booting, |
||||
then executes a command and checks that the output is more-or-less correct: |
||||
<programlisting> |
||||
machine.start() |
||||
machine.wait_for_unit("default.target") |
||||
if not "Linux" in machine.succeed("uname"): |
||||
raise Exception("Wrong OS") |
||||
</programlisting> |
||||
The first line is actually unnecessary; machines are implicitly started when |
||||
you first execute an action on them (such as <literal>wait_for_unit</literal> |
||||
or <literal>succeed</literal>). If you have multiple machines, you can speed |
||||
up the test by starting them in parallel: |
||||
<programlisting> |
||||
start_all() |
||||
</programlisting> |
||||
</para> |
||||
|
||||
<para> |
||||
The following methods are available on machine objects: |
||||
<variablelist> |
||||
<varlistentry> |
||||
<term> |
||||
<methodname>start</methodname> |
||||
</term> |
||||
<listitem> |
||||
<para> |
||||
Start the virtual machine. This method is asynchronous — it does not |
||||
wait for the machine to finish booting. |
||||
</para> |
||||
</listitem> |
||||
</varlistentry> |
||||
<varlistentry> |
||||
<term> |
||||
<methodname>shutdown</methodname> |
||||
</term> |
||||
<listitem> |
||||
<para> |
||||
Shut down the machine, waiting for the VM to exit. |
||||
</para> |
||||
</listitem> |
||||
</varlistentry> |
||||
<varlistentry> |
||||
<term> |
||||
<methodname>crash</methodname> |
||||
</term> |
||||
<listitem> |
||||
<para> |
||||
Simulate a sudden power failure, by telling the VM to exit immediately. |
||||
</para> |
||||
</listitem> |
||||
</varlistentry> |
||||
<varlistentry> |
||||
<term> |
||||
<methodname>block</methodname> |
||||
</term> |
||||
<listitem> |
||||
<para> |
||||
Simulate unplugging the Ethernet cable that connects the machine to the |
||||
other machines. |
||||
</para> |
||||
</listitem> |
||||
</varlistentry> |
||||
<varlistentry> |
||||
<term> |
||||
<methodname>unblock</methodname> |
||||
</term> |
||||
<listitem> |
||||
<para> |
||||
Undo the effect of <methodname>block</methodname>. |
||||
</para> |
||||
</listitem> |
||||
</varlistentry> |
||||
<varlistentry> |
||||
<term> |
||||
<methodname>screenshot</methodname> |
||||
</term> |
||||
<listitem> |
||||
<para> |
||||
Take a picture of the display of the virtual machine, in PNG format. The |
||||
screenshot is linked from the HTML log. |
||||
</para> |
||||
</listitem> |
||||
</varlistentry> |
||||
<varlistentry> |
||||
<term> |
||||
<methodname>get_screen_text_variants</methodname> |
||||
</term> |
||||
<listitem> |
||||
<para> |
||||
Return a list of different interpretations of what is currently visible |
||||
on the machine's screen using optical character recognition. The number |
||||
and order of the interpretations is not specified and is subject to |
||||
change, but if no exception is raised at least one will be returned. |
||||
</para> |
||||
<note> |
||||
<para> |
||||
This requires passing <option>enableOCR</option> to the test attribute |
||||
set. |
||||
</para> |
||||
</note> |
||||
</listitem> |
||||
</varlistentry> |
||||
<varlistentry> |
||||
<term> |
||||
<methodname>get_screen_text</methodname> |
||||
</term> |
||||
<listitem> |
||||
<para> |
||||
Return a textual representation of what is currently visible on the |
||||
machine's screen using optical character recognition. |
||||
</para> |
||||
<note> |
||||
<para> |
||||
This requires passing <option>enableOCR</option> to the test attribute |
||||
set. |
||||
</para> |
||||
</note> |
||||
</listitem> |
||||
</varlistentry> |
||||
<varlistentry> |
||||
<term> |
||||
<methodname>send_monitor_command</methodname> |
||||
</term> |
||||
<listitem> |
||||
<para> |
||||
Send a command to the QEMU monitor. This is rarely used, but allows doing |
||||
stuff such as attaching virtual USB disks to a running machine. |
||||
</para> |
||||
</listitem> |
||||
</varlistentry> |
||||
<varlistentry> |
||||
<term> |
||||
<methodname>send_key</methodname> |
||||
</term> |
||||
<listitem> |
||||
<para> |
||||
Simulate pressing keys on the virtual keyboard, e.g., |
||||
<literal>send_key("ctrl-alt-delete")</literal>. |
||||
</para> |
||||
</listitem> |
||||
</varlistentry> |
||||
<varlistentry> |
||||
<term> |
||||
<methodname>send_chars</methodname> |
||||
</term> |
||||
<listitem> |
||||
<para> |
||||
Simulate typing a sequence of characters on the virtual keyboard, e.g., |
||||
<literal>send_chars("foobar\n")</literal> will type the string |
||||
<literal>foobar</literal> followed by the Enter key. |
||||
</para> |
||||
</listitem> |
||||
</varlistentry> |
||||
<varlistentry> |
||||
<term> |
||||
<methodname>execute</methodname> |
||||
</term> |
||||
<listitem> |
||||
<para> |
||||
Execute a shell command, returning a list |
||||
<literal>(<replaceable>status</replaceable>, |
||||
<replaceable>stdout</replaceable>)</literal>. |
||||
</para> |
||||
</listitem> |
||||
</varlistentry> |
||||
<varlistentry> |
||||
<term> |
||||
<methodname>succeed</methodname> |
||||
</term> |
||||
<listitem> |
||||
<para> |
||||
Execute a shell command, raising an exception if the exit status |
||||
is not zero, otherwise returning the standard output. Commands |
||||
are run with <literal>set -euo pipefail</literal> set: |
||||
<itemizedlist> |
||||
<listitem> |
||||
<para> |
||||
If several commands are separated by <literal>;</literal> |
||||
and one fails, the command as a whole will fail. |
||||
</para> |
||||
</listitem> |
||||
<listitem> |
||||
<para> |
||||
For pipelines, the last non-zero exit status will be |
||||
returned (if there is one, zero will be returned |
||||
otherwise). |
||||
</para> |
||||
</listitem> |
||||
<listitem> |
||||
<para> |
||||
Dereferencing unset variables fail the command. |
||||
</para> |
||||
</listitem> |
||||
</itemizedlist> |
||||
</para> |
||||
</listitem> |
||||
</varlistentry> |
||||
<varlistentry> |
||||
<term> |
||||
<methodname>fail</methodname> |
||||
</term> |
||||
<listitem> |
||||
<para> |
||||
Like <methodname>succeed</methodname>, but raising an exception if the |
||||
command returns a zero status. |
||||
</para> |
||||
</listitem> |
||||
</varlistentry> |
||||
<varlistentry> |
||||
<term> |
||||
<methodname>wait_until_succeeds</methodname> |
||||
</term> |
||||
<listitem> |
||||
<para> |
||||
Repeat a shell command with 1-second intervals until it succeeds. |
||||
</para> |
||||
</listitem> |
||||
</varlistentry> |
||||
<varlistentry> |
||||
<term> |
||||
<methodname>wait_until_fails</methodname> |
||||
</term> |
||||
<listitem> |
||||
<para> |
||||
Repeat a shell command with 1-second intervals until it fails. |
||||
</para> |
||||
</listitem> |
||||
</varlistentry> |
||||
<varlistentry> |
||||
<term> |
||||
<methodname>wait_for_unit</methodname> |
||||
</term> |
||||
<listitem> |
||||
<para> |
||||
Wait until the specified systemd unit has reached the “active” state. |
||||
</para> |
||||
</listitem> |
||||
</varlistentry> |
||||
<varlistentry> |
||||
<term> |
||||
<methodname>wait_for_file</methodname> |
||||
</term> |
||||
<listitem> |
||||
<para> |
||||
Wait until the specified file exists. |
||||
</para> |
||||
</listitem> |
||||
</varlistentry> |
||||
<varlistentry> |
||||
<term> |
||||
<methodname>wait_for_open_port</methodname> |
||||
</term> |
||||
<listitem> |
||||
<para> |
||||
Wait until a process is listening on the given TCP port (on |
||||
<literal>localhost</literal>, at least). |
||||
</para> |
||||
</listitem> |
||||
</varlistentry> |
||||
<varlistentry> |
||||
<term> |
||||
<methodname>wait_for_closed_port</methodname> |
||||
</term> |
||||
<listitem> |
||||
<para> |
||||
Wait until nobody is listening on the given TCP port. |
||||
</para> |
||||
</listitem> |
||||
</varlistentry> |
||||
<varlistentry> |
||||
<term> |
||||
<methodname>wait_for_x</methodname> |
||||
</term> |
||||
<listitem> |
||||
<para> |
||||
Wait until the X11 server is accepting connections. |
||||
</para> |
||||
</listitem> |
||||
</varlistentry> |
||||
<varlistentry> |
||||
<term> |
||||
<methodname>wait_for_text</methodname> |
||||
</term> |
||||
<listitem> |
||||
<para> |
||||
Wait until the supplied regular expressions matches the textual contents |
||||
of the screen by using optical character recognition (see |
||||
<methodname>get_screen_text</methodname> and |
||||
<methodname>get_screen_text_variants</methodname>). |
||||
</para> |
||||
<note> |
||||
<para> |
||||
This requires passing <option>enableOCR</option> to the test attribute |
||||
set. |
||||
</para> |
||||
</note> |
||||
</listitem> |
||||
</varlistentry> |
||||
<varlistentry> |
||||
<term> |
||||
<methodname>wait_for_console_text</methodname> |
||||
</term> |
||||
<listitem> |
||||
<para> |
||||
Wait until the supplied regular expressions match a line of the serial |
||||
console output. This method is useful when OCR is not possibile or |
||||
accurate enough. |
||||
</para> |
||||
</listitem> |
||||
</varlistentry> |
||||
<varlistentry> |
||||
<term> |
||||
<methodname>wait_for_window</methodname> |
||||
</term> |
||||
<listitem> |
||||
<para> |
||||
Wait until an X11 window has appeared whose name matches the given |
||||
regular expression, e.g., <literal>wait_for_window("Terminal")</literal>. |
||||
</para> |
||||
</listitem> |
||||
</varlistentry> |
||||
<varlistentry> |
||||
<term> |
||||
<methodname>copy_from_host</methodname> |
||||
</term> |
||||
<listitem> |
||||
<para> |
||||
Copies a file from host to machine, e.g., |
||||
<literal>copy_from_host("myfile", "/etc/my/important/file")</literal>. |
||||
</para> |
||||
<para> |
||||
The first argument is the file on the host. The file needs to be |
||||
accessible while building the nix derivation. The second argument is the |
||||
location of the file on the machine. |
||||
</para> |
||||
</listitem> |
||||
</varlistentry> |
||||
<varlistentry> |
||||
<term> |
||||
<methodname>systemctl</methodname> |
||||
</term> |
||||
<listitem> |
||||
<para> |
||||
Runs <literal>systemctl</literal> commands with optional support for |
||||
<literal>systemctl --user</literal> |
||||
</para> |
||||
<para> |
||||
<programlisting> |
||||
machine.systemctl("list-jobs --no-pager") # runs `systemctl list-jobs --no-pager` |
||||
machine.systemctl("list-jobs --no-pager", "any-user") # spawns a shell for `any-user` and runs `systemctl --user list-jobs --no-pager` |
||||
</programlisting> |
||||
</para> |
||||
</listitem> |
||||
</varlistentry> |
||||
<varlistentry> |
||||
<term> |
||||
<methodname>shell_interact</methodname> |
||||
</term> |
||||
<listitem> |
||||
<para> |
||||
Allows you to directly interact with the guest shell. |
||||
This should only be used during test development, not in production tests. |
||||
Killing the interactive session with <literal>Ctrl-d</literal> or <literal>Ctrl-c</literal> also ends the guest session. |
||||
</para> |
||||
</listitem> |
||||
</varlistentry> |
||||
</variablelist> |
||||
</para> |
||||
|
||||
<para> |
||||
To test user units declared by <literal>systemd.user.services</literal> the |
||||
optional <literal>user</literal> argument can be used: |
||||
<programlisting> |
||||
machine.start() |
||||
machine.wait_for_x() |
||||
machine.wait_for_unit("xautolock.service", "x-session-user") |
||||
</programlisting> |
||||
This applies to <literal>systemctl</literal>, <literal>get_unit_info</literal>, |
||||
<literal>wait_for_unit</literal>, <literal>start_job</literal> and |
||||
<literal>stop_job</literal>. |
||||
</para> |
||||
|
||||
<para> |
||||
For faster dev cycles it's also possible to disable the code-linters (this shouldn't |
||||
be commited though): |
||||
<programlisting> |
||||
import ./make-test-python.nix { |
||||
skipLint = true; |
||||
machine = |
||||
{ config, pkgs, ... }: |
||||
{ <replaceable>configuration…</replaceable> |
||||
}; |
||||
|
||||
testScript = |
||||
'' |
||||
<replaceable>Python code…</replaceable> |
||||
''; |
||||
} |
||||
</programlisting> |
||||
This will produce a Nix warning at evaluation time. To fully disable the |
||||
linter, wrap the test script in comment directives to disable the Black linter |
||||
directly (again, don't commit this within the Nixpkgs repository): |
||||
<programlisting> |
||||
testScript = |
||||
'' |
||||
# fmt: off |
||||
<replaceable>Python code…</replaceable> |
||||
# fmt: on |
||||
''; |
||||
</programlisting> |
||||
</para> |
||||
</section> |
@ -0,0 +1,526 @@ |
||||
<section xmlns="http://docbook.org/ns/docbook" xmlns:xlink="http://www.w3.org/1999/xlink" xml:id="sec-writing-nixos-tests"> |
||||
<title>Writing Tests</title> |
||||
<para> |
||||
A NixOS test is a Nix expression that has the following structure: |
||||
</para> |
||||
<programlisting language="bash"> |
||||
import ./make-test-python.nix { |
||||
|
||||
# Either the configuration of a single machine: |
||||
machine = |
||||
{ config, pkgs, ... }: |
||||
{ configuration… |
||||
}; |
||||
|
||||
# Or a set of machines: |
||||
nodes = |
||||
{ machine1 = |
||||
{ config, pkgs, ... }: { … }; |
||||
machine2 = |
||||
{ config, pkgs, ... }: { … }; |
||||
… |
||||
}; |
||||
|
||||
testScript = |
||||
'' |
||||
Python code… |
||||
''; |
||||
} |
||||
</programlisting> |
||||
<para> |
||||
The attribute <literal>testScript</literal> is a bit of Python code |
||||
that executes the test (described below). During the test, it will |
||||
start one or more virtual machines, the configuration of which is |
||||
described by the attribute <literal>machine</literal> (if you need |
||||
only one machine in your test) or by the attribute |
||||
<literal>nodes</literal> (if you need multiple machines). For |
||||
instance, |
||||
<link xlink:href="https://github.com/NixOS/nixpkgs/blob/master/nixos/tests/login.nix"><literal>login.nix</literal></link> |
||||
only needs a single machine to test whether users can log in on the |
||||
virtual console, whether device ownership is correctly maintained |
||||
when switching between consoles, and so on. On the other hand, |
||||
<link xlink:href="https://github.com/NixOS/nixpkgs/blob/master/nixos/tests/nfs/simple.nix"><literal>nfs/simple.nix</literal></link>, |
||||
which tests NFS client and server functionality in the Linux kernel |
||||
(including whether locks are maintained across server crashes), |
||||
requires three machines: a server and two clients. |
||||
</para> |
||||
<para> |
||||
There are a few special NixOS configuration options for test VMs: |
||||
</para> |
||||
<variablelist> |
||||
<varlistentry> |
||||
<term> |
||||
<literal>virtualisation.memorySize</literal> |
||||
</term> |
||||
<listitem> |
||||
<para> |
||||
The memory of the VM in megabytes. |
||||
</para> |
||||
</listitem> |
||||
</varlistentry> |
||||
<varlistentry> |
||||
<term> |
||||
<literal>virtualisation.vlans</literal> |
||||
</term> |
||||
<listitem> |
||||
<para> |
||||
The virtual networks to which the VM is connected. See |
||||
<link xlink:href="https://github.com/NixOS/nixpkgs/blob/master/nixos/tests/nat.nix"><literal>nat.nix</literal></link> |
||||
for an example. |
||||
</para> |
||||
</listitem> |
||||
</varlistentry> |
||||
<varlistentry> |
||||
<term> |
||||
<literal>virtualisation.writableStore</literal> |
||||
</term> |
||||
<listitem> |
||||
<para> |
||||
By default, the Nix store in the VM is not writable. If you |
||||
enable this option, a writable union file system is mounted on |
||||
top of the Nix store to make it appear writable. This is |
||||
necessary for tests that run Nix operations that modify the |
||||
store. |
||||
</para> |
||||
</listitem> |
||||
</varlistentry> |
||||
</variablelist> |
||||
<para> |
||||
For more options, see the module |
||||
<link xlink:href="https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/virtualisation/qemu-vm.nix"><literal>qemu-vm.nix</literal></link>. |
||||
</para> |
||||
<para> |
||||
The test script is a sequence of Python statements that perform |
||||
various actions, such as starting VMs, executing commands in the |
||||
VMs, and so on. Each virtual machine is represented as an object |
||||
stored in the variable <literal>name</literal> if this is also the |
||||
identifier of the machine in the declarative config. If you didn't |
||||
specify multiple machines using the <literal>nodes</literal> |
||||
attribute, it is just <literal>machine</literal>. The following |
||||
example starts the machine, waits until it has finished booting, |
||||
then executes a command and checks that the output is more-or-less |
||||
correct: |
||||
</para> |
||||
<programlisting language="python"> |
||||
machine.start() |
||||
machine.wait_for_unit("default.target") |
||||
if not "Linux" in machine.succeed("uname"): |
||||
raise Exception("Wrong OS") |
||||
</programlisting> |
||||
<para> |
||||
The first line is actually unnecessary; machines are implicitly |
||||
started when you first execute an action on them (such as |
||||
<literal>wait_for_unit</literal> or <literal>succeed</literal>). If |
||||
you have multiple machines, you can speed up the test by starting |
||||
them in parallel: |
||||
</para> |
||||
<programlisting language="python"> |
||||
start_all() |
||||
</programlisting> |
||||
<para> |
||||
The following methods are available on machine objects: |
||||
</para> |
||||
<variablelist> |
||||
<varlistentry> |
||||
<term> |
||||
<literal>start</literal> |
||||
</term> |
||||
<listitem> |
||||
<para> |
||||
Start the virtual machine. This method is asynchronous — it |
||||
does not wait for the machine to finish booting. |
||||
</para> |
||||
</listitem> |
||||
</varlistentry> |
||||
<varlistentry> |
||||
<term> |
||||
<literal>shutdown</literal> |
||||
</term> |
||||
<listitem> |
||||
<para> |
||||
Shut down the machine, waiting for the VM to exit. |
||||
</para> |
||||
</listitem> |
||||
</varlistentry> |
||||
<varlistentry> |
||||
<term> |
||||
<literal>crash</literal> |
||||
</term> |
||||
<listitem> |
||||
<para> |
||||
Simulate a sudden power failure, by telling the VM to exit |
||||
immediately. |
||||
</para> |
||||
</listitem> |
||||
</varlistentry> |
||||
<varlistentry> |
||||
<term> |
||||
<literal>block</literal> |
||||
</term> |
||||
<listitem> |
||||
<para> |
||||
Simulate unplugging the Ethernet cable that connects the |
||||
machine to the other machines. |
||||
</para> |
||||
</listitem> |
||||
</varlistentry> |
||||
<varlistentry> |
||||
<term> |
||||
<literal>unblock</literal> |
||||
</term> |
||||
<listitem> |
||||
<para> |
||||
Undo the effect of <literal>block</literal>. |
||||
</para> |
||||
</listitem> |
||||
</varlistentry> |
||||
<varlistentry> |
||||
<term> |
||||
<literal>screenshot</literal> |
||||
</term> |
||||
<listitem> |
||||
<para> |
||||
Take a picture of the display of the virtual machine, in PNG |
||||
format. The screenshot is linked from the HTML log. |
||||
</para> |
||||
</listitem> |
||||
</varlistentry> |
||||
<varlistentry> |
||||
<term> |
||||
<literal>get_screen_text_variants</literal> |
||||
</term> |
||||
<listitem> |
||||
<para> |
||||
Return a list of different interpretations of what is |
||||
currently visible on the machine's screen using optical |
||||
character recognition. The number and order of the |
||||
interpretations is not specified and is subject to change, but |
||||
if no exception is raised at least one will be returned. |
||||
</para> |
||||
<note> |
||||
<para> |
||||
This requires passing <literal>enableOCR</literal> to the |
||||
test attribute set. |
||||
</para> |
||||
</note> |
||||
</listitem> |
||||
</varlistentry> |
||||
<varlistentry> |
||||
<term> |
||||
<literal>get_screen_text</literal> |
||||
</term> |
||||
<listitem> |
||||
<para> |
||||
Return a textual representation of what is currently visible |
||||
on the machine's screen using optical character recognition. |
||||
</para> |
||||
<note> |
||||
<para> |
||||
This requires passing <literal>enableOCR</literal> to the |
||||
test attribute set. |
||||
</para> |
||||
</note> |
||||
</listitem> |
||||
</varlistentry> |
||||
<varlistentry> |
||||
<term> |
||||
<literal>send_monitor_command</literal> |
||||
</term> |
||||
<listitem> |
||||
<para> |
||||
Send a command to the QEMU monitor. This is rarely used, but |
||||
allows doing stuff such as attaching virtual USB disks to a |
||||
running machine. |
||||
</para> |
||||
</listitem> |
||||
</varlistentry> |
||||
<varlistentry> |
||||
<term> |
||||
<literal>send_key</literal> |
||||
</term> |
||||
<listitem> |
||||
<para> |
||||
Simulate pressing keys on the virtual keyboard, e.g., |
||||
<literal>send_key("ctrl-alt-delete")</literal>. |
||||
</para> |
||||
</listitem> |
||||
</varlistentry> |
||||
<varlistentry> |
||||
<term> |
||||
<literal>send_chars</literal> |
||||
</term> |
||||
<listitem> |
||||
<para> |
||||
Simulate typing a sequence of characters on the virtual |
||||
keyboard, e.g., |
||||
<literal>send_chars("foobar\n")</literal> will type |
||||
the string <literal>foobar</literal> followed by the Enter |
||||
key. |
||||
</para> |
||||
</listitem> |
||||
</varlistentry> |
||||
<varlistentry> |
||||
<term> |
||||
<literal>execute</literal> |
||||
</term> |
||||
<listitem> |
||||
<para> |
||||
Execute a shell command, returning a list |
||||
<literal>(status, stdout)</literal>. |
||||
</para> |
||||
</listitem> |
||||
</varlistentry> |
||||
<varlistentry> |
||||
<term> |
||||
<literal>succeed</literal> |
||||
</term> |
||||
<listitem> |
||||
<para> |
||||
Execute a shell command, raising an exception if the exit |
||||
status is not zero, otherwise returning the standard output. |
||||
Commands are run with <literal>set -euo pipefail</literal> |
||||
set: |
||||
</para> |
||||
<itemizedlist> |
||||
<listitem> |
||||
<para> |
||||
If several commands are separated by <literal>;</literal> |
||||
and one fails, the command as a whole will fail. |
||||
</para> |
||||
</listitem> |
||||
<listitem> |
||||
<para> |
||||
For pipelines, the last non-zero exit status will be |
||||
returned (if there is one, zero will be returned |
||||
otherwise). |
||||
</para> |
||||
</listitem> |
||||
<listitem> |
||||
<para> |
||||
Dereferencing unset variables fail the command. |
||||
</para> |
||||
</listitem> |
||||
</itemizedlist> |
||||
</listitem> |
||||
</varlistentry> |
||||
<varlistentry> |
||||
<term> |
||||
<literal>fail</literal> |
||||
</term> |
||||
<listitem> |
||||
<para> |
||||
Like <literal>succeed</literal>, but raising an exception if |
||||
the command returns a zero status. |
||||
</para> |
||||
</listitem> |
||||
</varlistentry> |
||||
<varlistentry> |
||||
<term> |
||||
<literal>wait_until_succeeds</literal> |
||||
</term> |
||||
<listitem> |
||||
<para> |
||||
Repeat a shell command with 1-second intervals until it |
||||
succeeds. |
||||
</para> |
||||
</listitem> |
||||
</varlistentry> |
||||
<varlistentry> |
||||
<term> |
||||
<literal>wait_until_fails</literal> |
||||
</term> |
||||
<listitem> |
||||
<para> |
||||
Repeat a shell command with 1-second intervals until it fails. |
||||
</para> |
||||
</listitem> |
||||
</varlistentry> |
||||
<varlistentry> |
||||
<term> |
||||
<literal>wait_for_unit</literal> |
||||
</term> |
||||
<listitem> |
||||
<para> |
||||
Wait until the specified systemd unit has reached the |
||||
<quote>active</quote> state. |
||||
</para> |
||||
</listitem> |
||||
</varlistentry> |
||||
<varlistentry> |
||||
<term> |
||||
<literal>wait_for_file</literal> |
||||
</term> |
||||
<listitem> |
||||
<para> |
||||
Wait until the specified file exists. |
||||
</para> |
||||
</listitem> |
||||
</varlistentry> |
||||
<varlistentry> |
||||
<term> |
||||
<literal>wait_for_open_port</literal> |
||||
</term> |
||||
<listitem> |
||||
<para> |
||||
Wait until a process is listening on the given TCP port (on |
||||
<literal>localhost</literal>, at least). |
||||
</para> |
||||
</listitem> |
||||
</varlistentry> |
||||
<varlistentry> |
||||
<term> |
||||
<literal>wait_for_closed_port</literal> |
||||
</term> |
||||
<listitem> |
||||
<para> |
||||
Wait until nobody is listening on the given TCP port. |
||||
</para> |
||||
</listitem> |
||||
</varlistentry> |
||||
<varlistentry> |
||||
<term> |
||||
<literal>wait_for_x</literal> |
||||
</term> |
||||
<listitem> |
||||
<para> |
||||
Wait until the X11 server is accepting connections. |
||||
</para> |
||||
</listitem> |
||||
</varlistentry> |
||||
<varlistentry> |
||||
<term> |
||||
<literal>wait_for_text</literal> |
||||
</term> |
||||
<listitem> |
||||
<para> |
||||
Wait until the supplied regular expressions matches the |
||||
textual contents of the screen by using optical character |
||||
recognition (see <literal>get_screen_text</literal> and |
||||
<literal>get_screen_text_variants</literal>). |
||||
</para> |
||||
<note> |
||||
<para> |
||||
This requires passing <literal>enableOCR</literal> to the |
||||
test attribute set. |
||||
</para> |
||||
</note> |
||||
</listitem> |
||||
</varlistentry> |
||||
<varlistentry> |
||||
<term> |
||||
<literal>wait_for_console_text</literal> |
||||
</term> |
||||
<listitem> |
||||
<para> |
||||
Wait until the supplied regular expressions match a line of |
||||
the serial console output. This method is useful when OCR is |
||||
not possibile or accurate enough. |
||||
</para> |
||||
</listitem> |
||||
</varlistentry> |
||||
<varlistentry> |
||||
<term> |
||||
<literal>wait_for_window</literal> |
||||
</term> |
||||
<listitem> |
||||
<para> |
||||
Wait until an X11 window has appeared whose name matches the |
||||
given regular expression, e.g., |
||||
<literal>wait_for_window("Terminal")</literal>. |
||||
</para> |
||||
</listitem> |
||||
</varlistentry> |
||||
<varlistentry> |
||||
<term> |
||||
<literal>copy_from_host</literal> |
||||
</term> |
||||
<listitem> |
||||
<para> |
||||
Copies a file from host to machine, e.g., |
||||
<literal>copy_from_host("myfile", "/etc/my/important/file")</literal>. |
||||
</para> |
||||
<para> |
||||
The first argument is the file on the host. The file needs to |
||||
be accessible while building the nix derivation. The second |
||||
argument is the location of the file on the machine. |
||||
</para> |
||||
</listitem> |
||||
</varlistentry> |
||||
<varlistentry> |
||||
<term> |
||||
<literal>systemctl</literal> |
||||
</term> |
||||
<listitem> |
||||
<para> |
||||
Runs <literal>systemctl</literal> commands with optional |
||||
support for <literal>systemctl --user</literal> |
||||
</para> |
||||
<programlisting language="python"> |
||||
machine.systemctl("list-jobs --no-pager") # runs `systemctl list-jobs --no-pager` |
||||
machine.systemctl("list-jobs --no-pager", "any-user") # spawns a shell for `any-user` and runs `systemctl --user list-jobs --no-pager` |
||||
</programlisting> |
||||
</listitem> |
||||
</varlistentry> |
||||
<varlistentry> |
||||
<term> |
||||
<literal>shell_interact</literal> |
||||
</term> |
||||
<listitem> |
||||
<para> |
||||
Allows you to directly interact with the guest shell. This |
||||
should only be used during test development, not in production |
||||
tests. Killing the interactive session with |
||||
<literal>Ctrl-d</literal> or <literal>Ctrl-c</literal> also |
||||
ends the guest session. |
||||
</para> |
||||
</listitem> |
||||
</varlistentry> |
||||
</variablelist> |
||||
<para> |
||||
To test user units declared by |
||||
<literal>systemd.user.services</literal> the optional |
||||
<literal>user</literal> argument can be used: |
||||
</para> |
||||
<programlisting language="python"> |
||||
machine.start() |
||||
machine.wait_for_x() |
||||
machine.wait_for_unit("xautolock.service", "x-session-user") |
||||
</programlisting> |
||||
<para> |
||||
This applies to <literal>systemctl</literal>, |
||||
<literal>get_unit_info</literal>, <literal>wait_for_unit</literal>, |
||||
<literal>start_job</literal> and <literal>stop_job</literal>. |
||||
</para> |
||||
<para> |
||||
For faster dev cycles it's also possible to disable the code-linters |
||||
(this shouldn't be commited though): |
||||
</para> |
||||
<programlisting language="bash"> |
||||
import ./make-test-python.nix { |
||||
skipLint = true; |
||||
machine = |
||||
{ config, pkgs, ... }: |
||||
{ configuration… |
||||
}; |
||||
|
||||
testScript = |
||||
'' |
||||
Python code… |
||||
''; |
||||
} |
||||
</programlisting> |
||||
<para> |
||||
This will produce a Nix warning at evaluation time. To fully disable |
||||
the linter, wrap the test script in comment directives to disable |
||||
the Black linter directly (again, don't commit this within the |
||||
Nixpkgs repository): |
||||
</para> |
||||
<programlisting language="bash"> |
||||
testScript = |
||||
'' |
||||
# fmt: off |
||||
Python code… |
||||
# fmt: on |
||||
''; |
||||
</programlisting> |
||||
</section> |
Loading…
Reference in new issue