Merge pull request #122728 from cdepillabout/haskell-updates-workflow
haskell-updates: add documentation for workflowlaunchpad/nixpkgs/master
commit
718ab6dada
@ -0,0 +1,270 @@ |
||||
|
||||
## Maintainer Workflow |
||||
|
||||
The goal of the [@NixOS/haskell](https://github.com/orgs/NixOS/teams/haskell) |
||||
team is to keep the Haskell packages in Nixpkgs up-to-date, while making sure |
||||
there are no Haskell-related evaluation errors or build errors that get into |
||||
the Nixpkgs `master` branch. |
||||
|
||||
We do this by periodically merging an updated set of Haskell packages on the |
||||
`haskell-updates` branch into the `master` branch. Each member of the team |
||||
takes a two week period where they are in charge of merging the |
||||
`haskell-updates` branch into `master`. This is the documentation for this |
||||
workflow. |
||||
|
||||
The workflow generally proceeds in three main steps: |
||||
|
||||
1. create the initial `haskell-updates` PR, and update Stackage and Hackage snapshots |
||||
1. wait for contributors to fix newly broken Haskell packages |
||||
1. merge `haskell-updates` into `master` |
||||
|
||||
Each of these steps is described in a separate section. |
||||
|
||||
### Initial `haskell-updates` PR |
||||
|
||||
In this section we create the PR for merging `haskell-updates` into `master`. |
||||
|
||||
1. Make sure the `haskell-updates` branch is up-to-date with `master`. |
||||
|
||||
1. Update the Stackage Nightly resolver used by Nixpkgs and create a commit: |
||||
|
||||
```console |
||||
$ ./maintainers/scripts/haskell/update-stackage.sh --do-commit |
||||
``` |
||||
|
||||
1. Update the Hackage package set used by Nixpkgs and create a commit: |
||||
|
||||
```console |
||||
$ ./maintainers/scripts/haskell/update-hackage.sh --do-commit |
||||
``` |
||||
|
||||
1. Regenerate the Haskell package set used in Nixpkgs and create a commit: |
||||
|
||||
```console |
||||
$ ./maintainers/scripts/haskell/regenerate-hackage-packages.sh --do-commit |
||||
``` |
||||
|
||||
1. Push these commits to the Nixpkgs repository. |
||||
|
||||
1. Open a PR on Nixpkgs merging `haskell-updates` into `master`. |
||||
|
||||
Use the following message body: |
||||
|
||||
```markdown |
||||
### This Merge |
||||
|
||||
This PR is the regular merge of the `haskell-updates` branch into `master`. |
||||
|
||||
This branch is being continually built and tested by hydra at https://hydra.nixos.org/jobset/nixpkgs/haskell-updates. |
||||
|
||||
I will aim to merge this PR **by 2021-TODO-TODO**. If I can merge it earlier, there might be successor PRs in that time window. As part of our rotation @TODO will continue these merges from 2021-TODO-TODO to 2021-TODO-TODO. |
||||
|
||||
### haskellPackages Workflow Summary |
||||
|
||||
Our workflow is currently described in |
||||
[`pkgs/development/haskell-modules/HACKING.md`](https://github.com/NixOS/nixpkgs/blob/haskell-updates/pkgs/development/haskell-modules/HACKING.md). |
||||
|
||||
The short version is this: |
||||
* We regularly update the Stackage and Hackage pins on `haskell-updates` (normally at the beginning of a merge window). |
||||
* The community fixes builds of Haskell packages on that branch. |
||||
* We aim at at least one merge of `haskell-updates` into `master` every two weeks. |
||||
* We only do the merge if the `mergeable` job is succeeding on hydra. |
||||
* If a maintained package is still broken at the time of merge, we will only merge if the maintainer has been pinged 7 days in advance. (If you care about a Haskell package, become a maintainer!) |
||||
|
||||
--- |
||||
|
||||
This is the follow-up to #TODO. |
||||
``` |
||||
|
||||
Make sure to replace all TODO with the actual values. |
||||
|
||||
### Notify Maintainers and Fix Broken Packages |
||||
|
||||
After you've done the previous steps, Hydra will start building the new and |
||||
updated Haskell packages. You can see the progress Hydra is making at |
||||
https://hydra.nixos.org/jobset/nixpkgs/haskell-updates. This Hydra jobset is |
||||
defined in the file [release-haskell.nix](../../top-level/release-haskell.nix). |
||||
|
||||
#### Notify Maintainers |
||||
|
||||
When Hydra finishes building all the updated packages for the `haskell-updates` |
||||
jobset, you should generate a build report to notify maintainers of their |
||||
newly broken packages. You can do that with the following commands: |
||||
|
||||
```console |
||||
$ ./maintainers/scripts/haskell/hydra-report.hs get-report |
||||
$ ./maintainers/scripts/haskell/hydra-report.hs ping-maintainers |
||||
``` |
||||
|
||||
The `hyda-report.hs ping-maintainers` command generates a Markdown document |
||||
that you can paste in a GitHub comment on the PR opened above. This |
||||
comment describes which Haskell packages are now failing to build. It also |
||||
pings the maintainers so that they know to fix up their packages. |
||||
|
||||
It may be helpful to pipe `hydra-report.hs ping-maintainers` into `xclip` |
||||
(XOrg) or `wl-copy` (Wayland) in order to post on GitHub. |
||||
|
||||
This build report can be fetched and re-generated for new Hydra evaluations. |
||||
It may help contributors to try to keep the GitHub comment updated with the |
||||
most recent build report. |
||||
|
||||
Maintainers should be given at least 7 days to fix up their packages when they |
||||
break. If maintainers don't fix up their packages with 7 days, then they |
||||
may be marked broken before merging `haskell-updates` into `master`. |
||||
|
||||
#### Fix Broken Packages |
||||
|
||||
After getting the build report, you can see which packages and Hydra jobs are |
||||
failing to build. The most important jobs are the `maintained` and `mergeable` |
||||
jobs. These are both defined in |
||||
[`release-haskell.nix`](../../top-level/release-haskell.nix). |
||||
|
||||
`mergeable` is a set of the most important Haskell packages, including things |
||||
like Pandoc and XMonad. These packages are widely used. We would like to |
||||
always keep these building. |
||||
|
||||
`maintained` is a set of Haskell packages that have maintainers in Nixpkgs. |
||||
We should be proactive in working with maintainers to keep their packages |
||||
building. |
||||
|
||||
Steps to fix Haskell packages that are failing to build is out of scope for |
||||
this document, but it usually requires fixing up dependencies that are now |
||||
out-of-bounds. |
||||
|
||||
#### Mark Broken Packages |
||||
|
||||
Packages that do not get fixed can be marked broken with the following |
||||
commands. First check which packages are broken: |
||||
|
||||
```console |
||||
$ ./maintainers/scripts/haskell/hydra-report.hs get-report |
||||
$ ./maintainers/scripts/haskell/hydra-report.hs mark-broken-list |
||||
``` |
||||
|
||||
This shows a list of packages that reported a build failure on `x86_64-linux` on Hydra. |
||||
|
||||
Next, run the following command: |
||||
|
||||
```console |
||||
$ ./maintainers/scripts/haskell/mark-broken.sh --do-commit |
||||
``` |
||||
|
||||
This first opens up an editor with the broken package list. Some of these |
||||
packages may have a maintainer in Nixpkgs. If these maintainers have not been |
||||
given 7 days to fix up their package, then make sure to remove those packages |
||||
from the list before continuing. After saving and exiting the editor, the |
||||
following will happen: |
||||
|
||||
- Packages from the list will be added to |
||||
[`configuration-hackage2nix/broken.yaml`](configuration-hackage2nix/broken.yaml). |
||||
This is a list of Haskell packages that are known to be broken. |
||||
|
||||
- [`hackage-packages.nix`](hackage-packages.nix) will be regenerated. This |
||||
will mark all Haskell pacakges in `configuration-hackage2nix/broken.yaml` |
||||
as `broken`. |
||||
|
||||
- The |
||||
[`configuration-hackage2nix/transitive-broken.yaml`](configuration-hackage2nix/transitive-broken.yaml) |
||||
file will be updated. This is a list of Haskell packages that |
||||
depend on a package in `configuration-hackage2nix/broken.yaml` or |
||||
`configuration-hackage2nix/transitive-broken.yaml` |
||||
|
||||
- `hackage-packages.nix` will be regenerated again. This will set |
||||
`hydraPlatforms = none` for all the packages in |
||||
`configuration-hackage2nix/transitive-broken.yaml`. This makes |
||||
sure that Hydra does not try to build any of these packages. |
||||
|
||||
- All updated files will be committed. |
||||
|
||||
#### Merge `master` into `haskell-updates` |
||||
|
||||
You should occasionally merge the `master` branch into the `haskell-updates` |
||||
branch. |
||||
|
||||
In an ideal world, when we merge `haskell-updates` into `master`, it would |
||||
cause few Hydra rebuilds on `master`. Ideally, the `nixos-unstable` channel |
||||
would never be prevented from progressing because of needing to wait for |
||||
rebuilding Haskell packages. |
||||
|
||||
In order to make sure that there are a minimal number of rebuilds after merging |
||||
`haskell-updates` into `master`, `master` should occasionally be merged into |
||||
the `haskell-updates` branch. |
||||
|
||||
This is especially important after `staging-next` is merged into `master`, |
||||
since there is a high chance that this will cause all the Haskell packages to |
||||
rebuild. |
||||
|
||||
### Merge `haskell-updates` into `master` |
||||
|
||||
Now it is time to merge the `haskell-updates` PR you opened above. |
||||
|
||||
Before doing this, make sure of the following: |
||||
|
||||
- All Haskell packages that fail to build are correctly marked broken or |
||||
transitively broken. |
||||
|
||||
- The `maintained` and `mergeable` jobs are passing on Hydra. |
||||
|
||||
- The maintainers for any maintained Haskell packages that are newly broken |
||||
have been pinged on GitHub and given at least a week to fix their packages. |
||||
This is especially important for widely-used packages like `cachix`. |
||||
|
||||
- Make sure you first merge the `master` branch into `haskell-updates`. Wait |
||||
for Hydra to evaluate the new `haskell-updates` jobset. Make sure you only |
||||
merge `haskell-updates` into `master` when there are no evaluation errors. |
||||
|
||||
When you've double-checked these points, go ahead and merge the `haskell-updates` PR. |
||||
After merging, **make sure not to delete the `haskell-updates` branch**, since it |
||||
causes all currently open Haskell-related pull-requests to be automatically closed on GitHub. |
||||
|
||||
### Additional Info |
||||
|
||||
Here are some additional tips that didn't fit in above. |
||||
|
||||
- Hydra tries to evalute the `haskell-updates` branch (in the |
||||
[`nixpkgs:haskell-updates`](https://hydra.nixos.org/jobset/nixpkgs/haskell-updates) |
||||
jobset) every 4 hours. It is possible to force a new Hydra evaluation without |
||||
waiting 4 hours by the following steps: |
||||
|
||||
1. Log into Hydra with your GitHub or Google account. |
||||
1. Go to the [nixpkgs:haskell-updates](https://hydra.nixos.org/jobset/nixpkgs/haskell-updates) jobset. |
||||
1. Click the `Actions` button. |
||||
1. Select `Evaluate this jobset`. |
||||
1. If you refresh the page, there should be a new `Evaluation running since:` line. |
||||
1. Evaluations take about 10 minutes to finish. |
||||
|
||||
- It is sometimes helpful to update the version of |
||||
[`cabal2nix` / `hackage2nix`](https://github.com/NixOS/cabal2nix) that our |
||||
maintainer scripts use. This can be done with the |
||||
[`maintainers/scripts/haskell/update-cabal2nix-unstable.sh`](../../../maintainers/scripts/haskell/update-cabal2nix-unstable.sh) |
||||
script. |
||||
|
||||
You might want to do this if a user contributes a fix to `cabal2nix` that |
||||
will immediately fix a Haskell package in Nixpkgs. First, merge in |
||||
the PR to `cabal2nix`, then run `update-cabal2nix-upstable.sh`. Finally, run |
||||
[`regenerate-hackage-packages.sh`](../../../maintainers/scripts/haskell/regenerate-hackage-packages.sh) |
||||
to regenerate the Hackage package set with the updated version of `hackage2nix`. |
||||
|
||||
- Make sure never to update the Hackage package hashes in |
||||
[`pkgs/data/misc/hackage/`](../../../pkgs/data/misc/hackage/), or the |
||||
pinned Stackage Nightly versions on the release branches (like |
||||
`release-21.05`). |
||||
|
||||
This means that the |
||||
[`update-hackage.sh`](../../../maintainers/scripts/haskell/update-hackage.sh) |
||||
and |
||||
[`update-stackage.sh`](../../../maintainers/scripts/haskell/update-stackage.sh) |
||||
scripts should never be used on the release branches. |
||||
|
||||
However, changing other files in `./.` and regenerating the package set is encouraged. |
||||
This can be done with |
||||
[`regenerate-hackage-packages.sh`](../../../maintainers/scripts/haskell/regenerate-hackage-packages.sh) |
||||
as described above. |
||||
|
||||
- The Haskell team members generally hang out in the Matrix room |
||||
[#haskell-space:matrix.org](https://matrix.to/#/#haskell-space:matrix.org). |
||||
|
||||
## Contributor Workflow |
||||
|
||||
(TODO: this section is to describe the type of workflow for non-committers to |
||||
contribute to `haskell-updates`) |
Loading…
Reference in new issue