My personal project and infrastructure archive
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
nomicon/infra/libkookie/nixpkgs/unstable/pkgs/build-support/fetchzip/default.nix

70 lines
2.0 KiB

# This function downloads and unpacks an archive file, such as a zip
# or tar file. This is primarily useful for dynamically generated
# archives, such as GitHub's /archive URLs, where the unpacked content
# of the zip file doesn't change, but the zip file itself may
# (e.g. due to minor changes in the compression algorithm, or changes
# in timestamps).
{ lib, fetchurl, unzip }:
{ # Optionally move the contents of the unpacked tree up one level.
stripRoot ? true
, url ? ""
, urls ? []
, extraPostFetch ? ""
, name ? "source"
fetchzip: add extension parameter fetchzip downloads the file from specified URL, renames it to basename of that url, and then relies on unzip to do the unpacking. The first consequence is that this requires URL to end with proper extension—otherwise it will fail to unpack. This is not always the case and input-fonts workarounds this by adding “&.zip” query parameter (which is obviously a hack and is not guaranteed to work with every URL). The second consequence is that basename of the url must be a valid filename. I’ve tried to build a custom configuration of input-fonts and I get an error from mv that the filename is too long: > trying https://input.djr.com/build/?fontSelection=fourStyleFamily&regular=InputMonoNarrow-Regular&italic=InputMonoNarrow-Italic&bold=InputMonoNarrow-Bold&boldItalic=InputMonoNarrow-BoldItalic&a=0&g=0&i=topserif&l=serifs_round&zero=0&asterisk=height&braces=straight&preset=default&line-height=1.2&accept=I+do&email=&.zip > % Total % Received % Xferd Average Speed Time Time Time Current > Dload Upload Total Spent Left Speed > 100 406k 100 406k 0 0 230k 0 0:00:01 0:00:01 --:--:-- 230k > mv: failed to access '/build/?fontSelection=fourStyleFamily&regular=InputMonoNarrow-Regular&italic=InputMonoNarrow-Italic&bold=InputMonoNarrow-Bold&boldItalic=InputMonoNarrow-BoldItalic&a=0&g=0&i=topserif&l=serifs_round&zero=0&asterisk=height&braces=straight&preset=default&line-height=1.2&accept=I+do&email=&.zip': File name too long We could use “name” parameter as the filename (that’s how it is used in fetchurl). However, the previous attempt to do so (fc01353703426458d6990239936659ed3130d294) was reverted (24b5eb61eb944036b5716788ead1b204fba08719) because of the introduced regression—many fetchzip invocations use names without extension (also the default name is just “source”). This commit adds an optional “extension” parameter. If it is set, fetchzip renames the downloaded file to “download.${extension}” effectively solving both problems above without introducing a massive regression. This is a no-op for all existing packages. Tested by updating my NixOS setup + the extra inputs-fonts configuration mentioned above + tons of unstable emacs packages after a nix-collect-garbage (3Gb downloaded) with this patch applied.
3 years ago
, # Allows to set the extension for the intermediate downloaded
# file. This can be used as a hint for the unpackCmdHooks to select
# an appropriate unpacking tool.
extension ? null
, ... } @ args:
(fetchurl (let
fetchzip: add extension parameter fetchzip downloads the file from specified URL, renames it to basename of that url, and then relies on unzip to do the unpacking. The first consequence is that this requires URL to end with proper extension—otherwise it will fail to unpack. This is not always the case and input-fonts workarounds this by adding “&.zip” query parameter (which is obviously a hack and is not guaranteed to work with every URL). The second consequence is that basename of the url must be a valid filename. I’ve tried to build a custom configuration of input-fonts and I get an error from mv that the filename is too long: > trying https://input.djr.com/build/?fontSelection=fourStyleFamily&regular=InputMonoNarrow-Regular&italic=InputMonoNarrow-Italic&bold=InputMonoNarrow-Bold&boldItalic=InputMonoNarrow-BoldItalic&a=0&g=0&i=topserif&l=serifs_round&zero=0&asterisk=height&braces=straight&preset=default&line-height=1.2&accept=I+do&email=&.zip > % Total % Received % Xferd Average Speed Time Time Time Current > Dload Upload Total Spent Left Speed > 100 406k 100 406k 0 0 230k 0 0:00:01 0:00:01 --:--:-- 230k > mv: failed to access '/build/?fontSelection=fourStyleFamily&regular=InputMonoNarrow-Regular&italic=InputMonoNarrow-Italic&bold=InputMonoNarrow-Bold&boldItalic=InputMonoNarrow-BoldItalic&a=0&g=0&i=topserif&l=serifs_round&zero=0&asterisk=height&braces=straight&preset=default&line-height=1.2&accept=I+do&email=&.zip': File name too long We could use “name” parameter as the filename (that’s how it is used in fetchurl). However, the previous attempt to do so (fc01353703426458d6990239936659ed3130d294) was reverted (24b5eb61eb944036b5716788ead1b204fba08719) because of the introduced regression—many fetchzip invocations use names without extension (also the default name is just “source”). This commit adds an optional “extension” parameter. If it is set, fetchzip renames the downloaded file to “download.${extension}” effectively solving both problems above without introducing a massive regression. This is a no-op for all existing packages. Tested by updating my NixOS setup + the extra inputs-fonts configuration mentioned above + tons of unstable emacs packages after a nix-collect-garbage (3Gb downloaded) with this patch applied.
3 years ago
tmpFilename =
if extension != null
then "download.${extension}"
else baseNameOf (if url != "" then url else builtins.head urls);
in {
inherit name;
recursiveHash = true;
downloadToTemp = true;
postFetch =
''
unpackDir="$TMPDIR/unpack"
mkdir "$unpackDir"
cd "$unpackDir"
fetchzip: add extension parameter fetchzip downloads the file from specified URL, renames it to basename of that url, and then relies on unzip to do the unpacking. The first consequence is that this requires URL to end with proper extension—otherwise it will fail to unpack. This is not always the case and input-fonts workarounds this by adding “&.zip” query parameter (which is obviously a hack and is not guaranteed to work with every URL). The second consequence is that basename of the url must be a valid filename. I’ve tried to build a custom configuration of input-fonts and I get an error from mv that the filename is too long: > trying https://input.djr.com/build/?fontSelection=fourStyleFamily&regular=InputMonoNarrow-Regular&italic=InputMonoNarrow-Italic&bold=InputMonoNarrow-Bold&boldItalic=InputMonoNarrow-BoldItalic&a=0&g=0&i=topserif&l=serifs_round&zero=0&asterisk=height&braces=straight&preset=default&line-height=1.2&accept=I+do&email=&.zip > % Total % Received % Xferd Average Speed Time Time Time Current > Dload Upload Total Spent Left Speed > 100 406k 100 406k 0 0 230k 0 0:00:01 0:00:01 --:--:-- 230k > mv: failed to access '/build/?fontSelection=fourStyleFamily&regular=InputMonoNarrow-Regular&italic=InputMonoNarrow-Italic&bold=InputMonoNarrow-Bold&boldItalic=InputMonoNarrow-BoldItalic&a=0&g=0&i=topserif&l=serifs_round&zero=0&asterisk=height&braces=straight&preset=default&line-height=1.2&accept=I+do&email=&.zip': File name too long We could use “name” parameter as the filename (that’s how it is used in fetchurl). However, the previous attempt to do so (fc01353703426458d6990239936659ed3130d294) was reverted (24b5eb61eb944036b5716788ead1b204fba08719) because of the introduced regression—many fetchzip invocations use names without extension (also the default name is just “source”). This commit adds an optional “extension” parameter. If it is set, fetchzip renames the downloaded file to “download.${extension}” effectively solving both problems above without introducing a massive regression. This is a no-op for all existing packages. Tested by updating my NixOS setup + the extra inputs-fonts configuration mentioned above + tons of unstable emacs packages after a nix-collect-garbage (3Gb downloaded) with this patch applied.
3 years ago
renamed="$TMPDIR/${tmpFilename}"
mv "$downloadedFile" "$renamed"
unpackFile "$renamed"
''
+ (if stripRoot then ''
if [ $(ls "$unpackDir" | wc -l) != 1 ]; then
echo "error: zip file must contain a single file or directory."
echo "hint: Pass stripRoot=false; to fetchzip to assume flat list of files."
exit 1
fi
fn=$(cd "$unpackDir" && echo *)
if [ -f "$unpackDir/$fn" ]; then
mkdir $out
fi
mv "$unpackDir/$fn" "$out"
'' else ''
mv "$unpackDir" "$out"
'')
+ ''
${extraPostFetch}
''
# Remove non-owner write permissions
# Fixes https://github.com/NixOS/nixpkgs/issues/38649
+ ''
chmod 755 "$out"
'';
fetchzip: add extension parameter fetchzip downloads the file from specified URL, renames it to basename of that url, and then relies on unzip to do the unpacking. The first consequence is that this requires URL to end with proper extension—otherwise it will fail to unpack. This is not always the case and input-fonts workarounds this by adding “&.zip” query parameter (which is obviously a hack and is not guaranteed to work with every URL). The second consequence is that basename of the url must be a valid filename. I’ve tried to build a custom configuration of input-fonts and I get an error from mv that the filename is too long: > trying https://input.djr.com/build/?fontSelection=fourStyleFamily&regular=InputMonoNarrow-Regular&italic=InputMonoNarrow-Italic&bold=InputMonoNarrow-Bold&boldItalic=InputMonoNarrow-BoldItalic&a=0&g=0&i=topserif&l=serifs_round&zero=0&asterisk=height&braces=straight&preset=default&line-height=1.2&accept=I+do&email=&.zip > % Total % Received % Xferd Average Speed Time Time Time Current > Dload Upload Total Spent Left Speed > 100 406k 100 406k 0 0 230k 0 0:00:01 0:00:01 --:--:-- 230k > mv: failed to access '/build/?fontSelection=fourStyleFamily&regular=InputMonoNarrow-Regular&italic=InputMonoNarrow-Italic&bold=InputMonoNarrow-Bold&boldItalic=InputMonoNarrow-BoldItalic&a=0&g=0&i=topserif&l=serifs_round&zero=0&asterisk=height&braces=straight&preset=default&line-height=1.2&accept=I+do&email=&.zip': File name too long We could use “name” parameter as the filename (that’s how it is used in fetchurl). However, the previous attempt to do so (fc01353703426458d6990239936659ed3130d294) was reverted (24b5eb61eb944036b5716788ead1b204fba08719) because of the introduced regression—many fetchzip invocations use names without extension (also the default name is just “source”). This commit adds an optional “extension” parameter. If it is set, fetchzip renames the downloaded file to “download.${extension}” effectively solving both problems above without introducing a massive regression. This is a no-op for all existing packages. Tested by updating my NixOS setup + the extra inputs-fonts configuration mentioned above + tons of unstable emacs packages after a nix-collect-garbage (3Gb downloaded) with this patch applied.
3 years ago
} // removeAttrs args [ "stripRoot" "extraPostFetch" "extension" ])).overrideAttrs (x: {
# Hackety-hack: we actually need unzip hooks, too
nativeBuildInputs = x.nativeBuildInputs ++ [ unzip ];
})