From 39fb46f5384bd4d28f1b10580ac9227c4ed36da2 Mon Sep 17 00:00:00 2001 From: John Ericson Date: Sun, 22 Jan 2017 15:52:35 -0500 Subject: [PATCH] nixpkgs docs: Cross compilation docs --- doc/cross-compilation.xml | 153 ++++++++++++++++++++++++++++++++++++++ doc/manual.xml | 1 + 2 files changed, 154 insertions(+) create mode 100644 doc/cross-compilation.xml diff --git a/doc/cross-compilation.xml b/doc/cross-compilation.xml new file mode 100644 index 00000000000..e93d1a98f7f --- /dev/null +++ b/doc/cross-compilation.xml @@ -0,0 +1,153 @@ + + +Cross-compilation + +
+ Introduction + + "Cross-compilation" means compiling a program on one machine for another type of machine. + For example, a typical use of cross compilation is to compile programs for embedded devices. + These devices often don't have the computing power and memory to compile their own programs. + One might think that cross-compilation is a fairly niche concern, but there are advantages to being rigorous about distinguishing build-time vs run-time environments even when one is developing and deploying on the same machine. + Nixpkgs is increasingly adopting this opinion in that packages should be written with cross-compilation in mind, and nixpkgs should evaluate in a similar way (by minimizing cross-compilation-specific special cases) whether or not one is cross-compiling. + + + + This chapter will be organized in three parts. + First, it will describe the basics of how to package software in a way that supports cross-compilation. + Second, it will describe how to use Nixpkgs when cross-compiling. + Third, it will describe the internal infrastructure supporting cross-compilation. + +
+ + + +
+ Packing in a cross-friendly manner + +
+ Platform parameters + + The three GNU Autoconf platforms, build, host, and cross, are historically the result of much confusion. + clears this up somewhat but there is more to be said. + An important advice to get out the way is, unless you are packaging a compiler or other build tool, just worry about the build and host platforms. + Dealing with just two platforms usually better matches people's preconceptions, and in this case is completely correct. + + + In Nixpkgs, these three platforms are defined as attribute sets under the names buildPlatform, hostPlatform, and targetPlatform. + All are guaranteed to contain at least a platform field, which contains detailed information on the platform. + All three are always defined at the top level, so one can get at them just like a dependency in a function that is imported with callPackage: + { stdenv, buildPlatform, hostPlatform, fooDep, barDep, .. }: ... + + + These platforms should all have the same structure in all scenarios, but that is currently not the case. + When not cross-compiling, they will each contain a system field with a short 2-part, hyphen-separated summering string name for the platform. + But, when when cross compiling, hostPlatform and targetPlatform may instead contain config with a fuller 3- or 4-part string in the manner of LLVM. + We should have all 3 platforms always contain both, and maybe give config a better name while we are at it. + + + + buildPlatform + + The "build platform" is the platform on which a package is built. + Once someone has a built package, or pre-built binary package, the build platform should not matter and be safe to ignore. + + + + hostPlatform + + The "host platform" is the platform on which a package is run. + This is the simplest platform to understand, but also the one with the worst name. + + + + targetPlatform + + + The "target platform" is black sheep. + The other two intrinsically apply to all compiled software—or any build process with a notion of "build-time" followed by "run-time". + The target platform only applies to programming tools, and even then only is a good for for some of them. + Briefly, GCC, Binutils, GHC, and certain other tools are written in such a way such that a single build can only compiler code for a single platform. + Thus, when building them, one must think ahead about what platforms they wish to use the tool to produce machine code for, and build binaries for each. + + + There is no fundamental need to think about the target ahead of time like this. + LLVM, for example, was designed from the beginning with cross-compilation in mind, and so a normal LLVM binary will support every architecture that LLVM supports. + If the tool supports modular or pluggable backends, one might imagine specifying a set of target platforms / backends one wishes to support, rather than a single one. + + + The biggest reason for mess, if there is one, is that many compilers have the bad habit a build process that builds the compiler and standard library/runtime together. + Then the specifying target platform is essential, because it determines the host platform of the standard library/runtime. + Nixpkgs tries to avoid this where possible too, but still, because the concept of a target platform is so ingrained now in Autoconf and other tools, it is best to support it as is. + Tools like LLVM that don't need up-front target platforms can safely ignore it like normal packages, and it will do no harm. + + + + + + If you dig around nixpkgs, you may notice there is also stdenv.cross. + This field defined as hostPlatform when the host and build platforms differ, but otherwise not defined at all. + This field is obsolete and will soon disappear—please do not use it. + +
+ +
+ Specifying Dependencies + + As mentioned in the introduction to this chapter, one can think about a build time vs run time distinction whether cross-compiling or not. + In the case of cross-compilation, this corresponds with whether a derivation running on the native or foreign platform is produced. + An interesting thing to think about is how this corresponds with the three Autoconf platforms. + In the run-time case, the depending and depended-on package simply have matching build, host, and target platforms. + But in the build-time case, one can imagine "sliding" the platforms one over. + The depended-on package's host and target platforms (respectively) become the depending package's build and host platforms. + This is the most important guiding principle behind cross-compilation with Nixpkgs, and will be called the sliding window principle. + In this manner, given the 3 platforms for one package, we can determine the three platforms for all its transitive dependencies. + + + The depending package's target platform is unconstrained by the sliding window principle, which makes sense in that one can in principle build cross compilers targeting arbitrary platforms. + + + From the above, one would surmise that if a package is being built with a (build, host, target) platform triple of (foo, bar, bar), then its build-time dependencies would have a triple of (foo, foo, bar), and those packages' build-time dependencies would have triple of (foo, foo, foo). + In other words, it should take two "rounds" of following build-time dependency edges before one reaches a fixed point where, by the sliding window principle, the platform triple no longer changes. + Unfortunately, at the moment, we do not implement this correctly, and after only one round of following build-time dependencies is the fixed point reached, with target incorrectly kept different than the others. + + + How does this work in practice? Nixpkgs is now structured so that build-time dependencies are taken from from buildPackages, whereas run-time dependencies are taken from the top level attribute set. + For example, buildPackages.gcc should be used at build time, while gcc should be used at run time. + Now, for most of Nixpkgs's history, there was no buildPackages, and most packages have not been refactored to use it explicitly. + Instead, one can use the four attributes used for specifying dependencies as documented in . + We "splice" together the run-time and build-time package sets with callPackage, and then mkDerivation for each of four attributes pulls the right derivation out. + This splicing can be skipped when not cross compiling as the package sets are the same, but is a bit slow for cross compiling. + Because of this, a best-of-both-worlds solution is in the works with no splicing or explicit access of buildPackages needed. + For now, feel free to use either method. + +
+ +
+ + + +
+ Cross-building packages + + To be written. + This is basically unchanged so see the old wiki for now. + +
+ + + +
+ Cross-compilation infrastructure + To be written. + + If one explores nixpkgs, they will see derivations with names like gccCross. + Such *Cross derivations is a holdover from before we properly distinguished between the host and target platforms + —the derivation with "Cross" in the name covered the build = host != target case, while the other covered the host = target, with build platform the same or not based on whether one was using its .nativeDrv or .crossDrv. + This ugliness will disappear soon. + +
+ + diff --git a/doc/manual.xml b/doc/manual.xml index 1c0dac6e4df..75bd21557fd 100644 --- a/doc/manual.xml +++ b/doc/manual.xml @@ -13,6 +13,7 @@ +