@@ -4,105 +4,31 @@ This is an in-progress README which is targeted at helping to explain how Rust
44is bootstrapped and in general, some of the technical details of the build
55system.
66
7- ## Using rustbuild
7+ Note that this README only covers internal information, not how to use the tool.
8+ Please check [ bootstrapping dev guide] [ bootstrapping-dev-guide ] for further information.
89
9- The rustbuild build system has a primary entry point, a top level ` x.py ` script:
10+ [ bootstrapping-dev-guide ] : https://rustc-dev-guide.rust-lang.org/building/bootstrapping.html
1011
11- ``` sh
12- $ python ./x.py build
13- ```
14-
15- Note that if you're on Unix, you should be able to execute the script directly:
16-
17- ``` sh
18- $ ./x.py build
19- ```
20-
21- The script accepts commands, flags, and arguments to determine what to do:
22-
23- * ` build ` - a general purpose command for compiling code. Alone, ` build ` will
24- bootstrap the entire compiler, and otherwise, arguments passed indicate what to
25- build. For example:
26-
27- ```
28- # build the whole compiler
29- ./x.py build --stage 2
30-
31- # build the stage1 compiler
32- ./x.py build
33-
34- # build stage0 libstd
35- ./x.py build --stage 0 library/std
36-
37- # build a particular crate in stage0
38- ./x.py build --stage 0 library/test
39- ```
40-
41- If files that would normally be rebuilt from stage 0 are dirty, the rebuild can be
42- overridden using ` --keep-stage 0 ` . Using ` --keep-stage n ` will skip all steps
43- that belong to stage n or earlier:
44-
45- ```
46- # build stage 1, keeping old build products for stage 0
47- ./x.py build --keep-stage 0
48- ```
49-
50- * ` test ` - a command for executing unit tests. Like the ` build ` command, this
51- will execute the entire test suite by default, and otherwise, it can be used to
52- select which test suite is run:
53-
54- ```
55- # run all unit tests
56- ./x.py test
57-
58- # execute tool tests
59- ./x.py test tidy
60-
61- # execute the UI test suite
62- ./x.py test tests/ui
63-
64- # execute only some tests in the UI test suite
65- ./x.py test tests/ui --test-args substring-of-test-name
66-
67- # execute tests in the standard library in stage0
68- ./x.py test --stage 0 library/std
69-
70- # execute tests in the core and standard library in stage0,
71- # without running doc tests (thus avoid depending on building the compiler)
72- ./x.py test --stage 0 --no-doc library/core library/std
12+ ## Introduction
7313
74- # execute all doc tests
75- ./x.py test src/doc
76- ```
14+ The build system defers most of the complicated logic managing invocations
15+ of rustc and rustdoc to Cargo itself. However, moving through various stages
16+ and copying artifacts is still necessary for it to do. Each time rustbuild
17+ is invoked, it will iterate through the list of predefined steps and execute
18+ each serially in turn if it matches the paths passed or is a default rule.
19+ For each step rustbuild relies on the step internally being incremental and
20+ parallel. Note, though, that the ` -j ` parameter to rustbuild gets forwarded
21+ to appropriate test harnesses and such.
7722
78- * ` doc ` - a command for building documentation. Like above, can take arguments
79- for what to document.
80-
81- ## Configuring rustbuild
82-
83- rustbuild offers a TOML-based configuration system with a ` config.toml `
84- file. An example of this configuration can be found at ` config.toml.example ` ,
85- and the configuration file can also be passed as ` --config path/to/config.toml `
86- if the build system is being invoked manually (via the python script).
87-
88- You can generate a config.toml using ` ./configure ` options if you want to automate creating the file without having to edit it.
89-
90- Finally, rustbuild makes use of the [ cc-rs crate] which has [ its own
91- method] [ env-vars ] of configuring C compilers and C flags via environment
92- variables.
93-
94- [ cc-rs crate ] : https://github.com/alexcrichton/cc-rs
95- [ env-vars ] : https://github.com/alexcrichton/cc-rs#external-configuration-via-environment-variables
96-
97- ## Build stages
23+ ## Build phases
9824
9925The rustbuild build system goes through a few phases to actually build the
10026compiler. What actually happens when you invoke rustbuild is:
10127
102- 1 . The entry point script, ` x.py ` is run. This script is
103- responsible for downloading the stage0 compiler/Cargo binaries, and it then
104- compiles the build system itself (this folder). Finally, it then invokes the
105- actual ` bootstrap ` binary build system.
28+ 1 . The entry point script( ` x ` for unix like systems , ` x.ps1 ` for windows systems,
29+ ` x.py ` cross-platform) is run. This script is responsible for downloading the stage0
30+ compiler/Cargo binaries, and it then compiles the build system itself (this folder).
31+ Finally, it then invokes the actual ` bootstrap ` binary build system.
106322 . In Rust, ` bootstrap ` will slurp up all configuration, perform a number of
10733 sanity checks (whether compilers exist, for example), and then start building the
10834 stage0 artifacts.
@@ -115,24 +41,6 @@ compiler. What actually happens when you invoke rustbuild is:
11541The goal of each stage is to (a) leverage Cargo as much as possible and failing
11642that (b) leverage Rust as much as possible!
11743
118- ## Incremental builds
119-
120- You can configure rustbuild to use incremental compilation with the
121- ` --incremental ` flag:
122-
123- ``` sh
124- $ ./x.py build --incremental
125- ```
126-
127- The ` --incremental ` flag will store incremental compilation artifacts
128- in ` build/<host>/stage0-incremental ` . Note that we only use incremental
129- compilation for the stage0 -> stage1 compilation -- this is because
130- the stage1 compiler is changing, and we don't try to cache and reuse
131- incremental artifacts across different versions of the compiler.
132-
133- You can always drop the ` --incremental ` to build as normal (but you
134- will still be using the local nightly as your bootstrap).
135-
13644## Directory Layout
13745
13846This build system houses all output under the ` build ` directory, which looks
@@ -236,63 +144,31 @@ build/
236144 # system will link (using hard links) output from stageN-{std,rustc} into
237145 # each of these directories.
238146 #
239- # In theory, there is no extra build output in these directories.
147+ # In theory these are working rustc sysroot directories, meaning there is
148+ # no extra build output in these directories.
240149 stage1/
241150 stage2/
242151 stage3/
243152```
244153
245- ## Cargo projects
246-
247- The current build is unfortunately not quite as simple as ` cargo build ` in a
248- directory, but rather the compiler is split into three different Cargo projects:
249-
250- * ` library/std ` - the standard library
251- * ` library/test ` - testing support, depends on libstd
252- * ` compiler/rustc ` - the actual compiler itself
253-
254- Each "project" has a corresponding Cargo.lock file with all dependencies, and
255- this means that building the compiler involves running Cargo three times. The
256- structure here serves two goals:
257-
258- 1 . Facilitating dependencies coming from crates.io. These dependencies don't
259- depend on ` std ` , so libstd is a separate project compiled ahead of time
260- before the actual compiler builds.
261- 2 . Splitting "host artifacts" from "target artifacts". That is, when building
262- code for an arbitrary target, you don't need the entire compiler, but you'll
263- end up needing libraries like libtest that depend on std but also want to use
264- crates.io dependencies. Hence, libtest is split out as its own project that
265- is sequenced after ` std ` but before ` rustc ` . This project is built for all
266- targets.
267-
268- There is some loss in build parallelism here because libtest can be compiled in
269- parallel with a number of rustc artifacts, but in theory, the loss isn't too bad!
270-
271- ## Build tools
272-
273- We've actually got quite a few tools that we use in the compiler's build system
274- and for testing. To organize these, each tool is a project in ` src/tools ` with a
275- corresponding ` Cargo.toml ` . All tools are compiled with Cargo (currently having
276- independent ` Cargo.lock ` files) and do not currently explicitly depend on the
277- compiler or standard library. Compiling each tool is sequenced after the
278- appropriate libstd/libtest/librustc compile above.
279-
280154## Extending rustbuild
281155
282- So, you'd like to add a feature to the rustbuild build system or just fix a bug.
283- Great! One of the major motivational factors for moving away from ` make ` is that
284- Rust is in theory much easier to read, modify, and write. If you find anything
285- excessively confusing, please open an issue on this, and we'll try to get it
286- documented or simplified, pronto.
156+ When you use the bootstrap system, you'll call it through the entry point script
157+ (` x ` , ` x.ps1 ` , or ` x.py ` ). However, most of the code lives in ` src/bootstrap ` .
158+ ` bootstrap ` has a difficult problem: it is written in Rust, but yet it is run
159+ before the Rust compiler is built! To work around this, there are two components
160+ of bootstrap: the main one written in rust, and ` bootstrap.py ` . ` bootstrap.py `
161+ is what gets run by entry point script. It takes care of downloading the ` stage0 `
162+ compiler, which will then build the bootstrap binary written in Rust.
287163
288- First up, you'll probably want to read over the documentation above, as that'll
289- give you a high level overview of what rustbuild is doing. You also probably
290- want to play around a bit yourself by just getting it up and running before you
291- dive too much into the actual build system itself.
164+ Because there are two separate codebases behind ` x.py ` , they need to
165+ be kept in sync. In particular, both ` bootstrap.py ` and the bootstrap binary
166+ parse ` config.toml ` and read the same command line arguments. ` bootstrap.py `
167+ keeps these in sync by setting various environment variables, and the
168+ programs sometimes have to add arguments that are explicitly ignored, to be
169+ read by the other.
292170
293- After that, each module in rustbuild should have enough documentation to keep
294- you up and running. Some general areas that you may be interested in modifying
295- are:
171+ Some general areas that you may be interested in modifying are:
296172
297173* Adding a new build tool? Take a look at ` bootstrap/tool.rs ` for examples of
298174 other tools.
@@ -320,8 +196,9 @@ A 'major change' includes
320196Changes that do not affect contributors to the compiler or users
321197building rustc from source don't need an update to ` VERSION ` .
322198
323- If you have any questions, feel free to reach out on the ` #t-infra ` channel in
324- the [ Rust Zulip server] [ rust-zulip ] or ask on internals.rust-lang.org. When
325- you encounter bugs, please file issues on the rust-lang/rust issue tracker.
199+ If you have any questions, feel free to reach out on the ` #t-infra/bootstrap ` channel
200+ at [ Rust Bootstrap Zulip server] [ rust-bootstrap- zulip ] . When you encounter bugs,
201+ please file issues on the [ Rust issue tracker] [ rust-issue-tracker ] .
326202
327- [ rust-zulip ] : https://rust-lang.zulipchat.com/#narrow/stream/242791-t-infra
203+ [ rust-bootstrap-zulip ] : https://rust-lang.zulipchat.com/#narrow/stream/t-infra.2Fbootstrap
204+ [ rust-issue-tracker ] : https://github.com/rust-lang/rust/issues
0 commit comments