build-std: context

text/0000-build-std/0-introduction.md

@@ -0,0 +1,87 @@
+- Feature Name: `build-std`
+- Start Date: 2025-06-05
+- RFC PR: [rust-lang/rfcs#0000](https://github.com/rust-lang/rfcs/pull/0000)
+- Rust Issue: [rust-lang/rust#0000](https://github.com/rust-lang/rust/issues/0000)
+
+# Summary
+[summary]: #summary
+
+While Rust's pre-built standard library has proven itself sufficient for the
+majority of use cases, there are a handful of use cases that are not well
+supported:
+
+1. Rebuilding the standard library to match the user's profile
+2. Rebuilding the standard library with ABI-modifying flags
+3. Building the standard library for tier three targets
+
+This RFC is co-authored by [David Wood][davidtwco] and
+[Adam Gemmell][adamgemmell]. To improve the readability of this RFC, it does not
+follow the standard RFC template, while still aiming to capture all of the
+salient details that the template encourages. Due to the length of this RFC, it
+is split over multiple files to avoid rendering issues and slow loading on some
+platforms.
+
+### Scope
+[scope]: #scope
+
+build-std, as proposed by this RFC, has many restrictions and limitations that
+mean it will not support most use cases that those waiting for build-std hope
+that it will. This is an explicit and deliberate choice.
+
+This RFC will focus on resolving the key questions that will enable a MVP of
+build-std to be accepted and stabilised. This will lay the foundation for future
+proposals to lift restrictions and enable build-std to support more use cases,
+without those proposals having to survey the ten+ years of issues, pull requests
+and discussion that this RFC has.
+
+As a general rule, this RFC tries to answer the question "what crates of the
+standard library get built and when do they get built" and considers anything
+else as likely out-of-scope.
+
+### Terminology
+[terminology]: #terminology
+
+The following terminology is used throughout the RFC:
+
+- "the standard library" is used to refer to all of the crates that comprise the
+  standard library - `core`, `alloc` and `std`
+- "std" is used to refer only to the `std` crate, not the entirety of the standard
+  library
+
+# Contents
+[contents]: #contents
+
+This RFC has the following contents:
+
+1. [Summary][summary] (you are here)
+
+    - Introduction to the proposal, its scope, terminology/conventions used and
+      the structure of the RFC
+
+2. [Background](./1-background.md)
+
+    - Detailed explanations of how relevant and impacted parts of the Rust
+      toolchain currently work
+
+3. [History](./2-history.md)
+
+    - Chronological summary of the various proposals and discussions that have
+      taken place relating to the ability to rebuild the standard library, and
+      of the current experimental implementation in Cargo
+
+4. [Motivation](./3-motivation.md)
+
+    - Descriptions of the varied problems that build-std has been proposed as a
+      solution to
+
+5. [Appendix II: Exhaustive literature review](./4-appendix-literature-review.md)
+
+    - More detailed summaries of the relevant issues, discussions, pull requests
+      and proposals that comprise the history of the build-std feature since
+      2015
+
+    - [*History*](./2-history.md) aims to summarise this content further and
+      cover everything that should be necessary to understand the proposal
+
+[davidtwco]: https://github.com/davidtwco
+[adamgemmell]: https://github.com/adamgemmell

text/0000-build-std/1-background.md

@@ -0,0 +1,302 @@
+# Background
+[background]: #background
+
+See [*Implementation summary*][implementation-summary] for a summary of the
+current unstable build-std feature in Cargo. This section aims to introduce any
+relevant details about the standard library and compiler that are assumed
+knowledge by referenced sources and later sections.
+
+## Standard library
+[background-standard-library]: #standard-library
+
+Since the first stable release of Rust, the standard library has been distributed
+as a pre-built artifact via rustup, which has a variety of advantages and/or
+rationale:
+
+- It saves Rust users from having to rebuild the standard library whenever they
+  start a project or do a clean build
+- The standard library has and has had dependencies which require a more complicated
+  build environment than typical Rust projects
+  - e.g. requiring a working C toolchain to build `compiler-builtins`' `c` feature
+- To varying degrees at different times in its development, the standard library's
+  implementation has been tied to the compiler implementation and has had to change
+  in lockstep
+
+Not all targets support the standard library or have a pre-built standard
+library distributed via rustup. This depends on the tier of support for the
+target. According to rustc's [platform support][platform-support] documentation,
+for tier three targets:
+
+> Tier 3 targets are those which the Rust codebase has support for, but which
+> the Rust project does not build or test automatically, so they may or may not
+> work. Official builds are not available.
+
+..and tier two targets:
+
+> The Rust project builds official binary releases of the standard library (or,
+> in some cases, only the core library) for each tier 2 target, and automated
+> builds ensure that each tier 2 target can be used as build target after each
+> change.
+
+..and finally, tier one targets:
+
+> The Rust project builds official binary releases for each tier 1 target, and
+> automated testing ensures that each tier 1 target builds and passes tests
+> after each change.
+
+All of the standard library crates leverage permanently unstable features
+provided by the compiler that will never be stabilised and therefore require
+nightly to build.
+
+The configuration for the pre-built standard library build is spread across
+bootstrap, the standard library workspace, individual standard library crate
+manifests and the target specification. The pre-built standard library is
+installed into the sysroot.
+
+At the beginning of compilation, unless the crate has the `#![no_std]`
+attribute, the compiler will load the `libstd.rlib` file from the sysroot as a
+dependency of the current crate and add an implicit `extern crate std` for it.
+This is the mechanism by which every crate has an implicit dependency on the
+standard library.
+
+The standard library sources are distributed in the `rust-src` component by
+rustup and placed in the sysroot under `lib/rustlib/src/`. The sources consist
+of the `library/` workspace plus `src/llvm-project/libunwind`, which was
+required in the past to build the `unwind` crate on some targets.
+
+Cargo supports explicitly declaring a dependency on the standard library with
+a `path` source (e.g. `core = { path = "../my_core" }`), but crates with these
+dependencies are not accepted by crates.io. There are crates on GitHub that
+use this pattern, such as [embed-rs/stm32f7-discovery][embed-rs-cargo-toml],
+which are used as `git` dependencies of other crates on GitHub.
+
+### Dependencies
+[background-dependencies]: #dependencies
+
+Behind the facade, the standard library is split into multiple crates, some of
+which are in different repositories and included as submodules or using [JOSH].
+
+As well as local crates, the standard library depends on crates from crates.io.
+It needs to be able to point these crates' dependencies on the standard library
+at the sources of `core`, `alloc` and `std` in the current [rust-lang/rust]
+checkout.
+
+This is achieved through use of the `rustc-dep-of-std` feature. Crates used in
+the dependency graph of `std` declare a `rustc-dep-of-std` feature and when
+enabled, add new dependencies on `rustc-std-workspace-{core,alloc,std}`.
+`rustc-std-workspace-{core,alloc,std}` are empty crates published on crates.io.
+As part of the workspace for the standard library,
+`rustc-std-workspace-{core,alloc,std}` are patched with a `path` source to the
+directory for the corresponding crate.
+
+Historically, there have necessarily been C dependencies of the standard library,
+increasing the complexity of the build environment required. While these have
+largely been removed over time - for example, `libbacktrace` previously depended
+on `backtrace-sys` but now uses `gimli` ([rust#46439]) - there are still some C
+dependencies:
+
+- `libunwind` will either link to the LLVM `libunwind` or the system's
+  `libunwind`/`libgcc_s`. LLVM's `libunwind` is shipped as part of the
+  rustup component for the standard library and will be linked against
+  when `-Clink-self-contained` is used
+  - This only applies to Linux and Fuchsia targets
+- `compiler_builtins` has an optional `c` feature that will use optimised
+  routines from `compiler-rt` when enabled. It is enabled for the pre-built
+  standard library
+- `compiler_builtins` has an optional `mem` feature that provides symbols
+  for common memory routines (e.g. `memcpy`)
+  - It isn't used when `std` is built as `libc` provides these routines,
+    but is often used by `no_std` crates when there is not a system `libc`
+- To use sanitizers, the sanitizer runtimes from LLVM's compiler-rt need to
+  be linked against. Building of these is enabled in `bootstrap.toml`
+  ([`build.sanitizers`][bootstrap-sanitizers]) and they are
+  included in the rustup components shipped by the project.
+
+Dependencies of the standard library may use unstable or internal compiler and
+language features only when they are a dependency of the standard library.
+
+### Features
+[background-features]: #features
+
+There are a handful of features defined in the standard library crates'
+`Cargo.toml`s. There is currently no stable existing mechanism for users to
+enable or disable these features. The default set of features is determined by
+[logic in bootstrap][bootstrap-features-logic] and [the `rust.std-features`
+key in `bootstrap.toml`][bootstrap-features-toml]. The enabled features are
+often different depending on the target.
+
+### Target support
+[background-target-support]: #target-support
+
+The `std` crate's [`build.rs`][std-build.rs] checks for supported values of the
+`CARGO_CFG_TARGET_*` environment variables. These variables are akin to the
+conditional compilation [configuration options][conditional-compilation-config-options],
+and often correspond to parts of the target triple (for example,
+`CARGO_CFG_TARGET_OS` corresponds to the "os" part of a target triple - "linux"
+in "aarch64-unknown-linux-gnu"). This filtering is strict enough to distinguish
+between built-in targets but loose enough to match similar custom targets.
+
+When encountering an unknown or unsupported operating system then the
+`restricted_std` cfg is set. `restricted_std` marks the entire standard library
+as unstable, requiring `feature(restricted_std)` to be enabled on any crate that
+depends on it. There is no mechanism for users to enable the `restricted_std`
+feature on behalf of dependencies. There is also no such mechanism for `alloc`
+or `core`, only `std`.
+
+Cargo and rustc support custom targets, defined in JSON files according to an
+unstable schema defined in the compiler. On nightly, users can dump the
+target-spec-json for an existing target using `--print target-spec-json`. This
+JSON can be saved in a file, tweaked and used as the argument to `--target` even
+on stable toolchains, though the JSON format is unstable. Custom targets do not
+have a pre-built standard library and so must use `-Zbuild-std`. Custom targets
+may have `restricted_std` set depending on their `cfg` configuration options.
+
+## Prelude
+[background-prelude]: #prelude
+
+rustc has the concept of the "extern prelude" which are the set of crates that
+have been loaded by the compiler as direct dependencies. Originally this was
+populated by users writing `extern crate $crate` in their code for each direct
+dependency. Since the 2018 edition, crates passed via `--extern` are
+automatically loaded and added to the extern prelude.
+
+`std` is automatically loaded and added to the extern prelude. For `#![no_std]`
+crates, `core` is loaded and added to the extern prelude instead. For `std` or
+`core` as appropriate, an additional `use $crate::prelude::rust_20XX::*` is
+injected for common items that Rust does not require users import (e.g.
+`Option`).
+
+`extern crate` can still be used and will search for the dependency in locations
+where direct dependencies can be found, such as `-L crate=` paths or in the
+sysroot. `-L dependency=` paths will not be searched, as these directories only
+contain indirect dependencies (i.e. dependencies of direct dependencies).
+
+Although only `std` or `core` are added to the extern prelude automatically,
+users can still write `extern crate alloc` or `extern crate test` to load them
+from the sysroot.
+
+`--extern` has a `noprelude` modifier which will allow the user to use
+`--extern` to specify the location at which a crate can be found without adding
+it to the extern prelude. This could allow a path for crates like `alloc` or
+`test` to be provided without effecting the observable behaviour of the
+language.
+
+## Panic strategies
+[background-panic-strategies]: #panic-strategies
+
+Rust has the concept of a *panic handler*, which is a crate that is responsible
+for performing a panic. There are various panic handler crates on crates.io,
+such as [panic-abort] (which different from the `panic_abort` panic runtime!),
+[panic-halt], [panic-itm], and [panic-semihosting]. Panic handler crates define
+a function annotated with `#[panic_handler]`. There can only be one
+`#[panic_handler]` in the crate graph.
+
+`core` uses the panic handler to implement panics inserted by code generation
+(e.g. arithmetic overflow or out-of-bounds access) and the `core::panic!` macro
+immediately delegates to the panic handler crate.
+
+`std` is also a panic handler. `std`'s panic handler and `std::panic!` macro
+print panic information to stderr and delegate to a *panic runtime* to decide
+what to do next, determined by the *panic strategy*.
+
+There are two panic runtime crates in the standard library - `panic_unwind` and
+`panic_abort` - each with a corresponding panic strategy. Each target supported
+by rustc specifies a default panic strategy - either "unwind" or "abort" -
+though these are only relevant if `std`'s panic handler is used (i.e. the target
+isn't a `no_std` target or being used with a `no_std` crate).
+
+Rust's `-Cpanic` flag allows the user to choose the panic strategy, with the
+target's default as a fallback. If `-Cpanic=unwind` is provided then this
+doesn't guarantee that the unwind strategy is used, as the target may not
+support it.
+
+Both crates are compiled and shipped with the pre-built standard library for
+targets which support `std`. Some targets have a pre-built standard library with
+only the `core` and `alloc` crates, such as the `x86_64-unknown-none` target.
+While `x86_64-unknown-none` defaults to the `abort` panic strategy, as this
+target does not support the standard library, this default isn't actually
+relevant.
+
+The `std` crate has a `panic_unwind` feature that enables an optional dependency
+on the `panic_unwind` crate.
+
+`core` also has a `panic_immediate_abort` feature which modifies the
+`core::panic!` macro to immediately call the abort intrinsic without calling the
+panic handler. `std` and `alloc` have the same feature which enable the feature
+in `core`. `std`'s feature also adds an immediate abort to its `panic!` macro.
+
+## Cargo
+[background-cargo]: #cargo
+
+Cargo's building of the dependency graph is driven by the registry index.
+[Cargo registries][cargo-docs-registry], like crates.io, are centralised sources
+for crates. A registry's index is the interface between Cargo and the registry
+that Cargo queries to know which crates are available, what their dependencies
+are, etc. crates.io's registry index is a Git repository -
+[rust-lang/crates.io-index] - which is updated automatically by crates.io when
+crates are published, yanked, etc. Cargo can query registries using a Git
+protocol which caches the registry on disk, or using a sparse protocol which
+exposes the index over HTTP and allows Cargo to avoid Cargo having a local copy
+of the whole index, which has become quite large for crates.io.
+
+Each crates in the registry has a JSON file, following
+[a defined schema][cargo-json-schema]. Crates may refer to those in other
+registries, but all crates in the dependency graph must exist in a registry. As
+the registry index drives the building of Cargo's dependency graph, all crates
+that end up in the dependency graph must be present a registry.
+
+Registries can have different policies for what crates are accepted. For
+example, crates.io does not permit publishing packages named `std` or `core` but
+other registries might.
+
+### Public/private dependencies
+[background-pubpriv-dependencies]: #publicprivate-dependencies
+
+[Public and private dependencies][rust#44663] are an unstable feature which
+enables declaring which dependencies form part of a library's public interface,
+so as to make it easier to avoid breaking semver compatibility.
+
+With the `public-dependency` feature enabled, dependencies are marked as
+"private" by default which can be overridden with a `public = true` declaration.
+
+Private dependencies are passed to rustc with an `priv` modifier to the
+`--extern` flag. Dependencies without this modifier are treated as public by
+rustc for backwards compatibility reasons. rust emits the
+`exported-private-dependencies` lint if an item from a private dependency is
+re-exported.
+
+## Target modifiers
+[background-target-modifiers]: #target-modifiers
+
+[rfcs#3716] introduced the concept of *target modifiers* to rustc. Flags marked
+as target modifiers must match across the entire crate graph or the compilation
+will fail.
+
+For example, flags are made target modifiers when they change the ABI of
+generated code and could result in unsound ABI mismatches if two crates are
+linked together with different values of the flag set.
+
+[implementation-summary]: ./2-history.md#implementation-summary
+
+[JOSH]: https://josh-project.github.io/josh/intro.html
+[panic-abort]: https://crates.io/crates/panic-abort
+[panic-halt]: https://crates.io/crates/panic-halt
+[panic-itm]: https://crates.io/crates/panic-itm
+[panic-semihosting]: https://crates.io/crates/panic-semihosting
+[rust-lang/crates.io-index]: https://github.com/rust-lang/crates.io-index
+[rust-lang/rust]: https://github.com/rust-lang/rust
+
+[rfcs#3716]: https://rust-lang.github.io/rfcs/3716-target-modifiers.html
+[rust#46439]: https://github.com/rust-lang/rust/pull/46439
+[rust#44663]: https://github.com/rust-lang/rust/issues/44663
+
+[bootstrap-features-logic]: https://github.com/rust-lang/rust/blob/00b526212bbdd68872d6f964fcc9a14a66c36fd8/src/bootstrap/src/lib.rs#L732
+[bootstrap-features-toml]: https://github.com/rust-lang/rust/blob/00b526212bbdd68872d6f964fcc9a14a66c36fd8/bootstrap.example.toml#L816
+[bootstrap-sanitizers]: https://github.com/rust-lang/rust/blob/d13a431a6cc69cd65efe7c3eb7808251d6fd7a46/bootstrap.example.toml#L388
+[cargo-docs-registry]: https://doc.rust-lang.org/nightly/nightly-rustc/cargo/sources/registry/index.html
+[cargo-json-schema]: https://doc.rust-lang.org/cargo/reference/registry-index.html#json-schema
+[conditional-compilation-config-options]: https://doc.rust-lang.org/reference/conditional-compilation.html#set-configuration-options
+[embed-rs-cargo-toml]: https://github.com/embed-rs/stm32f7-discovery/blob/e2bf713263791c028c2a897f2eb1830d7f09eceb/Cargo.toml#L21
+[target-tier-policy]: https://doc.rust-lang.org/nightly/rustc/target-tier-policy.html
+[std-build.rs]: https://github.com/rust-lang/rust/blob/f315e6145802e091ff9fceab6db627a4b4ec2b86/library/std/build.rs#L17
+[platform-support]: https://doc.rust-lang.org/nightly/rustc/platform-support.html