Update cargo-vendor usage

Author: ehuss

CONTRIBUTING.md

@@ -179,7 +179,6 @@ Speaking of tests, Rust has a comprehensive test suite. More information about
 it can be found [here][rctd].
 
 ### External Dependencies
-[external-dependencies]: #external-dependencies
 
 Currently building Rust will also build the following external projects:
 
@@ -209,7 +208,6 @@ Breakage is not allowed in the beta and stable channels, and must be addressed
 before the PR is merged.
 
 #### Breaking Tools Built With The Compiler
-[breaking-tools-built-with-the-compiler]: #breaking-tools-built-with-the-compiler
 
 Rust's build system builds a number of tools that make use of the
 internals of the compiler. This includes
@@ -242,7 +240,7 @@ Here are those same steps in detail:
    `config.toml.example` in the root directory of the Rust repository.
    Set `submodules = false` in the `[build]` section. This will prevent `x.py`
    from resetting to the original branch after you make your changes. If you
-   need to [update any submodules to their latest versions][updating-submodules],
+   need to [update any submodules to their latest versions](#updating-submodules),
    see the section of this file about that for more information.
 2. (optional) Run `./x.py test src/tools/rustfmt` (substituting the submodule
    that broke for `rustfmt`). Fix any errors in the submodule (and possibly others).
@@ -256,7 +254,6 @@ Here are those same steps in detail:
 8. (optional) Send a PR to rust-lang/rust updating the submodule.
 
 #### Updating submodules
-[updating-submodules]: #updating-submodules
 
 These instructions are specific to updating `rustfmt`, however they may apply
 to the other submodules as well. Please help by improving these instructions
@@ -310,7 +307,6 @@ This should change the version listed in `Cargo.lock` to the new version you upd
 the submodule to. Running `./x.py build` should work now.
 
 ## Writing Documentation
-[writing-documentation]: #writing-documentation
 
 Documentation improvements are very welcome. The source of `doc.rust-lang.org`
 is located in `src/doc` in the tree, and standard API documentation is generated
@@ -337,7 +333,6 @@ tracker in that repo is also a great way to find things that need doing. There
 are issues for beginners and advanced compiler devs alike!
 
 ## Issue Triage
-[issue-triage]: #issue-triage
 
 Sometimes, an issue will stay open, even though the bug has been fixed. And
 sometimes, the original bug may go stale because something has changed in the
@@ -405,7 +400,6 @@ If you're looking for somewhere to start, check out the [E-easy][eeasy] tag.
 [rfcbot]: https://github.com/anp/rfcbot-rs/
 
 ## Out-of-tree Contributions
-[out-of-tree-contributions]: #out-of-tree-contributions
 
 There are a number of other ways to contribute to Rust that don't deal with
 this repository.
@@ -425,7 +419,6 @@ valuable!
 [community-library]: https://github.com/rust-lang/rfcs/labels/A-community-library
 
 ## Helpful Links and Information
-[helpful-info]: #helpful-info
 
 For people new to Rust, and just starting to contribute, or even for
 more seasoned developers, some useful places to look for information

README.md

@@ -6,19 +6,22 @@ standard library, and documentation.
 [Rust]: https://www.rust-lang.org
 
 ## Quick Start
-[quick-start]: #quick-start
 
 Read ["Installation"] from [The Book].
 
 ["Installation"]: https://doc.rust-lang.org/book/ch01-01-installation.html
 [The Book]: https://doc.rust-lang.org/book/index.html
 
 ## Installing from Source
-[building-from-source]: #building-from-source
 
-_Note: If you wish to contribute to the compiler, you should read
-[this chapter](https://rust-lang.github.io/rustc-guide/how-to-build-and-run.html)
-of the rustc-guide instead of this section._
+_Note: If you wish to contribute to the compiler, you should read [this
+chapter][rustcguidebuild] of the rustc-guide instead of this section._
+
+The Rust build system has a Python script called `x.py` to bootstrap building
+the compiler. More information about it may be found by running `./x.py --help`
+or reading the [rustc guide][rustcguidebuild].
+
+[rustcguidebuild]: https://rust-lang.github.io/rustc-guide/how-to-build-and-run.html
 
 ### Building on *nix
 1. Make sure you have installed the dependencies:
@@ -39,43 +42,36 @@ of the rustc-guide instead of this section._
 
 [source]: https://github.com/rust-lang/rust
 
-3. Build and install:
-
-    ```sh
-    $ ./x.py build && sudo ./x.py install
-    ```
+3. Configure the build settings:
 
-    If after running `sudo ./x.py install` you see an error message like
+    The Rust build system uses a file named `config.toml` in the root of the
+    source tree to determine various configuration settings for the build.
+    Copy the default `config.toml.example` to `config.toml` to get started.
 
-    ```
-    error: failed to load source for a dependency on 'cc'
+    ```sh
+    $ cp config.toml.example config.toml
     ```
 
-    then run these two commands and then try `sudo ./x.py install` again:
+    It is recommended that if you plan to use the Rust build system to create
+    an installation (using `./x.py install`) that you set the `prefix` value
+    in the `[install]` section to a directory that you have write permissions.
 
-    ```
-    $ cargo install cargo-vendor
-    ```
+4. Build and install:
 
+    ```sh
+    $ ./x.py build && ./x.py install
     ```
-    $ cargo vendor
-    ```
-
-    > ***Note:*** Install locations can be adjusted by copying the config file
-    > from `./config.toml.example` to `./config.toml`, and
-    > adjusting the `prefix` option under `[install]`. Various other options, such
-    > as enabling debug information, are also supported, and are documented in
-    > the config file.
 
-    When complete, `sudo ./x.py install` will place several programs into
-    `/usr/local/bin`: `rustc`, the Rust compiler, and `rustdoc`, the
+    When complete, `./x.py install` will place several programs into
+    `$PREFIX/bin`: `rustc`, the Rust compiler, and `rustdoc`, the
     API-documentation tool. This install does not include [Cargo],
-    Rust's package manager, which you may also want to build.
+    Rust's package manager. To build and install Cargo, you may
+    run `./x.py install cargo` or set the `build.extended` key in
+    `config.toml` to `true` to build and install all tools.
 
 [Cargo]: https://github.com/rust-lang/cargo
 
 ### Building on Windows
-[building-on-windows]: #building-on-windows
 
 There are two prominent ABIs in use on Windows: the native (MSVC) ABI used by
 Visual Studio, and the GNU ABI used by the GCC toolchain. Which version of Rust
@@ -85,7 +81,6 @@ for interop with GNU software built using the MinGW/MSYS2 toolchain use the GNU
 build.
 
 #### MinGW
-[windows-mingw]: #windows-mingw
 
 [MSYS2][msys2] can be used to easily build Rust on Windows:
 
@@ -126,7 +121,6 @@ build.
    ```
 
 #### MSVC
-[windows-msvc]: #windows-msvc
 
 MSVC builds of Rust additionally require an installation of Visual Studio 2017
 (or later) so `rustc` can use its linker.  The simplest way is to get the
@@ -155,7 +149,6 @@ by manually calling the appropriate vcvars file before running the bootstrap.
 ```
 
 #### Specifying an ABI
-[specifying-an-abi]: #specifying-an-abi
 
 Each specific ABI can also be used from either environment (for example, using
 the GNU ABI in PowerShell) by using an explicit build triple. The available
@@ -169,11 +162,10 @@ Windows build triples are:
 
 The build triple can be specified by either specifying `--build=<triple>` when
 invoking `x.py` commands, or by copying the `config.toml` file (as described
-in Building From Source), and modifying the `build` option under the `[build]`
-section.
+in [Installing From Source](#installing-from-source)), and modifying the
+`build` option under the `[build]` section.
 
 ### Configure and Make
-[configure-and-make]: #configure-and-make
 
 While it's not the recommended build system, this project also provides a
 configure script and makefile (the latter of which just invokes `x.py`).
@@ -188,7 +180,6 @@ When using the configure script, the generated `config.mk` file may override the
 `config.mk` file.
 
 ## Building Documentation
-[building-documentation]: #building-documentation
 
 If you’d like to build the documentation, it’s almost the same:
 
@@ -201,7 +192,6 @@ the ABI used. I.e., if the ABI was `x86_64-pc-windows-msvc`, the directory will
 `build\x86_64-pc-windows-msvc\doc`.
 
 ## Notes
-[notes]: #notes
 
 Since the Rust compiler is written in Rust, it must be built by a
 precompiled "snapshot" version of itself (made in an earlier stage of
@@ -224,7 +214,6 @@ There is more advice about hacking on Rust in [CONTRIBUTING.md].
 [CONTRIBUTING.md]: https://github.com/rust-lang/rust/blob/master/CONTRIBUTING.md
 
 ## Getting Help
-[getting-help]: #getting-help
 
 The Rust community congregates in a few places:
 
@@ -237,7 +226,6 @@ The Rust community congregates in a few places:
 [users.rust-lang.org]: https://users.rust-lang.org/
 
 ## Contributing
-[contributing]: #contributing
 
 To contribute to Rust, please see [CONTRIBUTING](CONTRIBUTING.md).
 
@@ -258,7 +246,6 @@ Also, you may find the [rustdocs for the compiler itself][rustdocs] useful.
 [rustdocs]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc/
 
 ## License
-[license]: #license
 
 Rust is primarily distributed under the terms of both the MIT license
 and the Apache License (Version 2.0), with portions covered by various
@@ -268,7 +255,6 @@ See [LICENSE-APACHE](LICENSE-APACHE), [LICENSE-MIT](LICENSE-MIT), and
 [COPYRIGHT](COPYRIGHT) for details.
 
 ## Trademark
-[trademark]: #trademark
 
 The Rust programming language is an open source, community project governed
 by a core team. It is also sponsored by the Mozilla Foundation (“Mozilla”),

src/bootstrap/bootstrap.py

@@ -735,6 +735,47 @@ def set_dev_environment(self):
         """Set download URL for development environment"""
         self._download_url = 'https://dev-static.rust-lang.org'
 
+    def check_vendored_status(self):
+        """Check that vendoring is configured properly"""
+        vendor_dir = os.path.join(self.rust_root, 'vendor')
+        if 'SUDO_USER' in os.environ and not self.use_vendored_sources:
+            if os.environ.get('USER') != os.environ['SUDO_USER']:
+                self.use_vendored_sources = True
+                print('info: looks like you are running this command under `sudo`')
+                print('      and so in order to preserve your $HOME this will now')
+                print('      use vendored sources by default.')
+                if not os.path.exists(vendor_dir):
+                    print('error: vendoring required, but vendor directory does not exist.')
+                    print('       Run `cargo vendor` without sudo to initialize the '
+                        'vendor directory.')
+                    raise Exception("{} not found".format(vendor_dir))
+
+        if self.use_vendored_sources:
+            if not os.path.exists('.cargo'):
+                os.makedirs('.cargo')
+            with output('.cargo/config') as cargo_config:
+                cargo_config.write(
+                    "[source.crates-io]\n"
+                    "replace-with = 'vendored-sources'\n"
+                    "registry = 'https://example.com'\n"
+                    "\n"
+                    "[source.vendored-sources]\n"
+                    "directory = '{}/vendor'\n"
+                .format(self.rust_root))
+        else:
+            if os.path.exists('.cargo'):
+                shutil.rmtree('.cargo')
+
+    def ensure_vendored(self):
+        """Ensure that the vendored sources are available if needed"""
+        vendor_dir = os.path.join(self.rust_root, 'vendor')
+        # Note that this does not handle updating the vendored dependencies if
+        # the rust git repository is updated. Normal development usually does
+        # not use vendoring, so hopefully this isn't too much of a problem.
+        if self.use_vendored_sources and not os.path.exists(vendor_dir):
+            run([self.cargo(), "vendor"],
+                verbose=self.verbose, cwd=self.rust_root)
+
 
 def bootstrap(help_triggered):
     """Configure, fetch, build and run the initial bootstrap"""
@@ -776,30 +817,7 @@ def bootstrap(help_triggered):
 
     build.use_locked_deps = '\nlocked-deps = true' in build.config_toml
 
-    if 'SUDO_USER' in os.environ and not build.use_vendored_sources:
-        if os.environ.get('USER') != os.environ['SUDO_USER']:
-            build.use_vendored_sources = True
-            print('info: looks like you are running this command under `sudo`')
-            print('      and so in order to preserve your $HOME this will now')
-            print('      use vendored sources by default. Note that if this')
-            print('      does not work you should run a normal build first')
-            print('      before running a command like `sudo ./x.py install`')
-
-    if build.use_vendored_sources:
-        if not os.path.exists('.cargo'):
-            os.makedirs('.cargo')
-        with output('.cargo/config') as cargo_config:
-            cargo_config.write("""
-                [source.crates-io]
-                replace-with = 'vendored-sources'
-                registry = 'https://example.com'
-
-                [source.vendored-sources]
-                directory = '{}/vendor'
-            """.format(build.rust_root))
-    else:
-        if os.path.exists('.cargo'):
-            shutil.rmtree('.cargo')
+    build.check_vendored_status()
 
     data = stage0_data(build.rust_root)
     build.date = data['date']
@@ -815,6 +833,7 @@ def bootstrap(help_triggered):
     build.build = args.build or build.build_triple()
     build.download_stage0()
     sys.stdout.flush()
+    build.ensure_vendored()
     build.build_bootstrap()
     sys.stdout.flush()
 

src/bootstrap/dist.rs

@@ -935,8 +935,6 @@ impl Step for Src {
     }
 }
 
-const CARGO_VENDOR_VERSION: &str = "0.1.22";
-
 #[derive(Debug, PartialOrd, Ord, Copy, Clone, Hash, PartialEq, Eq)]
 pub struct PlainSourceTarball;
 
@@ -998,26 +996,6 @@ impl Step for PlainSourceTarball {
 
         // If we're building from git sources, we need to vendor a complete distribution.
         if builder.rust_info.is_git() {
-            // Get cargo-vendor installed, if it isn't already.
-            let mut has_cargo_vendor = false;
-            let mut cmd = Command::new(&builder.initial_cargo);
-            for line in output(cmd.arg("install").arg("--list")).lines() {
-                has_cargo_vendor |= line.starts_with("cargo-vendor ");
-            }
-            if !has_cargo_vendor {
-                let mut cmd = builder.cargo(
-                    builder.compiler(0, builder.config.build),
-                    Mode::ToolBootstrap,
-                    builder.config.build,
-                    "install"
-                );
-                cmd.arg("--force")
-                   .arg("--debug")
-                   .arg("--vers").arg(CARGO_VENDOR_VERSION)
-                   .arg("cargo-vendor");
-                builder.run(&mut cmd);
-            }
-
             // Vendor all Cargo dependencies
             let mut cmd = Command::new(&builder.initial_cargo);
             cmd.arg("vendor")