Add BOM / dependency management support (com-lihaoyi#3924)

This PR adds support for user-specified BOMs and dependency management
in Mill.

BOM support allows users to pass the coordinates of an existing Maven
"Bill of Material" (BOM), such as [this
one](https://repo1.maven.org/maven2/io/quarkus/quarkus-bom/3.17.0/quarkus-bom-3.17.0.pom),
that contains versions of dependencies, meant to override those pulled
during dependency resolution. (They can also add exclusions to
dependencies.)
```scala
def bomDeps = Agg(
  ivy"io.quarkus:quarkus-bom:3.17.0"
)
```

It also allows users to specify the coordinates of a parent POM, which
are taken into account just like a BOM:
```scala
def parentDep = ivy"org.apache.spark::spark-parent:3.5.3"
```
(in line with `PublishModule#pomParentProject` that's been added
recently)

It allows users to specify "dependency management", which act like the
dependencies listed in a BOM: versions in dependency management override
those pulled transitively during dependency resolution, and exclusions
in its dependencies are added to the same dependencies during dependency
resolution.
```scala
def dependencyManagement = Agg(
  ivy"com.google.protobuf:protobuf-java:4.28.3",
  ivy"org.java-websocket:Java-WebSocket:_" // placeholder version - this one only adds exclusions, no version override
        .exclude(("org.slf4j", "slf4j-api"))
)
```

BOM and dependency management also allow for "placeholder" versions:
users can use `_` as version in their `ivyDeps`, and the version of that
dependency will be picked either in dependency management or in BOMs:
```scala
def bomDeps = Agg(
  ivy"com.google.cloud:libraries-bom:26.50.0"
)
def ivyDeps = Agg(
  ivy"com.google.protobuf:protobuf-java:_"
)
```

A tricky aspect of that PR is that details about BOMs and dependency
management have to be passed around via several paths:
- in the current module: BOMs and dependency management have to be taken
into account during dependency resolution of the module they're added to
- via `moduleDeps`: BOMs and dependency management of module
dependencies have to be applied to the dependencies of the module they
come from
- ~to transitive modules pulled via `moduleDeps`: BOMs and dependency
management of a module dependency have to be applied to the dependencies
of modules they pull transitively (if A depends on B and B depends on C,
from A, the BOMs and dep mgmt of B apply to C's dependencies too)~
(worked out-of-the-box with the previous point, via `transitiveIvyDeps`)
- via `ivy.xml`: when publishing to Ivy repositories (like during
`pubishLocal`), BOMs and dep mgmt details need to be written in the
`ivy.xml` file, so that they're taken into account when resolving that
module from the Ivy repo
- via POM files: when publishing to Maven repositories, BOMs and dep
mgmt details need to be written to POMs, so that they're taken into
account when resolving that module from the Maven repo


Fixes com-lihaoyi#1975

Author: alexarchambault

docs/modules/ROOT/pages/fundamentals/library-deps.adoc

@@ -96,6 +96,34 @@ def runIvyDeps = Agg(
 
 It is also possible to use a higher version of the same library dependencies already defined in `ivyDeps`, to ensure you compile against a minimal API version, but actually run with the latest available version.
 
+== Dependency management
+
+Dependency management consists in listing dependencies whose versions we want to force. Having
+a dependency in dependency management doesn't mean that this dependency will be fetched, only
+that
+
+* if it ends up being fetched transitively, its version will be forced to the one in dependency management
+
+* if its version is empty in an `ivyDeps` section in Mill, the version from dependency management will be used
+
+Dependency management also allows to add exclusions to dependencies, both explicit dependencies and
+transitive ones.
+
+Dependency management can be passed to Mill in two ways:
+
+* via external Maven BOMs, like https://repo1.maven.org/maven2/com/google/cloud/libraries-bom/26.50.0/libraries-bom-26.50.0.pom[this one],
+whose Maven coordinates are `com.google.cloud:libraries-bom:26.50.0`
+
+* via the `depManagement` task, that allows to directly list dependencies whose versions we want to enforce
+
+=== External BOMs
+
+include::partial$example/fundamentals/library-deps/bom-1-external-bom.adoc[]
+
+=== Dependency management task
+
+include::partial$example/fundamentals/library-deps/bom-2-dependency-management.adoc[]
+
 == Searching For Dependency Updates
 
 include::partial$example/fundamentals/dependencies/1-search-updates.adoc[]

docs/modules/ROOT/pages/javalib/dependencies.adoc

@@ -13,6 +13,11 @@ include::partial$example/javalib/dependencies/1-ivy-deps.adoc[]
 
 include::partial$example/javalib/dependencies/2-run-compile-deps.adoc[]
 
+== Dependency Management
+
+Mill has support for dependency management, see the
+xref:fundamentals/library-deps.adoc#_dependency_management[Dependency Management section]
+in xref:fundamentals/library-deps.adoc[].
 
 == Unmanaged Jars
 

docs/modules/ROOT/pages/kotlinlib/dependencies.adoc

@@ -19,6 +19,12 @@ include::partial$example/kotlinlib/dependencies/1-ivy-deps.adoc[]
 
 include::partial$example/kotlinlib/dependencies/2-run-compile-deps.adoc[]
 
+== Dependency Management
+
+Mill has support for dependency management, see the
+xref:fundamentals/library-deps.adoc#_dependency_management[Dependency Management section]
+in xref:fundamentals/library-deps.adoc[].
+
 == Unmanaged Jars
 
 include::partial$example/kotlinlib/dependencies/3-unmanaged-jars.adoc[]

docs/modules/ROOT/pages/scalalib/dependencies.adoc

@@ -16,6 +16,11 @@ include::partial$example/scalalib/dependencies/1-ivy-deps.adoc[]
 
 include::partial$example/scalalib/dependencies/2-run-compile-deps.adoc[]
 
+== Dependency Management
+
+Mill has support for dependency management, see the
+xref:fundamentals/library-deps.adoc#_dependency_management[Dependency Management section]
+in xref:fundamentals/library-deps.adoc[].
 
 == Unmanaged Jars
 

example/fundamentals/library-deps/bom-1-external-bom/build.mill

@@ -0,0 +1,25 @@
+// Pass an external BOM to a `JavaModule` / `ScalaModule` / `KotlinModule` with `bomDeps`, like
+
+//// SNIPPET:BUILD1
+package build
+import mill._, javalib._
+
+object foo extends JavaModule {
+  def bomDeps = Agg(
+    ivy"com.google.cloud:libraries-bom:26.50.0"
+  )
+  def ivyDeps = Agg(
+    ivy"io.grpc:grpc-protobuf"
+  )
+}
+
+// The version of grpc-protobuf (`io.grpc:grpc-protobuf`) isn't written down here, so the version
+// from the BOM, `1.67.1` is used.
+//
+// Also, by default, grpc-protobuf `1.67.1` pulls version `3.25.3` of protobuf-java (`com.google.protobuf:protobuf-java`) .
+// But the BOM specifies another version for that dependency, `4.28.3`, so
+// protobuf-java `4.28.3` ends up being pulled here.
+//
+// Several BOMs can be passed to `bomDeps`. If several specify a version for a dependency,
+// the version from the first one in the `bomDeps` list is used. If several specify exclusions
+// for a dependency, all exclusions are added to that dependency.

example/fundamentals/library-deps/bom-2-dependency-management/build.mill

@@ -0,0 +1,54 @@
+// Pass dependencies to `depManagement` in a `JavaModule` / `ScalaModule` / `KotlinModule`, like
+
+//// SNIPPET:BUILD1
+package build
+import mill._, javalib._
+
+object foo extends JavaModule {
+  def depManagement = Agg(
+    ivy"com.google.protobuf:protobuf-java:4.28.3",
+    ivy"io.grpc:grpc-protobuf:1.67.1"
+  )
+  def ivyDeps = Agg(
+    ivy"io.grpc:grpc-protobuf"
+  )
+}
+
+// The version of grpc-protobuf (`io.grpc:grpc-protobuf`) isn't written down here, so the version
+// found in `depManagement`, `1.67.1` is used.
+//
+// Also, by default, grpc-protobuf `1.67.1` pulls version `3.25.3` of protobuf-java (`com.google.protobuf:protobuf-java`) .
+// But `depManagement` specifies another version for that dependency, `4.28.3`, so
+// protobuf-java `4.28.3` ends up being pulled here.
+
+// One can also add exclusions via dependency management, like
+
+object bar extends JavaModule {
+  def depManagement = Agg(
+    ivy"io.grpc:grpc-protobuf:1.67.1"
+      .exclude(("com.google.protobuf", "protobuf-java"))
+  )
+  def ivyDeps = Agg(
+    ivy"io.grpc:grpc-protobuf"
+  )
+}
+
+// Here, grpc-protobuf has an empty version in `ivyDeps`, so the one in `depManagement`,
+// `1.67.1`, is used. Also, `com.google.protobuf:protobuf-java` is excluded from grpc-protobuf
+// in `depManagement`, so it ends up being excluded from it in `ivyDeps` too.
+
+// If one wants to add exclusions via `depManagement`, specifying a version is optional,
+// like
+
+object baz extends JavaModule {
+  def depManagement = Agg(
+    ivy"io.grpc:grpc-protobuf"
+      .exclude(("com.google.protobuf", "protobuf-java"))
+  )
+  def ivyDeps = Agg(
+    ivy"io.grpc:grpc-protobuf:1.67.1"
+  )
+}
+
+// Here, given that grpc-protobuf is fetched during dependency resolution,
+// `com.google.protobuf:protobuf-java` is excluded from it because of the dependency management.

example/package.mill

@@ -82,6 +82,7 @@ object `package` extends RootModule with Module {
     object cross extends Cross[ExampleCrossModule](build.listIn(millSourcePath / "cross"))
     object `out-dir` extends Cross[ExampleCrossModule](build.listIn(millSourcePath / "out-dir"))
     object libraries extends Cross[ExampleCrossModule](build.listIn(millSourcePath / "libraries"))
+    object `library-deps` extends Cross[ExampleCrossModule](build.listIn(millSourcePath / "library-deps"))
   }
 
   object depth extends Module {

main/util/src/mill/util/CoursierSupport.scala

@@ -31,6 +31,20 @@ trait CoursierSupport {
         ctx.fold(cache)(c => cache.withLogger(new TickerResolutionLogger(c)))
       }
 
+  private def isLocalTestDep(dep: Dependency): Option[Seq[PathRef]] = {
+    val org = dep.module.organization.value
+    val name = dep.module.name.value
+    val classpathKey = s"$org-$name"
+
+    val classpathResourceText =
+      try Some(os.read(
+          os.resource(getClass.getClassLoader) / "mill/local-test-overrides" / classpathKey
+        ))
+      catch { case e: os.ResourceNotFoundException => None }
+
+    classpathResourceText.map(_.linesIterator.map(s => PathRef(os.Path(s))).toSeq)
+  }
+
   /**
    * Resolve dependencies using Coursier.
    *
@@ -51,26 +65,8 @@ trait CoursierSupport {
       artifactTypes: Option[Set[Type]] = None,
       resolutionParams: ResolutionParams = ResolutionParams()
   ): Result[Agg[PathRef]] = {
-    def isLocalTestDep(dep: Dependency): Option[Seq[PathRef]] = {
-      val org = dep.module.organization.value
-      val name = dep.module.name.value
-      val classpathKey = s"$org-$name"
-
-      val classpathResourceText =
-        try Some(os.read(
-            os.resource(getClass.getClassLoader) / "mill/local-test-overrides" / classpathKey
-          ))
-        catch { case e: os.ResourceNotFoundException => None }
-
-      classpathResourceText.map(_.linesIterator.map(s => PathRef(os.Path(s))).toSeq)
-    }
-
-    val (localTestDeps, remoteDeps) = deps.iterator.toSeq.partitionMap(d =>
-      isLocalTestDep(d) match {
-        case None => Right(d)
-        case Some(vs) => Left(vs)
-      }
-    )
+    val (localTestDeps, remoteDeps) =
+      deps.iterator.toSeq.partitionMap(d => isLocalTestDep(d).toLeft(d))
 
     val resolutionRes = resolveDependenciesMetadataSafe(
       repositories,
@@ -262,6 +258,7 @@ trait CoursierSupport {
 
     val rootDeps = deps.iterator
       .map(d => mapDependencies.fold(d)(_.apply(d)))
+      .filter(dep => isLocalTestDep(dep).isEmpty)
       .toSeq
 
     val forceVersions = force.iterator

scalalib/src/mill/scalalib/CoursierModule.scala

@@ -231,6 +231,37 @@ object CoursierModule {
         sources: Boolean
     ): Agg[PathRef] =
       resolveDeps(deps, sources, None)
+
+    /**
+     * Processes dependencies and BOMs with coursier
+     *
+     * This makes coursier read and process BOM dependencies, and fill version placeholders
+     * in dependencies with the BOMs.
+     *
+     * Note that this doesn't throw when a version placeholder cannot be filled, and just leaves
+     * the placeholder behind.
+     *
+     * @param deps dependencies that might have placeholder versions ("_" as version)
+     * @param resolutionParams coursier resolution parameters
+     * @return dependencies with version placeholder filled
+     */
+    def processDeps[T: CoursierModule.Resolvable](
+        deps: IterableOnce[T],
+        resolutionParams: ResolutionParams = ResolutionParams()
+    ): Seq[Dependency] = {
+      val deps0 = deps
+        .map(implicitly[CoursierModule.Resolvable[T]].bind(_, bind))
+      val res = Lib.resolveDependenciesMetadataSafe(
+        repositories = repositories,
+        deps = deps0,
+        mapDependencies = mapDependencies,
+        customizer = customizer,
+        coursierCacheCustomizer = coursierCacheCustomizer,
+        ctx = ctx,
+        resolutionParams = resolutionParams
+      ).getOrThrow
+      res.processedRootDependencies
+    }
   }
 
   sealed trait Resolvable[T] {

scalalib/src/mill/scalalib/Dep.scala

@@ -118,6 +118,8 @@ object Dep {
     }
 
     (module.split(':') match {
+      case Array(a, b) => Dep(a, b, "_", cross = empty(platformed = false))
+      case Array(a, "", b) => Dep(a, b, "_", cross = Binary(platformed = false))
       case Array(a, b, c) => Dep(a, b, c, cross = empty(platformed = false))
       case Array(a, b, "", c) => Dep(a, b, c, cross = empty(platformed = true))
       case Array(a, "", b, c) => Dep(a, b, c, cross = Binary(platformed = false))