|
| 1 | +--- |
| 2 | +title: The hassle of transitive dependencies across sub-projects in Gradle |
| 3 | +canonical_url: 'https://blog.cronn.de/en/gradle/java/2021/05/06/gradle-dependencies-gotchas-in-subprojects.html' |
| 4 | +categories: [Code deep dives] |
| 5 | +tags: [gradle, java] |
| 6 | +--- |
| 7 | + |
| 8 | +> **_NOTE:_** Originally posted on https://blog.cronn.de/en/gradle/java/2021/05/06/gradle-dependencies-gotchas-in-subprojects.html |
| 9 | +
|
| 10 | +You have probably heard the term [dependency hell](https://blog.gradle.org/avoiding-dependency-hell-gradle-6). |
| 11 | +Modern build tools, like Gradle, handle that problem very well, especially by providing automatic resolution of transitive dependencies. |
| 12 | +By default, Gradle picks the highest version when many different versions of the same library are required. |
| 13 | +However, it still has some flaws, and we will focus on one of them, observed in a multi-project in Gradle. |
| 14 | + |
| 15 | +## What exactly is the problem? |
| 16 | + |
| 17 | +First of all, we should clarify the terminology. |
| 18 | +Dependencies can be divided into two categories: explicitly or implicitly declared. |
| 19 | +The former is called a *“first-level”* dependency, the latter a *“transitive”* dependency. |
| 20 | + |
| 21 | +Let's say in a multi-project we have two sub-projects: `project-a` and `project-b`. |
| 22 | +Our `project-a` has a first-level dependency on library `lib-x` in version `1.2`, while `project-b` has a dependency on library `lib-y`, which also has a dependency on `lib-x` but here it is in a **lower** version - let's say `1.1`. |
| 23 | +What's more `project-a` depends on `project-b`. It could be expressed as: |
| 24 | + |
| 25 | +``` |
| 26 | +project-a |
| 27 | +- project-b |
| 28 | +- lib-x 1.2 |
| 29 | +
|
| 30 | +project-b |
| 31 | +- lib-y |
| 32 | + - lib-x 1.1 |
| 33 | +``` |
| 34 | + |
| 35 | +Running `project-a` will resolve the transitive dependency `lib-x` in the highest version `1.2` as expected. |
| 36 | +However, running `project-b` (e.g. tests) will resolve `lib-x` in version `1.1`, so there will be an inconsistency between dependent projects. |
| 37 | +A simple solution would be for `project-b` to have a **higher** version of `lib-x`, as this would result in both resolved versions being the same for both sub-projects. But this isn't always the case. |
| 38 | + |
| 39 | +The dependency report for `project-a` with a **lower** version of `lib-x` in `project-b` would look like this: |
| 40 | +``` |
| 41 | ++--- project-b |
| 42 | +| +--- lib-y |
| 43 | +| | +--- lib-x 1.1 -> 1.2 |
| 44 | ++--- lib-x 1.2 |
| 45 | +``` |
| 46 | + |
| 47 | +Whereas the report for `project-b` alone will resolve to a different version: |
| 48 | +``` |
| 49 | ++--- lib-y |
| 50 | +| +--- lib-x 1.1 |
| 51 | +``` |
| 52 | + |
| 53 | +*Sure, but actually it makes sense, doesn't it?* |
| 54 | +Right, `project-b` knows nothing about the dependencies of `project-a`. |
| 55 | +Indeed, this is how [resolution strategy](https://docs.gradle.org/current/userguide/dependency_resolution.html#sub:resolution-strategy) works - Gradle picks the highest version from all requested modules in the given context. |
| 56 | + |
| 57 | +*So what's the problem then?* |
| 58 | +In most projects, we treat sub-projects as a whole. |
| 59 | +Therefore, if dependency resolution is not consistent across all sub-projects, it may result in unexpected behavior at runtime! |
| 60 | + |
| 61 | +*Could this really happen?* |
| 62 | +There are some examples of [binary incompatibility](https://stackoverflow.com/questions/43568116/is-there-a-way-to-stop-scala-2-12-breaking-the-jackson-object-mapper), but it's even worse when it comes to runtime differences because it's harder to spot. |
| 63 | +Besides, according to Murphy's Law: “If anything can go wrong, it will” ;) |
| 64 | + |
| 65 | + |
| 66 | +Now let's move on and consider a scenario when the inconsistency is between transitive dependencies. |
| 67 | +We modify our example so that `lib-x` in version `1.2` is now required by a first-level dependency `lib-z` declared in `project-a`. |
| 68 | + |
| 69 | +``` |
| 70 | +project-a |
| 71 | +- project-b |
| 72 | +- lib-z |
| 73 | + - lib-x 1.2 |
| 74 | +
|
| 75 | +project-b |
| 76 | +- lib-y |
| 77 | + - lib-x 1.1 |
| 78 | +``` |
| 79 | + |
| 80 | +This looks even less intuitive because the build tool should do [all the dirty work for us](https://rafael.cordones.me/blog/transitive-dependency-mediation-in-maven-and-gradle/#can-i-emhandleem-the-maven-truth), right? ;) |
| 81 | + |
| 82 | +Summarizing, we could consider two relevant scenarios of resolving conflicting versions between sub-projects (provided that one depends on another): |
| 83 | +1. first-level vs. transitive |
| 84 | +2. transitive vs. transitive |
| 85 | + |
| 86 | +## How to detect version conflicts? |
| 87 | + |
| 88 | +To do this we need a tool that resolves all dependencies from each project separately, then lists them together to find duplicates. |
| 89 | +In my case, the natural pick is *Intellij IDEA*. |
| 90 | +We can check dependencies in the `Libraries` view in the `Project structure` window. |
| 91 | +It gathers all resolved dependencies from *all configurations*. |
| 92 | +If `<group>:<name>` is listed more than once, it could mean you are in trouble :) |
| 93 | +The drawback is that you must find these conflicts with your own eyes. |
| 94 | +Another thing is that `buildSrc` dependencies are also included, which could be misleading. |
| 95 | + |
| 96 | +If you are not an Intellij user, you might try Gradle's `dependencies` task. |
| 97 | +However, it only generates reports for a single project. |
| 98 | +To use it for all sub-projects we should do [this trick](https://solidsoft.wordpress.com/2014/11/13/gradle-tricks-display-dependencies-for-all-subprojects-in-multi-project-build) and define our own task. We may also want to run it only for the `testRuntimeClasspath` configuration: |
| 99 | +```groovy |
| 100 | +allprojects { |
| 101 | + afterEvaluate { project -> |
| 102 | + if (project.configurations.findByName("testRuntimeClasspath")) { |
| 103 | + task allDeps(type: DependencyReportTask) { |
| 104 | + configuration = "testRuntimeClasspath" |
| 105 | + } |
| 106 | + } |
| 107 | + } |
| 108 | +} |
| 109 | +``` |
| 110 | + |
| 111 | +Then, we may try to parse the task output and find conflicting dependencies: |
| 112 | +```bash |
| 113 | +$ ./gradlew allDeps | sed -E 's/:[0-9][^ ]* -> /:/g' | grep -Po "[^ ]+:.+:\d[^ ]*" \ |
| 114 | +| sort | uniq | cut -d':' -f-2 | uniq -c | grep -v 1 |
| 115 | + 2 commons-beanutils:commons-beanutils |
| 116 | + 2 commons-collections:commons-collections |
| 117 | + ... |
| 118 | + ``` |
| 119 | +Let's explain this code. |
| 120 | +First by `sed -E 's/:[0-9][^ ]* -> /:/g'`, we want to make sure only resolved versions are considered. |
| 121 | +Thus, we cut off the declared version from transitive dependencies marked with an arrow (`->`). |
| 122 | +Next with `grep -Po "[^ ]+:.+:\d[^ ]*"` only GAVs (`<group>:<name>:<version>`) are extracted. |
| 123 | +Then we prepare a unique list of versions with `sort | uniq`. |
| 124 | +Next, we want to group by `<group>:<name>` to identify duplicated versions, so we cut off the version with `cut -d':' -f-2`. |
| 125 | +Lastly, we count each occurrence `uniq -c` and print only those with count greater than one: `grep -v 1`. |
| 126 | + |
| 127 | +*Okay, but is there really no way to tell Gradle to warn me?* |
| 128 | +Actually, there is. Gradle has a built-in check for transitive dependency conflicts. |
| 129 | +We could try the `failOnVersionConflict` resolution strategy. |
| 130 | +Then, with the help of the task `dependencyInsight`, we can narrow down the conflict to a single dependency. |
| 131 | +Unfortunately, it's not really helpful in our case, because single libraries or large BOMs could have many conflicts themselves (e.g. `org.springframework.boot:spring-boot-dependencies:2.2.7.RELEASE`). |
| 132 | + |
| 133 | +Nevertheless, there is some good news :) |
| 134 | +In future releases (Gradle 7.x) there are some plans to introduce a new flag in [resolution strategy tuning](https://docs.gradle.org/current/userguide/resolution_strategy_tuning.html) to fail on inconsistent dependency resolution across sub-projects. |
| 135 | +Currently we can only enforce versions for all configurations in a single project: |
| 136 | +```groovy |
| 137 | +java { |
| 138 | + consistentResolution { |
| 139 | + useCompileClasspathVersions() |
| 140 | + } |
| 141 | +} |
| 142 | +``` |
| 143 | + |
| 144 | +## Is there any solution? |
| 145 | + |
| 146 | +Luckily, the official documentation regarding transitives covers a wide range of options. |
| 147 | +I found a few approaches, including [custom plugins](https://github.com/nebula-plugins/nebula-dependency-recommender-plugin#54--transitive-dependencies). |
| 148 | +Currently the recommended solution for sharing dependency versions across projects is **platform**. |
| 149 | +Another solution, introduced as a feature preview in Gradle 7, is the `version catalogs`, but it doesn't offer so many [capabilities](https://docs.gradle.org/7.0/userguide/platforms.html#sub:platforms-vs-catalog). |
| 150 | + |
| 151 | +Our goal is clear: we want the highest common version across all projects. |
| 152 | +Does the *platform* seal the deal? Let's find out! |
| 153 | + |
| 154 | +The whole idea of the [*platform*](https://docs.gradle.org/current/userguide/platforms.html) came from Maven's BOM (Bill of Materials), which is a centralized set of dependency versions that “work well together”. |
| 155 | +This way we can define our first-level dependencies with just `<group>:<name>`, as long as they are defined in the *platform*. |
| 156 | +```groovy |
| 157 | +dependencies { |
| 158 | + implementation platform(project(":platform")) |
| 159 | + implementation "<group>:<name>" |
| 160 | +} |
| 161 | +``` |
| 162 | + |
| 163 | +This means we can use the same version all over the place. Sounds good! |
| 164 | +Yet, there are some pitfalls... The major one being that it's not really solving our problem. |
| 165 | + |
| 166 | +First, let's take a look at the example of first-level vs. transitive. |
| 167 | +When the **lower** version is defined in the base project `project-b`, then despite defining `lib-x` as a first-level dependency in the *platform*, different versions are still resolved. |
| 168 | +``` |
| 169 | +Platform |
| 170 | +- lib-x 1.1 (first-level) |
| 171 | +- lib-y |
| 172 | + - lib-x 1.2 (transitive) |
| 173 | +
|
| 174 | +project-a |
| 175 | +- Platform |
| 176 | +- project-b |
| 177 | +- lib-y |
| 178 | + - lib-x (version 1.2 resolved) |
| 179 | +
|
| 180 | +project-b |
| 181 | +- Platform |
| 182 | +- lib-x (version 1.1 resolved) |
| 183 | +``` |
| 184 | + |
| 185 | +The *platform* is not magically making versions consistent within itself - the same resolution rules still apply. |
| 186 | +To fix this and always use the version of the first-level dependency, we have to use `enforcedPlatform` instead: |
| 187 | +```groovy |
| 188 | +dependencies { |
| 189 | + implementation enforcedPlatform(project(":platform")) |
| 190 | +} |
| 191 | +``` |
| 192 | + |
| 193 | +Now we have the same version across all sub-projects, but actually it is the **lower** one. |
| 194 | +This means you can accidentally downgrade dependencies ;) |
| 195 | +Yet we don't want to downgrade dependencies, but just align them. |
| 196 | + |
| 197 | +Thus, we could consider two inspections here: |
| 198 | +- listing places where `platform` is used instead of `enforcedPlatform` |
| 199 | +- report dependencies that are resolved in a lower version than declared (the highest version should be OK, but it always better to upgrade related libraries with a lower version of transitive) |
| 200 | + |
| 201 | +Another snag is the case of transitive vs. transitive. |
| 202 | +Even with `enforcedPlatform` we still have to declare transitive dependency [explicitly to get rid of conflict](https://docs.gradle.org/current/userguide/dependency_constraints.html#sec:adding-constraints-transitive-deps). |
| 203 | +And again we might accidentally downgrade both dependencies, so we have to be careful. |
| 204 | + |
| 205 | +Therefore, enforcing the *platform* is not much different from using `force` in `resolutionStrategy`. |
| 206 | +Both cases are still hard to maintain because we need additional reports of unused constraints or unintentional downgrades. |
| 207 | + |
| 208 | +## Conclusions |
| 209 | +The *platform* seems to be the ideal solution, but it isn't, especially for maintainers of dependencies. |
| 210 | +Unexpected dependency conflicts still may occur, especially in multi-projects. |
| 211 | +Thus, every dependency update needs to be done carefully and wisely. There is no easy way. |
| 212 | +However, we can develop tools that could support us, like reports. |
| 213 | +Let's hope that the future Gradle releases will provide us with warnings about such conflicts. |
0 commit comments