[Backport 2.x] Switch from buildSrc/version.properties to Gradle version catalog (gradle/libs.versions.toml) to enable dependabot to perform automated upgrades on common libs (#16284) (#16508)

cwperks · reta · web-flow · commit 037c3ea568b3 · 2024-10-28T19:09:22.000-04:00
* Switch from `buildSrc/version.properties` to Gradle version catalog (`gradle/libs.versions.toml`) to enable dependabot to perform automated upgrades on common libs (#16284) * WIP on lib toml Signed-off-by: Craig Perkins <cwperx@amazon.com> * SpotlessApply Signed-off-by: Craig Perkins <cwperx@amazon.com> * Remove unnecessary lines Signed-off-by: Craig Perkins <cwperx@amazon.com> * Specify time when dependabot runs Signed-off-by: Craig Perkins <cwperx@amazon.com> * Refer to version from libs.versions.toml Signed-off-by: Craig Perkins <cwperx@amazon.com> * Use version Signed-off-by: Craig Perkins <cwperx@amazon.com> * Specify version catalog Signed-off-by: Craig Perkins <cwperx@amazon.com> * Call .get() Signed-off-by: Craig Perkins <cwperx@amazon.com> * Define version catalog Signed-off-by: Craig Perkins <cwperx@amazon.com> * Use libraries Signed-off-by: Craig Perkins <cwperx@amazon.com> * Downgrade purposefully Signed-off-by: Craig Perkins <cwperx@amazon.com> * Add mavenCentral Signed-off-by: Craig Perkins <cwperx@amazon.com> * Try w/o libraries section Signed-off-by: Craig Perkins <cwperx@amazon.com> * reinstate Signed-off-by: Craig Perkins <cwperx@amazon.com> * Remove version.properties Signed-off-by: Craig Perkins <cwperx@amazon.com> * Update syntax Signed-off-by: Craig Perkins <cwperx@amazon.com> * Change back to weekly Signed-off-by: Craig Perkins <cwperx@amazon.com> * Add grpc Signed-off-by: Craig Perkins <cwperx@amazon.com> * Get relative to project root. Relative path not working on windows bc windows had gradle wrapper path Signed-off-by: Craig Perkins <cwperx@amazon.com> * Add minimal version.properties with only opensearch version to accommodate external references Signed-off-by: Craig Perkins <cwperx@amazon.com> * singularize version.properties Signed-off-by: Craig Perkins <cwperx@amazon.com> * Get rootDir Signed-off-by: Craig Perkins <craig5008@gmail.com> * Fix issue loading snapshot Signed-off-by: Craig Perkins <craig5008@gmail.com> * Limit logic to generating version.properties file within buildSrc Signed-off-by: Craig Perkins <cwperx@amazon.com> * Remove unused exports Signed-off-by: Craig Perkins <cwperx@amazon.com> * Add import Signed-off-by: Craig Perkins <cwperx@amazon.com> * Remove unused code Signed-off-by: Craig Perkins <cwperx@amazon.com> * Remove mavenCentral from publication section Signed-off-by: Craig Perkins <cwperx@amazon.com> * Add to CHANGELOG Signed-off-by: Craig Perkins <cwperx@amazon.com> * Update reactor-netty version Signed-off-by: Craig Perkins <cwperx@amazon.com> * Only keep versions section in toml Signed-off-by: Craig Perkins <cwperx@amazon.com> * Replaces versions catalog TOML parsing with Gradle's VersionCatalogsExtension Signed-off-by: Andriy Redko <andriy.redko@aiven.io> * Update bundled_jdk Signed-off-by: Craig Perkins <cwperx@amazon.com> * Update bytebuddy and mockito Signed-off-by: Craig Perkins <cwperx@amazon.com> --------- Signed-off-by: Craig Perkins <cwperx@amazon.com> Signed-off-by: Craig Perkins <craig5008@gmail.com> Signed-off-by: Andriy Redko <andriy.redko@aiven.io> Signed-off-by: Daniel (dB.) Doubrovkine <dblock@amazon.com> Co-authored-by: Andriy Redko <andriy.redko@aiven.io> Co-authored-by: Daniel (dB.) Doubrovkine <dblock@amazon.com> (cherry picked from commit af7a067) * Update os version in libs.versions.toml Signed-off-by: Craig Perkins <cwperx@amazon.com> * tdigest 3.2 Signed-off-by: Craig Perkins <cwperx@amazon.com> * Update libs.versions.toml Signed-off-by: Andriy Redko <andriy.redko@aiven.io> * Update libs.versions.toml Signed-off-by: Andriy Redko <andriy.redko@aiven.io> --------- Signed-off-by: Craig Perkins <cwperx@amazon.com> Signed-off-by: Andriy Redko <andriy.redko@aiven.io> Co-authored-by: Andriy Redko <andriy.redko@aiven.io> Co-authored-by: Andriy Redko <drreta@gmail.com>
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -5,6 +5,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 
 ## [Unreleased 2.x]
 ### Added
+- Switch from `buildSrc/version.properties` to Gradle version catalog (`gradle/libs.versions.toml`) to enable dependabot to perform automated upgrades on common libs ([#16284](https://github.com/opensearch-project/OpenSearch/pull/16284))
 
 ### Dependencies
 
diff --git a/buildSrc/build.gradle b/buildSrc/build.gradle
@@ -57,7 +57,7 @@ if (project == rootProject) {
 // we update the version property to reflect if we are building a snapshot or a release build
 // we write this back out below to load it in the Build.java which will be shown in rest main action
 // to indicate this being a snapshot build or a release build.
-Properties props = VersionPropertiesLoader.loadBuildSrcVersion(project.file('version.properties'))
+Properties props = VersionPropertiesLoader.loadBuildSrcVersion(project)
 version = props.getProperty("opensearch")
 
 def generateVersionProperties = tasks.register("generateVersionProperties", WriteProperties) {
@@ -285,15 +285,19 @@ if (project != rootProject) {
   }
 }
 
-// Define this here because we need it early.
+// Define this here because we need it early. It uses VersionCatalogsExtension to extract all versions
+// and converts to a Java Properties object
 class VersionPropertiesLoader {
-  static Properties loadBuildSrcVersion(File input) throws IOException {
+  static Properties loadBuildSrcVersion(Project project) throws IOException {
     Properties props = new Properties();
-    InputStream is = new FileInputStream(input)
-    try {
-      props.load(is)
-    } finally {
-      is.close()
+
+    var catalogs = project.extensions.getByType(VersionCatalogsExtension)
+    var libs = catalogs.named("libs")
+    libs.getVersionAliases().forEach {
+       libs.findVersion(it).ifPresent { v ->
+         // Gradle replaces '_' with '.' so 'google_http_client' becomes 'google.http.client' instead
+         props.setProperty(it.replaceAll("[.]", "_"), v.requiredVersion)
+       }
     }
     loadBuildSrcVersion(props, System.getProperties())
     return props
diff --git a/buildSrc/settings.gradle b/buildSrc/settings.gradle
@@ -10,3 +10,11 @@
  */
 
 include 'reaper'
+
+dependencyResolutionManagement {
+ versionCatalogs {
+    libs {
+      from(files("../gradle/libs.versions.toml"))
+    }
+  }
+}
diff --git a/buildSrc/version.properties b/buildSrc/version.properties
@@ -1,77 +1 @@
 opensearch        = 2.19.0
-lucene            = 9.12.0
-
-bundled_jdk_vendor = adoptium
-bundled_jdk = 21.0.5+11
-
-# optional dependencies
-spatial4j         = 0.7
-jts               = 1.15.0
-jackson           = 2.17.2
-jackson_databind  = 2.17.2
-snakeyaml         = 2.1
-icu4j             = 75.1
-supercsv          = 2.4.0
-log4j             = 2.21.0
-slf4j             = 1.7.36
-asm               = 9.7
-jettison          = 1.5.4
-woodstox          = 6.4.0
-kotlin            = 1.7.10
-antlr4            = 4.13.1
-guava             = 32.1.1-jre
-protobuf          = 3.25.5
-jakarta_annotation = 1.3.5
-google_http_client = 1.44.1
-tdigest           = 3.2
-hdrhistogram      = 2.2.2
-grpc              = 1.68.0
-
-# when updating the JNA version, also update the version in buildSrc/build.gradle
-jna               = 5.13.0
-
-netty             = 4.1.114.Final
-joda              = 2.12.7
-
-# project reactor
-reactor_netty     = 1.1.23
-reactor           = 3.5.20
-
-# client dependencies
-httpclient        = 4.5.14
-httpcore          = 4.4.16
-httpasyncclient   = 4.1.5
-commonslogging    = 1.2
-commonscodec      = 1.16.1
-commonslang       = 3.14.0
-commonscompress   = 1.26.1
-commonsio         = 2.16.0
-# plugin dependencies
-aws               = 2.20.86
-reactivestreams   = 1.0.4
-
-# when updating this version, you need to ensure compatibility with:
-#  - plugins/ingest-attachment (transitive dependency, check the upstream POM)
-#  - distribution/tools/plugin-cli
-bouncycastle=1.78
-# test dependencies
-randomizedrunner  = 2.7.1
-junit             = 4.13.2
-hamcrest          = 2.1
-mockito           = 5.14.1
-objenesis         = 3.2
-bytebuddy         = 1.15.4
-
-# benchmark dependencies
-jmh               = 1.35
-
-# compression
-zstd              = 1.5.5-5
-
-jzlib             = 1.1.3
-
-resteasy          = 6.2.4.Final
-
-# opentelemetry dependencies
-opentelemetry         = 1.41.0
-opentelemetrysemconv  = 1.27.0-alpha
diff --git a/gradle/libs.versions.toml b/gradle/libs.versions.toml
@@ -0,0 +1,78 @@
+[versions]
+opensearch        = "2.19.0"
+lucene            = "9.12.0"
+
+bundled_jdk_vendor = "adoptium"
+bundled_jdk = "21.0.5+11"
+
+# optional dependencies
+spatial4j         = "0.7"
+jts               = "1.15.0"
+jackson           = "2.17.2"
+jackson_databind  = "2.17.2"
+snakeyaml         = "2.1"
+icu4j             = "75.1"
+supercsv          = "2.4.0"
+log4j             = "2.21.0"
+slf4j             = "1.7.36"
+asm               = "9.7"
+jettison          = "1.5.4"
+woodstox          = "6.4.0"
+kotlin            = "1.7.10"
+antlr4            = "4.13.1"
+guava             = "32.1.1-jre"
+protobuf          = "3.25.5"
+jakarta_annotation = "1.3.5"
+google_http_client = "1.44.1"
+tdigest           = "3.2"
+hdrhistogram      = "2.2.2"
+grpc              = "1.68.0"
+
+# when updating the JNA version, also update the version in buildSrc/build.gradle
+jna               = "5.13.0"
+
+netty             = "4.1.114.Final"
+joda              = "2.12.7"
+
+# project reactor
+reactor_netty     = "1.1.23"
+reactor           = "3.5.20"
+
+# client dependencies
+httpclient        = "4.5.14"
+httpcore          = "4.4.16"
+httpasyncclient   = "4.1.5"
+commonslogging    = "1.2"
+commonscodec      = "1.16.1"
+commonslang       = "3.14.0"
+commonscompress   = "1.26.1"
+commonsio         = "2.16.0"
+# plugin dependencies
+aws               = "2.20.86"
+reactivestreams   = "1.0.4"
+
+# when updating this version, you need to ensure compatibility with:
+#  - plugins/ingest-attachment (transitive dependency, check the upstream POM)
+#  - distribution/tools/plugin-cli
+bouncycastle="1.78"
+# test dependencies
+randomizedrunner  = "2.7.1"
+junit             = "4.13.2"
+hamcrest          = "2.1"
+mockito           = "5.14.1"
+objenesis         = "3.2"
+bytebuddy         = "1.15.4"
+
+# benchmark dependencies
+jmh               = "1.35"
+
+# compression
+zstd              = "1.5.5-5"
+
+jzlib             = "1.1.3"
+
+resteasy          = "6.2.4.Final"
+
+# opentelemetry dependencies
+opentelemetry         = "1.41.0"
+opentelemetrysemconv  = "1.27.0-alpha"
