feat(catalog): Add catalog loader and builder implementation for rest catalog #1580

lliangyu-lin · 2025-08-04T18:40:13Z

Which issue does this PR close?

Part of Implement catalog loader function. #1260
Closes Implement rest catalog loader. #1255

What changes are included in this PR?

Some of the implementations are directly coming from the design in Experiment implementation for catalog builder #1231
Added a new crate loader for loading catalog based on type.
Implement the CatalogBuilder trait for rest catalog

Are these changes tested?

Yes, by unit tests

CTTY · 2025-08-05T04:57:05Z

crates/catalog/loader/src/lib.rs

+use iceberg_catalog_rest::RestCatalogBuilder;
+
+#[async_trait]
+pub trait BoxedCatalogBuilder {


I think we can just put this in builder.rs under /crates/catalog/rest/src/

Thanks for taking a look!
I'm not sure if it's necessary because I think BoxedCatalogBuilder is suppose to be generic to use for all catalog builder implementations. Like in L42, we simply call the load method in the trait to return the corresponding catalog and can be extended to add more catalogs in load, so it's not really rest catalog specific.

Sorry I meant the common code like this can be put under crates/iceberg/src/catalog/mod.rs. I don't see the need of adding another module here, maybe I'm missing anything?

Also I'm thinking if Box<Self> will be common across all catalog builders, maybe we can just change the CatalogBuilder trait to be self: Arc<Self> or self: Box<Self>, so we don't need to define a wrapper trait like this, wdyt?

For putting these in crates/iceberg/src/catalog/mod.rs, I think it's fine, but if we want to add it as dependency to playground or integration test framework, does it mean we need to import the catalog crate individually?

For the Box<Self>, I think we are only consuming the CatalogBuilder and doesn't need to share it, so I think Box<Self> is fine. cc: @liurenjie1024 wondering if you have any suggestion on this?

For the Box, I think we are only consuming the CatalogBuilder and doesn't need to share it, so I think Box is fine.

+1. I think it's fine to put the BoxedCatalogBuilder here since it's mainly used by load function.

liurenjie1024

Thanks @lliangyu-lin for this pr, generally LGTM! Just missing some doc and a little change.

liurenjie1024 · 2025-08-07T10:23:39Z

crates/catalog/loader/src/lib.rs

+use iceberg_catalog_rest::RestCatalogBuilder;
+
+#[async_trait]
+pub trait BoxedCatalogBuilder {


For the Box, I think we are only consuming the CatalogBuilder and doesn't need to share it, so I think Box is fine.

+1. I think it's fine to put the BoxedCatalogBuilder here since it's mainly used by load function.

liurenjie1024 · 2025-08-07T10:25:06Z

crates/catalog/rest/src/catalog.rs

@@ -45,13 +46,18 @@ use crate::types::{
    RegisterTableRequest, RenameTableRequest,
 };

+const REST_CATALOG_PROP_URI: &str = "uri";
+const REST_CATALOG_PROP_WAREHOUSE: &str = "warehouse";
 const ICEBERG_REST_SPEC_VERSION: &str = "0.14.1";
 const CARGO_PKG_VERSION: &str = env!("CARGO_PKG_VERSION");
 const PATH_V1: &str = "v1";

 /// Rest catalog configuration.
 #[derive(Clone, Debug, TypedBuilder)]
 pub struct RestCatalogConfig {


We should mark this as crate private, and adding doc to explain how to load a RestCatalog with the new approach, e.g. using the RestCatalogBuilder. Otherwise user will be confusing since we will have two methods for building rest catalog.

I made the RestCatalogConfig crate private, seems like it's backward incompatible change as previous catalog new will require a RestCatalogConfig.

liurenjie1024 · 2025-08-08T09:37:52Z

crates/catalog/rest/src/catalog.rs

@@ -45,13 +46,113 @@ use crate::types::{
    RegisterTableRequest, RenameTableRequest,
 };

+const REST_CATALOG_PROP_URI: &str = "uri";


We should make them public as users are epxecting to use these constants rather than raw string.

liurenjie1024 · 2025-08-08T09:39:09Z

crates/catalog/rest/src/catalog.rs

+/// Builder for [`RestCatalog`].
+/// To build a rest catalog with configurations
+///
+/// # Example
+///
+/// ```rust, no_run
+/// use std::collections::HashMap;
+///
+/// use iceberg::CatalogBuilder;
+/// use iceberg_catalog_rest::RestCatalogBuilder;
+///
+/// #[tokio::main]
+/// async fn main() {
+///     let catalog = RestCatalogBuilder::default()
+///         .load(
+///             "rest",
+///             HashMap::from([
+///                 ("uri".to_string(), "http://localhost:8181".to_string()),
+///                 ("warehouse".to_string(), "s3://warehouse".to_string()),
+///             ]),
+///         )
+///         .await
+///         .unwrap();
+/// }
+/// ```


How about moving this crate's doc, e.g. in lib.rs? This should be the first thing a user sees when using this crate.

Yep, I think moving to lib.rs make sense.

liurenjie1024 · 2025-08-08T09:39:54Z

crates/catalog/rest/src/catalog.rs

+///         .load(
+///             "rest",
+///             HashMap::from([
+///                 ("uri".to_string(), "http://localhost:8181".to_string()),


We should use constants rather than raw string as better practice. Please change all occurences.

liurenjie1024

Thanks @lliangyu-lin for this pr!

CTTY · 2025-08-11T22:35:14Z

It seems this PR didn't include the Cargo.lock changes, I've put up a minor follow-up here: #1596

## Which issue does this PR close? - None ## What changes are included in this PR? - Auto-update `Cargo.lock` based on the changes from #1580 ## Are these changes tested? Using the existing uts

liurenjie1024 and others added 4 commits August 4, 2025 01:18

Add loader impl

b07c019

Add collect remaining props logic and add unit tests

4e8b320

Apply loader design from renjie

7d4d734

Fix issues

5e52118

CTTY reviewed Aug 5, 2025

View reviewed changes

liurenjie1024 reviewed Aug 7, 2025

View reviewed changes

Make rest catalog config crate private and prefer to use catalog loader

84b398c

liurenjie1024 reviewed Aug 8, 2025

View reviewed changes

lliangyu-lin added 2 commits August 8, 2025 14:50

Make property public and use defined property

a929435

Minor update on doc and use name in playground catalog name

c3a2d57

liurenjie1024 approved these changes Aug 11, 2025

View reviewed changes

liurenjie1024 merged commit b174c45 into apache:main Aug 11, 2025
18 checks passed

CTTY mentioned this pull request Aug 11, 2025

minor: Update cargo lock for rest catalog loader #1596

Merged

feat(catalog): Add catalog loader and builder implementation for rest catalog #1580

feat(catalog): Add catalog loader and builder implementation for rest catalog #1580

Uh oh!

Conversation

lliangyu-lin commented Aug 4, 2025

Which issue does this PR close?

What changes are included in this PR?

Are these changes tested?

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

liurenjie1024 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

liurenjie1024 Aug 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

liurenjie1024 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

CTTY commented Aug 11, 2025

Uh oh!

Uh oh!

liurenjie1024 Aug 8, 2025 •

edited

Loading