Allow to pass botocore session to REST Catalog (S3 Tables) like Glue Catalog does

[rudolfix]

Feature Request / Improvement

Background
Glue catalogs allows to setup access credentials when creating boto3.Session. Unfortunately, REST Catalog is able to use default session only in class SigV4Adapter(HTTPAdapter):.
Ideally exactly the same mechanism could be used here (OFC with constants renamed)

session = boto3.Session(
                profile_name=properties.get(GLUE_PROFILE_NAME),
                region_name=get_first_property_value(properties, GLUE_REGION, AWS_REGION),
                botocore_session=properties.get(BOTOCORE_SESSION),
                aws_access_key_id=get_first_property_value(properties, GLUE_ACCESS_KEY_ID, AWS_ACCESS_KEY_ID),
                aws_secret_access_key=get_first_property_value(properties, GLUE_SECRET_ACCESS_KEY, AWS_SECRET_ACCESS_KEY),
                aws_session_token=get_first_property_value(properties, GLUE_SESSION_TOKEN, AWS_SESSION_TOKEN),
            )

Implementation
I'm happy to implement the change. I have it working locally. I can also remove some of the code duplication (FileIO also creates boto session - it looks as above)

Pls tell me if you are interested in PR

[potatochipcoconut]

@rudolfix what's the implication of how it currently works? because I am seeing some weird iam errors when writing data in sometimes w/lambda, not sure if related.

[rudolfix]

@rudolfix what's the implication of how it currently works? because I am seeing some weird iam errors when writing data in sometimes w/lambda, not sure if related.

I can't easily configure S3 Table access credentials - I have to provide default AWS profile in .aws or AWS env variables. If that is not done then I get this:

  File "/home/rudolfix/src/dlt-plus/playground/demo_aws_catalogs/error_catalog_credentials.py", line 24, in <module>
    raise e
  File "/home/rudolfix/src/dlt-plus/playground/demo_aws_catalogs/error_catalog_credentials.py", line 22, in <module>
    rest_catalog = RestCatalog(name="s3tables", uri=uri, **properties)
  File "/home/rudolfix/src/iceberg-python/pyiceberg/catalog/rest/__init__.py", line 265, in __init__
    self._fetch_config()
  File "/home/rudolfix/src/iceberg-python/pyiceberg/catalog/rest/__init__.py", line 382, in _fetch_config
    response = session.get(self.url(Endpoints.get_config, prefixed=False), params=params)
  File "/home/rudolfix/src/dlt-plus/.venv/lib/python3.10/site-packages/requests/sessions.py", line 602, in get
    return self.request("GET", url, **kwargs)
  File "/home/rudolfix/src/dlt-plus/.venv/lib/python3.10/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/rudolfix/src/dlt-plus/.venv/lib/python3.10/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
  File "/home/rudolfix/src/dlt-plus/.venv/lib/python3.10/site-packages/requests/adapters.py", line 641, in send
    self.add_headers(
  File "/home/rudolfix/src/iceberg-python/pyiceberg/catalog/rest/__init__.py", line 484, in add_headers
    credentials = boto_session.get_credentials().get_frozen_credentials()
AttributeError: 'NoneType' object has no attribute 'get_frozen_credentials'

we use pyiceberg from our library (dlt) where many configurations should coexist as the same time and such "hardcoded" session makes that harder

[YangKian]

Hello!
We're encountering the same issue using PyIceberg to access AWS S3 tables with the AWS S3 Table REST Catalog. Our use case involves multiple clients accessing different S3 tables concurrently, so we need PyIceberg to support dynamically specifying credentials in code.

[YangKian]

Hi @rudolfix , just wonder do you still have plans to implement this change?

[rudolfix]

looks like it is implemented here: #2071 and this exactly what I planned to do!