S3Exception when metadata value contains non-ASCII characters

[cbromberg]

S3Client throws a S3Exception with message "The request signature we calculated does not match the signature you provided. Check your key and signing method. (Service: S3, Status Code: 403..."

Expected Behavior

The documentation says Amazon S3 allows arbitrary Unicode characters in your metadata values. https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingMetadata.html
Then it goes on that you should conform to using US-ASCII, the error message says nothing with that regard (and Status Code: 403)
I would expect that the SDK would transparently encode or unencode (e.g. Unicode escape), but having the setting that is mentioned on this aws/aws-sdk-net#1587 would also be nice. Even if you leave it up to the client code it would be great to have more to the point error message.

Current Behavior

When sending the request S3Exception with message "The request signature we calculated does not match the signature you provided. Check your key and signing method..." is thrown

Possible Solution

Adapt docs, e.g. in Java the PutObjectRequestBuilder.metadata doesn't say a word about only US-ASCII, and / or guard clause to protect against non-ASCII data
Ideally a setting as mentioned in aws/aws-sdk-net#1587, as I understand you cannot change the default behaviour

Steps to Reproduce (for bugs)

@Test
void createSimpleItemWithUmlaut() {
     s3Client.putObject(PutObjectRequest.builder().bucket(TEST_BUCKET).key(System.currentTimeMillis() + "/" + "ä")
            .metadata(Map.of( "some-key", "/ä"))
            .build(), RequestBody.empty());
}

Context

Just trying to send also non-ASCII data

Your Environment

• AWS Java SDK version used: software.amazon.awssdk:s3:2.16.6
• JDK version used: openjdk 11.06
• Operating System and version: MacOS 11.5

[debora-ito]

Hi @cbromberg, we got the exact same ask on Java SDK v1 (aws/aws-sdk-java#2577), and when that issue came out I was going to reach out for the S3 team for guidelines on the subject, because (1) the documentation is not very clear on what is the expected behavior (as you showed), and (2) it's not possible to add non US-ASCII characters in the object metadata via AWS Console, it throws the same signature error - I think this one is an indication that it's not supported by S3.

I'll contact the S3 API team and ask for instructions. Depending on their response, if S3 expects the SDK to automatically encode the metadata input, this would be a cross-sdk feature.

[cbromberg]

Hi @debora-ito thanks for the feedback,
from your response I conclude that there is no setting as described in the aws/aws-sdk-net#1587 issue in Java SDK and as of now we would need to encode ourselves (which I don't mind).

[debora-ito]

That's right, there's no configuration setting for that in the Java SDK, you'd need to encode the input yourself.

[lon9man]

@debora-ito

we are trying to upload files into S3 with user metadata including non US-ASCII characters.

we upload from:

1. JS (using POST)
2. Android (using SDK)
3. iOS (using SDK)

IMPORTANT! each upload-way should make THE SAME DETERMINISTIC result of encoding, because after upload we parse S3-metadata using 1 worker and it should recognize/decode data correctly independently to which upload-way was used.

we see that JS-way works as expected without any encode on our side:
basic upload flow:

1. get file from user's browser
2. prepare metadata (including non US-ASCII). we don't encode anything here.

example of metadata from Chrome Developer Tools

acl: public-read
key: XXX/YYY/Тест альбома utf8/картинка 1 test.jpg.1631795880028
success_action_status: 201
x-amz-meta-key: XXX/YYY
x-amz-meta-user: 123
x-amz-meta-team: 456
x-amz-meta-album: Тест альбома utf8
X-Amz-Credential: ZZZ/20210916/us-east-1/s3/aws4_request
X-Amz-Algorithm: AWS4-HMAC-SHA256
X-Amz-Date: 20210916T123759Z
Policy: ...
X-Amz-Signature: ...
Content-Type: image/jpeg
x-amz-meta-filename: картинка 1 test.jpg
x-amz-meta-test: АБВГДЕ
x-amz-meta-fullpath: /Тест альбома utf8/картинка 1 test.jpg
x-amz-meta-filesize: 1503826
file: (binary)

3. make signature and etc
4. send POST-request to S3

in result upload is successful, on S3-console we got encoded string in the metadata using MIME encoding.
For example:
x-amz-meta-test --> =?UTF-8?B?w5DCkMOQwpHDkMKS?=

main result of JS-upload - IT WORKS WITHOUT ANY ENCODE of non US-ASCII characters before upload.

now we are trying to upload from Android/iOS using SDK and got signature-error, which is described in this issue or here aws/aws-sdk-java#2577.

questions:

1. why JS-way works as expected, BUT SDK-way don't? AFAIU both of them make POST-request to S3-endpoint and in theory it should work for both.
2. why Android/iOS SDK don't make things the same as it works in JS-way WITHOUT ANY additional encoding/modifying from our side, which you suggested to do in you'd need to encode the input yourself. do you/team suggest to make own custom code for EACH developer worldwide, which use your SDK to fix non US-ASCII issue?
3. let's assume: if anyway we will need to make our custom code.. we need to understand which way we should to encode it to make it the same to JS-way, which is already works correctly
4. am i missing something?

[gshondal]

Hi @lon9man, while the original topic is referencing a cross SDK feature, your specific issue should be created under https://github.com/aws-amplify/aws-sdk-android or https://github.com/aws-amplify/aws-sdk-ios.

Just a heads up, if this is a new project we highly recommend using the AWS Amplify Framework. AWS Amplify is a set of tools and services that can be used together or on their own, to help front-end web and mobile developers build scalable full stack applications, powered by AWS. With Amplify, you can configure app backends and connect your app in minutes, deploy static web apps in a few clicks, and easily manage app content outside the AWS console.

Amplify supports popular web frameworks including JavaScript, React, Angular, Vue, Next.js, and mobile platforms including Android, iOS, React Native, Ionic, Flutter. Get to market faster with AWS Amplify.

[debora-ito]

@lon9man yes, the Javascript SDK behavior diverged from the Java SDK v2's. We aim to avoid different SDKs to have different behavior, but in this case we missed something with the metadata encoding.

We can't change the Java SDK to start encoding automatically, this is a breaking change. Yes, the suggestion is to encode the metadata if you use the Java SDK v2.

Marking this to auto-close soon. Let us know if anyone has follow-up questions.