Unable to generate lambda from cross account s3 bucket

[ArendAMZN]

Hello, I'm trying to define the following setup in CDK

• s3Stack: defined in account A and it generates an s3 bucket
• lambdaStack: defined in account B and it generates a lambda using code from the bucket in s3Stack

I'm able to generate the s3Stack fine, but when I try to generate the lambdaStack it's unable to reference the s3 bucket cross account

Is this workflow possible in cdk?

Reproduction Steps

Code:

import lambda = require('@aws-cdk/aws-lambda');
import s3 = require('@aws-cdk/aws-s3');
import iam = require('@aws-cdk/aws-iam');
import kms = require('@aws-cdk/aws-kms');
import { RemovalPolicy, App, Stack, StackProps } from '@aws-cdk/core';
 
const app = new App();
 
const devEnv = {region: 'us-west-2', account: '363575151959'}
const toolEnv = {region: 'us-west-2', account: '776362166164'}
 
const s3Stack = new Stack(app, 's3Stack', { env: devEnv });
const s3Bucket = new s3.Bucket(s3Stack, 'bucket', {
  bucketName: 'castaren-s3-crossaccount-testing',
  removalPolicy: RemovalPolicy.DESTROY,
  encryption: s3.BucketEncryption.KMS,
  encryptionKey: new kms.Key(s3Stack, 'CrossAccountKmsKey',
 {
    removalPolicy: RemovalPolicy.DESTROY,
  }),
});
 
const s3Role = new iam.Role(s3Stack, 'CrossAccountRole', {
  roleName: 's3CrossAccountRoleName',
  assumedBy: new iam.AccountPrincipal(toolEnv['account']),
});
 
var s3CrossPolicy = new iam.PolicyStatement();
s3CrossPolicy.addAllResources();
s3CrossPolicy.addActions('s3:*');
s3Role.addToPolicy(s3CrossPolicy);
 
const lambdaStack = new Stack(app, 'lambdaStack', { env: toolEnv });
 
const lambdaFunction = new lambda.Function(lambdaStack, 'function ', {
  runtime: lambda.Runtime.PYTHON_3_7,
  handler: 'index.handler',
  code: lambda.Code.fromBucket(s3Bucket,'BETA'),
});
 
var funcCrossPolicy = new iam.PolicyStatement()
funcCrossPolicy.addResources(s3Role.roleArn);
funcCrossPolicy.addActions('sts:AssumeRole');
lambdaFunction.addToRolePolicy(funcCrossPolicy);

Commands

npm run build
cdk deploy s3Stack
cdk deploy lambdaStack

Error log below is from lambdaStack deployment

Error Log

00:06:56 castaren-> cdk deploy lambdaStack                                                                                                                                           
lambdaStack: deploying...
lambdaStack: creating CloudFormation changeset...
 0/5 | 4:44:15 PM | CREATE_IN_PROGRESS   | AWS::IAM::Role        | function /ServiceRole (functionServiceRoleD810D9B4) 
 0/5 | 4:44:15 PM | CREATE_IN_PROGRESS   | AWS::CDK::Metadata    | CDKMetadata 
 0/5 | 4:44:16 PM | CREATE_IN_PROGRESS   | AWS::IAM::Role        | function /ServiceRole (functionServiceRoleD810D9B4) Resource creation Initiated
 0/5 | 4:44:17 PM | CREATE_IN_PROGRESS   | AWS::CDK::Metadata    | CDKMetadata Resource creation Initiated
 1/5 | 4:44:17 PM | CREATE_COMPLETE      | AWS::CDK::Metadata    | CDKMetadata 
 2/5 | 4:44:34 PM | CREATE_COMPLETE      | AWS::IAM::Role        | function /ServiceRole (functionServiceRoleD810D9B4) 
 2/5 | 4:44:36 PM | CREATE_IN_PROGRESS   | AWS::IAM::Policy      | function /ServiceRole/DefaultPolicy (functionServiceRoleDefaultPolicy618511A6) 
 2/5 | 4:44:37 PM | CREATE_IN_PROGRESS   | AWS::IAM::Policy      | function /ServiceRole/DefaultPolicy (functionServiceRoleDefaultPolicy618511A6) Resource creation Initiated
 3/5 | 4:44:46 PM | CREATE_COMPLETE      | AWS::IAM::Policy      | function /ServiceRole/DefaultPolicy (functionServiceRoleDefaultPolicy618511A6) 
 3/5 | 4:44:48 PM | CREATE_IN_PROGRESS   | AWS::Lambda::Function | function  (function7F8023C1) 
 4/5 | 4:44:48 PM | CREATE_FAILED        | AWS::Lambda::Function | function  (function7F8023C1) Your access has been denied by S3, please make sure your request credentials have permission to GetObject for castaren-s3-crossaccount-testing/BETA. S3 Error Code: AccessDenied. S3 Error Message: Access Denied (Service: AWSLambdaInternal; Status Code: 403; Error Code: AccessDeniedException; Request ID: 059e22ba-ab44-44d0-9791-7ceca24faec1)
        new Function (/workspace/castaren/SearchResourceManager_PoC/node_modules/@aws-cdk/aws-lambda/lib/function.ts:436:35)
        \_ Object.<anonymous> (/workspace/castaren/SearchResourceManager_PoC/bin/search_resource_manager_poc.ts:104:24)
        \_ Module._compile (internal/modules/cjs/loader.js:778:30)
        \_ Module.m._compile (/workspace/castaren/SearchResourceManager_PoC/node_modules/ts-node/src/index.ts:493:23)
        \_ Module._extensions..js (internal/modules/cjs/loader.js:789:10)
        \_ Object.require.extensions.(anonymous function) [as .ts] (/workspace/castaren/SearchResourceManager_PoC/node_modules/ts-node/src/index.ts:496:12)
        \_ Module.load (internal/modules/cjs/loader.js:653:32)
        \_ tryModuleLoad (internal/modules/cjs/loader.js:593:12)
        \_ Function.Module._load (internal/modules/cjs/loader.js:585:3)
        \_ Function.Module.runMain (internal/modules/cjs/loader.js:831:12)
        \_ Object.<anonymous> (/workspace/castaren/SearchResourceManager_PoC/node_modules/ts-node/src/bin.ts:158:12)
        \_ Module._compile (internal/modules/cjs/loader.js:778:30)
        \_ Object.Module._extensions..js (internal/modules/cjs/loader.js:789:10)
        \_ Module.load (internal/modules/cjs/loader.js:653:32)
        \_ tryModuleLoad (internal/modules/cjs/loader.js:593:12)
        \_ Function.Module._load (internal/modules/cjs/loader.js:585:3)
        \_ Function.Module.runMain (internal/modules/cjs/loader.js:831:12)
        \_ findNodeScript.then.existing (/usr/lib/node_modules/npm/node_modules/libnpx/index.js:268:14)
 4/5 | 4:44:49 PM | ROLLBACK_IN_PROGRESS | AWS::CloudFormation::Stack | lambdaStack The following resource(s) failed to create: [function7F8023C1]. . Rollback requested by user.
 4/5 | 4:45:03 PM | DELETE_IN_PROGRESS   | AWS::CDK::Metadata    | CDKMetadata 
 5/5 | 4:45:03 PM | DELETE_COMPLETE      | AWS::Lambda::Function | function  (function7F8023C1) 
 5/5 | 4:45:04 PM | DELETE_IN_PROGRESS   | AWS::IAM::Policy      | function /ServiceRole/DefaultPolicy (functionServiceRoleDefaultPolicy618511A6) 
 6/5 | 4:45:05 PM | DELETE_COMPLETE      | AWS::CDK::Metadata    | CDKMetadata 
 7/5 | 4:45:05 PM | DELETE_COMPLETE      | AWS::IAM::Policy      | function /ServiceRole/DefaultPolicy (functionServiceRoleDefaultPolicy618511A6) 
 7/5 | 4:45:06 PM | DELETE_IN_PROGRESS   | AWS::IAM::Role        | function /ServiceRole (functionServiceRoleD810D9B4) 
 
 ❌  lambdaStack failed: Error: The stack named lambdaStack failed creation, it may need to be manually deleted from the AWS console: ROLLBACK_COMPLETE
 8/5 | 4:45:07 PM | DELETE_COMPLETE      | AWS::IAM::Role        | function /ServiceRole (functionServiceRoleD810D9B4) 
 9/5 | 4:45:08 PM | ROLLBACK_COMPLETE    | AWS::CloudFormation::Stack | lambdaStack 
The stack named lambdaStack failed creation, it may need to be manually deleted from the AWS console: ROLLBACK_COMPLETE

Environment

• CLI Version : cdk --version = 1.13.1 (build 96cfc63), cat pacakge.json = all 1.13.1
• Framework Version: ?
• OS : Amazon Linux 2 (internal)
• Language : typescript

[eladb]

@ArendAMZN is this still an issue?

[ArendAMZN]

@iliapolo This is still an issue

I pushed this to the side to focus on other tasks (even though I still need this to work eventually), but I hit it again with today's task

 13/34 | 5:22:20 PM | CREATE_FAILED        | AWS::Lambda::Function   | CommentingFunction (CommentingFunction5C76628F) Your access has been denied by S3, please make sure your reque
st credentials have permission to GetObject for keacodebuildprevents-source-code-us-west-2-326471334405/dev. S3 Error Code: AccessDenied. S3 Error Message: Access Denied (Service: A
WSLambdaInternal; Status Code: 403; Error Code: AccessDeniedException; Request ID: 9fc6a3c6-7832-4186-94a0-562957ba0cbd)

[ArendAMZN]

I'm wondering if lambdaFunction.role is actually the thing that needs access to the bucket? Since it's accessing the bucket to create the lambda (rather than execute it) maybe it's using a different role?

[iliapolo]

Acking this issue.

@ArendAMZN Apologies for the long delay, will get to it this week promptly.

[iliapolo]

Hi @ArendAMZN

Im trying to figure out the correct way to configure this. Though honestly, i'm not sure this is even possible. Lambda requires the source code bucket to be in the same region as the lambda, it doesn't mention anything about requiring the same account, but that doesn't seem like a stretch to assume.

While I investigate this further, I do have a few observations.

The following code you posted:

var funcCrossPolicy = new iam.PolicyStatement()
funcCrossPolicy.addResources(s3Role.roleArn);
funcCrossPolicy.addActions('sts:AssumeRole');
lambdaFunction.addToRolePolicy(funcCrossPolicy);

Doesn't help because the lambda isn't actively attempting to assume any role, so giving it permissions to do so is void.

The entity that needs access to the bucket in the different account is actually the lambda service itself, not any specific function. I would try using iam.ServicePrincipal to grant the lambda.amazonaws.com service the required permissions.

Unfortunately I don't have a full solution for you just yet, but i wanted to give my input so maybe we can figure this out together :)

Also, any chance you elaborate on the use-case for attempting this? Doing cross-account references might not be the best road to follow. Perhaps we can think of a better way to achieve what you need.