-
Notifications
You must be signed in to change notification settings - Fork 4.3k
feat(sagemaker-alpha): add Pipeline import methods for existing SageMaker Pipelines #35230
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
…ageMaker Pipelines Add Pipeline class with import methods to enable importing existing SageMaker Pipelines into CDK applications for use with EventBridge Scheduler and other AWS services. - Add Pipeline.fromPipelineArn() for importing by ARN (cross-account/region support) - Add Pipeline.fromPipelineName() for importing by name (same account/region) - Add Pipeline.fromPipelineAttributes() for advanced import scenarios - Add AWS-compliant pipeline name validation (1-256 chars, alphanumeric with hyphens) - Add grantStartPipelineExecution() IAM method for proper permission management - Add full EventBridge Scheduler integration with SageMakerStartPipelineExecution target - Add comprehensive unit tests (17 tests) and integration test - Add complete documentation with usage examples and EventBridge integration patterns This enables separation of concerns where data scientists manage ML pipelines using SageMaker tools while infrastructure teams can schedule and orchestrate them via CDK. Closes aws#35220
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
(This review is outdated)
✅ Updated pull request passes all PRLinter validations. Dismissing previous PRLinter review.
| @@ -0,0 +1,189 @@ | |||
| import { Construct } from 'constructs'; | |||
| import { Grant, IGrantable } from 'aws-cdk-lib/aws-iam'; | |||
| import { IPipeline } from 'aws-cdk-lib/aws-sagemaker'; | |||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not sure if we should duplicate the IPipeline in the alpha module. Tentatively import it from stable module for now. Please advise if we should duplicate instead.
|
@pahud what's stopping this PR to be reviewed and merged? I don't understand the processes here. If this isn't merged and released soon, I'll have to find a workaround. |
|
@lorenzwalthert this PR has been in the reviewing backlog of the team. I'll bring this up to the team for inputs here. Thank you for the callout. |
Issue # (if applicable)
Closes #35220.
Reason for this change
Users need the ability to import existing SageMaker Pipelines into their CDK applications to integrate with other AWS services like EventBridge Scheduler. This addresses a common use case where data scientists develop and manage ML pipelines outside of CDK, but infrastructure teams need to schedule and orchestrate these pipelines as part of larger workflows.
The current
@aws-cdk/aws-sagemaker-alphapackage had anIPipelineinterface but no concrete implementation with import methods, making it impossible to reference existing pipelines in CDK applications.Description of changes
This change adds import-only Pipeline functionality to the
@aws-cdk/aws-sagemaker-alphapackage. Note: This construct cannot create new SageMaker Pipelines - it only imports existing ones for infrastructure integration purposes.Pipelineclass implementing theIPipelineinterface fromaws-cdk-lib/aws-sagemakerwith static import methodsPipeline.fromPipelineArn()- Import pipeline by ARN (supports cross-account/region)Pipeline.fromPipelineName()- Import pipeline by name (same account/region as stack)Pipeline.fromPipelineAttributes()- Import with custom attributes for advanced scenariosgrantStartPipelineExecution()method for proper permission managementSageMakerStartPipelineExecutiontarget by implementing the standardIPipelineinterfacevalidatePipelineName()function is exported for reuse by other constructs and user validationThe implementation follows established CDK patterns used by other AWS services (S3, Lambda, IAM) for consistency and familiarity.
Key design decisions:
IPipelineinterface from the stableaws-cdk-lib/aws-sagemakerpackage instead of duplicating it in the alpha package. This ensures compatibility with existing CDK services (like EventBridge Scheduler targets) and prevents interface conflicts when both packages are used together.validatePipelineName()function is also exported as a utility for other constructs and user validation.Describe any new or updated permissions being added
IAM Permissions: The
grantStartPipelineExecution()method grants thesagemaker:StartPipelineExecutionpermission on the specific pipeline resource. This is the minimal permission required for EventBridge Scheduler and other services to trigger pipeline executions.No additional resource access patterns are introduced - this follows the standard CDK grant pattern used throughout the framework.
Description of how you validated changes
Unit Tests: 20 comprehensive unit tests covering all functionality:
fromPipelineArn,fromPipelineName,fromPipelineAttributes)Integration Test: Real-world deployment test (
integ.pipeline-import.ts) that:fromPipelineArnandfromPipelineName)Checklist
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license