Skip to content

v0.2.0

Pre-release
Pre-release

Choose a tag to compare

View all tags
@JohnChe88 JohnChe88 released this 28 Feb 20:27
· 47 commits to develop since this release
v0.2.0
ecb27f3
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Verified
Learn about vigilant mode.

Release v0.2.0 introduces several new features and improvements, including:

  • Architecture to submit the PySpark script from Amazon S3 on AWS Lambda using Spark on Docker. This feature enables users to easily run PySpark jobs on AWS Lambda and impact less when pyspark code requires update.

  • SAM (Serverless Application Model) templates to automatically build and deploy Docker images to AWS ECR (Elastic Container Registry) and AWS Lambda. This feature makes it easy to deploy and manage Docker images on AWS Lambda using SAM templates.

  • Apache Hudi integration with Spark on AWS Lambda. This feature enables users to use Apache Hudi, a storage system for managing small to medium (up to 200MB payload) and complex data sets on Amazon S3.

These features enhance the usability and scalability of Spark on AWS Lambda, providing users with more flexibility and options for running PySpark jobs on AWS Lambda.

Assets 2
Loading