Buildpacks support Auto sync

Signed-off-by: David Gageot <[email protected]>

Author: dgageot

docs/content/en/docs/pipeline-stages/filesync.md

@@ -9,9 +9,42 @@ aliases: [/docs/how-tos/filesync]
 Skaffold supports copying changed files to a deployed container so as to avoid the need to rebuild, redeploy, and restart the corresponding pod.
 The file copying is enabled by adding a `sync` section with _sync rules_ to the `artifact` in the `skaffold.yaml`.
 Under the hood, Skaffold creates a tar file with changed files that match the sync rules.
-This tar file is sent to and extracted on the corresponding containers. 
+This tar file is sent to and extracted on the corresponding containers.
+
+Multiple types of sync are supported by Skaffold:
+
+ + `manual`: The user must specify both the files in their local workspace and the destination in the running container.
+   This is supported by every type of artifact.
+
+ + `infer`: The destinations for each changed file is inferred from instructions in a Dockerfile.
+   This is supported by docker and kaniko artifacts and also for custom artifacts that declare a
+   dependency on a Dockerfile.
+
+ + `auto`: Skaffold automatically configures the sync.  This is only supported by Jib and Buildpacks artifacts.
+
+### Manual sync mode
+
+A manual sync rule must specify the `src` and `dest` field.
+The `src` field is a glob pattern to match files relative to the artifact _context_ directory, which may contain `**` to match nested files.
+The `dest` field is the absolute or relative destination path in the container.
+If the destination is a relative path, an absolute path will be inferred by prepending the path with the container's `WORKDIR`.
+By default, matched files are transplanted with their whole directory hierarchy below the artifact context directory onto the destination.
+The optional `strip` field can cut off some levels from the directory hierarchy.
+The following example showcases manual filesync:
+
+{{% readfile file="samples/filesync/filesync-manual.yaml" %}}
+
+- The first rule synchronizes the file `.filebaserc` to the `/etc` folder in the container.
+- The second rule synchronizes all `html` files in the `static-html` folder into the `<WORKDIR>/static` folder in the container.
+  Note that this pattern does not match files in sub-folders below `static-html` (e.g. `static-html/a.html` but not `static-html/sub/a.html`).
+- The third rule synchronizes all `png` files from any sub-folder into the `assets` folder on the container.
+  For example, `img.png` ↷ `assets/img.png` or `sub/img.png` ↷ `assets/sub/img.png`.
+- The last rule synchronizes all `md` files below the `content/en` directory into the `content` folder on the container.
+  The `strip` directive ensures that only the directory hierarchy below `content/en` is re-created at the destination.
+  For example, `content/en/index.md` ↷ `content/index.md` or `content/en/sub/index.md` ↷ `content/sub/index.md`.
 
 ### Inferred sync mode
+
 For docker artifacts, Skaffold knows how to infer the desired destination from the artifact's `Dockerfile`.
 To enable syncing, you only need to specify which files are eligible for syncing in the sync rules.
 The sync rules for inferred sync mode is just a list of glob patterns.
@@ -42,26 +75,21 @@ And a `skaffold.yaml` with the following sync configuration:
 Inferred sync mode only applies to modified and added files.
 File deletion will always cause a complete rebuild.
 
-### Manual sync mode
+### Auto sync mode (Buildpacks)
 
-A manual sync rule must specify the `src` and `dest` field.
-The `src` field is a glob pattern to match files relative to the artifact _context_ directory, which may contain `**` to match nested files.
-The `dest` field is the absolute or relative destination path in the container.
-If the destination is a relative path, an absolute path will be inferred by prepending the path with the container's `WORKDIR`.
-By default, matched files are transplanted with their whole directory hierarchy below the artifact context directory onto the destination.
-The optional `strip` field can cut off some levels from the directory hierarchy.
-The following example showcases manual filesync:
+Skaffold requires special collaboration from the Buildpacks for the `auto` sync to work.
+The [gcr.io/buildpacks/builder](https://github.com/GoogleCloudPlatform/buildpacks) supports Skaffold
+out of the box, currently for Go and NodeJS.
 
-{{% readfile file="samples/filesync/filesync-manual.yaml" %}}
+Cloud Native Buildpacks set a `io.buildpacks.build.metadata` label on the images they create.
+This labels points to json description of the [Bill-of-Materials, aka BOM](https://github.com/buildpacks/spec/blob/master/buildpack.md#bill-of-materials-toml) of the build.
+In the BOM, under the `metadata.devmode.sync` key, Buildpacks that want to collaborate with Skaffold
+have to output the sync rules based on their exploration of the source and the build process they had to apply to it.
+Those sync rules will then be used by Skaffold without the user having to configure them manually.
 
-- The first rule synchronizes the file `.filebaserc` to the `/etc` folder in the container.
-- The second rule synchronizes all `html` files in the `static-html` folder into the `<WORKDIR>/static` folder in the container.
-  Note that this pattern does not match files in sub-folders below `static-html` (e.g. `static-html/a.html` but not `static-html/sub/a.html`).
-- The third rule synchronizes all `png` files from any sub-folder into the `assets` folder on the container.
-  For example, `img.png` ↷ `assets/img.png` or `sub/img.png` ↷ `assets/sub/img.png`.
-- The last rule synchronizes all `md` files below the `content/en` directory into the `content` folder on the container.
-  The `strip` directive ensures that only the directory hierarchy below `content/en` is re-created at the destination.
-  For example, `content/en/index.md` ↷ `content/index.md` or `content/en/sub/index.md` ↷ `content/sub/index.md`.
+Another thing the Buildpacks have to do is support the `GOOGLE_DEVMODE` environment variable. Skaffold will
+set it to `1` when running `skaffold dev` with sync configured to `auto: {}`. The Buildpacks can then use that
+signal to change the way the application is built so that it reloads the changes or rebuilds the app on each change.
 
 ## Limitations
 

docs/content/en/schemas/v2beta3.json

@@ -1947,16 +1947,16 @@
       "properties": {
         "auto": {
           "$ref": "#/definitions/Auto",
-          "description": "delegates discovery of sync rules to the build system. Currently only available for jib.",
-          "x-intellij-html-description": "delegates discovery of sync rules to the build system. Currently only available for jib."
+          "description": "delegates discovery of sync rules to the build system. Only available for jib and buildpacks.",
+          "x-intellij-html-description": "delegates discovery of sync rules to the build system. Only available for jib and buildpacks."
         },
         "infer": {
           "items": {
             "type": "string"
           },
           "type": "array",
-          "description": "file patterns which may be synced into the container. The container destination is inferred by the builder. Currently only available for docker artifacts.",
-          "x-intellij-html-description": "file patterns which may be synced into the container. The container destination is inferred by the builder. Currently only available for docker artifacts.",
+          "description": "file patterns which may be synced into the container The container destination is inferred by the builder based on the instructions of a Dockerfile. Available for docker and kaniko artifacts and custom artifacts that declare dependencies on a dockerfile.",
+          "x-intellij-html-description": "file patterns which may be synced into the container The container destination is inferred by the builder based on the instructions of a Dockerfile. Available for docker and kaniko artifacts and custom artifacts that declare dependencies on a dockerfile.",
           "default": "[]"
         },
         "manual": {

examples/buildpacks-node/skaffold.yaml

@@ -5,3 +5,5 @@ build:
   - image: skaffold-buildpacks-node
     buildpack:
       builder: "gcr.io/buildpacks/builder"
+    sync:
+      auto: {}

examples/buildpacks/skaffold.yaml

@@ -6,8 +6,9 @@ build:
     buildpack:
       builder: "gcr.io/buildpacks/builder"
       env:
-      - GOVERSION=1.13.1
       - GOPROXY={{.GOPROXY}}
+    sync:
+      auto: {}
 profiles:
 - name: gcb
   build:

integration/examples/buildpacks-node/skaffold.yaml

@@ -5,3 +5,5 @@ build:
   - image: skaffold-buildpacks-node
     buildpack:
       builder: "gcr.io/buildpacks/builder"
+    sync:
+      auto: {}

integration/examples/buildpacks/skaffold.yaml

@@ -7,6 +7,8 @@ build:
       builder: "gcr.io/buildpacks/builder"
       env:
       - GOPROXY={{.GOPROXY}}
+    sync:
+      auto: {}
 profiles:
 - name: gcb
   build:

pkg/skaffold/build/buildpacks/build_test.go

@@ -78,7 +78,7 @@ func TestBuild(t *testing.T) {
 		},
 		{
 			description: "dev mode",
-			artifact:    withSync(&latest.Sync{Infer: []string{"**/*"}}, buildpacksArtifact("another/builder", "another/run")),
+			artifact:    withSync(&latest.Sync{Auto: &latest.Auto{}}, buildpacksArtifact("another/builder", "another/run")),
 			tag:         "img:tag",
 			api:         &testutil.FakeAPIClient{},
 			devMode:     true,

pkg/skaffold/build/buildpacks/lifecycle.go

@@ -59,7 +59,7 @@ func (b *Builder) build(ctx context.Context, out io.Writer, a *latest.Artifact,
 		return "", fmt.Errorf("unable to evaluate env variables: %w", err)
 	}
 
-	if b.devMode && a.Sync != nil && len(a.Sync.Infer) > 0 {
+	if b.devMode && a.Sync != nil && a.Sync.Auto != nil {
 		env = append(env, "GOOGLE_DEVMODE=1")
 	}
 

pkg/skaffold/build/cache/hash.go

@@ -106,7 +106,7 @@ func artifactConfig(a *latest.Artifact, devMode bool) (string, error) {
 		return "", fmt.Errorf("marshalling the artifact's configuration for %q: %w", a.ImageName, err)
 	}
 
-	if devMode && a.BuildpackArtifact != nil && a.Sync != nil && len(a.Sync.Infer) > 0 {
+	if devMode && a.BuildpackArtifact != nil && a.Sync != nil && a.Sync.Auto != nil {
 		return string(buf) + ".DEV", nil
 	}
 

pkg/skaffold/build/cache/hash_test.go

@@ -184,7 +184,7 @@ func TestArtifactConfigDevMode(t *testing.T) {
 			},
 		}
 		sync := &latest.Sync{
-			Infer: []string{"**/*"},
+			Auto: &latest.Auto{},
 		}
 
 		config, err := artifactConfig(&latest.Artifact{