Convert a serialization config JSON to a OCI runtime configuration.

[philips]

From @s-urbaniak on May 25, 2016 1:44

As per opencontainers/image-spec#82 (comment): Currently, the oci-image-tool supports a simple unpacking mechanism to inspect deflated images and validation.

More concretely the config/image JSON https://github.com/opencontainers/image-spec/blob/master/serialization.md#image-json-description has to be converted to a https://github.com/opencontainers/runtime-spec/blob/master/config.md#container-configuration-file.

Copied from original issue: opencontainers/image-spec#87

[philips]

From @wking on May 25, 2016 3:17

On Tue, May 24, 2016 at 06:44:43PM -0700, Sergiusz Urbaniak wrote:

More concretely the config/image JSON
https://github.com/opencontainers/image-spec/blob/master/serialization.md#image-json-description
has to be converted to a
https://github.com/opencontainers/runtime-spec/blob/master/config.md#container-configuration-file.

I don't think this is possible. For example, the image-spec schema
currently has ‘ExposedPorts’, but networking is out-of-scope for the
runtime config 1. You'd have to select a particular network manager
(that isn't covered by an OCI spec) and setup hooks to drive that
network manager.

Stepping back, I see no reason for the image-spec repo to care about
what is in the runtime config (see also @stevvooe's “decouple”
wording in 2). I think imaging tools need to know “this is a
directory that I'm (de)serializing”. Then the image spec can add on
distribution-oriented layers like “distribute that directory using
layered tarballs” or “sign name ↔ directory associations”. But baking
knowledge of the filesystem directory (e.g. there's a ‘config.json’
specified by runtime-spec, there may be a ‘rootfs’ directory
referenced from ‘root.path’ in that config.json, …) into the image
spec seems like it's breaking a perfectly good abstraction boundary
for no purpose.

If the “we don't care what's inside the directory” approach from my
last paragraph gains traction, there will still be a need for some
conversion logic for backwards compatibility with existing images.
But I think we want the image spec to get out of that business (and
let the runtimes handle it).

 Subject: Do we have any plan for network and storage specs?
 Date: Fri, 14 Aug 2015 02:36:15 -0700 (PDT)
 Message-Id: <c056925e-6ecf-4457-8c16-7ab34d279604@opencontainers.org>

[philips]

@wking exposed ports is useful metadata from the image. I think it needs to exist to retain useful UX from builder to runner.

I am really really unclear on what you want to see here.

Furthermore, retaining the UX of container runtimes is a desirable part of the spec and explicitly called out in the README and image format spec project proposal:

docker run example.com/org/app:v1.0.0
rkt run example.com/org/app,version=v1.0.0

[philips]

From @wking on May 25, 2016 15:44

On Wed, May 25, 2016 at 08:22:16AM -0700, Brandon Philips wrote:

@wking exposed ports is useful metadata from the image. I think it
needs to exist to retain useful UX from builder to runner.

Then I think you want to file a PR against runtime-spec (or a
networking OCI spec?) to get it supported. I don't think image-spec
is a good place to specify container networking.

I am really really unclear on what you want to see here.

As I understand it, #82 handles unpacking a local directory from layer
tarballs. I want to see image-spec drop
application/vnd.oci.image.serialization.config.v1+json, and have
runtime-spec claim a MIME type for config.json. Then
application/vnd.oci.image.manifest.v1+json is just a list of layers
(and optional annotations). Then runtime-spec can wander around
however it likes and image-spec would still be able to distribute the
new configs without changes. And folks could add alternative runtime
configs, and image-spec could distribute them without changes. And I
could package up a non-container package (e.g. a Gentoo stage3) and
use image-spec tooling to distribute that too (although I obviously
wouldn't want to point runC at the config.json-less unpacked
directory).

Teaching image-spec about what's inside the distributed directory
pokes holes in the runtime's abstraction, and defining a local config
that has to be translated to a runtime-spec config pokes big holes in
the abstraction. Sometimes that sort of abstraction-breaking is
useful, but in this case I don't see an upside, and I see flexibility
and maintenance downsides.

[philips]

@wking the runtime can IGNORE this information and it can be used by a higher level system. I will file an issue about how to convert it but I don't think dropping it makes sense.

[philips]

@wking I disagree with your suggestion of the split. The image format needs a way of expressing runtime defaults that aren't host specific which the runtime spec has not tackled and made difficult by recombining everything into a single config.json.

[philips]

From @wking on May 25, 2016 20:21

On Wed, May 25, 2016 at 09:11:49AM -0700, Brandon Philips wrote:

the runtime can IGNORE this information and it can be used by a
higher level system.

And I'm fine with that, although I think we want to be careful about
how external tools like network managers are included [1]. I just
don't think the image-spec should be in the business of orchestrating
this. You can have:

• Orchestration layer
  • Call image-spec tooling to fetch a generic directory by name
  • Call network-setup tooling to convert a config file's ExposedPorts
    entries to libnetwork hooks (or some such).
  • Call ocitools to translate fromContainer entries in the config.
  • … Call other tools to localize the config …
  • Call runtime-spec tooling (e.g. runC) to launch the adapted config.

I think that such a narrowly-scoped image-spec would be easier to
maintain and use than a broadly-scoped image-spec like:

• Orchestration layer
  • image-spec
    • Call image-spec tooling to fetch a generic directory by name
    • Call network-setup tooling to convert a config file's
      ExposedPorts entries to libnetwork hooks (or some such).
    • Call ocitools to translate fromContainer entries in the config.
    • … Call other tools to localize the config …
  • Call runtime-spec tooling (e.g. runC) to launch the adapted config.

With the narrowly-scoped image-spec, you can add as many localization
steps to your orchestration layer as you like, and don't have to file
image-spec PRs for “please have ocitools translate fromContainer”,
etc.

On Wed, May 25, 2016 at 09:13:45AM -0700, Brandon Philips wrote:

The image format needs a way of expressing runtime defaults that
aren't host specific which the runtime spec has not tackled and made
difficult by recombining everything into a single config.json.

The orchestration layer needs to do that sort of thing, but I don't
see how it benefits from being tied into image-spec (which is about
efficiently and securely shipping directories between hosts).

[philips]

I think what we should do is just have an annotation on the runtime spec config that lists the exposed ports. Something like:

{
    "ociVersion": "0.2.0",
...
    "annotations": {
        "opencontainers.org/image-spec/exposedPorts": "8080,53/udp"
    }
}

Then that can be consumed by whatever tool is setting up the networking.

[philips]

From @wking on June 7, 2016 16:57

On Mon, Jun 06, 2016 at 10:50:41PM -0700, Brandon Philips wrote:

I think what we should do is just have an annotation on the runtime
spec config…

This approach doesn't work as well for more complex values like the
‘history’ structure 1. I'd rather drop the hashmap restriction on
‘annotations’ 2 and embed the whole serialization config there, or
copy the full serialization config to separate file in the unpacked
bundle. I'm not entirely clear on what the ‘history’ field is for (UI
hints?), but having it and a record of DiffIDs in the unpacked bundle
may make it easier for tools to build a new image that recycles a
parent's images layer tarballs. You'll probably want a copy of the
manifest too, so you can use the same digests as the parent image 3
without having to maintain an external DiffID → digest lookup service.

[philips]

To what end do we add stuff to the runtime config.json though? We could simply say that the original manifest sits inside the bundle instead.

The reason this annotation is needed in the container runtime is because presumably an OCI network plugin would need to read this to setup forwarding firewalls or something.

cc @opencontainers/runtime-spec-maintainers to discuss. @mrunalp lets put this on the agenda tomorrow on how these things interoperate.

[philips]

From @wking on June 7, 2016 23:45

On Tue, Jun 07, 2016 at 04:40:25PM -0700, Brandon Philips wrote:

To what end do we add stuff to the runtime config.json though? We
could simply say that the original manifest sits inside the bundle
instead.

“copy the full serialization config to separate file” was one of my
options 1.

The reason this annotation is needed in the container runtime is
because presumably an OCI network plugin would need to read this to
setup forwarding firewalls or something.

And they can access that information in the bundle directory without
us needing to put it in the annotations field 2.

[philips]

@wking fair point. I think the runtime spec maintainers should chime in on this point of external files vs serialized metadata into the config.json.

[philips]

From @wking on June 8, 2016 4:17

On Tue, Jun 07, 2016 at 04:50:56PM -0700, Brandon Philips wrote:

I think the runtime spec maintainers should chime in on this point
of external files vs serialized metadata into the config.json.

I've filed opencontainers/runtime-spec#492 to see if they have any
generic opinions on this sort of thing.