RFC: Handling of Alternative Multibases

[kevina]

As we switch to using CidV1 base32 encoding the question is how we should handle this displaying of CID in alternative multibases.

This issue is only about the output. For the input will we already handle CIDv1 in any base supported by go-multibase.

Here are the alternatives, as I see it, and what I estimate it will take to do it.

1. Don't Preserve the multibase, Use Settings or Defaults to determine multibase

Accept CidV1 in any multibase, but ignore the base that is used. Output the CIDs based on either a global --cid-base flag or a config value.

This means that the string a user passes in may not be the same string that is returned. For example

$ ipfs pin add zb2rhfwrxLo7wm3ZT2ishCdimPKWdRCe3KuxHFQYMR8Kyj96x
pinned bafkreiekhi5qqy64ovh7jckh5ewnyt6lcvfzqfubv3l66cazo5dknu6dre recursively

This also means that #5234 (ipfs resolve should respect CID base) will not be implemented.

The most straightforward way to implement this is replacing every call of cid.String() with something like cid.Encode(base) (see ipfs/go-cid#60) and make sure the base is available at the point when this method needs to be called. The WithValue functionally on contexts can be helpful here.

As most CID's are converted to strings on the server side, rather than the client, doing a quick hack such as setting a global variable is not going to work.

Another way to implement this, at least when working with the API, is to modify our API to make CID's a special type and then set the multibase as part of the writer that formats the output for this user. This means extra CPU cycles as the CID is converted from a binary represenation to a string twice. If we go this route I recommend the API be hardcoded to use base32 as is a more efficient encoding than our current default of base58btc.

2. Preserve the multibase. Children of DAGs inherit multi-base of parent when known.

When a CID is decoded preserve the multibase used with the CID so when it is converted back to a string the same multibase will be used. In addition associate the CID of any children of the DAG with the same multibase prefix. When a CID does not have a multibase associated use the settings from the command line or config option.

ipfs pin add will then display the same base. In addition commands like ipfs ls will use the same base to display directory entries as the hash used for the parent.

$ ipfs ls f017012205be9e545b52def2d18a922146c6cdec6bcbf4007b6ee7bec3341673bbf1d8c06
f01551220f6ba0f2491f2371dee0ea66e8812331b78f162697d2f41b5d55b667e5ae6bc67 3606  diff.go
f01551220e43f16413e6e1aa73eee0194d40363b7082d06faae8d6e94f18bd5f8cc08dfee 18299 object.go
f015512201985aa3462ff163c46de0f5e6634bde41f2bd537780ccbe5401aea1cccc49577 9057  patch.go

Implementing this will involve associating a multibase with CID's when it is known (see ipfs/go-cid#66). It will also involve setting the multibase of children CID's in the DAG code. New CID's will also need to get the base set somewhere.

After thinking about this I think this will involve less code change than #1 as the base does not need to be set everywhere, although the changes may not be as straightforward.

I think this is the most user friendly approach.

3. Attempt to Preserve the multibase.

This solution does not store the multi-base with the CID but makes an attempt to output the same multibase on a ad-hoc bases. For the current implementation of ipfs resolve it turned out to be easy. ipfs pin add is more difficult. It was in fact after looking at ipfs pin add that I decided that #2 will be required to do it properly, especially when multiple CIDs are given on the command line.

Implementing this involves doing #1 and in addition detecting the multibase of any CIDs used and setting and using the correct base for the output. Things can get tricky when CID's with different mutlibases are used at once. This will involve more code that either other solution.

#5289 started effort towards this solution (see the code for ipfs resolve and ipfs ls)

I do not like this solution as it will only work some of the time and prefer we go with #1 or #2 for more predictable output.

[kevina]

New CID's will also need to get the base set somewhere.

If we allow CIDs to carry a multibase, setting the base should be fairly easy to do with the new Builder interface (see ipfs/go-cid#53) and would require very little code change.

[Stebalien]

2. Preserve the multibase. Children of DAGs inherit multi-base of parent when known.

I'm not actually sure this is user-friendly. If I run:

ipfs pin f017012205be9e545b52def2d18a922146c6cdec6bcbf4007b6ee7bec3341673bbf1d8c06 zb2rhfwrxLo7wm3ZT2ishCdimPKWdRCe3KuxHFQYMR8Kyj96x

And get back a bunch of different CIDs in different bases, that's going to be a bit confusing.

3. Attempt to Preserve the multibase.

I agree with your analysis here. Predictability is key.

---

Personally, I'd go with 1 and add a global flag (possibly with an environment variable as a fallback?).

As most CID's are converted to strings on the server side, rather than the client, doing a quick hack such as setting a global variable is not going to work.

Damn... We should be converting them back and forth (well, we should be using a binary RPC transport). Oh well.

Regardless, we can use an environment variable, we just have to forward it as an argument.

We could also start converting them back. That is, just replace every "string that is actually a CID" with a CID (and make sure that go can unmarshal everything correctly). That would actually be better from a backwards compat standpoint. How does that sound?

---

We could also mix in a bit of 3 but I'm wary about that. It's easy for commands that take one argument and spit out one argument but not something we can do reliably for commands that take many arguments.

[Stebalien]

Regardless of what we do, I really don't want to carry display information with CIDs themselves.

[kevina]

I'm not actually sure this is user-friendly. If I run:
ipfs pin f017012205be9e545b52def2d18a922146c6cdec6bcbf4007b6ee7bec3341673bbf1d8c06 zb2rhfwrxLo7wm3ZT2ishCdimPKWdRCe3KuxHFQYMR8Kyj96x
And get back a bunch of different CIDs in different bases, that's going to be a bit confusing.

We will have to agree to disagree. I don't think it will be confusing at all if we return the same string. I.e.

$ ipfs pin add f017012205be9e545b52def2d18a922146c6cdec6bcbf4007b6ee7bec3341673bbf1d8c06 zb2rhfwrxLo7wm3ZT2ishCdimPKWdRCe3KuxHFQYMR8Kyj96x
pinned f017012205be9e545b52def2d18a922146c6cdec6bcbf4007b6ee7bec3341673bbf1d8c06 recursively
pinned zb2rhfwrxLo7wm3ZT2ishCdimPKWdRCe3KuxHFQYMR8Kyj96x recursively

[kevina]

That is, just replace every "string that is actually a CID" with a CID (and make sure that go can unmarshal everything correctly). That would actually be better from a backwards compat standpoint. How does that sound?

This will be an API change. As {"cid": "<cid-string>"} will be replaced with {"cid": {"/": "<cid-string>"}} based on how the UnmarshalJSON and MarshalJSON functions are written: https://github.com/ipfs/go-cid/blob/master/cid.go#L387

[kevina]

@lidel @lgierth I would also appreciate some feedback on this from a users perspective.

[lidel]

@kevina If our endgame is to move to base32 as the universal default, users should not need to think about getting "the right" encoding unless they explicitly choose to do so.

If we look at this from that perspective, option 1 is not a problem, it is a feature. It may be actually a good UX idea to "normalize" to base32 by default, enforcing its universal use. If user needs to interact or get different encoding then config or CLI --cid-base ah-hoc flag should be enough for those rare cases.

That being said, option 2 is also fine (but not ideal, as users may need to always pass --cid-base base32 to normalize all outputs to base32).

[Stebalien]

We will have to agree to disagree. I don't think it will be confusing at all if we return the same string. I.e.

In that case, I agree. I was thinking about the refs -r case.

This will be an API change. As {"cid": ""} will be replaced with {"cid": {"/": ""}} based on how the UnmarshalJSON and MarshalJSON functions are written:

Hm. Yeah, I forgot about that (technically the correct solution but a massive breaking change).

[kevina]

This will be an API change. As {"cid": ""} will be replaced with {"cid": {"/": ""}} based on how the UnmarshalJSON and MarshalJSON functions are written.

One way to fix this is to introduce a new type, maybe called 'APICid` that has a type:

  var APICidBase := mbase.Base58BTC // effects the as String method, not JSON output
  const APICidJSONBase := mbase.Base32 
  type APICid stuct {
    str string // always in APICidJSONBase
  }

  func FromCid(*cid Cid) {}

  func FromString(str string) {} // may not be necessary

  func (c APICid) AsString() {
    return WithBase(APICidBase)
  }

  func (c APICid) WithBase(enc mbase.Encoder) string {
    if CidV0 string or already in enc {
      return c.str
    }
    // convert Cid to base enc
  }
 
  func (c APICid) AsCid() *cid.Cid {...}

  func (c APICid) UnmarshalJSON(b []byte) error {...}

  // encodes the cid as a JSON string
  func (c APICid) MarshalJSON() ([]byte, error) {
    return json.Marshal(c.str)
  }

Then convert the type of anything that is a Cid String to APICid. We might be able to get away with just calling APICid as Cid, especially if we put this in its own package.

To change the global base the variable APICidBase can be changed. This will be a client side change so we can use a global variable now.

[Stebalien]

Another solution is to use refmt. It has support for explicitly specifying how different types should be encoded to/from JSON.