Setting up unified CI for js repos

[achingbrain]

It would be great to get similar syncing of the .github/workflows folder for js repos as maintaining them is tedious work.

There are a few different setups with different CI requirements:

1. monorepos
2. repos that only target node.js
3. repos that target node.js and the browser

I spotted https://github.com/protocol/.github/tree/master/templates/.github/workflows - can we parameterise the repos in config.json to copy from subdirectories under templates? Something simple like:

[{
  "target": "filecoin-project/go-amt-ipld",
  "type": "go"
}, {
  "target": "libp2/libp2p-tcp",
  "type": "js/node"
}, {
  "target": "ipfs/js-datastore-s3",
  "type": "js/isomorphic"
}, {
  "target": "ipfs/js-unixfs",
  "type": "js/monorepo"
}
// etc

Then we can have templates/go, templates/js/monorepo, templates/js/node, templates/js/isomorphic, etc.

What do you think? Some other way?

Also, some repos have unique build requirements, like js-ipfs testing itself with various repos from the https://github.com/ipfs-examples org - I guess maybe the copying of the templates could just overwrite files that are in the templates and leave others alone? Then js-ipfs could have a .github/workflows/examples.yml file that would survive an update to templates/js/monorepo in this repo.

[marten-seemann]

Then we can have templates/go, templates/js/monorepo, templates/js/node, templates/js/isomorphic, etc.

That's an interesting idea. Are we reasonable sure that there won't be a lot of overlap (we could check in symlinks in git afaik, but that's probably not the nicest design. Alternatively, we could also have a separate section in the config mapping the type to an array[file].

Another alternative would be to have multiple config files (config-go.json, config-js-monorepo.json`, ...).

We will also have to rework the copy workflow a bit. It has accumulated a bunch of Go-specific stuff. We'd probably need to refactor those steps into a separate file, and include that file depending on the type.

[achingbrain]

I guess there would be a bit of overlap - the isomorphic CI would be a superset of the node CI, for example, though it's not a huge amount of code.

We could just have a js-monorepo build and a js-non-monorepo build - they'd invoke npm scripts with the --if-present flag - test:node, test:browser and so on. Repos can opt-in to these jobs by declaring npm scripts with these names. We have npm scripts like this everywhere already.

multiple config files

This sounds fine, it'd make the contents of the files a bit more concise too.

[achingbrain]

We will also have to rework the copy workflow a bit. It has accumulated a bunch of Go-specific stuff. We'd probably need to refactor those steps into a separate file, and include that file depending on the type.

Do you think you can take that on? I can PR the js workflows in & configure repos once I know where to put them.

[galargh]

Do you think you can take that on? I can PR the js workflows in & configure repos once I know where to put them.

Sure! Added to my after bootcamp TODO list. It finishes this Monday so I should be able to get to it fairly quickly.

[rvagg]

We also have repos that don't use the aegir testing pattern and also have auto-release-on-merge. I'm slowly unifying the patterns used across all of those and am planning on updating some outstanding ones this coming week. But they're quite different to the current patterns for the repos that Alex is maintaining, so can we please make sure there's room for alternative patterns in this.

[galargh]

@achingbrain Could you possibly share examples of js repos that follow different testing patterns(monorepo, node, isomorphic)? I wanted to check them out to get a feel of what those differences are and maybe come up with some ideas on how to best support the customisation on the unified CI side.

I guess maybe the copying of the templates could just overwrite files that are in the templates and leave others alone?

That's exactly how it behaves now. You can have your custom workflows living next to the ones managed by unified CI. Here's an example of that: https://github.com/ipfs/ipfs-update/tree/master/.github/workflows sharness.yml is a custom workflow here and all the others are managed by unified CI.

@rvagg Could you share the links to the repos you're thinking of too? I'll keep them in mind when working out the approach to take here.

[achingbrain]

• isomorphic - https://github.com/ipfs/js-ipfs-utils
• node (actually runs on electron too) - https://github.com/ipfs/js-datastore-fs
• monorepo - https://github.com/ipfs/js-ipfs-unixfs

N.b. I very much want automated releases to come out of this work for the ipfs/libp2p stack - we've wanted this everywhere for ages.

I have no strong opinion on which tools to use for this, more than happy to adopt whatever other people find useful.

[achingbrain]

This is nice - rvagg/polendina#16 (comment) - having to add "the changes from this PR were shipped in foo@bar" to PRs is a chore but is really useful for contributors.

[rvagg]

OK, so for my JS workflows, I've spent this week updating stuff and I have these projects roughly in sync now:

• https://github.com/multiformats/js-multiformats
• https://github.com/ipld/js-dag-cbor
• https://github.com/ipld/js-dag-json
• https://github.com/ipld/js-dag-pb
• https://github.com/ipld/js-car
• https://github.com/multiformats/js-blake2
• https://github.com/multiformats/js-sha3
• https://github.com/multiformats/js-murmur3

The common parts of these are: auto-releases using semantic-release, dependabot configs, ipjs build and typescript pipeline, and testing structure. The .github/ directories should be the same between them now (there used to be differences due to windows and Node.js 12 awkwardness but those are solved and I've just removed Node.js 12 from CI now). Their package.json's are roughly equivalent and I'm trying to keep some consistency there but they have different bits. When you diff the package.json's, the main differences should just be names, but there are some other differences (that could potentially be minimised with some additional work). The "release" field in package.json should be the same between them all and it'd be nice to keep that in sync automatically .. somehow. Keeping package.json's in sync would be sweet. tsconfig.json should be the same, except for the "paths" field which is derived from the name in package.json, these files could potentially be kept in sync if that field was left untouched.

These projects use the same auto-release and dependabot patterns but not much else from the above:

• https://github.com/filecoin-shipyard/js-lotus-client
• https://github.com/filecoin-shipyard/js-lotus-client-schema
• https://github.com/filecoin-shipyard/js-lotus-client-provider-nodejs
• https://github.com/filecoin-shipyard/js-lotus-client-provider-browser

Other repos that are currently in use, some of which are in my personal GitHub (they could be moved, location not so important) that would be nice to keep in sync but need a bit more attention are:

• https://github.com/ipld/js-ipld-schema
• https://github.com/rvagg/cborg
• https://github.com/rvagg/iamap
• https://github.com/rvagg/js-ipld-hashmap
• https://github.com/rvagg/js-ipld-schema-validator
• https://github.com/rvagg/js-ipld-schema-describer

There's more, but we're getting lower down the importance hierarchy and I need to do a fair bit of work to update some older components that have fallen into disrepair (e.g. bitcoin and zcash codecs and their dependencies).

---

The major outstanding concern I have about the current auto-release workflow is that I'm not pinning versions of the semantic-release components. If you have a look in the Actions scripts you'll see use npm and npx for packages that are unversioned. I don't want these packages in the package.json because they're only relevant for releases in Actions (nobody else should need to install them and I try to keep devDependency bloat to a minimum). This presents a breakage risk, and arguably a small security risk (although those same risks exist for any semver ranged dependency versions!). I can't use dependabot to upgrade pinned versions, so if I were to pin versions then I'd have to go and update everything that uses this script. Ideally I'd have npm install .pkg@version and npx pkg@version in there.

If we had this coming out of web3-bot updates then it'd be easier. Another way might be to set up a custom GitHub Action that has all of this packaged in one place and could be updated with dependabot (or web3-bot).