Add a set of GitHub Actions to perform GitGitGadget's job

[dscho]

This PR is part 3 of addressing #609, and it is stacked on top of #1980 and #1981 (and therefore contains also the commits of those PRs), therefore I will leave this in draft mode until those PRs are merged.

The grand idea is to bundle the CIHelper class together with all its direct and transitive dependencies into one big, honking dist/index.js, and then add a set of really minimal GitHub Actions that call into CIHelper. The Actions are added in sub-directories so that they can be called in GitHub workflows via e.g. - uses: gitgitgadget/gitgitgadget/update-prs@1.

The component used for bundling CIHelper is @vercel/ncc . To support acting as a GitHub Action, @actions/core is installed.

To allow for really minimal GitHub Actions, the CIHelper class is augmented accordingly to re-implement more logic that is currently either in misc-helper.ts or in the (non-public 😞) Azure Pipelines definitions.

The naming convention for specifying the necessary tokens as GitHub Actions inputs is:

• upstream-repo-token: This is to comment on PRs in git/git
• pr-repo-token: This is to comment on PRs in gitgitgadget/git (as well as to be able to push to that repository)
• test-repo-token: This is to comment on PRs in dscho/git (used exclusively for testing)

To clarify, here is a diagram:

graph TD
    user["user (contributor)"]
    upstream-repo["upstream-repo (authoritative project repo)"]
    pr-repo["pr-repo (GitGitGadget-enabled GitHub repo)"]
    GitGitGadget["GitGitGadget"]
    mailing-list["mailing-list"]

    user -->|"opens PR"| pr-repo
    user -->|"opens PR (if GitHub App installed)"| upstream-repo

    upstream-repo -->|"GitGitGadget syncs branches to"| pr-repo

    pr-repo -->|"slash commands"| GitGitGadget
    upstream-repo -->|"slash commands (if App installed)"| GitGitGadget

    GitGitGadget -->|"sends patch series"| mailing-list

Loading

[webstech]

Could the input names be less GitGitGadget specific to make it simpler for other projects? Maybe more purpose oriented.

[dscho]

I would have loved that! Unfortunately, it seems that the same access token cannot be reused for multiple repositories unless they are in the same org (see here, the underlying limitation is that the access tokens can only be created per installation, which is per-org).

So the main problem here is GitGitGadget's now-problematic historical development, where it first used its own Git fork for all the work (gitgitgadget/git), but was later extended to work on git/git itself (albeit with a different GitHub App, one that lacks write permission to the repository, for security reasons).

There is probably a good way to talk about this kind of setup, something like "primary-repo" and "own-fork" would be descriptive but probably too unintuitive to be useful when extending GitGitGadget to support 3rd-party projects.

Hmm. @webstech @mjcheetham how about pr-repo-token? And git-git-token would become upstream-repo-token? And dscho-git-token would either go away (that would be the easiest) or would become test-repo-token?

Maybe I should throw in a diagram?

graph TD
    user["user (contributor)"]
    upstream-repo["upstream-repo (authoritative project repo)"]
    pr-repo["pr-repo (GitGitGadget-enabled GitHub repo)"]
    GitGitGadget["GitGitGadget"]
    mailing-list["mailing-list"]

    user -->|"opens PR"| pr-repo
    user -->|"opens PR (if GitHub App installed)"| upstream-repo

    upstream-repo -->|"GitGitGadget syncs branches to"| pr-repo

    pr-repo -->|"slash commands"| GitGitGadget
    upstream-repo -->|"slash commands (if App installed)"| GitGitGadget

    GitGitGadget -->|"sends patch series"| mailing-list

Loading

[mjcheetham]

I like the pr-repo-*, upstream-repo-*, test-repo* abstract naming.

[dscho]

Done: https://github.com/gitgitgadget/gitgitgadget/compare/ca225d2c2501de62ec48e4af09c28d7ccf8aff57..659f8b6fa823d34df9d196e5a3f6d9270fb170ba

[webstech]

The branch name should probably be in the config or obtained from the remote (ugh). I mention this without knowing if it will be a factor later. Just remember there was a LOT of discussion about main.

[dscho]

Oh, I should have mentioned that the --initial-branch main parameter is there just to suppress the ugly warning. It serves no other purpose because upstream Git does not even update its own main branch anymore (it has been deleted in the meantime, even, showing just how the Git maintainer cared about said LOT of discussion).

[dscho]

Maybe changing that branch name from main to unused would clear things up?

[dscho]

Done.

[mjcheetham]

This path is Debian specific, since they use lib rather than libexec here. We should at least add a comment here, less we hit issues in the future with moving to different distros or if Debian changes their minds again.

[dscho]

Hmm. I added that mostly because in my VS Code-based testing, GIT_EXEC_PATH was set and referred to /usr/local/lib/git-core/ (because I installed microsoft/git there)...

I guess I'll just remove this statement and see whether things still work on the runners.

[dscho]

Ah no, I was wrong... Dugite forcefully sets the GIT_EXEC_PATH, and it uses an incorrect one for Ubuntu's git: https://github.com/desktop/dugite/blob/v2.7.1/lib/git-environment.ts#L44-L64. So yes, we do need to set the exec path.

[dscho]

Made generic.

[mjcheetham]

I like the pr-repo-*, upstream-repo-*, test-repo* abstract naming.

[webstech]

Should repo be checked against this.config.repo.name? This seems git specific.

[webstech]

The name and desc are unclear. This is really the state tracking repo isn't it? The pr can be on multiple (well at least two) forks. Just trying to see the outside user viewpoint. If a model project is going to be provided (that was the direction I wanted), the desc can be clearer there.

[webstech]

Where does this come from? comment.id and other context variables have the data this contains.

[dscho]

The context would contain comment.id and other relevant metadata iff the workflow was triggered by issue_comment in pr-repo (or in upstream-repo or in test-repo). But it doesn't even run in the same GitHub org...

So why not provide the metadata split up, as comment-id, owner, repo? For ease of use: GitHub occasionally misses sending webhook events, or cannot reach the Azure Function, or something else goes awry. In such a case, someone with write permissions to gitgitgadget-workflows can conveniently trigger the run by simply copying the link to the respective PR comment and pasting it when triggering the workflow manually.

[webstech]

re: support for other projects

Probably need to add an optional config input. Maybe also a static method in CIHelper to get a user specified input var.

[dscho]

Right, but I'd probably make that more automagic by teaching CIHelper's constructor to call core.getInput("config"), and if it is set (and not empty!), parse it as JSON and use it instead of the default config.

Then, I'd add those as optional Action inputs, and then I'd set the config as a repository variable in the gitgitgadget-workflows repository and then modify the workflow definitions to pass that variable through.

Having said all that, I'd want to do that in a separate set of PRs, after making those modifications and testing them out on Cygwin contributions. In other words: later.

[webstech]

re: support for other projects

Need to call the previously mentioned CIHelper function to get an input var for the config prior to this.

[dscho]

As mentioned above, I'd put that into the constructor of CIHelper, to heed the DRY principle insofar possible.

[webstech]

Why is this a different name from the previous action? (still leaves the question of why other context information is not used for this)

[dscho]

For PR pushes, we don't have a PR issue, and therefore no PR issue URL, that's why 🤭

And we don't have the PR issue webhook event context to work with here, as GitHub Actions' architecture does not allow for cross-repository triggers: The workflow runs in gitgitgadget-workflows, not in, say, gitgitgadget/git.

[webstech]

Could be the same name that both parsePRCommentURLInput and parsePRURLInput use for getInput. The contents are contextual based on the workflow. Just a different point of view.

[dscho]

They could, but I think it is much clearer if the input keys are as different as the exact regular expressions their values must match, i.e. I want to keep pr-url and pr-comment-url as-are.

[webstech]

Should this be a config value to support other projects with a similar setup?

[dscho]

Yes!

[dscho]

@webstech I force-pushed quite a few updates in response to your feedback (thank you!):

range-diff

• 1: ba4fd1d ! 1: 5183e81 Install @actions/core

  @@ Metadata
    ## Commit message ##
       Install @actions/core
   
  -    This will be used to convert GitGitGadget to a GitHub Action.
  +    This will be used to migrate GitGitGadget a set of Azure Pipelines to a
  +    set of GitHub Actions and workflows.
   
       Signed-off-by: Johannes Schindelin <[email protected]>
   

• 2: 53d0e7c ! 2: 7d538e5 CIHelper: add a convenient function to initialize e.g. the Git worktree

  @@ Commit message
       To make this somewhat efficient, let's try to use a partial
       (semi-shallow) clone.
   
  -    And since there will be at least one GitHub workflow that runs several
  -    of GitGitGadget's GitHub Actions, let's play nice and reuse the
  -    initialized Git worktree.
  +    Let's play nice and reuse an already-existing Git worktree (if any), to
  +    make local debugging easier.
  +
  +    The strategy chosen here is to initialize a partial, blob-less clone,
  +    then initialize the `gitgitgadget` Git notes by shallowly fetching the
  +    ref with depth 1 (i.e. the tip commit) including all blobs (using
  +    `--filter=blob:limit=1g` instead of `--no-filter` to avoid running into
  +    bugs introduced in Git v2.48.0-rc0~58^2 (index-pack: repack local links
  +    into promisor packs, 2024-11-01) where it falls over its own feet when
  +    promisor and non-promisor packs are mixed), and then unshallows the ref
  +    with a tree-less fetch.
   
       There are a couple of other things we want to set up in the upcoming
       GitHub Actions, e.g. handling the installation access tokens specified
  -    as GitHub Action inputs; Let's combine all of that setup in a single
  +    as GitHub Action inputs; Let's combine all of that setup within a single
       method of the `CIHelper` class.
   
       Signed-off-by: Johannes Schindelin <[email protected]>
  @@ lib/ci-helper.ts: export class CIHelper {
   +        }
   +
   +        // get the access tokens via the inputs of the GitHub Action
  -+        this.setAccessToken("gitgitgadget", core.getInput("pr-repo-token"));
  -+        this.setAccessToken("git", core.getInput("upstream-repo-token"));
  -+        this.setAccessToken("dscho", core.getInput("test-repo-token"));
  ++        this.setAccessToken(this.config.repo.owner, core.getInput("pr-repo-token"));
  ++        this.setAccessToken(this.config.repo.baseOwner, core.getInput("upstream-repo-token"));
  ++        if (this.config.repo.testOwner) {
  ++            this.setAccessToken(this.config.repo.testOwner, core.getInput("test-repo-token"));
  ++        }
   +
   +        // eslint-disable-next-line security/detect-non-literal-fs-filename
   +        if (!fs.existsSync(this.workDir)) await git(["init", "--bare", "--initial-branch", "unused", this.workDir]);
  @@ lib/ci-helper.ts: export class CIHelper {
   +        console.time("fetch Git notes");
   +        await git(
   +            [
  -+                "-c",
  -+                "remote.origin.promisor=false", // let's fetch the notes with all of their blobs
   +                "fetch",
  ++                "--filter=blob:limit=1g", // let's fetch the notes with all of their blobs
   +                "--no-tags",
   +                "origin",
   +                "--depth=1",
  @@ lib/ci-helper.ts: export class CIHelper {
        public setAccessToken(repositoryOwner: string, token: string): void {
            this.github.setAccessToken(repositoryOwner, token);
            if (this.config.repo.owner === repositoryOwner) {
  +
  + ## lib/gitgitgadget-config.ts ##
  +@@ lib/gitgitgadget-config.ts: const defaultConfig: IConfig = {
  +         name: "git",
  +         owner: "gitgitgadget",
  +         baseOwner: "git",
  ++        testOwner: "dscho",
  +         owners: ["gitgitgadget", "git", "dscho"],
  +         branches: ["maint", "seen"],
  +         closingBranches: ["maint", "master"],
  +
  + ## lib/project-config.ts ##
  +@@ lib/project-config.ts: export interface IConfig {
  +     repo: {
  +         name: string; // name of the repo
  +         owner: string; // owner of repo holding the notes (tracking data)
  +-        baseOwner: string; // owner of base repo
  ++        baseOwner: string; // owner of upstream ("base") repo
  ++        testOwner?: string; // owner of the test repo (if any)
  +         owners: string[]; // owners of clones being monitored (PR checking)
  +         branches: string[]; // remote branches to fetch - just use trackingBranches?
  +         closingBranches: string[]; // close if the pr is added to this branch

• 3: 46bc653 ! 3: 4a99b4e CIHelper.setupGitHubAction(): also process the SMTP inputs, if any

  @@ Commit message
       CIHelper.setupGitHubAction(): also process the SMTP inputs, if any
   
       The upcoming `handle-pr-comment` Action wants to send emails, and for
  -    that reason needs to accept SMTP information. This will be passed into
  -    the Action via the `smtp-options` input, which is a JSON-formatted
  -    triplet of user/host/pass info.
  +    that reason needs to accept SMTP credentials. These can be provided via
  +    the `smtp-host`, `smtp-user` and `smtp-pass` Action inputs. Additional
  +    options (that might be necessary to support projects other than Git) can
  +    be passed in via the `smtp-opts` input, a JSON-formatted object to
  +    augment/oeverride nodemailer's `SMTPTransport.Options` when sending the
  +    emails.
   
       Since most of GitGitGadget's Actions do not want to send emails, it is
  -    not an error if there are no `smtp-options` when `setupGitHubAction()`
  -    is called.
  +    silently ignored if these Action inputs are unset or empty when
  +    `setupGitHubAction()` is called.
   
       Signed-off-by: Johannes Schindelin <[email protected]>
   
    ## lib/ci-helper.ts ##
   @@ lib/ci-helper.ts: export class CIHelper {
  -         this.setAccessToken("git", core.getInput("upstream-repo-token"));
  -         this.setAccessToken("dscho", core.getInput("test-repo-token"));
  +             this.setAccessToken(this.config.repo.testOwner, core.getInput("test-repo-token"));
  +         }
    
   +        // set the SMTP options
   +        try {
   +            const options = {
  -+                user: core.getInput("smtp-user"),
  -+                host: core.getInput("smtp-host"),
  -+                pass: core.getInput("smtp-pass"),
  -+                opts: core.getInput("smtp-opts"),
  ++                smtpUser: core.getInput("smtp-user"),
  ++                smtpHost: core.getInput("smtp-host"),
  ++                smtpPass: core.getInput("smtp-pass"),
  ++                smtpOpts: core.getInput("smtp-opts"),
   +            };
  -+            if (options.user && options.host && options.pass) {
  -+                this.setSMTPOptions({
  -+                    smtpUser: options.user,
  -+                    smtpHost: options.host,
  -+                    smtpPass: options.pass,
  -+                    smtpOpts: options.opts,
  -+                });
  ++            if (options.smtpUser && options.smtpHost && options.smtpPass) {
  ++                this.setSMTPOptions(options);
   +            }
   +        } catch (e) {
   +            // Ignore, for now

• 4: 0e41096 ! 4: 912f1dc CIHelper.setupGitHubAction: configure Git committer information

  @@ Metadata
    ## Commit message ##
       CIHelper.setupGitHubAction: configure Git committer information
   
  -    GitGitGadget will enquire it via `git var` when sending mails, and also
  -    use it when tagging the patch series' iterations.
  +    GitGitGadget will obtain this information via `git var` whenever sending
  +    mails, and also when tagging the patch series' iterations.
   
       Signed-off-by: Johannes Schindelin <[email protected]>
   
  @@ lib/ci-helper.ts: export class CIHelper {
   +            .join(" ");
   +
            // get the access tokens via the inputs of the GitHub Action
  -         this.setAccessToken("gitgitgadget", core.getInput("pr-repo-token"));
  -         this.setAccessToken("git", core.getInput("upstream-repo-token"));
  +         this.setAccessToken(this.config.repo.owner, core.getInput("pr-repo-token"));
  +         this.setAccessToken(this.config.repo.baseOwner, core.getInput("upstream-repo-token"));

• 5: 5900704 ! 5: 1b8d5fd Install @vercel/ncc

  @@ Commit message
       install `@vercel/ncc`, a 'simple CLI for compiling a Node.js module into
       a single file, together with all its dependencies, gcc-style.'. This
       will allow us to bundle a minimal set of Javascript files and publish
  -    the result via a tag.
  +    the result via a tag, ready to be consumed in GitHub workflows.
   
       The idea is to run `ncc build -s -m lib/ci-helper.ts -o dist/` to obtain
       a minimized `dist/index.js` that has no additional dependencies, and
  -    then have subdirectories (e.g. `update-prs/`) with the individual
  -    `action.yml` and `index.js` that starts with the following line
  +    then have subdirectories (e.g. `update-prs/`) defining a GitHub Action
  +    via an `action.yml` file and a `index.js` file that starts with the
  +    following line
   
               const { CIHelper } = require('../dist/index.js')
   

• 6: 0a11624 = 6: 5d7df21 Run ncc via npm run build-dist

• 7: f02a5ed ! 7: e975b59 Ensure that res/WELCOME.md is found even after compiling with ncc

  @@ Commit message
       seem to work in my hands).
   
       Having said that, the promise was that `ncc` would include this asset in
  -    the output, but this is not true for me.
  +    the `dist/index.js` file, but no matter what I tried, I could not make
  +    this work. For that reason, the `res/` folder will need to be copied
  +    into `dist/` before publishing as a GitHub Action.
   
       Signed-off-by: Johannes Schindelin <[email protected]>
   

• 8: f2cc47c ! 8: b8437bb CIHelper: offer a convenient method to parse a PR comment URL

  @@ Commit message
   
       This will be used in the upcoming `handle-pr-comment` GitHub Action.
   
  +    Note that the reason why this information needs to be passed to the
  +    respective GitHub Action is that the corresponding GitHub workflows will
  +    not run inside `gitgitgadget/git` and will therefore not automatically
  +    receive the comment ID and PR number as part of the `github` context.
  +    GitGitGadget's GitHub App will need to pass that information along when
  +    receiving the webhook event that a PR comment has been created or
  +    modified.
  +
       Signed-off-by: Johannes Schindelin <[email protected]>
   
    ## lib/ci-helper.ts ##
  @@ lib/ci-helper.ts: export class CIHelper {
   +        const prCommentUrl = core.getInput("pr-comment-url");
   +        const [, owner, repo, prNumber, commentId] =
   +            prCommentUrl.match(/^https:\/\/github\.com\/([^/]+)\/([^/]+)\/pull\/(\d+)#issuecomment-(\d+)$/) || [];
  -+        if (!this.config.repo.owners.includes(owner) || repo !== "git") {
  ++        if (!this.config.repo.owners.includes(owner) || repo !== this.config.repo.name) {
   +            throw new Error(`Invalid PR comment URL: ${prCommentUrl}`);
   +        }
   +        return { owner, repo, prNumber: parseInt(prNumber, 10), commentId: parseInt(commentId, 10) };

• 9: cc73002 ! 9: 7bdd196 Add the handle-pr-comment GitHub Action

  @@ Commit message
         - GitGitGadget PR Handler (dscho):
           https://dev.azure.com/gitgitgadget/git/_build?definitionId=12
   
  +    Note that this Action receives the full `pr-comment-url` as Action
  +    input; Readers might wonder why the Action cannot receive the
  +    information in a more fine-grained manner, seeing as the `github`
  +    context of a GitHub workflow triggered by a new PR comment already has
  +    all the necessary values. But this would require adding a
  +    GitGitGadget-specific workflow to the main `git/git` repository,
  +    something that was considered in
  +    https://github.com/gitgitgadget/gitgitgadget/issues/861 but which would
  +    be _extremely_ difficult to convince the core Git contributors to
  +    accept. Therefore, the GitHub workflow using this Action will be a
  +    `workflow_dispatch`-triggered one, for more details see
  +    https://github.com/gitgitgadget-workflows/gitgitgadget-workflows/pull/1.
  +
       Signed-off-by: Johannes Schindelin <[email protected]>
   
    ## handle-pr-comment/action.yml (new) ##
  @@ handle-pr-comment/action.yml (new)
   +author: 'Johannes Schindelin'
   +inputs:
   +  pr-repo-token:
  -+    description: 'The access token to work on gitgitgadget/git'
  ++    description: 'The access token to work on the repository that holds PRs and state'
   +    required: true
   +  upstream-repo-token:
  -+    description: 'The access token to work on git/git'
  ++    description: 'The access token to work on PRs in the upstream repository'
   +    required: true
   +  test-repo-token:
  -+    description: 'The access token to work on dscho/git'
  ++    description: 'The access token to work on PRs in the test repository'
   +    required: true
   +  smtp-host:
   +    description: 'The SMTP host to use for sending emails'

• 10: 944adc1 ! 10: 3a84ca1 CIHelper: offer a convenient method to parse a PR URL

  @@ Commit message
   
       This will be used in the upcoming `handle-pr-push` GitHub Action.
   
  +    Just like the `handle-pr-comment` Action, this Action will be used in a
  +    GitHub workflow that runs in a different repository than the one where
  +    the PR lives that is to be processed, therefore the information cannot
  +    be passed in via the `github` context, and combined with ease of
  +    debugging, I settled on the full PR URL as the unit of information that
  +    the Action input should accept.
  +
       Signed-off-by: Johannes Schindelin <[email protected]>
   
  + ## .vscode/settings.json ##
  +@@
  +         "CIsmtphost",
  +         "CIsmtppass",
  +         "CIsmtpopts",
  ++        "parsePRURLInput",
  +         "users\\.noreply\\.github\\.com",
  +     ],
  +     "cSpell.language": "en-US",
  +
    ## lib/ci-helper.ts ##
   @@ lib/ci-helper.ts: export class CIHelper {
            return { owner, repo, prNumber: parseInt(prNumber, 10), commentId: parseInt(commentId, 10) };
  @@ lib/ci-helper.ts: export class CIHelper {
   +
   +        const [, owner, repo, prNumber] =
   +            prCommentUrl.match(/^https:\/\/github\.com\/([^/]+)\/([^/]+)\/pull\/(\d+)$/) || [];
  -+        if (!this.config.repo.owners.includes(owner) || repo !== "git") {
  ++        if (!this.config.repo.owners.includes(owner) || repo !== this.config.repo.name) {
   +            throw new Error(`Invalid PR comment URL: ${prCommentUrl}`);
   +        }
   +        return { owner, repo, prNumber: parseInt(prNumber, 10) };

• 11: e275249 ! 11: 7979f53 Add the handle-pr-push GitHub Action

  @@ Commit message
         https://dev.azure.com/gitgitgadget/git/_build?definitionId=12
   
       This Action will be used in a GitHub workflow that will replace those
  -    trusty Azure Pipelines we used for so long.
  +    trusty Azure Pipelines we used for so long. This workflow lives in a
  +    different org, for full details see
  +    https://github.com/gitgitgadget-workflows/gitgitgadget-workflows/pull/1.
   
       Signed-off-by: Johannes Schindelin <[email protected]>
   
  @@ handle-pr-push/action.yml (new)
   +author: 'Johannes Schindelin'
   +inputs:
   +  pr-repo-token:
  -+    description: 'The access token to work on gitgitgadget/git'
  ++    description: 'The access token to work on the repository that holds PRs and state'
   +    required: true
   +  upstream-repo-token:
  -+    description: 'The access token to work on git/git'
  ++    description: 'The access token to work on PRs in the upstream repository'
   +    required: true
   +  test-repo-token:
  -+    description: 'The access token to work on dscho/git'
  ++    description: 'The access token to work on PRs in the test repository'
   +    required: true
   +  pr-url:
   +    description: 'The URL of the PR that was pushed'

• 12: 29949d3 ! 12: 083a1a1 CIHelper: optionally clone the Git mailing list mirror

  @@ Commit message
       no time is wasted on a full clone every time GitGitGadget wants to look
       at mails.
   
  +    The not-so-obvious idea is to un-shallow the clone using the same
  +    strategy (and in fact, the same function) as for the Git "worktree":
  +    When I tried to simply delete the `shallow` file during the development
  +    of this patch, assuming that Git's partial clone functionality would
  +    gracefully fetch any missing commit e.g. when computing a merge base,
  +    reality taught me the harsh lesson that it would fail hard in such
  +    scenarios. Instead, the `git fetch --unshallow` command is called with
  +    `--filter=tree:0` so that _only_ the missing commits (and all of them)
  +    are fetched. This still seems to strike a good compromise between saving
  +    time on the clone and saving on network round-trips when fetching
  +    missing Git objects.
  +
       Signed-off-by: Johannes Schindelin <[email protected]>
   
  + ## .vscode/settings.json ##
  +@@
  +         "CIsmtphost",
  +         "CIsmtppass",
  +         "CIsmtpopts",
  ++        "mailrepo",
  +         "parsePRURLInput",
  +         "users\\.noreply\\.github\\.com",
  +     ],
  +
    ## lib/ci-helper.ts ##
   @@ lib/ci-helper.ts: export class CIHelper {
        private notesPushToken: string | undefined;
  @@ lib/ci-helper.ts: export class CIHelper {
            await unshallow(this.workDir);
   +
   +        if (setupOptions?.needsMailingListMirror) {
  -+            this.mailingListMirror = "git-mailing-list-mirror.git";
  -+            const epoch = 1;
  ++            this.mailingListMirror = "mailing-list-mirror.git";
  ++            const epoch = this.config.mailrepo.public_inbox_epoch ?? 1;
   +
   +            // eslint-disable-next-line security/detect-non-literal-fs-filename
   +            if (!fs.existsSync(this.mailingListMirror)) {
  @@ lib/ci-helper.ts: export class CIHelper {
   +
   +            // First fetch from GitGitGadget's mirror, which supports partial clones
   +            for (const [key, value] of [
  -+                ["remote.mirror.url", `https://github.com/${this.config.repo.owner}/git-mailing-list-mirror`],
  ++                ["remote.mirror.url", this.config.mailrepo.mirrorURL || this.config.mailrepo.url],
   +                ["remote.mirror.promisor", "true"],
   +                ["remote.mirror.partialCloneFilter", "blob:none"],
   +            ]) {
  @@ lib/ci-helper.ts: export class CIHelper {
            await git(["fetch"], { workDir: mailArchiveGitDir });
            const prFilter = !onlyPRs
                ? undefined
  +
  + ## lib/gitgitgadget-config.ts ##
  +@@ lib/gitgitgadget-config.ts: const defaultConfig: IConfig = {
  +         branch: "master",
  +         host: "lore.kernel.org",
  +         url: "https://lore.kernel.org/git/",
  ++        public_inbox_epoch: 1,
  ++        mirrorURL: "https://github.com/gitgitgadget/git-mailing-list-mirror",
  +         descriptiveName: "lore.kernel/git",
  +     },
  +     mail: {
  +
  + ## lib/project-config.ts ##
  +@@ lib/project-config.ts: export interface IConfig {
  +         branch: string;
  +         host: string;
  +         url: string;
  ++        public_inbox_epoch?: number;
  ++        mirrorURL?: string;
  +         descriptiveName: string;
  +     };
  +     mail: {

• 13: a42a21d ! 13: 113de57 Add the handle-new-mails GitHub Action

  @@ Commit message
       GitGitGadget's PRs" Azure Pipeline over at
       https://dev.azure.com/gitgitgadget/git/_build?definitionId=5.
   
  -    This will obviously require a GitHub workflow, too.
  +    This will obviously require a GitHub workflow, too, which is added in
  +    https://github.com/gitgitgadget-workflows/gitgitgadget-workflows/pull/1
   
       Signed-off-by: Johannes Schindelin <[email protected]>
   
  @@ handle-new-mails/action.yml (new)
   +author: 'Johannes Schindelin'
   +inputs:
   +  pr-repo-token:
  -+    description: 'The access token to work on gitgitgadget/git'
  ++    description: 'The access token to work on the repository that holds PRs and state'
   +    required: true
   +  upstream-repo-token:
  -+    description: 'The access token to work on git/git'
  ++    description: 'The access token to work on PRs in the upstream repository'
   +    required: true
   +  test-repo-token:
  -+    description: 'The access token to work on dscho/git'
  ++    description: 'The access token to work on PRs in the test repository'
   +    required: true
   +runs:
   +  using: 'node20'

• 14: c1e4c12 ! 14: 2849f5e CIHelper: offer to pre-populate the upstream branches

  @@ Commit message
   
       Note that we use a "funny" `--filter=blob:limit=1g` option here instead
       of the more natural `--no-filter`: Git introduced a subtle bug (that I
  -    have no time to pursue any further) in v2.48.0-rc0~58^2 (index-pack:
  -    repack local links into promisor packs, 2024-11-01) that would
  -    occasionally result in this strange error when fetching in the partial
  -    clone after an occasional `--no-filter` fetch:
  +    have no time nor patience to pursue any further) in v2.48.0-rc0~58^2
  +    (index-pack: repack local links into promisor packs, 2024-11-01) that
  +    would occasionally result in this strange error when fetching in the
  +    partial clone after an occasional `--no-filter` fetch:
   
         git -C git.git fetch upstream
         remote: Enumerating objects: 2, done.
  @@ Commit message
         fatal: index-pack failed
   
       The most likely reason is that the logic needs to improved to
  -    accommodate to situations where commits are fetched into a non-promisor
  -    pack as part of fetch that notices that some of the reachable commits
  -    are present locally (but they are in promisor packs). But again, I do
  -    not have time to pursue this any further. Keeping out non-promisor packs
  -    avoids the bug, and that's that.
  +    accommodate for situations where commits are fetched into a non-promisor
  +    pack as part of fetch that notices that some of the commits reachable
  +    from that pack are present locally (but they are in promisor packs,
  +    which the naïve `pack-objects` call fails to anticipate). But again, I
  +    do not have time to pursue this any further. Keeping out non-promisor
  +    packs avoids the bug, and that's that.
   
       Signed-off-by: Johannes Schindelin <[email protected]>
   

• 15: d364126 ! 15: 57a6c9b Add the update-prs GitHub Action

  @@ Commit message
       Azure Pipeline over at
       https://dev.azure.com/gitgitgadget/git/_build?definitionId=2.
   
  -    This will obviously require a GitHub workflow, too.
  +    This will obviously require a GitHub workflow, which is added in
  +    https://github.com/gitgitgadget-workflows/gitgitgadget-workflows/pull/1
   
       Signed-off-by: Johannes Schindelin <[email protected]>
   
  @@ update-prs/action.yml (new)
   +author: 'Johannes Schindelin'
   +inputs:
   +  pr-repo-token:
  -+    description: 'The access token to work on gitgitgadget/git'
  ++    description: 'The access token to work on the repository that holds PRs and state'
   +    required: true
   +  upstream-repo-token:
  -+    description: 'The access token to work on git/git'
  ++    description: 'The access token to work on PRs in the upstream repository'
   +    required: true
   +  test-repo-token:
  -+    description: 'The access token to work on dscho/git'
  ++    description: 'The access token to work on PRs in the test repository'
   +    required: true
   +runs:
   +  using: 'node20'

• 16: 17030ef ! 16: 35466fa CIHelper: offer to update the mail-to-commit and commit-to-mail notes

  @@ Commit message
       logic to a method of the `CIHelper` class that will be used in a GitHub
       Action so that the Azure Pipeline can be retired.
   
  +    This was the only Azure Pipeline of GitGitGadget which did not,
  +    actually, use the `misc-helper.ts` script, but instead was opaquely
  +    implemented as a lengthy shell scriptlet inside the Pipeline definition
  +    that called two of the shell scripts in the `gitgitgadget/gitgitgadget`
  +    repository: `lookup-commit.sh` and `update-mail-to-commit-notes.sh`.
  +
       Since the scripts that are called by the new method expect a persisted,
       non-partial clone of the Git mailing list, we need to play a couple of
       games here to make it work in a GitHub workflow (that runs on ephemeral
       runners where the repository has to be cloned afresh in every run).
   
  +    Also: These shell scripts, by virtue of needing to be interpreted by a
  +    shell interpreter outside of node.js, need to be copied into the
  +    `dist/script/` subdirectory before publishing the GitHub Action, so that
  +    they can be found and interpreted as expected.
  +
       Signed-off-by: Johannes Schindelin <[email protected]>
   
  + ## .vscode/settings.json ##
  +@@
  +         "CIsmtphost",
  +         "CIsmtppass",
  +         "CIsmtpopts",
  ++        "GITGIT_(|DIR|GIT_REMOTE|MAIL_REMOTE|MAIL_EPOCH)",
  +         "mailrepo",
  +         "parsePRURLInput",
  +         "users\\.noreply\\.github\\.com",
  +
    ## lib/ci-helper.ts ##
   @@ lib/ci-helper.ts: import * as core from "@actions/core";
    import * as fs from "fs";
  @@ lib/ci-helper.ts: export class CIHelper {
   +        }
            await git(
                [
  -                 "-c",
  +                 "fetch",
   @@ lib/ci-helper.ts: export class CIHelper {
                    "--no-tags",
                    "origin",

• 17: 659f8b6 ! 17: 0f5158b Add the update-mail-to-commit-notes GitHub Action

  @@ Commit message
       https://dev.azure.com/gitgitgadget/git/_build?definitionId=9, making use
       of the just-ported logic in `CIHelper`.
   
  -    This will obviously require a GitHub workflow, too.
  +    This will obviously require a GitHub workflow, too, which is added in
  +    https://github.com/gitgitgadget-workflows/gitgitgadget-workflows/pull/1
   
       Signed-off-by: Johannes Schindelin <[email protected]>
   
  @@ update-mail-to-commit-notes/action.yml (new)
   +author: 'Johannes Schindelin'
   +inputs:
   +  pr-repo-token:
  -+    description: 'The access token to work on gitgitgadget/git'
  ++    description: 'The access token to work on the repository that holds PRs and state'
   +    required: true
   +runs:
   +  using: 'node20'

[webstech]

Should lore be in the config? ie could someone follow a similar pattern for a mailing list but not quite follow the lore.kernel.org pattern?

[dscho]

Oh, you're right! I just called git ls-remote https://inbox.sourceware.org/cygwin-patches/0 and it turns out that its HEAD ref is called master...

[dscho]

Actually, it would appear that I was playing some games here, seeing as there are already two epochs of the Git mailing list repository, where the "0th" epoch had a different tree layout (a wide one, which contained all emails in the tip tree, as opposed to the current layout, where the tip tree contains the most recent email as the HEAD:m file). To be able to fetch both into the same mirror repository, I fetched the "0th epoch" into lore-0 and the "1st epoch" into lore-1.

So it's probably safe to assume that whatever public-inbox-based mailing list mirror is used will have to potentially deal with different epochs, too, and come up with a similar naming scheme as I did.

But yes, it's unlikely to be the naming scheme refs/heads/lore-${epoch} that I chose ;-) Therefore, I should totally move the actual ref name into the config, similar to the mirror URL of the Git repository.

For the record, I do believe that we must work with a mailing list mirror repository from which one can make a partial clone; Trying to clone with --filter from, say, lore.kernel.org will result in this warning: "warning: filtering not recognized by server, ignoring".

[dscho]

Will push an update in a moment to this specific end.

[webstech]

Noting this will pull from dscho repo when it is not needed in normal runs.

[dscho]

It's determining which PRs are open in dscho/git, which is currently 0...

[webstech]

I have finished reviewing.

[dscho]

@dscho dscho force-pushed the add-the-action branch from 0f5158b to a209ca3

range-diff

• 1: 083a1a1 ! 1: b38439c CIHelper: optionally clone the Git mailing list mirror

  @@ lib/ci-helper.ts: export class CIHelper {
   +                    "fetch",
   +                    "mirror",
   +                    "--depth=50",
  -+                    `+refs/heads/lore-${epoch}:refs/heads/lore-${epoch}`,
  ++                    "+REF:REF".replace("REF", this.config.mailrepo.mirrorRef || `refs/heads/lore-${epoch}`),
   +                ],
   +                {
   +                    workDir: this.mailingListMirror,
  @@ lib/gitgitgadget-config.ts: const defaultConfig: IConfig = {
            url: "https://lore.kernel.org/git/",
   +        public_inbox_epoch: 1,
   +        mirrorURL: "https://github.com/gitgitgadget/git-mailing-list-mirror",
  ++        mirrorRef: "refs/heads/lore-1",
            descriptiveName: "lore.kernel/git",
        },
        mail: {
  @@ lib/project-config.ts: export interface IConfig {
            url: string;
   +        public_inbox_epoch?: number;
   +        mirrorURL?: string;
  ++        mirrorRef?: string;
            descriptiveName: string;
        };
        mail: {

• 2: 113de57 = 2: 90989ff Add the handle-new-mails GitHub Action

• 3: 2849f5e = 3: 0339122 CIHelper: offer to pre-populate the upstream branches

• 4: 57a6c9b = 4: 40f0d48 Add the update-prs GitHub Action

• 5: 35466fa ! 5: 9340912 CIHelper: offer to update the mail-to-commit and commit-to-mail notes

  @@ lib/ci-helper.ts: export class CIHelper {
                        "mirror",
   -                    "--depth=50",
   +                    `--depth=${setupOptions?.needsMailToCommitNotes ? 5000 : 50}`,
  -                     `+refs/heads/lore-${epoch}:refs/heads/lore-${epoch}`,
  +                     "+REF:REF".replace("REF", this.config.mailrepo.mirrorRef || `refs/heads/lore-${epoch}`),
                    ],
                    {
   @@ lib/ci-helper.ts: export class CIHelper {

• 6: 0f5158b = 6: a209ca3 Add the update-mail-to-commit-notes GitHub Action

[webstech]

Does mirrorRef need to include refs/heads/? I suppose you could put things pretty much anywhere you want in a repo.

[dscho]

It does not have to, but it avoids ambiguity.