Create config validation workflow

[grahamalama]

Currently, when we merge JBI config changes, there is no shared workflow among the team for validating that those config changes are sound. Things that might make a config change invalid are:

• invalid Jira project name
• our Jira service account not having the correct permissions on a project
• steps are in an order that don't make sense

We should establish some shared workflow for ensuring that proposed config changes are sound before promoting them to prod. This might take the form of:

• a manual testing checklist
  • for us (developers)
  • for customers (so they can QA their own workflow)
• some sort of automated integration test
• some combination of the above

[leplatrem]

A manual testing checklist seems like the bare minimum.

Linking all sub-issues, and Jira admin tickets on the configuration change PR could help too.

Some tests are not obvious to do manually, like for example how could you check that the project permissions in prod?
One idea could be do to spawn an instance with the stage or prod config, as part of CI ideally, and check that the heartbeat is green there.

[bsieber-mozilla]

One idea could be do to spawn an instance with the stage or prod config, as part of CI ideally, and check that the heartbeat is green there.

+1 to this

---

• Prerequistes:

  • Jira Project exists...
    • in prod: if not confirm the use of jira (is this just general interest--if so suggest using JBI Bin[JB] project tag for testing.
    • in nonprod: if not create a support ticket to reflect the project to mozit-test (and it's settings/configurations
      /etc)
  • Jira Permissions are set (default permissions: create/edit/delete issue, create comment, browse_project)
    • in prod: if project is new to JBI--it will likely need permissions updated; file a support request for the above permissions
    • in nonprod: if project is new to JBI--it will likely need permissions updated; file a support request for the above permissions (combine with other support request if project is new to nonprod env)
  • Expected Bugzilla Product Webhooks are being tracked...
    • in prod: check jbi webhooks--if expected product is missing create new webhook (currently manual--one way we can display this could be to present an endpoint that displays the results of the new Webhook GET API in BMO
    • in nonprod: same as in prod

• Testing

  • SysEng provides testing of default workflow; that is as follows:
    • Create Bug in Bugzilla with whiteboard-tag; confirm Jira issue creation and see_also in bugzilla--edit bug title, add new comment, close issue, reopen issue, then finally close issue; confirm JBI acts as expected.
    • Create Bug in Bugzilla without whiteboard-tag; edit bug to add tag--confirm Jira issue creation and see_also in bugzilla--edit bug title, add new comment, close issue, reopen issue, then finally close issue; confirm JBI acts as expected.
  • Beyond this testing; ask customer to test out their specific interactions/workflow--preferably in nonprod.