Gatsby Adapters causes regressions & issues with headers & redirects for NETLIFY builds on >=5.12.0

[techfg]

Preliminary Checks

• This issue is not a duplicate. Before opening a new issue, please search existing issues: https://github.com/gatsbyjs/gatsby/issues
• This issue is not a question, feature request, RFC, or anything other than a bug report directly related to Gatsby. Please post those things in GitHub Discussions: https://github.com/gatsbyjs/gatsby/discussions

Description

The forced usage of Gatsby Adapters has caused several regressions and issues related to headers & redirects for builds running on NETLIFY. As of v5.12.0, Gatsby will detect that the build is running on NETLIFY and add the gatsby-adapter-netlify adapter to the build and remove the gatsby-plugin-netlify plugin without any stock way to "opt-out" of the behavior.

Unfortunately, the gatsby-adapter-netlify does not have feature parity with gatsby-plugin-netlify and also contains a few bugs. The combination of these two things results in build output being different for anyone building on NETLIFY.

A couple of references to bugs/issues already filed:

1. gatsby-adapter-netlify not respecting "force" option passed to createRedirect #38541
2. gatsby-adapter-netlify not generating headers for redirects #38538

After digging through the output and code some more, I believe this is a much broader set of issues than those two issues alone so creating this issue as an umbrella. If the preference is to have a single issue for each individual item identified, please just let me know.

The issues discovered are:

1. Headers on redirects are not emitted
2. force is not honored (needs to include !)
3. external redirects have a forced / added to the beginning of them (Netlify supports domain redirects - example: domain alias to redirect to primary site).
4. Conditions on redirects are not honored
5. The following features of gatsby-plugin-netlify are not available and therefore have no effect:
   1. allPageHeaders
   2. transformHeaders
   3. mergeSecurityHeaders
   4. mergeCachingHeaders
   5. generateMatchPathRewrites
6. Link paths are not converted to their corresponding hash
7. Client-only routes have incorrect header
8. pathPrefix & assetPrefix
   1. Not honored in headers (note that gatsby-plugin-netlify had some issues in this area as well).
   2. Should asset routes even be passed through to the adapter in RoutesManifest when assetPrefix is an absolute URL?
9. _headers file size could become significantly large, especially on larger sites - gatsby-adapter-netlify emits explicit headers for each route path as opposed to passing wildcards, etc. through to the adapter. This comes with three potential downsides:
   1. Does not give the adapter, which understands its own platform best, a way to optimize headers. While each platform will likely be able to process explicit path's, there may be optimizations that the platform can take with wildcards, supporting regex's, etc. that would better serve its implementation.
   2. Could lead to performance degradation depending on the way the platform evaluates path matching on headers at runtime. For example, if the platform looks through every "header" key for a match of a page path to determine all the header matches before setting the headers on the response, a large file would slow down page requests. This would take full evaluation of the platform to determine true impact (if any) but, as the first item mentions, the platform adapter should know best how to optimize the headers for itself.
   3. Could result in failed builds because of the file size. For example, Netlify specifically states that if the combined size of _redirects & netlify.toml is too large, it could fail the deploy. While it doesn't mention this for _headers, the same is likely true (would require validation). In the repro, the file size of _headers using gatsby-plugin-netlify is 2,408 bytes while the file size using gatsby-adapter-netlify is 59,515 bytes (using the extrapages build which has ~105 pages), nearly a 25x increase.

All of the above (and possibly some more I missed), will impact customers relying on any of these as soon as they build on Netlify with a version >= 5.12.0.

As mentioned here as it relates to builds breaking, 5.12.0 likely should have been a major version update as all of these are breaking changes and since most likely have a semver of ^5.x.x, customers relying on any of these features/functionality will be impacted if building on NETLIFY.

Most of the above issues are problems/missing functionality in the core Gatsby Adapter logic, not in the gatsby-adapter-netlify adapter itself. For example, when IRedirectRoute is created, force is not passed through so the adapter doesn't receive the value even though there is code in the adapter to process it - same holds true for the conditions property. Additionally, createRedirect allows for any set of [key: string]: unknown which aren't passed through (they wouldn't be used in this case but they should be passed through since createRedirect API and IRedirectRoute support it).

I had created feature requests for the ability to disable adapters and add allPageHeaders & transformHeaders, however the more that I've run up against, after thinking it through, I believe these aren't features but regression defects given Adapters was a minor release version and not a major. At the very least, adding a way to disable adapters as soon as possible is likely warranted.

Workarounds

In the meantime, for those that come across this, there are a few options to avoid the above issues:

1. Stay on a version of Gatsby < 5.12.0 - This is the simplest and recommended approach if any of the above would effect your site
2. Implement the noop Adapter solution that I describe in the discussion
3. Define your own redirects & headers via (_redirects/_headers) instead of relying on Gatsby adapter to generate. You can place these files in static directory and they will get copied over to public during the build. Gatsby adapter will merge in any changes it makes to these files for a combined output so you could define everything you need or just the redirects/rules that Gatsby adapter doesn't support properly.

Note

For any anyone (especially Gatsby cloud customers moving over) moving to Netlify and planning on using transformHeaders in gatsby-plugin-netlify (same existed in gatsby-plugin-gatsby-cloud), note that there is a bug where the path parameter is not passed as expected. Assuming the PR I submitted gets merged, then workaround 1 or 2 will achieve the equivalent output as something like gatsby-plugin-gatsby-cloud would have.

I have created a full-set of repros that demonstrate three scenarios related to the output of headers & redirects - again, this only impacts builds on NETLIFY:

1. No Adapter - What the headers & redirects are when no adapter is used and gatsby-plugin-netlify is configured

• main branch
• codesandbox

2. With Adapter - What the exact same site & config from No Adapter would generate for headers & redirects

• with netlify adapter branch
• codesandbox

3. WIth Adapter & Gatsby Headers Config - As much as possible, use the headers config to replicate some of the functionality that is lost with Adapters compared to gatsby-plugin-netlify

• with netlify adapter and headeres config branch
• codesandbox

Each branch has the following files in root directory:

1. npm run build - Standard site
   1. sanitized_headers - The output of _headers after each build
   2. sanitized_redirects - The output of _redirects after each build
2. npm run build:pathprefix - Includes a path prefix
   1. sanitized_headers_pathprefix - The output of _headers after each build
   2. sanitized_redirects_pathprefix - The output of _redirects after each build
3. npm run build:assetprefix - Includes a path & asset prefix
   1. sanitized_headers_assetprefix - The output of _headers after each build
   2. sanitized_redirects_assetprefix - The output of _redirects after each build
4. npm run build:extrapages - Standard site with 100 extra pages
   1. sanitized_headers_extrapages - The output of _headers after each build
   2. sanitized_redirects_extrapages - The output of _redirects after each build

I have manually adjusted these files from their original to align sort order some. The output between using an adapter & gatsby-plugin-netlify is quite different (e.g., plugin uses /*, adapter emits same headers for each path when /* is used) but hoping this makes it a little more readable. You can review the PRs to see the differences between building with just gatsby-plugin-netlify

Reproduction Link

https://github.com/techfg/gatsby-adapter-issues

Steps to Reproduce

Run one of the four builds in all three branches and compare the output of public/_headers and public/_redirects.

Expected Result

The headers & redirects in a build that uses gatsby-adapter-netlify should be same as build using gatsby-plugin-netlify (or better in some cases like security improvements & issues with prefix paths).

Actual Result

The headers & redirects are not identical resulting in different build output without any changes to code/config of the site due to Adapters not being an "opt-in" feature.

Environment

System:
    OS: Linux 5.15 Ubuntu 20.04.6 LTS (Focal Fossa)
    CPU: (20) x64 12th Gen Intel(R) Core(TM) i9-12900HK
    Shell: 5.0.17 - /bin/bash
  Binaries:
    Node: 18.17.1 - ~/.nvm/versions/node/v18.17.1/bin/node
    Yarn: 1.22.19 - ~/.nvm/versions/node/v18.17.1/bin/yarn
    npm: 9.8.1 - ~/.nvm/versions/node/v18.17.1/bin/npm
  Browsers:
    Chrome: 116.0.5845.179
  npmPackages:
    gatsby: ^5.11.0 => 5.12.4
    gatsby-adapter-netlify: ^1.0.0 => 1.0.0
    gatsby-plugin-netlify: ^5.1.0 => 5.1.0
  npmGlobalPackages:
    gatsby-cli: 5.12.1

Config Flags

N/A

[zachmerrill]

I'm getting error latestAdapters.find is not a function almost every build on Netlify too. I keep having to clear my cache and retry.

[techfg]

Found a couple more issues...

1. Changed title to ...causes regressions & issues... from ...causes regressions...
2. Added Issues 8 & 9

[laradevitt]

After upgrading one of our sites to 5.12.0:

(time period is between "clean" builds, about 26 days)

• "header rules processed" went from 0 to 563
• cached dependencies went from 1.1 GB to 25.2 GB
• build minutes from 2m50s to 14m9s

Yikes! Not sure where to go with this other than to downgrade...

[seanloock]

Ever since Netlify started automatically installing gatsby-adapter-netlify our custom Template page is being generated as [...].html instead of index.html resulting in a 404 when viewing page(s) from built site

The page is created in gatsby-node.js as follows

  const CheckoutPageTemplate = path.resolve(`src/templates/checkout.tsx`)
  createPage({
    path: process.env.GATSBY_PREFIX_PATH
      ? `${process.env.GATSBY_PREFIX_PATH}/checkout/`
      : "/checkout/",
    matchPath: process.env.GATSBY_PREFIX_PATH
      ? `${process.env.GATSBY_PREFIX_PATH}/checkout/*`
      : "/checkout/*",
    component: CheckoutPageTemplate,
  })

Can resolve this by renaming the [...].html file to index.html after generation or locking Gatsby to v5.10.0

However, both of these solutions are quite hacky and not ideal in long run

[henrik-m]

We were hit by this as well and only found this in production when we discovered that all URLs which supported clientside routes were delivered with 404 status codes instead of 200 when accessed via URL.

Simple repro with latest versions of gatsby (5.13.1) and gatsby-adapter-netlify (1.1.1):
Repository: https://gitlab.com/henrik-m/gatsby-redirect-bug-repro
Published Page: https://main--verdant-granita-458d34.netlify.app/en/settings/

The settings page (which is not even a client-only page) and all client-only pages below it respond with a 404 status code when directly accessed.

Workaround: Downgrade to gatsby 5.11.x and gatsby-plugin netlify:
Merge Request: https://gitlab.com/henrik-m/gatsby-redirect-bug-repro/-/merge_requests/1
Deploy Preview: https://deploy-preview-1--verdant-granita-458d34.netlify.app/en/settings/

The settings pages of the deploy preview are delivered with a 200 status code, as they should be.

[JamesToohey]

Adding our tale and some some keywords in here to help others, we were seeing builds intermittently failing on Netlify with Gatsby 5.12.

We had a JSON file imported in gatsby-node.ts that was being compiled by webpack (which can't be configured for this particular part of the build process...) at build time into [name].[contenthash].js, but when this file was accessed slightly later in the build process contenthash was entirely different, meaning the file couldn't be found.

Clearing the Netlify build cache solved this temporarily, but it appears that downgrading to 5.11 has resolved it. I have no idea if it's related to this plugin, but can't find a more relevant place to leave this comment and am unsure how to reproduce in order to open a separate issue.