Serving SQLite database files with sql.js-httpvfs broken due to gzip encoding on HEAD requests

[Milegoo]

Select Topic Area

Question

Body

Hello,

I have been using GitHub Pages to serve SQLite database files for a client-side app with sql.js-httpvfs, which requires HTTP Range requests to read parts of the database file.

Until recently, this setup worked correctly. However, now the sql.js-httpvfs worker throws an error:
Length of the file not known. It must either be supplied in the config or given by the HTTP server.

Upon inspecting the HTTP response headers, I see that HEAD requests respond with content-encoding: gzip. According to sql.js-httpvfs, this breaks range requests because the server returns compressed data on HEAD but serves uncompressed data on range requests, making the byte length calculation incorrect.

Example response headers from a HEAD request:

content-encoding: gzip accept-ranges: bytes content-length: 1288

This behavior causes sql.js-httpvfs to fail when trying to read the file size and ranges properly.

Could you please confirm if there has been a recent change in how GitHub Pages serves static files or handles gzip encoding on HEAD and Range requests? Is there a recommended workaround to properly serve files that need range requests without gzip interfering?

Thank you very much for your help!

[bPavan16]

Answer: GitHub Pages and HTTP Range Requests with Gzip

GitHub Pages uses a CDN (typically backed by Fastly) that automatically applies gzip compression to certain file types, even for HEAD requests. This can cause issues when a client relies on accurate byte-level control, such as with sql.js-httpvfs.

What's happening?

• sql.js-httpvfs expects the HEAD response to reflect the actual byte length of the file.
• GitHub Pages responds to HEAD requests with Content-Encoding: gzip, which means the Content-Length represents the compressed size.
• However, Range requests return uncompressed content (as expected), causing a mismatch.

This inconsistency leads to the error:
"Length of the file not known. It must either be supplied in the config or given by the HTTP server."

Has GitHub Pages behavior changed?

Yes, GitHub Pages and its underlying CDN may update behavior periodically, including how it handles caching and compression. If this setup previously worked, it’s possible a recent change now forces gzip encoding on HEAD responses by default.

Workarounds

1. Self-host the file

Host the SQLite file on a server where you can fully control HTTP headers, including disabling gzip for specific routes.

2. Manually set the file length

If you know the exact size of the SQLite file (in bytes), you can pass it directly in the sql.js-httpvfs config:

{
  fileSize: 1234567 // exact byte size of the SQLite file
}

[bPavan16]

Please mark it as answer if you feel it helpful

[rahulps1000]

Hello,

Thank you for providing detailed context and headers—this is a great question and an increasingly relevant scenario for client-side applications relying on byte-accurate file delivery.

---

Recent Changes & GitHub Pages Behavior

GitHub Pages serves static assets behind a CDN, which may apply optimizations like transparent gzip compression. In recent months, there have been updates on how HEAD and GET/Range requests are processed, particularly regarding content-encoding. If you notice content-encoding: gzip returned on HEAD requests, but not on Range requests (which remain uncompressed), this can indeed cause byte length mismatches for libraries (like sql.js-httpvfs) that depend on consistent file size reporting for HTTP Range operations.

This discrepancy—compressed on HEAD but uncompressed on Range—means the content-length header doesn't always reflect the true byte size required for range-based file access.

---

Workarounds & Recommendations

1. Disable Compression for Database Files

GitHub Pages/CDN does not allow end-user configuration to disable compression for specific file types. All files may be subject to gzip if the CDN considers it appropriate, especially for .db or other non-standard static files.

2. Supply File Length in Client Config

As a workaround, sql.js-httpvfs allows you to explicitly provide the file length in your config object:

{
  //...
  fileLength: 123456 // Actual byte size of your SQLite file
}

You can determine this by checking the uncompressed file size locally or via a build step. This bypasses the need for HEAD content-length accuracy.

3. Serve From a Custom Domain or Different Host

If you need stricter control over HTTP headers (including compression), consider using an alternative static hosting provider (e.g., Cloudflare Pages, Netlify, AWS S3 with static hosting) where you can configure headers to ensure no compression is applied to database files.

4. File Suffix Workarounds

Some users have reported success by serving files with uncommon extensions (e.g., renaming .sqlite to .bin) to discourage automatic gzip. However, this is not officially supported and may break in the future.

---

Summary

• GitHub Pages/CDN behavior can cause content-encoding: gzip on HEAD but not Range requests, which breaks byte-accurate range operations.
• You cannot currently configure GitHub Pages to disable gzip for specific files.
• Recommended: Explicitly supply the uncompressed byte length to sql.js-httpvfs as a workaround.
• For long-term stability, consider static hosting that allows custom header management for your assets.

If you have further questions or need help with alternative hosting setups or automation for file size detection, feel free to ask!

[bdon]

I'm encountering a related issue with gzip encoding on HTTP Range (206) responses, specific to Firefox.

Minimal reproduction of the issue here, which only fails on the combination of GitHub Pages server + Firefox browser: https://github.com/bdon/ghpages-firefox-range-bug