deployed site missing dependency

[thinkingfish]

Describe the bug
I have a Framework project that uses apache-arrow, this dependency is declared in package.json, and when I execute npm run dev the pages works fine. However, when I deploy it, either to observable cloud or to netlify, it throws an error about missing package:

RuntimeError: error loading dynamically imported module: https://rezolus-ui.iop.systems/_npm/[email protected]/_esm.js

Locally I can see this particular package being put under pages/.observablehq/cache/_npm during dev, but when I run npm run build, I don't see it under either dist/_npm or dist/_observablehq

[mbostock]

Are you intending to use an npm import or a node import?

An npm import does not need to be declared in your package.json, and is imported using the npm: protocol, e.g., npm:apache-arrow. A node import does need to be declared in your package.json (and installed using your package manager), and is imported as a bare module specifier, e.g., apache-arrow. It sounds like you are using an npm import, so you don’t need to declare apache-arrow in your package.json (but it won’t hurt, either).

It sounds like Framework isn’t statically detecting your import of apache-arrow. Can you share more details about how you are importing apache-arrow into your code? What does the import statement look like, and where does this import statement live?

[thinkingfish]

Oh just realized I should have replied instead of adding another comment. Hopefully not spamming you but in case you missed my other comment, I added some more details.

[thinkingfish]

Thanks for the clarification, I didn't know that distinction.

Originally I did not have an import statement for apache-arrow at all but I believe observablehq depends on it already, so it was cached and I was able to use statements like file.arrow(). I accidentally left the dependency in package.json, but it doesn't lead to a _node/ directory to be created in dist/.

I then removed the dependency in package.json, and all commands still executed successfully. However, the deployed version is looking for [BASE_URL]/_npm/[email protected]/_esm.js which isn't there.

I then added a import "apache-arrow"; statement in my index.js file, which when building creates the module under dist/_node/. However, since I'm indirectly invoking this module through the Inputs.file object, the runtime looks for the module under _npm which is not there.

So it seems this runs fine locally because auto-caching puts the module under src/.observablehq/cache/_npm, but somehow that doesn't happen when building.

[thinkingfish]

Just checked the source—apache-arrow is imported by _observablehq/stdlib.js

async arrow(){const[e,r]=await Promise.all([import("../_npm/[email protected]/_esm.js"),c(this)]);

[mbostock]

If you’re using the Arrow built-in, then yes, you don’t need to declare a dependency in your package.json file, and you definitely don’t want a import "apache-arrow" (because that will load a second copy of Apache Arrow from your node_modules that you’re not using). The Arrow built-in is equivalent to:

import * as Arrow from "npm:apache-arrow";

so you can use that if you need to reference Arrow directly (rather than just using FileAttachment.arrow).

Can you try clearing your cache in src/.observablehq/cache and restarting your preview server. Does it still work in that case?

[mbostock]

Also, where and how are you calling file.arrow()? It seems like Framework isn’t noticing that you’re calling file.arrow() and so therefore isn’t registering the dependency on npm:apache-arrow.

[thinkingfish]

My current workaround is indeed using a dummy import "npm:apache-arrow"; statement even though I don't call Arrow explicitly. This brings the module under _npm and that solves the problem. (See http://rezolus-ui.iop.systems/)

I nuked the src/.observablehq directory and it builds fine.

I have two JS cells in my index.md that are involved in data ingestion, as below.

const file = view(
  Inputs.file({ label: 'Metric file', accept: '.arrow', required: true })
);

const data = file.arrow();

Attaching the console output of my npm run dev after nuking the cache. Note toward the end of the log I locally picked a file which triggered the async download of the arrow package.
dev.log

[mbostock]

Okay, yeah, Framework doesn’t know that file there is a FileAttachment (because it’s dynamically constructed by Inputs.file), and so Framework doesn’t know that you depend on Apache Arrow. It can only detect code of the form FileAttachment("foo").arrow().

Adding an explicit import of npm:apache-arrow is the recommended fix for this problem!