[BUG] Unable to install untracked peer optional dependency

[baileympearson]

Is there an existing issue for this?

• I have searched the existing issues

This issue exists in the latest npm version

• I am using the latest npm

Current Behavior

npm i <dependency name> succeeds with exit code 0 but the resultant dependency is not present, if <dependency name> is an untracked (not included in prod or dev dependencies) optional peer dependency.

I believe this is caused by #8431.

Expected Behavior

npm i <dependency name> should always install the dependency if the dependency has no version incompatibility issues.

Steps To Reproduce

Easiest way to see this is working in the mongodb package, because this scenario arose in our CI.

1. Clone node-mongodb-native.
2. run npm i
3. install kerberos: npm i kerberos
4. attempt to use kerberos: `node -p "require('kerberos')":

node:internal/modules/cjs/loader:1408
  throw err;
  ^

Error: Cannot find module 'kerberos'

Environment

• npm: 11.5.0+
• Node.js: doesn't matter
• OS Name: doesn't matter
• System Model Name: doesn't matter
• npm config: doesn't matter

[kchindam-infy]

@baileympearson i tried to reproduce the issue in a clean environment by installing mongodb and then installing kerberos separately using npm install kerberos. In my test, the package installed correctly, appeared in node_modulesand could be required without issue.
Here is my output:

➜  node-mongodb-native git:(main) npm install kerberos

added 2 packages, and audited 693 packages in 2s

109 packages are looking for funding
  run `npm fund` for details

4 vulnerabilities (1 low, 3 high)

To address all issues, run:
  npm audit fix

Run `npm audit` for details.
➜  node-mongodb-native git:(main) ✗ node -p "require('kerberos')"
<ref *1> {
  initializeClient: [Function (anonymous)],
  initializeServer: [Function (anonymous)],
  principalDetails: [Function (anonymous)],
  checkPassword: [Function (anonymous)],
  GSS_C_DELEG_FLAG: 1,
  GSS_C_MUTUAL_FLAG: 2,
  GSS_C_REPLAY_FLAG: 4,
  GSS_C_SEQUENCE_FLAG: 8,
  GSS_C_CONF_FLAG: 16,
  GSS_C_INTEG_FLAG: 32,
  GSS_C_ANON_FLAG: 64,
  GSS_C_PROT_READY_FLAG: 128,
  GSS_C_TRANS_FLAG: 256,
  GSS_C_NO_OID: 0,
  GSS_MECH_OID_KRB5: 9,
  GSS_MECH_OID_SPNEGO: 6,
  Kerberos: [Circular *1],
  version: '2.2.2',
  processes: { MongoAuthProcess: [class MongoAuthProcess] }
}
➜  node-mongodb-native git:(main) ✗ ls node_modules/kerberos
binding.gyp  build        HISTORY.md   lib          LICENSE      node_modules package.json README.md    src
➜  node-mongodb-native git:(main) ✗ 

[baileympearson]

Hey @kchindam-infy , which npm version are you using? The issue is only present npm 11.5.0+

[G-Rath]

hmm ok I think this has revealed a different bug(s?) in npm related to how it preserves the group when installing existing dependencies, so if you do e.g. npm install once with once as a dev dependency that's where it'll stay.

This also happens for peerDependencies assumingly because they're now installed by default, but that assumption is wrong if its an optional peer dependency since those only get installed if they're present in a group other than peerDependencies - I would say this is a bug in npm and that it should be ignoring the peer group if a dependency is also optional (aka, what my fix was in regard to pruning).

Possibly another bug in npm is that it seems to favor the existing peer group even if you explicitly use --save-prod, which I think is less critical but if it was working in the strictness sense you'd be able to use that here - as it is, you should be able to use --save-dev to work around this because that results in a different devOptional flag that my change isn't using, which I think itself is a bug but we can revisit that after dealing with this (my fix works fine with dev dependencies, doesn't seem to need to be checking devOptional)

[G-Rath]

ah ok so actually I think there's sort of an element of user error here as peer dependencies cannot be specified as production dependencies, only dev dependencies so for what you're trying to do you should be specifying --save-dev:

cli/workspaces/arborist/lib/add-rm-pkg-deps.js

Lines 17 to 33 in d4e56b2

	if (addSaveType === 'prod') {
	// a production dependency can only exist as production (rpj ensures it
	// doesn't coexist w/ optional)
	deleteSubKey(pkg, 'devDependencies', name, 'dependencies')
	deleteSubKey(pkg, 'peerDependencies', name, 'dependencies')
	} else if (addSaveType === 'dev') {
	// a dev dependency may co-exist as peer, or optional, but not production
	deleteSubKey(pkg, 'dependencies', name, 'devDependencies')
	} else if (addSaveType === 'optional') {
	// an optional dependency may co-exist as dev (rpj ensures it doesn't
	// coexist w/ prod)
	deleteSubKey(pkg, 'peerDependencies', name, 'optionalDependencies')
	} else { // peer or peerOptional is all that's left
	// a peer dependency may coexist as dev
	deleteSubKey(pkg, 'dependencies', name, 'peerDependencies')
	deleteSubKey(pkg, 'optionalDependencies', name, 'peerDependencies')
	}

I think this is made more confusing by npm install being the "install packages", "add new package(s)", and "update package(s)" command - so in this case npm is thinking "oh you want to update the dependency" and that's what it does (without installing it because its optional), but actually you want to add this optional dependency as a dev dependency so you can use it...

[baileympearson]

I think this is made more confusing by npm install being the "install packages", "add new package(s)", and "update package(s)" command - so in this case npm is thinking "oh you want to update the dependency" and that's what it does (without installing it because its optional), but actually you want to add this optional dependency as a dev dependency so you can use it...

Could you clarify this? If kerberos isn't a package currently tracked in either dev or prod dependencies but is an optional peer dependency, I'd assume the npm i would result in "add new package". Is that incorrect?

r.e. using --save-dev: I see the rationale for not allowing optional prod dependencies, that does feel inherently contradictory. Installing kerberos as a dev dependency would work for our use case and we can make the change.

We do feel that the experience of silently pruning the installed dependency is a bad UI. In this case, npm i kerberos succeeds with exit code 0 and no errors but the package is pruned. Even npm i -ddd kerberos doesn't print anything in the logs to indicate that something has caused kerberos to be pruned. Would printing a warning in this scenario be feasible or adding something to the debug logs indicating why the packages were pruned?

[G-Rath]

Could you clarify this? If kerberos isn't a package currently tracked in either dev or prod dependencies but is an optional peer dependency, I'd assume the npm i would result in "add new package". Is that incorrect?

The dependency is being tracked as a peer dependency and by definition production and peer dependencies are mutually exclusive which is regardless of the "optional"ness of the peer (really for the same reason: as soon as a dependency is in dependencies it is no longer peer or optional).

In order for npm install to function as add, install, and update, it effectively has a default group of "existing or production" and so when it sees your dependency existing already as a peer dependency it preserves that but then doesn't install it because its an optional peer and you've not specified it as a dev dependency.

We do feel that the experience of silently pruning the installed dependency is a bad UI. In this case, npm i kerberos succeeds with exit code 0 and no errors but the package is pruned

I think it could be good to have a warning or error when you do npm install for an optional peer dependency without specifying --save-dev, but I'll let the maintainers make the final call on that - one possible complication is that by the time npm can know a package is an optional peer it might no longer know that you've run "install" rather than "update" as under the hood they all dovetail into the same set of functions just with different options

[acburdine]

not sure if it's related to the original posted issue, but #8431 does appear to have broken something in my npm workspaces setup indirectly. I wrote up a gist with a minimal repro case: https://gist.github.com/acburdine/ca28093a0772434cfc24119dffff5c14.

the tl;dr is that npm at least since latest v10 seems to use the "last" encounter with a particular dependency as the final resolution of whether it's optional/dev/peer/etc, so my particular optional (non-peer) dev dep is getting flagged as an optional peer dep because the "last" workspace in my setup happens to only depend on it as an optional peer.

I'll look around to see if this bug is noted elsewhere in the repo, but thought it worth noting here because the change in #8431 when combined with the noted bug breaks our setup 😬

[chabb]

Same here, it seems #8431 has broken our ci. We've some projects that fail, as some optional dependencies are removed

[olivier-lxo]

We're having similar issues with peer dependencies not being install anymore with the upgrade to npm@11.5