Query supported devices before graph compilation

[anssiko]

[ Spun off from PR #809 review comment by @fdwr and a discussion with @inexorabletash, @reillyeon, @RafaelCintron, @philloooo -- thanks! ]

(Emphasis mine:)

This afternoon we were talking with {@inexorabletash, @reillyeon, @RafaelCintron, Phillis} about the likely need for a caller to know whether a particular device is supported or not, because an app may want to (if say GPU is not supported) use a different more performant fallback than for WebNN to silently fall back to CPU. For example, if GPU was unavailable (even though you preferred high performance), then it might be faster to execute the model with WebGPU shaders than WebNN CPU, or it might be okay to use CPU, but the app could load a different model that's more CPU-friendly, if it knew that was the case.

This need is the reverse of what we currently have, querying the context for what kind of devices it actually supports rather than prescribing a preferred device type. Now, preferred device type prescription may return someday as a weaker hint (or list of hints, or list of antihints...) as we learn more about usage/needs, but for either case prescription or post-query, having device enums could be useful. 🤔

To kick off this discussion, I'll start with a simple and naive API further abstracted from the GPUDevice.features setlike interface:

// only MLPowerPreference supported at context creation time
const context = await navigator.ml.createContext({ powerPreference: 'high-performance' });

// query the newly created context
if (context.devices.has('cpu') && context.devices.has('npu')) {
   // context supports both CPU and NPU, choose the model accordingly
   // how the work is split across CPU and NPU is an implementation detail
} else if (!context.devices.has('gpu')) {
   //  no GPU available, perhaps try WebGPU?
} else {
   // ...
}

To help tease out more informed proposals, I'd like folks to test this imaginary API against the following:

• Address key use cases
• Design based on user needs, not the underlying capability
• Implementability / support the full range of devices and platforms
• Use care when exposing APIs for selecting or enumerating devices
• Is extensible and future-proof
• Exposed data is minimized
• Other considerations?

@mwyrzykowski would this work with CoreML MLComputeUnits? A subset of possible MLDeviceType combinations is supported but there's also a catch-all all?

How about other backends?

This issue welcomes further discussion and proposals on the topic of post-context creation query mechanism for supported devices.

[mwyrzykowski]

@mwyrzykowski would this work with CoreML MLComputeUnits? A subset of possible MLDeviceType combinations is supported but there's also a catch-all all?

It would work but perhaps of limited usefulness? context.devices.has(...) would return true but there is no indication which device a particular model will run on. E.g., usage of float32 arithmetic will prevent the model from running on Apple's NPUs. Whereas on possibly another computing device, usage of float16 arithmetic could prevent running on a certain processor unit.

I am not sure how we answer the question from #809 (comment) without providing some fine grained limits / capabilities as noted in #749. The concern there, mentioned in #749 (comment) was the privacy / security aspect from my understanding.

[reillyeon]

Fine grained signals are not necessary to answer the question from #809 (comment). A developer who has developed GPU- and CPU-optimized models wants to implement this flowchart:

flowchart TD
    A[Device has GPU?] -->|Yes| B(Load GPU model)
    A -->|No| C(Load CPU model)
    B --> D(Can execute on GPU?)
    D -->|Yes| E(Execute on GPU)
    D -->|No| C
    C --> F(Execute on CPU)

Loading

This only requires knowing whether the GPU is an option for a given context (to avoid loading a GPU model if it has no chance of working) and whether a graph can run (or has run) on the GPU or not.

The actual flowchart may be more complex, as it could include non-WebNN fallbacks such as WASM and WebGPU as well as models with different system performance requirements.

[zolkis]

If we remove MLDeviceType (as in #809), the only way to do the flowchart above is trying to create a WebGPU context, and if that fails, then create a generic context - and then this issue argues for adding query support to those contexts.

Just to note, earlier Chai (?) mentioned we need a way to easily create a GPU context without a WebGPU device. We are losing that with #809. Querying context capabilities of generic contexts will not solve that.

It seems there is still a strong use case to query the platform capabilities and match that with the requirements of the model at hand. Moreover, I could imagine a more complex flowchart, where also want to query other context specific constraints.

We could go and create a generic context using the current options and then query for capabilities. But this does not guarantee getting a GPU context, even when one would be available.

I propose we walk a middle path, e.g. by saying:

const context = await navigator.ml.createContext({ 
    powerPreference: 'high-performance', 
    devicePreference: 'gpu'
});

This would only guarantee that GPU will be involved (e.g. a combination of GPU and NPU/CPU), otherwise it will fail.

Note that passing "npu" alone might be too little information. So I do I see the reason in passing other context creation options/constraints, too, as suggested in #749 (comment)

const context = await navigator.ml.createContext({ 
    powerPreference: 'low-power', 
    devicePreference: 'npu',
    supportLimits: { output: [ "float16" ] }
});

I am not sure do we also need the query function proposed there (the one that might make fingerprinting easier):

partial interface ML {
  record<USVString, MLOpSupportLimits> opSupportLimitsPerAdapter();
};

If possible, I'd try doing the options above without this query function, at the first attempt.
@mwyrzykowski would that work for you? @reillyeon?

If yes, we could do the following:

• I close Remove MLDeviceType #809 (i.e. cancel the PR) -- or modify it as follows.
• keep the MLDeviceType enum
• support a new context creation option as MLOpSupportLimits supportLimits, as proposed in #749 (comment)
• write the related algorithmic steps.

WDYT?

[anssiko]

Thanks for your swift responses and suggestions, everyone. (Markdown-friendly Mermaid is a great tool for quick diagrams, we should use it more often in the spec-land.)

@zolkis, I see consensus emerging on PR #809, so I'd like to see that through and work on the query mechanism on top of it. If we need to revive some bits from that PR later, that is OK.

[mwyrzykowski]

If we remove MLDeviceType (as in #809), the only way to do the flowchart above is trying to create a WebGPU context, and if that fails, then create a generic context - and then this issue argues for adding query support to those contexts.

Technically, there is nothing in the WebGPU specification which requires a GPU (it was intentionally designed this way).

const context = await navigator.ml.createContext({
powerPreference: 'low-power',
devicePreference: 'npu',
supportLimits: { output: [ "float16" ] }
});

If possible, I'd try doing the options above without this query function, at the first attempt. @mwyrzykowski would that work for you? @reillyeon?

The concept that an npu device has certain operations and a gpu device has others, or a cpu device can run all operations is something WebKit thinks should remain out of the WebNN specification and leave these details to the UA.

Foreseeably, one vendor's npu device could have the capabilities of another vendor's gpu device. Or one vendor's gpu has all the same operations / capabilities as the cpu with different performance characteristics.

I understand there are privacy concerns, but achieving this use case seems better suited to something like WebGPU's supportedLimits / supportedFeatures:
https://www.w3.org/TR/webgpu/#gpusupportedlimits
https://www.w3.org/TR/webgpu/#gpusupportedfeatures

Perhaps we could have a similar approach with allowances to maintain privacy?

[reillyeon]

I imagine the "Device has GPU?" check would be implemented by running,

const context = await navigator.ml.createContext({
    powerPreference: 'high-performance'
});
if (context.devices.has('gpu')) {
  // Yes, load GPU model.
} else {
  // No, load CPU model.
}

Given the constraints of the Core ML framework the "Can execute on GPU?" check needs to actually invoke the graph,

const [test_input, test_output] = await Promise.all([
    context.createTensor({...}),
    context.createTensor({...}),
]);
context.dispatch(graph, { input: test_input }, { output: test_output });
await context.readTensor(test_output);
if (graph.devices.has('gpu')) {
  // Yes.
} else {
  // No, load CPU model.
}

The naming and how exactly graph.devices gets populated is TBD but would be based on the history of previous inference runs.

[zolkis]

OK, so there seems to be a preference for creating a device-agnostic context, leave it to the UA to select the device(s) based on context options (currently only powerPreference) (any other hints to add to context options about device selection?), then apps would query the context for more detailed capabilities.

@mwyrzykowski
I understand there are privacy concerns, but achieving this use case seems better suited to something like WebGPU's supportedLimits / supportedFeatures:
https://www.w3.org/TR/webgpu/#gpusupportedlimits
https://www.w3.org/TR/webgpu/#gpusupportedfeatures

Sounds good, especially the latter, for our use case. So what about the following.

enum MLDeviceType {
  "cpu",
  "gpu",
  "npu"
};

interface MLDevices {
    readonly setlike<MLDeviceType>;
};

partial interface MLContext {
    [SameObject] readonly attribute MLDevices devices;
}

So then the examples by @anssiko and @reillyeon would work:

if (context.devices.has('gpu')) {

[huningxin]

@fdwr

This afternoon we were talking with @inexorabletash, @reillyeon, @RafaelCintron, @philloooo about the likely need for a caller to know whether a particular device is supported or not,

Thanks for the discussion. I'd like to understand more about the use case.

because an app may want to (if say GPU is not supported) use a different more performant fallback than for WebNN to silently fall back to CPU. For example, if GPU was unavailable (even though you preferred high performance), then it might be faster to execute the model with WebGPU shaders than WebNN CPU,

As I commented, WebNN implementation could use WebGPU shaders to execute graph on GPU for some devices where IHV-optimized GPU kernels are not available. Native ML frameworks are doing similar thing, for example ONNX Runtime native WebGPU execution provider. I suppose when high performance execution is preferred, silently falling back to sub-optimal CPU execution is an implementation issue.

or it might be okay to use CPU, but the app could load a different model that's more CPU-friendly, if it knew that was the case.

May I know what aspects make a model more CPU-friendly? Does it use CPU-friendly operators or data types? Are there any gaps indicating a WebNN implementation to select CPU device when the app loads a CPU-friendly model?

[reillyeon]

May I know what aspects make a model more CPU-friendly? Does it use CPU-friendly operators or data types?

I am not an expert here but my understanding is that a CPU-friendly model may use different data types or simply be smaller so that it can meet the application's performance needs while providing acceptably reduced inference quality.

Are there any gaps indicating a WebNN implementation to select CPU device when the app loads a CPU-friendly model?

I think there was some discussion of adding a specific "cpu-only" flag to context creation if the current power preference flags are insufficient.

[mwyrzykowski]

Sounds good, especially the latter, for our use case. So what about the following.

enum MLDeviceType {
"cpu",
"gpu",
"npu"
};

interface MLDevices {
readonly setlike;
};

partial interface MLContext {
[SameObject] readonly attribute MLDevices devices;
}
So then the examples by @anssiko and @reillyeon would work:

if (context.devices.has('gpu')) {

@zolkis What information does the 'cpu', 'gpu', 'npu' label really provide? The concern is it does not provide information over the chips capabilities.

Instead, we could have a mechanism for saying that a given UA can, for example:

• support the following MLOperandDataType values (e.g., float16, float32, int64, float64)
• create tensors of maximum shape = K
• other distinguishing limits?

whether or not the UA is running on a device with an npu or gpu is not really meaningful. One vendor's NPU could be equal in ML capabilities to another vendor's GPU for instance.

[zolkis]

@mwyrzykowski yes, the idea for exposing capabilities has been recorded. I was thinking to remove device types (PR #809), then experiment with how to expose the capabilities, and/or other mappings between context options/properties and device selection.

But it seems that #809 made people re-thinking what we'd lose and what we'd need, spanning this issue.

IIUC, the capabilities approach does not necessarily conflict with having another mechanism to poll whether a given type (as standardized string label) of accelerator/device is associated with the context just created on the given platform/environment.
In my current understanding, in most use cases CPU/GPU/NPU still have meanings (as well as symbolic mappings like low-power and high-performance), though it might be blurry and conflated with capabilities, and NPU alone is a rather diverse category.

I agree that knowing the current device types is not enough information, but I don't think this is useless information, at least today.

If you have a strong view on avoiding CPU/GPU/NPU device names altogether and focus only on capabilities, we need to discuss this during the next call (@anssiko please check). The problem with capabilities is that they are not well defined ATM, and appear to be (far) more complex to use. We need time to figure that out. Yet, I'd agree this might be the most flexible approach, and also the most exact approach.

[inexorabletash]

Given discussion today on the telecon and @mwyrzykowski's comments above, WDYT about something like this:

const support = await context.querySupport({
  dataTypes: ['float16', 'int8'],
  maximumRank: 6,
  operators: ['lstm', 'hardSwish'],
});
console.log(support); // "optimized" or "fallback"