feat: ability to execute external providers

[leseb]

What does this PR do?

Providers that live outside of the llama-stack codebase are now
supported.
A new property external_providers_dir has been added to the main
config and can be configured as follow:

external_providers_dir: /etc/llama-stack/providers.d/

Where the expected structure is:

providers.d/
  inference/
    custom_ollama.yaml
    vllm.yaml
  vector_io/
    qdrant.yaml

Where custom_ollama.yaml is:

adapter:
  adapter_type: custom_ollama
  pip_packages: ["ollama", "aiohttp"]
  config_class: llama_stack_ollama_provider.config.OllamaImplConfig
  module: llama_stack_ollama_provider
api_dependencies: []
optional_api_dependencies: []

Obviously the package must be installed on the system, here is the
llama_stack_ollama_provider example:

$ uv pip show llama-stack-ollama-provider
Using Python 3.10.16 environment at: /Users/leseb/Documents/AI/llama-stack/.venv
Name: llama-stack-ollama-provider
Version: 0.1.0
Location: /Users/leseb/Documents/AI/llama-stack/.venv/lib/python3.10/site-packages
Editable project location: /private/var/folders/mq/rnm5w_7s2d3fxmtkx02knvhm0000gn/T/tmp.ZBHU5Ezxg4/ollama/llama-stack-ollama-provider
Requires:
Required-by:

Closes: #658

[leseb]

TODO: add doc instruction on how to write an external provider and configure a llama-stack instance of it.

[booxter]

Would it make sense to use some manager for entrypoints like stevedore that would refer to python package names instead of file paths?

[leseb]

stevedore

Never heard of it, I'll do some more exploration. Thanks!

[leseb]

Closing for now, discussed offline with @ashwinb and we prefer to have an RFC first, which will supersede #658. Thanks!

[ashwinb]

cc @raghotham let's consider this and allow folks to "bring your own providers to the Stack"

[ashwinb]

See inline comment about where the extension point should be executed.

[booxter]

This seems like a good direction. I wonder if the YAML format for provider specs should be adopted for in-tree providers too, to avoid two separate code paths to define these. (python vs yaml)

[booxter]

It's fine. One thing that could be improved is de-duplicating the code for test provider. Basically, instead of copy pasting the implementation to another directory, we could prepare a tmpdir with the necessary files (most of them copied verbatim from the "real" ollama provider subtree). Otherwise, we'll have to patch the test provider every time we patch the "real" provider.)

[leseb]

It's fine. One thing that could be improved is de-duplicating the code for test provider. Basically, instead of copy pasting the implementation to another directory, we could prepare a tmpdir with the necessary files (most of them copied verbatim from the "real" ollama provider subtree). Otherwise, we'll have to patch the test provider every time we patch the "real" provider.)

Yes, I was trying to avoid that with symlinks first (but the package install would fail). Agree that the duplication is not ideal I can give this another try but would like to move fw with this too :)

[leseb]

It's fine. One thing that could be improved is de-duplicating the code for test provider. Basically, instead of copy pasting the implementation to another directory, we could prepare a tmpdir with the necessary files (most of them copied verbatim from the "real" ollama provider subtree). Otherwise, we'll have to patch the test provider every time we patch the "real" provider.)

Yes, I was trying to avoid that with symlinks first (but the package install would fail). Agree that the duplication is not ideal I can give this another try but would like to move fw with this too :)

@booxter no more dups!

[ashwinb]

We want the dependencies of the config class to be minimal, it should not pick up dependencies of the implementation so lots of automation can work with typed configs without needing dependency bloat. At the very least, we can say this in the documentation here for now. Later, it might be a good idea to enforce it via a test.

[ashwinb]

amazing! thank you so much and many apologies for being AWOL. 🙏

[leseb]

amazing! thank you so much and many apologies for being AWOL. 🙏

I’ve noted your latest review comments, assuming they’re non-blocking, I’ll go ahead and merge this. I’ll follow up with a separate PR to address the minor nits. Thanks again!

[leseb]

Just gave it a final spin after a rebase, and it's all green! Merging now :)