Skip to content

Commit 66a3618

Browse files
lesebleseb
committed
feat: ability to execute external providers
Providers that live outside of the llama-stack codebase are now supported. A new property `external_providers_dir` has been added to the main config and can be configured as follow: ``` external_providers_dir: /etc/llama-stack/providers.d/ ``` Where the expected structure is: ``` providers.d/ inference/ custom_ollama.yaml vllm.yaml vector_io/ qdrant.yaml ``` Where `custom_ollama.yaml` is: ``` adapter: adapter_type: custom_ollama pip_packages: ["ollama", "aiohttp"] config_class: llama_stack_ollama_provider.config.OllamaImplConfig module: llama_stack_ollama_provider api_dependencies: [] optional_api_dependencies: [] ``` Obviously the package must be installed on the system, here is the `llama_stack_ollama_provider` example: ``` $ uv pip show llama-stack-ollama-provider Using Python 3.10.16 environment at: /Users/leseb/Documents/AI/llama-stack/.venv Name: llama-stack-ollama-provider Version: 0.1.0 Location: /Users/leseb/Documents/AI/llama-stack/.venv/lib/python3.10/site-packages Editable project location: /private/var/folders/mq/rnm5w_7s2d3fxmtkx02knvhm0000gn/T/tmp.ZBHU5Ezxg4/ollama/llama-stack-ollama-provider Requires: Required-by: ``` Signed-off-by: Sébastien Han <[email protected]>
1 parent cfd30d2 commit 66a3618

File tree

8 files changed

+677
-9
lines changed

8 files changed

+677
-9
lines changed

‎.github/workflows/test-external-providers.yml‎

Lines changed: 89 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,89 @@
1+
name: Test External Providers
2+
3+
on:
4+
push:
5+
branches: [ main ]
6+
pull_request:
7+
branches: [ main ]
8+
9+
jobs:
10+
test-external-providers:
11+
runs-on: ubuntu-latest
12+
steps:
13+
- name: Checkout repository
14+
uses: actions/checkout@v4
15+
16+
- name: Install uv
17+
uses: astral-sh/setup-uv@v5
18+
with:
19+
python-version: "3.10"
20+
21+
- name: Install Ollama
22+
run: |
23+
curl -fsSL https://ollama.com/install.sh | sh
24+
25+
- name: Pull Ollama image
26+
run: |
27+
ollama pull llama3.2:3b-instruct-fp16
28+
29+
- name: Start Ollama in background
30+
run: |
31+
nohup ollama run llama3.2:3b-instruct-fp16 > ollama.log 2>&1 &
32+
33+
- name: Set Up Environment and Install Dependencies
34+
run: |
35+
uv sync --extra dev --extra test
36+
uv pip install -e .
37+
38+
- name: Clone Ollama provider
39+
run: |
40+
git clone https://github.com/leseb/llama-stack-provider-ollama.git
41+
cd llama-stack-provider-ollama
42+
uv pip install .
43+
44+
- name: Create provider configuration
45+
run: |
46+
mkdir -p /tmp/providers.d/remote/inference
47+
cp llama-stack-provider-ollama/custom_ollama.yaml /tmp/providers.d/remote/inference/custom_ollama.yaml
48+
49+
- name: Wait for Ollama to start
50+
run: |
51+
echo "Waiting for Ollama..."
52+
for i in {1..30}; do
53+
if curl -s http://localhost:11434 | grep -q "Ollama is running"; then
54+
echo "Ollama is running!"
55+
exit 0
56+
fi
57+
sleep 1
58+
done
59+
echo "Ollama failed to start"
60+
ollama ps
61+
ollama.log
62+
exit 1
63+
64+
- name: Start Llama Stack server in background
65+
env:
66+
INFERENCE_MODEL: "meta-llama/Llama-3.2-3B-Instruct"
67+
run: |
68+
source .venv/bin/activate
69+
nohup uv run llama stack run llama-stack-provider-ollama/run.yaml --image-type venv > server.log 2>&1 &
70+
71+
- name: Wait for Llama Stack server to be ready
72+
run: |
73+
echo "Waiting for Llama Stack server..."
74+
for i in {1..30}; do
75+
if curl -s http://localhost:8321/v1/health | grep -q "OK"; then
76+
echo "Llama Stack server is up!"
77+
if grep -q "remote::custom_ollama from /tmp/providers.d/remote/inference/custom_ollama.yaml" server.log; then
78+
echo "Llama Stack server is using custom Ollama provider"
79+
exit 0
80+
else
81+
echo "Llama Stack server is not using custom Ollama provider"
82+
exit 1
83+
fi
84+
fi
85+
sleep 1
86+
done
87+
echo "Llama Stack server failed to start"
88+
cat server.log
89+
exit 1

‎docs/source/providers/external.md‎

Lines changed: 226 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,226 @@
1+
# External Providers
2+
3+
Llama Stack supports external providers that live outside of the main codebase. This allows you to:
4+
- Create and maintain your own providers independently
5+
- Share providers with others without contributing to the main codebase
6+
- Keep provider-specific code separate from the core Llama Stack code
7+
8+
## Configuration
9+
10+
To enable external providers, you need to configure the `external_providers_dir` in your Llama Stack configuration. This directory should contain your external provider specifications:
11+
12+
```yaml
13+
external_providers_dir: /etc/llama-stack/providers.d/
14+
```
15+
16+
## Directory Structure
17+
18+
The external providers directory should follow this structure:
19+
20+
```
21+
providers.d/
22+
remote/
23+
inference/
24+
custom_ollama.yaml
25+
vllm.yaml
26+
vector_io/
27+
qdrant.yaml
28+
safety/
29+
llama-guard.yaml
30+
inline/
31+
inference/
32+
custom_ollama.yaml
33+
vllm.yaml
34+
vector_io/
35+
qdrant.yaml
36+
safety/
37+
llama-guard.yaml
38+
```
39+
40+
Each YAML file in these directories defines a provider specification for that particular API.
41+
42+
## Provider Types
43+
44+
Llama Stack supports two types of external providers:
45+
46+
1. **Remote Providers**: Providers that communicate with external services (e.g., cloud APIs)
47+
2. **Inline Providers**: Providers that run locally within the Llama Stack process
48+
49+
### Remote Provider Specification
50+
51+
Remote providers are used when you need to communicate with external services. Here's an example for a custom Ollama provider:
52+
53+
```yaml
54+
adapter:
55+
adapter_type: custom_ollama
56+
pip_packages:
57+
- ollama
58+
- aiohttp
59+
config_class: llama_stack_ollama_provider.config.OllamaImplConfig
60+
module: llama_stack_ollama_provider
61+
api_dependencies: []
62+
optional_api_dependencies: []
63+
```
64+
65+
#### Adapter Configuration
66+
67+
The `adapter` section defines how to load and configure the provider:
68+
69+
- `adapter_type`: A unique identifier for this adapter
70+
- `pip_packages`: List of Python packages required by the provider
71+
- `config_class`: The full path to the configuration class
72+
- `module`: The Python module containing the provider implementation
73+
74+
### Inline Provider Specification
75+
76+
Inline providers run locally within the Llama Stack process. Here's an example for a custom vector store provider:
77+
78+
```yaml
79+
module: llama_stack_vector_provider
80+
config_class: llama_stack_vector_provider.config.VectorStoreConfig
81+
pip_packages:
82+
- faiss-cpu
83+
- numpy
84+
api_dependencies:
85+
- inference
86+
optional_api_dependencies:
87+
- vector_io
88+
provider_data_validator: llama_stack_vector_provider.validator.VectorStoreValidator
89+
container_image: custom-vector-store:latest # optional
90+
```
91+
92+
#### Inline Provider Fields
93+
94+
- `module`: The Python module containing the provider implementation
95+
- `config_class`: The full path to the configuration class
96+
- `pip_packages`: List of Python packages required by the provider
97+
- `api_dependencies`: List of Llama Stack APIs that this provider depends on
98+
- `optional_api_dependencies`: List of optional Llama Stack APIs that this provider can use
99+
- `provider_data_validator`: Optional validator for provider data
100+
- `container_image`: Optional container image to use instead of pip packages
101+
102+
## Required Implementation
103+
104+
### Remote Providers
105+
106+
Remote providers must expose a `get_adapter_impl()` function in their module that takes two arguments:
107+
1. `config`: An instance of the provider's config class
108+
2. `deps`: A dictionary of API dependencies
109+
110+
This function must return an instance of the provider's adapter class that implements the required protocol for the API.
111+
112+
Example:
113+
```python
114+
async def get_adapter_impl(
115+
config: OllamaImplConfig, deps: Dict[Api, Any]
116+
) -> OllamaInferenceAdapter:
117+
return OllamaInferenceAdapter(config)
118+
```
119+
120+
### Inline Providers
121+
122+
Inline providers must expose a `get_provider_impl()` function in their module that takes two arguments:
123+
1. `config`: An instance of the provider's config class
124+
2. `deps`: A dictionary of API dependencies
125+
126+
Example:
127+
```python
128+
async def get_provider_impl(
129+
config: VectorStoreConfig, deps: Dict[Api, Any]
130+
) -> VectorStoreImpl:
131+
impl = VectorStoreImpl(config, deps[Api.inference])
132+
await impl.initialize()
133+
return impl
134+
```
135+
136+
## Dependencies
137+
138+
The provider package must be installed on the system. For example:
139+
140+
```bash
141+
$ uv pip show llama-stack-ollama-provider
142+
Name: llama-stack-ollama-provider
143+
Version: 0.1.0
144+
Location: /path/to/venv/lib/python3.10/site-packages
145+
```
146+
147+
## Example: Custom Ollama Provider
148+
149+
Here's a complete example of creating and using a custom Ollama provider:
150+
151+
1. First, create the provider package:
152+
153+
```bash
154+
mkdir -p llama-stack-provider-ollama
155+
cd llama-stack-provider-ollama
156+
git init
157+
uv init
158+
```
159+
160+
2. Edit `pyproject.toml`:
161+
162+
```toml
163+
[project]
164+
name = "llama-stack-provider-ollama"
165+
version = "0.1.0"
166+
description = "Ollama provider for Llama Stack"
167+
requires-python = ">=3.10"
168+
dependencies = ["llama-stack", "pydantic", "ollama", "aiohttp"]
169+
```
170+
171+
3. Create the provider specification:
172+
173+
```yaml
174+
# /etc/llama-stack/providers.d/remote/inference/custom_ollama.yaml
175+
adapter:
176+
adapter_type: custom_ollama
177+
pip_packages: ["ollama", "aiohttp"]
178+
config_class: llama_stack_provider_ollama.config.OllamaImplConfig
179+
module: llama_stack_provider_ollama
180+
api_dependencies: []
181+
optional_api_dependencies: []
182+
```
183+
184+
4. Install the provider:
185+
186+
```bash
187+
uv pip install -e .
188+
```
189+
190+
5. Configure Llama Stack to use external providers:
191+
192+
```yaml
193+
external_providers_dir: /etc/llama-stack/providers.d/
194+
```
195+
196+
The provider will now be available in Llama Stack with the type `remote::custom_ollama`.
197+
198+
## Best Practices
199+
200+
1. **Package Naming**: Use the prefix `llama-stack-provider-` for your provider packages to make them easily identifiable.
201+
202+
2. **Version Management**: Keep your provider package versioned and compatible with the Llama Stack version you're using.
203+
204+
3. **Dependencies**: Only include the minimum required dependencies in your provider package.
205+
206+
4. **Documentation**: Include clear documentation in your provider package about:
207+
- Installation requirements
208+
- Configuration options
209+
- Usage examples
210+
- Any limitations or known issues
211+
212+
5. **Testing**: Include tests in your provider package to ensure it works correctly with Llama Stack.
213+
You can refer to the [integration tests
214+
guide](https://github.com/meta-llama/llama-stack/blob/main/tests/integration/README.md) for more
215+
information. Execute the test for the Provider type you are developing.
216+
217+
## Troubleshooting
218+
219+
If your external provider isn't being loaded:
220+
221+
1. Check that the `external_providers_dir` path is correct and accessible.
222+
2. Verify that the YAML files are properly formatted.
223+
3. Ensure all required Python packages are installed.
224+
4. Check the Llama Stack server logs for any error messages - turn on debug logging to get more
225+
information using `LLAMA_STACK_LOGGING=all=debug`.
226+
5. Verify that the provider package is installed in your Python environment.

‎docs/source/providers/index.md‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,10 @@ Providers come in two flavors:
1111

1212
Importantly, Llama Stack always strives to provide at least one fully inline provider for each API so you can iterate on a fully featured environment locally.
1313

14+
## External Providers
15+
16+
Llama Stack supports external providers that live outside of the main codebase. This allows you to create and maintain your own providers independently. See the [External Providers Guide](external) for details.
17+
1418
## Agents
1519
Run multi-step agentic workflows with LLMs with tool usage, memory (RAG), etc.
1620

@@ -50,6 +54,7 @@ The following providers (i.e., databases) are available for Vector IO:
5054
```{toctree}
5155
:maxdepth: 1
5256
57+
external
5358
vector_io/faiss
5459
vector_io/sqlite-vec
5560
vector_io/chromadb

‎llama_stack/distribution/datatypes.py‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -312,6 +312,11 @@ class StackRunConfig(BaseModel):
312312
description="Configuration for the HTTP(S) server",
313313
)
314314

315+
external_providers_dir: Optional[str] = Field(
316+
default=None,
317+
description="Path to directory containing external provider implementations.",
318+
)
319+
315320

316321
class BuildConfig(BaseModel):
317322
version: str = LLAMA_STACK_BUILD_CONFIG_VERSION

0 commit comments

Comments
 (0)