Failed to create agent avatar

[Hello-World-Traveler]

Unable to create agent, it creates the agent but can not start or edit it.

=ERROR msg="Failed to create agent avatar" error="failed to generate image prompt: error, status code: 500, status: 500 Internal Server Error, message: could not load model - all backends returned error: [llama-cpp]: failed to load model with internal loader: could not load model: rpc error: code = Canceled desc = \n[llama-cpp-fallback]: failed to load model with internal loader: could not load model: rpc error: code = Canceled desc = \n[stablediffusion-ggml]: failed to load model with internal loader: could not load model: rpc error: code = Unknown desc = could not load model\n[whisper]: failed to load model with internal loader: could not load model: rpc error: code = Unknown desc = stat /build/models/arcee-agent: no such file or directory\n[bark-cpp]: failed to load model with internal loader: could not load model: rpc error: code = Unknown desc = inference failed\n[piper]: failed to load model with internal loader: could not load model: rpc error: code = Unknown desc = unsupported model type /build/models/arcee-agent (should end with .onnx)\n[silero-vad]: failed to load model with internal loader: could not load model: rpc error: code = Unknown desc = create silero detector: failed to create session: Load model from /build/models/arcee-agent failed:Load model /build/models/arcee-agent failed. File doesn't exist\n[huggingface]: failed to load model with internal loader: could not load model: rpc error: code = Unknown desc = no huggingface token provided\n[/build/backend/python/faster-whisper/run.sh]: failed to load model with internal loader: backend not found: /tmp/localai/backend_data/backend-assets/grpc/build/backend/python/faster-whisper/run.sh\n[/build/backend/python/vllm/run.sh]: failed to load model with internal loader: backend not found: /tmp/localai/backend_data/backend-assets/grpc/build/backend/python/vllm/run.sh\n[/build/backend/python/exllama2/run.sh]: failed to load model with internal loader: backend not found: /tmp/localai/backend_data/backend-assets/grpc/build/backend/python/exllama2/run.sh\n[/build/backend/python/bark/run.sh]: failed to load model with internal loader: backend not found: /tmp/localai/backend_data/backend-assets/grpc/build/backend/python/bark/run.sh\n[/build/backend/python/transformers/run.sh]: failed to load model with internal loader: backend not found: /tmp/localai/backend_data/backend-assets/grpc/build/backend/python/transformers/run.sh\n[/build/backend/python/rerankers/run.sh]: failed to load model with internal loader: backend not found: /tmp/localai/backend_data/backend-assets/grpc/build/backend/python/rerankers/run.sh\n[/build/backend/python/autogptq/run.sh]: failed to load model with internal loader: backend not found: /tmp/localai/backend_data/backend-assets/grpc/build/backend/python/autogptq/run.sh\n[/build/backend/python/kokoro/run.sh]: failed to load model with internal loader: backend not found: /tmp/localai/backend_data/backend-assets/grpc/build/backend/python/kokoro/run.sh\n[/build/backend/python/diffusers/run.sh]: failed to load model with internal loader: backend not found: /tmp/localai/backend_data/backend-assets/grpc/build/backend/python/diffusers/run.sh\n[/build/backend/python/coqui/run.sh]: failed to load model with internal loader: backend not found: /tmp/localai/backend_data/backend-assets/grpc/build/backend/python/coqui/run.sh" source.file=/work/core/state/pool.go source.L=16

level=ERROR msg="Agent not found" name=Infobot source.file=/work/webui/routes.go source.L=204

image: quay.io/mudler/localagi:master
    environment:
      - LOCALAGI_MODEL=${MODEL_NAME:-arcee-agent}
      - LOCALAGI_MULTIMODAL_MODEL=${MULTIMODAL_MODEL:-minicpm-v-2_6}
      - LOCALAGI_IMAGE_MODEL=${IMAGE_MODEL:-sd-1.5-ggml}

No python in default image, what does this mean for docker compose as it's the default image?

[richiejp]

The default image model uses stablediffusion-ggml which is written in C++, so that is what should load the model, but if it fails then LocalAI will cycle through all the backends to see if any work which is maybe where you are getting the Python not found error.

I'm not sure the root cause is included in the log, could you post more of the log?

As a temporary workaround you can copy one of the `docker-compose.{intel,nvidia}.yaml' files and replace the localagi service with:

  localagi:
    extends:
      file: docker-compose.yaml
      service: localagi
    environment:
      - LOCALAGI_MODEL=${MODEL_NAME:-gemma-3-12b-it-qat}
      - LOCALAGI_MULTIMODAL_MODEL=${MULTIMODAL_MODEL:-minicpm-v-2_6}
      - LOCALAGI_IMAGE_MODEL=
      - LOCALAGI_LLM_API_URL=http://localai:8080
      - LOCALAGI_LOCALRAG_URL=http://localrecall:8080
      - LOCALAGI_STATE_DIR=/pool
      - LOCALAGI_TIMEOUT=5m
      - LOCALAGI_ENABLE_CONVERSATIONS_LOGGING=false

or edit the base file. This unsets the image model so it won't try to create avatars.

[Hello-World-Traveler]

I have all those environment settings with default models that are already downloaded. The only thing I changed was the port to be the port of Local-AI
LOCALAGI_LLM_API_URL OpenAI-compatible API server URL` Would this be the Local-AI server?

It connects to the main AI

DBG context local model name not found, setting to the first model first model name=LocalAI-functioncall-phi-4-v0.3`

DBG Prompt (after templating): Generate a prompt that I can use to create a random avatar for the bot

DBG Grammar: space ::= " "?

freestring ::= (

I think the issue is that there is no model to create the avatar. I am not able to create images with the master image.

Using the outdatad image localai/localai:latest-aio-gpu-nvidia-cuda-12 I was able to create images, using latest localai/localai:master-cublas-cuda12-ffmpeg-core I am not, therefore the system wouldn’t be able to create one? I also have all stablediffusion models installed.

[richiejp]

I suspect that the core image doesn't include the Python diffusers library and the AIO image does. You should be fine using the AIO image assuming by 'outdated' you mean we changed it from AIO to core?

If this is the case then there are two possible fixes; 1. we default to a different image and the other is we fix stablediffusion-ggml in LocalAI mudler/LocalAI#5236

[Hello-World-Traveler]

Outdated as in haven't been able to update yet for AIO.

Your pull request will fix diffusers for core images?

[richiejp]

Outdated as in haven't been able to update yet for AIO.

OK, it'll be interesting to see if it still works on the latest version

Your pull request will fix diffusers for core images?

No, it is supposed to fix stablediffusion-ggml, but it appears there is still an issue with that and I am struggling to isolate what is wrong. So probably your best bet is to keep using diffusers for now in the AIO image. I don't know if we should add diffusers to the core image, I assume it drags in a lot of deps.

[Hello-World-Traveler]

No, it is supposed to fix stablediffusion-ggml, but it appears there is still an issue with that and I am struggling to isolate what is wrong. So probably your best bet is to keep using diffusers for now in the AIO image. I don't know if we should add diffusers to the core image, I assume it drags in a lot of deps.

Core images are about 3-5Gb and AIO is about 17Gb, most of that are dependences and models I think.

Running diffusion-3-medium on the core image and it tries to load python. There is no python setup.

9:44AM INF BackendLoader starting backend=diffusers modelID=stable-diffusion-3-medium o.model=v2ray/stable-diffusion-3-medium-diffusers

9:44AM DBG Loading model in memory from file: /build/models/v2ray/stable-diffusion-3-medium-diffusers

9:44AM DBG Loading Model stable-diffusion-3-medium with gRPC (file: /build/models/v2ray/stable-diffusion-3-medium-diffusers) (backend: diffusers): {backendString:diffusers model:v2ray/stable-diffusion-3-medium-diffusers modelID:stable-diffusion-3-medium assetDir:/tmp/localai/backend_data context:{emptyCtx:{}} gRPCOptions:0xc0006fe908 externalBackends:map[autogptq:/build/backend/python/autogptq/run.sh bark:/build/backend/python/bark/run.sh coqui:/build/backend/python/coqui/run.sh diffusers:/build/backend/python/diffusers/run.sh exllama2:/build/backend/python/exllama2/run.sh faster-whisper:/build/backend/python/faster-whisper/run.sh kokoro:/build/backend/python/kokoro/run.sh rerankers:/build/backend/python/rerankers/run.sh transformers:/build/backend/python/transformers/run.sh vllm:/build/backend/python/vllm/run.sh] grpcAttempts:20 grpcAttemptsDelay:2 parallelRequests:false}

9:44AM DBG Loading external backend: /build/backend/python/diffusers/run.sh

9:44AM DBG external backend is file: &{name:run.sh size:73 mode:448 modTime:{wall:0 ext:63880309059 loc:0x5a4f5da0} sys:{Dev:2097162 Ino:55186172 Nlink:1 Mode:33216 Uid:0 Gid:0 X__pad0:0 Rdev:0 Size:73 Blksize:4096 Blocks:8 Atim:{Sec:1745377230 Nsec:376934073} Mtim:{Sec:1744712259 Nsec:0} Ctim:{Sec:1745377230 Nsec:375934048} X__unused:[0 0 0]}}

9:44AM DBG Loading GRPC Process: /build/backend/python/diffusers/run.sh

9:44AM DBG GRPC Service for stable-diffusion-3-medium will be running at: '127.0.0.1:32939'

9:44AM DBG GRPC Service state dir: /tmp/go-processmanager101947073

9:44AM DBG GRPC Service Started

9:44AM DBG Wait for the service to start up

9:44AM DBG Options: ContextSize:1024  Seed:2095585644  NBatch:512  MMap:true  NGPULayers:99999999  Threads:10  PipelineType:"StableDiffusion3Pipeline"  CUDA:true

9:44AM DBG GRPC(stable-diffusion-3-medium-127.0.0.1:32939): stdout Initializing libbackend for diffusers

9:44AM DBG GRPC(stable-diffusion-3-medium-127.0.0.1:32939): stderr ./../common/libbackend.sh: line 94: uv: command not found

9:44AM DBG GRPC(stable-diffusion-3-medium-127.0.0.1:32939): stderr ./../common/libbackend.sh: line 100: /build/backend/python/diffusers/venv/bin/activate: No such file or directory

9:44AM DBG GRPC(stable-diffusion-3-medium-127.0.0.1:32939): stdout virtualenv created

9:44AM DBG GRPC(stable-diffusion-3-medium-127.0.0.1:32939): stdout virtualenv activated

9:44AM DBG GRPC(stable-diffusion-3-medium-127.0.0.1:32939): stdout activated virtualenv has been ensured

9:44AM DBG GRPC(stable-diffusion-3-medium-127.0.0.1:32939): stderr ./../common/libbackend.sh: line 183: exec: python: not found

Core Image

In the python folder within the container: autogptq bark common coqui diffusers exllama2 faster-whisper kokoro rerankers transformers vllm

In diffuers folder:

Makefile   backend.py  requirements-cpu.txt       requirements-cublas12.txt  requirements-intel.txt  run.sh   test.sh
README.md  install.sh  requirements-cublas11.txt  requirements-hipblas.txt   requirements.txt        test.py

AIO Image:

Makefile   backend.py      backend_pb2_grpc.py  requirements-cpu.txt       requirements-cublas12.txt  requirements-intel.txt  run.sh   test.sh
README.md  backend_pb2.py  install.sh           requirements-cublas11.txt  requirements-hipblas.txt   requirements.txt        test.py  venv

The All in one image has ven, backend_pb2.py and backend_pb2_grpc.py

stable-diffusion-3-medium.yaml

backend: diffusers
diffusers:
  cuda: true
  enable_parameters: negative_prompt,num_inference_steps
  pipeline_type: StableDiffusion3Pipeline
f16: false
name: stable-diffusion-3-medium
parameters:
  model: v2ray/stable-diffusion-3-medium-diffusers
step: 25

10:08AM DBG context local model name not found, setting to default defaultModelName=stablediffusion

10:08AM DBG Model file loaded: sd3.5_large-Q4_0.gguf architecture=llama bosTokenID=-1 eosTokenID=-1 modelName=

10:08AM DBG guessDefaultsFromFile: family not identified

10:08AM DBG Parameter Config: &{PredictionOptions:{BasicM

After a restart and selecting Dreamshaper using the AIO image (outdated image on my end)

10:13AM DBG Loading GRPC Process: /build/backend/python/diffusers/run.sh

10:13AM DBG GRPC Service for dreamshaper will be running at: '127.0.0.1:33685'

10:13AM DBG GRPC Service state dir: /tmp/go-processmanager896307259

10:13AM DBG GRPC Service Started

10:13AM DBG Wait for the service to start up

10:13AM DBG Options: ContextSize:1024 Seed:103804155 NBatch:512 F16Memory:true MMap:true NGPULayers:99999999 Threads:10 PipelineType:"StableDiffusionPipeline" SchedulerType:"k_dpmpp_2m" CUDA:true

10:13AM DBG GRPC(dreamshaper-127.0.0.1:33685): stdout Initializing libbackend for diffusers

10:13AM DBG GRPC(dreamshaper-127.0.0.1:33685): stdout virtualenv activated

10:13AM DBG GRPC(dreamshaper-127.0.0.1:33685): stdout activated virtualenv has been ensured

10:13AM DBG GRPC(dreamshaper-127.0.0.1:33685): stderr /build/backend/python/diffusers/venv/lib/python3.10/site-packages/google/protobuf/runtime_version.py:98: UserWarning: Protobuf gencode version 5.29.0 is exactly one major version older than the runtime version 6.30.2 at backend.proto. Please update the gencode to avoid compatibility violations in the next runtime release.

10:13AM DBG GRPC(dreamshaper-127.0.0.1:33685): stderr   warnings.warn(

10:13AM DBG GRPC(dreamshaper-127.0.0.1:33685): stderr /build/backend/python/diffusers/venv/lib/python3.10/site-packages/transformers/utils/hub.py:105: FutureWarning: Using `TRANSFORMERS_CACHE` is deprecated and will be removed in v5 of Transformers. Use `HF_HOME` instead.

10:13AM DBG GRPC(dreamshaper-127.0.0.1:33685): stderr   warnings.warn(

10:13AM DBG GRPC(dreamshaper-127.0.0.1:33685): stderr Server started. Listening on: 127.0.0.1:33685

10:13AM DBG GRPC Service Ready

Creates the image.

What's missing from the core image or could be added to the core image?