Skip to content

Does we need the GGUF files download by link support in a set_handlers? #217

@bigcat88

Description

@bigcat88

e.g.

MODEL_NAME = "https://huggingface.co/TheBloke/Mistral-7B-Instruct-v0.2-GGUF/blob/main/mistral-7b-instruct-v0.2.Q5_K_M.gguf"


@asynccontextmanager
async def lifespan(_app: FastAPI):
    ex_app.set_handlers(APP, enabled_handler, models_to_fetch={MODEL_NAME: {}})
    yield
    

with automatic progress display in AppAPI UI and with ETag support probably to autoupdate model if file was changes on server?

we can check start of string for "https/http" to detect if this is direct link to file

Or something else(additional files?) will be needed for GGUF models and we need a more complex struct?

cc @kyteinsky ?

edited: ETags support added only for huggingface(they use sha256 of file and X-Linked-ETag field), it should be enough.

Metadata

Metadata

Assignees

No one assigned

    Labels

    questionFurther information is requested

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions