Error when calling `InferenceClient.conversational`

[philipp-zettl]

Describe the bug

Calling InferenceClient.conversational according to the docs results in a 400 Client Error.

Reproduction

from huggingface_hub import InferenceClient
InferenceClient().conversational('test')

Logs

Traceback (most recent call last):
  File "~/.venv/lib/python3.11/site-packages/huggingface_hub/utils/_errors.py", line 270, in hf_raise_for_status
    response.raise_for_status()
  File "~/.venv/lib/python3.11/site-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: https://api-inference.huggingface.co/models/facebook/blenderbot-400M-distill

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "~/.venv/lib/python3.11/site-packages/huggingface_hub/inference/_client.py", line 396, in conversational
    response = self.post(json=payload, model=model, task="conversational")
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/.venv/lib/python3.11/site-packages/huggingface_hub/inference/_client.py", line 240, in post
    hf_raise_for_status(response)
  File "~/.venv/lib/python3.11/site-packages/huggingface_hub/utils/_errors.py", line 326, in hf_raise_for_status
    raise BadRequestError(message, response=response) from e
huggingface_hub.utils._errors.BadRequestError:  (Request ID: EmFefRo_g2_zE3De0TRwJ)

Bad request:
`args[0]`: {'text': 'test'} have the wrong format. The should be either of type `str` or type `list`

System info

- huggingface_hub version: 0.19.4
- Platform: Linux-6.1.71-1-MANJARO-x86_64-with-glibc2.38
- Python version: 3.11.6
- Running in iPython ?: No
- Running in notebook ?: No
- Running in Google Colab ?: No
- Token path ?: ~/.cache/huggingface/token
- Has saved token ?: True
- Who am I ?: philipp-zettl
- Configured git credential helpers: 
- FastAI: N/A
- Tensorflow: N/A
- Torch: N/A
- Jinja2: N/A
- Graphviz: N/A
- Pydot: N/A
- Pillow: 10.2.0
- hf_transfer: N/A
- gradio: N/A
- tensorboard: N/A
- numpy: N/A
- pydantic: N/A
- aiohttp: 3.9.1
- ENDPOINT: https://huggingface.co
- HF_HUB_CACHE: ~/.cache/huggingface/hub
- HF_ASSETS_CACHE: ~/.cache/huggingface/assets
- HF_TOKEN_PATH: ~/.cache/huggingface/token
- HF_HUB_OFFLINE: False
- HF_HUB_DISABLE_TELEMETRY: False
- HF_HUB_DISABLE_PROGRESS_BARS: None
- HF_HUB_DISABLE_SYMLINKS_WARNING: False
- HF_HUB_DISABLE_EXPERIMENTAL_WARNING: False
- HF_HUB_DISABLE_IMPLICIT_TOKEN: False
- HF_HUB_ENABLE_HF_TRANSFER: False
- HF_HUB_ETAG_TIMEOUT: 10
- HF_HUB_DOWNLOAD_TIMEOUT: 10

[romanmeister]

+ 1

[Wauplin]

Hi @philipp-zettl @romanmeister I'm sorry about the inconvenience here. We are actually making a revampt of the conversational API on the Hub. As a consequence, it broke the default usage of InferenceClient.conversational. By default, it calls the API for facebook/blenderbot-400M-distill which is now tagged as "text-generation" pipeline and not "conversational". A solution if you really want to use conversational with this model is to explicitly call the conversational pipeline as such:

>>> from huggingface_hub import InferenceClient
>>> client = InferenceClient(model="https://api-inference.huggingface.co/pipeline/conversational/facebook/blenderbot-400M-distill")
>>> client.conversational("test")
{'generated_text': " Have you ever taken a test? It's a test that tests your ability to perform certain tasks.", 'conversation': {'generated_responses': [" Have you ever taken a test? It's a test that tests your ability to perform certain tasks."], 'past_user_inputs': ['test']}}

The short to mid-term goal is to deprecate/revampt the InferenceClient.conversational method as well in favor of a more OpenAI-style API. In fact, the current conversational API (and task) has quite low usage and few compatible models, excluding most of the currently trending LLMs. The current best solution to use our Serverless Endpoint for a chatbot usage is use the text-generation task. You'll have to pass as input a prompt generated from your list of messages using AutoTokenizer.apply_chat_template.

cc @julien-c @osanseviero

[philipp-zettl]

Hi @philipp-zettl @romanmeister I'm sorry about the inconvenience here. We are actually making a revampt of the conversational API on the Hub. As a consequence, it broke the default usage of InferenceClient.conversational. By default, it calls the API for facebook/blenderbot-400M-distill which is now tagged as "text-generation" pipeline and not "conversational". A solution if you really want to use conversational with this model is to explicitly call the conversational pipeline as such:

>>> from huggingface_hub import InferenceClient
>>> client = InferenceClient(model="https://api-inference.huggingface.co/pipeline/conversational/facebook/blenderbot-400M-distill")
>>> client.conversational("test")
{'generated_text': " Have you ever taken a test? It's a test that tests your ability to perform certain tasks.", 'conversation': {'generated_responses': [" Have you ever taken a test? It's a test that tests your ability to perform certain tasks."], 'past_user_inputs': ['test']}}

The short to mid-term goal is to deprecate/revampt the InferenceClient.conversational method as well in favor of a more OpenAI-style API. In fact, the current conversational API (and task) has quite low usage and few compatible models, excluding most of the currently trending LLMs. The current best solution to use our Serverless Endpoint for a chatbot usage is use the text-generation task. You'll have to pass as input a prompt generated from your list of messages using AutoTokenizer.apply_chat_template.

cc @julien-c @osanseviero

To clarify, I didn't explicitly select a model, I was using the suggested one.

Great, thanks for the quick reply and explanation!
Albeit the provided solution is working, it seemingly requires a lot of extra work to enforce coherent responses from the model, compared to the conversational task.
I can life with this for now and will keep an eye open for updates.
Is there anything I can follow to be notified once the OpenAI-style API or conversational is updated?

[Wauplin]

To clarify, I didn't explicitly select a model, I was using the suggested one.

Yes, that's what I expected as well. Just wanted to mention the possibility to use the conversational with the "old fashion" API.

Albeit the provided solution is working, it seemingly requires a lot of extra work to enforce coherent responses from the model, compared to the conversational task.

Agree and that's why we are working on it! 3 weeks ago @drbh added support for the openai chat completion API to TGI and that's only the beginning before supporting it everywhere in our ecosystem :)

[Wauplin]

And related blog post from yesterday: https://huggingface.co/blog/tgi-messages-api

[Wauplin]

Closing since conversational is now officially deprecated. Please use chat_completion instead.