Skip to content

'Could not automatically map SimilarityCurie001 to a tokeniser. Please use tiktok.get_encoding to explicitly get the tokeniser you expect.' #2854

@viksing

Description

@viksing

This is related to AzureOpenAI call.

import os
import tiktoken
from langchain.embeddings import OpenAIEmbeddings
from langchain.llms import AzureOpenAI

os.environ["OPENAI_API_TYPE"] = "azure"
os.environ["OPENAI_API_BASE"] = "https://xxxxxxx.openai.azure.com/"
os.environ["OPENAI_API_KEY"] = "xxxx"

embeddings = OpenAIEmbeddings(model="SimilarityCurie001-AzureDeploymentName")

text = "This is a test document."
query_result = embeddings.embed_query(text)

Getting error on the execution of 'query_result = embeddings.embed_query(text)' line.

MODEL_TO_ENCODING variable is having all the encoding mapping against the real names of the models.
but we specify AzureDeploymentName of the the model in embeddings = OpenAIEmbeddings(model="SimilarityCurie001-AzureDeploymentName").

and the look up fails.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions