Friday, April 5, 2024

openai/tiktoken: tiktoken is a fast BPE tokeniser for use with OpenAI's models.