Tiktoken Tutorial: OpenAI's Python Library for Tokenizing Text

Particle participle is a basic step in dealing with natural language processing (NLP) tasks. It involves breaking text into smaller units, called markers, which can be words, subwords, or characters.

Efficient word segmentation is critical to the performance of language models, making it an important step in a variety of NLP tasks such as text generation, translation, and abstraction.

Tiktoken is a fast and efficient thesaurus developed by OpenAI. It provides a powerful solution for converting text into tags and vice versa. Its speed and efficiency make it an excellent choice for developers and data scientists who work with large data sets and complex models.

This guide is designed for developers, data scientists, and anyone who plans to use Tiktoken and needs a practical guide that contains examples.

Basics of OpenAI

Get Started with OpenAI API and more!

Start nowYou can view the code for the Tiktoken open source Python version in the following GitHub repository.

To import the library, we run:

<code>pip install tiktoken</code>

Coding Model

The encoding model in Tiktoken determines the rules for breaking text into tags. These models are crucial because they define how text is segmented and encoded, which affects the efficiency and accuracy of language processing tasks. Different OpenAI models use different encodings.

<code>import tiktoken</code>

Tiktoken provides three coding models optimized for different use cases:

• o200k_base: encoding of the latest GPT-4o-Mini model.
• cl100k_base: Coding models for newer OpenAI models such as GPT-4 and GPT-3.5-Turbo.
• p50k_base: Codex models that are used in code applications.
• r50k_base: Older encoding for different versions of GPT-3.

All of these models are available for OpenAI's API. Note that the API provides much more models than those listed here. Fortunately, the Tiktoken library provides an easy way to check which encoding should be used with which model.

For example, if I need to know what encoding model the text-embedding-3-small model uses, I can run the following command and get the answer as output:

<code>pip install tiktoken</code>

We get as output. Before we use Tiktoken directly, I would like to mention that OpenAI has a tokenized web application where you can see how different strings are tokenized - you can access it here. There is also a third-party online tagger, Tiktokenizer, which supports non-OpenAI models.

Encode text as marker

To encode text as a tag using Tiktoken, you first need to get the encoded object. There are two ways to initialize it. First, you can do this using the name of the tokenizer:

<code>import tiktoken</code>

Alternatively, you can run the encoding_for_model function mentioned earlier to get the encoder for a specific model:

<code>print(tiktoken.encoding_for_model('text-embedding-3-small'))</code>

Now, we can run the encode method of the encode object to encode the string. For example, we can encode the "I love DataCamp" string as follows - here I use the cl100k_base encoder:

<code>encoding = tiktoken.get_encoding("[标记器名称]")</code>

We get [40, 3021, 2956, 34955] as output.

Decode the mark into text

To decode the mark back to text, we can use the .decode() method on the encoded object.

Let's decode the following tag [40, 4048, 264, 2763, 505, 2956, 34955]:

<code>encoding = tiktoken.encoding_for_model("[模型名称]")</code>

These marks are decoded as "I learned a lot from DataCamp".

Practical use cases and tips

In addition to encoding and decoding, I also thought of two other use cases.

Cost Estimation and Management

Understanding tag counting before sending a request to the OpenAI API can help you manage costs efficiently. Because OpenAI's billing is based on the number of tags processed, pre-tagged text allows you to estimate the cost of API usage. Here is how to calculate tags in text using Tiktoken:

<code>print(encoding.encode("我爱 DataCamp"))</code>

We just need to check the length of the array to see how many marks we get. By knowing the number of tags ahead of time, you can decide whether to shorten text or adjust usage to stay within your budget.

You can read more about this method in this tutorial on estimating the cost of GPT using the tiktoken library in Python.

Input length verification

When using OpenAI models from the API, you are limited by the maximum number of markers input and output. Exceeding these limits can result in errors or output truncated. With Tiktoken, you can verify the input length and make sure it complies with the marking limit.

Conclusion

Tiktoken is an open source thesaurus that provides speed and efficiency tailored to the OpenAI language model.

Learning how to use Tiktoken to encode and decode text and its various coding models can greatly enhance your work with large language models.

Get top AI certification

Prove that you can use AI effectively and responsibly. Get certified, get hired

The above is the detailed content of Tiktoken Tutorial: OpenAI's Python Library for Tokenizing Text. For more information, please follow other related articles on the PHP Chinese website!