Tokens
Last updated
What is Tokens?
Tokens are fundamental units of text that LLMs process, created by breaking down text into smaller components like words, subwords, or characters. Understanding tokens is crucial because models predict the next token in sequences, API costs are based on token count, and models have maximum token limits for input and output.
Free resources to learn Tokens
More in stage 2: Sampling Parameters
Tokens: common questions
Can I learn Tokens in Hindi?
This lesson is written in English. Once you sign in, open it in the interactive course and ask the AI tutor to explain Tokens in Hindi or Hinglish. The tutor answers from this lesson’s own text, so it stays on the syllabus, and it runs on your own Google, OpenAI or Anthropic key.
Is this Tokens tutorial for beginners?
It is lesson 12 of 46 in the free Prompt Engineering roadmap, in stage 2, Sampling Parameters. If you are new to Prompt Engineering, start from stage 1 and come back here in order; if you already know the earlier stages, this lesson stands on its own.