Generative AI with Node js in Hindi #5 what is token in Open AI | how to calculate tokens

Open AI token


Steps:-

  1. What is Token
  2. Why token is important
  3. How to check and calculate token
  4. Important points about token
  5. tiktoken npm package


What is Token?

  1. Unit of data that a AI model processes
  2. AI Models calculate bills according to token
  3. Request and response data caculated in token


For example:

  1. "ChatGPT" = 1 token
  2. "Hello" = 1 token
  3. "I love coding!" = 4 tokens (["I", " love", " coding", "!"])

When you use the API, OpenAI counts how many tokens are in your input + output, and that’s how it calculates cost and limits.


Open AI tokenizer Link :- https://platform.openai.com/tokenizer

API Pricing Link :- https://platform.openai.com/docs/pricing

Tiktoken Npm package link :- https://www.npmjs.com/package/js-tiktoken



import OpenAI from "openai"
import dotenv from 'dotenv'
import { encoding_for_model } from "tiktoken";
dotenv.config();

const client= new OpenAI({apiKey:process.env.openAI_Key})

// const response = await client.responses.create({
// instructions:'give result in 10 word',
// input:"tell me one best color",
// model:"gpt-4o-mini"
// });

const prompt =`Many words map to one token, but some don't: indivisible.

Unicode characters like emojis may be split into many tokens containing the underlying bytes: 🤚🏾

Sequences of characters commonly found next to each other may be grouped together: 1234567890`;
const model="gpt-4o-mini"

const response = await client.responses.create({
input:[
// {role:'system',content:'answer in 20 words'},
// {role:'developer',content:'gave a basic example in JS'},
{role:'user',content:prompt}
],
model
});
console.log(response.usage);

function calculateToken(){
const encoder= encoding_for_model(model);
const tokenData = encoder.encode(prompt);
console.log(tokenData);
}

calculateToken()