Chunk text by tokens API

Splits text into overlapping token-window chunks with start/end token offsets — the core RAG ingestion step. Answers 'How do I chunk this text into N-token windows with overlap?', 'What are the token chunks for embedding?'.

Price$0.03per request

MethodPOST

Route/v1/retrieval/chunk-by-tokens

StatusLive

MIME typeapplication/json

Rate limit120/minute

Cache0s public

chunktokenssplitoverlapragingestionwindowembedding

API URLhttps://x402.hexl.dev/v1/retrieval/chunk-by-tokens

Integration docs

Example request

{
  "text": "a b c d e f g",
  "chunkSize": 3,
  "overlap": 1
}

Example response

{
  "chunkSize": 3,
  "overlap": 1,
  "totalTokens": 7,
  "chunkCount": 3,
  "chunks": [
    {
      "index": 0,
      "text": "a b c",
      "tokenCount": 3,
      "startToken": 0,
      "endToken": 3
    },
    {
      "index": 1,
      "text": "c d e",
      "tokenCount": 3,
      "startToken": 2,
      "endToken": 5
    },
    {
      "index": 2,
      "text": "e f g",
      "tokenCount": 3,
      "startToken": 4,
      "endToken": 7
    }
  ]
}

Input schema

{
  "type": "object",
  "required": [
    "text"
  ],
  "properties": {
    "text": {
      "type": "string"
    },
    "chunkSize": {
      "type": "integer"
    },
    "overlap": {
      "type": "integer"
    }
  }
}

Output schema

{
  "type": "object",
  "additionalProperties": true
}