mcp-tokenizer mcp

v1.0.7 · MCP Tool · nlp · registry.pascalai.org

Text tokenization utilities: count tokens, encode text to token IDs, truncate to a token limit, split text into chunks, estimate cost.

tokenizertokensnlpllmtextsplitlinux

Input Parameters

ParameterTypeDescription
operationrequired string Operation: count (token count), encode (text → token IDs), truncate (cut to maxTokens), split (chunk with overlap), estimate (cost estimate). One of: count, encode, truncate, split, estimate.
textrequired string Input text to process.
modeloptional string Target model name (e.g. gpt-4, claude-3). Affects tokenization heuristic.
maxTokensoptional integer Token limit for truncate/split. Default: 4096.
overlapoptional integer Overlap tokens between chunks for split. Default: 0.

Output Fields

FieldTypeDescription
No output fields defined

Examples

No examples available.

Install & Discovery

Install

ppm install mcp-tokenizer

Get JSON Schema

GET /v1/packages/mcp-tokenizer/1.0.7/schema

Discover by keyword

GET /v1/mcp/discover?q=tokenizer
Discovery hint: Install with ppm install mcp-tokenizer or invoke remotely via POST /v1/invoke/mcp-tokenizer on the MCP Service.