Counting tokens before sending a prompt to an LLM, in PHP
I need a rough token count before sending, mostly to reject prompts that would blow the context window and to estimate cost. The real tokenizer is BPE and I do not want to ship a megabyte of merge rules into a PHP service if I can avoid it. For English text the chars-divided-by-four heuristic is in the right ballpark but it falls apart on code and on heavy punctuation.
What are people using in production for a pre-flight estimate that does not need a native extension? Accuracy within ten percent would be plenty for a guardrail.
For a guardrail, do not chase exactness. chars/4 underestimates code because code has more short tokens, so I bias it: count words and punctuation runs separately. Roughly, tokens are about 0.75 times word count for prose, and code lands closer to 1 token per 3 characters. If you only need a reject threshold, estimate high and add a safety margin so you never accept a prompt that actually overflows.
A surprisingly decent estimator without merge tables is to count by a regex that splits on word boundaries, whitespace runs, and individual punctuation, then weight. Something like preg_match_all on /\w+|[^\w\s]/u gives you a unit count that correlates better with real BPE than raw chars/4, especially for code. It overcounts long words a bit, which is the safe direction for a guard.
Keep in mind whitespace and newlines are not free, they become tokens too, and code is full of them. The chars/4 rule quietly assumes prose density. If your inputs are mostly source files, calibrate against a few real samples where you know the true count from the API response usage field, then pick a divisor that fits your traffic. We landed on chars/3.3 for our mixed code-and-text payloads.
The calibration idea is what I was missing. The API already returns the real token count in the usage field on every response, so I logged estimate versus actual for a day, plotted the ratio, and just fit a divisor. For our traffic 3.4 chars per token with the punctuation-aware regex as a sanity check keeps me inside five percent. Good enough to gate on.
One caution: do your length check on the exact string you send, after you have assembled the full message array including the system prompt and any tool schemas. People estimate the user text and forget the system prompt and function definitions, which can be hundreds of tokens. The overflow always comes from the part you did not measure.
```php blocks are runnable.