Seven benchmarking mistakes that produce misleading PHP results
A practical review checklist for PHP microbenchmarks: equivalent work, consumed outputs, warm-up, sample rotation, units, environment and production relevance.
It is easy to time two code snippets. It is much harder to make the result mean what the headline claims. Most misleading benchmarks are not fabricated; they simply omit a condition that changes the interpretation.
The following mistakes are drawn from the controls used in the ASAMB build and from the limitations that remain even after those controls are applied.
1. Comparing different work
Two variants may produce similar-looking output while differing in validation, escaping, error handling, allocation, typing or failure behavior. A faster variant that omits required work has not won the same task.
Document semantic differences before timing. When exact equivalence is impossible, state what changed and restrict the conclusion accordingly.
2. Letting the runtime discard or simplify the result
A benchmark should consume enough of the output to demonstrate that the operation completed. Otherwise, compiler or runtime behavior can make the measured loop less representative than expected.
Consumption does not need to dominate the test, but it should prevent an unused result from becoming a special case.
3. Reporting one run
A single sample can include scheduler interruption, filesystem noise, thermal changes or unrelated host activity. The current ASAMB build uses 7 measured rounds and reports a median plus middle spread rather than selecting the most flattering run.
More samples do not remove every source of noise, but they make unusual values visible.
4. Running every variant in the same fixed order
Fixed ordering can couple a variant to cache warmth, allocator state or changing host load. Rotate or randomize the order across rounds, then retain the mapping between samples and variants.
5. Publishing percentages without absolute units
A 500% difference sounds decisive until both operations are shown to be sub-microsecond and called once per request. Conversely, a 10% gap can matter in a dominant batch stage.
Publish nanoseconds or microseconds per operation, operations per second where useful, sample counts and the expected production call frequency.
6. Hiding the environment and fixture
PHP version, extensions, processor, operating system, SAPI, OPcache state and fixture shape can change rankings. “PHP is faster with X” is not a complete claim without those conditions.
The current ASAMB snapshot records PHP 8.4.23 on AMD EPYC-Rome Processor, but readers should rerun the source on their own target host before adopting a local ranking.
7. Treating a microbenchmark as a production profile
An isolated benchmark removes database latency, framework behavior, network waits, contention and application branching. That isolation is useful for answering a narrow question, not for estimating total request improvement.
Use production profiling to find hot paths, then use focused benchmarks to investigate a candidate change. Reverse that order and the team may optimize code that users never notice.
A defensible benchmark narrows a decision. It does not replace profiling, load testing or failure analysis.
Minimum publication checklist
- State the question and the behavior being held constant.
- Publish fixture generation and executable source.
- Record environment and runtime configuration.
- Use repeated samples and disclose the summary statistic.
- Show absolute units, not only percentages.
- Name security, compatibility and failure-mode constraints.
- Explain what the result cannot establish.