All the following commands are executed from the root of the project.
Launch the benchmark
- you can specify the task to run (
encodeordecode) - you can control run durations with flags (defaults shown below)
- you have to specify the benchmark tag, which will be used to identify the benchmark run
- benchmark results are stored in
./benchmark/output/benchee/<TASK>-<HHMMSS>-<BENCHMARK_TAG>.benchee
mix protox.benchmark.run [--task <TASK>] [--warmup 2] [--time 5] [--memory-time 2] [--reduction-time 2] <BENCHMARK_TAG>
Examples:
# Short run (defaults)
mix protox.benchmark.run --task decode my_tag
# Longer run to reduce variance
mix protox.benchmark.run --task encode --warmup 3 --time 15 --memory-time 5 --reduction-time 5 my_long_tag
As a sanity check, before measuring anything, the run re-encodes every message in the corpus and checks it against the size recorded when the corpus was generated.
You can restrict any run to a subset of the corpus:
mix protox.benchmark.run --task decode --input maps,test_all_types_proto3 my_tag
Profile the code
--profile-after switches the task into a profiling mode: Benchee runs each scenario exactly once under a profiler, after everything else.
mix protox.benchmark.run --task encode --profile-after tprof --profile-type memory --input synthetic_200
| Flag | Values | Default |
|---|---|---|
--profile-after | tprof, cprof, eprof, fprof | — |
--profile-type | memory, calls, time (tprof only) | memory |
--profile-scope | module, codec, all (tprof only) | module |
Aggregate the results
mix protox.benchmark.report <BENCHMARK_RESULT_PATHS>
Example:
mix protox.benchmark.report ./benchmark/output/benchee/encode-*
This prints a console comparison and writes an HTML report to ./benchmark/output/html/.
The corpus
Every run reads ./benchmark/benchmark_payloads.bin, a compressed :erlang.term_to_binary map of input_name => {module, [{message, size, bytes}]}.
Regenerate it with:
mix protox.benchmark.generate.payloads
[!WARNING] Generation is not seeded, so regenerating produces an entirely different corpus and makes every previously saved
.bencheefile incomparable.
Add new .proto files in compile_benchmark_protos.ex.
Value distributions
Payloads are generated with Protox.RandomInit, using a profile:
Protox.Benchmark.RealisticValues: production-like values for benchmarking.Protox.RandomInit.EdgeCase: the profile the test suite uses, which deliberately over-produces NaN, Infinity, full-range integers and non-ASCII strings.
The inputs
| Input | Source | Notes |
|---|---|---|
google_message1_proto3 / _proto2 | captured | Real Google production payloads, 228 bytes each |
synthetic_5 … synthetic_200 | generated | Synthetic messages with 5 to 200 fields |
maps | generated | Map-heavy; the synthetic distribution contains no maps |
otel_traces_data | generated | OpenTelemetry TracesData, real schema |
prometheus_write_request | generated | Prometheus remote-write, real schema |
test_all_types_proto3 | generated | The conformance suite's mega-message |
synthetic_100_edge | generated | EdgeCase twin of synthetic_100 |
edge_* | generated | One saturated encoder path each |
Provenance of the synthetic protos
protos/synthetic_*.proto are frozen artifacts. They were generated once using field types and labels from the weighted distribution used by Google's own protobuf benchmarks: