On-Device AI Benchmark Record Schema

Direct answer

A benchmark result is only interpretable when its surrounding conditions are recorded with it. A useful schema binds the app build, OS, device identity, model artifact, runtime and backend configuration, workload revision, input fixture, measurement boundaries, output acceptance, and final result. If one of those fields is missing, a reviewer may not know whether two numbers describe the same task, whether a backend changed, or whether a failed run was silently dropped. The purpose of a schema is not to make every result comparable automatically; it is to make the limits and provenance of a comparison visible. Separate stable identity from an observation. A run identity should reference exact app and runtime revisions, artifact hash or stable artifact ID, backend request and observed backend when available, device/OS build, and benchmark protocol revision. The observation stores measured events and values, not claims: start/end timestamps, sample sequence, status, errors, cancellation, fallback, and measurement units. Keep the raw event record immutable; derived summaries should reference the source records and the aggregation rule. This allows a future correction to a parser or unit conversion without rewriting history. Define event names before collecting data. For example, “request submitted,” “first accepted output,” and “task terminal” are not interchangeable with application launch, artifact verification, or engine initialization. A benchmark that reports only one duration may hide whether waiting came from storage, initialization, first output, or the rest of generation. Store both the event markers and a short protocol definition. If a device thermal signal or battery state is included, record the platform source and the time it was sampled; do not invent a universal pass threshold from a field in the schema. Preserve every outcome. A valid sample can be successful, failed, cancelled, timed out, rejected, or discarded for an explicitly documented instrumentation defect. The record should say which and why. Never average away a crash, exclude slow runs without a predeclared rule, or substitute a retry result for the original attempt. For sustained sequences, retain ordered samples so reviewers can examine changes over time rather than only an average. Missing data should be `null` or an explicit absent-state reason, not a fabricated zero. Keep privacy boundaries in the schema. Use fixture IDs and content hashes where useful, but do not put user prompts, photos, audio, document text, or sensitive filenames into analytics or public benchmark exports. If test content itself is necessary for independent reproduction, use consented or synthetic fixtures stored under a separate controlled process. A schema that is technically complete but leaks user content is not an acceptable measurement system. Validate records like application input. Require known schema versions, enumerated status values, explicit units, and a reference to the protocol. Reject records that have a measured value but no event definition, that claim a fallback without its cause, or that refer to an artifact ID absent from the release manifest. Keep unknown fields during forward-compatible ingestion only when the consumer can safely ignore them; never reinterpret an unknown unit. Version migrations should be tested with old fixtures, not applied silently to historical records. This method is especially useful when several teams contribute measurements or when a performance claim must be audited later. It does not create benchmark data by itself and does not establish a device-support list for Cove. The fact cards explicitly warn that benchmark conclusions require reproducible method and raw evidence. A schema can help collect that evidence, but it cannot replace physical-device testing, calibrated instrumentation, or a reviewer deciding whether the workload represents the actual user task. Publish only the scope supported by records and describe what the dataset omits.

A team receives two incompatible “time to answer” spreadsheets

One tester timed from tapping Send, another started after model initialization, and neither retained failed or cancelled attempts. The team needs to compare a build change without inventing what the old records mean.

Decision

Define an event-based record schema and preserve the original observations instead of normalizing ambiguous durations after the fact.

Expected outcome

Future results identify their setup, task, outcome, and measurement boundaries; historical ambiguity remains explicitly unresolved.

Evidence to collect

The claim becomes reviewable only when the following evidence is attached to the same artifact and test run:

• Schema-valid raw records
• Ordered sustained sequence
• All failures

A release claim is valid only for the recorded configuration and acceptance set. Generalization requires additional records.

Implementation workflow

1. Define a run identity. Bind app and runtime revisions, artifact identity, requested and observed backend, device/OS build, fixture ID, and protocol version. Use stable identifiers rather than human-entered model family names alone.
2. Specify events and units. Document the exact start and end events for every metric, its unit, clock source, and sampling method. Keep launch, artifact verification, engine readiness, first output, and terminal completion distinct.
3. Represent all outcomes and samples. Use explicit success, failure, cancel, timeout, fallback, and invalid-instrumentation states. Preserve ordered samples and errors; define exclusions before the test rather than after looking at results.
4. Apply a privacy-safe fixture reference. Store fixture identifiers and approved hashes instead of prompt or media content. If external reproduction is needed, use a separately governed synthetic or consented fixture set.
5. Validate schema and migration fixtures. Reject unknown units, absent protocol versions, unbound artifact IDs, and success records without required events. Test readers against earlier schema fixtures before releasing a migration.

Keep each transition observable. A failure should identify the stage, artifact, runtime, and recovery action without logging private user content.

Acceptance criteria

• Every numeric observation names its event boundaries, units, workload, app/runtime revisions, and artifact identity.
• Failures, cancellation, fallback, and ordered sustained samples remain present rather than being silently filtered from summaries.
• Schema validation rejects ambiguous units or missing provenance and migration tests preserve historical meaning.
• No user prompt, media payload, document text, or sensitive filename is stored in a routine benchmark record.

Decision table

Failure patterns to prevent

• Publishing aggregates without raw method
• Guessing missing fields

Also prevent silent fallback, unpinned artifacts, missing cancellation, and conclusions that combine unlike configurations. Store unsuccessful runs alongside successful ones.

Minimum reproducibility record

Release checklist

• ☐ The primary query is answered without an unsupported number.
• ☐ Every artifact and runtime is pinned.
• ☐ The representative task and failure policy are explicit.
• ☐ Lifecycle, cancellation, cleanup, and fallback are tested.
• ☐ User-content and network boundaries are documented.
• ☐ Result wording applies only to the recorded configuration.
• ☐ The page links to raw method or evidence when results are added.

Sources and related evidence

• Cove benchmark method
• Android Macrobenchmark setup and result output
• Android thermal status API reference

Chinese deployment and troubleshooting content is organized in the .奇连 AI 端侧专题.

Frequently asked questions