Reproducibility Limits in On-Device AI

Direct answer

Repeated model runs are not automatically expected to produce byte-identical text. An observed difference can come from a changed artifact, runtime revision, backend, sampling configuration, thread or execution settings, preprocessing, prompt, or input—not only from a regression in model quality. A reproducibility protocol pins the known variables, records the actual configuration for each run, and defines what kind of equivalence the task requires. This differs from a generic answer-regression suite: the immediate question is why two runs differ and which conditions must be controlled before comparing them. Start with an immutable run manifest. Record exact model artifact identity or digest, runtime and delegate/backend version, app build, device and OS, prompt/template revision, tokenizer identity where available, input fixture, preprocessing, decoding parameters, and relevant thread/execution settings. Distinguish requested configuration from observed behavior. A flag requesting one backend does not prove it executed; a prompt in source control does not prove it rendered identically after template expansion. Keep the manifest alongside output properties and measurement method. Define the repeatability claim before running. For deterministic parsing or pre/post-processing, exact equality may be appropriate. For generated language, decide whether the requirement is byte equality, structured validity, preservation of critical facts, bounded values, correct citation grounding, or an acceptable human-reviewed outcome. Byte equality is a strong condition and should not be imposed without a reason. Accepting any fluent result is too weak for many tasks. The protocol should name required properties and allowed variation. Run repeated trials in a predeclared sequence. Keep artifact and input fixed; if exploring a sampler parameter, change only that parameter and create a new condition label. Store all outputs in a privacy-safe fixture report or store only reviewed derived properties when content must remain protected. Preserve failures, malformed outputs, empty results, cancellations, and fallback events. Do not select the most favorable run or silently rerun until a desired response appears. If the runtime has nondeterministic device/backend behavior, report what the test observed rather than claiming every source of variation is controlled. Separate configuration drift from run-to-run variation. First verify manifests match on artifact, software, template, preprocessing, and device setup. If they differ, classify the pair as different conditions instead of calling it nondeterminism. If manifests match, inspect declared stochastic decoding parameters, backend behavior, thread scheduling, and application state. The protocol should not assume an undocumented runtime guarantee. Consult exact runtime documentation and record the version; absence of an explicit guarantee means do not promise determinism. Use paired tests to investigate meaningful changes. A fixed input set can reveal whether important properties changed when runtime or artifact changed, but does not automatically establish a universal quality score. Record the rubric, reviewer, and permitted alternatives. For citations or structured output, validate syntax and source linkage separately from wording. For translation or transcription, use task-specific human-reviewed labels rather than comparing raw strings without accounting for legitimate alternatives. Protect test data. Reproducibility does not require uploading user prompts or media. Use synthetic or consented fixtures and record IDs, versions, and hashes as appropriate. Keep private output out of issue trackers and ordinary CI logs. When retaining generated text is necessary, set access and retention policy. Project fact cards require benchmark claims to be grounded in reproducible method and actual evidence; a manifest makes runs inspectable but does not create evidence of accuracy, speed, or device compatibility. Conclude narrowly. Report how many runs used matching configuration, which properties were compared, which differences remained, and the exact artifact/runtime/device scope. If a property is stable in the tested fixture set, say so without promising every future generation will match. If the run was not reproducible because the runtime silently selected a different backend or artifact identity was missing, report the evidence gap and rerun under a traceable setup. Honest uncertainty is preferable to a fabricated deterministic guarantee.

The same local prompt produces different JSON formatting

A developer repeats a structured extraction test and gets valid output in one run and an extra explanatory sentence in another. The app, prompt, and backend were not recorded with the first result.

Decision

Pin and compare the complete run manifest, define schema validity as the acceptance property, and retain both runs rather than choosing the better one.

Expected outcome

The team can distinguish configuration drift from observed run variation and state exactly what the test established.

Evidence to collect

The claim becomes reviewable only when the following evidence is attached to the same artifact and test run:

• Complete run configuration
• Repeated ordered runs
• Property-level assertions

A release claim is valid only for the recorded configuration and acceptance set. Generalization requires additional records.

Implementation workflow

1. Create the complete run manifest. Record artifact identity, runtime/backend, app and OS builds, prompt/template, tokenizer where known, preprocessing, sampling, threads, input fixture, and requested versus observed execution.
2. Choose an equivalence rule for the task. Decide whether the task requires exact bytes, parseable structure, preserved facts, grounded citations, or reviewed acceptability. Justify the rule before seeing trial outputs.
3. Repeat a fixed condition and preserve every outcome. Run a predeclared number and order of trials with the same fixture. Keep errors, cancellation, fallback, malformed and empty output; do not retry selectively until a preferred answer appears.
4. Classify drift before claiming variability. Compare manifests first. Treat changed artifact, template, preprocessing, device, or backend as a different condition; only then inspect behavior under matching records.
5. Report scope and protect content. Use consented or synthetic fixtures, redact outputs where required, and state tested artifact/runtime/device/property coverage. Do not infer general accuracy or deterministic guarantees.

Keep each transition observable. A failure should identify the stage, artifact, runtime, and recovery action without logging private user content.

Acceptance criteria

• Every compared run has a manifest identifying artifact, runtime, backend, prompt, input fixture, and relevant execution settings.
• The task-specific equivalence rule is defined before output review and is stronger than fluency alone where safety or structure matters.
• All declared trials, failures, cancellations, malformed outputs, and fallback outcomes remain accounted for.
• The report does not promise byte identity or universal repeatability unless runtime contract and evidence support it.

Decision table

Failure patterns to prevent

• Promising byte-identical output universally
• Ignoring backend changes

Also prevent silent fallback, unpinned artifacts, missing cancellation, and conclusions that combine unlike configurations. Store unsuccessful runs alongside successful ones.

Minimum reproducibility record

Release checklist

• ☐ The primary query is answered without an unsupported number.
• ☐ Every artifact and runtime is pinned.
• ☐ The representative task and failure policy are explicit.
• ☐ Lifecycle, cancellation, cleanup, and fallback are tested.
• ☐ User-content and network boundaries are documented.
• ☐ Result wording applies only to the recorded configuration.
• ☐ The page links to raw method or evidence when results are added.

Sources and related evidence

• Cove benchmark method
• Google LiteRT overview
• LiteRT-LM Android configuration and sampling options
• Android benchmark methodology and controlled setup

Chinese deployment and troubleshooting content is organized in the .奇连 AI 端侧专题.

Frequently asked questions