Build On-Device AI Test Fixtures

Direct answer

A test fixture is a controlled example that lets the team exercise an input path and evaluate an outcome again after a change. For on-device AI, a fixture includes more than a text prompt: it may include a source image or audio clip, its provenance and consent status, preprocessing revision, task instructions, expected properties, and a rule for human review. If any of these change without a new version, a before-and-after result may no longer be comparable. The project fact card calls for reproducible workloads and privacy-safe records; fixture design is the practical place to enforce both. Use distinct fixture classes for distinct questions. A parser fixture can be a small synthetic JSON response with a known schema; an OCR fixture can be a licensed or self-created image with a verified transcription; an audio fixture can include a consented recording plus timing and transcript annotations. A benchmark fixture may be intentionally representative, while a boundary fixture targets empty input, unusually long content, mixed scripts, or a malformed value. Do not mix them into one unlabelled folder or treat a few handpicked “happy path” examples as a comprehensive quality set. Every fixture should carry a stable ID, version, source, consent or license record, input hash, preprocessing version, expected output properties, and a retention owner. The hash helps detect accidental mutation but does not replace the source record. Do not place user content inside source code, screenshots, open issue trackers, or CI logs. Use synthetic content where possible. If real media is required to test a property, minimize it, obtain permission, restrict access, and specify deletion. An anonymized filename alone does not make an identifiable voice or photo safe to publish. Write assertions at the right level. Some tests can compare exact deterministic transformations, such as a parser accepting a known structure. Model generation can vary; use property assertions when exact text is not justified: required named entities preserved, output parses to a schema, citations refer to supplied passages, or a response stays within a declared safety boundary. Human review should use a rubric that records the question, expected behavior, allowed alternatives, and severity. Never silently update the expected answer just to make a regression test green; require review and a new fixture version. Keep fixture execution isolated from user state. A test should not upload private content, modify a user's saved notes, or leave temporary model files behind. Record the fixture ID and result, not its private bytes. When a test needs a large model artifact, pin its revision and integrity evidence separately from the fixture. A changed preprocessing algorithm should create a new pipeline version; otherwise, a regression could be misattributed to a model update. Use both repeatability and coverage deliberately. Replaying the same fixture catches some regressions; it does not cover all languages, lighting, noise, formatting, or user intent. Maintain a coverage map by task, input modality, failure boundary, locale/script where applicable, and expected outcome. Add a new fixture when a real bug is found, but sanitize it and explain which behavior it captures. Keep a minimal smoke set for routine tests and a broader review set for release decisions. Do not claim representative coverage unless the sampling plan supports that statement. Finally, make fixture governance part of code review. Reviewers check provenance, access, retention, expected-property justification, and version changes alongside code. CI should report fixture IDs, versions, and pass/fail states without printing sensitive source data. If a fixture expires or its license changes, remove it from executable collections and document any remaining historical record. A good fixture set is maintained data, not a pile of files that silently accumulates. The output of this process is an auditable test corpus, not a guarantee of real-world accuracy. The repository's RAG-01 and benchmark facts support source traceability and reproducible method; they do not certify that an example suite covers every user or that one model passes a task. Publish conclusions only for the fixtures, preprocessing, artifact, runtime, and review rubric that were actually exercised.

A prompt edit appears to fix a document-summary regression

The team replays a handful of documents copied from support tickets, but the files change between runs and nobody knows which preprocessing code produced the expected summary. Reviewers cannot tell whether a difference is model behavior or merely a changed fixture.

Decision

Replace copied user records with governed fixtures whose input, preprocessing, expected properties, and version are explicit.

Expected outcome

The same tests can be repeated without leaking user content or silently redefining a passing answer.

Evidence to collect

The claim becomes reviewable only when the following evidence is attached to the same artifact and test run:

• Fixture hashes
• Expected output policy
• Preprocessing revision

A release claim is valid only for the recorded configuration and acceptance set. Generalization requires additional records.

Implementation workflow

1. Choose a fixture for one test question. Label whether the example tests parsing, model behavior, OCR, audio, retrieval, or a boundary case. Keep representative coverage, stress cases, and smoke fixtures distinguishable.
2. Record provenance and permission. Give each fixture an ID and version, source/license or consent record, input hash, retention rule, preprocessing revision, and a human owner for renewal or deletion.
3. Define expected properties before running. Specify exact output only when justified. Otherwise record required fields, preserved facts, allowed alternatives, refusal boundary, and reviewer rubric so stochastic generation is judged consistently.
4. Run without user-state side effects. Use isolated test accounts and controlled storage. Log fixture identifiers and outcomes rather than private bytes, and pin the model artifact independently from the input set.
5. Review updates as data changes. When a bug fix changes preprocessing or expected behavior, create a new fixture version with rationale and review. Do not edit the expected result simply to silence a failing test.

Keep each transition observable. A failure should identify the stage, artifact, runtime, and recovery action without logging private user content.

Acceptance criteria

• Each fixture has stable provenance, permission/license, version, preprocessing identity, and retention ownership.
• Expected outputs use exact matching only where justified; variable model responses use reviewed, testable properties.
• CI and routine logs expose IDs and outcomes without printing prompts, images, audio, or private document content.
• A changed fixture or pipeline creates an auditable version and does not silently rewrite previous test meaning.

Decision table

Failure patterns to prevent

• Using private user data
• Changing fixtures between candidates

Also prevent silent fallback, unpinned artifacts, missing cancellation, and conclusions that combine unlike configurations. Store unsuccessful runs alongside successful ones.

Minimum reproducibility record

Release checklist

• ☐ The primary query is answered without an unsupported number.
• ☐ Every artifact and runtime is pinned.
• ☐ The representative task and failure policy are explicit.
• ☐ Lifecycle, cancellation, cleanup, and fallback are tested.
• ☐ User-content and network boundaries are documented.
• ☐ Result wording applies only to the recorded configuration.
• ☐ The page links to raw method or evidence when results are added.

Sources and related evidence

• Cove benchmark method
• Android testing fundamentals
• Android test doubles and controlled dependencies

Chinese deployment and troubleshooting content is organized in the .奇连 AI 端侧专题.

Frequently asked questions