Build a Device Matrix for On-Device AI

Direct answer

A device matrix for on-device AI is a set of traceable test records, not a compatibility list copied from chipset specifications. Each row should identify one exact device and software configuration, one model artifact, one runtime revision, one requested and—if observable—actual execution path, and one fixed task with explicit acceptance criteria. A result belongs to that row only. If the device, OS build, artifact quantization, runtime, prompt, or backend changes, create a new row rather than silently reusing an old pass. This makes the matrix useful for engineering decisions while keeping its claims bounded. Start with what you can verify directly: product and device identifier, manufacturer-reported model, Android version and API level, supported ABI, app build, runtime package and resolved revision, exact artifact reference or digest if available, and task fixture version. Record unavailable values as unknown; do not infer an accelerator from a marketing name, infer performance from a chipset family, or call a requested GPU/NPU path confirmed unless the runtime exposes evidence that it executed. Google’s LiteRT-LM run documentation describes runtime and acceleration paths, but the existence of a documented path is not proof that a particular app configuration used it. Define the task before gathering results. A text summarization row, an OCR row, and a speech transcription row are not interchangeable workload evidence. Include input modality, preprocessing version, prompt or template, output parser, start and stop boundaries, and acceptance criteria. Use reviewed synthetic fixtures when possible. Record outcomes such as passed, failed, unsupported by observed evidence, not run, or inconclusive; avoid a single green “device supported” cell that collapses distinct stages like artifact verification, load, task completion, and output validation. The matrix should preserve failures and missing data. Keep a stable run ID, timestamp, operator or automation version, raw measurements if collected, error class, and recovery result. For a benchmark result, the project evidence policy calls for device, chipset, OS, artifact, runtime, backend, input, timing, memory, thermal state, and failure outcome. A matrix that lacks raw measurements can still guide reproducibility and coverage, but it cannot justify a numeric performance headline. Keep device-specific observations separate from tier summaries; a tier needs a defined membership rule and sufficient data for the claim it makes. Use the matrix to decide the next experiment, not to create false certainty. If two rows differ in several variables, the result cannot isolate which one caused a behavior change. Add controlled rows that vary one dimension where practical, preserve the prior evidence, and repeat results that materially affect a release decision. The deliverable is an auditable record of what was actually tested and what remains unknown—not a declaration that all devices with a particular chip can run every on-device model.

A successful test on one phone is being used to approve a device family

A release review has one passing local inference run from a phone with a familiar chipset label. The team has not pinned the artifact digest, runtime revision, OS build, or backend evidence, and now needs to plan a defensible validation matrix without claiming support beyond tested configurations.

Decision

Make every run a unique configuration row, record unknown fields explicitly, and keep task acceptance separate from device-family conclusions.

Expected outcome

Reviewers can identify exact coverage, missing dimensions, and the next high-value experiment without treating a single pass as a compatibility certification.

Evidence to collect

The claim becomes reviewable only when the following evidence is attached to the same artifact and test run:

• Physical device identity
• Exact software stack
• Completed and failed results

A release claim is valid only for the recorded configuration and acceptance set. Generalization requires additional records.

Implementation workflow

1. Define the unit of evidence. Use one row for one exact device/software/artifact/runtime/task configuration. Give it a stable run ID and version the schema. A change in artifact, runtime, OS build, prompt, modality, or backend request creates a new row.
2. Capture identity and provenance. Record device model and relevant build properties, Android/API level, supported ABI, app build, runtime coordinate and resolved version, artifact revision/format/quantization, and the trusted digest when one exists. Mark absent evidence unknown rather than filling it from a family name.
3. Separate requested from observed execution. Store the backend preference or request in one field and the runtime’s execution evidence in another. If the runtime does not expose actual backend identity, label it unconfirmed. Do not infer NPU/GPU support from chipset marketing or a settings toggle.
4. Freeze the task and acceptance contract. Identify modality, input fixture, preprocessing, prompt/template, decoding configuration, parser revision, and expected properties. Keep text, image, and audio runs distinct; a task that loads successfully but fails output assertions is not a pass for the feature.
5. Retain failures and review coverage. Record pass/fail/not-run/inconclusive, error stage, cancellation or recovery, and raw data where collected. Compare rows only after listing their differences. Use the coverage table to select controlled follow-up runs and state conclusions at the exact scope the data supports.

Keep each transition observable. A failure should identify the stage, artifact, runtime, and recovery action without logging private user content.

Acceptance criteria

• Every matrix row binds an outcome to an exact device, OS, app, artifact, runtime, execution-path evidence, task, and fixture version.
• Requested backend and observed backend are separate; unavailable evidence is marked unknown rather than inferred.
• Failures, cancellations, unrun configurations, and inconclusive outcomes remain visible and are not collapsed into supported/unsupported without criteria.
• Any summary or compatibility statement names the tested scope and does not generalize from chipset family or a single phone.

Decision table

Failure patterns to prevent

• Using chipset specs alone
• Turning one device into a support family

Also prevent silent fallback, unpinned artifacts, missing cancellation, and conclusions that combine unlike configurations. Store unsuccessful runs alongside successful ones.

Minimum reproducibility record

Release checklist

• ☐ The primary query is answered without an unsupported number.
• ☐ Every artifact and runtime is pinned.
• ☐ The representative task and failure policy are explicit.
• ☐ Lifecycle, cancellation, cleanup, and fallback are tested.
• ☐ User-content and network boundaries are documented.
• ☐ Result wording applies only to the recorded configuration.
• ☐ The page links to raw method or evidence when results are added.

Sources and related evidence

• Cove benchmark method
• Gemma run guide
• Android Build device and build information API
• Google Gemma and LiteRT-LM run guide

Chinese deployment and troubleshooting content is organized in the .奇连 AI 端侧专题.

Frequently asked questions