Accessibility Testing for Offline AI Results

Direct answer

Accessibility testing for an on-device AI feature should verify what a person using TalkBack, switch access, keyboard navigation, or another assistive technology can perceive and operate during a changing task—not only whether a model returned text. Progress, partial streaming output, cancellation, errors, retry controls, and final results need meaningful labels, focus behavior, and announcements. A stream that updates visually may overwhelm a screen reader if every token is announced; a silent update may leave the user unaware that work completed. Define a usable announcement policy and verify it through the real interaction path. Start with a semantic map of the screen. Identify input, submit action, progress state, partial-output region, stop/cancel control, error message, retry action, and final result. Check that each has an accessible name, role, state, and relationship where appropriate. Focus must not jump unpredictably when a chunk arrives or become trapped in a progress element. The user should still be able to stop work while generation is running. Error text should describe the problem and offer a next step without exposing native stack traces or private prompt content. Streaming needs an explicit announcement strategy. Decide whether to announce task start and completion, meaningful sentence-sized updates, or provide a user-controlled “read current result” action. Avoid announcing every token. When output changes, do not repeatedly reset screen-reader focus; provide a stable result region while letting the user remain in control. Test cancellation during streaming, navigation away and back, empty output, timeout, and retry. A spinner stopping is insufficient if semantic state still says “in progress”; do not announce completion before output validation finishes. Use Android accessibility testing guidance and Compose semantics tests where applicable, then manually traverse the feature with TalkBack and keyboard or switch navigation on a test device. Automated checks can find missing labels and some semantics problems, but they do not establish full usability or legal compliance. Use synthetic inputs and outputs rather than private conversations. The repository benchmark fact makes this a validation method, not an accessibility score or device benchmark. The on-device definition describes where inference runs; it does not imply that a task is usable to every person or that network and privacy boundaries are automatically correct. Record the app build, Android version, input modality, accessibility service/settings, task fixture, semantic state transitions, announcements heard, focus order, errors, and recovery. If a screen reader or test API cannot expose a behavior, mark it unobserved and add manual review. The outcome is a scoped test record and actionable defects—not a blanket claim that the app is accessible or compliant. Dynamic content requires special review. Do not force TalkBack to read every newly appended token or move focus to the result after each update. A live-region semantic may be appropriate, but its behavior and interruption policy must be tested. Error and cancellation announcements should be near the relevant action and remain discoverable. If a label changes while generation continues, verify that assistive technology reports the current state rather than stale cached text. Conclusions apply only to the tested flow and settings. Passing a TalkBack test does not prove keyboard navigation, magnification, switch access, or every locale. Record which assistive technologies, task sequence, build, and device were used. Without a full audit or user study, avoid claims such as “fully accessible” or “compliant with all standards.”

A streamed answer updates visually but is not usable with TalkBack

A local assistant shows a spinner and appends generated text during inference. A TalkBack user hears no completion announcement, while focus returns to the submit button after screen updates. The model output itself appears valid.

Decision

Test semantic state, stable focus, announcement cadence, cancellation, and error recovery with synthetic output and manual assistive-technology traversal.

Expected outcome

The team identifies whether the gap is semantics, announcements, focus, or task state without confusing accessibility with model quality.

Evidence to collect

The claim becomes reviewable only when the following evidence is attached to the same artifact and test run:

• Screen-reader traversal
• Focus order
• Error and progress announcements

A release claim is valid only for the recorded configuration and acceptance set. Generalization requires additional records.

Implementation workflow

1. Map accessible controls and states. List input, submit, progress, partial output, cancel, error, retry, and result elements. Define accessible name/role/state, focus order, and which changes a user must hear.
2. Choose a streaming announcement policy. Decide between start/end announcements, meaningful sentence-sized updates, or user-triggered reading of current output. Do not announce every token or steal focus on each chunk.
3. Exercise task and recovery states. Use a fixed synthetic response to test start, partial output, completion, cancellation, empty output, timeout, retry, navigation, and return. Confirm semantic state matches visible state at every transition.
4. Run automated semantics and manual traversal. Use Compose semantics and Android accessibility test tools where applicable, then manually navigate with TalkBack and keyboard or switch access. Treat automation as partial coverage, not certification.
5. Record defects and bound conclusions. Log build, OS, service/settings, fixture, focus changes, announcements, and recovery. Use synthetic content, redact production diagnostics, and mark unobservable behaviors for manual follow-up.

Keep each transition observable. A failure should identify the stage, artifact, runtime, and recovery action without logging private user content.

Acceptance criteria

• Each control and task state has a meaningful accessible name/role/state and predictable focus behavior.
• Streaming does not announce every token, repeatedly steal focus, or leave completion/cancellation semantically unreported.
• Error, empty-output, timeout, cancellation, and retry states are manually traversable and match visible state.
• Reports identify the tested app/OS/service and do not claim universal accessibility certification or model quality.

Decision table

Failure patterns to prevent

• Color-only status
• Unlabeled streaming output

Also prevent silent fallback, unpinned artifacts, missing cancellation, and conclusions that combine unlike configurations. Store unsuccessful runs alongside successful ones.

Minimum reproducibility record

Release checklist

• ☐ The primary query is answered without an unsupported number.
• ☐ Every artifact and runtime is pinned.
• ☐ The representative task and failure policy are explicit.
• ☐ Lifecycle, cancellation, cleanup, and fallback are tested.
• ☐ User-content and network boundaries are documented.
• ☐ Result wording applies only to the recorded configuration.
• ☐ The page links to raw method or evidence when results are added.

Sources and related evidence

• Cove benchmark method
• On-device AI architecture overview
• Android accessibility testing guidance
• Compose accessibility and semantics testing

Chinese deployment and troubleshooting content is organized in the .奇连 AI 端侧专题.

Frequently asked questions