Verify an AI App Really Works Offline

Direct answer

An “offline” test must cover the user-visible task from its real starting state, not just one inference call. First declare what setup already happened: model or language assets installed, account state established, permissions granted, and any cache warmed. Then block connectivity and execute the same steps a user would rely on, including capture or import, preprocessing, inference, result display, save, and optional output. `ARCH-01` defines on-device AI by where inference executes, while `PRIVACY-01` cautions that local inference does not prove the application makes no other requests. Use both facts to scope claims precisely: a task may have a local inference stage while setup, account, update, telemetry, or an optional feature still depends on a network. Separate cold and prepared conditions. If a model download was completed earlier, say so; do not call installation offline merely because the later task runs without connectivity. Test a fresh installation separately from a device whose assets are already present. Record model identity and app build, but do not publish an unsupported device verdict. A cache hit is not equivalent to executing the intended local pipeline. Use a unique input that cannot be mistaken for a saved prior output, clear the app's relevant cache where safe, and verify the observed result corresponds to the new request. Use more than airplane mode as evidence. Airplane mode is a useful controlled condition, but platform radios or developer settings can complicate interpretation, and cached services may still respond. Combine controlled network denial, endpoint capture during setup and task, code-path review, and dependency inventory. Android Studio Network Inspector displays supported HttpsURLConnection and OkHttp traffic; other networking libraries may not appear. Record that limitation. Compare before, during, and after the task, including reconnect behavior, so delayed uploads or telemetry are not confused with inference traffic. The task matrix should isolate stages. Test model absent, model present, account unavailable if relevant, permission denied, network lost before launch, network lost during a request, app backgrounded, process recreated, and optional speech or export unavailable. Note whether each state yields a local result, an honest blocked state, or a separate remote route. Do not silently retry remotely after local failure. If a speech recognizer, voice, map, or account check is platform-managed, determine its behavior from the specific official documentation and the target configuration rather than assuming that the AI model's location settles the question. Define acceptance criteria before running the test: which outcome is acceptable, what counts as a visible network dependency, and what failure must be explained. Keep a run sheet with build hash, exact steps, connectivity controls, starting cache state, route, output identifier, request capture, and any error stage. Avoid recording private user inputs; synthetic fixtures are preferable. Repeat on each configuration for which a public claim is made, and restrict the claim to those records. One successful run supports only the recorded setup and workflow; it is not evidence that every feature, language, model, or Android device is offline-capable. Finally, communicate offline boundaries in the interface. Tell the user which assets must be downloaded first, what remains available without a connection, and which optional capabilities may be unavailable. If the app queues work for later, say so and disclose what will be sent when connectivity returns. An offline badge should reflect the current route, not a static product identity. The result is an auditable workflow that distinguishes local execution from complete offline operation and does not overstate privacy based on a single model call.

A translator task is tested after model setup

A QA engineer downloads the required artifact, then wants to verify that importing a new phrase, translating it, saving the result, and replaying the output works with connectivity blocked. A warm cache could hide a dependency.

Decision

Record the prepared starting state, use a fresh synthetic phrase, inspect network behavior around the complete flow, and test optional output separately.

Expected outcome

The test report says exactly which stages completed locally, what was unavailable, and whether any non-inference request occurred in the tested build.

Evidence to collect

The claim becomes reviewable only when the following evidence is attached to the same artifact and test run:

• Setup state
• End-to-end task result
• Traffic after reconnect

A privacy claim needs a data inventory and traffic evidence. A local-model label by itself is not proof.

Implementation workflow

1. Define the offline claim and starting state. Name the user task, app build, artifact identity, account state, permissions, prior setup, and cache condition. Distinguish offline installation from an offline task after setup.
2. Create a fresh end-to-end fixture. Use unique synthetic content, clear only the relevant cache safely, and write the expected visible result or acceptance property before running. Do not reuse a prior output as proof.
3. Block and observe connectivity. Use a controlled network block and capture supported endpoints during setup, task, and reconnect. Review call sites and dependencies for clients the capture tool cannot observe.
4. Exercise failure and optional paths. Test missing assets, permissions, account checks, interruption, process recreation, remote fallback conditions, speech output, and export. Ensure no route switch or queued transfer is hidden.
5. Limit the conclusion to the record. Store the build, exact steps, network state, cache state, observed route, outcomes, and gaps. Repeat on any additional configuration before widening a claim.

Keep each transition observable. A failure should identify the stage, artifact, runtime, and recovery action without logging private user content.

Acceptance criteria

• The report distinguishes pre-downloaded setup from the tested task and uses input that cannot be confused with a stale cached answer.
• Every feature stage has a result under connectivity denial, and any unavailable or queued remote step is visible and described.
• Network observations identify the inspected libraries and blind spots; the published statement is limited to the tested build, task, and configuration.

Decision table

Failure patterns to prevent

• Testing model only
• Account dependency hidden

Also prevent silent fallback, unpinned artifacts, missing cancellation, and conclusions that combine unlike configurations. Store unsuccessful runs alongside successful ones.

Minimum reproducibility record

Release checklist

• ☐ The primary query is answered without an unsupported number.
• ☐ Every artifact and runtime is pinned.
• ☐ The representative task and failure policy are explicit.
• ☐ Lifecycle, cancellation, cleanup, and fallback are tested.
• ☐ User-content and network boundaries are documented.
• ☐ Result wording applies only to the recorded configuration.
• ☐ The page links to raw method or evidence when results are added.

Sources and related evidence

• On-device AI architecture overview
• Android Studio Network Inspector
• Android Data Saver and network use
• Android reading network state

Chinese deployment and troubleshooting content is organized in the .奇连 AI 端侧专题.

Frequently asked questions