Telemetry Without User Content in AI Apps

Direct answer

Telemetry for an on-device AI feature should answer a bounded product question using a reviewed event schema, not copy the user's task into an analytics field. Start by writing down which operational decisions the events must support: for example, whether a task was started, which broad route was selected, whether it ended, and which non-content failure category occurred. Then define each field, its permitted values, its retention purpose, and the component that emits it. Reject free-form strings by default. A local inference path removes the need to send the prompt to a remote inference server, but it does not establish that every other request is absent. This distinction is captured by `PRIVACY-01` and the separately documented Cove boundary in `PRIVACY-02`; neither fact card authorizes an unreviewed analytics payload. Treat event design as a data-flow change. A developer may intend to send only a failure code while an exception message, URL, filename, model path, or serialized request is attached automatically. Android security guidance warns against sensitive information in logs; analytics wrappers and crash hooks deserve the same field-by-field inspection. Build an allowlist with enums and bounded numeric values. Do not assume a field is anonymous because it lacks a name: a stable installation token, rare task sequence, timestamp, or path can still distinguish activity. Where a measurement does not change a product decision, remove it rather than collect it “just in case.” A useful event can be as narrow as `task_started` with a task category selected from a finite set, an execution route such as `local` or `remote` only when that distinction exists in the implementation, and a schema version. A completion event can record success, cancellation, or a reviewed failure class. It should not contain the input, output, transcript, image URI, document title, prompt template expansion, or generated rationale. For an error, prefer a stable app-owned category over an exception's raw message. Keep debugging material in a separately controlled developer build or a user-reviewed diagnostic export; do not silently turn it into telemetry. Prove the contract at several boundaries. Unit-test the event encoder with sensitive canary strings and assert that none appear in serialized output. Inspect the network request body in a test environment, then inspect the release build's dependencies and destinations. Android Studio Network Inspector can display supported `HttpsURLConnection` and OkHttp traffic in an inspected process, but it does not see every possible networking library; its empty timeline alone is not proof of no network activity. Combine source review, dependency inventory, endpoint capture, and a complete task run. Document what was observed, what was not observable, and the exact build tested. Consent and controls need truthful scope. If telemetry is optional, the opt-out must affect actual event emission, including failure and startup paths. If some service events are required for a specific operation, explain that separately rather than presenting one global toggle as a universal network switch. Define retention and access in the product's own policy from verified implementation and service settings; do not invent a retention duration. If schema or SDK changes, rerun the canary, payload, endpoint, offline-task, and opt-out tests. A clean test run is evidence for the build and routes inspected—not a permanent guarantee about future dependencies or every user environment. The release record should identify the event schema revision, app build, inspected destinations, tests, and any remaining limitations. This makes “we do not place user content in telemetry” a testable design statement instead of an assumption based on local inference.

A failure dashboard must not become a prompt archive

A team wants to know whether a local summarization task starts, completes, is cancelled, or fails during model initialization. Its first analytics proposal includes an exception string and a request object “to make debugging easier.”

Decision

Replace the open-ended request object with a versioned event allowlist, stable failure categories, and a separately user-reviewed diagnostic path.

Expected outcome

The dashboard can distinguish task outcomes while automated tests demonstrate that chosen prompt and output canaries do not enter event payloads.

Evidence to collect

The claim becomes reviewable only when the following evidence is attached to the same artifact and test run:

• Event schema
• Redaction tests
• Opt-out behavior

A privacy claim needs a data inventory and traffic evidence. A local-model label by itself is not proof.

Implementation workflow

1. Write the decision before the event. For every proposed event, state the product decision it will support and the minimum field needed. Delete fields that do not change that decision. This prevents broad “debug context” objects from becoming an undocumented collection surface.
2. Define a closed schema. Use event names, schema versions, finite task and route enums, and reviewed result categories. Prohibit free-form prompts, generated output, exception messages, file paths, content URIs, account identifiers, and model conversation objects unless a separately reviewed requirement justifies them.
3. Test serialization with canaries. Pass unique synthetic strings through success, cancellation, initialization failure, and parser-failure branches. Inspect the exact serialized request and assert that the canaries and sensitive field names are absent. Include nested exceptions and automatic SDK context in the test plan.
4. Inspect the release network path. Inventory dependencies and destinations, exercise a complete task, capture supported requests with Network Inspector or an equivalent test proxy, and review source for other clients. Record the inspector limitation for libraries it cannot observe; do not interpret a blank capture as proof.
5. Verify controls and repeat after change. Exercise telemetry enabled and disabled states, including process startup and error handling. Re-run the same payload and endpoint checks after SDK, event schema, or routing changes, and update the reviewed data-flow record.

Keep each transition observable. A failure should identify the stage, artifact, runtime, and recovery action without logging private user content.

Acceptance criteria

• Every emitted field has a documented purpose, allowed value set, and owner; there is no generic serialized prompt, transcript, image, document, or exception field.
• Synthetic content canaries are absent from payloads across success and failure paths, and the release dependency and destination inventory is recorded.
• The opt-out test proves event emission changes in all covered paths, while documentation limits the claim to the build and network surfaces actually inspected.

Decision table

Failure patterns to prevent

• Prompt text in events
• File paths in crash context

Also prevent silent fallback, unpinned artifacts, missing cancellation, and conclusions that combine unlike configurations. Store unsuccessful runs alongside successful ones.

Minimum reproducibility record

Release checklist

• ☐ The primary query is answered without an unsupported number.
• ☐ Every artifact and runtime is pinned.
• ☐ The representative task and failure policy are explicit.
• ☐ Lifecycle, cancellation, cleanup, and fallback are tested.
• ☐ User-content and network boundaries are documented.
• ☐ Result wording applies only to the recorded configuration.
• ☐ The page links to raw method or evidence when results are added.

Sources and related evidence

• Cove network audit
• Android security and privacy overview
• Android Studio Network Inspector
• Android log information disclosure risks

Chinese deployment and troubleshooting content is organized in the .奇连 AI 端侧专题.

Frequently asked questions