LiteRT-LM Streaming Output on Android

Direct answer

Streaming output is a sequence of events, not a promise that every emitted fragment is complete or safe to display as a final answer. LiteRT-LM’s Kotlin API guide demonstrates collecting generated output from a conversation. The application still owns threading, request identity, UI state, cancellation, parsing, and resource lifetime. Initialize and run inference away from the main thread; deliver bounded events to a ViewModel or repository; and distinguish partial text, completion, error, and cancellation. A streaming UI should have an explicit state machine: idle, initializing, generating, finalizing, complete, cancelled, and failed. When a new request starts, attach a unique request ID and clear or archive the prior result intentionally. Every event must be checked against the active ID so delayed chunks from an older stream cannot overwrite a newer request. Recomposition should only render state; it must not create another conversation or re-collect the native stream in a way that duplicates inference. Partial output may contain an unfinished sentence, malformed JSON, an incomplete tool proposal, or sensitive text the user later cancels. Decide whether to show fragments immediately, show them as draft, or buffer until a safe boundary. If the output is structured, do not run a parser or tool handler on arbitrary intermediate fragments; wait for the protocol boundary, parse strictly, then validate. A “stop” action should propagate to the actual coroutine and runtime cancellation path where supported. Removing a screen collector does not prove native generation stopped. Define ownership across layers. The inference coordinator owns the engine and conversation according to the library contract. The ViewModel owns task state and the active job; the UI owns a rendering lifecycle, not native resources. On navigation away, decide whether generation continues or cancels. On process death, an in-process stream is gone; do not restore a spinner as if it were still running. If resumable work is required, persist task inputs and a stage marker, then reinitialize rather than serialize a native conversation. Error handling should preserve the stage: model initialization, conversation creation, streaming, output validation, cancellation, or close. Do not show a partial response as success if the stream ended with an error. If the model emits no tokens, state that the request produced no result rather than inventing a fallback answer. Keep sensitive prompt and output text out of logs unless the user chooses a support export. Test with fixed prompts that cover short output, multi-event output, empty output, malformed structured output, cancellation between chunks, a second request arriving before the first completes, screen rotation, navigation, process recreation, and close. Record event order and final state. The test proves only the exact artifact/runtime path and UI policy; it is not an inference speed or model-quality benchmark. Implementation sequence: 1. Build an app-owned request state machine with request ID, partial content, final content, error, and cancellation state. 2. Collect output on a background coroutine and forward immutable events; keep Compose recomposition free of engine creation and native stream subscriptions. 3. Route stop to the runtime owner, cancel collectors, close conversations according to the API contract, and ignore chunks from stale request IDs. 4. Validate final output only at its documented completion boundary; structured actions require schema checks and host authorization. 5. Verify every event sequence with deterministic fixtures and confirm no task is presented as complete after an error or process death. Streaming is a UI and lifecycle contract layered over generation. It does not establish that a specific Cove release exposes streaming or that every runtime supports identical cancellation behavior.

A second prompt arrives while the first stream is still generating

An Android user submits a question and sees partial text. Before completion, they change the request and tap Generate again. The old stream later emits more text while the new request is active.

Decision

Assign unique request identities, define whether the previous job is cancelled, and discard stale events rather than merging streams into one answer.

Expected outcome

The screen shows a single coherent state and never labels an interrupted or stale partial response as the new request’s final answer.

Evidence to collect

The claim becomes reviewable only when the following evidence is attached to the same artifact and test run:

• Pinned runtime revision
• Ordered partial-output trace
• Cancel and restart result

A runtime comparison is publishable only when artifact identity, build configuration, selected backend, task fixture, and failures are recorded.

Implementation workflow

1. Define event and request identity. Represent initialization, partial output, completion, cancellation, and failure as distinct immutable events tied to one request ID. Decide how a new request affects any active stream.
2. Keep collection outside UI recomposition. Start collection in an application-owned coroutine scope or ViewModel job, emit state updates, and ensure Compose only renders current state. Recomposition must not create another engine or duplicate request.
3. Handle partial output safely. Choose a draft display policy, wait for complete boundaries before parsing structured data, and route cancellation to the runtime where supported. Ignore any chunk whose request ID is no longer active.
4. Exercise lifecycle and failure cases. Test empty stream, parser error, stop between chunks, overlapping requests, rotation, navigation, process death, and close. Verify the final UI state matches the actual terminal event.

Keep each transition observable. A failure should identify the stage, artifact, runtime, and recovery action without logging private user content.

Acceptance criteria

• Events are associated with one request ID and stale chunks cannot modify the active result.
• Cancellation reaches the execution owner and the UI does not report partial output as complete.
• Initialization, generation, validation, cancellation, and close errors remain distinguishable.

Decision table

Failure patterns to prevent

• Updating Compose state from native callbacks
• Treating a dropped collector as native cancellation

Also prevent silent fallback, unpinned artifacts, missing cancellation, and conclusions that combine unlike configurations. Store unsuccessful runs alongside successful ones.

Minimum reproducibility record

Release checklist

• ☐ The primary query is answered without an unsupported number.
• ☐ Every artifact and runtime is pinned.
• ☐ The representative task and failure policy are explicit.
• ☐ Lifecycle, cancellation, cleanup, and fallback are tested.
• ☐ User-content and network boundaries are documented.
• ☐ Result wording applies only to the recorded configuration.
• ☐ The page links to raw method or evidence when results are added.

Sources and related evidence

• LiteRT-LM repository
• Google AI Edge LiteRT-LM Kotlin API getting started
• Android Developers — processes and threads
• Android Developers — activity lifecycle

Chinese deployment and troubleshooting content is organized in the .奇连 AI 端侧专题.

Frequently asked questions