Measure Time to First Token on Device

Direct answer

A time-to-first-token measurement is meaningful only after its start and end events are defined. For a user-facing request, a useful end-to-end interval can start when the application accepts the request and end when the first model-generated content becomes visible. A runtime-level interval may instead start when the request is submitted to the inference API and end at the first output event. These answer different questions. Neither should silently include cold model loading in one run and omit it in another. Record model-load readiness, preprocessing, prompt construction, request submission, first runtime output, first rendered output, and completion as separate timestamps where the integration exposes them. Do not assume that an API described as streaming emits one token per callback. LiteRT-LM’s Kotlin guide shows asynchronous conversation output being collected, but the application should use the documented event semantics for its exact version. If callbacks contain chunks, label the metric “time to first output chunk” unless token-level output is actually observable. If output is returned only after generation finishes, the integration cannot measure first-token latency; record that as unavailable rather than estimating it from elapsed time or character count. Use a monotonic clock for intervals and capture timestamps from one clock source. Android’s SystemClock reference distinguishes elapsed realtime clocks from wall clock time, which can change due to user or network updates. Record app build, artifact, runtime, backend request and observed path if available, prompt/template, input fixture, output budget, device, OS, and cold/warm process state. Keep initial artifact download and engine initialization outside a request-level TTFT interval, but measure them separately when the product question is first-use readiness. Prepare a fixed non-sensitive fixture set and run ordered trials with the same configuration. Report every attempt, including failures, timeouts, cancellations, and runs with no output. Define whether user-perceived timing includes queue wait, input preprocessing, request admission, prefill, stream transport, UI rendering, and accessibility announcement. Do not subtract phases unless the corresponding timestamps are observed and the calculation is described. A warm model and a cold process are separate cohorts, not values to average together. The project does not contain a complete real-device benchmark dataset, so this page publishes a method rather than a TTFT result, device ranking, or speed promise. If first-token events are chunked, the record should state the chunk contract. If only UI text is observable, call it first visible output and include rendering cost. The useful result is a timing definition tied to exact artifacts and event boundaries, with enough raw timestamps to reproduce the computation.

A team wants to compare the first visible response after a model change

A local assistant feels slower after an update, but the team has only a stopwatch time from tapping the button to seeing text. The newer build performs preprocessing differently and may start with a cold model, while the old test ran warm.

Decision

Define request-level, runtime-output, and UI-visible boundaries separately; record load and preprocessing as their own stages and compare matched states.

Expected outcome

The team can say which interval changed for the exact fixture without claiming a universal token rate or hiding cold-start cost.

Evidence to collect

The claim becomes reviewable only when the following evidence is attached to the same artifact and test run:

• Clock boundaries
• Cold and warm labels
• All run records

A release claim is valid only for the recorded configuration and acceptance set. Generalization requires additional records.

Implementation workflow

1. Choose the question and interval. Decide whether the metric is user-action-to-visible-output, API-submit-to-first-event, or another named interval. Document whether queue wait, preprocessing, prompt build, model load, and rendering are included.
2. Verify event granularity. Read the exact runtime version’s output contract. Record whether the callback is a token, chunk, or final string. If token-level events are not exposed, do not label a chunk or completed response as first-token latency.
3. Pin clock and configuration. Use one monotonic clock source and record app/artifact/runtime, backend request/evidence, prompt, fixture, output budget, device/OS, and cold/warm state. Measure initial load separately from a request-level interval.
4. Run ordered matched trials. Repeat the same non-sensitive input under each declared process state. Save all timestamps and include failed, cancelled, timed-out, or no-output trials; keep cold and warm cohorts separate.
5. Report boundaries and calculation. Show start/end definitions, raw timestamps, included phases, event granularity, number of trials, and exclusions. If measuring only UI output, state that rendering is part of the interval; avoid deriving missing internal phases.

Keep each transition observable. A failure should identify the stage, artifact, runtime, and recovery action without logging private user content.

Acceptance criteria

• The metric has an explicit start event, end event, clock source, and inclusion/exclusion policy.
• Token-level wording is used only when the runtime exposes token-level events; otherwise the metric names chunk or visible output.
• Cold load, preprocessing, and request-level output timing are separate and cold/warm trials are not mixed.
• All runs and failures are retained; no speed result or device comparison exceeds raw timestamp evidence.

Decision table

Failure patterns to prevent

• Starting after preprocessing
• Publishing the fastest run

Also prevent silent fallback, unpinned artifacts, missing cancellation, and conclusions that combine unlike configurations. Store unsuccessful runs alongside successful ones.

Minimum reproducibility record

Release checklist

• ☐ The primary query is answered without an unsupported number.
• ☐ Every artifact and runtime is pinned.
• ☐ The representative task and failure policy are explicit.
• ☐ Lifecycle, cancellation, cleanup, and fallback are tested.
• ☐ User-content and network boundaries are documented.
• ☐ Result wording applies only to the recorded configuration.
• ☐ The page links to raw method or evidence when results are added.

Sources and related evidence

• Cove benchmark method
• LiteRT-LM official Kotlin streaming conversation guide
• Android SystemClock monotonic time reference

Chinese deployment and troubleshooting content is organized in the .奇连 AI 端侧专题.

Frequently asked questions