Test Names and Numbers in Offline Translation

Direct answer

Test names and numbers as their own acceptance category instead of hiding them inside a single “translation quality” score. A fluent sentence can still change a person’s name, reverse day and month, alter a decimal separator, omit a unit, or normalize an identifier that must remain exact. Before running a model, create a versioned bilingual fixture set and annotate which spans must be preserved exactly, which may be localized according to a written rule, and which require user confirmation. Run each source-to-target direction separately. A passing result in one direction is not evidence for the reverse direction, another model artifact, an image workflow, or speech recognition. This is a test-method article, not a claim about offline translation accuracy or language coverage. The repository’s benchmark fact explicitly says it lacks a complete real-device on-device AI dataset and permits methodology/schema content while requiring raw records before result claims. The system’s on-device definition also distinguishes where inference runs; it does not certify linguistic quality. Google’s model documentation may describe variants and supported task families, but a family-level statement does not establish that a particular app, artifact, runtime, device, or language pair preserves the fixture’s entities correctly. Measure the app path you intend to ship, and label every result with the exact inputs. Design fixtures from realistic but non-sensitive examples: a traveler’s booking code, a medicine label only when reviewed by qualified people, a street address, a decimal amount, a date with ambiguous ordering, a telephone number, and names with diacritics or transliteration conventions. For each item, set a rule before viewing outputs: exact-character match, normalized equivalence, semantic requirement, or reject-and-ask behavior. Preserve the source beside the translation so a human can verify uncertain spans. Never infer a medical, legal, or financial consequence from a translated entity; the test checks transformation behavior, not professional correctness. Run the same fixture under controlled conditions: freeze the artifact, runtime, prompt, preprocessing, decoding settings, and app build; record language direction; execute every case; compare outputs against the annotations; and classify errors by entity type. If a case fails, preserve the failed example in an approved synthetic corpus, adjust one component, and replay the complete set. Report denominators and error classes, not just a cherry-picked success. If no raw device records exist, publish the protocol and leave performance or quality conclusions blank. An honest test suite makes regression visible without promising that every real-world name or number will be translated perfectly.

An offline travel translator changes a booking code in a fluent sentence

A team is preparing a release candidate and notices that one translated itinerary sounds natural but changes the final character of a booking reference. The app has multiple modalities, but the suspected issue came from typed text on one exact artifact and direction.

Decision

Create entity-specific acceptance rules and run a versioned typed-text fixture in both directions separately. Preserve source and output for authorized review, while keeping any public statement scoped to the test set.

Expected outcome

The team can identify whether the defect is exact-span preservation, direction-specific behavior, preprocessing, or rendering, and can add a repeatable regression case without claiming universal accuracy.

Evidence to collect

The claim becomes reviewable only when the following evidence is attached to the same artifact and test run:

• Versioned bilingual entity corpus
• Direction-specific acceptance rules
• Exact-span preservation and failure outcomes

A release claim is valid only for the recorded configuration and acceptance set. Generalization requires additional records.

Implementation workflow

1. Define entity classes and rules. List names, dates, times, addresses, quantities, units, currency, and identifiers. For each class, decide whether exact bytes, normalized equivalence, semantic meaning, or a user-confirmation step is required. Make the rule explicit before testing.
2. Build a versioned, reviewed fixture set. Use synthetic or permission-cleared examples with a source-language label, target-language label, direction, entity spans, expected rule, and reviewer note. Include ambiguous dates, decimals, diacritics, and identifiers that resemble ordinary text. Do not put real personal records into public test assets.
3. Freeze and label the application path. Record artifact identity, runtime revision, prompt/template, preprocessing, app build, device and OS for the run, and whether input is typed, spoken, or extracted from an image. Start with one modality; do not combine unlike pipelines in the same result.
4. Score per entity and per direction. Execute source-to-target A and target-to-source B independently. Compare each annotated span using its own rule, retain the original source next to output, and classify changed, omitted, added, ambiguous, and correctly preserved values.
5. Turn failures into regression and disclosure. Keep approved synthetic failures, change one variable at a time, and replay all cases. Report the fixture denominator, exact configuration, and unresolved classes. For high-consequence data, ask the user to verify rather than silently asserting a corrected translation.

Keep each transition observable. A failure should identify the stage, artifact, runtime, and recovery action without logging private user content.

Acceptance criteria

• Each fixture has a written entity-level rule and reviewer-approved expected interpretation before model output is inspected.
• Every language direction and modality is reported separately, with artifact, runtime, prompt, preprocessing, app build, device, and OS recorded.
• Changed, missing, extra, ambiguous, and preserved values are separate result classes; one fluent example is not the whole quality score.
• No claim of universal offline translation accuracy, supported language pairs, or high-consequence correctness is made from this method alone.

Decision table

Failure patterns to prevent

• Scoring fluency alone
• Normalizing identifiers without user review

Also prevent silent fallback, unpinned artifacts, missing cancellation, and conclusions that combine unlike configurations. Store unsuccessful runs alongside successful ones.

Minimum reproducibility record

Release checklist

• ☐ The primary query is answered without an unsupported number.
• ☐ Every artifact and runtime is pinned.
• ☐ The representative task and failure policy are explicit.
• ☐ Lifecycle, cancellation, cleanup, and fallback are tested.
• ☐ User-content and network boundaries are documented.
• ☐ Result wording applies only to the recorded configuration.
• ☐ The page links to raw method or evidence when results are added.

Sources and related evidence

• Cove benchmark method
• On-device AI architecture overview
• Google Gemma model overview and task variants
• Google Gemma audio guidance with explicit transcription-format examples

Chinese deployment and troubleshooting content is organized in the .奇连 AI 端侧专题.

Frequently asked questions