Code-Switching Tests for Offline Translation

Direct answer

A code-switching test should annotate language spans inside a sentence instead of assigning one language label to the entire input. Real short messages may combine a sentence in one language, a place name in another script, a borrowed term, digits, and punctuation. A useful evaluation record keeps the original text intact, labels spans and boundaries, identifies the requested source and target direction, and states how the app should behave when a span is uncertain or unsupported. Evaluate every direction separately and distinguish language identification, translation, formatting, and user-interface rendering. This is a test plan; it is not evidence that a particular model supports a given language pair or produces a specified accuracy. The project’s benchmark fact allows methodology and schemas but requires raw records before result claims. Its on-device definition says where inference occurs, not whether mixed-language translation is correct. Google’s Gemma documentation describes task variants and language capabilities at a model-family level, but a family page does not validate a particular converted artifact, runtime, device, prompt, or code-switch fixture. The product path must be tested in its actual configuration, and unsupported spans must not be silently guessed into a fluent sentence. Build fixtures that exercise different boundaries: a grammatical switch between clauses; an embedded person or place name; script changes in the middle of a word or phrase; quoted text that should remain unchanged; an unknown abbreviation; a target language with a different writing system; and punctuation or digits shared across spans. Include clean single-language controls so a failure is not misclassified as code switching when the base task itself is not working. Keep each fixture synthetic or explicitly reviewed for public use, and version the expected span annotations. Before executing, decide what “correct” means for each span. Some spans may be translated, some preserved, some transliterated under a written rule, and some rejected with an uncertainty prompt. The test harness should preserve span offsets through normalization and rendering, because removing diacritics or changing Unicode normalization can shift annotation positions. Do not require a single universal policy for proper nouns. A human reviewer should approve fixture labels and ambiguous expected outputs; a fluent response alone cannot establish that the named entity or script behavior is right. Record artifact identity, runtime, app build, prompt/template revision, preprocessing, input direction, and whether the task completed with network access blocked. Keep separate outcomes for span detection, translated meaning, preserved names, script rendering, and uncertainty handling. If the runtime does not expose internal language spans, evaluate the final output against the fixed annotations and note that internal routing is unobservable. Report all runs and failures. Do not convert a small hand-authored corpus into a broad language-support claim, and do not publish result percentages without raw, reproducible device records.

A message mixes English instructions with a local-language address

A traveler types a short request containing an English sentence, a street name written in a local script, a borrowed product name, and numerals. The translator returns a smooth sentence but changes the address, and the team wants a regression fixture that identifies which span failed.

Decision

Annotate language and script spans, define per-span preserve/translate/ask behavior, and compare both language directions separately using a reviewed synthetic fixture.

Expected outcome

The regression identifies whether the failure was span detection, entity handling, translation, or rendering, without implying unmeasured broad language support.

Evidence to collect

The claim becomes reviewable only when the following evidence is attached to the same artifact and test run:

• Code-switch fixture corpus
• Language-span annotations
• Per-span translation, rejection, and recovery results

A release claim is valid only for the recorded configuration and acceptance set. Generalization requires additional records.

Implementation workflow

1. Define spans and boundaries. Represent each test as original text plus ordered spans, language/script labels where known, entity type, and boundary offsets. Include punctuation and whitespace ownership. Mark genuinely ambiguous text as uncertain instead of forcing a label.
2. Write a policy per span type. For each label, specify whether the app translates, preserves, transliterates under an approved rule, or asks the user. Names, addresses, quoted strings, identifiers, and borrowed words may need different behavior; do not silently normalize them all.
3. Create controls and direction pairs. For every mixed-language fixture, add a clean single-language control and, when relevant, a reverse-direction counterpart. Freeze the artifact, app build, runtime, prompt, preprocessing, and exact input so changes can be attributed.
4. Score each stage separately. Compare span detection evidence when available, translation meaning, exact-preservation rules, Unicode/script rendering, and uncertainty behavior as separate fields. Keep raw input next to expected output in the reviewed test corpus, but never use private traveler data.
5. Turn ambiguous cases into safe outcomes. When a span cannot be reliably classified or the required preservation rule cannot be checked, make the UI preserve the source and request review rather than invent a confident translation. Repeat every fixture after template, artifact, or runtime changes.

Keep each transition observable. A failure should identify the stage, artifact, runtime, and recovery action without logging private user content.

Acceptance criteria

• Every fixture has reviewed span boundaries, direction, expected per-span behavior, and explicit treatment for uncertainty before execution.
• Single-language controls and mixed-language cases are reported separately; one direction does not stand in for its reverse.
• Unknown or unsupported spans remain visible and are not silently converted into a confident-looking output.
• Published findings name the exact tested app path and corpus scope and do not claim general language coverage or accuracy.

Decision table

Failure patterns to prevent

• Assigning one language to a mixed sentence
• Hiding uncertain span detection

Also prevent silent fallback, unpinned artifacts, missing cancellation, and conclusions that combine unlike configurations. Store unsuccessful runs alongside successful ones.

Minimum reproducibility record

Release checklist

• ☐ The primary query is answered without an unsupported number.
• ☐ Every artifact and runtime is pinned.
• ☐ The representative task and failure policy are explicit.
• ☐ Lifecycle, cancellation, cleanup, and fallback are tested.
• ☐ User-content and network boundaries are documented.
• ☐ Result wording applies only to the recorded configuration.
• ☐ The page links to raw method or evidence when results are added.

Sources and related evidence

• On-device AI architecture overview
• Cove benchmark method
• Google Gemma model overview and task variants
• Google guidance on tasks in spoken languages with Gemma

Chinese deployment and troubleshooting content is organized in the .奇连 AI 端侧专题.

Frequently asked questions