Validate a Converted On-Device Model

Direct answer

A converted model is a new release artifact, not a renamed copy of its source. Freeze the source model revision, license reference, tokenizer and prompt assets, converter version, command options, target format, and output manifest. After conversion, give the result its own immutable identity and digest when a trusted digest can be generated. Inspect the output structure with the format’s official tooling, load it using the intended runtime, and run fixed acceptance fixtures. If output behavior differs, treat that as an evaluation result, not proof that the converter itself is defective; source and target artifacts may have different tokenization, metadata, operators, or preprocessing assumptions. The repository’s facts do not certify any converted Cove artifact. COVE-MODEL-02 says the current E2B digest is empty, while the conversion seed explicitly requires output identity and a reference-task comparison. BENCHMARK-01 permits methodology and schemas but requires raw device records before results are claimed. Therefore a page can describe validation procedure but must not publish accuracy, latency, compatibility, or quality claims without actual records. Keep “the converter emitted a file,” “the format parser accepted it,” “the runtime loaded it,” and “the target task passed” as separate statements. A reproducible conversion record needs more than one command line. Preserve exact input and output revisions, converter repository commit and dependencies, configuration flags, supported-op policy, calibration or representative data provenance if quantization is involved, and deterministic seed settings where applicable. Record warnings, skipped operators, external companion files, and output sizes as observations rather than universal guarantees. Do not include private prompts or user examples in the manifest. When conversion uses sample data, use fixtures with appropriate rights and documented expected properties. Validation should cover structure before execution. Confirm required files exist, checksums are recorded, package metadata matches the selected runtime family, tokenizer and model configuration refer to compatible revisions, and any external weights are attached to the correct package. Then load the candidate in a clean test environment. A parser accepting metadata does not prove the runtime can execute every operation. A successful engine initialization does not prove representative tasks work. Task validation should compare stable properties such as schema validity, required fields, or controlled reference outputs, while acknowledging stochastic outputs and not inventing a numerical benchmark. Keep source and converted artifacts side by side until release acceptance. Do not replace the source record with the converted artifact or lose license provenance during export. If a later conversion is attempted, assign a new revision even when the filename is unchanged. Compare changes one factor at a time: converter commit, flags, tokenizer, calibration corpus, or runtime. If multiple inputs change together, a regression cannot be attributed. Preserve a known-good candidate for rollback and invalidate only derived caches that depend on the old artifact representation. ### Procedure 1. Freeze the source package identity, license record, tokenizer, prompt assets, converter commit, environment, and exact flags. Save a machine-readable conversion manifest and keep user data out of it. 2. Produce output into a clean, revision-specific location. Record file inventory, lengths, hashes when available, metadata, warnings, external companion files, and conversion exit state. 3. Inspect format structure using official tooling for that format. Check required sections, model configuration, tokenizer association, weight references, and supported runtime family; reject ambiguous or mixed packages. 4. Load the exact output with the target runtime in a clean test app. Capture the load result and failure stage separately from conversion success. 5. Run fixed task fixtures, compare documented acceptance properties, and retain the source artifact until all release gates pass. Assign a new identity to every changed output and test rollback. ### Acceptance criteria - The output has a distinct identity linked to source revision, converter version, flags, and companion assets. - Structural inspection and target-runtime load are both recorded and cannot be inferred from successful conversion alone. - Task acceptance uses versioned fixtures and declared properties; no unsupported benchmark or device claim is added. - License/provenance records and the previous known-good artifact remain recoverable through release and rollback. ### Comparison | Evidence stage | What it answers | What it does not answer | |---|---|---| | Converter exit and output file | Did a tool produce an output path? | Is the package structurally complete or runtime-compatible? | | Format inspection and manifest | Are expected sections and identities present? | Does inference execute the intended task correctly? | | Runtime plus task fixture | Does this exact candidate load and meet defined checks? | Does it work on untested devices or have a universal quality level? | ### FAQ **If conversion exits successfully, is the model ready to ship?** No. A successful process only says the converter returned according to its own exit behavior. Inspect package structure, load the exact output in the target runtime, run task fixtures, and retain source provenance. **Can a converted file reuse the source model’s checksum or identity?** No. The output bytes are different and require a new identity and digest record. Preserve a link to the source revision and conversion manifest rather than reusing its checksum. **Does a runtime load prove conversion preserved quality?** No. Loading checks one boundary. Use versioned task fixtures and documented acceptance properties to compare the output; avoid claiming a numerical quality result unless raw evidence exists. **What if a conversion needs calibration data?** Record the exact dataset or fixture provenance, permitted use, preprocessing, and converter configuration. Do not use private user content or silently substitute another dataset between candidate revisions.

A converted artifact loads but differs on a fixed task

A developer converts a source checkpoint into a mobile runtime package. The converter exits successfully and the runtime opens the file, but a fixed task fixture produces a different structure than expected. The team must distinguish a valid package from an accepted product candidate.

Decision

Preserve source and converter provenance, inspect the output format, then validate runtime and task behavior as separate gates before activation.

Expected outcome

The converted artifact is accepted, rejected, or held for investigation with an exact record; no source-model property is silently copied onto the converted bytes.

Evidence to collect

The claim becomes reviewable only when the following evidence is attached to the same artifact and test run:

• Converter revision and flags
• Output artifact digest
• Reference task comparison

A runtime comparison is publishable only when artifact identity, build configuration, selected backend, task fixture, and failures are recorded.

Implementation workflow

1. Freeze source and conversion inputs. Record source revision, license reference, tokenizer, prompt template, converter revision, environment, options, and calibration fixture provenance. Preserve each input separately so future comparisons can identify the changed variable.
2. Create a new artifact identity. Write the output to a clean revision-specific path and create a manifest that points back to the source and converter. Compute a digest when the release process can obtain a trusted expected value; do not reuse the source hash.
3. Inspect package structure. Use official format tools to check required sections, metadata, tokenizer/config relationship, and external weights. Record warnings and unsupported operations rather than suppressing them.
4. Load the exact candidate in the target runtime. Use a clean environment and capture initialization separately from conversion. A successful parser or file open does not imply all operations can execute.
5. Run acceptance fixtures and preserve rollback. Test stable task properties with versioned inputs, keep a known-good artifact until approval, and assign a new identity whenever source, converter flags, or output bytes change.

Keep each transition observable. A failure should identify the stage, artifact, runtime, and recovery action without logging private user content.

Acceptance criteria

• The converted output has a new identity and traceable source, converter, flags, tokenizer, and companion files.
• Format inspection and target-runtime loading are separately successful for the exact candidate.
• Task checks use versioned fixtures and do not claim unsupported quality or device performance.
• The source provenance and prior accepted artifact remain available for audit and rollback.

Decision table

Failure patterns to prevent

• Assuming conversion preserves behavior
• Publishing source-model facts for converted bytes

Also prevent silent fallback, unpinned artifacts, missing cancellation, and conclusions that combine unlike configurations. Store unsuccessful runs alongside successful ones.

Minimum reproducibility record

Release checklist

• ☐ The primary query is answered without an unsupported number.
• ☐ Every artifact and runtime is pinned.
• ☐ The representative task and failure policy are explicit.
• ☐ Lifecycle, cancellation, cleanup, and fallback are tested.
• ☐ User-content and network boundaries are documented.
• ☐ Result wording applies only to the recorded configuration.
• ☐ The page links to raw method or evidence when results are added.

Sources and related evidence

• Cove benchmark method
• Google AI Edge LiteRT-LM model builder and package inspection
• LiteRT-LM package header schema
• Google LiteRT converter API reference

Chinese deployment and troubleshooting content is organized in the .奇连 AI 端侧专题.

Frequently asked questions