Feature Flags for Local AI Runtime Changes

Direct answer

A feature flag for an on-device model is a configuration input that changes an execution path; it is not proof that the new path works. If a flag selects an artifact, backend, parser, prompt policy, or concurrency setting, every test result must record the flag's effective value and the configuration version that produced it. Otherwise, a dashboard may compare results from different runtimes as if they were identical. A flag can reduce rollout risk, but it also adds branches and combinations that need explicit validation. Keep the configuration source in scope. A locally bundled flag is different from a remotely fetched value: a remote source introduces network availability, privacy, cache, and consent questions. This guide focuses on deterministic app configuration recorded with the build or test, not on a remote feature-flag service. Avoid adding a cloud dependency merely to toggle local inference. For every flag, document its owner, allowed values, default, compatibility constraints, and whether changing it requires recreating the runtime. Never let a flag silently send private input to a remote backend or switch the meaning of an “offline” claim. Use one variable at a time for diagnosis. If a release changes model artifact, backend, and parser simultaneously, a pass or regression cannot be attributed. Define a configuration receipt containing app build, flag-set revision, exact artifact ID, runtime/backend choice, parser revision, and task policy. The test harness should read the effective receipt rather than the developer's intended value. If a requested backend falls back, record requested and observed behavior separately. A flag name such as useGpu is not evidence that a GPU executed the task. Design rollback as a state transition. Disabling a feature should stop new operations from entering the path, while in-flight work follows an explicit cancel-or-finish policy. If the previous configuration uses a different model session or parser state, do not reuse incompatible runtime objects. The coordinator should drain or cancel work safely, close resources it owns, select the prior configuration, and initialize the corresponding state before accepting another request. Old callbacks must be tied to the old configuration and ignored when no longer active. Test the flag matrix intentionally. Cover default, enabled, disabled, unknown value, configuration mismatch, and rollback. Not every Cartesian combination of flags is worth testing; select combinations based on dependencies and risk, and write down the omitted combinations. Verify that startup, model loading, task result, cancellation, and next request behave under each supported configuration. Use a fake runtime to assert branch selection and a real pinned artifact/runtime test for claims about actual execution. A passing unit test cannot prove backend compatibility. Keep measurement and product copy connected to the receipt. A release note that describes one backend or artifact must not be attached to a run that used another. If the app has no measured performance evidence, flags cannot create it. Do not infer support from a flag being present, and do not infer compatibility from a successful build. The repository's benchmark and model facts require conclusions to be tied to exact artifact and test evidence. Finally, minimize flag lifetime. A temporary experiment should have an owner and removal date. After rollback or successful adoption, delete obsolete branches and their stale configuration paths instead of leaving a permanent matrix nobody understands. Keep a short audit trail showing why the flag existed, what was tested, and which version became default. This reduces the chance that a future task silently enters an unmaintained inference route.

A parser change is gated while the current runtime stays available

A team wants to test a new structured-output parser for a local model. Some requests may produce malformed data, and the old parser must remain available for controlled rollback. The team needs each result tied to the parser variant that actually ran.

Decision

Version the parser selection with the app-local configuration receipt, route each task through one recorded variant, and test rollback while preserving in-flight state policy.

Expected outcome

Results identify which parser and runtime were used, while switching does not silently rerun or mislabel a task.

Evidence to collect

The claim becomes reviewable only when the following evidence is attached to the same artifact and test run:

• Flag inventory
• Configuration receipt
• Rollback test

A release claim is valid only for the recorded configuration and acceptance set. Generalization requires additional records.

Implementation workflow

1. Define the flag contract and owner. Write its default, allowed values, owner, expiry, compatibility requirements, and whether a change requires rebuilding or closing runtime state. Identify privacy-sensitive effects explicitly.
2. Produce an effective configuration receipt. Bind the selected value to app build, artifact, runtime/backend, parser, and task policy. Record effective configuration in each test; do not trust an intended value from a developer note.
3. Change one dimension for diagnosis. Keep artifact, backend, parser, and workload stable except for the one flag under investigation. Record requested versus observed backend and keep performance claims out unless measured.
4. Exercise rollback and in-flight work. Disable the path, apply the declared cancel-or-finish rule, drain or close incompatible state, and verify old callbacks cannot change the newly selected configuration.
5. Retire temporary branches. Assign an expiration date and remove obsolete flag paths after adoption or rollback. Preserve the tested decision and new default in release evidence.

Keep each transition observable. A failure should identify the stage, artifact, runtime, and recovery action without logging private user content.

Acceptance criteria

• Each test record identifies the effective flag revision, artifact, runtime/backend, parser, and task policy.
• A runtime-affecting flag change has a defined owner, default, accepted values, and resource-recreation policy.
• Rollback does not accept incompatible in-flight state or apply stale callbacks to the new variant.
• No flag is treated as evidence of actual backend execution, device compatibility, privacy, or performance.

Decision table

Failure patterns to prevent

• Changing backend without telemetry label
• Leaving incompatible state after rollback

Also prevent silent fallback, unpinned artifacts, missing cancellation, and conclusions that combine unlike configurations. Store unsuccessful runs alongside successful ones.

Minimum reproducibility record

Release checklist

• ☐ The primary query is answered without an unsupported number.
• ☐ Every artifact and runtime is pinned.
• ☐ The representative task and failure policy are explicit.
• ☐ Lifecycle, cancellation, cleanup, and fallback are tested.
• ☐ User-content and network boundaries are documented.
• ☐ Result wording applies only to the recorded configuration.
• ☐ The page links to raw method or evidence when results are added.

Sources and related evidence

• Cove benchmark method
• Android build variants and configuration
• Android Macrobenchmark and reproducible target setup

Chinese deployment and troubleshooting content is organized in the .奇连 AI 端侧专题.

Frequently asked questions