Retry Policy for Large Model Downloads

Direct answer

A safe retry policy begins with classification, not a counter. Retry only when the failure is plausibly transient and the next attempt can use the same pinned artifact identity. Stop automatic retries for integrity mismatch, malformed content, incompatible revision, user cancellation, or a configuration failure until the cause changes. Respect scheduler constraints and cancellation, preserve only partial bytes whose identity and response history are known, and expose a manual recovery action when the system cannot decide. A repeated HTTP request is not automatically a safe retry: if the body belongs to another revision, appending it can produce a corrupt file even though the request succeeded. The repository's current ModelDownloader tries a configured primary URL and, after timeout or DNS resolution failure, may try a configured mirror. Other exceptions produce an error rather than silently falling through to another source. It requests a Range beginning at the existing temporary-file length, accepts HTTP 200 or 206, and optionally hashes the completed temporary file only when a configured SHA-256 exists. These are current implementation facts, not a general recommendation that every retry path is correct. The current E2B digest is empty; a reader must not conclude a digest mismatch can be detected in that configuration. Any retry policy must specify what the application knows about artifact identity and byte provenance before it resumes. Classify outcomes using observable evidence. A temporary DNS or socket timeout may justify a later attempt, but distinguish it from an HTTP response such as an authorization failure, not-found artifact, unexpected content type, or server error. A user-selected cancel is terminal for that attempt and should not be retried in the background against the user's choice. A scheduler stopping work because constraints changed is different from a transfer error: persistent-work APIs allow work to be constrained and cancelled, while the worker must cooperate with stop requests. Disk-space failure is not repaired by waiting for a network retry. A checksum mismatch should quarantine or delete only the candidate partial artifact and require a trusted replacement; retrying identical bytes indefinitely does not fix the cause. Backoff should avoid synchronized retry storms and conserve battery and network, but it must not hide state. Record attempt number, failure category, chosen delay class, artifact revision, partial-file state, and user control. Do not invent a fixed “best” delay without measuring the system and product policy. Let the scheduler enforce constraints such as network conditions when appropriate; also re-check the state at execution time because conditions can change after enqueue. Respect platform cancellation promptly and close streams and connections. A visible state might say waiting for an unmetered connection, retry scheduled, action required, cancelled, or terminal failure. Do not make a notification the sole source of truth. Resuming safely requires more than retaining a temporary partial file. Persist a transfer record that binds the partial path to model id, immutable revision or URL identity, expected length source, expected digest if available, last response status/range, and bytes actually written. When the next response arrives, verify that the status and Content-Range match the requested offset and exact representation. If a server ignores Range and returns a full body, restart into a fresh temporary file rather than append. If the representation changed or metadata is missing, discard or quarantine partial bytes and restart only after policy permits. Never promote a file merely because a previous attempt wrote the expected number of bytes. The retry test matrix should use deterministic fakes or a controlled test server for timeout, DNS failure, temporary server error, not found, authorization failure, invalid range, ignored range, body truncation, changed revision, storage exhaustion, checksum mismatch, user cancellation, and scheduler stop. Assert both attempt behavior and data safety: no duplicate transfers, no cross-revision append, no deleted user data, no retry after cancellation, and no “ready” state for an incomplete candidate. Add process recreation between scheduling and execution to verify that state is durable enough for the product's claim. A successful transient retry should still pass verification and runtime-load acceptance before the app reports the feature ready. ### Procedure 1. Define a failure taxonomy and action table. For each observable failure, label it transient, constraint wait, user action, integrity/security terminal, artifact missing, or configuration error. Pair it with retry eligibility, user-visible status, and the next safe action. 2. Bind retries to artifact provenance. Persist model id, pinned revision, URL/source identity, expected metadata, partial path, and last range state. Refuse to reuse bytes when a revision or representation cannot be established. 3. Implement scheduler-aware retry and cancellation. Use persistent work only when the job needs to survive process loss; express constraints explicitly, observe stop signals, close active resources, and do not schedule another attempt after user cancellation. 4. Test deterministic failures and recovery. Simulate response classes, partial files, network changes, process death, and storage pressure. Verify bytes and state across every attempt, not only final success. 5. Audit retries for privacy and resource cost. Ensure logs contain failure classes but not private prompts or credentials, and ensure network and battery behavior follows an explicit product rule rather than an infinite retry loop. ### Acceptance criteria - Every failure category has a documented retry eligibility decision and a user-visible next step. - Retry uses the same verified artifact identity; unknown or changed representation cannot reuse prior partial bytes. - Cancellation and platform stop signals prevent a later automatic retry and release active resources. - Integrity, authorization, missing-artifact, and permanent configuration failures do not loop without a changed input or explicit user action. ### Comparison | Outcome | Correct policy | Unsafe shortcut | |---|---|---| | Temporary connection failure | Back off and retry the pinned representation if still eligible | Retry immediately in a tight loop | | Constraint changed or work stopped | Persist safe state and wait for scheduler policy | Treat every stop as corruption or as success | | Digest or representation mismatch | Reject candidate and obtain a trusted replacement | Retry identical bytes forever or mark ready | ### FAQ **Should every failed request retry automatically?** No. Classify the result first. A timeout may be transient, while a missing artifact, authorization error, digest mismatch, or user cancellation requires a different action and often must stop automatic work. **Is WorkManager retry the same as resuming the file?** No. Scheduling a later Worker and appending a valid partial representation are separate problems. The transfer layer still has to verify artifact identity, range response, and persisted byte offset. **What should happen after checksum mismatch?** Do not promote the candidate. Quarantine or remove only that model candidate, preserve unrelated user data, report a repairable integrity failure, and fetch a trusted replacement only when policy allows. **Can retry continue after the user presses cancel?** Not for that cancelled operation. Persist cancellation as terminal for its work identity; a new attempt should require a deliberate user action or a clearly documented product event.

A background model transfer fails while the network policy changes

A user starts a model transfer on a constrained connection and then moves between networks. The current attempt may stop, the temporary file may contain bytes, and a Worker can be stopped because a constraint is no longer met. The app must distinguish waiting from a broken artifact.

Decision

Classify the event, persist artifact and partial-response identity, and let only retry-eligible failures schedule another attempt. A stopped Worker is not itself proof of corruption or successful completion.

Expected outcome

The next attempt either resumes a provably matching representation, safely restarts, waits for constraints, or requests user action; it never silently mixes bytes or loops forever.

Evidence to collect

The claim becomes reviewable only when the following evidence is attached to the same artifact and test run:

• Failure classes
• Backoff policy
• Terminal user action

A completed byte stream is not enough. Final placement, integrity, runtime load, and recovery must also pass.

Implementation workflow

1. Create explicit retry categories. Map timeout, DNS, server status, auth, missing artifact, malformed content, range mismatch, storage pressure, checksum mismatch, cancellation, and scheduler stop to separate categories. Define retry eligibility, stop condition, and user guidance for each category.
2. Persist transfer identity before scheduling. Store model id, immutable revision, source identity, expected metadata, partial path, byte offset, and prior response-range state. Treat absent or inconsistent identity as unsafe partial data; do not attach it to a new revision because the filename happens to match.
3. Respect constraints and cancellation. If the job must survive process death, schedule persistent work with the necessary constraints and observe its stop signal. Close streams and connections, preserve only verified transfer metadata, and ensure an explicit user cancel does not create a new automatic attempt.
4. Validate a resumed response before writing. Check response status, requested range, returned range, representation identity, and body type before opening the temporary file in append mode. If the server returns a full representation or a different revision, restart safely or reject it according to policy.
5. Verify recovery with fault injection. Use a test server and controlled storage fixtures to cover transient and permanent failures, process recreation, constraint changes, partial writes, and integrity mismatch. Assert the data on disk and user-visible state after each attempt, not only eventual success.

Keep each transition observable. A failure should identify the stage, artifact, runtime, and recovery action without logging private user content.

Acceptance criteria

• Each failure type maps to a documented retry, wait, terminal, or user-action policy.
• No partial bytes are reused unless model revision and response-range provenance match.
• Cancellation and Worker stop close active resources and cannot trigger an automatic retry after the chosen user action.
• Only a verified and load-accepted candidate can become the selected ready model.

Decision table

Failure patterns to prevent

• Retrying corrupt bytes forever
• Hiding storage errors as network failures

Also prevent silent fallback, unpinned artifacts, missing cancellation, and conclusions that combine unlike configurations. Store unsuccessful runs alongside successful ones.

Minimum reproducibility record

Release checklist

• ☐ The primary query is answered without an unsupported number.
• ☐ Every artifact and runtime is pinned.
• ☐ The representative task and failure policy are explicit.
• ☐ Lifecycle, cancellation, cleanup, and fallback are tested.
• ☐ User-content and network boundaries are documented.
• ☐ Result wording applies only to the recorded configuration.
• ☐ The page links to raw method or evidence when results are added.

Sources and related evidence

• Android WorkManager: manage and cancel work
• Android WorkManager persistent work
• IETF HTTP Semantics: range requests
• Kotlin cancellation and timeouts

Chinese deployment and troubleshooting content is organized in the .奇连 AI 端侧专题.

Frequently asked questions