[justinwlin]

Note

Companion PR: runpod/flash#394. This PR documents runpod-flash 1.21.0 as released, including the 90-second first-run failure and the broken flash deploy curl example. #394 fixes both in the SDK. The pages were tested in the same live runs. Review them together, and revisit the 90-second guidance once #394 ships.

Brings the Flash quickstart and the SDXL and text-generation tutorials in line with what actually happens on runpod-flash 1.21.0. Every page was run end to end against live endpoints, and the code, outputs, and timings come from those runs.

What changed

• Deploy step: the quickstart and both tutorials now run flash deploy before the script. Since 1.15.0, calling an @Endpoint function looks up a deployed endpoint, so without it the first run fails with endpoint ... not found. Each page shows the real deploy output.
• Model caching: both tutorials load the model once per worker (module-level cache plus global) and report model_cached. Warm SDXL calls went from about 25s to 7s end to end. Both have a "Complete file" accordion that matches the steps.
• Build size: accelerate is gone from both tutorials. It pulls torch and several GB of CUDA wheels into the local build on Linux, which is the "Disk quota exceeded" failure. SDXL builds to 50 MB and GPT-2 to 38 MB.
• First run past 90 seconds: documents what 1.21.0 does when a cold worker takes longer than 90s: timed out after 90s ... may not be deployed, or KeyError on the output. The fix is to run the script again. FLASH_SENTINEL_TIMEOUT is no longer recommended because above 90 it turns the timeout into the KeyError. Also covers the /health check (throttled means waiting for a GPU).
• Call from any language: new quickstart Step 9 with /run and /status curl calls (including exporting RUNPOD_API_KEY, which flash login doesn't set), and a warning that {"input": {}} (the example flash deploy prints in 1.21.0) fails after about 40s.
• Quickstart code: the matrix multiply now runs on the GPU with torch. Before, it used numpy on the worker's CPU and only read the GPU name.
• Corrections: measured timings replace guessed ones (SDXL 30 steps is about 4-5s on an RTX 4090, not 20-40s). Also fixed:
  • max_new_tokens instead of max_length
  • the flash init layout (gpu_worker.py, not workers/gpu/endpoint.py)
  • stale model IDs (stabilityai/stable-diffusion-2-1 returns 401)
  • SDXL Turbo needs a redeploy
  • the network volume example now sets HF_HOME, so models are actually cached
  • the real auth error text
  • the endpoint defined in multiple files error
  • redeploying during a request
  • warm workers can serve the previous version for a few minutes after a redeploy
  • shared troubleshooting no longer says scripts provision endpoints on their own
• Python versions: local 3.10 to 3.14 works. The worker version comes from --python-version, then python_version on a resource config such as LiveServerless, then 3.12. A conflict fails the build. @Endpoint has no python_version parameter.

Verification

• Quickstart: deployed the exact page code with 1.21.0 and ran Steps 5, 6, 8, and 9 plus the cleanup commands. Cold first run took 45-49s, and once failed at 90s, which is why the page warns about it. Warm runs took 1-3s. The build is 0.2 MB.
• SDXL: deployed the exact "Complete file". Cold first run took 74-82s (load 18s, generate 4.9s), and warm runs took 7s (generate 3.9s). Three prompts in parallel finished in 18s on one warm worker.
• Text generation: deployed the exact "Complete file". The first two runs failed at 90s while the worker was throttled waiting for a GPU. The third took 51s, and warm runs took 1.4s. Step 8 returned three completions in 3.3s.
• Errors: reproduced KeyError with FLASH_SENTINEL_TIMEOUT=300, the 401 without credentials, {"input": {}} failing after 40s, and defined in multiple files. All test endpoints and apps were deleted afterwards.
• Checks: mintlify broken-links found none. The only parse error is in README.md, which was already there.
• Review: five Codex passes against the 1.21.0 source, plus a fresh pass on the full PR that found 4 more issues (all fixed). The last pass found no defects.

Related Flash PRs

runpod-flash stops getting fixes on 2026-10-29, so the runpod/flash changes need to merge and release before then.

PR	What it does	Status (2026-10-27)	Requires before merge
runpod-workers/flash#113	Installs torchvision in the GPU load-balancer image	Merged, not released. Ships with release PR runpod-workers/flash#112 (1.6.1).	Done.
runpod/flash#390	Keeps torch's dependency tree out of flash build archives	Approved	runpod-workers/flash#112 merged so the 1.6.1 worker image is released.
runpod/flash#394	Waits for cold-start jobs instead of failing at 90s, and prints a working curl example after flash deploy	Approved, review round 1 addressed	Nothing.
#923	Matches the Flash quickstart and tutorials to a live run on 1.21.0	Approved, review round 1 addressed	Nothing. Documents 1.21.0 as released. Revisit the 90-second guidance once runpod/flash#394 ships.
runpod/flash#369	Reads runpodctl's top-level apikey as a credentials fallback	Draft	Nothing.

[mintlify]

Preview deployment for your docs. Learn more about Mintlify Previews.

Project	Status	Preview	Updated
runpod-docs	🟢 Ready	View Preview	Oct 27, 2026, 6:17 PM

💡 Tip: Enable Automations to automatically generate PRs for you.

[OneAsterisk]

Claude review, round 1

Verdict: Approved. The docs match the runpod-flash 1.21.0 source on every claim checked; the remaining items are small accuracy fixes.
Approval gate: 0 Critical, 0 High, 6 Medium+Low (1 Medium, 5 Low) → approved.

Existing feedback

• @lavanya-gunreddi: one review at 710c60f (dismissed, no body, no inline comments). Nothing to verify.
• mintlify[bot]: preview deploy only. CI: Check tooltip imports and Mintlify Deployment both pass.

PR description vs code

Checked against runpod/flash tag v1.21.0 (c0f7843).

Claim	Status	Where
Calling an @Endpoint function needs a prior flash deploy (since 1.15.0); "not found" error text	✅	flash_sentinel.py:105-109; client.py:236-251
90s timeout message; FLASH_SENTINEL_TIMEOUT > 90 turns into KeyError	✅	flash_sentinel.py:32,97-102,130 (data.get("output", data) returns the IN_PROGRESS dict)
Python resolution: flag → resource python_version (e.g. LiveServerless) → 3.12; conflict fails; no python_version on Endpoint	✅	cli/commands/build_utils/manifest.py:424-479; endpoint.py:392-419
Local Python 3.10 to 3.14	✅	pyproject.toml requires-python = ">=3.10,<3.15"
flash init creates gpu_worker.py, cpu_worker.py, lb_worker.py	✅	cli/utils/skeleton_template/
flash deploy prints a curl example with {"input": {}}	✅	cli/commands/deploy.py:119-127 (but the shown deploy output omits it, see Low)
endpoint defined in multiple files when a second file imports the function	✅	scanner.py:552-613, manifest.py:496-504
flash login doesn't set RUNPOD_API_KEY	✅	saves to ~/.runpod/config.toml (core/credentials.py)
Auto-excluded set is torch, torchvision, torchaudio, triton (no numpy)	✅	build.py:168-175
Live timings, build sizes, 40s empty-input failure, stale warm workers	⚠️ not verifiable from source

Findings

Medium

• "For load-balanced endpoints" label points at an error deployed scripts never hit: flash/troubleshooting.mdx:376. Inline.

Low

• Example output shows values round(x, 1) can't produce (18s, 0s): SDXL :372, :382; text-gen :318. Inline.
• Unsupported-Python error shown with a ValueError: prefix the CLI doesn't print: flash/troubleshooting.mdx:235. Inline.
• --no-deps offered as a disk-space fix without a caveat: flash/quickstart.mdx:150. Inline.
• Documented flash deploy output omits the curl block that Step 9's warning refers to: flash/quickstart.mdx:141 (also SDXL :357, text-gen :294). Inline.
• The 90-second guidance is copied verbatim into three pages: quickstart :443-467, SDXL :587-611, text-gen :402-426, plus three Step warnings. The PR body says to revisit it once runpod/flash#394 ships, so that's six places to edit. A shared snippet under snippets/ with the error key as a prop would make it one edit. Fine as a follow-up.

Nits

• flash/quickstart.mdx:467 (and SDXL :611, text-gen :426): idle / running are counts nested under workers and always present. "A non-zero workers.idle, workers.ready, or workers.running" is accurate.
• flash/quickstart.mdx:352 links API keys to /user/settings; :189 on the same page uses /user/credentials. Pick one.
• flash/troubleshooting.mdx:389: "deploy and call with the same environment" doesn't say how. Mention --env on deploy and FLASH_ENV for the script.
• flash/quickstart.mdx:180-181: double blank line.

Simplification opportunities

1. Move the "First run fails after 90 seconds" section and the Step 7 warning into one snippet (about 60 duplicated lines become one snippet of about 25). Follow-up PR, and pairs with the #394 revisit.

Verified clean

• Every step snippet matches its "Complete file" accordion exactly (SDXL, text-gen, quickstart Step 3 vs Step 7).
• Module-level _pipeline / _model caching with global is correct and persists: the generated handler imports the module once (handler_generator.py).
• unexpected keyword argument after a redeploy matches func(**job_input) in the handler template.
• Deploy output lines (✓ built … N deps, No app '…' found…, ✓ uploaded, ✓ deployed to production) match build.py/deploy.py. Dep counts exclude torch, so "0 deps" and "2 deps" are consistent.
• Endpoint params used in samples exist: workers as int or tuple, idle_timeout, execution_timeout_ms, volume, env, list gpu.
• flash undeploy list|<name>|--all|--force exist.
• /runsync 90s default wait matches serverless/endpoints/operation-reference.mdx:41.
• No inbound links to the removed #python_version anchor in parameters.mdx.

Questions for the author

• Out of diff: text-generation-with-transformers.mdx:501 (unchanged) says generate_text and chat share a config, but chat has its own name="chat". Worth a one-line fix here, or leave it?
• FYI, not a docs bug: the SDK's own --python-version help says "3.10, 3.11, or 3.12" (cli/commands/build.py:568), and the python_version field description says it defaults to the local interpreter (serverless.py:268-271). The docs are right and the SDK text is stale. Is that worth a line in runpod/flash#394?

---

_{Reviewed by Claude (claude-opus-5-5) via the pr-review skill. Tokens: 170 in / 8,431 out / 7,542,389 cache read / 325,634 cache write = 7,876,624. Estimated cost: $3.31 at Opus 5.5 rates ($4 in, $20 out, $0.20 cache read, $5 cache write assuming 5-minute TTL). Round 1 of this PR. This is automated assistance; a human owns the final call.}

[OneAsterisk]

Claude review, round 2

Verdict: Approved. Every round 1 finding and nit is fixed at d78a8f8, and each fix matches the runpod-flash 1.21.0 source.
Approval gate: 0 Critical, 0 High, 2 Medium+Low (0 Medium, 2 Low) → approved.

Round 1's approval at 632611e was dismissed by the new push. This round re-approves at d78a8f8.

Status of prior findings

#	Finding	Severity	Status	Evidence
R1-1	"For load-balanced endpoints" error only appears under flash dev	Medium	Fixed	flash/troubleshooting.mdx:371-377. Block removed. Deployed LB calls raise the same not found error (flash_sentinel.py:290-294).
R1-2	round(x, 1) output shown as 18s / 0s	Low	Fixed	SDXL :377, :387; text-gen :323 now show 18.0s / 0.0s
R1-3	Unsupported-Python error had a ValueError: prefix	Low	Fixed	flash/troubleshooting.mdx:235 now shows ✗, matching print_error (cli/utils/formatting.py:8-10)
R1-4	--no-deps offered without a caveat	Low	Fixed	flash/quickstart.mdx:155 leads with --exclude <package> and limits --no-deps to full dependency lists
R1-5	Deploy output omitted the curl block	Low	Fixed	quickstart :142-145, SDXL :358-361, text-gen :295-298. Matches _print_curl_example (cli/commands/deploy.py:119-127) character for character.
R1-6	90-second guidance copied into three pages	Low	Not addressed (deferred)	Round 1 marked it fine as a follow-up. Still counts as one open Low.
R1 nits	/health field names, API key link, --env / FLASH_ENV, double blank line	Nit	Fixed	FLASH_ENV matches flash_context.py:52; --env matches deploy.py:27
R1 Q1	Text-gen note said generate_text and chat share a config	Question	Fixed	text-generation-with-transformers.mdx:506 now names both configs

Existing feedback

• @lavanya-gunreddi: one review at 710c60f (dismissed, no body). Nothing to verify.
• mintlify[bot]: preview deploy only. Check tooltip imports passes. Mintlify Deployment was still pending at review time.

PR description vs code

Round 1's claim table still holds. The delta adds no new claims. The body's "Related Flash PRs" table lists this PR as Approved. That was true at 632611e, and it is true again after this round.

Findings

Low

• Tutorials now show the {"input": {}} curl block with no warning: tutorials/flash/image-generation-with-sdxl.mdx:361 and tutorials/flash/text-generation-with-transformers.mdx:298. New from the R1-5 fix. Inline.
• R1-6 carried forward: shared snippet for the 90-second guidance. Fine as a follow-up.

Verified clean

• The three added curl blocks match 1.21.0 output exactly, including the blank line before them (deploy.py:192-194).
• ValueError from --python-version is caught and printed through print_error (cli/commands/build.py:410-413).
• FLASH_ENV=staging python your_script.py reaches sentinel lookup through get_flash_context() (endpoint.py:923-925, client.py:238).
• The /health wording now uses the nested workers.* counts.
• No stray changes outside the four files the commit names.

---

_{Reviewed by Claude (claude-opus-5-5) via the pr-review skill. Tokens: 22 in / 4,738 out / 594,435 cache read / 54,887 cache write = 654,082. Estimated cost: $0.49 at the same Opus 5.5 rates as round 1 ($4 in, $20 out, $0.20 cache read, $5 cache write assuming 5-minute TTL). Round 2 of this PR. This is automated assistance; a human owns the final call.}

[OneAsterisk]

Low: The quickstart follows this block with "Don't run the printed curl example as is in 1.21.0" and explains why in Step 9. The tutorials show the same {"input": {}} block with no warning. A reader who pastes it waits about 40s for job timed out after 1 retries. Add one line after the block, for example: "Don't run the printed curl example as is: its empty input fails in 1.21.0. See Call the endpoint from any language." Same at text-generation-with-transformers.mdx:298.

[OneAsterisk]

Low: Same as the SDXL tutorial: this empty-input curl block needs the one-line warning the quickstart has at flash/quickstart.mdx:148.