Lab systems: improvement report
1. Summary
The backend is careful work. The GPU lease, the durable submission journal, the fsync discipline on the Qwen path and the strict input checks are better than in most hobby projects, and also than in many paid ones. Most of the problems are on the other side: what the user sees while a job runs, and what they can do with a result after it exists.
The five changes with the most value for the effort:
- Keep the result on screen. On a 1440×900 screen, the output panel is not visible when you press Generate in Krea or Qwen. The image appears in the middle of a 3,400 px page. A sticky output column fixes this in a few lines of CSS (details).
- Queue instead of HTTP 409. If another tab holds the GPU, Qwen and H3 fail immediately with "lab coordinator is busy". There is no owner name, no wait and no retry. A small wait-and-retry in the frontend helps. A real queue in the gateway is better (details).
- Progress and finish signals. No app shows step progress or an ETA. Nothing tells you that a 5-minute video is done when you are in another tab. ComfyUI already sends step progress on its websocket, and the gateway already sees every phase change.
- Gallery actions. No app can delete an item, mark a favorite, filter, or send a result to another app (for example, a Qwen image as the H3 start frame). The Qwen gallery loads full-size PNGs and has no paging or retention.
- Fix the H3 frontend state bugs and move H3 to the shared studio UI. H3 forgets a running job on reload, stops polling on one network error, and has its own look and CSS. Moving it to
image-studioremoves these bugs and the inconsistency together.
On the backend, the most important bug is a timeout mismatch: the gateway waits 30 s for an H3/Qwen unload, but the backend can take up to 75 s. A slow unload then shows as a false Krea 502 (BUG-1). I verified this and three other findings myself in the code. The rest of the bug list comes from code review without a GPU, so treat "medium" confidence items as leads, not facts.
Safety note from this session. This Mac runs a real Lemonade server on
127.0.0.1:13305. Krea had that address hard-coded, so my first mock run pointed the gateway at the real server. I stopped it within seconds, before any generation or unload request. I only sent GET requests, and the model stayed loaded. The mock stack now uses its own ports, refuses to start if any port is in use, and Krea has an optional KREA_LEMONADE_URL override (default unchanged).2. Suggested roadmap
Phase 1: quick wins (1–2 days)
- Sticky output column, batch result strip (UX-1, UX-2)
- Ctrl/Cmd+Enter, block Enter-to-submit in number fields (UX-6)
- H3: resume job after reload, retry polling, fix Stop (BUG-6…8)
- Gateway unload timeout, idle "dirty" flag (BUG-1, BUG-2)
- Wait-and-retry on 409 with owner name (FEAT-1a)
- Tab title / favicon badge when a job finishes (FEAT-3)
scripts/test-all.shand a pre-push hook (OPS-1)
Phase 2: daily-use features (1–2 weeks)
- Step progress and ETA via ComfyUI websocket (FEAT-2)
- Thumbnails, gallery paging, delete, favorites (FEAT-4, FEAT-5, PERF-1)
- Send image to H3 start frame / Qwen edit (FEAT-6)
- Drag-and-drop and paste for images (UX-7)
- Persist form state per app (UX-5)
- Gateway queue (FEAT-1b)
- Deploy script with drain mode (OPS-2)
Phase 3: structure (ongoing)
- Move H3 UI to image-studio (ARCH-5)
- Share job poller / batch loop in image-studio (ARCH-6)
- Merge H3 and Qwen submit/reconcile pipelines (ARCH-2)
- Split
h3-lab/server.pyand Qwenapp.js(ARCH-1, ARCH-7) - Model/LoRA manifest as data (ARCH-4, FEAT-11)
- Move dated validation notes out of the runbooks (OPS-5)
3. How I tested
I added a mock stack under dev/mock/. It replaces only the two GPU services, so the real gateway, Krea server and H3/Qwen server run unchanged:
mock_backends.pyhas a fake ComfyUI (/prompt,/queue,/history,/free,/interrupt,/object_info,/system_stats) and a fake Lemonade (generation, load/unload, health, SDA capability). Outputs are placeholder PNGs and MP4s with audio. They show the prompt, seed and node list, so you can see what the backend really sent.run.shcreates sparse model files with the exact sizes the readiness checks need, then starts everything. The UI is athttp://127.0.0.1:7861/.- A prompt with
MOCK_ERRORfails the job. A prompt withMOCK_SLOWmakes it 5× longer, which is useful to test contention and reloads. - Not mocked: local Qwen PE (needs the isolated runtime) and the remote LLM enhancer (rejects loopback targets and uses Linux
getifaddrs).
I then drove all three tabs with a headless Chromium at 1440×900 and 390×844: single images, 4-image batches, cross-tab contention, an H3 reload during a job, the lightbox and the galleries. Four review agents read the gateway, the H3/Qwen backend, the Krea backend with repo tooling, and all frontend code. I checked their main claims against the source.
4. UI walkthrough
4.1 The result is never on screen
Krea and Qwen use a two-column layout. The output panel stretches to the height of the form, and the image is centered vertically in it. The Qwen form is about 2,400 px tall. So the result appears around y=1,400: below the first screen and above the Generate button. At no scroll position can you see the Generate button and the result together. During a batch you see a dark empty panel and one status line.
Fix: position: sticky; top: 16px; align-self: start; max-height: calc(100vh - 32px) on the output panel (in image-studio.css, so both apps get it). Show batch results as a strip of thumbnails under the main image. Keep the previous image under a busy overlay, do not hide it.
4.2 "Lab coordinator is busy"
I started a slow 4-image Krea batch, switched to the Qwen tab and pressed Generate. Qwen got HTTP 409 and stopped. The error panel says only lab coordinator is busy. The status line under the button still said "Submitting…" (a stale status). The Krea batch finished 50 s later, and nothing retried.
Qwen already has a retry loop for 409, but only for image 2 and later of its own batch (static-qwen/app.js:1363). Krea batches release the lease between images, so a well-timed submit from another tab can also take the GPU in the gap and make the Krea batch fail on its next image.
Fix, small: in all three apps, on 409, read /api/coordination/status, show "Waiting for Krea (2 m 10 s)…", and retry until it is free or the user cancels. Fix, complete: a FIFO queue in the gateway (FEAT-1b). The coordinator already serializes all work, so a queue is a natural next step. It also fixes batch interleaving.
4.3 H3 forgets a running job on reload
I started a slow H3 video and reloaded the page. The form came back empty: no prompt, no status, Generate enabled. The gateway still showed generating h3. The video later appeared in the gallery, but the UI did not say anything. Qwen solves this with sessionStorage (static-qwen/app.js:1251-1273). H3 keeps the job ID only in a closure.
4.4 Noise and density
The remote LLM enhancer panel shows four paragraphs of warnings even when the gateway has no enhancer configured, and then says "Remote enhancement is not configured". In Krea, this pushes the LoRA stack and the Generate button down to y≈1,770. In Qwen, a "Model availability" box with red operator diagnostics ("Pinned llama.cpp executable is unavailable") is the fourth element on the page.
Fix: hide the enhancer panel completely when it is not configured, and collapse the warnings to one line with a "details" link when it is. Move "Model availability" into a collapsed <details>, and show it open only when the selected checkpoint or profile is unavailable. Put only the controls you change often above the fold: prompt, size, count, Generate.
4.5 Other observations
- The lightbox is good. It has navigation, zoom, metadata, Reuse settings and "Use output as reference". It needs Delete, Favorite, "Same settings, new seed" and "Send to H3".
- Mobile works without horizontal scroll, but the shell header and the app header use ~40% of the first screen, and the page scrolls inside the iframe. On mobile the red "Unload all models" button is the largest element. Make it smaller or put it in a menu.
- The shell status pill is useful. It could also show the elapsed time, a dot on the tab that owns the GPU, and a Stop button for H3/Qwen jobs.
- Krea "Lemonade ready" is static text. It does not come from a health check.
5. All ideas
Effort: S hours, M days, L a week or more. Value is my estimate for a single-user lab. seen means I reproduced it in the mock UI. verified means I checked it in the code myself. Other items come from the review agents and have a confidence note where it matters. File references are relative to apps/ unless they start with another top folder.
5.1 Features
| ID | Idea | What and why | Effort | Value | Refs |
|---|---|---|---|---|---|
| FEAT-1a | Wait and retry on 409 | On a 409, show the owner and elapsed time from /api/coordination/status and retry until the lease is free or the user cancels. Today the first image fails at once. seen | S | high | h3-lab/static-qwen/app.js:1363; h3-lab/static/app.js; krea-ui/static/app.js |
| FEAT-1b | Gateway job queue | FIFO of stored request bodies with gateway job IDs and a queued phase. Submit four Qwen prompts and a video, then walk away. Also stops batches from different apps from interleaving. | L | high | lab-ui/server.py (Coordinator) |
| FEAT-2 | Step progress, ETA, latent preview | Connect to ComfyUI /ws with the existing client IDs, relay progress (step/max) and optional preview frames through /api/jobs/<id> or SSE. For Krea, show an ETA from a rolling median of past durations. No app shows progress today. verified | M | high | h3-lab/server.py:3157, 3437 (client_id); start.sh (--preview-method) |
| FEAT-3 | "Done" signal | Change the tab title and favicon when the shell sees generating → idle, plus an optional sound. Browser notifications need HTTPS, so for the LAN the option is an ntfy/webhook URL that the gateway calls on lease release. | S | high | lab-ui/static/app.js:69-79; lab-ui/server.py:316 |
| FEAT-4 | Gallery delete, favorites, filter | No app has a delete endpoint. Krea's ring buffer removes old images with no choice. Add DELETE /api/gallery/<id> (refuse non-terminal jobs), a favorite flag that retention keeps, and a text/mode/seed filter. | M | high | krea-ui/gallery.py; h3-lab/server.py:2507, 2782 |
| FEAT-5 | A/B compare in the lightbox | Two-up view of two results (same seed, different profile or checkpoint). This is the core workflow of a lab with several checkpoints and Turbo profiles. | M | med | packages/image-studio/image-studio.mjs |
| FEAT-6 | Cross-app hand-off | "Send to H3 as start frame" and "Use as Qwen edit input" from any lightbox. All apps are same-origin, so a BroadcastChannel("lab") message plus a tab switch is enough. The lightbox already has an extraActions hook. | M | high | image-studio.mjs:58; h3-lab/static/app.js (image.onchange) |
| FEAT-7 | Video continuation | "Use last frame as start frame" on an H3 result: seek to the end, draw to canvas, feed the crop editor. With FEAT-6 this gives chained clips. | M | med | h3-lab/static/app.js |
| FEAT-8 | Seed variations | One click: same settings, seed+1 or random. Today this needs Reuse settings, then a seed edit, then Generate. | S | med | lightbox actions |
| FEAT-9 | Preview the local Qwen PE rewrite | Local PE runs only inside generation, so you cannot read or edit the rewrite before a 2K render. Add POST /qwen/api/rewrite and let generate reuse the stored rewrite. A content-addressed cache would also skip the unload/reload cycle on repeats. | M | med | h3-lab/server.py:3168-3211, 102 |
| FEAT-10 | Settings inside the files | Write a compact lab iTXt chunk (prompt, effective prompt, seed, model, profile) into PNGs, and metadata tags into MP4s, so downloaded files keep their settings. | S | med | h3-lab/server.py:2373 (finalize); krea-ui/gallery.py |
| FEAT-11 | LoRA registry | Krea LoRAs and H3 EXTRA_LORAS are hard-coded, and Qwen has none. A loras.json (file, AIR, family, default strength, size/sha) read at startup makes adding one a data change. | M | med | krea-ui/server.py:58; h3-lab/server.py:291 |
| FEAT-12 | Upscale / hires pass | "Upscale this" from the gallery: a model upscaler or a draft→2K img2img pass as a Qwen profile. Generate drafts at 1 MP, then upscale only the good ones. | M | med | h3-lab/server.py (build_qwen_workflow) |
| FEAT-13 | Unified history | A "Recent" view in the shell that merges the three galleries by time, with deep links into each tab. | M | med | lab-ui/static |
| FEAT-14 | Show what is loaded | Combine Lemonade loaded models and ComfyUI /system_stats into the status: "Loaded: Qwen 2.1 · VRAM 21.3 GB". The Unload button can then say what it frees. | M | med | lab-ui/server.py (status) |
| FEAT-15 | Stop button in the shell | When the owner is h3, show Stop in the shell. H3 already has /api/interrupt. Make it job-scoped first (BUG-11). | S | med | h3-lab/server.py:3488 |
5.2 UX
| ID | Idea | What and why | Effort | Value | Refs |
|---|---|---|---|---|---|
| UX-1 | Sticky output column | See 4.1. One CSS rule in image-studio. seen | S | high | packages/image-studio/image-studio.css |
| UX-2 | Batch strip, keep previous image | Show all images of a batch as thumbnails. Keep the last image under a busy overlay during generation. Today Krea hides it and both apps show only the last image. seen | S | med | krea-ui/static/app.js:345; static-qwen/app.js |
| UX-3 | Hide the unconfigured enhancer | Do not render the enhancer panel when the gateway has no enhancer. When configured, collapse the warnings to one line. seen | S | med | lab-ui/static/prompt-enhancer.js |
| UX-4 | Collapse "Model availability" | Show it only when the selected option is unavailable. Put operator diagnostics in the API docs or a status page. seen | S | med | h3-lab/static-qwen/index.html |
| UX-5 | Persist form state | Save prompt, mode, size, steps, seed and LoRAs per app in localStorage, with "Reset to defaults". The existing reuse-settings code can do the restore. | M | high | all app.js |
| UX-6 | Keyboard: Ctrl/Cmd+Enter, no Enter-submit | Enter in Steps, Seed, Width or Height submits the form and can start an 8-image 2K batch. Block implicit submit. Add Ctrl/Cmd+Enter as one shared helper. | S | high | krea-ui/static/index.html; static-qwen/index.html |
| UX-7 | Drag-and-drop and paste | For Qwen references, H3 start frame and references, and enhancer attachments. All paths already end in a File[] function. | M | high | static-qwen/index.html:52; h3-lab/static/index.html:34, 45 |
| UX-8 | GPU owner on the tabs | A dot on the tab that owns the lease and a "busy" hint on the others before the user submits. | S | med | lab-ui/static/app.js:53-68 |
| UX-9 | H3 gallery: poster, lightbox, paging | Grey cards, no lightbox, all details on the card, and preload=metadata for every video. Use the shared lightbox with a video slot and a poster frame. seen | M | med | h3-lab/static/app.js:520-608 |
| UX-10 | Mobile header | Combine the shell and app headers on narrow screens and make Unload smaller. seen | S | low | lab-ui/static/app.css; h3-lab/static/app.css |
| UX-11 | Accessibility | Krea announces its 100 ms timer through aria-live. The shell announces the countdown every 2 s. The H3 crop canvas has no keyboard control. The Krea LoRA card is one big label. Polling continues in hidden tabs. | S | med | krea-ui/static/index.html:95; lab-ui/static/index.html:21; h3-lab/static/app.js:123-157 |
| UX-12 | Live Krea status pill | "Lemonade ready" is static text. Use the coordinator or Lemonade health. verified | S | low | krea-ui/static/index.html:19 |
5.3 Bugs
| ID | Bug | What happens | Effort | Value | Refs |
|---|---|---|---|---|---|
| BUG-1 | Unload timeout 30 s vs 75 s | The gateway coordinator HTTP client has a 30 s timeout. The H3 control unload waits up to 75 s for ComfyUI /free. A slow unload gives a false Krea 502 and a failed explicit/idle unload, while the unload still finishes. verified | S | high | lab-ui/server.py:1359; h3-lab/server.py:696 |
| BUG-2 | Idle unload repeats forever | After a successful unload, the next deadline is set again, so both backends get an unload call every 30 min (~48 per day) with nothing loaded, and the UI always shows a countdown. Add a "dirty" flag. verified | S | med | lab-ui/server.py:512-513, 547 |
| BUG-3 | Idle worker can die silently | _idle_loop has no try/except. If it dies, idle unload and H3 monitor recovery stop with no sign in the status. Confidence: medium. | S | med | lab-ui/server.py:534-554 |
| BUG-4 | Krea lease released while Krea may still run | On an upstream timeout or read error, the Krea lease is released although Lemonade may still be generating. A following H3 claim then unloads Lemonade under it. Krea leases are also memory-only. Confidence: medium. | M | med | lab-ui/server.py:907-926 |
| BUG-5 | H3 gallery metadata not fsynced | H3 writes {id}.json with write_text and replace and no fsync. The runbook says it fsyncs. Qwen uses atomic_json. verified | S | med | h3-lab/server.py:3428-3432, 617 |
| BUG-6 | H3 reload loses the job | See 4.3. Copy the Qwen sessionStorage resume. seen | S | high | h3-lab/static/app.js:338-391 |
| BUG-7 | H3 stops polling on one error | Any fetch or JSON error ends polling and the UI shows a failure while the job runs on. Qwen retries with backoff. | S | high | h3-lab/static/app.js:380-390 |
| BUG-8 | H3 Stop and Reuse races | Stop does not check the response, and a late poll can undo the "interrupted" state. "Reuse settings" in Ref2VA mode re-enables Generate during a job. | S | med | h3-lab/static/app.js:507-510, 613-616 |
| BUG-9 | Qwen stale "Submitting…" status | After a 409, the error panel shows the error but the status under the button still says "Submitting…". seen | S | low | static-qwen/app.js (jobStatus) |
| BUG-10 | INPUT_DIR grows without limit | Uploads up to 256 MiB, restored references and start images are never deleted. A start image is also leaked when later validation fails. | S | high | h3-lab/server.py:1355-1375, 1617 |
| BUG-11 | /api/interrupt is not job-scoped | It interrupts whatever ComfyUI runs, and the H3 UI reports "interrupted" even on 502. Accept a job ID: remove it if pending, interrupt only if running. | S | med | h3-lab/server.py:3488-3495 |
| BUG-12 | Journal rewrites on each poll | Polling a completed job rewrites the journal with four fsyncs every time. H3 status also globs every gallery JSON per poll. Skip unchanged writes and read {gallery_id}.json directly. | S | med | h3-lab/server.py:2479, 545-550, 2726, 2759 |
| BUG-13 | Non-atomic timing update | persist_generation_timing uses a fixed .json.tmp name from concurrent poll threads with no fsync. Readers can see a partial file. Confidence: medium. | S | med | h3-lab/server.py:2759-2779 |
| BUG-14 | Krea gallery resets after each image | After each image, Krea reloads page 1 of the gallery, so older pages the user loaded disappear and the grid flickers during a batch. | S | med | krea-ui/static/app.js:379, 280 |
| BUG-15 | Krea tests run three times | from test_gallery import HTTPTests makes discovery collect the HTTP tests again, and a subclass inherits them a third time. ~14 of 40 tests are duplicates and most of the 20 s runtime. verified | S | med | krea-ui/test_sda.py:9, 61 |
| BUG-16 | Gateway test fails on macOS | One enhancer test does not patch _local_addresses, so it gets 503 instead of 504 on non-Linux. | S | low | lab-ui/tests/test_prompt_enhancer.py:462 |
| BUG-17 | Small Krea validation holes | An integer with 400 digits raises OverflowError, which kills the handler with no 400. A non-dict Lemonade health reply raises AttributeError. Floats and booleans are truncated to ints. | S | low | krea-ui/server.py:99-106, 35, 43 |
| BUG-18 | Safe default binds | Krea defaults to 0.0.0.0:7861 (the gateway port!) and H3 to 0.0.0.0:7862. A manual start bypasses the coordinator on the LAN. Default to 127.0.0.1. | S | med | krea-ui/server.py:350; h3-lab/server.py:3500 |
| BUG-19 | Small gateway edge cases | --idle-timeout 0 gives a tight unload loop. /api/generate/ takes the lease and unloads H3, then Krea returns 404. Streaming reads use 128 KiB blocks (would stall SSE). | S | low | lab-ui/server.py:169, 835, 1105 |
| BUG-20 | H3 crop re-encodes on each keystroke | The enhancer listens to every input event and snapshots the H3 crop as a 1280 px JPEG data URL each time. | S | low | lab-ui/static/prompt-enhancer.js:205; h3-lab/static/app.js:624 |
5.4 Performance and storage
| ID | Idea | What and why | Effort | Value | Refs |
|---|---|---|---|---|---|
| PERF-1 | Thumbnails and caching | Gallery cards load full 2–4 MP PNGs with no-store. Write a 512 px WebP at finalize (and an H3 poster frame) and serve it as immutable. Krea image IDs are never reused, so they can be cached too. | M | high | h3-lab/server.py:2373, 3290; krea-ui/server.py:227 |
| PERF-2 | Gallery paging and index | The Qwen and H3 galleries stat and parse every item on each request, with no paging and no retention. H3 also reloads every 30 s. Add ?before=&limit= and a small SQLite index. | M | med | h3-lab/server.py:2507, 2782; h3-lab/static/app.js:619 |
| PERF-3 | Cache enhancer status probes | /qwen/api/config runs the enhancer status three times, including nvidia-smi and a subprocess probe. The UI polls this endpoint. | S | med | h3-lab/server.py:2904-2906, 932-947 |
| PERF-4 | Fewer image decodes | References are fully decoded several times per request (validate, build, persist, fingerprint). Read sizes without load() and memoize per request. | S | low | h3-lab/server.py:1767-1780, 1966 |
| PERF-5 | Stream uploads to disk | Video uploads up to 256 MiB are read fully into RAM, then copied into BytesIO. | S | low | h3-lab/server.py:3404, 1342 |
| PERF-6 | Disk report and retention | A script that shows sizes per category and finds orphans (journals without gallery, gallery without journal, old inputs), with an opt-in prune. | S | med | scripts/ |
5.5 Architecture and maintainability
| ID | Idea | What and why | Effort | Value | Refs |
|---|---|---|---|---|---|
| ARCH-1 | Split h3-lab/server.py | 3,504 lines in one file. Seams: config, storage/journal, comfy client, enhancer, h3, qwen, http. Keep server.py as the entry point. Do it after ARCH-2, with tests split to match. | L | high | h3-lab/server.py |
| ARCH-2 | One submit and reconcile pipeline | H3 and Qwen have the same seven-way exception ladder for submit, and the same reconcile algorithm in qwen_job/h3_job. One parametrized function removes ~150 lines and fixes BUG-5 and BUG-10 for H3 as a side effect. | M | high | h3-lab/server.py:3083-3254, 3410-3477, 2994-3082 |
| ARCH-3 | One range-serving function, request validators | HTTP Range handling is copied three times. Seed, int and prompt checks are repeated. Small helpers make the workflow builders ~30% shorter. | S | med | h3-lab/server.py:2855, 3301-3357 |
| ARCH-4 | Model pins as data | Sizes and hashes are in server.py, download scripts, MODELS.md and README. One models.json read by all of them stops drift. | M | med | h3-lab/server.py:25-70, 120-266; scripts/; models/MODELS.md |
| ARCH-5 | H3 on image-studio | H3 has its own minified CSS, header, gallery and no lightbox. Moving it to the shared package gives one look, a lightbox, mobile breakpoints, and fewer bugs. seen | M | high | h3-lab/static/; packages/image-studio/sync-assets.sh:34 |
| ARCH-6 | More shared frontend code | Two or three copies each of: requestJSON, job poller with resume, batch loop with "stop after current", setBusy, formatTime, gallery card builder. Put them in image-studio with tests. | M | high | static-qwen/app.js:1239-1330; krea-ui/static/app.js:116-135, 300-396 |
| ARCH-7 | Split Qwen app.js | 2,161 lines and ~30 module-level variables. Start with the guide editor (~400 lines, own state). Replace the regex/vm test harness with real module tests. | L | med | static-qwen/app.js:852-1237; h3-lab/test_qwen_presets.mjs:79-85 |
| ARCH-8 | Serve shared assets from the gateway | The gateway already serves /prompt-enhancer.js to all apps. It could serve image-studio too, which removes the copy-and-check sync. Keep the copies only if standalone deploys matter. If you keep them, enforce sync-assets.sh --check in a hook or test. | S | med | packages/image-studio/sync-assets.sh |
| ARCH-9 | Split gateway server.py | 1,472 lines: coordinator, proxy, http utils, cli. Coordinator tests then do not need the HTTP layer. Body-length parsing is duplicated three times. | M | med | lab-ui/server.py:150-581, 766-817, 1019-1032 |
| ARCH-10 | Workflow graph builder | Node IDs are hand-numbered strings, and some branches overwrite node "6". A tiny Graph.add() builder removes collision risk. The tests assert exact graphs, so plan the test updates. | M | low | h3-lab/server.py:1995, 2031 |
| ARCH-11 | Krea: constants and pure functions | SDA limits are written in four places, and the capability check compares an ordered list. Pull out gallery_settings() and one header helper. | S | med | krea-ui/server.py:44-48, 128, 176, 319-329 |
5.6 Operations and tooling
| ID | Idea | What and why | Effort | Value | Refs |
|---|---|---|---|---|---|
| OPS-1 | One test command | scripts/test-all.sh: sync check, three Python suites, node tests, node --check, OpenAPI JSON, bash -n, mock tests. Run it in a pre-push hook, or in Gitea Actions if a runner exists. Then the runbooks can drop hard-coded test counts. | S | high | krea-lab.md:131; lab-ui.md:117; h3-video-lab.md:385 |
| OPS-2 | Deploy scripts with drain | Add POST /api/coordination/drain (new claims get 409 "draining", running jobs continue). A deploy script then: runs tests, drains, waits for idle, installs from a manifest, compares hashes, restarts, smoke-tests. Today this is a manual runbook sequence with a race between "check idle" and "restart". | M | high | lab-ui.md (safe deployment); krea-lab.md:204 |
| OPS-3 | Log coordinator transitions | Claims, releases, unload results and journal restores only change in-memory fields. One log line per transition makes incidents traceable. Also drop the status-poll access logs (one line per browser every 2 s). | S | high | lab-ui/server.py:1218 |
| OPS-4 | /healthz | Backend reachability, idle worker alive, journal state, source hash. One curl replaces seven in the runbook. | S | med | lab-ui/server.py |
| OPS-5 | Slim the runbooks | h3-video-lab.md is 76 KB with lines of 2,500 characters. Dated validation stories are history, not operations. Move them to docs/validation/ and keep the runbook to current state and commands. | M | med | h3-video-lab.md:400-460; README.md (Qwen row) |
| OPS-6 | Container build | The image clones and installs ComfyUI twice. The BF16 PE runtime (a second PyTorch stack, "rollback only") could be a build target. Prefix ComfyUI and server logs, keep stderr from the CDI step, add a HEALTHCHECK. | S | med | h3-lab/Containerfile; Containerfile.base; start.sh; run.sh |
| OPS-7 | Unit dependencies | Requires=krea-ui.service restarts the gateway when Krea restarts, which kills H3 proxy threads. Wants= plus After= is enough because the gateway fails closed. | S | med | lab-ui/deploy/lab-ui.service:5 |
| OPS-8 | Lint baseline | ruff --select F,E9 is already clean, so it costs nothing to add. Do not adopt the formatter (17 of 24 files would change). | S | low | repo root |
| OPS-9 | Mock stack improvements | Mock the remote enhancer (needs a dev flag for the loopback check and a non-Linux address lookup) and the local Qwen PE. Add a ComfyUI websocket to the mock when FEAT-2 starts. | S | med | dev/mock/; lab-ui/prompt_enhancer.py:146 |
5.7 Security (only what matters on a trusted LAN)
| ID | Idea | What and why | Effort | Value | Refs |
|---|---|---|---|---|---|
| SEC-1 | Host header allowlist | Without --public-origin, the expected origin comes from Host, so a DNS-rebinding page passes the same-origin check. It could then read galleries, start jobs, unload models and use the enhancer. Check Host against a short allowlist. | S | med | lab-ui/server.py:1307-1308 |
| SEC-2 | Network limits in the unit | IPAddressDeny=any plus IPAddressAllow= localhost, LAN and the enhancer host. | S | low | lab-ui/deploy/lab-ui.service |
| SEC-3 | Limit concurrent enhancer calls | Each call can hold a thread for 30 min with up to 24 MiB of body. A semaphore of about 4 is enough. | S | low | lab-ui/prompt_enhancer.py:26, 34 |
6. Things I would not do
- Do not merge the gallery storage backends. Krea's SQLite blobs and Qwen's per-item directories with references serve different needs. Align the API (paging, delete, favorites), not the storage.
- Do not add a web framework or a frontend build step. The stdlib servers and vanilla JS are small enough. The problems are file size and duplication, not tooling.
- Do not adopt a code formatter now. The churn would hide real changes in review.
- Do not create a shared Python HTTP package yet. It would need the same copy-sync as image-studio for little gain. Split
h3-lab/server.pyfirst.
7. Appendix
Changes made in this session
dev/mock/mock_backends.py,dev/mock/run.sh,dev/mock/test_mock_backends.py: GPU-free mock stack and its tests.apps/krea-ui/server.py: optionalKREA_LEMONADE_URL(loopback only, default unchanged) andtest_lemonade_url.py. This was needed so the mock stack never talks to a real Lemonade.README.md: short section on the mock stack..gitignore:dev/mock/.state/.
Run it
dev/mock/run.sh, then open http://127.0.0.1:7861/. Put MOCK_SLOW or MOCK_ERROR in a prompt to simulate long or failing jobs.
