# Backsplash Visualizer (Next.js + ONNX pipeline)

This project serves a backsplash visualizer built with Next.js 15. The server side runs a multi-model segmentation stack (ONNX/Sharp) to find tiles, occluders, and foreground objects before computing lighting masks for rendering.

## What the backend does today
- **Foreground (plants/props) mask** via U²-Net (`models/u2net_finetune.onnx` prioritized, then `u2net.onnx`, `u2netp.onnx`).
- **Depth backplane** via Depth-Anything V2 (ONNX) to estimate the backsplash plane and auto-mark closer objects as occluders.
- **Tile segmentation** via a sliding-window U-Net (`models/unet_latest/unet_tiles.onnx`) that labels background / grout / tiles and reports per-class coverage.
- **Occluder fusion** optionally merges YOLO/SAM-based masks (Python worker or local fallback) with the tile mask so foreground objects stay in front of tiles.
- **Light/shadow extraction** (`lib/illumination.ts`) builds blurred low-frequency light maps, respecting the foreground mask.
- **APIs**: `/api/mask-debug` (returns PNG mask for a test image) and `/api/light-masks` (returns light/shadow masks; pulls `fg_mask` from `analyze-mix` if not provided).

## Runtime pipeline (per request)
1) **Load models**
   - U²-Net sessions are created once using `U2NET_MODEL_PRIORITY` and `U2NET_USE_ALL_MODELS`. Env overrides sizes (`U2NET_PRIMARY_SIZE`, `U2NET_SECONDARY_SIZE`) and thresholds.
   - Tile U-Net session loads from `models/unet_latest/` with optional class/color metadata.
2) **Foreground mask** (`lib/segmentation/u2net.ts`)
   - Image is resized to primary/secondary sizes, run through U²-Net, optionally merged across all models, and post-processed with grow/threshold knobs (`U2NET_FALLBACK_THRESHOLD`, `U2NET_GROW_*`).
3) **Tile mask** (`lib/segmentation/tile-unet.ts`)
   - Image is scaled to a max dimension (`TILE_UNET_MAX_DIMENSION`), tiled with overlap, run through the ONNX model, and stitched back; coverage per class is reported.
4) **Occluders (optional)** (`lib/segmentation/occluder.ts`)
   - If configured, fetches a mask from a Python worker (`OCCLUDER_WORKER_URL`) or local CLI fallback; resized to the request dimensions.
5) **Mask fusion** (`lib/segmentation/tile-mask.ts`)
   - Occluder mask is punched out of the tile mask; combined and raw variants are encoded as data URLs with coverage metadata.
6) **Illumination** (`lib/illumination.ts`)
   - Builds shadow/light PNG overlays from low-frequency luminance; foreground areas can be excluded when a mask is present.

The UI consumes these data URLs plus numeric coverage so reviewers can sanity check each stage.

## Project layout
- `models/` – ONNX assets (`u2net_finetune.onnx`, `u2net.onnx`, `u2netp.onnx`, `unet_latest/…`).
- `lib/segmentation/` – U²-Net, tile U-Net, occluder fusion, and full-pipeline helpers.
- `pages/api/` – `/api/mask-debug` and `/api/light-masks` entrypoints.
- `scripts/` – Python occluder helpers (`occluder_segment.py`, `occluder_worker.py`).
- `postbuild.js` / `.next/standalone` – production bundle used by PM2.

## Running locally
- Prereqs: Node.js 18+, npm; Python 3.9+ only if you need the occluder worker/CLI.
- Install JS deps: `npm install`
- Dev server: `npm run dev`
- Build: `npm run build`

Optional: create a Python venv and install `pillow numpy ultralytics onnxruntime` to run the occluder scripts.

## Key environment knobs
- `U2NET_MODEL_PRIORITY` (comma list), `U2NET_USE_ALL_MODELS`, `U2NET_PRIMARY_SIZE`, `U2NET_SECONDARY_SIZE`, `U2NET_FALLBACK_THRESHOLD`, `U2NET_GROW_SEED`, `U2NET_GROW_RADIUS`, `U2NET_GROW_MIN_COVERAGE`, `U2NET_STRETCH_MIN_SIDE` – control U²-Net loading and post-processing.
- `TILE_UNET_MAX_DIMENSION` – max side length before tiling/resizing.
- `OCCLUDER_WORKER_URL` (+ `workerTimeoutMs`, `preferWorker`) – external Python mask source.
- `OCCLUDER_METHOD` – occluder backend selector (`u2net_refine`, `sam_hq`, `hybrid`, `distilled_unsup`).
- `OCCLUDER_DISTILL_MIN_PROXY`, `OCCLUDER_DISTILL_MIN_COVERAGE`, `OCCLUDER_DISTILL_MAX_COVERAGE`, `OCCLUDER_DISTILL_MAX_TRANS_DENSITY`, `OCCLUDER_DISTILL_MAX_TILE_LEAK` – no-GT quality gates for `distilled_unsup`.
- `OCCLUDER_ALT_SPILL_PRUNE_*` – optional spill-prune controls used by headless sample export tuning.
  - includes recall-guarded acceptance for GT eval:
    - `OCCLUDER_ALT_SPILL_PRUNE_RECALL_GUARD_ENABLED`
    - `OCCLUDER_ALT_SPILL_PRUNE_MAX_FN_INCREASE_PIXELS`
    - `OCCLUDER_ALT_SPILL_PRUNE_MAX_FN_INCREASE_FRAC`
- `NEXT_RUNTIME_BASE_PATH`, `PORT`, `HOST`, `NODE_ENV` – Next.js/PM2 runtime.

## Production-oriented distilled occluder mode
- Enable with: `OCCLUDER_METHOD=distilled_unsup`
- Behavior:
  - Runs base and SAM-prompted candidates.
  - Builds a distilled merge candidate and scores each using no-GT proxy quality.
  - Chooses highest proxy score and records diagnostics in response meta (`meta.distilled`).
  - Falls back to base candidate if proxy score is below `OCCLUDER_DISTILL_MIN_PROXY`.
- Benchmark command:
  - `bench/occluder_breakthrough/.venv/bin/python bench/occluder_breakthrough/run.py --samples sample_a,sample_b --method sam_matte_hybrid,distilled_unsup,fewshot_rf --clean`

## Production (PM2)
- Process: `nextjs` via `ecosystem.config.js` (Next standalone output). Env is picked up on start/restart.
- Restart with updated env: `pm2 restart ecosystem.config.js --only nextjs --update-env` (ensure `PM2_HOME` points to the site user’s `.pm2`).
- Logs: `~/.pm2/logs/nextjs-out.log` and `nextjs-error.log` (per user).

## Reviewer checklist
- Confirm models exist under `models/` and env priorities point at the finetuned U²-Net.
- Exercise `/api/mask-debug` with a known image to inspect `fg_mask` coverage and overlays.
- Exercise `/api/light-masks` to verify light/shadow PNGs and response size (<4 MB warning in logs).
- Verify PM2 env after restart includes the U²-Net tuning vars and port/base-path settings.