feat(deps): add unified dependency resolver using uv pip compile (#2589)

* feat(deps): add unified dependency resolver using uv pip compile

- Add UnifiedDepResolver module with 7 FRs: collect, compile, install pipeline
- Integrate startup batch resolution in prestartup_script.py (module scope)
- Skip per-node pip install in execute_install_script() when unified mode active
- Add use_unified_resolver config flag following use_uv pattern
- Input sanitization: reject -r, -e, --find-links, @ file://, path separators
- Handle --index-url/--extra-index-url separation with credential redaction
- Fallback to per-node pip on resolver failure or uv unavailability
- Add 98 unit tests across 20 test classes
- Add PRD and Design docs with cm_global integration marked as DEFERRED

* fix(deps): reset use_unified_resolver flag on startup fallback

When the unified resolver fails at startup (compile error, install
error, uv unavailable, or generic exception), the runtime flag was
not being reset to False. This caused subsequent runtime installs
to incorrectly defer pip dependencies instead of falling back to
per-node pip install.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* test(deps): add manual test cases for unified dependency resolver

Add environment setup guide and 16 test cases covering:
- Normal batch resolution (TC-1), disabled state (TC-2)
- Fallback paths: uv unavailable (TC-3), compile fail (TC-4),
  install fail (TC-5), generic exception (TC-16)
- install.py preservation (TC-6), runtime defer (TC-13)
- Input sanitization: dangerous patterns (TC-7), path separators
  (TC-8), index-url separation (TC-9), credential redaction (TC-10)
- Disabled pack exclusion (TC-11), no-deps path (TC-12)
- Both unified resolver guard paths (TC-14), post-fallback (TC-15)

Includes API reference, traceability matrix, and out-of-scope items.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(deps): prevent read_config() from overriding resolver fallback state

read_config() in manager_core.py unconditionally re-read
use_unified_resolver from config.ini, undoing the False set by
prestartup_script.py on resolver fallback. This caused runtime
installs to still defer deps even after a startup batch failure.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(deps): support multiple index URLs per line and optimize downgrade check

- Rewrite _split_index_url() to handle multiple --index-url /
  --extra-index-url options on a single requirements.txt line using
  regex-based parsing instead of single split.
- Cache installed_packages snapshot in collect_requirements() to avoid
  repeated subprocess calls during downgrade blacklist checks.
- Add unit tests for multi-URL lines and bare --index-url edge case.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* test(deps): add E2E scripts and update test documentation

Add automated E2E test scripts for unified dependency resolver:
- setup_e2e_env.sh: idempotent environment setup (clone ComfyUI,
  create venv, install deps, symlink Manager, write config.ini)
- start_comfyui.sh: foreground-blocking launcher using
  tail -f | grep -q readiness detection
- stop_comfyui.sh: graceful SIGTERM → SIGKILL shutdown

Update test documentation reflecting E2E testing findings:
- TEST-environment-setup.md: add automated script usage, document
  caveats (PYTHONPATH, config.ini path, Manager v4 /v2/ prefix,
  Blocked by policy, bash ((var++)) trap, git+https:// rejection)
- TEST-unified-dep-resolver.md: add TC-17 (restart dependency
  detection), TC-18 (real node pack integration), Validated
  Behaviors section, normalize API port to 8199

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(deps): harden input sanitization, expand test coverage, bump version

Security:
- Add _INLINE_DANGEROUS_OPTIONS regex to catch pip options after package
  names (--find-links, --constraint, --requirement, --editable, --trusted-host,
  --global-option, --install-option and short forms)
- Stage index URLs in pending_urls, commit only after full line validation
  to prevent URL injection from rejected lines

Tests:
- Add 50 new tests: inline sanitization, false-positive guards, parse
  helpers (_parse_conflicts, _parse_install_output), exception paths
  (91 → 141 total, all pass)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* feat(cli): add uv-compile command and --uv-compile flag for batch dependency resolution

Add two CLI entry points for the unified dependency resolver:

- `cm_cli uv-compile`: standalone batch resolution of all installed
  node pack dependencies via uv pip compile
- `cm_cli install --uv-compile`: skip per-node pip, batch-resolve all
  deps after install completes (mutually exclusive with --no-deps)

Both use a shared `_run_unified_resolve()` helper that passes real
cm_global values (pip_blacklist, pip_overrides, pip_downgrade_blacklist)
and guarantees PIPFixer.fix_broken() runs via try/finally.

Update DESIGN, PRD, and TEST docs for consistency.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Dr.Lt.Data
2026-03-07 06:51:53 +09:00
committed by GitHub
parent 0d88a3874d
commit f042d73b72
15 changed files with 4667 additions and 8 deletions

View File

@ -0,0 +1,241 @@
#!/usr/bin/env bash
# setup_e2e_env.sh — Automated E2E environment setup for ComfyUI + Manager
#
# Creates an isolated ComfyUI installation with ComfyUI-Manager for E2E testing.
# Idempotent: skips setup if marker file and key artifacts already exist.
#
# Input env vars:
# E2E_ROOT — target directory (default: auto-generated via mktemp)
# MANAGER_ROOT — manager repo root (default: auto-detected from script location)
# COMFYUI_BRANCH — ComfyUI branch to clone (default: master)
# PYTHON — Python executable (default: python3)
#
# Output (last line of stdout):
# E2E_ROOT=/path/to/environment
#
# Exit: 0=success, 1=failure
set -euo pipefail
# --- Constants ---
COMFYUI_REPO="https://github.com/comfyanonymous/ComfyUI.git"
PYTORCH_CPU_INDEX="https://download.pytorch.org/whl/cpu"
CONFIG_INI_CONTENT="[default]
use_uv = true
use_unified_resolver = true
file_logging = false"
# --- Logging helpers ---
log() { echo "[setup_e2e] $*"; }
err() { echo "[setup_e2e] ERROR: $*" >&2; }
die() { err "$@"; exit 1; }
# --- Detect manager root by walking up from script dir to find pyproject.toml ---
detect_manager_root() {
local dir
dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
while [[ "$dir" != "/" ]]; do
if [[ -f "$dir/pyproject.toml" ]]; then
echo "$dir"
return 0
fi
dir="$(dirname "$dir")"
done
return 1
}
# --- Validate prerequisites ---
validate_prerequisites() {
local py="${PYTHON:-python3}"
local missing=()
command -v git >/dev/null 2>&1 || missing+=("git")
command -v uv >/dev/null 2>&1 || missing+=("uv")
command -v "$py" >/dev/null 2>&1 || missing+=("$py")
if [[ ${#missing[@]} -gt 0 ]]; then
die "Missing prerequisites: ${missing[*]}"
fi
# Verify Python version >= 3.9
local py_version
py_version=$("$py" -c "import sys; print(f'{sys.version_info.major}.{sys.version_info.minor}')")
local major minor
major="${py_version%%.*}"
minor="${py_version##*.}"
if [[ "$major" -lt 3 ]] || { [[ "$major" -eq 3 ]] && [[ "$minor" -lt 9 ]]; }; then
die "Python 3.9+ required, found $py_version"
fi
log "Prerequisites OK (python=$py_version, git=$(git --version | awk '{print $3}'), uv=$(uv --version 2>&1 | awk '{print $2}'))"
}
# --- Check idempotency: skip if already set up ---
check_already_setup() {
local root="$1"
if [[ -f "$root/.e2e_setup_complete" ]] \
&& [[ -d "$root/comfyui" ]] \
&& [[ -d "$root/venv" ]] \
&& [[ -f "$root/comfyui/user/__manager/config.ini" ]] \
&& [[ -L "$root/comfyui/custom_nodes/ComfyUI-Manager" ]]; then
log "Environment already set up at $root (marker file exists). Skipping."
echo "E2E_ROOT=$root"
exit 0
fi
}
# --- Verify the setup ---
verify_setup() {
local root="$1"
local manager_root="$2"
local venv_py="$root/venv/bin/python"
local errors=0
log "Running verification checks..."
# Check ComfyUI directory
if [[ ! -f "$root/comfyui/main.py" ]]; then
err "Verification FAIL: comfyui/main.py not found"
((errors++))
fi
# Check venv python
if [[ ! -x "$venv_py" ]]; then
err "Verification FAIL: venv python not executable"
((errors++))
fi
# Check symlink
local link_target="$root/comfyui/custom_nodes/ComfyUI-Manager"
if [[ ! -L "$link_target" ]]; then
err "Verification FAIL: symlink $link_target does not exist"
((errors++))
elif [[ "$(readlink -f "$link_target")" != "$(readlink -f "$manager_root")" ]]; then
err "Verification FAIL: symlink target mismatch"
((errors++))
fi
# Check config.ini
if [[ ! -f "$root/comfyui/user/__manager/config.ini" ]]; then
err "Verification FAIL: config.ini not found"
((errors++))
fi
# Check Python imports
# comfy is a local package inside ComfyUI (not pip-installed), and
# comfyui_manager.__init__ imports from comfy — both need PYTHONPATH
if ! PYTHONPATH="$root/comfyui" "$venv_py" -c "import comfy" 2>/dev/null; then
err "Verification FAIL: 'import comfy' failed"
((errors++))
fi
if ! PYTHONPATH="$root/comfyui" "$venv_py" -c "import comfyui_manager" 2>/dev/null; then
err "Verification FAIL: 'import comfyui_manager' failed"
((errors++))
fi
if [[ "$errors" -gt 0 ]]; then
die "Verification failed with $errors error(s)"
fi
log "Verification OK: all checks passed"
}
# ===== Main =====
# Resolve MANAGER_ROOT
if [[ -z "${MANAGER_ROOT:-}" ]]; then
MANAGER_ROOT="$(detect_manager_root)" || die "Cannot detect MANAGER_ROOT. Set it explicitly."
fi
MANAGER_ROOT="$(cd "$MANAGER_ROOT" && pwd)"
log "MANAGER_ROOT=$MANAGER_ROOT"
# Validate prerequisites
validate_prerequisites
PYTHON="${PYTHON:-python3}"
COMFYUI_BRANCH="${COMFYUI_BRANCH:-master}"
# Create or use E2E_ROOT
CREATED_BY_US=false
if [[ -z "${E2E_ROOT:-}" ]]; then
E2E_ROOT="$(mktemp -d -t e2e_comfyui_XXXXXX)"
CREATED_BY_US=true
log "Created E2E_ROOT=$E2E_ROOT"
else
mkdir -p "$E2E_ROOT"
log "Using E2E_ROOT=$E2E_ROOT"
fi
# Idempotency check
check_already_setup "$E2E_ROOT"
# Cleanup trap: remove E2E_ROOT on failure only if we created it
cleanup_on_failure() {
local exit_code=$?
if [[ "$exit_code" -ne 0 ]] && [[ "$CREATED_BY_US" == "true" ]]; then
err "Setup failed. Cleaning up $E2E_ROOT"
rm -rf "$E2E_ROOT"
fi
}
trap cleanup_on_failure EXIT
# Step 1: Clone ComfyUI
log "Step 1/8: Cloning ComfyUI (branch=$COMFYUI_BRANCH)..."
if [[ -d "$E2E_ROOT/comfyui/.git" ]]; then
log " ComfyUI already cloned, skipping"
else
git clone --depth=1 --branch "$COMFYUI_BRANCH" "$COMFYUI_REPO" "$E2E_ROOT/comfyui"
fi
# Step 2: Create virtual environment
log "Step 2/8: Creating virtual environment..."
if [[ -d "$E2E_ROOT/venv" ]]; then
log " venv already exists, skipping"
else
uv venv "$E2E_ROOT/venv"
fi
VENV_PY="$E2E_ROOT/venv/bin/python"
# Step 3: Install ComfyUI dependencies
log "Step 3/8: Installing ComfyUI dependencies (CPU-only)..."
uv pip install \
--python "$VENV_PY" \
-r "$E2E_ROOT/comfyui/requirements.txt" \
--extra-index-url "$PYTORCH_CPU_INDEX"
# Step 4: Install ComfyUI-Manager (non-editable, production-like)
log "Step 4/8: Installing ComfyUI-Manager..."
uv pip install --python "$VENV_PY" "$MANAGER_ROOT"
# Step 5: Create symlink for custom_nodes discovery
log "Step 5/8: Creating custom_nodes symlink..."
mkdir -p "$E2E_ROOT/comfyui/custom_nodes"
local_link="$E2E_ROOT/comfyui/custom_nodes/ComfyUI-Manager"
if [[ -L "$local_link" ]]; then
log " Symlink already exists, updating"
rm -f "$local_link"
fi
ln -s "$MANAGER_ROOT" "$local_link"
# Step 6: Write config.ini to correct path
log "Step 6/8: Writing config.ini..."
mkdir -p "$E2E_ROOT/comfyui/user/__manager"
echo "$CONFIG_INI_CONTENT" > "$E2E_ROOT/comfyui/user/__manager/config.ini"
# Step 7: Create HOME isolation directories
log "Step 7/8: Creating HOME isolation directories..."
mkdir -p "$E2E_ROOT/home/.config"
mkdir -p "$E2E_ROOT/home/.local/share"
mkdir -p "$E2E_ROOT/logs"
# Step 8: Verify setup
log "Step 8/8: Verifying setup..."
verify_setup "$E2E_ROOT" "$MANAGER_ROOT"
# Write marker file
date -Iseconds > "$E2E_ROOT/.e2e_setup_complete"
# Clear the EXIT trap since setup succeeded
trap - EXIT
log "Setup complete."
echo "E2E_ROOT=$E2E_ROOT"

View File

@ -0,0 +1,129 @@
#!/usr/bin/env bash
# start_comfyui.sh — Foreground-blocking ComfyUI launcher for E2E tests
#
# Starts ComfyUI in the background, then blocks the foreground until the server
# is ready (or timeout). This makes it safe to call from subprocess.run() or
# Claude's Bash tool — the call returns only when ComfyUI is accepting requests.
#
# Input env vars:
# E2E_ROOT — (required) path to E2E environment from setup_e2e_env.sh
# PORT — ComfyUI listen port (default: 8199)
# TIMEOUT — max seconds to wait for readiness (default: 120)
#
# Output (last line on success):
# COMFYUI_PID=<pid> PORT=<port>
#
# Exit: 0=ready, 1=timeout/failure
set -euo pipefail
# --- Defaults ---
PORT="${PORT:-8199}"
TIMEOUT="${TIMEOUT:-120}"
# --- Logging helpers ---
log() { echo "[start_comfyui] $*"; }
err() { echo "[start_comfyui] ERROR: $*" >&2; }
die() { err "$@"; exit 1; }
# --- Validate environment ---
[[ -n "${E2E_ROOT:-}" ]] || die "E2E_ROOT is not set"
[[ -d "$E2E_ROOT/comfyui" ]] || die "ComfyUI not found at $E2E_ROOT/comfyui"
[[ -x "$E2E_ROOT/venv/bin/python" ]] || die "venv python not found at $E2E_ROOT/venv/bin/python"
[[ -f "$E2E_ROOT/.e2e_setup_complete" ]] || die "Setup marker not found. Run setup_e2e_env.sh first."
PY="$E2E_ROOT/venv/bin/python"
COMFY_DIR="$E2E_ROOT/comfyui"
LOG_DIR="$E2E_ROOT/logs"
LOG_FILE="$LOG_DIR/comfyui.log"
PID_FILE="$LOG_DIR/comfyui.pid"
mkdir -p "$LOG_DIR"
# --- Check/clear port ---
if ss -tlnp 2>/dev/null | grep -q ":${PORT}\b"; then
log "Port $PORT is in use. Attempting to stop existing process..."
# Try to read existing PID file
if [[ -f "$PID_FILE" ]]; then
OLD_PID="$(cat "$PID_FILE")"
if kill -0 "$OLD_PID" 2>/dev/null; then
kill "$OLD_PID" 2>/dev/null || true
sleep 2
fi
fi
# Fallback: kill by port pattern
if ss -tlnp 2>/dev/null | grep -q ":${PORT}\b"; then
pkill -f "main\\.py.*--port $PORT" 2>/dev/null || true
sleep 2
fi
# Final check
if ss -tlnp 2>/dev/null | grep -q ":${PORT}\b"; then
die "Port $PORT is still in use after cleanup attempt"
fi
log "Port $PORT cleared."
fi
# --- Start ComfyUI ---
log "Starting ComfyUI on port $PORT..."
# Create empty log file (ensures tail -f works from the start)
: > "$LOG_FILE"
# Launch with unbuffered Python output so log lines appear immediately
PYTHONUNBUFFERED=1 \
HOME="$E2E_ROOT/home" \
nohup "$PY" "$COMFY_DIR/main.py" \
--cpu \
--enable-manager \
--port "$PORT" \
> "$LOG_FILE" 2>&1 &
COMFYUI_PID=$!
echo "$COMFYUI_PID" > "$PID_FILE"
log "ComfyUI PID=$COMFYUI_PID, log=$LOG_FILE"
# Verify process didn't crash immediately
sleep 1
if ! kill -0 "$COMFYUI_PID" 2>/dev/null; then
err "ComfyUI process died immediately. Last 30 lines of log:"
tail -n 30 "$LOG_FILE" >&2
rm -f "$PID_FILE"
exit 1
fi
# --- Block until ready ---
# tail -n +1 -f: read from file start AND follow new content (no race condition)
# grep -q -m1: exit on first match → tail gets SIGPIPE → pipeline ends
# timeout: kill the pipeline after TIMEOUT seconds
log "Waiting up to ${TIMEOUT}s for ComfyUI to become ready..."
if timeout "$TIMEOUT" bash -c \
"tail -n +1 -f '$LOG_FILE' 2>/dev/null | grep -q -m1 'To see the GUI'"; then
log "ComfyUI startup message detected."
else
err "Timeout (${TIMEOUT}s) waiting for ComfyUI. Last 30 lines of log:"
tail -n 30 "$LOG_FILE" >&2
kill "$COMFYUI_PID" 2>/dev/null || true
rm -f "$PID_FILE"
exit 1
fi
# Verify process is still alive after readiness detected
if ! kill -0 "$COMFYUI_PID" 2>/dev/null; then
err "ComfyUI process died after readiness signal. Last 30 lines:"
tail -n 30 "$LOG_FILE" >&2
rm -f "$PID_FILE"
exit 1
fi
# Optional HTTP health check
if command -v curl >/dev/null 2>&1; then
if curl -sf "http://127.0.0.1:${PORT}/system_stats" >/dev/null 2>&1; then
log "HTTP health check passed (/system_stats)"
else
log "HTTP health check skipped (endpoint not yet available, but startup message detected)"
fi
fi
log "ComfyUI is ready."
echo "COMFYUI_PID=$COMFYUI_PID PORT=$PORT"

View File

@ -0,0 +1,75 @@
#!/usr/bin/env bash
# stop_comfyui.sh — Graceful ComfyUI shutdown for E2E tests
#
# Stops a ComfyUI process previously started by start_comfyui.sh.
# Uses SIGTERM first, then SIGKILL after a grace period.
#
# Input env vars:
# E2E_ROOT — (required) path to E2E environment
# PORT — ComfyUI port for fallback pkill (default: 8199)
#
# Exit: 0=stopped, 1=failed
set -euo pipefail
PORT="${PORT:-8199}"
GRACE_PERIOD=10
# --- Logging helpers ---
log() { echo "[stop_comfyui] $*"; }
err() { echo "[stop_comfyui] ERROR: $*" >&2; }
die() { err "$@"; exit 1; }
# --- Validate ---
[[ -n "${E2E_ROOT:-}" ]] || die "E2E_ROOT is not set"
PID_FILE="$E2E_ROOT/logs/comfyui.pid"
# --- Read PID ---
COMFYUI_PID=""
if [[ -f "$PID_FILE" ]]; then
COMFYUI_PID="$(cat "$PID_FILE")"
log "Read PID=$COMFYUI_PID from $PID_FILE"
fi
# --- Graceful shutdown via SIGTERM ---
if [[ -n "$COMFYUI_PID" ]] && kill -0 "$COMFYUI_PID" 2>/dev/null; then
log "Sending SIGTERM to PID $COMFYUI_PID..."
kill "$COMFYUI_PID" 2>/dev/null || true
# Wait for graceful shutdown
elapsed=0
while kill -0 "$COMFYUI_PID" 2>/dev/null && [[ "$elapsed" -lt "$GRACE_PERIOD" ]]; do
sleep 1
elapsed=$((elapsed + 1))
done
# Force kill if still alive
if kill -0 "$COMFYUI_PID" 2>/dev/null; then
log "Process still alive after ${GRACE_PERIOD}s. Sending SIGKILL..."
kill -9 "$COMFYUI_PID" 2>/dev/null || true
sleep 1
fi
fi
# --- Fallback: kill by port pattern ---
if ss -tlnp 2>/dev/null | grep -q ":${PORT}\b"; then
log "Port $PORT still in use. Attempting pkill fallback..."
pkill -f "main\\.py.*--port $PORT" 2>/dev/null || true
sleep 2
if ss -tlnp 2>/dev/null | grep -q ":${PORT}\b"; then
pkill -9 -f "main\\.py.*--port $PORT" 2>/dev/null || true
sleep 1
fi
fi
# --- Cleanup PID file ---
rm -f "$PID_FILE"
# --- Verify port is free ---
if ss -tlnp 2>/dev/null | grep -q ":${PORT}\b"; then
die "Port $PORT is still in use after shutdown"
fi
log "ComfyUI stopped."