youngkingdom/ComfyUI-Manager
1
0
Fork 0
mirror of https://github.com/Comfy-Org/ComfyUI-Manager.git synced 2026-05-03 16:57:51 +08:00
Code Issues Packages Projects Releases Wiki Activity

test(deps): add E2E scripts and update test documentation

Browse Source 
Add automated E2E test scripts for unified dependency resolver:
- setup_e2e_env.sh: idempotent environment setup (clone ComfyUI,
  create venv, install deps, symlink Manager, write config.ini)
- start_comfyui.sh: foreground-blocking launcher using
  tail -f | grep -q readiness detection
- stop_comfyui.sh: graceful SIGTERM → SIGKILL shutdown

Update test documentation reflecting E2E testing findings:
- TEST-environment-setup.md: add automated script usage, document
  caveats (PYTHONPATH, config.ini path, Manager v4 /v2/ prefix,
  Blocked by policy, bash ((var++)) trap, git+https:// rejection)
- TEST-unified-dep-resolver.md: add TC-17 (restart dependency
  detection), TC-18 (real node pack integration), Validated
  Behaviors section, normalize API port to 8199

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Dr.Lt.Data
2026-02-28 08:46:59 +09:00
parent 3d9c9ca8de
commit ca8698533d
5 changed files with 657 additions and 24 deletions
Download Patch File Download Diff File Expand all files Collapse all files

124
docs/dev/TEST-environment-setup.md
View File

@ -2,13 +2,48 @@
Procedures for setting up a ComfyUI environment with ComfyUI-Manager installed for functional testing.
## Automated Setup (Recommended)
Three shell scripts in `tests/e2e/scripts/` automate the entire lifecycle:
```bash
# 1. Setup: clone ComfyUI, create venv, install deps, symlink Manager
E2E_ROOT=/tmp/e2e_test MANAGER_ROOT=/path/to/comfyui-manager-draft4 \
bash tests/e2e/scripts/setup_e2e_env.sh
# 2. Start: launches ComfyUI in background, blocks until ready
E2E_ROOT=/tmp/e2e_test bash tests/e2e/scripts/start_comfyui.sh
# 3. Stop: graceful SIGTERM → SIGKILL shutdown
E2E_ROOT=/tmp/e2e_test bash tests/e2e/scripts/stop_comfyui.sh
# 4. Cleanup
rm -rf /tmp/e2e_test
```
### Script Details
| Script | Purpose | Input | Output |
|--------|---------|-------|--------|
| `setup_e2e_env.sh` | Full environment setup (8 steps) | `E2E_ROOT`, `MANAGER_ROOT`, `COMFYUI_BRANCH` (default: master), `PYTHON` (default: python3) | `E2E_ROOT=<path>` on last line |
| `start_comfyui.sh` | Foreground-blocking launcher | `E2E_ROOT`, `PORT` (default: 8199), `TIMEOUT` (default: 120s) | `COMFYUI_PID=<pid> PORT=<port>` |
| `stop_comfyui.sh` | Graceful shutdown | `E2E_ROOT`, `PORT` (default: 8199) | — |
**Idempotent**: `setup_e2e_env.sh` checks for a `.e2e_setup_complete` marker file and skips setup if the environment already exists.
**Blocking mechanism**: `start_comfyui.sh` uses `tail -n +1 -f | grep -q -m1 'To see the GUI'` to block until ComfyUI is ready. No polling loop needed.
---
## Prerequisites
- Python 3.9+
- Git
- `uv` (pip alternative, install via `pip install uv`)
- `uv` (install via `pip install uv` or [standalone](https://docs.astral.sh/uv/getting-started/installation/))
## Environment Setup
## Manual Setup (Reference)
For understanding or debugging, the manual steps are documented below. The automated scripts execute these same steps.
### 1. ComfyUI Clone
@ -50,17 +85,44 @@ uv pip install -e "$MANAGER_ROOT"
> **Note**: Editable mode (`-e`) reflects code changes without reinstalling.
> For production-like testing, use `uv pip install "$MANAGER_ROOT"` (non-editable).
### 5. ComfyUI Launch
### 5. Symlink Manager into custom_nodes
```bash
ln -s "$MANAGER_ROOT" "$COMFY_ROOT/custom_nodes/ComfyUI-Manager"
```
### 6. Write config.ini
```bash
mkdir -p "$COMFY_ROOT/user/__manager"
cat > "$COMFY_ROOT/user/__manager/config.ini" << 'EOF'
[default]
use_uv = true
use_unified_resolver = true
EOF
```
> **IMPORTANT**: The config path is `$COMFY_ROOT/user/__manager/config.ini`, resolved by `folder_paths.get_system_user_directory("manager")`. It is NOT inside the symlinked Manager directory.
### 7. HOME Isolation
```bash
export HOME=/tmp/e2e_home
mkdir -p "$HOME/.config" "$HOME/.local/share"
```
### 8. ComfyUI Launch
```bash
cd "$COMFY_ROOT"
python main.py --enable-manager --cpu
PYTHONUNBUFFERED=1 python main.py --enable-manager --cpu --port 8199
```
| Flag | Purpose |
|------|---------|
| `--enable-manager` | Enable ComfyUI-Manager (disabled by default) |
| `--cpu` | Run without GPU (for functional testing) |
| `--port 8199` | Use non-default port to avoid conflicts |
| `--enable-manager-legacy-ui` | Enable legacy UI (optional) |
| `--listen` | Allow remote connections (optional) |
@ -85,24 +147,48 @@ When Manager loads successfully, the following log lines appear:
[START] ComfyUI-Manager # manager_server.py loaded
```
The `Blocked by policy` message for Manager in custom_nodes is **expected** — `should_be_disabled()` in `comfyui_manager/__init__.py` prevents legacy double-loading when Manager is already pip-installed.
---
## Caveats & Known Issues
### PYTHONPATH for `comfy` imports
ComfyUI's `comfy` package is a **local package** inside the ComfyUI directory — it is NOT pip-installed. Any code that imports from `comfy` (including `comfyui_manager.__init__`) requires `PYTHONPATH` to include the ComfyUI directory:
```bash
PYTHONPATH="$COMFY_ROOT" python -c "import comfy"
PYTHONPATH="$COMFY_ROOT" python -c "import comfyui_manager"
```
The automated scripts handle this via `PYTHONPATH` in verification checks and the ComfyUI process inherits it implicitly by running from the ComfyUI directory.
### config.ini path
The config file must be at `$COMFY_ROOT/user/__manager/config.ini`, **NOT** inside the Manager symlink directory. This is resolved by `folder_paths.get_system_user_directory("manager")` at `prestartup_script.py:65-73`.
### Manager v4 endpoint prefix
All Manager endpoints use the `/v2/` prefix (e.g., `/v2/manager/queue/status`, `/v2/snapshot/get_current`). Paths without the prefix will return 404.
### `Blocked by policy` is expected
When Manager detects that it's loaded as a custom_node but is already pip-installed, it prints `Blocked by policy` and skips legacy loading. This is intentional behavior in `comfyui_manager/__init__.py:39-51`.
### Bash `((var++))` trap
Under `set -e`, `((0++))` evaluates the pre-increment value (0), and `(( 0 ))` returns exit code 1, killing the script. Use `var=$((var + 1))` instead.
### `git+https://` URLs in requirements.txt
Some node packs (e.g., Impact Pack's SAM2 dependency) use `git+https://github.com/...` URLs. The unified resolver correctly rejects these with "rejected path separator" — they must be installed separately.
---
## Cleanup
```bash
deactivate
rm -rf "$COMFY_ROOT"
```
## Quick Reference (Copy-Paste)
```bash
# Full setup in one block
COMFY_ROOT=$(mktemp -d)/ComfyUI
MANAGER_ROOT=/path/to/comfyui-manager-draft4
git clone https://github.com/comfyanonymous/ComfyUI.git "$COMFY_ROOT"
cd "$COMFY_ROOT"
uv venv .venv && source .venv/bin/activate
uv pip install -r requirements.txt --extra-index-url https://download.pytorch.org/whl/cpu
uv pip install -e "$MANAGER_ROOT"
python main.py --enable-manager --cpu
```

112
docs/dev/TEST-unified-dep-resolver.md
View File

@ -21,10 +21,12 @@ use_unified_resolver = true
Node pack installation at runtime uses the task queue API:
```
POST http://localhost:8188/v2/manager/queue/task
POST http://localhost:8199/v2/manager/queue/task
Content-Type: application/json
```
> **Port**: E2E tests use port 8199 to avoid conflicts with running ComfyUI instances. Replace with your actual port if different.
**Payload** (`QueueTaskItem`):
| Field | Type | Description |
@ -186,7 +188,7 @@ chmod -R u+w "$(python -c 'import site; print(site.getsitepackages()[0])')"
**Steps**:
1. While ComfyUI is running, install a node pack that has both `install.py` and `requirements.txt` via API:
```bash
curl -X POST http://localhost:8188/v2/manager/queue/task \
curl -X POST http://localhost:8199/v2/manager/queue/task \
-H "Content-Type: application/json" \
-d '{
"ui_id": "test-installpy",
@ -334,7 +336,7 @@ private-pkg --index-url https://user:token123@pypi.private.com/simple
1. Start ComfyUI and confirm batch resolution succeeds
2. While ComfyUI is running, install a new node pack via API:
```bash
curl -X POST http://localhost:8188/v2/manager/queue/task \
curl -X POST http://localhost:8199/v2/manager/queue/task \
-H "Content-Type: application/json" \
-d '{
"ui_id": "test-defer-1",
@ -377,7 +379,7 @@ Verify both code locations that guard per-node pip install behave correctly in u
**Path 1 — Runtime API install (class method)**:
```bash
# While ComfyUI is running:
curl -X POST http://localhost:8188/v2/manager/queue/task \
curl -X POST http://localhost:8199/v2/manager/queue/task \
-H "Content-Type: application/json" \
-d '{
"ui_id": "test-path1",
@ -426,7 +428,7 @@ echo "['$COMFY_ROOT/custom_nodes/test_pack_lazy', '#LAZY-INSTALL-SCRIPT', '$PYTH
1. Set up conflicting deps (as in TC-4) and start ComfyUI (resolver fails, flag reset to `False`)
2. While still running, install a new node pack via API:
```bash
curl -X POST http://localhost:8188/v2/manager/queue/task \
curl -X POST http://localhost:8199/v2/manager/queue/task \
-H "Content-Type: application/json" \
-d '{
"ui_id": "test-postfallback",
@ -478,6 +480,101 @@ chmod u+r "$COMFY_ROOT/custom_nodes"
---
## TC-17: Restart Dependency Detection [P0]
**Precondition**: `use_unified_resolver = true`, automated E2E scripts available
This test verifies that the resolver correctly detects and installs dependencies for node packs added between restarts, incrementally building the dependency set.
**Steps**:
1. Boot ComfyUI with no custom node packs (Boot 1 — baseline)
2. Verify baseline deps only (Manager's own deps)
3. Stop ComfyUI
4. Clone `ComfyUI-Impact-Pack` into `custom_nodes/`
5. Restart ComfyUI (Boot 2)
6. Verify Impact Pack deps are installed (`cv2`, `skimage`, `dill`, `scipy`, `matplotlib`)
7. Stop ComfyUI
8. Clone `ComfyUI-Inspire-Pack` into `custom_nodes/`
9. Restart ComfyUI (Boot 3)
10. Verify Inspire Pack deps are installed (`cachetools`, `webcolors`)
**Expected log (each boot)**:
```
[UnifiedDepResolver] Collected N deps from M sources (skipped S)
[UnifiedDepResolver] running: ... uv pip compile ...
[UnifiedDepResolver] running: ... uv pip install ...
[UnifiedDepResolver] startup batch resolution succeeded
```
**Verify**:
- Boot 1: ~10 deps from ~10 sources; `cv2`, `dill`, `cachetools` are NOT installed
- Boot 2: ~19 deps from ~18 sources; `cv2`, `skimage`, `dill`, `scipy`, `matplotlib` all importable
- Boot 3: ~24 deps from ~21 sources; `cachetools`, `webcolors` also importable
- Both packs show as loaded in logs
**Automation**: Use `tests/e2e/scripts/` (setup → start → stop) with node pack cloning between boots.
---
## TC-18: Real Node Pack Integration [P0]
**Precondition**: `use_unified_resolver = true`, network access to GitHub + PyPI
Full pipeline test with real-world node packs (`ComfyUI-Impact-Pack` + `ComfyUI-Inspire-Pack`) to verify the resolver handles production requirements.txt files correctly.
**Steps**:
1. Set up E2E environment
2. Clone both Impact Pack and Inspire Pack into `custom_nodes/`
3. Direct-mode: instantiate `UnifiedDepResolver`, call `collect_requirements()` and `resolve_and_install()`
4. Boot-mode: start ComfyUI and verify via logs
**Expected behavior (direct mode)**:
```
--- Discovered node packs (3) --- # Manager, Impact, Inspire
ComfyUI-Impact-Pack
ComfyUI-Inspire-Pack
ComfyUI-Manager
--- Phase 1: Collect Requirements ---
Total requirements: ~24
Skipped: 1 # SAM2 git+https:// URL
Extra index URLs: set()
```
**Verify**:
- `git+https://github.com/facebookresearch/sam2.git` is correctly rejected with "rejected path separator"
- All other dependencies are collected and resolved
- After install, `cv2`, `PIL`, `scipy`, `skimage`, `matplotlib` are all importable
- No conflicting version errors during compile
**Automation**: Use `tests/e2e/scripts/` (setup → clone packs → start) with direct-mode resolver invocation.
---
## Validated Behaviors (from E2E Testing)
The following behaviors were confirmed during manual E2E testing:
### Resolver Pipeline
- **3-phase pipeline**: Collect → `uv pip compile` → `uv pip install` works end-to-end
- **Incremental detection**: Resolver discovers new node packs on each restart without reinstalling existing deps
- **Dependency deduplication**: Overlapping deps from multiple packs are resolved to compatible versions
### Security & Filtering
- **`git+https://` rejection**: URLs like `git+https://github.com/facebookresearch/sam2.git` are rejected with "rejected path separator" — SAM2 is the only dependency skipped from Impact Pack
- **Blacklist filtering**: `PackageRequirement` objects have `.name`, `.spec`, `.source` attributes; `collected.skipped` returns `[(spec_string, reason_string)]` tuples
### Manager Integration
- **Manager v4 endpoints**: All endpoints use `/v2/` prefix (e.g., `/v2/manager/queue/status`)
- **`Blocked by policy`**: Expected when Manager is pip-installed and also symlinked in `custom_nodes/`; prevents legacy double-loading
- **config.ini path**: Must be at `$COMFY_ROOT/user/__manager/config.ini`, not in the symlinked Manager dir
### Environment
- **PYTHONPATH requirement**: `comfy` is a local package (not pip-installed); `comfyui_manager` imports from `comfy`, so both require `PYTHONPATH=$COMFY_ROOT`
- **HOME isolation**: `HOME=$E2E_ROOT/home` prevents host config contamination during boot
---
## Summary
| TC | P | Scenario | Key Verification |
@ -498,6 +595,8 @@ chmod u+r "$COMFY_ROOT/custom_nodes"
| 14 | P0 | Both unified resolver paths | runtime API (class method) + startup lazy install |
| 15 | P1 | Post-fallback behavior | Per-node pip resumes in same process |
| 16 | P1 | Generic exception fallback | Distinct from uv-absent and batch-failed |
| 17 | P0 | Restart dependency detection | Incremental node pack discovery across restarts |
| 18 | P0 | Real node pack integration | Impact + Inspire Pack full pipeline |
### Traceability
@ -513,3 +612,6 @@ chmod u+r "$COMFY_ROOT/custom_nodes"
| Fallback behavior | TC-3, TC-4, TC-5, TC-15, TC-16 |
| Disabled node pack exclusion | TC-11 |
| Runtime defer behavior | TC-13, TC-14 |
| FR-8: Restart discovery | TC-17 |
| FR-9: Real-world compatibility | TC-17, TC-18 |
| FR-2: Input sanitization (git URLs) | TC-8, TC-18 |