youngkingdom/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-05-04 01:18:05 +08:00
Code Issues Packages Projects Releases Wiki Activity

feat(ssrf_proxy): Add dev-mode and tests for ssrf_proxy

Browse Source 
Signed-off-by: -LAN- <laipz8200@outlook.com>
This commit is contained in:
-LAN-
2025-09-01 13:17:17 +08:00
parent 42110a8217
commit 6a54980824
8 changed files with 446 additions and 21 deletions
Download Patch File Download Diff File Expand all files Collapse all files

50
api/tests/integration_tests/ssrf_proxy/TEST_CASES_README.md
View File

@ -36,6 +36,21 @@ Run tests from a specific YAML file:
uv run python tests/integration_tests/ssrf_proxy/test_ssrf_proxy.py --test-file test_cases_extended.yaml
```
### Development Mode Testing
**WARNING: Development mode DISABLES all SSRF protections! Only use in development environments!**
Test the development mode configuration (used by docker-compose.middleware.yaml):
```bash
uv run python tests/integration_tests/ssrf_proxy/test_ssrf_proxy.py --dev-mode
```
Development mode:
- Mounts `conf.d.dev/` configuration that allows ALL requests
- Uses `test_cases_dev_mode.yaml` by default (all tests expect ALLOW)
- Verifies that private networks, cloud metadata, and non-standard ports are accessible
- Should NEVER be used in production environments
### Command Line Options
- `--host HOST`: Proxy host (default: localhost)
@ -44,6 +59,7 @@ uv run python tests/integration_tests/ssrf_proxy/test_ssrf_proxy.py --test-file
- `--save-results`: Save test results to JSON file
- `--test-file FILE`: Path to YAML file containing test cases
- `--list-tests`: List all test cases without running them
- `--dev-mode`: Run in development mode (DISABLES all SSRF protections - DO NOT use in production!)
## YAML Test Case Format
@ -63,10 +79,11 @@ test_categories:
## Available Test Files
1. **test_cases.yaml** - Standard test suite with essential test cases
1. **test_cases.yaml** - Standard test suite with essential test cases (default)
2. **test_cases_extended.yaml** - Extended test suite with additional edge cases and scenarios
3. **test_cases_dev_mode.yaml** - Development mode test suite (all requests should be allowed)
Both files are located in `api/tests/integration_tests/ssrf_proxy/`
All files are located in `api/tests/integration_tests/ssrf_proxy/`
## Categories
@ -107,6 +124,35 @@ The tests validate the SSRF proxy configuration files in `docker/ssrf_proxy/`:
- `squid.conf.template` - Squid proxy configuration
- `docker-entrypoint.sh` - Container initialization script
- `conf.d/` - Additional configuration files (if present)
- `conf.d.dev/` - Development mode configuration (when using --dev-mode)
## Development Mode Configuration
Development mode provides a zero-configuration environment for local development:
- Mounts `conf.d.dev/` instead of `conf.d/`
- Allows ALL requests including private networks and cloud metadata
- Enables access to any port
- Disables all SSRF protections
### Using Development Mode with Docker Compose
From the main Dify repository root:
```bash
# Use the development overlay
docker-compose -f docker-compose.middleware.yaml -f docker/ssrf_proxy/docker-compose.dev.yaml up ssrf_proxy
```
Or manually mount the development configuration:
```bash
docker run -d \
--name ssrf-proxy-dev \
-p 3128:3128 \
-v ./docker/ssrf_proxy/conf.d.dev:/etc/squid/conf.d:ro \
# ... other volumes
ubuntu/squid:latest
```
**CRITICAL**: Never use this configuration in production!
## Benefits

168
api/tests/integration_tests/ssrf_proxy/test_cases_dev_mode.yaml Normal file
View File

@ -0,0 +1,168 @@
# Development Mode Test Cases for SSRF Proxy
# These test cases verify that development mode correctly disables all SSRF protections
# WARNING: All requests should be ALLOWED in development mode
test_categories:
private_networks:
name: "Private Networks (Dev Mode)"
description: "In dev mode, private networks should be ALLOWED"
test_cases:
- name: "Loopback (127.0.0.1)"
url: "http://127.0.0.1"
expected_blocked: false # ALLOWED in dev mode
description: "IPv4 loopback - normally blocked, allowed in dev mode"
- name: "Localhost"
url: "http://localhost"
expected_blocked: false # ALLOWED in dev mode
description: "Localhost hostname - normally blocked, allowed in dev mode"
- name: "Private 10.x.x.x"
url: "http://10.0.0.1"
expected_blocked: false # ALLOWED in dev mode
description: "RFC 1918 private network - normally blocked, allowed in dev mode"
- name: "Private 172.16.x.x"
url: "http://172.16.0.1"
expected_blocked: false # ALLOWED in dev mode
description: "RFC 1918 private network - normally blocked, allowed in dev mode"
- name: "Private 192.168.x.x"
url: "http://192.168.1.1"
expected_blocked: false # ALLOWED in dev mode
description: "RFC 1918 private network - normally blocked, allowed in dev mode"
- name: "Link-local"
url: "http://169.254.1.1"
expected_blocked: false # ALLOWED in dev mode
description: "Link-local address - normally blocked, allowed in dev mode"
- name: "This network"
url: "http://0.0.0.0"
expected_blocked: false # ALLOWED in dev mode
description: "'This' network address - normally blocked, allowed in dev mode"
cloud_metadata:
name: "Cloud Metadata (Dev Mode)"
description: "In dev mode, cloud metadata endpoints should be ALLOWED"
test_cases:
- name: "AWS Metadata"
url: "http://169.254.169.254/latest/meta-data/"
expected_blocked: false # ALLOWED in dev mode
description: "AWS EC2 metadata - normally blocked, allowed in dev mode"
- name: "Azure Metadata"
url: "http://169.254.169.254/metadata/instance"
expected_blocked: false # ALLOWED in dev mode
description: "Azure metadata - normally blocked, allowed in dev mode"
non_standard_ports:
name: "Non-Standard Ports (Dev Mode)"
description: "In dev mode, all ports should be ALLOWED"
test_cases:
- name: "Port 8080"
url: "http://example.com:8080"
expected_blocked: false # ALLOWED in dev mode
description: "Alternative HTTP port - normally blocked, allowed in dev mode"
- name: "Port 3000"
url: "http://example.com:3000"
expected_blocked: false # ALLOWED in dev mode
description: "Node.js development port - normally blocked, allowed in dev mode"
- name: "SSH Port 22"
url: "http://example.com:22"
expected_blocked: false # ALLOWED in dev mode
description: "SSH port - normally blocked, allowed in dev mode"
- name: "Database Port 3306"
url: "http://example.com:3306"
expected_blocked: false # ALLOWED in dev mode
description: "MySQL port - normally blocked, allowed in dev mode"
- name: "Database Port 5432"
url: "http://example.com:5432"
expected_blocked: false # ALLOWED in dev mode
description: "PostgreSQL port - normally blocked, allowed in dev mode"
- name: "Redis Port 6379"
url: "http://example.com:6379"
expected_blocked: false # ALLOWED in dev mode
description: "Redis port - normally blocked, allowed in dev mode"
- name: "MongoDB Port 27017"
url: "http://example.com:27017"
expected_blocked: false # ALLOWED in dev mode
description: "MongoDB port - normally blocked, allowed in dev mode"
- name: "High Port 12345"
url: "http://example.com:12345"
expected_blocked: false # ALLOWED in dev mode
description: "Random high port - normally blocked, allowed in dev mode"
localhost_ports:
name: "Localhost with Various Ports (Dev Mode)"
description: "In dev mode, localhost with any port should be ALLOWED"
test_cases:
- name: "Localhost:8080"
url: "http://localhost:8080"
expected_blocked: false # ALLOWED in dev mode
description: "Localhost with port 8080 - normally blocked, allowed in dev mode"
- name: "Localhost:3000"
url: "http://localhost:3000"
expected_blocked: false # ALLOWED in dev mode
description: "Localhost with port 3000 - normally blocked, allowed in dev mode"
- name: "127.0.0.1:9200"
url: "http://127.0.0.1:9200"
expected_blocked: false # ALLOWED in dev mode
description: "Loopback with Elasticsearch port - normally blocked, allowed in dev mode"
- name: "127.0.0.1:5001"
url: "http://127.0.0.1:5001"
expected_blocked: false # ALLOWED in dev mode
description: "Loopback with API port - normally blocked, allowed in dev mode"
public_internet:
name: "Public Internet (Dev Mode)"
description: "Public internet should still work in dev mode"
test_cases:
- name: "Example.com"
url: "http://example.com"
expected_blocked: false
description: "Public website - always allowed"
- name: "Google HTTPS"
url: "https://www.google.com"
expected_blocked: false
description: "HTTPS public website - always allowed"
- name: "GitHub API"
url: "https://api.github.com"
expected_blocked: false
description: "Public API over HTTPS - always allowed"
special_cases:
name: "Special Cases (Dev Mode)"
description: "Edge cases that should all be allowed in dev mode"
test_cases:
- name: "Decimal IP notation"
url: "http://2130706433"
expected_blocked: false # ALLOWED in dev mode
description: "127.0.0.1 in decimal - normally blocked, allowed in dev mode"
- name: "Private network in subdomain"
url: "http://192-168-1-1.example.com"
expected_blocked: false
description: "Domain that looks like private IP - always allowed as it resolves externally"
- name: "IPv6 Loopback"
url: "http://[::1]"
expected_blocked: false # ALLOWED in dev mode
description: "IPv6 loopback - normally blocked, allowed in dev mode"
- name: "IPv6 Link-local"
url: "http://[fe80::1]"
expected_blocked: false # ALLOWED in dev mode
description: "IPv6 link-local - normally blocked, allowed in dev mode"

95
api/tests/integration_tests/ssrf_proxy/test_ssrf_proxy.py
View File

@ -47,18 +47,32 @@ class TestCase:
@final
class SSRFProxyTester:
def __init__(self, proxy_host: str = "localhost", proxy_port: int = 3128, test_file: str | None = None):
def __init__(
self,
proxy_host: str = "localhost",
proxy_port: int = 3128,
test_file: str | None = None,
dev_mode: bool = False,
):
self.proxy_host = proxy_host
self.proxy_port = proxy_port
self.proxy_url = f"http://{proxy_host}:{proxy_port}"
self.container_name = "ssrf-proxy-test"
self.container_name = "ssrf-proxy-test-dev" if dev_mode else "ssrf-proxy-test"
self.image = "ubuntu/squid:latest"
self.results: list[dict[str, object]] = []
self.test_file = test_file or "test_cases.yaml"
self.dev_mode = dev_mode
# Use dev mode test cases by default when in dev mode
if dev_mode and test_file is None:
self.test_file = "test_cases_dev_mode.yaml"
else:
self.test_file = test_file or "test_cases.yaml"
def start_proxy_container(self) -> bool:
"""Start the SSRF proxy container"""
print(f"{Colors.YELLOW}Starting SSRF proxy container...{Colors.NC}")
mode_str = " (DEVELOPMENT MODE)" if self.dev_mode else ""
print(f"{Colors.YELLOW}Starting SSRF proxy container{mode_str}...{Colors.NC}")
if self.dev_mode:
print(f"{Colors.RED}WARNING: Development mode DISABLES all SSRF protections!{Colors.NC}")
# Stop and remove existing container if exists
_ = subprocess.run(["docker", "stop", self.container_name], capture_output=True, text=True)
@ -70,6 +84,12 @@ class SSRFProxyTester:
project_root = os.path.abspath(os.path.join(script_dir, "..", "..", "..", ".."))
docker_config_dir = os.path.join(project_root, "docker", "ssrf_proxy")
# Choose configuration template based on mode
if self.dev_mode:
config_template = "squid.conf.dev.template"
else:
config_template = "squid.conf.template"
# Start container
cmd = [
"docker",
@ -82,7 +102,7 @@ class SSRFProxyTester:
"-p",
"8194:8194",
"-v",
f"{docker_config_dir}/squid.conf.template:/etc/squid/squid.conf.template:ro",
f"{docker_config_dir}/{config_template}:/etc/squid/squid.conf.template:ro",
"-v",
f"{docker_config_dir}/docker-entrypoint.sh:/docker-entrypoint-mount.sh:ro",
"-e",
@ -102,11 +122,16 @@ class SSRFProxyTester:
"cp /docker-entrypoint-mount.sh /docker-entrypoint.sh && sed -i 's/\\r$//' /docker-entrypoint.sh && chmod +x /docker-entrypoint.sh && /docker-entrypoint.sh", # noqa: E501
]
# Add conf.d mount if directory exists
conf_d_path = f"{docker_config_dir}/conf.d"
if os.path.exists(conf_d_path) and os.listdir(conf_d_path):
cmd.insert(-3, "-v")
cmd.insert(-3, f"{conf_d_path}:/etc/squid/conf.d:ro")
# Mount configuration directory (only in normal mode)
# In dev mode, the dev template already allows everything
if not self.dev_mode:
# Normal mode: mount regular conf.d if it exists
conf_d_path = f"{docker_config_dir}/conf.d"
if os.path.exists(conf_d_path) and os.listdir(conf_d_path):
cmd.insert(-3, "-v")
cmd.insert(-3, f"{conf_d_path}:/etc/squid/conf.d:ro")
else:
print(f"{Colors.YELLOW}Using development mode configuration (all SSRF protections disabled){Colors.NC}")
result = subprocess.run(cmd, capture_output=True, text=True)
@ -159,9 +184,17 @@ class SSRFProxyTester:
else:
# Other HTTP errors mean the request went through
is_blocked = False
except (urllib.error.URLError, OSError, TimeoutError):
# Connection errors mean blocked by proxy
is_blocked = True
except (urllib.error.URLError, OSError, TimeoutError) as e:
# In dev mode, connection errors to 169.254.x.x addresses are expected
# These addresses don't exist locally, so timeout is normal
# The proxy allowed the request, but the destination is unreachable
if self.dev_mode and "169.254" in test_case.url:
# In dev mode, if we're testing 169.254.x.x addresses,
# a timeout means the proxy allowed it (not blocked)
is_blocked = False
else:
# In normal mode, or for other addresses, connection errors mean blocked
is_blocked = True
except Exception as e:
# Unexpected error
print(f"{Colors.YELLOW}Warning: Unexpected error testing {test_case.url}: {e}{Colors.NC}")
@ -205,7 +238,13 @@ class SSRFProxyTester:
test_cases = self.get_test_cases()
print("=" * 50)
print(" SSRF Proxy Test Suite")
if self.dev_mode:
print(" SSRF Proxy Test Suite (DEV MODE)")
print("=" * 50)
print(f"{Colors.RED}WARNING: Testing with SSRF protections DISABLED!{Colors.NC}")
print(f"{Colors.YELLOW}All requests should be ALLOWED in dev mode.{Colors.NC}")
else:
print(" SSRF Proxy Test Suite")
print("=" * 50)
# Group tests by category
@ -295,9 +334,19 @@ class SSRFProxyTester:
print(f"Tests Skipped: {Colors.YELLOW}{skipped}{Colors.NC}")
if failed == 0:
print(f"\n{Colors.GREEN}✓ All tests passed! SSRF proxy is configured correctly.{Colors.NC}")
if hasattr(self, "dev_mode") and self.dev_mode:
print(f"\n{Colors.GREEN}✓ All tests passed! Development mode is working correctly.{Colors.NC}")
print(
f"{Colors.YELLOW}Remember: Dev mode DISABLES all SSRF protections - "
f"use only for development!{Colors.NC}"
)
else:
print(f"\n{Colors.GREEN}✓ All tests passed! SSRF proxy is configured correctly.{Colors.NC}")
else:
print(f"\n{Colors.RED}✗ Some tests failed. Please review the configuration.{Colors.NC}")
if hasattr(self, "dev_mode") and self.dev_mode:
print(f"\n{Colors.RED}✗ Some tests failed. Dev mode should allow ALL requests!{Colors.NC}")
else:
print(f"\n{Colors.RED}✗ Some tests failed. Please review the configuration.{Colors.NC}")
print("\nFailed tests:")
for r in self.results:
if r["result"] == "failed":
@ -330,6 +379,7 @@ def main():
save_results: bool = False
test_file: str | None = None
list_tests: bool = False
dev_mode: bool = False
def parse_args() -> Args:
parser = argparse.ArgumentParser(description="Test SSRF Proxy Configuration")
@ -345,6 +395,11 @@ def main():
"--test-file", type=str, help="Path to YAML file containing test cases (default: test_cases.yaml)"
)
_ = parser.add_argument("--list-tests", action="store_true", help="List all test cases without running them")
_ = parser.add_argument(
"--dev-mode",
action="store_true",
help="Run in development mode (DISABLES all SSRF protections - DO NOT use in production!)",
)
# Parse arguments - argparse.Namespace has Any-typed attributes
# This is a known limitation of argparse in Python's type system
@ -360,18 +415,22 @@ def main():
save_results=bool(namespace.save_results), # pyright: ignore[reportAny]
test_file=namespace.test_file if namespace.test_file else None, # pyright: ignore[reportAny]
list_tests=bool(namespace.list_tests), # pyright: ignore[reportAny]
dev_mode=bool(namespace.dev_mode), # pyright: ignore[reportAny]
)
args = parse_args()
tester = SSRFProxyTester(args.host, args.port, args.test_file)
tester = SSRFProxyTester(args.host, args.port, args.test_file, args.dev_mode)
# If --list-tests flag is set, just list the tests and exit
if args.list_tests:
test_cases = tester.get_test_cases()
mode_str = " (DEVELOPMENT MODE)" if args.dev_mode else ""
print("\n" + "=" * 50)
print(" Available Test Cases")
print(f" Available Test Cases{mode_str}")
print("=" * 50)
if args.dev_mode:
print(f"\n{Colors.RED}WARNING: Dev mode test cases expect ALL requests to be ALLOWED!{Colors.NC}")
# Group by category for display
categories: dict[str, list[TestCase]] = {}