Commit Graph

296 Commits

Author SHA1 Message Date
8a62c1d915 chore(api): pyright + ruff cleanup for openapi/cli surface
Type and lint pass over the openapi controllers, auth pipeline, and
oauth bearer/device-flow plumbing. Down from 36 pyright errors and 16
ruff errors to 0/0; 93 openapi unit tests pass.

Logic fixes:
- libs/oauth_bearer.py: drop private-naming on the friend-API methods
  consumed by _VariantResolver (cache_get / cache_set_positive /
  cache_set_negative / hard_expire / session_factory). They were always
  cross-class accessors — leading underscore was misleading. Add public
  registry property on BearerAuthenticator. _hard_expire row_id widened
  to UUID | str (matches the StringUUID column type).
- libs/oauth_bearer.py: type validate_bearer / bearer_feature_required
  with ParamSpec / PEP-695 so wrapped routes preserve their signature.
- libs/rate_limit.py: same — typed rate_limit decorator.
- services/oauth_device_flow.py: mint_oauth_token / _upsert accept
  Session | scoped_session (Flask-SQLAlchemy proxy). Guard row-is-None
  after upsert.
- controllers/openapi/{chat,completion,workflow}_messages.py: tuple-vs-
  Mapping shape narrowing on AppGenerateService.generate return —
  production returns Mapping, tests mock as (body, status). Validate
  through Pydantic Response model in both shapes.
- controllers/openapi/oauth_device.py: replace flask_restx.reqparse (banned)
  with Pydantic Request/Query models — DeviceCodeRequest, DevicePollRequest,
  DeviceLookupQuery, DeviceMutateRequest. Two PEP-695 generic helpers
  (_validate_json / _validate_query) translate ValidationError to BadRequest.
- controllers/openapi/auth/strategies.py: Protocol param-name match
  (subject_type), Optional narrowing on app/tenant/account_id/subject_email.
- controllers/openapi/auth/steps.py: subject_type-is-None guard before
  mounter dispatch.
- core/app/apps/workflow/generate_task_pipeline.py + models/workflow.py:
  add WorkflowAppLogCreatedFrom.OPENAPI + matching match-case branch.
  Fixes match-exhaustiveness and possibly-unbound created_from.
- libs/device_flow_security.py: pyright ignore on flask after_request
  hook (registered by the framework, pyright sees as unused).
- services/oauth_device_flow.py: rename Exceptions to *Error suffix
  (StateNotFoundError / InvalidTransitionError / UserCodeExhaustedError);
  same for libs/oauth_bearer.py (InvalidBearerError / TokenExpiredError).
  Update all callers across openapi controllers.
- controllers/openapi/{oauth_device,oauth_device_sso}.py +
  services/oauth_device_flow.py: switch logger.error in except blocks
  to logger.exception (TRY400) — keeps the traceback for ops.
- configs/feature/__init__.py: OPENAPI_KNOWN_CLIENT_IDS computed_field
  needs an @property alongside for pyright to see it as a value, not a
  method. Matches the existing line-451 pattern.

Plus ruff format + import-sort across the openapi tree (pure formatting).
2026-04-28 21:44:54 -07:00
9b2a37ceff feat(openapi): cookie auth for device-flow approval routes
Adds the openapi blueprint branch in load_user_from_request so that
account-branch device-flow approval routes (approve / deny /
approval-context) can authenticate via the console session cookie
under @login_required.

Splits extract_access_token into two helpers:
- extract_console_cookie_token (cookie-only) — used by openapi
  approval routes that must not fall through to the Authorization
  header, where dfoa_/dfoe_ bearers live (those aren't JWTs and
  PassportService.verify would crash on them).
- extract_access_token retains both code paths for legacy callers.
2026-04-28 20:41:38 -07:00
0b3b0b5ce8 feat(api): retire legacy /v1/* and /console/api device-flow mounts (Phase F)
Web and CLI consumers now hit /openapi/v1/* directly, so the dual-mount
shims can go:
  - controllers/oauth_device_sso.py (legacy /v1/oauth/device/sso-* + /v1/device/sso-complete)
  - controllers/service_api/oauth.py (legacy /v1/oauth/device/*, /v1/me, /v1/oauth/authorizations/self)
  - controllers/console/auth/oauth_device.py (placeholder for legacy /console/api/oauth/device/{approve,deny})
  - the deferred _register_legacy_console_mount() inside openapi/oauth_device.py

Imports in controllers/console/__init__.py, controllers/service_api/__init__.py,
and extensions/ext_blueprints.py pruned. Tests rewritten to openapi-only.
2026-04-27 00:45:10 -07:00
218ef6a447 feat(api): CORS posture for /openapi/v1 (Phase A.5)
OPENAPI_CORS_ALLOW_ORIGINS env var defaults to empty (same-origin only).
Operators expand for third-party integrations via comma-separated list.
Allowed headers: Authorization, Content-Type, X-CSRF-Token. Methods:
GET POST PATCH DELETE OPTIONS. Max-Age 600s. supports_credentials=True
so cookie-authed approve/deny work once Phase D moves them in.

Disallowed origins receive a normal 200 OPTIONS response without the
Access-Control-Allow-Origin header — flask-cors's standard behavior;
browser blocks the cross-origin request from the disallowed origin.

Plan: docs/superpowers/plans/2026-04-26-openapi-migration.md (in difyctl repo).
2026-04-26 23:30:27 -07:00
f5f224f49d feat(api): scaffold /openapi/v1 blueprint (Phase A.1)
New Flask blueprint at /openapi/v1/ that will host user-scoped
programmatic endpoints (device flow, identity, sessions, workspaces).
Ships only a smoke route GET /openapi/v1/_health for now; subsequent
phases lift handlers in from service_api, console, and the orphan
oauth_device_sso.py.

CORS is intentionally omitted here and configured in step A.5 once
the allowlist envvar lands.

Plan: docs/superpowers/plans/2026-04-26-openapi-migration.md (in difyctl repo).
2026-04-26 23:08:15 -07:00
fe8510ad1a feat(api,web): OAuth 2.0 device flow + bearer auth (RFC 8628)
Adds a CLI-friendly authorization flow so difyctl (and future
non-browser clients) can obtain user-scoped tokens without copy-
pasting cookies or raw API keys. Two grant paths share one device
flow surface:

  1. Account branch — user signs in via the existing /signin
     methods, /device page calls console-authed approve, mints a
     dfoa_ token tied to (account_id, tenant).
  2. External-SSO branch (EE) — /v1/oauth/device/sso-initiate signs
     an SSOState envelope, hands off to Enterprise's external ACS,
     receives a signed external-subject assertion, mints a dfoe_
     token tied to (subject_email, subject_issuer).

API surface (all under /v1, EE-only endpoints 404 on CE):

  POST   /v1/oauth/device/code              — RFC 8628 start
  POST   /v1/oauth/device/token             — RFC 8628 poll
  GET    /v1/oauth/device/lookup            — pre-validate user_code
  GET    /v1/oauth/device/sso-initiate      — SSO branch entry
  GET    /v1/device/sso-complete            — SSO callback sink
  GET    /v1/oauth/device/approval-context  — /device cookie probe
  POST   /v1/oauth/device/approve-external  — SSO approve
  GET    /v1/me                             — bearer subject lookup
  DELETE /v1/oauth/authorizations/self      — self-revoke
  POST   /console/api/oauth/device/approve  — account approve
  POST   /console/api/oauth/device/deny     — account deny

Core primitives:
- libs/oauth_bearer.py: prefix-keyed TokenKindRegistry +
  BearerAuthenticator + validate_bearer decorator. Two-tier scope
  (full vs apps:run) stamped from the registry, never from the DB.
- libs/jws.py: HS256 compact JWS keyed on the shared Dify
  SECRET_KEY — same key-set verifies the SSOState envelope, the
  external-subject assertion (minted by Enterprise), and the
  approval-grant cookie.
- libs/device_flow_security.py: enterprise_only gate, approval-
  grant cookie mint/verify/consume (Path=/v1/oauth/device,
  HttpOnly, SameSite=Lax, Secure follows is_secure()), anti-
  framing headers.
- libs/rate_limit.py: typed RateLimit / RateLimitScope dispatch
  with composite-key buckets; both decorator + imperative form.
- services/oauth_device_flow.py: Redis state machine (PENDING ->
  APPROVED|DENIED with atomic consume-on-poll), token mint via
  partial unique index uq_oauth_active_per_device (rotates in
  place), env-driven TTL policy.

Storage: oauth_access_tokens table with partial unique index on
(subject_email, subject_issuer, client_id, device_label) WHERE
revoked_at IS NULL. account_id NULL distinguishes external-SSO
rows. Hard-expire is CAS UPDATE (revoked_at + nullify token_hash)
so audit events keep their token_id. Retention pruner DELETEs
revoked + zombie-expired rows past OAUTH_ACCESS_TOKEN_RETENTION_DAYS.

Frontend: /device page with code-entry, chooser (account vs SSO),
authorize-account, authorize-sso views. SSO branch detaches from
the URL user_code and reads everything from the cookie via
/approval-context. Anti-framing headers on all responses.

Wiring: ENABLE_OAUTH_BEARER feature flag; ext_oauth_bearer binds
the authenticator at startup; clean_oauth_access_tokens_task
scheduled in ext_celery.

Spec: docs/specs/v1.0/server/{device-flow,tokens,middleware,security}.md
2026-04-26 20:06:43 -07:00
3193e8a712 chore: reorg imports (#35308)
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
2026-04-16 08:50:02 +00:00
53a22aa41b feat: collaboration (#30781)
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: yyh <yuanyouhuilyz@gmail.com>
Co-authored-by: yyh <92089059+lyzno1@users.noreply.github.com>
2026-04-16 02:21:04 +00:00
1bcc7f78c7 refactor: replace bare dict with dict[str, Any] in models, providers, extensions, libs, and test utilities (#35186) 2026-04-14 17:51:25 +00:00
736880e046 feat: support configurable redis key prefix (#35139) 2026-04-14 09:31:41 +00:00
eeebedcfe8 refactor: replace bare dict with dict[str, Any] in core provider services and misc modules (#35124) 2026-04-14 03:03:08 +00:00
ed83f5369e refactor: replace bare dict with dict[str, Any] in entities, workflow nodes, and tasks (#35109) 2026-04-14 03:02:39 +00:00
44ebfa3bb8 refactor(api): type _get_cluster_connection_health_params with TypedDict (#34999) 2026-04-13 00:52:12 +00:00
674495680d refactor(api): type Redis connection param builder functions with TypedDicts (#34875) 2026-04-10 11:36:39 +00:00
af55665ff2 refactor(otel): replace Any with Tracer and [T] generics (#34883) 2026-04-10 07:37:14 +00:00
a31c1d2c69 refactor(api): type Celery SSL options and Sentinel transport dicts with TypedDicts (#34871) 2026-04-10 00:33:23 +00:00
27e484e7f8 feat: redis add retry logic (#34566)
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
2026-04-09 03:08:25 +00:00
ae9fcc2969 refactor: use sessionmaker in controllers, events, models, and tasks 1 (#34693) 2026-04-07 23:47:20 +00:00
8d37116fec refactor(api): type storage statistics with StorageStatisticsDict TypedDict (#34609)
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
2026-04-07 00:59:32 +00:00
e83a4090ac fix: lighten the health checks for the Worker and Worker Beat services, and disable them by default (#34572) 2026-04-06 02:26:26 +00:00
0f051d5886 refactor(api): type celery sqlcommenter tags with CelerySqlcommenterTagsDict TypedDict (#34526) 2026-04-03 12:06:15 +00:00
99
318a3d0308 refactor(api): tighten login and wrapper typing (#34447) 2026-04-02 09:36:58 +00:00
99
8f9dbf269e chore(api): align Python support with 3.12 (#34419)
Co-authored-by: Asuka Minato <i@asukaminato.eu.org>
2026-04-02 05:07:32 +00:00
c51cd42cb4 refactor(api): replace json.loads with Pydantic validation in controllers and infra layers (#34277)
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
2026-04-01 05:41:44 +00:00
f7b78b08fd refactor(api): narrow otel instrumentor typing (#33853)
Co-authored-by: 复试资料 <study@example.com>
Co-authored-by: Asuka Minato <i@asukaminato.eu.org>
2026-03-31 10:13:31 +08:00
lif
ae9a16a397 fix: upgrade langfuse SDK to v3+ for LLM-as-judge support (#34265)
Signed-off-by: majiayu000 <1835304752@qq.com>
2026-03-30 13:06:55 +00:00
99
40591a7c50 refactor(api): use standalone graphon package (#34209)
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
2026-03-27 21:05:32 +00:00
5a8a68cab8 feat: enterprise otel exporter (#33138)
Co-authored-by: QuantumGhost <obelisk.reg+git@gmail.com>
Co-authored-by: Yunlu Wen <yunlu.wen@dify.ai>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
2026-03-27 07:56:31 +00:00
2ea85d3ba2 refactor: use EnumText for model_type and WorkflowNodeExecution.status (#34093)
Co-authored-by: Krishna Chaitanya <krishnabkc15@gmail.com>
2026-03-26 20:34:44 +00:00
99
52e7492cbc refactor(api): rename dify_graph to graphon (#34095) 2026-03-25 21:58:56 +08:00
56593f20b0 refactor(api): continue decoupling dify_graph from API concerns (#33580)
Signed-off-by: -LAN- <laipz8200@outlook.com>
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
Co-authored-by: WH-2099 <wh2099@pm.me>
2026-03-25 20:32:24 +08:00
a2c4345c00 fix: pass default root to OpenDAL Operator for fs scheme (#33678)
Co-authored-by: QuantumGhost <obelisk.reg+git@gmail.com>
2026-03-19 15:30:13 +08:00
8a22cc06c9 refactor: migrate db.session.query to select in infra layer (#33694)
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
2026-03-19 15:48:29 +09:00
a717519822 refactor(api): tighten phase 1 shared type contracts (#33453) 2026-03-17 17:50:51 +08:00
18ff5d9288 feat: add metrics to clean message and workflow-run task (#33143)
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
Co-authored-by: hj24 <mambahj24@gmail.com>
2026-03-17 13:55:28 +08:00
fb41b215c8 refactor(api): move workflow knowledge nodes and trigger nodes (#33445) 2026-03-15 15:24:59 +08:00
e64f4d6039 refactor: replace sa.String with EnumText in mapped_column for type s… (#33332)
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-14 13:38:27 +09:00
a0f0c97133 chore(deps): bump opentelemetry-propagator-b3 from 1.28.0 to 1.40.0 in /api in the opentelemetry group (#33308)
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Asuka Minato <i@asukaminato.eu.org>
2026-03-12 00:37:48 +09:00
eaf86c521f feat: Improve SQL Comment Context for Celery Worker Queries (#33058) 2026-03-10 12:37:26 +08:00
05ab107e73 feat: add export app messages (#32990) 2026-03-07 11:27:15 +08:00
2f4c740d46 feat: support redis xstream (#32586) 2026-03-04 13:18:55 +08:00
b584434e28 feat: redis connection support max connections (#32935)
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
2026-03-04 11:52:43 +08:00
4fd6b52808 refactor(api): move model_runtime into dify_graph (#32858) 2026-03-02 20:15:32 +08:00
c917838f9c refactor: move workflow package to dify_graph (#32844) 2026-03-02 18:42:30 +08:00
1f0fca89a8 refactor(workflow): move variables package into core.workflow (#32750) 2026-03-01 03:15:09 +08:00
eea1cf17ef refactor(workflow): inject redis into graph engine manager (#32622) 2026-02-27 13:29:52 +08:00
349d2d8e4e fix: replace deprecated SpanAttributes and ResourceAttributes with new semconv imports (#32661)
Co-authored-by: User <user@example.com>
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
2026-02-27 08:53:45 +09:00
2eefb585f9 fix: add type annotations to BaseStorage.exists and BaseStorage.download (#32652)
Signed-off-by: edvatar <88481784+toroleapinc@users.noreply.github.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
2026-02-27 07:35:30 +09:00
33e0dae2b2 ci: try from main repo (#32620)
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
2026-02-26 16:30:10 +09:00
99
7656d514b9 refactor(workflow-file): move core.file to core.workflow.file (#32252)
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
2026-02-16 22:38:19 +08:00