mirror of
https://github.com/langgenius/dify.git
synced 2026-02-19 01:25:40 +08:00
Compare commits
18 Commits
test/log-r
...
chore/ssrf
| Author | SHA1 | Date | |
|---|---|---|---|
| ed6ac97854 | |||
| 0b74b82394 | |||
| d01931dd52 | |||
| 4ea43f93ae | |||
| 44c5f7ec5c | |||
| 895b847204 | |||
| 4d184c98de | |||
| 5ea168f03b | |||
| b7c87245a3 | |||
| 6a54980824 | |||
| 42110a8217 | |||
| fb36069f1c | |||
| 1e971bd20d | |||
| 621ede0f7b | |||
| 99ee64c864 | |||
| 1a49febc02 | |||
| 9e2b6325f3 | |||
| 23c97ec7f7 |
@ -1,4 +1,4 @@
|
||||
FROM mcr.microsoft.com/devcontainers/python:3.12-bookworm
|
||||
FROM mcr.microsoft.com/devcontainers/python:3.12-bullseye
|
||||
|
||||
RUN apt-get update && export DEBIAN_FRONTEND=noninteractive \
|
||||
&& apt-get -y install libgmp-dev libmpfr-dev libmpc-dev
|
||||
|
||||
3
.github/workflows/api-tests.yml
vendored
3
.github/workflows/api-tests.yml
vendored
@ -67,6 +67,9 @@ jobs:
|
||||
cp docker/.env.example docker/.env
|
||||
cp docker/middleware.env.example docker/middleware.env
|
||||
|
||||
- name: Setup SSRF Proxy for Testing
|
||||
run: sh docker/ssrf_proxy/setup-testing.sh
|
||||
|
||||
- name: Expose Service Ports
|
||||
run: sh .github/workflows/expose_service_ports.sh
|
||||
|
||||
|
||||
6
.gitignore
vendored
6
.gitignore
vendored
@ -228,10 +228,14 @@ web/public/fallback-*.js
|
||||
api/.env.backup
|
||||
/clickzetta
|
||||
|
||||
# SSRF Proxy - ignore the conf.d directory that's created for testing/local overrides
|
||||
docker/ssrf_proxy/conf.d/
|
||||
|
||||
# Benchmark
|
||||
scripts/stress-test/setup/config/
|
||||
scripts/stress-test/reports/
|
||||
|
||||
# mcp
|
||||
.playwright-mcp/
|
||||
.serena/
|
||||
.serena/
|
||||
|
||||
|
||||
@ -6,7 +6,7 @@
|
||||
|
||||
本指南和 Dify 一样在不断完善中。如果有任何滞后于项目实际情况的地方，恳请谅解，我们也欢迎任何改进建议。
|
||||
|
||||
关于许可证，请花一分钟阅读我们简短的[许可和贡献者协议](../../LICENSE)。同时也请遵循社区[行为准则](https://github.com/langgenius/.github/blob/main/CODE_OF_CONDUCT.md)。
|
||||
关于许可证，请花一分钟阅读我们简短的[许可和贡献者协议](../LICENSE)。同时也请遵循社区[行为准则](https://github.com/langgenius/.github/blob/main/CODE_OF_CONDUCT.md)。
|
||||
|
||||
## 开始之前
|
||||
|
||||
@ -6,7 +6,7 @@ Wir müssen wendig sein und schnell liefern, aber wir möchten auch sicherstelle
|
||||
|
||||
Dieser Leitfaden ist, wie Dify selbst, in ständiger Entwicklung. Wir sind dankbar für Ihr Verständnis, falls er manchmal hinter dem eigentlichen Projekt zurückbleibt, und begrüßen jedes Feedback zur Verbesserung.
|
||||
|
||||
Bitte nehmen Sie sich einen Moment Zeit, um unsere [Lizenz- und Mitwirkungsvereinbarung](../../LICENSE) zu lesen. Die Community hält sich außerdem an den [Verhaltenskodex](https://github.com/langgenius/.github/blob/main/CODE_OF_CONDUCT.md).
|
||||
Bitte nehmen Sie sich einen Moment Zeit, um unsere [Lizenz- und Mitwirkungsvereinbarung](../LICENSE) zu lesen. Die Community hält sich außerdem an den [Verhaltenskodex](https://github.com/langgenius/.github/blob/main/CODE_OF_CONDUCT.md).
|
||||
|
||||
## Bevor Sie loslegen
|
||||
|
||||
@ -6,7 +6,7 @@ Necesitamos ser ágiles y enviar rápidamente dado donde estamos, pero también
|
||||
|
||||
Esta guía, como Dify mismo, es un trabajo en constante progreso. Agradecemos mucho tu comprensión si a veces se queda atrás del proyecto real, y damos la bienvenida a cualquier comentario para que podamos mejorar.
|
||||
|
||||
En términos de licencia, por favor tómate un minuto para leer nuestro breve [Acuerdo de Licencia y Colaborador](../../LICENSE). La comunidad también se adhiere al [código de conducta](https://github.com/langgenius/.github/blob/main/CODE_OF_CONDUCT.md).
|
||||
En términos de licencia, por favor tómate un minuto para leer nuestro breve [Acuerdo de Licencia y Colaborador](../LICENSE). La comunidad también se adhiere al [código de conducta](https://github.com/langgenius/.github/blob/main/CODE_OF_CONDUCT.md).
|
||||
|
||||
## Antes de empezar
|
||||
|
||||
@ -6,7 +6,7 @@ Nous devons être agiles et livrer rapidement compte tenu de notre position, mai
|
||||
|
||||
Ce guide, comme Dify lui-même, est un travail en constante évolution. Nous apprécions grandement votre compréhension si parfois il est en retard par rapport au projet réel, et nous accueillons tout commentaire pour nous aider à nous améliorer.
|
||||
|
||||
En termes de licence, veuillez prendre une minute pour lire notre bref [Accord de Licence et de Contributeur](../../LICENSE). La communauté adhère également au [code de conduite](https://github.com/langgenius/.github/blob/main/CODE_OF_CONDUCT.md).
|
||||
En termes de licence, veuillez prendre une minute pour lire notre bref [Accord de Licence et de Contributeur](../LICENSE). La communauté adhère également au [code de conduite](https://github.com/langgenius/.github/blob/main/CODE_OF_CONDUCT.md).
|
||||
|
||||
## Avant de vous lancer
|
||||
|
||||
@ -6,7 +6,7 @@ Difyに貢献しようとお考えですか？素晴らしいですね。私た
|
||||
|
||||
このガイドは、Dify自体と同様に、常に進化し続けています。実際のプロジェクトの進行状況と多少のずれが生じる場合もございますが、ご理解いただけますと幸いです。改善のためのフィードバックも歓迎いたします。
|
||||
|
||||
ライセンスについては、[ライセンスと貢献者同意書](../../LICENSE)をご一読ください。また、コミュニティは[行動規範](https://github.com/langgenius/.github/blob/main/CODE_OF_CONDUCT.md)に従っています。
|
||||
ライセンスについては、[ライセンスと貢献者同意書](../LICENSE)をご一読ください。また、コミュニティは[行動規範](https://github.com/langgenius/.github/blob/main/CODE_OF_CONDUCT.md)に従っています。
|
||||
|
||||
## 始める前に
|
||||
|
||||
@ -6,7 +6,7 @@ Dify에 기여하려고 하시는군요 - 정말 멋집니다, 당신이 무엇
|
||||
|
||||
이 가이드는 Dify 자체와 마찬가지로 끊임없이 진행 중인 작업입니다. 때로는 실제 프로젝트보다 뒤처질 수 있다는 점을 이해해 주시면 감사하겠으며, 개선을 위한 피드백은 언제든지 환영합니다.
|
||||
|
||||
라이센스 측면에서, 간략한 [라이센스 및 기여자 동의서](../../LICENSE)를 읽어보는 시간을 가져주세요. 커뮤니티는 또한 [행동 강령](https://github.com/langgenius/.github/blob/main/CODE_OF_CONDUCT.md)을 준수합니다.
|
||||
라이센스 측면에서, 간략한 [라이센스 및 기여자 동의서](../LICENSE)를 읽어보는 시간을 가져주세요. 커뮤니티는 또한 [행동 강령](https://github.com/langgenius/.github/blob/main/CODE_OF_CONDUCT.md)을 준수합니다.
|
||||
|
||||
## 시작하기 전에
|
||||
|
||||
@ -6,7 +6,7 @@ Precisamos ser ágeis e entregar rapidamente considerando onde estamos, mas tamb
|
||||
|
||||
Este guia, como o próprio Dify, é um trabalho em constante evolução. Agradecemos muito a sua compreensão se às vezes ele ficar atrasado em relação ao projeto real, e damos as boas-vindas a qualquer feedback para que possamos melhorar.
|
||||
|
||||
Em termos de licenciamento, por favor, dedique um minuto para ler nosso breve [Acordo de Licença e Contribuidor](../../LICENSE). A comunidade também adere ao [código de conduta](https://github.com/langgenius/.github/blob/main/CODE_OF_CONDUCT.md).
|
||||
Em termos de licenciamento, por favor, dedique um minuto para ler nosso breve [Acordo de Licença e Contribuidor](../LICENSE). A comunidade também adere ao [código de conduta](https://github.com/langgenius/.github/blob/main/CODE_OF_CONDUCT.md).
|
||||
|
||||
## Antes de começar
|
||||
|
||||
@ -6,7 +6,7 @@ Bulunduğumuz noktada çevik olmamız ve hızlı hareket etmemiz gerekiyor, anca
|
||||
|
||||
Bu rehber, Dify'ın kendisi gibi, sürekli gelişen bir çalışmadır. Bazen gerçek projenin gerisinde kalırsa anlayışınız için çok minnettarız ve gelişmemize yardımcı olacak her türlü geri bildirimi memnuniyetle karşılıyoruz.
|
||||
|
||||
Lisanslama konusunda, lütfen kısa [Lisans ve Katkıda Bulunan Anlaşmamızı](../../LICENSE) okumak için bir dakikanızı ayırın. Topluluk ayrıca [davranış kurallarına](https://github.com/langgenius/.github/blob/main/CODE_OF_CONDUCT.md) da uyar.
|
||||
Lisanslama konusunda, lütfen kısa [Lisans ve Katkıda Bulunan Anlaşmamızı](../LICENSE) okumak için bir dakikanızı ayırın. Topluluk ayrıca [davranış kurallarına](https://github.com/langgenius/.github/blob/main/CODE_OF_CONDUCT.md) da uyar.
|
||||
|
||||
## Başlamadan Önce
|
||||
|
||||
@ -6,7 +6,7 @@
|
||||
|
||||
這份指南與 Dify 一樣，都在持續完善中。如果指南內容有落後於實際專案的情況，還請見諒，也歡迎提供改進建議。
|
||||
|
||||
關於授權部分，請花點時間閱讀我們簡短的[授權和貢獻者協議](../../LICENSE)。社群也需遵守[行為準則](https://github.com/langgenius/.github/blob/main/CODE_OF_CONDUCT.md)。
|
||||
關於授權部分，請花點時間閱讀我們簡短的[授權和貢獻者協議](../LICENSE)。社群也需遵守[行為準則](https://github.com/langgenius/.github/blob/main/CODE_OF_CONDUCT.md)。
|
||||
|
||||
## 開始之前
|
||||
|
||||
@ -6,7 +6,7 @@ Chúng tôi cần phải nhanh nhẹn và triển khai nhanh chóng, nhưng cũn
|
||||
|
||||
Hướng dẫn này, giống như Dify, đang được phát triển liên tục. Chúng tôi rất cảm kích sự thông cảm của bạn nếu đôi khi nó chưa theo kịp dự án thực tế, và hoan nghênh mọi phản hồi để cải thiện.
|
||||
|
||||
Về giấy phép, vui lòng dành chút thời gian đọc [Thỏa thuận Cấp phép và Người đóng góp](../../LICENSE) ngắn gọn của chúng tôi. Cộng đồng cũng tuân theo [quy tắc ứng xử](https://github.com/langgenius/.github/blob/main/CODE_OF_CONDUCT.md).
|
||||
Về giấy phép, vui lòng dành chút thời gian đọc [Thỏa thuận Cấp phép và Người đóng góp](../LICENSE) ngắn gọn của chúng tôi. Cộng đồng cũng tuân theo [quy tắc ứng xử](https://github.com/langgenius/.github/blob/main/CODE_OF_CONDUCT.md).
|
||||
|
||||
## Trước khi bắt đầu
|
||||
|
||||
1
Makefile
1
Makefile
@ -26,6 +26,7 @@ prepare-web:
|
||||
@echo "🌐 Setting up web environment..."
|
||||
@cp -n web/.env.example web/.env 2>/dev/null || echo "Web .env already exists"
|
||||
@cd web && pnpm install
|
||||
@cd web && pnpm build
|
||||
@echo "✅ Web environment prepared (not started)"
|
||||
|
||||
# Step 3: Prepare API environment
|
||||
|
||||
24
README.md
24
README.md
@ -40,18 +40,18 @@
|
||||
|
||||
<p align="center">
|
||||
<a href="./README.md"><img alt="README in English" src="https://img.shields.io/badge/English-d9d9d9"></a>
|
||||
<a href="./docs/zh-TW/README.md"><img alt="繁體中文文件" src="https://img.shields.io/badge/繁體中文-d9d9d9"></a>
|
||||
<a href="./docs/zh-CN/README.md"><img alt="简体中文文件" src="https://img.shields.io/badge/简体中文-d9d9d9"></a>
|
||||
<a href="./docs/ja-JP/README.md"><img alt="日本語のREADME" src="https://img.shields.io/badge/日本語-d9d9d9"></a>
|
||||
<a href="./docs/es-ES/README.md"><img alt="README en Español" src="https://img.shields.io/badge/Español-d9d9d9"></a>
|
||||
<a href="./docs/fr-FR/README.md"><img alt="README en Français" src="https://img.shields.io/badge/Français-d9d9d9"></a>
|
||||
<a href="./docs/tlh/README.md"><img alt="README tlhIngan Hol" src="https://img.shields.io/badge/Klingon-d9d9d9"></a>
|
||||
<a href="./docs/ko-KR/README.md"><img alt="README in Korean" src="https://img.shields.io/badge/한국어-d9d9d9"></a>
|
||||
<a href="./docs/ar-SA/README.md"><img alt="README بالعربية" src="https://img.shields.io/badge/العربية-d9d9d9"></a>
|
||||
<a href="./docs/tr-TR/README.md"><img alt="Türkçe README" src="https://img.shields.io/badge/Türkçe-d9d9d9"></a>
|
||||
<a href="./docs/vi-VN/README.md"><img alt="README Tiếng Việt" src="https://img.shields.io/badge/Ti%E1%BA%BFng%20Vi%E1%BB%87t-d9d9d9"></a>
|
||||
<a href="./docs/de-DE/README.md"><img alt="README in Deutsch" src="https://img.shields.io/badge/German-d9d9d9"></a>
|
||||
<a href="./docs/bn-BD/README.md"><img alt="README in বাংলা" src="https://img.shields.io/badge/বাংলা-d9d9d9"></a>
|
||||
<a href="./README/README_TW.md"><img alt="繁體中文文件" src="https://img.shields.io/badge/繁體中文-d9d9d9"></a>
|
||||
<a href="./README/README_CN.md"><img alt="简体中文版自述文件" src="https://img.shields.io/badge/简体中文-d9d9d9"></a>
|
||||
<a href="./README/README_JA.md"><img alt="日本語のREADME" src="https://img.shields.io/badge/日本語-d9d9d9"></a>
|
||||
<a href="./README/README_ES.md"><img alt="README en Español" src="https://img.shields.io/badge/Español-d9d9d9"></a>
|
||||
<a href="./README/README_FR.md"><img alt="README en Français" src="https://img.shields.io/badge/Français-d9d9d9"></a>
|
||||
<a href="./README/README_KL.md"><img alt="README tlhIngan Hol" src="https://img.shields.io/badge/Klingon-d9d9d9"></a>
|
||||
<a href="./README/README_KR.md"><img alt="README in Korean" src="https://img.shields.io/badge/한국어-d9d9d9"></a>
|
||||
<a href="./README/README_AR.md"><img alt="README بالعربية" src="https://img.shields.io/badge/العربية-d9d9d9"></a>
|
||||
<a href="./README/README_TR.md"><img alt="Türkçe README" src="https://img.shields.io/badge/Türkçe-d9d9d9"></a>
|
||||
<a href="./README/README_VI.md"><img alt="README Tiếng Việt" src="https://img.shields.io/badge/Ti%E1%BA%BFng%20Vi%E1%BB%87t-d9d9d9"></a>
|
||||
<a href="./README/README_DE.md"><img alt="README in Deutsch" src="https://img.shields.io/badge/German-d9d9d9"></a>
|
||||
<a href="./README/README_BN.md"><img alt="README in বাংলা" src="https://img.shields.io/badge/বাংলা-d9d9d9"></a>
|
||||
</p>
|
||||
|
||||
Dify is an open-source platform for developing LLM applications. Its intuitive interface combines agentic AI workflows, RAG pipelines, agent capabilities, model management, observability features, and more—allowing you to quickly move from prototype to production.
|
||||
|
||||
@ -1,4 +1,4 @@
|
||||
|
||||
|
||||
|
||||
<p align="center">
|
||||
<a href="https://cloud.dify.ai">Dify Cloud</a> ·
|
||||
@ -35,19 +35,17 @@
|
||||
</p>
|
||||
|
||||
<p align="center">
|
||||
<a href="../../README.md"><img alt="README in English" src="https://img.shields.io/badge/English-d9d9d9"></a>
|
||||
<a href="../zh-TW/README.md"><img alt="繁體中文文件" src="https://img.shields.io/badge/繁體中文-d9d9d9"></a>
|
||||
<a href="../zh-CN/README.md"><img alt="简体中文文件" src="https://img.shields.io/badge/简体中文-d9d9d9"></a>
|
||||
<a href="../ja-JP/README.md"><img alt="日本語のREADME" src="https://img.shields.io/badge/日本語-d9d9d9"></a>
|
||||
<a href="../es-ES/README.md"><img alt="README en Español" src="https://img.shields.io/badge/Español-d9d9d9"></a>
|
||||
<a href="../fr-FR/README.md"><img alt="README en Français" src="https://img.shields.io/badge/Français-d9d9d9"></a>
|
||||
<a href="../tlh/README.md"><img alt="README tlhIngan Hol" src="https://img.shields.io/badge/Klingon-d9d9d9"></a>
|
||||
<a href="../ko-KR/README.md"><img alt="README in Korean" src="https://img.shields.io/badge/한국어-d9d9d9"></a>
|
||||
<a href="../ar-SA/README.md"><img alt="README بالعربية" src="https://img.shields.io/badge/العربية-d9d9d9"></a>
|
||||
<a href="../tr-TR/README.md"><img alt="Türkçe README" src="https://img.shields.io/badge/Türkçe-d9d9d9"></a>
|
||||
<a href="../vi-VN/README.md"><img alt="README Tiếng Việt" src="https://img.shields.io/badge/Ti%E1%BA%BFng%20Vi%E1%BB%87t-d9d9d9"></a>
|
||||
<a href="../de-DE/README.md"><img alt="README in Deutsch" src="https://img.shields.io/badge/German-d9d9d9"></a>
|
||||
<a href="../bn-BD/README.md"><img alt="README in বাংলা" src="https://img.shields.io/badge/বাংলা-d9d9d9"></a>
|
||||
<a href="../README.md"><img alt="README in English" src="https://img.shields.io/badge/English-d9d9d9"></a>
|
||||
<a href="./README_CN.md"><img alt="简体中文版自述文件" src="https://img.shields.io/badge/简体中文-d9d9d9"></a>
|
||||
<a href="./README_JA.md"><img alt="日本語のREADME" src="https://img.shields.io/badge/日本語-d9d9d9"></a>
|
||||
<a href="./README_ES.md"><img alt="README en Español" src="https://img.shields.io/badge/Español-d9d9d9"></a>
|
||||
<a href="./README_FR.md"><img alt="README en Français" src="https://img.shields.io/badge/Français-d9d9d9"></a>
|
||||
<a href="./README_KL.md"><img alt="README tlhIngan Hol" src="https://img.shields.io/badge/Klingon-d9d9d9"></a>
|
||||
<a href="./README_KR.md"><img alt="README in Korean" src="https://img.shields.io/badge/한국어-d9d9d9"></a>
|
||||
<a href="./README_AR.md"><img alt="README بالعربية" src="https://img.shields.io/badge/العربية-d9d9d9"></a>
|
||||
<a href="./README_TR.md"><img alt="Türkçe README" src="https://img.shields.io/badge/Türkçe-d9d9d9"></a>
|
||||
<a href="./README_VI.md"><img alt="README Tiếng Việt" src="https://img.shields.io/badge/Ti%E1%BA%BFng%20Vi%E1%BB%87t-d9d9d9"></a>
|
||||
<a href="./README_BN.md"><img alt="README in বাংলা" src="https://img.shields.io/badge/বাংলা-d9d9d9"></a>
|
||||
</p>
|
||||
|
||||
<div style="text-align: right;">
|
||||
@ -99,7 +97,7 @@
|
||||
|
||||
</br>
|
||||
|
||||
أسهل طريقة لبدء تشغيل خادم Dify هي تشغيل ملف [docker-compose.yml](../../docker/docker-compose.yaml) الخاص بنا. قبل تشغيل أمر التثبيت، تأكد من تثبيت [Docker](https://docs.docker.com/get-docker/) و [Docker Compose](https://docs.docker.com/compose/install/) على جهازك:
|
||||
أسهل طريقة لبدء تشغيل خادم Dify هي تشغيل ملف [docker-compose.yml](docker/docker-compose.yaml) الخاص بنا. قبل تشغيل أمر التثبيت، تأكد من تثبيت [Docker](https://docs.docker.com/get-docker/) و [Docker Compose](https://docs.docker.com/compose/install/) على جهازك:
|
||||
|
||||
```bash
|
||||
cd docker
|
||||
@ -113,7 +111,7 @@ docker compose up -d
|
||||
|
||||
## الخطوات التالية
|
||||
|
||||
إذا كنت بحاجة إلى تخصيص الإعدادات، فيرجى الرجوع إلى التعليقات في ملف [.env.example](../../docker/.env.example) وتحديث القيم المقابلة في ملف `.env`. بالإضافة إلى ذلك، قد تحتاج إلى إجراء تعديلات على ملف `docker-compose.yaml` نفسه، مثل تغيير إصدارات الصور أو تعيينات المنافذ أو نقاط تحميل وحدات التخزين، بناءً على بيئة النشر ومتطلباتك الخاصة. بعد إجراء أي تغييرات، يرجى إعادة تشغيل `docker-compose up -d`. يمكنك العثور على قائمة كاملة بمتغيرات البيئة المتاحة [هنا](https://docs.dify.ai/getting-started/install-self-hosted/environments).
|
||||
إذا كنت بحاجة إلى تخصيص الإعدادات، فيرجى الرجوع إلى التعليقات في ملف [.env.example](docker/.env.example) وتحديث القيم المقابلة في ملف `.env`. بالإضافة إلى ذلك، قد تحتاج إلى إجراء تعديلات على ملف `docker-compose.yaml` نفسه، مثل تغيير إصدارات الصور أو تعيينات المنافذ أو نقاط تحميل وحدات التخزين، بناءً على بيئة النشر ومتطلباتك الخاصة. بعد إجراء أي تغييرات، يرجى إعادة تشغيل `docker-compose up -d`. يمكنك العثور على قائمة كاملة بمتغيرات البيئة المتاحة [هنا](https://docs.dify.ai/getting-started/install-self-hosted/environments).
|
||||
|
||||
يوجد مجتمع خاص بـ [Helm Charts](https://helm.sh/) وملفات YAML التي تسمح بتنفيذ Dify على Kubernetes للنظام من الإيجابيات العلوية.
|
||||
|
||||
@ -187,4 +185,12 @@ docker compose up -d
|
||||
|
||||
## الرخصة
|
||||
|
||||
هذا المستودع متاح تحت [رخصة البرنامج الحر Dify](../../LICENSE)، والتي تعتبر بشكل أساسي Apache 2.0 مع بعض القيود الإضافية.
|
||||
هذا المستودع متاح تحت [رخصة البرنامج الحر Dify](../LICENSE)، والتي تعتبر بشكل أساسي Apache 2.0 مع بعض القيود الإضافية.
|
||||
|
||||
## الكشف عن الأمان
|
||||
|
||||
لحماية خصوصيتك، يرجى تجنب نشر مشكلات الأمان على GitHub. بدلاً من ذلك، أرسل أسئلتك إلى <security@dify.ai> وسنقدم لك إجابة أكثر تفصيلاً.
|
||||
|
||||
## الرخصة
|
||||
|
||||
هذا المستودع متاح تحت [رخصة البرنامج الحر Dify](../LICENSE)، والتي تعتبر بشكل أساسي Apache 2.0 مع بعض القيود الإضافية.
|
||||
@ -1,4 +1,4 @@
|
||||
|
||||
|
||||
|
||||
<p align="center">
|
||||
📌 <a href="https://dify.ai/blog/introducing-dify-workflow-file-upload-a-demo-on-ai-podcast">ডিফাই ওয়ার্কফ্লো ফাইল আপলোড পরিচিতি: গুগল নোটবুক-এলএম পডকাস্ট পুনর্নির্মাণ</a>
|
||||
@ -39,19 +39,18 @@
|
||||
</p>
|
||||
|
||||
<p align="center">
|
||||
<a href="../../README.md"><img alt="README in English" src="https://img.shields.io/badge/English-d9d9d9"></a>
|
||||
<a href="../zh-TW/README.md"><img alt="繁體中文文件" src="https://img.shields.io/badge/繁體中文-d9d9d9"></a>
|
||||
<a href="../zh-CN/README.md"><img alt="简体中文文件" src="https://img.shields.io/badge/简体中文-d9d9d9"></a>
|
||||
<a href="../ja-JP/README.md"><img alt="日本語のREADME" src="https://img.shields.io/badge/日本語-d9d9d9"></a>
|
||||
<a href="../es-ES/README.md"><img alt="README en Español" src="https://img.shields.io/badge/Español-d9d9d9"></a>
|
||||
<a href="../fr-FR/README.md"><img alt="README en Français" src="https://img.shields.io/badge/Français-d9d9d9"></a>
|
||||
<a href="../tlh/README.md"><img alt="README tlhIngan Hol" src="https://img.shields.io/badge/Klingon-d9d9d9"></a>
|
||||
<a href="../ko-KR/README.md"><img alt="README in Korean" src="https://img.shields.io/badge/한국어-d9d9d9"></a>
|
||||
<a href="../ar-SA/README.md"><img alt="README بالعربية" src="https://img.shields.io/badge/العربية-d9d9d9"></a>
|
||||
<a href="../tr-TR/README.md"><img alt="Türkçe README" src="https://img.shields.io/badge/Türkçe-d9d9d9"></a>
|
||||
<a href="../vi-VN/README.md"><img alt="README Tiếng Việt" src="https://img.shields.io/badge/Ti%E1%BA%BFng%20Vi%E1%BB%87t-d9d9d9"></a>
|
||||
<a href="../de-DE/README.md"><img alt="README in Deutsch" src="https://img.shields.io/badge/German-d9d9d9"></a>
|
||||
<a href="../bn-BD/README.md"><img alt="README in বাংলা" src="https://img.shields.io/badge/বাংলা-d9d9d9"></a>
|
||||
<a href="../README.md"><img alt="README in English" src="https://img.shields.io/badge/English-d9d9d9"></a>
|
||||
<a href="./README_CN.md"><img alt="简体中文版自述文件" src="https://img.shields.io/badge/简体中文-d9d9d9"></a>
|
||||
<a href="./README_JA.md"><img alt="日本語のREADME" src="https://img.shields.io/badge/日本語-d9d9d9"></a>
|
||||
<a href="./README_ES.md"><img alt="README en Español" src="https://img.shields.io/badge/Español-d9d9d9"></a>
|
||||
<a href="./README_FR.md"><img alt="README en Français" src="https://img.shields.io/badge/Français-d9d9d9"></a>
|
||||
<a href="./README_KL.md"><img alt="README tlhIngan Hol" src="https://img.shields.io/badge/Klingon-d9d9d9"></a>
|
||||
<a href="./README_KR.md"><img alt="README in Korean" src="https://img.shields.io/badge/한국어-d9d9d9"></a>
|
||||
<a href="./README_AR.md"><img alt="README بالعربية" src="https://img.shields.io/badge/العربية-d9d9d9"></a>
|
||||
<a href="./README_TR.md"><img alt="Türkçe README" src="https://img.shields.io/badge/Türkçe-d9d9d9"></a>
|
||||
<a href="./README_VI.md"><img alt="README Tiếng Việt" src="https://img.shields.io/badge/Ti%E1%BA%BFng%20Vi%E1%BB%87t-d9d9d9"></a>
|
||||
<a href="./README_DE.md"><img alt="README in Deutsch" src="https://img.shields.io/badge/German-d9d9d9"></a>
|
||||
<a href="./README_BN.md"><img alt="README in বাংলা" src="https://img.shields.io/badge/বাংলা-d9d9d9"></a>
|
||||
</p>
|
||||
|
||||
ডিফাই একটি ওপেন-সোর্স LLM অ্যাপ ডেভেলপমেন্ট প্ল্যাটফর্ম। এটি ইন্টুইটিভ ইন্টারফেস, এজেন্টিক AI ওয়ার্কফ্লো, RAG পাইপলাইন, এজেন্ট ক্যাপাবিলিটি, মডেল ম্যানেজমেন্ট, মনিটরিং সুবিধা এবং আরও অনেক কিছু একত্রিত করে, যা দ্রুত প্রোটোটাইপ থেকে প্রোডাকশন পর্যন্ত নিয়ে যেতে সহায়তা করে।
|
||||
@ -65,7 +64,7 @@
|
||||
|
||||
</br>
|
||||
|
||||
ডিফাই সার্ভার চালু করার সবচেয়ে সহজ উপায় [docker compose](../../docker/docker-compose.yaml) মাধ্যমে। নিম্নলিখিত কমান্ডগুলো ব্যবহার করে ডিফাই চালানোর আগে, নিশ্চিত করুন যে আপনার মেশিনে [Docker](https://docs.docker.com/get-docker/) এবং [Docker Compose](https://docs.docker.com/compose/install/) ইনস্টল করা আছে :
|
||||
ডিফাই সার্ভার চালু করার সবচেয়ে সহজ উপায় [docker compose](docker/docker-compose.yaml) মাধ্যমে। নিম্নলিখিত কমান্ডগুলো ব্যবহার করে ডিফাই চালানোর আগে, নিশ্চিত করুন যে আপনার মেশিনে [Docker](https://docs.docker.com/get-docker/) এবং [Docker Compose](https://docs.docker.com/compose/install/) ইনস্টল করা আছে :
|
||||
|
||||
```bash
|
||||
cd dify
|
||||
@ -129,7 +128,7 @@ GitHub-এ ডিফাইকে স্টার দিয়ে রাখুন
|
||||
|
||||
## Advanced Setup
|
||||
|
||||
যদি আপনার কনফিগারেশনটি কাস্টমাইজ করার প্রয়োজন হয়, তাহলে অনুগ্রহ করে আমাদের [.env.example](../../docker/.env.example) ফাইল দেখুন এবং আপনার `.env` ফাইলে সংশ্লিষ্ট মানগুলি আপডেট করুন। এছাড়াও, আপনার নির্দিষ্ট এনভায়রনমেন্ট এবং প্রয়োজনীয়তার উপর ভিত্তি করে আপনাকে `docker-compose.yaml` ফাইলে সমন্বয় করতে হতে পারে, যেমন ইমেজ ভার্সন পরিবর্তন করা, পোর্ট ম্যাপিং করা, অথবা ভলিউম মাউন্ট করা।
|
||||
যদি আপনার কনফিগারেশনটি কাস্টমাইজ করার প্রয়োজন হয়, তাহলে অনুগ্রহ করে আমাদের [.env.example](docker/.env.example) ফাইল দেখুন এবং আপনার `.env` ফাইলে সংশ্লিষ্ট মানগুলি আপডেট করুন। এছাড়াও, আপনার নির্দিষ্ট এনভায়রনমেন্ট এবং প্রয়োজনীয়তার উপর ভিত্তি করে আপনাকে `docker-compose.yaml` ফাইলে সমন্বয় করতে হতে পারে, যেমন ইমেজ ভার্সন পরিবর্তন করা, পোর্ট ম্যাপিং করা, অথবা ভলিউম মাউন্ট করা।
|
||||
যেকোনো পরিবর্তন করার পর, অনুগ্রহ করে `docker-compose up -d` পুনরায় চালান। ভেরিয়েবলের সম্পূর্ণ তালিকা [এখানে] (https://docs.dify.ai/getting-started/install-self-hosted/environments) খুঁজে পেতে পারেন।
|
||||
|
||||
যদি আপনি একটি হাইলি এভেইলেবল সেটআপ কনফিগার করতে চান, তাহলে কমিউনিটি [Helm Charts](https://helm.sh/) এবং YAML ফাইল রয়েছে যা Dify কে Kubernetes-এ ডিপ্লয় করার প্রক্রিয়া বর্ণনা করে।
|
||||
@ -176,7 +175,7 @@ GitHub-এ ডিফাইকে স্টার দিয়ে রাখুন
|
||||
|
||||
## Contributing
|
||||
|
||||
যারা কোড অবদান রাখতে চান, তাদের জন্য আমাদের [অবদান নির্দেশিকা](https://github.com/langgenius/dify/blob/main/CONTRIBUTING.md) দেখুন।
|
||||
যারা কোড অবদান রাখতে চান, তাদের জন্য আমাদের [অবদান নির্দেশিকা] দেখুন (https://github.com/langgenius/dify/blob/main/CONTRIBUTING.md)।
|
||||
একই সাথে, সোশ্যাল মিডিয়া এবং ইভেন্ট এবং কনফারেন্সে এটি শেয়ার করে Dify কে সমর্থন করুন।
|
||||
|
||||
> আমরা ম্যান্ডারিন বা ইংরেজি ছাড়া অন্য ভাষায় Dify অনুবাদ করতে সাহায্য করার জন্য অবদানকারীদের খুঁজছি। আপনি যদি সাহায্য করতে আগ্রহী হন, তাহলে আরও তথ্যের জন্য [i18n README](https://github.com/langgenius/dify/blob/main/web/i18n-config/README.md) দেখুন এবং আমাদের [ডিসকর্ড কমিউনিটি সার্ভার](https://discord.gg/8Tpq4AcN9c) এর `গ্লোবাল-ইউজারস` চ্যানেলে আমাদের একটি মন্তব্য করুন।
|
||||
@ -204,4 +203,4 @@ GitHub-এ ডিফাইকে স্টার দিয়ে রাখুন
|
||||
|
||||
## লাইসেন্স
|
||||
|
||||
এই রিপোজিটরিটি [ডিফাই ওপেন সোর্স লাইসেন্স](../../LICENSE) এর অধিনে , যা মূলত অ্যাপাচি ২.০, তবে কিছু অতিরিক্ত বিধিনিষেধ রয়েছে।
|
||||
এই রিপোজিটরিটি [ডিফাই ওপেন সোর্স লাইসেন্স](../LICENSE) এর অধিনে , যা মূলত অ্যাপাচি ২.০, তবে কিছু অতিরিক্ত বিধিনিষেধ রয়েছে।
|
||||
@ -1,4 +1,4 @@
|
||||
|
||||
|
||||
|
||||
<div align="center">
|
||||
<a href="https://cloud.dify.ai">Dify 云服务</a> ·
|
||||
@ -35,19 +35,17 @@
|
||||
</p>
|
||||
|
||||
<div align="center">
|
||||
<a href="../../README.md"><img alt="README in English" src="https://img.shields.io/badge/English-d9d9d9"></a>
|
||||
<a href="../zh-TW/README.md"><img alt="繁體中文文件" src="https://img.shields.io/badge/繁體中文-d9d9d9"></a>
|
||||
<a href="../zh-CN/README.md"><img alt="简体中文文件" src="https://img.shields.io/badge/简体中文-d9d9d9"></a>
|
||||
<a href="../ja-JP/README.md"><img alt="日本語のREADME" src="https://img.shields.io/badge/日本語-d9d9d9"></a>
|
||||
<a href="../es-ES/README.md"><img alt="README en Español" src="https://img.shields.io/badge/Español-d9d9d9"></a>
|
||||
<a href="../fr-FR/README.md"><img alt="README en Français" src="https://img.shields.io/badge/Français-d9d9d9"></a>
|
||||
<a href="../tlh/README.md"><img alt="README tlhIngan Hol" src="https://img.shields.io/badge/Klingon-d9d9d9"></a>
|
||||
<a href="../ko-KR/README.md"><img alt="README in Korean" src="https://img.shields.io/badge/한국어-d9d9d9"></a>
|
||||
<a href="../ar-SA/README.md"><img alt="README بالعربية" src="https://img.shields.io/badge/العربية-d9d9d9"></a>
|
||||
<a href="../tr-TR/README.md"><img alt="Türkçe README" src="https://img.shields.io/badge/Türkçe-d9d9d9"></a>
|
||||
<a href="../vi-VN/README.md"><img alt="README Tiếng Việt" src="https://img.shields.io/badge/Ti%E1%BA%BFng%20Vi%E1%BB%87t-d9d9d9"></a>
|
||||
<a href="../de-DE/README.md"><img alt="README in Deutsch" src="https://img.shields.io/badge/German-d9d9d9"></a>
|
||||
<a href="../bn-BD/README.md"><img alt="README in বাংলা" src="https://img.shields.io/badge/বাংলা-d9d9d9"></a>
|
||||
<a href="../README.md"><img alt="README in English" src="https://img.shields.io/badge/English-d9d9d9"></a>
|
||||
<a href="./README_CN.md"><img alt="简体中文版自述文件" src="https://img.shields.io/badge/简体中文-d9d9d9"></a>
|
||||
<a href="./README_JA.md"><img alt="日本語のREADME" src="https://img.shields.io/badge/日本語-d9d9d9"></a>
|
||||
<a href="./README_ES.md"><img alt="README en Español" src="https://img.shields.io/badge/Español-d9d9d9"></a>
|
||||
<a href="./README_FR.md"><img alt="README en Français" src="https://img.shields.io/badge/Français-d9d9d9"></a>
|
||||
<a href="./README_KL.md"><img alt="README tlhIngan Hol" src="https://img.shields.io/badge/Klingon-d9d9d9"></a>
|
||||
<a href="./README_KR.md"><img alt="README in Korean" src="https://img.shields.io/badge/한국어-d9d9d9"></a>
|
||||
<a href="./README_AR.md"><img alt="README بالعربية" src="https://img.shields.io/badge/العربية-d9d9d9"></a>
|
||||
<a href="./README_TR.md"><img alt="Türkçe README" src="https://img.shields.io/badge/Türkçe-d9d9d9"></a>
|
||||
<a href="./README_VI.md"><img alt="README Tiếng Việt" src="https://img.shields.io/badge/Ti%E1%BA%BFng%20Vi%E1%BB%87t-d9d9d9"></a>
|
||||
<a href="./README_BN.md"><img alt="README in বাংলা" src="https://img.shields.io/badge/বাংলা-d9d9d9"></a>
|
||||
</div>
|
||||
|
||||
#
|
||||
@ -113,7 +111,7 @@ Dify 是一个开源的 LLM 应用开发平台。其直观的界面结合了 AI
|
||||
|
||||
### 快速启动
|
||||
|
||||
启动 Dify 服务器的最简单方法是运行我们的 [docker-compose.yml](../../docker/docker-compose.yaml) 文件。在运行安装命令之前，请确保您的机器上安装了 [Docker](https://docs.docker.com/get-docker/) 和 [Docker Compose](https://docs.docker.com/compose/install/)：
|
||||
启动 Dify 服务器的最简单方法是运行我们的 [docker-compose.yml](docker/docker-compose.yaml) 文件。在运行安装命令之前，请确保您的机器上安装了 [Docker](https://docs.docker.com/get-docker/) 和 [Docker Compose](https://docs.docker.com/compose/install/)：
|
||||
|
||||
```bash
|
||||
cd docker
|
||||
@ -125,7 +123,7 @@ docker compose up -d
|
||||
|
||||
### 自定义配置
|
||||
|
||||
如果您需要自定义配置，请参考 [.env.example](../../docker/.env.example) 文件中的注释，并更新 `.env` 文件中对应的值。此外，您可能需要根据您的具体部署环境和需求对 `docker-compose.yaml` 文件本身进行调整，例如更改镜像版本、端口映射或卷挂载。完成任何更改后，请重新运行 `docker-compose up -d`。您可以在[此处](https://docs.dify.ai/getting-started/install-self-hosted/environments)找到可用环境变量的完整列表。
|
||||
如果您需要自定义配置，请参考 [.env.example](docker/.env.example) 文件中的注释，并更新 `.env` 文件中对应的值。此外，您可能需要根据您的具体部署环境和需求对 `docker-compose.yaml` 文件本身进行调整，例如更改镜像版本、端口映射或卷挂载。完成任何更改后，请重新运行 `docker-compose up -d`。您可以在[此处](https://docs.dify.ai/getting-started/install-self-hosted/environments)找到可用环境变量的完整列表。
|
||||
|
||||
#### 使用 Helm Chart 或 Kubernetes 资源清单（YAML）部署
|
||||
|
||||
@ -182,7 +180,7 @@ docker compose up -d
|
||||
|
||||
## Contributing
|
||||
|
||||
对于那些想要贡献代码的人，请参阅我们的[贡献指南](./CONTRIBUTING.md)。
|
||||
对于那些想要贡献代码的人，请参阅我们的[贡献指南](https://github.com/langgenius/dify/blob/main/CONTRIBUTING/CONTRIBUTING_CN.md)。
|
||||
同时，请考虑通过社交媒体、活动和会议来支持 Dify 的分享。
|
||||
|
||||
> 我们正在寻找贡献者来帮助将 Dify 翻译成除了中文和英文之外的其他语言。如果您有兴趣帮助，请参阅我们的[i18n README](https://github.com/langgenius/dify/blob/main/web/i18n-config/README.md)获取更多信息，并在我们的[Discord 社区服务器](https://discord.gg/8Tpq4AcN9c)的`global-users`频道中留言。
|
||||
@ -198,7 +196,7 @@ docker compose up -d
|
||||
我们欢迎您为 Dify 做出贡献，以帮助改善 Dify。包括：提交代码、问题、新想法，或分享您基于 Dify 创建的有趣且有用的 AI 应用程序。同时，我们也欢迎您在不同的活动、会议和社交媒体上分享 Dify。
|
||||
|
||||
- [GitHub Discussion](https://github.com/langgenius/dify/discussions). 👉：分享您的应用程序并与社区交流。
|
||||
- [GitHub Issues](https://github.com/langgenius/dify/issues)。👉：使用 Dify.AI 时遇到的错误和问题，请参阅[贡献指南](./CONTRIBUTING.md)。
|
||||
- [GitHub Issues](https://github.com/langgenius/dify/issues)。👉：使用 Dify.AI 时遇到的错误和问题，请参阅[贡献指南](../CONTRIBUTING.md)。
|
||||
- [电子邮件支持](mailto:hello@dify.ai?subject=%5BGitHub%5DQuestions%20About%20Dify)。👉：关于使用 Dify.AI 的问题。
|
||||
- [Discord](https://discord.gg/FngNHpbcY7)。👉：分享您的应用程序并与社区交流。
|
||||
- [X(Twitter)](https://twitter.com/dify_ai)。👉：分享您的应用程序并与社区交流。
|
||||
@ -210,4 +208,4 @@ docker compose up -d
|
||||
|
||||
## License
|
||||
|
||||
本仓库遵循 [Dify Open Source License](../../LICENSE) 开源协议，该许可证本质上是 Apache 2.0，但有一些额外的限制。
|
||||
本仓库遵循 [Dify Open Source License](../LICENSE) 开源协议，该许可证本质上是 Apache 2.0，但有一些额外的限制。
|
||||
@ -1,4 +1,4 @@
|
||||
|
||||
|
||||
|
||||
<p align="center">
|
||||
📌 <a href="https://dify.ai/blog/introducing-dify-workflow-file-upload-a-demo-on-ai-podcast">Einführung in Dify Workflow File Upload: Google NotebookLM Podcast nachbilden</a>
|
||||
@ -39,19 +39,18 @@
|
||||
</p>
|
||||
|
||||
<p align="center">
|
||||
<a href="../../README.md"><img alt="README in English" src="https://img.shields.io/badge/English-d9d9d9"></a>
|
||||
<a href="../zh-TW/README.md"><img alt="繁體中文文件" src="https://img.shields.io/badge/繁體中文-d9d9d9"></a>
|
||||
<a href="../zh-CN/README.md"><img alt="简体中文文件" src="https://img.shields.io/badge/简体中文-d9d9d9"></a>
|
||||
<a href="../ja-JP/README.md"><img alt="日本語のREADME" src="https://img.shields.io/badge/日本語-d9d9d9"></a>
|
||||
<a href="../es-ES/README.md"><img alt="README en Español" src="https://img.shields.io/badge/Español-d9d9d9"></a>
|
||||
<a href="../fr-FR/README.md"><img alt="README en Français" src="https://img.shields.io/badge/Français-d9d9d9"></a>
|
||||
<a href="../tlh/README.md"><img alt="README tlhIngan Hol" src="https://img.shields.io/badge/Klingon-d9d9d9"></a>
|
||||
<a href="../ko-KR/README.md"><img alt="README in Korean" src="https://img.shields.io/badge/한국어-d9d9d9"></a>
|
||||
<a href="../ar-SA/README.md"><img alt="README بالعربية" src="https://img.shields.io/badge/العربية-d9d9d9"></a>
|
||||
<a href="../tr-TR/README.md"><img alt="Türkçe README" src="https://img.shields.io/badge/Türkçe-d9d9d9"></a>
|
||||
<a href="../vi-VN/README.md"><img alt="README Tiếng Việt" src="https://img.shields.io/badge/Ti%E1%BA%BFng%20Vi%E1%BB%87t-d9d9d9"></a>
|
||||
<a href="../de-DE/README.md"><img alt="README in Deutsch" src="https://img.shields.io/badge/German-d9d9d9"></a>
|
||||
<a href="../bn-BD/README.md"><img alt="README in বাংলা" src="https://img.shields.io/badge/বাংলা-d9d9d9"></a>
|
||||
<a href="../README.md"><img alt="README in English" src="https://img.shields.io/badge/English-d9d9d9"></a>
|
||||
<a href="./README_CN.md"><img alt="简体中文版自述文件" src="https://img.shields.io/badge/简体中文-d9d9d9"></a>
|
||||
<a href="./README_JA.md"><img alt="日本語のREADME" src="https://img.shields.io/badge/日本語-d9d9d9"></a>
|
||||
<a href="./README_ES.md"><img alt="README en Español" src="https://img.shields.io/badge/Español-d9d9d9"></a>
|
||||
<a href="./README_FR.md"><img alt="README en Français" src="https://img.shields.io/badge/Français-d9d9d9"></a>
|
||||
<a href="./README_KL.md"><img alt="README tlhIngan Hol" src="https://img.shields.io/badge/Klingon-d9d9d9"></a>
|
||||
<a href="./README_KR.md"><img alt="README in Korean" src="https://img.shields.io/badge/한국어-d9d9d9"></a>
|
||||
<a href="./README_AR.md"><img alt="README بالعربية" src="https://img.shields.io/badge/العربية-d9d9d9"></a>
|
||||
<a href="./README_TR.md"><img alt="Türkçe README" src="https://img.shields.io/badge/Türkçe-d9d9d9"></a>
|
||||
<a href="./README_VI.md"><img alt="README Tiếng Việt" src="https://img.shields.io/badge/Ti%E1%BA%BFng%20Vi%E1%BB%87t-d9d9d9"></a>
|
||||
<a href="./README_DE.md"><img alt="README in Deutsch" src="https://img.shields.io/badge/German-d9d9d9"></a>
|
||||
<a href="./README_BN.md"><img alt="README in বাংলা" src="https://img.shields.io/badge/বাংলা-d9d9d9"></a>
|
||||
</p>
|
||||
|
||||
Dify ist eine Open-Source-Plattform zur Entwicklung von LLM-Anwendungen. Ihre intuitive Benutzeroberfläche vereint agentenbasierte KI-Workflows, RAG-Pipelines, Agentenfunktionen, Modellverwaltung, Überwachungsfunktionen und mehr, sodass Sie schnell von einem Prototyp in die Produktion übergehen können.
|
||||
@ -65,7 +64,7 @@ Dify ist eine Open-Source-Plattform zur Entwicklung von LLM-Anwendungen. Ihre in
|
||||
|
||||
</br>
|
||||
|
||||
Der einfachste Weg, den Dify-Server zu starten, ist über [docker compose](../../docker/docker-compose.yaml). Stellen Sie vor dem Ausführen von Dify mit den folgenden Befehlen sicher, dass [Docker](https://docs.docker.com/get-docker/) und [Docker Compose](https://docs.docker.com/compose/install/) auf Ihrem System installiert sind:
|
||||
Der einfachste Weg, den Dify-Server zu starten, ist über [docker compose](docker/docker-compose.yaml). Stellen Sie vor dem Ausführen von Dify mit den folgenden Befehlen sicher, dass [Docker](https://docs.docker.com/get-docker/) und [Docker Compose](https://docs.docker.com/compose/install/) auf Ihrem System installiert sind:
|
||||
|
||||
```bash
|
||||
cd dify
|
||||
@ -128,7 +127,7 @@ Star Dify auf GitHub und lassen Sie sich sofort über neue Releases benachrichti
|
||||
|
||||
## Erweiterte Einstellungen
|
||||
|
||||
Falls Sie die Konfiguration anpassen müssen, lesen Sie bitte die Kommentare in unserer [.env.example](../../docker/.env.example)-Datei und aktualisieren Sie die entsprechenden Werte in Ihrer `.env`-Datei. Zusätzlich müssen Sie eventuell Anpassungen an der `docker-compose.yaml`-Datei vornehmen, wie zum Beispiel das Ändern von Image-Versionen, Portzuordnungen oder Volumen-Mounts, je nach Ihrer spezifischen Einsatzumgebung und Ihren Anforderungen. Nachdem Sie Änderungen vorgenommen haben, starten Sie `docker-compose up -d` erneut. Eine vollständige Liste der verfügbaren Umgebungsvariablen finden Sie [hier](https://docs.dify.ai/getting-started/install-self-hosted/environments).
|
||||
Falls Sie die Konfiguration anpassen müssen, lesen Sie bitte die Kommentare in unserer [.env.example](docker/.env.example)-Datei und aktualisieren Sie die entsprechenden Werte in Ihrer `.env`-Datei. Zusätzlich müssen Sie eventuell Anpassungen an der `docker-compose.yaml`-Datei vornehmen, wie zum Beispiel das Ändern von Image-Versionen, Portzuordnungen oder Volumen-Mounts, je nach Ihrer spezifischen Einsatzumgebung und Ihren Anforderungen. Nachdem Sie Änderungen vorgenommen haben, starten Sie `docker-compose up -d` erneut. Eine vollständige Liste der verfügbaren Umgebungsvariablen finden Sie [hier](https://docs.dify.ai/getting-started/install-self-hosted/environments).
|
||||
|
||||
Falls Sie eine hochverfügbare Konfiguration einrichten möchten, gibt es von der Community bereitgestellte [Helm Charts](https://helm.sh/) und YAML-Dateien, die es ermöglichen, Dify auf Kubernetes bereitzustellen.
|
||||
|
||||
@ -174,14 +173,14 @@ Stellen Sie Dify mit einem Klick in AKS bereit, indem Sie [Azure Devops Pipeline
|
||||
|
||||
## Contributing
|
||||
|
||||
Falls Sie Code beitragen möchten, lesen Sie bitte unseren [Contribution Guide](./CONTRIBUTING.md). Gleichzeitig bitten wir Sie, Dify zu unterstützen, indem Sie es in den sozialen Medien teilen und auf Veranstaltungen und Konferenzen präsentieren.
|
||||
Falls Sie Code beitragen möchten, lesen Sie bitte unseren [Contribution Guide](https://github.com/langgenius/dify/blob/main/CONTRIBUTING/CONTRIBUTING_DE.md). Gleichzeitig bitten wir Sie, Dify zu unterstützen, indem Sie es in den sozialen Medien teilen und auf Veranstaltungen und Konferenzen präsentieren.
|
||||
|
||||
> Wir suchen Mitwirkende, die dabei helfen, Dify in weitere Sprachen zu übersetzen – außer Mandarin oder Englisch. Wenn Sie Interesse an einer Mitarbeit haben, lesen Sie bitte die [i18n README](https://github.com/langgenius/dify/blob/main/web/i18n-config/README.md) für weitere Informationen und hinterlassen Sie einen Kommentar im `global-users`-Kanal unseres [Discord Community Servers](https://discord.gg/8Tpq4AcN9c).
|
||||
|
||||
## Gemeinschaft & Kontakt
|
||||
|
||||
- [GitHub Discussion](https://github.com/langgenius/dify/discussions). Am besten geeignet für: den Austausch von Feedback und das Stellen von Fragen.
|
||||
- [GitHub Issues](https://github.com/langgenius/dify/issues). Am besten für: Fehler, auf die Sie bei der Verwendung von Dify.AI stoßen, und Funktionsvorschläge. Siehe unseren [Contribution Guide](./CONTRIBUTING.md).
|
||||
- [GitHub Issues](https://github.com/langgenius/dify/issues). Am besten für: Fehler, auf die Sie bei der Verwendung von Dify.AI stoßen, und Funktionsvorschläge. Siehe unseren [Contribution Guide](https://github.com/langgenius/dify/blob/main/CONTRIBUTING.md).
|
||||
- [Discord](https://discord.gg/FngNHpbcY7). Am besten geeignet für: den Austausch von Bewerbungen und den Austausch mit der Community.
|
||||
- [X(Twitter)](https://twitter.com/dify_ai). Am besten geeignet für: den Austausch von Bewerbungen und den Austausch mit der Community.
|
||||
|
||||
@ -201,4 +200,4 @@ Um Ihre Privatsphäre zu schützen, vermeiden Sie es bitte, Sicherheitsprobleme
|
||||
|
||||
## Lizenz
|
||||
|
||||
Dieses Repository steht unter der [Dify Open Source License](../../LICENSE), die im Wesentlichen Apache 2.0 mit einigen zusätzlichen Einschränkungen ist.
|
||||
Dieses Repository steht unter der [Dify Open Source License](../LICENSE), die im Wesentlichen Apache 2.0 mit einigen zusätzlichen Einschränkungen ist.
|
||||
@ -1,4 +1,4 @@
|
||||
|
||||
|
||||
|
||||
<p align="center">
|
||||
<a href="https://cloud.dify.ai">Dify Cloud</a> ·
|
||||
@ -35,19 +35,17 @@
|
||||
</p>
|
||||
|
||||
<p align="center">
|
||||
<a href="../../README.md"><img alt="README in English" src="https://img.shields.io/badge/English-d9d9d9"></a>
|
||||
<a href="../zh-TW/README.md"><img alt="繁體中文文件" src="https://img.shields.io/badge/繁體中文-d9d9d9"></a>
|
||||
<a href="../zh-CN/README.md"><img alt="简体中文文件" src="https://img.shields.io/badge/简体中文-d9d9d9"></a>
|
||||
<a href="../ja-JP/README.md"><img alt="日本語のREADME" src="https://img.shields.io/badge/日本語-d9d9d9"></a>
|
||||
<a href="../es-ES/README.md"><img alt="README en Español" src="https://img.shields.io/badge/Español-d9d9d9"></a>
|
||||
<a href="../fr-FR/README.md"><img alt="README en Français" src="https://img.shields.io/badge/Français-d9d9d9"></a>
|
||||
<a href="../tlh/README.md"><img alt="README tlhIngan Hol" src="https://img.shields.io/badge/Klingon-d9d9d9"></a>
|
||||
<a href="../ko-KR/README.md"><img alt="README in Korean" src="https://img.shields.io/badge/한국어-d9d9d9"></a>
|
||||
<a href="../ar-SA/README.md"><img alt="README بالعربية" src="https://img.shields.io/badge/العربية-d9d9d9"></a>
|
||||
<a href="../tr-TR/README.md"><img alt="Türkçe README" src="https://img.shields.io/badge/Türkçe-d9d9d9"></a>
|
||||
<a href="../vi-VN/README.md"><img alt="README Tiếng Việt" src="https://img.shields.io/badge/Ti%E1%BA%BFng%20Vi%E1%BB%87t-d9d9d9"></a>
|
||||
<a href="../de-DE/README.md"><img alt="README in Deutsch" src="https://img.shields.io/badge/German-d9d9d9"></a>
|
||||
<a href="../bn-BD/README.md"><img alt="README in বাংলা" src="https://img.shields.io/badge/বাংলা-d9d9d9"></a>
|
||||
<a href="../README.md"><img alt="README in English" src="https://img.shields.io/badge/English-d9d9d9"></a>
|
||||
<a href="./README_CN.md"><img alt="简体中文版自述文件" src="https://img.shields.io/badge/简体中文-d9d9d9"></a>
|
||||
<a href="./README_JA.md"><img alt="日本語のREADME" src="https://img.shields.io/badge/日本語-d9d9d9"></a>
|
||||
<a href="./README_ES.md"><img alt="README en Español" src="https://img.shields.io/badge/Español-d9d9d9"></a>
|
||||
<a href="./README_FR.md"><img alt="README en Français" src="https://img.shields.io/badge/Français-d9d9d9"></a>
|
||||
<a href="./README_KL.md"><img alt="README tlhIngan Hol" src="https://img.shields.io/badge/Klingon-d9d9d9"></a>
|
||||
<a href="./README_KR.md"><img alt="README in Korean" src="https://img.shields.io/badge/한국어-d9d9d9"></a>
|
||||
<a href="./README_AR.md"><img alt="README بالعربية" src="https://img.shields.io/badge/العربية-d9d9d9"></a>
|
||||
<a href="./README_TR.md"><img alt="Türkçe README" src="https://img.shields.io/badge/Türkçe-d9d9d9"></a>
|
||||
<a href="./README_VI.md"><img alt="README Tiếng Việt" src="https://img.shields.io/badge/Ti%E1%BA%BFng%20Vi%E1%BB%87t-d9d9d9"></a>
|
||||
<a href="./README_BN.md"><img alt="README in বাংলা" src="https://img.shields.io/badge/বাংলা-d9d9d9"></a>
|
||||
</p>
|
||||
|
||||
#
|
||||
@ -110,7 +108,7 @@ Dale estrella a Dify en GitHub y serás notificado instantáneamente de las nuev
|
||||
|
||||
</br>
|
||||
|
||||
La forma más fácil de iniciar el servidor de Dify es ejecutar nuestro archivo [docker-compose.yml](../../docker/docker-compose.yaml). Antes de ejecutar el comando de instalación, asegúrate de que [Docker](https://docs.docker.com/get-docker/) y [Docker Compose](https://docs.docker.com/compose/install/) estén instalados en tu máquina:
|
||||
La forma más fácil de iniciar el servidor de Dify es ejecutar nuestro archivo [docker-compose.yml](docker/docker-compose.yaml). Antes de ejecutar el comando de instalación, asegúrate de que [Docker](https://docs.docker.com/get-docker/) y [Docker Compose](https://docs.docker.com/compose/install/) estén instalados en tu máquina:
|
||||
|
||||
```bash
|
||||
cd docker
|
||||
@ -124,7 +122,7 @@ Después de ejecutarlo, puedes acceder al panel de control de Dify en tu navegad
|
||||
|
||||
## Próximos pasos
|
||||
|
||||
Si necesita personalizar la configuración, consulte los comentarios en nuestro archivo [.env.example](../../docker/.env.example) y actualice los valores correspondientes en su archivo `.env`. Además, es posible que deba realizar ajustes en el propio archivo `docker-compose.yaml`, como cambiar las versiones de las imágenes, las asignaciones de puertos o los montajes de volúmenes, según su entorno de implementación y requisitos específicos. Después de realizar cualquier cambio, vuelva a ejecutar `docker-compose up -d`. Puede encontrar la lista completa de variables de entorno disponibles [aquí](https://docs.dify.ai/getting-started/install-self-hosted/environments).
|
||||
Si necesita personalizar la configuración, consulte los comentarios en nuestro archivo [.env.example](docker/.env.example) y actualice los valores correspondientes en su archivo `.env`. Además, es posible que deba realizar ajustes en el propio archivo `docker-compose.yaml`, como cambiar las versiones de las imágenes, las asignaciones de puertos o los montajes de volúmenes, según su entorno de implementación y requisitos específicos. Después de realizar cualquier cambio, vuelva a ejecutar `docker-compose up -d`. Puede encontrar la lista completa de variables de entorno disponibles [aquí](https://docs.dify.ai/getting-started/install-self-hosted/environments).
|
||||
|
||||
. Después de realizar los cambios, ejecuta `docker-compose up -d` nuevamente. Puedes ver la lista completa de variables de entorno [aquí](https://docs.dify.ai/getting-started/install-self-hosted/environments).
|
||||
|
||||
@ -172,7 +170,7 @@ Implementa Dify en AKS con un clic usando [Azure Devops Pipeline Helm Chart by @
|
||||
|
||||
## Contribuir
|
||||
|
||||
Para aquellos que deseen contribuir con código, consulten nuestra [Guía de contribución](./CONTRIBUTING.md).
|
||||
Para aquellos que deseen contribuir con código, consulten nuestra [Guía de contribución](https://github.com/langgenius/dify/blob/main/CONTRIBUTING/CONTRIBUTING_ES.md).
|
||||
Al mismo tiempo, considera apoyar a Dify compartiéndolo en redes sociales y en eventos y conferencias.
|
||||
|
||||
> Estamos buscando colaboradores para ayudar con la traducción de Dify a idiomas que no sean el mandarín o el inglés. Si estás interesado en ayudar, consulta el [README de i18n](https://github.com/langgenius/dify/blob/main/web/i18n-config/README.md) para obtener más información y déjanos un comentario en el canal `global-users` de nuestro [Servidor de Comunidad en Discord](https://discord.gg/8Tpq4AcN9c).
|
||||
@ -186,7 +184,7 @@ Al mismo tiempo, considera apoyar a Dify compartiéndolo en redes sociales y en
|
||||
## Comunidad y Contacto
|
||||
|
||||
- [Discusión en GitHub](https://github.com/langgenius/dify/discussions). Lo mejor para: compartir comentarios y hacer preguntas.
|
||||
- [Reporte de problemas en GitHub](https://github.com/langgenius/dify/issues). Lo mejor para: errores que encuentres usando Dify.AI y propuestas de características. Consulta nuestra [Guía de contribución](./CONTRIBUTING.md).
|
||||
- [Reporte de problemas en GitHub](https://github.com/langgenius/dify/issues). Lo mejor para: errores que encuentres usando Dify.AI y propuestas de características. Consulta nuestra [Guía de contribución](https://github.com/langgenius/dify/blob/main/CONTRIBUTING.md).
|
||||
- [Discord](https://discord.gg/FngNHpbcY7). Lo mejor para: compartir tus aplicaciones y pasar el rato con la comunidad.
|
||||
- [X(Twitter)](https://twitter.com/dify_ai). Lo mejor para: compartir tus aplicaciones y pasar el rato con la comunidad.
|
||||
|
||||
@ -200,4 +198,12 @@ Para proteger tu privacidad, evita publicar problemas de seguridad en GitHub. En
|
||||
|
||||
## Licencia
|
||||
|
||||
Este repositorio está disponible bajo la [Licencia de Código Abierto de Dify](../../LICENSE), que es esencialmente Apache 2.0 con algunas restricciones adicionales.
|
||||
Este repositorio está disponible bajo la [Licencia de Código Abierto de Dify](../LICENSE), que es esencialmente Apache 2.0 con algunas restricciones adicionales.
|
||||
|
||||
## Divulgación de Seguridad
|
||||
|
||||
Para proteger tu privacidad, evita publicar problemas de seguridad en GitHub. En su lugar, envía tus preguntas a security@dify.ai y te proporcionaremos una respuesta más detallada.
|
||||
|
||||
## Licencia
|
||||
|
||||
Este repositorio está disponible bajo la [Licencia de Código Abierto de Dify](../LICENSE), que es esencialmente Apache 2.0 con algunas restricciones adicionales.
|
||||
@ -1,4 +1,4 @@
|
||||
|
||||
|
||||
|
||||
<p align="center">
|
||||
<a href="https://cloud.dify.ai">Dify Cloud</a> ·
|
||||
@ -35,19 +35,17 @@
|
||||
</p>
|
||||
|
||||
<p align="center">
|
||||
<a href="../../README.md"><img alt="README in English" src="https://img.shields.io/badge/English-d9d9d9"></a>
|
||||
<a href="../zh-TW/README.md"><img alt="繁體中文文件" src="https://img.shields.io/badge/繁體中文-d9d9d9"></a>
|
||||
<a href="../zh-CN/README.md"><img alt="简体中文文件" src="https://img.shields.io/badge/简体中文-d9d9d9"></a>
|
||||
<a href="../ja-JP/README.md"><img alt="日本語のREADME" src="https://img.shields.io/badge/日本語-d9d9d9"></a>
|
||||
<a href="../es-ES/README.md"><img alt="README en Español" src="https://img.shields.io/badge/Español-d9d9d9"></a>
|
||||
<a href="../fr-FR/README.md"><img alt="README en Français" src="https://img.shields.io/badge/Français-d9d9d9"></a>
|
||||
<a href="../tlh/README.md"><img alt="README tlhIngan Hol" src="https://img.shields.io/badge/Klingon-d9d9d9"></a>
|
||||
<a href="../ko-KR/README.md"><img alt="README in Korean" src="https://img.shields.io/badge/한국어-d9d9d9"></a>
|
||||
<a href="../ar-SA/README.md"><img alt="README بالعربية" src="https://img.shields.io/badge/العربية-d9d9d9"></a>
|
||||
<a href="../tr-TR/README.md"><img alt="Türkçe README" src="https://img.shields.io/badge/Türkçe-d9d9d9"></a>
|
||||
<a href="../vi-VN/README.md"><img alt="README Tiếng Việt" src="https://img.shields.io/badge/Ti%E1%BA%BFng%20Vi%E1%BB%87t-d9d9d9"></a>
|
||||
<a href="../de-DE/README.md"><img alt="README in Deutsch" src="https://img.shields.io/badge/German-d9d9d9"></a>
|
||||
<a href="../bn-BD/README.md"><img alt="README in বাংলা" src="https://img.shields.io/badge/বাংলা-d9d9d9"></a>
|
||||
<a href="../README.md"><img alt="README in English" src="https://img.shields.io/badge/English-d9d9d9"></a>
|
||||
<a href="./README_CN.md"><img alt="简体中文版自述文件" src="https://img.shields.io/badge/简体中文-d9d9d9"></a>
|
||||
<a href="./README_JA.md"><img alt="日本語のREADME" src="https://img.shields.io/badge/日本語-d9d9d9"></a>
|
||||
<a href="./README_ES.md"><img alt="README en Español" src="https://img.shields.io/badge/Español-d9d9d9"></a>
|
||||
<a href="./README_FR.md"><img alt="README en Français" src="https://img.shields.io/badge/Français-d9d9d9"></a>
|
||||
<a href="./README_KL.md"><img alt="README tlhIngan Hol" src="https://img.shields.io/badge/Klingon-d9d9d9"></a>
|
||||
<a href="./README_KR.md"><img alt="README in Korean" src="https://img.shields.io/badge/한국어-d9d9d9"></a>
|
||||
<a href="./README_AR.md"><img alt="README بالعربية" src="https://img.shields.io/badge/العربية-d9d9d9"></a>
|
||||
<a href="./README_TR.md"><img alt="Türkçe README" src="https://img.shields.io/badge/Türkçe-d9d9d9"></a>
|
||||
<a href="./README_VI.md"><img alt="README Tiếng Việt" src="https://img.shields.io/badge/Ti%E1%BA%BFng%20Vi%E1%BB%87t-d9d9d9"></a>
|
||||
<a href="./README_BN.md"><img alt="README in বাংলা" src="https://img.shields.io/badge/বাংলা-d9d9d9"></a>
|
||||
</p>
|
||||
|
||||
#
|
||||
@ -110,7 +108,7 @@ Mettez une étoile à Dify sur GitHub et soyez instantanément informé des nouv
|
||||
|
||||
</br>
|
||||
|
||||
La manière la plus simple de démarrer le serveur Dify est d'exécuter notre fichier [docker-compose.yml](../../docker/docker-compose.yaml). Avant d'exécuter la commande d'installation, assurez-vous que [Docker](https://docs.docker.com/get-docker/) et [Docker Compose](https://docs.docker.com/compose/install/) sont installés sur votre machine:
|
||||
La manière la plus simple de démarrer le serveur Dify est d'exécuter notre fichier [docker-compose.yml](docker/docker-compose.yaml). Avant d'exécuter la commande d'installation, assurez-vous que [Docker](https://docs.docker.com/get-docker/) et [Docker Compose](https://docs.docker.com/compose/install/) sont installés sur votre machine:
|
||||
|
||||
```bash
|
||||
cd docker
|
||||
@ -124,7 +122,7 @@ Après l'exécution, vous pouvez accéder au tableau de bord Dify dans votre nav
|
||||
|
||||
## Prochaines étapes
|
||||
|
||||
Si vous devez personnaliser la configuration, veuillez vous référer aux commentaires dans notre fichier [.env.example](../../docker/.env.example) et mettre à jour les valeurs correspondantes dans votre fichier `.env`. De plus, vous devrez peut-être apporter des modifications au fichier `docker-compose.yaml` lui-même, comme changer les versions d'image, les mappages de ports ou les montages de volumes, en fonction de votre environnement de déploiement et de vos exigences spécifiques. Après avoir effectué des modifications, veuillez réexécuter `docker-compose up -d`. Vous pouvez trouver la liste complète des variables d'environnement disponibles [ici](https://docs.dify.ai/getting-started/install-self-hosted/environments).
|
||||
Si vous devez personnaliser la configuration, veuillez vous référer aux commentaires dans notre fichier [.env.example](docker/.env.example) et mettre à jour les valeurs correspondantes dans votre fichier `.env`. De plus, vous devrez peut-être apporter des modifications au fichier `docker-compose.yaml` lui-même, comme changer les versions d'image, les mappages de ports ou les montages de volumes, en fonction de votre environnement de déploiement et de vos exigences spécifiques. Après avoir effectué des modifications, veuillez réexécuter `docker-compose up -d`. Vous pouvez trouver la liste complète des variables d'environnement disponibles [ici](https://docs.dify.ai/getting-started/install-self-hosted/environments).
|
||||
|
||||
Si vous souhaitez configurer une configuration haute disponibilité, la communauté fournit des [Helm Charts](https://helm.sh/) et des fichiers YAML, à travers lesquels vous pouvez déployer Dify sur Kubernetes.
|
||||
|
||||
@ -170,7 +168,7 @@ Déployez Dify sur AKS en un clic en utilisant [Azure Devops Pipeline Helm Chart
|
||||
|
||||
## Contribuer
|
||||
|
||||
Pour ceux qui souhaitent contribuer du code, consultez notre [Guide de contribution](./CONTRIBUTING.md).
|
||||
Pour ceux qui souhaitent contribuer du code, consultez notre [Guide de contribution](https://github.com/langgenius/dify/blob/main/CONTRIBUTING/CONTRIBUTING_FR.md).
|
||||
Dans le même temps, veuillez envisager de soutenir Dify en le partageant sur les réseaux sociaux et lors d'événements et de conférences.
|
||||
|
||||
> Nous recherchons des contributeurs pour aider à traduire Dify dans des langues autres que le mandarin ou l'anglais. Si vous êtes intéressé à aider, veuillez consulter le [README i18n](https://github.com/langgenius/dify/blob/main/web/i18n-config/README.md) pour plus d'informations, et laissez-nous un commentaire dans le canal `global-users` de notre [Serveur communautaire Discord](https://discord.gg/8Tpq4AcN9c).
|
||||
@ -184,7 +182,7 @@ Dans le même temps, veuillez envisager de soutenir Dify en le partageant sur le
|
||||
## Communauté & Contact
|
||||
|
||||
- [Discussion GitHub](https://github.com/langgenius/dify/discussions). Meilleur pour: partager des commentaires et poser des questions.
|
||||
- [Problèmes GitHub](https://github.com/langgenius/dify/issues). Meilleur pour: les bogues que vous rencontrez en utilisant Dify.AI et les propositions de fonctionnalités. Consultez notre [Guide de contribution](./CONTRIBUTING.md).
|
||||
- [Problèmes GitHub](https://github.com/langgenius/dify/issues). Meilleur pour: les bogues que vous rencontrez en utilisant Dify.AI et les propositions de fonctionnalités. Consultez notre [Guide de contribution](https://github.com/langgenius/dify/blob/main/CONTRIBUTING.md).
|
||||
- [Discord](https://discord.gg/FngNHpbcY7). Meilleur pour: partager vos applications et passer du temps avec la communauté.
|
||||
- [X(Twitter)](https://twitter.com/dify_ai). Meilleur pour: partager vos applications et passer du temps avec la communauté.
|
||||
|
||||
@ -198,4 +196,12 @@ Pour protéger votre vie privée, veuillez éviter de publier des problèmes de
|
||||
|
||||
## Licence
|
||||
|
||||
Ce référentiel est disponible sous la [Licence open source Dify](../../LICENSE), qui est essentiellement l'Apache 2.0 avec quelques restrictions supplémentaires.
|
||||
Ce référentiel est disponible sous la [Licence open source Dify](../LICENSE), qui est essentiellement l'Apache 2.0 avec quelques restrictions supplémentaires.
|
||||
|
||||
## Divulgation de sécurité
|
||||
|
||||
Pour protéger votre vie privée, veuillez éviter de publier des problèmes de sécurité sur GitHub. Au lieu de cela, envoyez vos questions à security@dify.ai et nous vous fournirons une réponse plus détaillée.
|
||||
|
||||
## Licence
|
||||
|
||||
Ce référentiel est disponible sous la [Licence open source Dify](../LICENSE), qui est essentiellement l'Apache 2.0 avec quelques restrictions supplémentaires.
|
||||
@ -1,4 +1,4 @@
|
||||
|
||||
|
||||
|
||||
<p align="center">
|
||||
<a href="https://cloud.dify.ai">Dify Cloud</a> ·
|
||||
@ -35,19 +35,17 @@
|
||||
</p>
|
||||
|
||||
<p align="center">
|
||||
<a href="../../README.md"><img alt="README in English" src="https://img.shields.io/badge/English-d9d9d9"></a>
|
||||
<a href="../zh-TW/README.md"><img alt="繁體中文文件" src="https://img.shields.io/badge/繁體中文-d9d9d9"></a>
|
||||
<a href="../zh-CN/README.md"><img alt="简体中文文件" src="https://img.shields.io/badge/简体中文-d9d9d9"></a>
|
||||
<a href="../ja-JP/README.md"><img alt="日本語のREADME" src="https://img.shields.io/badge/日本語-d9d9d9"></a>
|
||||
<a href="../es-ES/README.md"><img alt="README en Español" src="https://img.shields.io/badge/Español-d9d9d9"></a>
|
||||
<a href="../fr-FR/README.md"><img alt="README en Français" src="https://img.shields.io/badge/Français-d9d9d9"></a>
|
||||
<a href="../tlh/README.md"><img alt="README tlhIngan Hol" src="https://img.shields.io/badge/Klingon-d9d9d9"></a>
|
||||
<a href="../ko-KR/README.md"><img alt="README in Korean" src="https://img.shields.io/badge/한국어-d9d9d9"></a>
|
||||
<a href="../ar-SA/README.md"><img alt="README بالعربية" src="https://img.shields.io/badge/العربية-d9d9d9"></a>
|
||||
<a href="../tr-TR/README.md"><img alt="Türkçe README" src="https://img.shields.io/badge/Türkçe-d9d9d9"></a>
|
||||
<a href="../vi-VN/README.md"><img alt="README Tiếng Việt" src="https://img.shields.io/badge/Ti%E1%BA%BFng%20Vi%E1%BB%87t-d9d9d9"></a>
|
||||
<a href="../de-DE/README.md"><img alt="README in Deutsch" src="https://img.shields.io/badge/German-d9d9d9"></a>
|
||||
<a href="../bn-BD/README.md"><img alt="README in বাংলা" src="https://img.shields.io/badge/বাংলা-d9d9d9"></a>
|
||||
<a href="../README.md"><img alt="README in English" src="https://img.shields.io/badge/English-d9d9d9"></a>
|
||||
<a href="./README_CN.md"><img alt="简体中文版自述文件" src="https://img.shields.io/badge/简体中文-d9d9d9"></a>
|
||||
<a href="./README_JA.md"><img alt="日本語のREADME" src="https://img.shields.io/badge/日本語-d9d9d9"></a>
|
||||
<a href="./README_ES.md"><img alt="README en Español" src="https://img.shields.io/badge/Español-d9d9d9"></a>
|
||||
<a href="./README_FR.md"><img alt="README en Français" src="https://img.shields.io/badge/Français-d9d9d9"></a>
|
||||
<a href="./README_KL.md"><img alt="README tlhIngan Hol" src="https://img.shields.io/badge/Klingon-d9d9d9"></a>
|
||||
<a href="./README_KR.md"><img alt="README in Korean" src="https://img.shields.io/badge/한국어-d9d9d9"></a>
|
||||
<a href="./README_AR.md"><img alt="README بالعربية" src="https://img.shields.io/badge/العربية-d9d9d9"></a>
|
||||
<a href="./README_TR.md"><img alt="Türkçe README" src="https://img.shields.io/badge/Türkçe-d9d9d9"></a>
|
||||
<a href="./README_VI.md"><img alt="README Tiếng Việt" src="https://img.shields.io/badge/Ti%E1%BA%BFng%20Vi%E1%BB%87t-d9d9d9"></a>
|
||||
<a href="./README_BN.md"><img alt="README in বাংলা" src="https://img.shields.io/badge/বাংলা-d9d9d9"></a>
|
||||
</p>
|
||||
|
||||
#
|
||||
@ -111,7 +109,7 @@ GitHub上でDifyにスターを付けることで、Difyに関する新しいニ
|
||||
|
||||
</br>
|
||||
|
||||
Difyサーバーを起動する最も簡単な方法は、[docker-compose.yml](../../docker/docker-compose.yaml)ファイルを実行することです。インストールコマンドを実行する前に、マシンに[Docker](https://docs.docker.com/get-docker/)と[Docker Compose](https://docs.docker.com/compose/install/)がインストールされていることを確認してください。
|
||||
Difyサーバーを起動する最も簡単な方法は、[docker-compose.yml](docker/docker-compose.yaml)ファイルを実行することです。インストールコマンドを実行する前に、マシンに[Docker](https://docs.docker.com/get-docker/)と[Docker Compose](https://docs.docker.com/compose/install/)がインストールされていることを確認してください。
|
||||
|
||||
```bash
|
||||
cd docker
|
||||
@ -125,7 +123,7 @@ docker compose up -d
|
||||
|
||||
## 次のステップ
|
||||
|
||||
設定をカスタマイズする必要がある場合は、[.env.example](../../docker/.env.example) ファイルのコメントを参照し、`.env` ファイルの対応する値を更新してください。さらに、デプロイ環境や要件に応じて、`docker-compose.yaml` ファイル自体を調整する必要がある場合があります。たとえば、イメージのバージョン、ポートのマッピング、ボリュームのマウントなどを変更します。変更を加えた後は、`docker-compose up -d` を再実行してください。利用可能な環境変数の全一覧は、[こちら](https://docs.dify.ai/getting-started/install-self-hosted/environments)で確認できます。
|
||||
設定をカスタマイズする必要がある場合は、[.env.example](docker/.env.example) ファイルのコメントを参照し、`.env` ファイルの対応する値を更新してください。さらに、デプロイ環境や要件に応じて、`docker-compose.yaml` ファイル自体を調整する必要がある場合があります。たとえば、イメージのバージョン、ポートのマッピング、ボリュームのマウントなどを変更します。変更を加えた後は、`docker-compose up -d` を再実行してください。利用可能な環境変数の全一覧は、[こちら](https://docs.dify.ai/getting-started/install-self-hosted/environments)で確認できます。
|
||||
|
||||
高可用性設定を設定する必要がある場合、コミュニティは[Helm Charts](https://helm.sh/)とYAMLファイルにより、DifyをKubernetesにデプロイすることができます。
|
||||
|
||||
@ -171,7 +169,7 @@ docker compose up -d
|
||||
|
||||
## 貢献
|
||||
|
||||
コードに貢献したい方は、[Contribution Guide](./CONTRIBUTING.md)を参照してください。
|
||||
コードに貢献したい方は、[Contribution Guide](https://github.com/langgenius/dify/blob/main/CONTRIBUTING/CONTRIBUTING_JA.md)を参照してください。
|
||||
同時に、DifyをSNSやイベント、カンファレンスで共有してサポートしていただけると幸いです。
|
||||
|
||||
> Difyを英語または中国語以外の言語に翻訳してくれる貢献者を募集しています。興味がある場合は、詳細については[i18n README](https://github.com/langgenius/dify/blob/main/web/i18n-config/README.md)を参照してください。また、[Discordコミュニティサーバー](https://discord.gg/8Tpq4AcN9c)の`global-users`チャンネルにコメントを残してください。
|
||||
@ -185,10 +183,10 @@ docker compose up -d
|
||||
## コミュニティ & お問い合わせ
|
||||
|
||||
- [GitHub Discussion](https://github.com/langgenius/dify/discussions). 主に: フィードバックの共有や質問。
|
||||
- [GitHub Issues](https://github.com/langgenius/dify/issues). 主に: Dify.AIを使用する際に発生するエラーや問題については、[貢献ガイド](./CONTRIBUTING.md)を参照してください
|
||||
- [GitHub Issues](https://github.com/langgenius/dify/issues). 主に: Dify.AIを使用する際に発生するエラーや問題については、[貢献ガイド](../CONTRIBUTING/CONTRIBUTING_JA.md)を参照してください
|
||||
- [Discord](https://discord.gg/FngNHpbcY7). 主に: アプリケーションの共有やコミュニティとの交流。
|
||||
- [X(Twitter)](https://twitter.com/dify_ai). 主に: アプリケーションの共有やコミュニティとの交流。
|
||||
|
||||
## ライセンス
|
||||
|
||||
このリポジトリは、Dify Open Source License にいくつかの追加制限を加えた[Difyオープンソースライセンス](../../LICENSE)の下で利用可能です。
|
||||
このリポジトリは、Dify Open Source License にいくつかの追加制限を加えた[Difyオープンソースライセンス](../LICENSE)の下で利用可能です。
|
||||
@ -1,4 +1,4 @@
|
||||
|
||||
|
||||
|
||||
<p align="center">
|
||||
<a href="https://cloud.dify.ai">Dify Cloud</a> ·
|
||||
@ -35,19 +35,17 @@
|
||||
</p>
|
||||
|
||||
<p align="center">
|
||||
<a href="../../README.md"><img alt="README in English" src="https://img.shields.io/badge/English-d9d9d9"></a>
|
||||
<a href="../zh-TW/README.md"><img alt="繁體中文文件" src="https://img.shields.io/badge/繁體中文-d9d9d9"></a>
|
||||
<a href="../zh-CN/README.md"><img alt="简体中文文件" src="https://img.shields.io/badge/简体中文-d9d9d9"></a>
|
||||
<a href="../ja-JP/README.md"><img alt="日本語のREADME" src="https://img.shields.io/badge/日本語-d9d9d9"></a>
|
||||
<a href="../es-ES/README.md"><img alt="README en Español" src="https://img.shields.io/badge/Español-d9d9d9"></a>
|
||||
<a href="../fr-FR/README.md"><img alt="README en Français" src="https://img.shields.io/badge/Français-d9d9d9"></a>
|
||||
<a href="../tlh/README.md"><img alt="README tlhIngan Hol" src="https://img.shields.io/badge/Klingon-d9d9d9"></a>
|
||||
<a href="../ko-KR/README.md"><img alt="README in Korean" src="https://img.shields.io/badge/한국어-d9d9d9"></a>
|
||||
<a href="../ar-SA/README.md"><img alt="README بالعربية" src="https://img.shields.io/badge/العربية-d9d9d9"></a>
|
||||
<a href="../tr-TR/README.md"><img alt="Türkçe README" src="https://img.shields.io/badge/Türkçe-d9d9d9"></a>
|
||||
<a href="../vi-VN/README.md"><img alt="README Tiếng Việt" src="https://img.shields.io/badge/Ti%E1%BA%BFng%20Vi%E1%BB%87t-d9d9d9"></a>
|
||||
<a href="../de-DE/README.md"><img alt="README in Deutsch" src="https://img.shields.io/badge/German-d9d9d9"></a>
|
||||
<a href="../bn-BD/README.md"><img alt="README in বাংলা" src="https://img.shields.io/badge/বাংলা-d9d9d9"></a>
|
||||
<a href="../README.md"><img alt="README in English" src="https://img.shields.io/badge/English-d9d9d9"></a>
|
||||
<a href="./README_CN.md"><img alt="简体中文版自述文件" src="https://img.shields.io/badge/简体中文-d9d9d9"></a>
|
||||
<a href="./README_JA.md"><img alt="日本語のREADME" src="https://img.shields.io/badge/日本語-d9d9d9"></a>
|
||||
<a href="./README_ES.md"><img alt="README en Español" src="https://img.shields.io/badge/Español-d9d9d9"></a>
|
||||
<a href="./README_FR.md"><img alt="README en Français" src="https://img.shields.io/badge/Français-d9d9d9"></a>
|
||||
<a href="./README_KL.md"><img alt="README tlhIngan Hol" src="https://img.shields.io/badge/Klingon-d9d9d9"></a>
|
||||
<a href="./README_KR.md"><img alt="README in Korean" src="https://img.shields.io/badge/한국어-d9d9d9"></a>
|
||||
<a href="./README_AR.md"><img alt="README بالعربية" src="https://img.shields.io/badge/العربية-d9d9d9"></a>
|
||||
<a href="./README_TR.md"><img alt="Türkçe README" src="https://img.shields.io/badge/Türkçe-d9d9d9"></a>
|
||||
<a href="./README_VI.md"><img alt="README Tiếng Việt" src="https://img.shields.io/badge/Ti%E1%BA%BFng%20Vi%E1%BB%87t-d9d9d9"></a>
|
||||
<a href="./README_BN.md"><img alt="README in বাংলা" src="https://img.shields.io/badge/বাংলা-d9d9d9"></a>
|
||||
</p>
|
||||
|
||||
#
|
||||
@ -110,7 +108,7 @@ Star Dify on GitHub and be instantly notified of new releases.
|
||||
|
||||
</br>
|
||||
|
||||
The easiest way to start the Dify server is to run our [docker-compose.yml](../../docker/docker-compose.yaml) file. Before running the installation command, make sure that [Docker](https://docs.docker.com/get-docker/) and [Docker Compose](https://docs.docker.com/compose/install/) are installed on your machine:
|
||||
The easiest way to start the Dify server is to run our [docker-compose.yml](docker/docker-compose.yaml) file. Before running the installation command, make sure that [Docker](https://docs.docker.com/get-docker/) and [Docker Compose](https://docs.docker.com/compose/install/) are installed on your machine:
|
||||
|
||||
```bash
|
||||
cd docker
|
||||
@ -124,7 +122,7 @@ After running, you can access the Dify dashboard in your browser at [http://loca
|
||||
|
||||
## Next steps
|
||||
|
||||
If you need to customize the configuration, please refer to the comments in our [.env.example](../../docker/.env.example) file and update the corresponding values in your `.env` file. Additionally, you might need to make adjustments to the `docker-compose.yaml` file itself, such as changing image versions, port mappings, or volume mounts, based on your specific deployment environment and requirements. After making any changes, please re-run `docker-compose up -d`. You can find the full list of available environment variables [here](https://docs.dify.ai/getting-started/install-self-hosted/environments).
|
||||
If you need to customize the configuration, please refer to the comments in our [.env.example](docker/.env.example) file and update the corresponding values in your `.env` file. Additionally, you might need to make adjustments to the `docker-compose.yaml` file itself, such as changing image versions, port mappings, or volume mounts, based on your specific deployment environment and requirements. After making any changes, please re-run `docker-compose up -d`. You can find the full list of available environment variables [here](https://docs.dify.ai/getting-started/install-self-hosted/environments).
|
||||
|
||||
If you'd like to configure a highly-available setup, there are community-contributed [Helm Charts](https://helm.sh/) and YAML files which allow Dify to be deployed on Kubernetes.
|
||||
|
||||
@ -183,7 +181,10 @@ At the same time, please consider supporting Dify by sharing it on social media
|
||||
|
||||
## Community & Contact
|
||||
|
||||
- [GitHub Discussion](https://github.com/langgenius/dify/discussions). Best for: sharing feedback and asking questions.
|
||||
- \[GitHub Discussion\](https://github.com/langgenius/dify/discussions
|
||||
|
||||
). Best for: sharing feedback and asking questions.
|
||||
|
||||
- [GitHub Issues](https://github.com/langgenius/dify/issues). Best for: bugs you encounter using Dify.AI, and feature proposals. See our [Contribution Guide](https://github.com/langgenius/dify/blob/main/CONTRIBUTING.md).
|
||||
- [Discord](https://discord.gg/FngNHpbcY7). Best for: sharing your applications and hanging out with the community.
|
||||
- [X(Twitter)](https://twitter.com/dify_ai). Best for: sharing your applications and hanging out with the community.
|
||||
@ -198,4 +199,4 @@ To protect your privacy, please avoid posting security issues on GitHub. Instead
|
||||
|
||||
## License
|
||||
|
||||
This repository is available under the [Dify Open Source License](../../LICENSE), which is essentially Apache 2.0 with a few additional restrictions.
|
||||
This repository is available under the [Dify Open Source License](../LICENSE), which is essentially Apache 2.0 with a few additional restrictions.
|
||||
@ -1,4 +1,4 @@
|
||||
|
||||
|
||||
|
||||
<p align="center">
|
||||
<a href="https://cloud.dify.ai">Dify 클라우드</a> ·
|
||||
@ -35,19 +35,17 @@
|
||||
</p>
|
||||
|
||||
<p align="center">
|
||||
<a href="../../README.md"><img alt="README in English" src="https://img.shields.io/badge/English-d9d9d9"></a>
|
||||
<a href="../zh-TW/README.md"><img alt="繁體中文文件" src="https://img.shields.io/badge/繁體中文-d9d9d9"></a>
|
||||
<a href="../zh-CN/README.md"><img alt="简体中文文件" src="https://img.shields.io/badge/简体中文-d9d9d9"></a>
|
||||
<a href="../ja-JP/README.md"><img alt="日本語のREADME" src="https://img.shields.io/badge/日本語-d9d9d9"></a>
|
||||
<a href="../es-ES/README.md"><img alt="README en Español" src="https://img.shields.io/badge/Español-d9d9d9"></a>
|
||||
<a href="../fr-FR/README.md"><img alt="README en Français" src="https://img.shields.io/badge/Français-d9d9d9"></a>
|
||||
<a href="../tlh/README.md"><img alt="README tlhIngan Hol" src="https://img.shields.io/badge/Klingon-d9d9d9"></a>
|
||||
<a href="../ko-KR/README.md"><img alt="README in Korean" src="https://img.shields.io/badge/한국어-d9d9d9"></a>
|
||||
<a href="../ar-SA/README.md"><img alt="README بالعربية" src="https://img.shields.io/badge/العربية-d9d9d9"></a>
|
||||
<a href="../tr-TR/README.md"><img alt="Türkçe README" src="https://img.shields.io/badge/Türkçe-d9d9d9"></a>
|
||||
<a href="../vi-VN/README.md"><img alt="README Tiếng Việt" src="https://img.shields.io/badge/Ti%E1%BA%BFng%20Vi%E1%BB%87t-d9d9d9"></a>
|
||||
<a href="../de-DE/README.md"><img alt="README in Deutsch" src="https://img.shields.io/badge/German-d9d9d9"></a>
|
||||
<a href="../bn-BD/README.md"><img alt="README in বাংলা" src="https://img.shields.io/badge/বাংলা-d9d9d9"></a>
|
||||
<a href="../README.md"><img alt="README in English" src="https://img.shields.io/badge/English-d9d9d9"></a>
|
||||
<a href="./README_CN.md"><img alt="简体中文版自述文件" src="https://img.shields.io/badge/简体中文-d9d9d9"></a>
|
||||
<a href="./README_JA.md"><img alt="日本語のREADME" src="https://img.shields.io/badge/日本語-d9d9d9"></a>
|
||||
<a href="./README_ES.md"><img alt="README en Español" src="https://img.shields.io/badge/Español-d9d9d9"></a>
|
||||
<a href="./README_FR.md"><img alt="README en Français" src="https://img.shields.io/badge/Français-d9d9d9"></a>
|
||||
<a href="./README_KL.md"><img alt="README tlhIngan Hol" src="https://img.shields.io/badge/Klingon-d9d9d9"></a>
|
||||
<a href="./README_KR.md"><img alt="README in Korean" src="https://img.shields.io/badge/한국어-d9d9d9"></a>
|
||||
<a href="./README_AR.md"><img alt="README بالعربية" src="https://img.shields.io/badge/العربية-d9d9d9"></a>
|
||||
<a href="./README_TR.md"><img alt="Türkçe README" src="https://img.shields.io/badge/Türkçe-d9d9d9"></a>
|
||||
<a href="./README_VI.md"><img alt="README Tiếng Việt" src="https://img.shields.io/badge/Ti%E1%BA%BFng%20Vi%E1%BB%87t-d9d9d9"></a>
|
||||
<a href="./README_BN.md"><img alt="README in বাংলা" src="https://img.shields.io/badge/বাংলা-d9d9d9"></a>
|
||||
</p>
|
||||
|
||||
Dify는 오픈 소스 LLM 앱 개발 플랫폼입니다. 직관적인 인터페이스를 통해 AI 워크플로우, RAG 파이프라인, 에이전트 기능, 모델 관리, 관찰 기능 등을 결합하여 프로토타입에서 프로덕션까지 빠르게 전환할 수 있습니다. 주요 기능 목록은 다음과 같습니다:</br> </br>
|
||||
@ -104,7 +102,7 @@ GitHub에서 Dify에 별표를 찍어 새로운 릴리스를 즉시 알림 받
|
||||
|
||||
</br>
|
||||
|
||||
Dify 서버를 시작하는 가장 쉬운 방법은 [docker-compose.yml](../../docker/docker-compose.yaml) 파일을 실행하는 것입니다. 설치 명령을 실행하기 전에 [Docker](https://docs.docker.com/get-docker/) 및 [Docker Compose](https://docs.docker.com/compose/install/)가 머신에 설치되어 있는지 확인하세요.
|
||||
Dify 서버를 시작하는 가장 쉬운 방법은 [docker-compose.yml](docker/docker-compose.yaml) 파일을 실행하는 것입니다. 설치 명령을 실행하기 전에 [Docker](https://docs.docker.com/get-docker/) 및 [Docker Compose](https://docs.docker.com/compose/install/)가 머신에 설치되어 있는지 확인하세요.
|
||||
|
||||
```bash
|
||||
cd docker
|
||||
@ -118,7 +116,7 @@ docker compose up -d
|
||||
|
||||
## 다음 단계
|
||||
|
||||
구성을 사용자 정의해야 하는 경우 [.env.example](../../docker/.env.example) 파일의 주석을 참조하고 `.env` 파일에서 해당 값을 업데이트하십시오. 또한 특정 배포 환경 및 요구 사항에 따라 `docker-compose.yaml` 파일 자체를 조정해야 할 수도 있습니다. 예를 들어 이미지 버전, 포트 매핑 또는 볼륨 마운트를 변경합니다. 변경 한 후 `docker-compose up -d`를 다시 실행하십시오. 사용 가능한 환경 변수의 전체 목록은 [여기](https://docs.dify.ai/getting-started/install-self-hosted/environments)에서 찾을 수 있습니다.
|
||||
구성을 사용자 정의해야 하는 경우 [.env.example](docker/.env.example) 파일의 주석을 참조하고 `.env` 파일에서 해당 값을 업데이트하십시오. 또한 특정 배포 환경 및 요구 사항에 따라 `docker-compose.yaml` 파일 자체를 조정해야 할 수도 있습니다. 예를 들어 이미지 버전, 포트 매핑 또는 볼륨 마운트를 변경합니다. 변경 한 후 `docker-compose up -d`를 다시 실행하십시오. 사용 가능한 환경 변수의 전체 목록은 [여기](https://docs.dify.ai/getting-started/install-self-hosted/environments)에서 찾을 수 있습니다.
|
||||
|
||||
Dify를 Kubernetes에 배포하고 프리미엄 스케일링 설정을 구성했다는 커뮤니티가 제공하는 [Helm Charts](https://helm.sh/)와 YAML 파일이 존재합니다.
|
||||
|
||||
@ -164,7 +162,7 @@ Dify를 Kubernetes에 배포하고 프리미엄 스케일링 설정을 구성했
|
||||
|
||||
## 기여
|
||||
|
||||
코드에 기여하고 싶은 분들은 [기여 가이드](./CONTRIBUTING.md)를 참조하세요.
|
||||
코드에 기여하고 싶은 분들은 [기여 가이드](https://github.com/langgenius/dify/blob/main/CONTRIBUTING/CONTRIBUTING_KR.md)를 참조하세요.
|
||||
동시에 Dify를 소셜 미디어와 행사 및 컨퍼런스에 공유하여 지원하는 것을 고려해 주시기 바랍니다.
|
||||
|
||||
> 우리는 Dify를 중국어나 영어 이외의 언어로 번역하는 데 도움을 줄 수 있는 기여자를 찾고 있습니다. 도움을 주고 싶으시다면 [i18n README](https://github.com/langgenius/dify/blob/main/web/i18n-config/README.md)에서 더 많은 정보를 확인하시고 [Discord 커뮤니티 서버](https://discord.gg/8Tpq4AcN9c)의 `global-users` 채널에 댓글을 남겨주세요.
|
||||
@ -178,7 +176,7 @@ Dify를 Kubernetes에 배포하고 프리미엄 스케일링 설정을 구성했
|
||||
## 커뮤니티 & 연락처
|
||||
|
||||
- [GitHub 토론](https://github.com/langgenius/dify/discussions). 피드백 공유 및 질문하기에 적합합니다.
|
||||
- [GitHub 이슈](https://github.com/langgenius/dify/issues). Dify.AI 사용 중 발견한 버그와 기능 제안에 적합합니다. [기여 가이드](./CONTRIBUTING.md)를 참조하세요.
|
||||
- [GitHub 이슈](https://github.com/langgenius/dify/issues). Dify.AI 사용 중 발견한 버그와 기능 제안에 적합합니다. [기여 가이드](https://github.com/langgenius/dify/blob/main/CONTRIBUTING.md)를 참조하세요.
|
||||
- [디스코드](https://discord.gg/FngNHpbcY7). 애플리케이션 공유 및 커뮤니티와 소통하기에 적합합니다.
|
||||
- [트위터](https://twitter.com/dify_ai). 애플리케이션 공유 및 커뮤니티와 소통하기에 적합합니다.
|
||||
|
||||
@ -192,4 +190,4 @@ Dify를 Kubernetes에 배포하고 프리미엄 스케일링 설정을 구성했
|
||||
|
||||
## 라이선스
|
||||
|
||||
이 저장소는 기본적으로 몇 가지 추가 제한 사항이 있는 Apache 2.0인 [Dify 오픈 소스 라이선스](../../LICENSE)에 따라 사용할 수 있습니다.
|
||||
이 저장소는 기본적으로 몇 가지 추가 제한 사항이 있는 Apache 2.0인 [Dify 오픈 소스 라이선스](../LICENSE)에 따라 사용할 수 있습니다.
|
||||
@ -1,4 +1,4 @@
|
||||
|
||||
|
||||
|
||||
<p align="center">
|
||||
📌 <a href="https://dify.ai/blog/introducing-dify-workflow-file-upload-a-demo-on-ai-podcast">Introduzindo o Dify Workflow com Upload de Arquivo: Recrie o Podcast Google NotebookLM</a>
|
||||
@ -39,20 +39,18 @@
|
||||
</p>
|
||||
|
||||
<p align="center">
|
||||
<a href="../../README.md"><img alt="README em Inglês" src="https://img.shields.io/badge/English-d9d9d9"></a>
|
||||
<a href="../zh-TW/README.md"><img alt="繁體中文文件" src="https://img.shields.io/badge/繁體中文-d9d9d9"></a>
|
||||
<a href="../zh-CN/README.md"><img alt="简体中文文件" src="https://img.shields.io/badge/简体中文-d9d9d9"></a>
|
||||
<a href="../ja-JP/README.md"><img alt="日本語のREADME" src="https://img.shields.io/badge/日本語-d9d9d9"></a>
|
||||
<a href="../es-ES/README.md"><img alt="README em Espanhol" src="https://img.shields.io/badge/Español-d9d9d9"></a>
|
||||
<a href="../fr-FR/README.md"><img alt="README em Francês" src="https://img.shields.io/badge/Français-d9d9d9"></a>
|
||||
<a href="../tlh/README.md"><img alt="README tlhIngan Hol" src="https://img.shields.io/badge/Klingon-d9d9d9"></a>
|
||||
<a href="../ko-KR/README.md"><img alt="README em Coreano" src="https://img.shields.io/badge/한국어-d9d9d9"></a>
|
||||
<a href="../ar-SA/README.md"><img alt="README em Árabe" src="https://img.shields.io/badge/العربية-d9d9d9"></a>
|
||||
<a href="../tr-TR/README.md"><img alt="README em Turco" src="https://img.shields.io/badge/Türkçe-d9d9d9"></a>
|
||||
<a href="../vi-VN/README.md"><img alt="README em Vietnamita" src="https://img.shields.io/badge/Ti%E1%BA%BFng%20Vi%E1%BB%87t-d9d9d9"></a>
|
||||
<a href="../pt-BR/README.md"><img alt="README em Português - BR" src="https://img.shields.io/badge/Portugu%C3%AAs-BR?style=flat&label=BR&color=d9d9d9"></a>
|
||||
<a href="../de-DE/README.md"><img alt="README in Deutsch" src="https://img.shields.io/badge/German-d9d9d9"></a>
|
||||
<a href="../bn-BD/README.md"><img alt="README in বাংলা" src="https://img.shields.io/badge/বাংলা-d9d9d9"></a>
|
||||
<a href="../README.md"><img alt="README em Inglês" src="https://img.shields.io/badge/English-d9d9d9"></a>
|
||||
<a href="./README_CN.md"><img alt="简体中文版自述文件" src="https://img.shields.io/badge/简体中文-d9d9d9"></a>
|
||||
<a href="./README_JA.md"><img alt="日本語のREADME" src="https://img.shields.io/badge/日本語-d9d9d9"></a>
|
||||
<a href="./README_ES.md"><img alt="README em Espanhol" src="https://img.shields.io/badge/Español-d9d9d9"></a>
|
||||
<a href="./README_FR.md"><img alt="README em Francês" src="https://img.shields.io/badge/Français-d9d9d9"></a>
|
||||
<a href="./README_KL.md"><img alt="README tlhIngan Hol" src="https://img.shields.io/badge/Klingon-d9d9d9"></a>
|
||||
<a href="./README_KR.md"><img alt="README em Coreano" src="https://img.shields.io/badge/한국어-d9d9d9"></a>
|
||||
<a href="./README_AR.md"><img alt="README em Árabe" src="https://img.shields.io/badge/العربية-d9d9d9"></a>
|
||||
<a href="./README_TR.md"><img alt="README em Turco" src="https://img.shields.io/badge/Türkçe-d9d9d9"></a>
|
||||
<a href="./README_VI.md"><img alt="README em Vietnamita" src="https://img.shields.io/badge/Ti%E1%BA%BFng%20Vi%E1%BB%87t-d9d9d9"></a>
|
||||
<a href="./README_PT.md"><img alt="README em Português - BR" src="https://img.shields.io/badge/Portugu%C3%AAs-BR?style=flat&label=BR&color=d9d9d9"></a>
|
||||
<a href="./README_BN.md"><img alt="README in বাংলা" src="https://img.shields.io/badge/বাংলা-d9d9d9"></a>
|
||||
</p>
|
||||
|
||||
Dify é uma plataforma de desenvolvimento de aplicativos LLM de código aberto. Sua interface intuitiva combina workflow de IA, pipeline RAG, capacidades de agente, gerenciamento de modelos, recursos de observabilidade e muito mais, permitindo que você vá rapidamente do protótipo à produção. Aqui está uma lista das principais funcionalidades:
|
||||
@ -110,7 +108,7 @@ Dê uma estrela no Dify no GitHub e seja notificado imediatamente sobre novos la
|
||||
|
||||
</br>
|
||||
|
||||
A maneira mais fácil de iniciar o servidor Dify é executar nosso arquivo [docker-compose.yml](../../docker/docker-compose.yaml). Antes de rodar o comando de instalação, certifique-se de que o [Docker](https://docs.docker.com/get-docker/) e o [Docker Compose](https://docs.docker.com/compose/install/) estão instalados na sua máquina:
|
||||
A maneira mais fácil de iniciar o servidor Dify é executar nosso arquivo [docker-compose.yml](docker/docker-compose.yaml). Antes de rodar o comando de instalação, certifique-se de que o [Docker](https://docs.docker.com/get-docker/) e o [Docker Compose](https://docs.docker.com/compose/install/) estão instalados na sua máquina:
|
||||
|
||||
```bash
|
||||
cd docker
|
||||
@ -124,7 +122,7 @@ Após a execução, você pode acessar o painel do Dify no navegador em [http://
|
||||
|
||||
## Próximos passos
|
||||
|
||||
Se precisar personalizar a configuração, consulte os comentários no nosso arquivo [.env.example](../../docker/.env.example) e atualize os valores correspondentes no seu arquivo `.env`. Além disso, talvez seja necessário fazer ajustes no próprio arquivo `docker-compose.yaml`, como alterar versões de imagem, mapeamentos de portas ou montagens de volumes, com base no seu ambiente de implantação específico e nas suas necessidades. Após fazer quaisquer alterações, execute novamente `docker-compose up -d`. Você pode encontrar a lista completa de variáveis de ambiente disponíveis [aqui](https://docs.dify.ai/getting-started/install-self-hosted/environments).
|
||||
Se precisar personalizar a configuração, consulte os comentários no nosso arquivo [.env.example](docker/.env.example) e atualize os valores correspondentes no seu arquivo `.env`. Além disso, talvez seja necessário fazer ajustes no próprio arquivo `docker-compose.yaml`, como alterar versões de imagem, mapeamentos de portas ou montagens de volumes, com base no seu ambiente de implantação específico e nas suas necessidades. Após fazer quaisquer alterações, execute novamente `docker-compose up -d`. Você pode encontrar a lista completa de variáveis de ambiente disponíveis [aqui](https://docs.dify.ai/getting-started/install-self-hosted/environments).
|
||||
|
||||
Se deseja configurar uma instalação de alta disponibilidade, há [Helm Charts](https://helm.sh/) e arquivos YAML contribuídos pela comunidade que permitem a implantação do Dify no Kubernetes.
|
||||
|
||||
@ -170,7 +168,7 @@ Implante o Dify no AKS com um clique usando [Azure Devops Pipeline Helm Chart by
|
||||
|
||||
## Contribuindo
|
||||
|
||||
Para aqueles que desejam contribuir com código, veja nosso [Guia de Contribuição](./CONTRIBUTING.md).
|
||||
Para aqueles que desejam contribuir com código, veja nosso [Guia de Contribuição](https://github.com/langgenius/dify/blob/main/CONTRIBUTING/CONTRIBUTING_PT.md).
|
||||
Ao mesmo tempo, considere apoiar o Dify compartilhando-o nas redes sociais e em eventos e conferências.
|
||||
|
||||
> Estamos buscando contribuidores para ajudar na tradução do Dify para idiomas além de Mandarim e Inglês. Se você tiver interesse em ajudar, consulte o [README i18n](https://github.com/langgenius/dify/blob/main/web/i18n-config/README.md) para mais informações e deixe-nos um comentário no canal `global-users` em nosso [Servidor da Comunidade no Discord](https://discord.gg/8Tpq4AcN9c).
|
||||
@ -184,7 +182,7 @@ Ao mesmo tempo, considere apoiar o Dify compartilhando-o nas redes sociais e em
|
||||
## Comunidade e contato
|
||||
|
||||
- [Discussões no GitHub](https://github.com/langgenius/dify/discussions). Melhor para: compartilhar feedback e fazer perguntas.
|
||||
- [Problemas no GitHub](https://github.com/langgenius/dify/issues). Melhor para: relatar bugs encontrados no Dify.AI e propor novos recursos. Veja nosso [Guia de Contribuição](./CONTRIBUTING.md).
|
||||
- [Problemas no GitHub](https://github.com/langgenius/dify/issues). Melhor para: relatar bugs encontrados no Dify.AI e propor novos recursos. Veja nosso [Guia de Contribuição](https://github.com/langgenius/dify/blob/main/CONTRIBUTING.md).
|
||||
- [Discord](https://discord.gg/FngNHpbcY7). Melhor para: compartilhar suas aplicações e interagir com a comunidade.
|
||||
- [X(Twitter)](https://twitter.com/dify_ai). Melhor para: compartilhar suas aplicações e interagir com a comunidade.
|
||||
|
||||
@ -198,4 +196,4 @@ Para proteger sua privacidade, evite postar problemas de segurança no GitHub. E
|
||||
|
||||
## Licença
|
||||
|
||||
Este repositório está disponível sob a [Licença de Código Aberto Dify](../../LICENSE), que é essencialmente Apache 2.0 com algumas restrições adicionais.
|
||||
Este repositório está disponível sob a [Licença de Código Aberto Dify](../LICENSE), que é essencialmente Apache 2.0 com algumas restrições adicionais.
|
||||
@ -1,4 +1,4 @@
|
||||
|
||||
|
||||
|
||||
<p align="center">
|
||||
📌 <a href="https://dify.ai/blog/introducing-dify-workflow-file-upload-a-demo-on-ai-podcast">Predstavljamo nalaganje datotek Dify Workflow: znova ustvarite Google NotebookLM Podcast</a>
|
||||
@ -36,20 +36,18 @@
|
||||
</p>
|
||||
|
||||
<p align="center">
|
||||
<a href="../../README.md"><img alt="README in English" src="https://img.shields.io/badge/English-d9d9d9"></a>
|
||||
<a href="../zh-TW/README.md"><img alt="繁體中文文件" src="https://img.shields.io/badge/繁體中文-d9d9d9"></a>
|
||||
<a href="../zh-CN/README.md"><img alt="简体中文文件" src="https://img.shields.io/badge/简体中文-d9d9d9"></a>
|
||||
<a href="../ja-JP/README.md"><img alt="日本語のREADME" src="https://img.shields.io/badge/日本語-d9d9d9"></a>
|
||||
<a href="../es-ES/README.md"><img alt="README en Español" src="https://img.shields.io/badge/Español-d9d9d9"></a>
|
||||
<a href="../fr-FR/README.md"><img alt="README en Français" src="https://img.shields.io/badge/Français-d9d9d9"></a>
|
||||
<a href="../tlh/README.md"><img alt="README tlhIngan Hol" src="https://img.shields.io/badge/Klingon-d9d9d9"></a>
|
||||
<a href="../ko-KR/README.md"><img alt="README in Korean" src="https://img.shields.io/badge/한국어-d9d9d9"></a>
|
||||
<a href="../ar-SA/README.md"><img alt="README بالعربية" src="https://img.shields.io/badge/العربية-d9d9d9"></a>
|
||||
<a href="../tr-TR/README.md"><img alt="Türkçe README" src="https://img.shields.io/badge/Türkçe-d9d9d9"></a>
|
||||
<a href="../vi-VN/README.md"><img alt="README Tiếng Việt" src="https://img.shields.io/badge/Ti%E1%BA%BFng%20Vi%E1%BB%87t-d9d9d9"></a>
|
||||
<a href="../sl-SI/README.md"><img alt="README Slovenščina" src="https://img.shields.io/badge/Sloven%C5%A1%C4%8Dina-d9d9d9"></a>
|
||||
<a href="../de-DE/README.md"><img alt="README in Deutsch" src="https://img.shields.io/badge/German-d9d9d9"></a>
|
||||
<a href="../bn-BD/README.md"><img alt="README in বাংলা" src="https://img.shields.io/badge/বাংলা-d9d9d9"></a>
|
||||
<a href="../README.md"><img alt="README in English" src="https://img.shields.io/badge/English-d9d9d9"></a>
|
||||
<a href="./README_CN.md"><img alt="简体中文版自述文件" src="https://img.shields.io/badge/简体中文-d9d9d9"></a>
|
||||
<a href="./README_JA.md"><img alt="日本語のREADME" src="https://img.shields.io/badge/日本語-d9d9d9"></a>
|
||||
<a href="./README_ES.md"><img alt="README en Español" src="https://img.shields.io/badge/Español-d9d9d9"></a>
|
||||
<a href="./README_FR.md"><img alt="README en Français" src="https://img.shields.io/badge/Français-d9d9d9"></a>
|
||||
<a href="./README_KL.md"><img alt="README tlhIngan Hol" src="https://img.shields.io/badge/Klingon-d9d9d9"></a>
|
||||
<a href="./README_KR.md"><img alt="README in Korean" src="https://img.shields.io/badge/한국어-d9d9d9"></a>
|
||||
<a href="./README_AR.md"><img alt="README بالعربية" src="https://img.shields.io/badge/العربية-d9d9d9"></a>
|
||||
<a href="./README_TR.md"><img alt="Türkçe README" src="https://img.shields.io/badge/Türkçe-d9d9d9"></a>
|
||||
<a href="./README_VI.md"><img alt="README Tiếng Việt" src="https://img.shields.io/badge/Ti%E1%BA%BFng%20Vi%E1%BB%87t-d9d9d9"></a>
|
||||
<a href="./README_SI.md"><img alt="README Slovenščina" src="https://img.shields.io/badge/Sloven%C5%A1%C4%8Dina-d9d9d9"></a>
|
||||
<a href="./README_BN.md"><img alt="README in বাংলা" src="https://img.shields.io/badge/বাংলা-d9d9d9"></a>
|
||||
</p>
|
||||
|
||||
Dify je odprtokodna platforma za razvoj aplikacij LLM. Njegov intuitivni vmesnik združuje agentski potek dela z umetno inteligenco, cevovod RAG, zmogljivosti agentov, upravljanje modelov, funkcije opazovanja in več, kar vam omogoča hiter prehod od prototipa do proizvodnje.
|
||||
@ -171,7 +169,7 @@ Z enim klikom namestite Dify v AKS z uporabo [Azure Devops Pipeline Helm Chart b
|
||||
|
||||
## Prispevam
|
||||
|
||||
Za tiste, ki bi radi prispevali kodo, si oglejte naš [vodnik za prispevke](https://github.com/langgenius/dify/blob/main/CONTRIBUTING.md). Hkrati vas prosimo, da podprete Dify tako, da ga delite na družbenih medijih ter na dogodkih in konferencah.
|
||||
Za tiste, ki bi radi prispevali kodo, si oglejte naš vodnik za prispevke . Hkrati vas prosimo, da podprete Dify tako, da ga delite na družbenih medijih ter na dogodkih in konferencah.
|
||||
|
||||
> Iščemo sodelavce za pomoč pri prevajanju Difyja v jezike, ki niso mandarinščina ali angleščina. Če želite pomagati, si oglejte i18n README za več informacij in nam pustite komentar v global-userskanalu našega strežnika skupnosti Discord .
|
||||
|
||||
@ -198,4 +196,4 @@ Zaradi zaščite vaše zasebnosti se izogibajte objavljanju varnostnih vprašanj
|
||||
|
||||
## Licenca
|
||||
|
||||
To skladišče je na voljo pod [odprtokodno licenco Dify](../../LICENSE) , ki je v bistvu Apache 2.0 z nekaj dodatnimi omejitvami.
|
||||
To skladišče je na voljo pod [odprtokodno licenco Dify](../LICENSE) , ki je v bistvu Apache 2.0 z nekaj dodatnimi omejitvami.
|
||||
@ -1,4 +1,4 @@
|
||||
|
||||
|
||||
|
||||
<p align="center">
|
||||
<a href="https://cloud.dify.ai">Dify Bulut</a> ·
|
||||
@ -35,19 +35,17 @@
|
||||
</p>
|
||||
|
||||
<p align="center">
|
||||
<a href="../../README.md"><img alt="README in English" src="https://img.shields.io/badge/English-d9d9d9"></a>
|
||||
<a href="../zh-TW/README.md"><img alt="繁體中文文件" src="https://img.shields.io/badge/繁體中文-d9d9d9"></a>
|
||||
<a href="../zh-CN/README.md"><img alt="简体中文文件" src="https://img.shields.io/badge/简体中文-d9d9d9"></a>
|
||||
<a href="../ja-JP/README.md"><img alt="日本語のREADME" src="https://img.shields.io/badge/日本語-d9d9d9"></a>
|
||||
<a href="../es-ES/README.md"><img alt="README en Español" src="https://img.shields.io/badge/Español-d9d9d9"></a>
|
||||
<a href="../fr-FR/README.md"><img alt="README en Français" src="https://img.shields.io/badge/Français-d9d9d9"></a>
|
||||
<a href="../tlh/README.md"><img alt="README tlhIngan Hol" src="https://img.shields.io/badge/Klingon-d9d9d9"></a>
|
||||
<a href="../ko-KR/README.md"><img alt="README in Korean" src="https://img.shields.io/badge/한국어-d9d9d9"></a>
|
||||
<a href="../ar-SA/README.md"><img alt="README بالعربية" src="https://img.shields.io/badge/العربية-d9d9d9"></a>
|
||||
<a href="../tr-TR/README.md"><img alt="Türkçe README" src="https://img.shields.io/badge/Türkçe-d9d9d9"></a>
|
||||
<a href="../vi-VN/README.md"><img alt="README Tiếng Việt" src="https://img.shields.io/badge/Ti%E1%BA%BFng%20Vi%E1%BB%87t-d9d9d9"></a>
|
||||
<a href="../de-DE/README.md"><img alt="README in Deutsch" src="https://img.shields.io/badge/German-d9d9d9"></a>
|
||||
<a href="../bn-BD/README.md"><img alt="README in বাংলা" src="https://img.shields.io/badge/বাংলা-d9d9d9"></a>
|
||||
<a href="../README.md"><img alt="README in English" src="https://img.shields.io/badge/English-d9d9d9"></a>
|
||||
<a href="./README_CN.md"><img alt="简体中文版自述文件" src="https://img.shields.io/badge/简体中文-d9d9d9"></a>
|
||||
<a href="./README_JA.md"><img alt="日本語のREADME" src="https://img.shields.io/badge/日本語-d9d9d9"></a>
|
||||
<a href="./README_ES.md"><img alt="README en Español" src="https://img.shields.io/badge/Español-d9d9d9"></a>
|
||||
<a href="./README_FR.md"><img alt="README en Français" src="https://img.shields.io/badge/Français-d9d9d9"></a>
|
||||
<a href="./README_KL.md"><img alt="README tlhIngan Hol" src="https://img.shields.io/badge/Klingon-d9d9d9"></a>
|
||||
<a href="./README_KR.md"><img alt="README in Korean" src="https://img.shields.io/badge/한국어-d9d9d9"></a>
|
||||
<a href="./README_AR.md"><img alt="README بالعربية" src="https://img.shields.io/badge/العربية-d9d9d9"></a>
|
||||
<a href="./README_TR.md"><img alt="Türkçe README" src="https://img.shields.io/badge/Türkçe-d9d9d9"></a>
|
||||
<a href="./README_VI.md"><img alt="README Tiếng Việt" src="https://img.shields.io/badge/Ti%E1%BA%BFng%20Vi%E1%BB%87t-d9d9d9"></a>
|
||||
<a href="./README_BN.md"><img alt="README in বাংলা" src="https://img.shields.io/badge/বাংলা-d9d9d9"></a>
|
||||
</p>
|
||||
|
||||
Dify, açık kaynaklı bir LLM uygulama geliştirme platformudur. Sezgisel arayüzü, AI iş akışı, RAG pipeline'ı, ajan yetenekleri, model yönetimi, gözlemlenebilirlik özellikleri ve daha fazlasını birleştirerek, prototipten üretime hızlıca geçmenizi sağlar. İşte temel özelliklerin bir listesi:
|
||||
@ -104,7 +102,7 @@ GitHub'da Dify'a yıldız verin ve yeni sürümlerden anında haberdar olun.
|
||||
> - RAM >= 4GB
|
||||
|
||||
</br>
|
||||
Dify sunucusunu başlatmanın en kolay yolu, [docker-compose.yml](../../docker/docker-compose.yaml) dosyamızı çalıştırmaktır. Kurulum komutunu çalıştırmadan önce, makinenizde [Docker](https://docs.docker.com/get-docker/) ve [Docker Compose](https://docs.docker.com/compose/install/)'un kurulu olduğundan emin olun:
|
||||
Dify sunucusunu başlatmanın en kolay yolu, [docker-compose.yml](docker/docker-compose.yaml) dosyamızı çalıştırmaktır. Kurulum komutunu çalıştırmadan önce, makinenizde [Docker](https://docs.docker.com/get-docker/) ve [Docker Compose](https://docs.docker.com/compose/install/)'un kurulu olduğundan emin olun:
|
||||
|
||||
```bash
|
||||
cd docker
|
||||
@ -118,7 +116,7 @@ docker compose up -d
|
||||
|
||||
## Sonraki adımlar
|
||||
|
||||
Yapılandırmayı özelleştirmeniz gerekiyorsa, lütfen [.env.example](../../docker/.env.example) dosyamızdaki yorumlara bakın ve `.env` dosyanızdaki ilgili değerleri güncelleyin. Ayrıca, spesifik dağıtım ortamınıza ve gereksinimlerinize bağlı olarak `docker-compose.yaml` dosyasının kendisinde de, imaj sürümlerini, port eşlemelerini veya hacim bağlantılarını değiştirmek gibi ayarlamalar yapmanız gerekebilir. Herhangi bir değişiklik yaptıktan sonra, lütfen `docker-compose up -d` komutunu tekrar çalıştırın. Kullanılabilir tüm ortam değişkenlerinin tam listesini [burada](https://docs.dify.ai/getting-started/install-self-hosted/environments) bulabilirsiniz.
|
||||
Yapılandırmayı özelleştirmeniz gerekiyorsa, lütfen [.env.example](docker/.env.example) dosyamızdaki yorumlara bakın ve `.env` dosyanızdaki ilgili değerleri güncelleyin. Ayrıca, spesifik dağıtım ortamınıza ve gereksinimlerinize bağlı olarak `docker-compose.yaml` dosyasının kendisinde de, imaj sürümlerini, port eşlemelerini veya hacim bağlantılarını değiştirmek gibi ayarlamalar yapmanız gerekebilir. Herhangi bir değişiklik yaptıktan sonra, lütfen `docker-compose up -d` komutunu tekrar çalıştırın. Kullanılabilir tüm ortam değişkenlerinin tam listesini [burada](https://docs.dify.ai/getting-started/install-self-hosted/environments) bulabilirsiniz.
|
||||
|
||||
Yüksek kullanılabilirliğe sahip bir kurulum yapılandırmak isterseniz, Dify'ın Kubernetes üzerine dağıtılmasına olanak tanıyan topluluk katkılı [Helm Charts](https://helm.sh/) ve YAML dosyaları mevcuttur.
|
||||
|
||||
@ -163,7 +161,7 @@ Dify'ı bulut platformuna tek tıklamayla dağıtın [terraform](https://www.ter
|
||||
|
||||
## Katkıda Bulunma
|
||||
|
||||
Kod katkısında bulunmak isteyenler için [Katkı Kılavuzumuza](./CONTRIBUTING.md) bakabilirsiniz.
|
||||
Kod katkısında bulunmak isteyenler için [Katkı Kılavuzumuza](https://github.com/langgenius/dify/blob/main/CONTRIBUTING/CONTRIBUTING_TR.md) bakabilirsiniz.
|
||||
Aynı zamanda, lütfen Dify'ı sosyal medyada, etkinliklerde ve konferanslarda paylaşarak desteklemeyi düşünün.
|
||||
|
||||
> Dify'ı Mandarin veya İngilizce dışındaki dillere çevirmemize yardımcı olacak katkıda bulunanlara ihtiyacımız var. Yardımcı olmakla ilgileniyorsanız, lütfen daha fazla bilgi için [i18n README](https://github.com/langgenius/dify/blob/main/web/i18n-config/README.md) dosyasına bakın ve [Discord Topluluk Sunucumuzdaki](https://discord.gg/8Tpq4AcN9c) `global-users` kanalında bize bir yorum bırakın.
|
||||
@ -177,7 +175,7 @@ Aynı zamanda, lütfen Dify'ı sosyal medyada, etkinliklerde ve konferanslarda p
|
||||
## Topluluk & iletişim
|
||||
|
||||
- [GitHub Tartışmaları](https://github.com/langgenius/dify/discussions). En uygun: geri bildirim paylaşmak ve soru sormak için.
|
||||
- [GitHub Sorunları](https://github.com/langgenius/dify/issues). En uygun: Dify.AI kullanırken karşılaştığınız hatalar ve özellik önerileri için. [Katkı Kılavuzumuza](./CONTRIBUTING.md) bakın.
|
||||
- [GitHub Sorunları](https://github.com/langgenius/dify/issues). En uygun: Dify.AI kullanırken karşılaştığınız hatalar ve özellik önerileri için. [Katkı Kılavuzumuza](https://github.com/langgenius/dify/blob/main/CONTRIBUTING.md) bakın.
|
||||
- [Discord](https://discord.gg/FngNHpbcY7). En uygun: uygulamalarınızı paylaşmak ve toplulukla vakit geçirmek için.
|
||||
- [X(Twitter)](https://twitter.com/dify_ai). En uygun: uygulamalarınızı paylaşmak ve toplulukla vakit geçirmek için.
|
||||
|
||||
@ -191,4 +189,4 @@ Gizliliğinizi korumak için, lütfen güvenlik sorunlarını GitHub'da paylaşm
|
||||
|
||||
## Lisans
|
||||
|
||||
Bu depo, temel olarak Apache 2.0 lisansı ve birkaç ek kısıtlama içeren [Dify Açık Kaynak Lisansı](../../LICENSE) altında kullanıma sunulmuştur.
|
||||
Bu depo, temel olarak Apache 2.0 lisansı ve birkaç ek kısıtlama içeren [Dify Açık Kaynak Lisansı](../LICENSE) altında kullanıma sunulmuştur.
|
||||
@ -1,4 +1,4 @@
|
||||
|
||||
|
||||
|
||||
<p align="center">
|
||||
📌 <a href="https://dify.ai/blog/introducing-dify-workflow-file-upload-a-demo-on-ai-podcast">介紹 Dify 工作流程檔案上傳功能：重現 Google NotebookLM Podcast</a>
|
||||
@ -39,18 +39,18 @@
|
||||
</p>
|
||||
|
||||
<p align="center">
|
||||
<a href="../../README.md"><img alt="README in English" src="https://img.shields.io/badge/English-d9d9d9"></a>
|
||||
<a href="../zh-TW/README.md"><img alt="繁體中文文件" src="https://img.shields.io/badge/繁體中文-d9d9d9"></a>
|
||||
<a href="../zh-CN/README.md"><img alt="简体中文文件" src="https://img.shields.io/badge/简体中文-d9d9d9"></a>
|
||||
<a href="../ja-JP/README.md"><img alt="日本語のREADME" src="https://img.shields.io/badge/日本語-d9d9d9"></a>
|
||||
<a href="../es-ES/README.md"><img alt="README en Español" src="https://img.shields.io/badge/Español-d9d9d9"></a>
|
||||
<a href="../fr-FR/README.md"><img alt="README en Français" src="https://img.shields.io/badge/Français-d9d9d9"></a>
|
||||
<a href="../tlh/README.md"><img alt="README tlhIngan Hol" src="https://img.shields.io/badge/Klingon-d9d9d9"></a>
|
||||
<a href="../ko-KR/README.md"><img alt="README in Korean" src="https://img.shields.io/badge/한국어-d9d9d9"></a>
|
||||
<a href="../ar-SA/README.md"><img alt="README بالعربية" src="https://img.shields.io/badge/العربية-d9d9d9"></a>
|
||||
<a href="../tr-TR/README.md"><img alt="Türkçe README" src="https://img.shields.io/badge/Türkçe-d9d9d9"></a>
|
||||
<a href="../vi-VN/README.md"><img alt="README Tiếng Việt" src="https://img.shields.io/badge/Ti%E1%BA%BFng%20Vi%E1%BB%87t-d9d9d9"></a>
|
||||
<a href="../de-DE/README.md"><img alt="README in Deutsch" src="https://img.shields.io/badge/German-d9d9d9"></a>
|
||||
<a href="../README.md"><img alt="README in English" src="https://img.shields.io/badge/English-d9d9d9"></a>
|
||||
<a href="./README_TW.md"><img alt="繁體中文文件" src="https://img.shields.io/badge/繁體中文-d9d9d9"></a>
|
||||
<a href="./README_CN.md"><img alt="简体中文版自述文件" src="https://img.shields.io/badge/简体中文-d9d9d9"></a>
|
||||
<a href="./README_JA.md"><img alt="日本語のREADME" src="https://img.shields.io/badge/日本語-d9d9d9"></a>
|
||||
<a href="./README_ES.md"><img alt="README en Español" src="https://img.shields.io/badge/Español-d9d9d9"></a>
|
||||
<a href="./README_FR.md"><img alt="README en Français" src="https://img.shields.io/badge/Français-d9d9d9"></a>
|
||||
<a href="./README_KL.md"><img alt="README tlhIngan Hol" src="https://img.shields.io/badge/Klingon-d9d9d9"></a>
|
||||
<a href="./README_KR.md"><img alt="README in Korean" src="https://img.shields.io/badge/한국어-d9d9d9"></a>
|
||||
<a href="./README_AR.md"><img alt="README بالعربية" src="https://img.shields.io/badge/العربية-d9d9d9"></a>
|
||||
<a href="./README_TR.md"><img alt="Türkçe README" src="https://img.shields.io/badge/Türkçe-d9d9d9"></a>
|
||||
<a href="./README_VI.md"><img alt="README Tiếng Việt" src="https://img.shields.io/badge/Ti%E1%BA%BFng%20Vi%E1%BB%87t-d9d9d9"></a>
|
||||
<a href="./README_DE.md"><img alt="README in Deutsch" src="https://img.shields.io/badge/German-d9d9d9"></a>
|
||||
</p>
|
||||
|
||||
Dify 是一個開源的 LLM 應用程式開發平台。其直觀的界面結合了智能代理工作流程、RAG 管道、代理功能、模型管理、可觀察性功能等，讓您能夠快速從原型進展到生產環境。
|
||||
@ -64,7 +64,7 @@ Dify 是一個開源的 LLM 應用程式開發平台。其直觀的界面結合
|
||||
|
||||
</br>
|
||||
|
||||
啟動 Dify 伺服器最簡單的方式是透過 [docker compose](../../docker/docker-compose.yaml)。在使用以下命令運行 Dify 之前，請確保您的機器已安裝 [Docker](https://docs.docker.com/get-docker/) 和 [Docker Compose](https://docs.docker.com/compose/install/)：
|
||||
啟動 Dify 伺服器最簡單的方式是透過 [docker compose](docker/docker-compose.yaml)。在使用以下命令運行 Dify 之前，請確保您的機器已安裝 [Docker](https://docs.docker.com/get-docker/) 和 [Docker Compose](https://docs.docker.com/compose/install/)：
|
||||
|
||||
```bash
|
||||
cd dify
|
||||
@ -128,7 +128,7 @@ Dify 的所有功能都提供相應的 API，因此您可以輕鬆地將 Dify
|
||||
|
||||
## 進階設定
|
||||
|
||||
如果您需要自定義配置，請參考我們的 [.env.example](../../docker/.env.example) 文件中的註釋，並在您的 `.env` 文件中更新相應的值。此外，根據您特定的部署環境和需求，您可能需要調整 `docker-compose.yaml` 文件本身，例如更改映像版本、端口映射或卷掛載。進行任何更改後，請重新運行 `docker-compose up -d`。您可以在[這裡](https://docs.dify.ai/getting-started/install-self-hosted/environments)找到可用環境變數的完整列表。
|
||||
如果您需要自定義配置，請參考我們的 [.env.example](docker/.env.example) 文件中的註釋，並在您的 `.env` 文件中更新相應的值。此外，根據您特定的部署環境和需求，您可能需要調整 `docker-compose.yaml` 文件本身，例如更改映像版本、端口映射或卷掛載。進行任何更改後，請重新運行 `docker-compose up -d`。您可以在[這裡](https://docs.dify.ai/getting-started/install-self-hosted/environments)找到可用環境變數的完整列表。
|
||||
|
||||
如果您想配置高可用性設置，社區貢獻的 [Helm Charts](https://helm.sh/) 和 Kubernetes 資源清單（YAML）允許在 Kubernetes 上部署 Dify。
|
||||
|
||||
@ -173,7 +173,7 @@ Dify 的所有功能都提供相應的 API，因此您可以輕鬆地將 Dify
|
||||
|
||||
## 貢獻
|
||||
|
||||
對於想要貢獻程式碼的開發者，請參閱我們的[貢獻指南](./CONTRIBUTING.md)。
|
||||
對於想要貢獻程式碼的開發者，請參閱我們的[貢獻指南](https://github.com/langgenius/dify/blob/main/CONTRIBUTING/CONTRIBUTING_TW.md)。
|
||||
同時，也請考慮透過在社群媒體和各種活動與會議上分享 Dify 來支持我們。
|
||||
|
||||
> 我們正在尋找貢獻者協助將 Dify 翻譯成中文和英文以外的語言。如果您有興趣幫忙，請查看 [i18n README](https://github.com/langgenius/dify/blob/main/web/i18n-config/README.md) 獲取更多資訊，並在我們的 [Discord 社群伺服器](https://discord.gg/8Tpq4AcN9c) 的 `global-users` 頻道留言給我們。
|
||||
@ -181,7 +181,7 @@ Dify 的所有功能都提供相應的 API，因此您可以輕鬆地將 Dify
|
||||
## 社群與聯絡方式
|
||||
|
||||
- [GitHub Discussion](https://github.com/langgenius/dify/discussions)：最適合分享反饋和提問。
|
||||
- [GitHub Issues](https://github.com/langgenius/dify/issues)：最適合報告使用 Dify.AI 時遇到的問題和提出功能建議。請參閱我們的[貢獻指南](./CONTRIBUTING.md)。
|
||||
- [GitHub Issues](https://github.com/langgenius/dify/issues)：最適合報告使用 Dify.AI 時遇到的問題和提出功能建議。請參閱我們的[貢獻指南](https://github.com/langgenius/dify/blob/main/CONTRIBUTING.md)。
|
||||
- [Discord](https://discord.gg/FngNHpbcY7)：最適合分享您的應用程式並與社群互動。
|
||||
- [X(Twitter)](https://twitter.com/dify_ai)：最適合分享您的應用程式並與社群互動。
|
||||
|
||||
@ -201,4 +201,4 @@ Dify 的所有功能都提供相應的 API，因此您可以輕鬆地將 Dify
|
||||
|
||||
## 授權條款
|
||||
|
||||
本代碼庫採用 [Dify 開源授權](../../LICENSE)，這基本上是 Apache 2.0 授權加上一些額外限制條款。
|
||||
本代碼庫採用 [Dify 開源授權](../LICENSE)，這基本上是 Apache 2.0 授權加上一些額外限制條款。
|
||||
@ -1,4 +1,4 @@
|
||||
|
||||
|
||||
|
||||
<p align="center">
|
||||
<a href="https://cloud.dify.ai">Dify Cloud</a> ·
|
||||
@ -35,19 +35,17 @@
|
||||
</p>
|
||||
|
||||
<p align="center">
|
||||
<a href="../../README.md"><img alt="README in English" src="https://img.shields.io/badge/English-d9d9d9"></a>
|
||||
<a href="../zh-TW/README.md"><img alt="繁體中文文件" src="https://img.shields.io/badge/繁體中文-d9d9d9"></a>
|
||||
<a href="../zh-CN/README.md"><img alt="简体中文文件" src="https://img.shields.io/badge/简体中文-d9d9d9"></a>
|
||||
<a href="../ja-JP/README.md"><img alt="日本語のREADME" src="https://img.shields.io/badge/日本語-d9d9d9"></a>
|
||||
<a href="../es-ES/README.md"><img alt="README en Español" src="https://img.shields.io/badge/Español-d9d9d9"></a>
|
||||
<a href="../fr-FR/README.md"><img alt="README en Français" src="https://img.shields.io/badge/Français-d9d9d9"></a>
|
||||
<a href="../tlh/README.md"><img alt="README tlhIngan Hol" src="https://img.shields.io/badge/Klingon-d9d9d9"></a>
|
||||
<a href="../ko-KR/README.md"><img alt="README in Korean" src="https://img.shields.io/badge/한국어-d9d9d9"></a>
|
||||
<a href="../ar-SA/README.md"><img alt="README بالعربية" src="https://img.shields.io/badge/العربية-d9d9d9"></a>
|
||||
<a href="../tr-TR/README.md"><img alt="Türkçe README" src="https://img.shields.io/badge/Türkçe-d9d9d9"></a>
|
||||
<a href="../vi-VN/README.md"><img alt="README Tiếng Việt" src="https://img.shields.io/badge/Ti%E1%BA%BFng%20Vi%E1%BB%87t-d9d9d9"></a>
|
||||
<a href="../de-DE/README.md"><img alt="README in Deutsch" src="https://img.shields.io/badge/German-d9d9d9"></a>
|
||||
<a href="../bn-BD/README.md"><img alt="README in বাংলা" src="https://img.shields.io/badge/বাংলা-d9d9d9"></a>
|
||||
<a href="../README.md"><img alt="README in English" src="https://img.shields.io/badge/English-d9d9d9"></a>
|
||||
<a href="./README_CN.md"><img alt="简体中文版自述文件" src="https://img.shields.io/badge/简体中文-d9d9d9"></a>
|
||||
<a href="./README_JA.md"><img alt="日本語のREADME" src="https://img.shields.io/badge/日本語-d9d9d9"></a>
|
||||
<a href="./README_ES.md"><img alt="README en Español" src="https://img.shields.io/badge/Español-d9d9d9"></a>
|
||||
<a href="./README_FR.md"><img alt="README en Français" src="https://img.shields.io/badge/Français-d9d9d9"></a>
|
||||
<a href="./README_KL.md"><img alt="README tlhIngan Hol" src="https://img.shields.io/badge/Klingon-d9d9d9"></a>
|
||||
<a href="./README_KR.md"><img alt="README in Korean" src="https://img.shields.io/badge/한국어-d9d9d9"></a>
|
||||
<a href="./README_AR.md"><img alt="README بالعربية" src="https://img.shields.io/badge/العربية-d9d9d9"></a>
|
||||
<a href="./README_TR.md"><img alt="Türkçe README" src="https://img.shields.io/badge/Türkçe-d9d9d9"></a>
|
||||
<a href="./README_VI.md"><img alt="README Tiếng Việt" src="https://img.shields.io/badge/Ti%E1%BA%BFng%20Vi%E1%BB%87t-d9d9d9"></a>
|
||||
<a href="./README_BN.md"><img alt="README in বাংলা" src="https://img.shields.io/badge/বাংলা-d9d9d9"></a>
|
||||
</p>
|
||||
|
||||
Dify là một nền tảng phát triển ứng dụng LLM mã nguồn mở. Giao diện trực quan kết hợp quy trình làm việc AI, mô hình RAG, khả năng tác nhân, quản lý mô hình, tính năng quan sát và hơn thế nữa, cho phép bạn nhanh chóng chuyển từ nguyên mẫu sang sản phẩm. Đây là danh sách các tính năng cốt lõi:
|
||||
@ -105,7 +103,7 @@ Yêu thích Dify trên GitHub và được thông báo ngay lập tức về cá
|
||||
|
||||
</br>
|
||||
|
||||
Cách dễ nhất để khởi động máy chủ Dify là chạy tệp [docker-compose.yml](../../docker/docker-compose.yaml) của chúng tôi. Trước khi chạy lệnh cài đặt, hãy đảm bảo rằng [Docker](https://docs.docker.com/get-docker/) và [Docker Compose](https://docs.docker.com/compose/install/) đã được cài đặt trên máy của bạn:
|
||||
Cách dễ nhất để khởi động máy chủ Dify là chạy tệp [docker-compose.yml](docker/docker-compose.yaml) của chúng tôi. Trước khi chạy lệnh cài đặt, hãy đảm bảo rằng [Docker](https://docs.docker.com/get-docker/) và [Docker Compose](https://docs.docker.com/compose/install/) đã được cài đặt trên máy của bạn:
|
||||
|
||||
```bash
|
||||
cd docker
|
||||
@ -119,7 +117,7 @@ Sau khi chạy, bạn có thể truy cập bảng điều khiển Dify trong tr
|
||||
|
||||
## Các bước tiếp theo
|
||||
|
||||
Nếu bạn cần tùy chỉnh cấu hình, vui lòng tham khảo các nhận xét trong tệp [.env.example](../../docker/.env.example) của chúng tôi và cập nhật các giá trị tương ứng trong tệp `.env` của bạn. Ngoài ra, bạn có thể cần điều chỉnh tệp `docker-compose.yaml`, chẳng hạn như thay đổi phiên bản hình ảnh, ánh xạ cổng hoặc gắn kết khối lượng, dựa trên môi trường triển khai cụ thể và yêu cầu của bạn. Sau khi thực hiện bất kỳ thay đổi nào, vui lòng chạy lại `docker-compose up -d`. Bạn có thể tìm thấy danh sách đầy đủ các biến môi trường có sẵn [tại đây](https://docs.dify.ai/getting-started/install-self-hosted/environments).
|
||||
Nếu bạn cần tùy chỉnh cấu hình, vui lòng tham khảo các nhận xét trong tệp [.env.example](docker/.env.example) của chúng tôi và cập nhật các giá trị tương ứng trong tệp `.env` của bạn. Ngoài ra, bạn có thể cần điều chỉnh tệp `docker-compose.yaml`, chẳng hạn như thay đổi phiên bản hình ảnh, ánh xạ cổng hoặc gắn kết khối lượng, dựa trên môi trường triển khai cụ thể và yêu cầu của bạn. Sau khi thực hiện bất kỳ thay đổi nào, vui lòng chạy lại `docker-compose up -d`. Bạn có thể tìm thấy danh sách đầy đủ các biến môi trường có sẵn [tại đây](https://docs.dify.ai/getting-started/install-self-hosted/environments).
|
||||
|
||||
Nếu bạn muốn cấu hình một cài đặt có độ sẵn sàng cao, có các [Helm Charts](https://helm.sh/) và tệp YAML do cộng đồng đóng góp cho phép Dify được triển khai trên Kubernetes.
|
||||
|
||||
@ -164,7 +162,7 @@ Triển khai Dify lên AKS chỉ với một cú nhấp chuột bằng [Azure De
|
||||
|
||||
## Đóng góp
|
||||
|
||||
Đối với những người muốn đóng góp mã, xem [Hướng dẫn Đóng góp](./CONTRIBUTING.md) của chúng tôi.
|
||||
Đối với những người muốn đóng góp mã, xem [Hướng dẫn Đóng góp](https://github.com/langgenius/dify/blob/main/CONTRIBUTING/CONTRIBUTING_VI.md) của chúng tôi.
|
||||
Đồng thời, vui lòng xem xét hỗ trợ Dify bằng cách chia sẻ nó trên mạng xã hội và tại các sự kiện và hội nghị.
|
||||
|
||||
> Chúng tôi đang tìm kiếm người đóng góp để giúp dịch Dify sang các ngôn ngữ khác ngoài tiếng Trung hoặc tiếng Anh. Nếu bạn quan tâm đến việc giúp đỡ, vui lòng xem [README i18n](https://github.com/langgenius/dify/blob/main/web/i18n-config/README.md) để biết thêm thông tin và để lại bình luận cho chúng tôi trong kênh `global-users` của [Máy chủ Cộng đồng Discord](https://discord.gg/8Tpq4AcN9c) của chúng tôi.
|
||||
@ -178,7 +176,7 @@ Triển khai Dify lên AKS chỉ với một cú nhấp chuột bằng [Azure De
|
||||
## Cộng đồng & liên hệ
|
||||
|
||||
- [Thảo luận GitHub](https://github.com/langgenius/dify/discussions). Tốt nhất cho: chia sẻ phản hồi và đặt câu hỏi.
|
||||
- [Vấn đề GitHub](https://github.com/langgenius/dify/issues). Tốt nhất cho: lỗi bạn gặp phải khi sử dụng Dify.AI và đề xuất tính năng. Xem [Hướng dẫn Đóng góp](./CONTRIBUTING.md) của chúng tôi.
|
||||
- [Vấn đề GitHub](https://github.com/langgenius/dify/issues). Tốt nhất cho: lỗi bạn gặp phải khi sử dụng Dify.AI và đề xuất tính năng. Xem [Hướng dẫn Đóng góp](https://github.com/langgenius/dify/blob/main/CONTRIBUTING.md) của chúng tôi.
|
||||
- [Discord](https://discord.gg/FngNHpbcY7). Tốt nhất cho: chia sẻ ứng dụng của bạn và giao lưu với cộng đồng.
|
||||
- [X(Twitter)](https://twitter.com/dify_ai). Tốt nhất cho: chia sẻ ứng dụng của bạn và giao lưu với cộng đồng.
|
||||
|
||||
@ -192,4 +190,4 @@ Triển khai Dify lên AKS chỉ với một cú nhấp chuột bằng [Azure De
|
||||
|
||||
## Giấy phép
|
||||
|
||||
Kho lưu trữ này có sẵn theo [Giấy phép Mã nguồn Mở Dify](../../LICENSE), về cơ bản là Apache 2.0 với một vài hạn chế bổ sung.
|
||||
Kho lưu trữ này có sẵn theo [Giấy phép Mã nguồn Mở Dify](../LICENSE), về cơ bản là Apache 2.0 với một vài hạn chế bổ sung.
|
||||
@ -427,8 +427,8 @@ CODE_EXECUTION_POOL_MAX_KEEPALIVE_CONNECTIONS=20
|
||||
CODE_EXECUTION_POOL_KEEPALIVE_EXPIRY=5.0
|
||||
CODE_MAX_NUMBER=9223372036854775807
|
||||
CODE_MIN_NUMBER=-9223372036854775808
|
||||
CODE_MAX_STRING_LENGTH=400000
|
||||
TEMPLATE_TRANSFORM_MAX_LENGTH=400000
|
||||
CODE_MAX_STRING_LENGTH=80000
|
||||
TEMPLATE_TRANSFORM_MAX_LENGTH=80000
|
||||
CODE_MAX_STRING_ARRAY_LENGTH=30
|
||||
CODE_MAX_OBJECT_ARRAY_LENGTH=30
|
||||
CODE_MAX_NUMBER_ARRAY_LENGTH=1000
|
||||
|
||||
@ -50,7 +50,6 @@ def initialize_extensions(app: DifyApp):
|
||||
ext_commands,
|
||||
ext_compress,
|
||||
ext_database,
|
||||
ext_elasticsearch,
|
||||
ext_hosting_provider,
|
||||
ext_import_modules,
|
||||
ext_logging,
|
||||
@ -83,7 +82,6 @@ def initialize_extensions(app: DifyApp):
|
||||
ext_migrate,
|
||||
ext_redis,
|
||||
ext_storage,
|
||||
ext_elasticsearch,
|
||||
ext_celery,
|
||||
ext_login,
|
||||
ext_mail,
|
||||
|
||||
292
api/commands.py
292
api/commands.py
@ -1824,295 +1824,3 @@ def migrate_oss(
|
||||
except Exception as e:
|
||||
db.session.rollback()
|
||||
click.echo(click.style(f"Failed to update DB storage_type: {str(e)}", fg="red"))
|
||||
|
||||
|
||||
# Elasticsearch Migration Commands
|
||||
@click.group()
|
||||
def elasticsearch():
|
||||
"""Elasticsearch migration and management commands."""
|
||||
pass
|
||||
|
||||
|
||||
@elasticsearch.command()
|
||||
@click.option(
|
||||
"--tenant-id",
|
||||
help="Migrate data for specific tenant only",
|
||||
)
|
||||
@click.option(
|
||||
"--start-date",
|
||||
help="Start date for migration (YYYY-MM-DD format)",
|
||||
)
|
||||
@click.option(
|
||||
"--end-date",
|
||||
help="End date for migration (YYYY-MM-DD format)",
|
||||
)
|
||||
@click.option(
|
||||
"--data-type",
|
||||
type=click.Choice(["workflow_runs", "app_logs", "node_executions", "all"]),
|
||||
default="all",
|
||||
help="Type of data to migrate",
|
||||
)
|
||||
@click.option(
|
||||
"--batch-size",
|
||||
type=int,
|
||||
default=1000,
|
||||
help="Number of records to process in each batch",
|
||||
)
|
||||
@click.option(
|
||||
"--dry-run",
|
||||
is_flag=True,
|
||||
help="Perform a dry run without actually migrating data",
|
||||
)
|
||||
def migrate(
|
||||
tenant_id: str | None,
|
||||
start_date: str | None,
|
||||
end_date: str | None,
|
||||
data_type: str,
|
||||
batch_size: int,
|
||||
dry_run: bool,
|
||||
):
|
||||
"""
|
||||
Migrate workflow log data from PostgreSQL to Elasticsearch.
|
||||
"""
|
||||
from datetime import datetime
|
||||
|
||||
from extensions.ext_elasticsearch import elasticsearch as es_extension
|
||||
from services.elasticsearch_migration_service import ElasticsearchMigrationService
|
||||
|
||||
if not es_extension.is_available():
|
||||
click.echo("Error: Elasticsearch is not available. Please check your configuration.", err=True)
|
||||
return
|
||||
|
||||
# Parse dates
|
||||
start_dt = None
|
||||
end_dt = None
|
||||
|
||||
if start_date:
|
||||
try:
|
||||
start_dt = datetime.strptime(start_date, "%Y-%m-%d")
|
||||
except ValueError:
|
||||
click.echo(f"Error: Invalid start date format '{start_date}'. Use YYYY-MM-DD.", err=True)
|
||||
return
|
||||
|
||||
if end_date:
|
||||
try:
|
||||
end_dt = datetime.strptime(end_date, "%Y-%m-%d")
|
||||
except ValueError:
|
||||
click.echo(f"Error: Invalid end date format '{end_date}'. Use YYYY-MM-DD.", err=True)
|
||||
return
|
||||
|
||||
# Initialize migration service
|
||||
migration_service = ElasticsearchMigrationService(batch_size=batch_size)
|
||||
|
||||
click.echo(f"Starting {'dry run' if dry_run else 'migration'} to Elasticsearch...")
|
||||
click.echo(f"Tenant ID: {tenant_id or 'All tenants'}")
|
||||
click.echo(f"Date range: {start_date or 'No start'} to {end_date or 'No end'}")
|
||||
click.echo(f"Data type: {data_type}")
|
||||
click.echo(f"Batch size: {batch_size}")
|
||||
click.echo()
|
||||
|
||||
total_stats = {
|
||||
"workflow_runs": {},
|
||||
"app_logs": {},
|
||||
"node_executions": {},
|
||||
}
|
||||
|
||||
try:
|
||||
# Migrate workflow runs
|
||||
if data_type in ["workflow_runs", "all"]:
|
||||
click.echo("Migrating WorkflowRun data...")
|
||||
stats = migration_service.migrate_workflow_runs(
|
||||
tenant_id=tenant_id,
|
||||
start_date=start_dt,
|
||||
end_date=end_dt,
|
||||
dry_run=dry_run,
|
||||
)
|
||||
total_stats["workflow_runs"] = stats
|
||||
|
||||
click.echo(f" Total records: {stats['total_records']}")
|
||||
click.echo(f" Migrated: {stats['migrated_records']}")
|
||||
click.echo(f" Failed: {stats['failed_records']}")
|
||||
if stats.get("duration"):
|
||||
click.echo(f" Duration: {stats['duration']:.2f}s")
|
||||
click.echo()
|
||||
|
||||
# Migrate app logs
|
||||
if data_type in ["app_logs", "all"]:
|
||||
click.echo("Migrating WorkflowAppLog data...")
|
||||
stats = migration_service.migrate_workflow_app_logs(
|
||||
tenant_id=tenant_id,
|
||||
start_date=start_dt,
|
||||
end_date=end_dt,
|
||||
dry_run=dry_run,
|
||||
)
|
||||
total_stats["app_logs"] = stats
|
||||
|
||||
click.echo(f" Total records: {stats['total_records']}")
|
||||
click.echo(f" Migrated: {stats['migrated_records']}")
|
||||
click.echo(f" Failed: {stats['failed_records']}")
|
||||
if stats.get("duration"):
|
||||
click.echo(f" Duration: {stats['duration']:.2f}s")
|
||||
click.echo()
|
||||
|
||||
# Migrate node executions
|
||||
if data_type in ["node_executions", "all"]:
|
||||
click.echo("Migrating WorkflowNodeExecution data...")
|
||||
stats = migration_service.migrate_workflow_node_executions(
|
||||
tenant_id=tenant_id,
|
||||
start_date=start_dt,
|
||||
end_date=end_dt,
|
||||
dry_run=dry_run,
|
||||
)
|
||||
total_stats["node_executions"] = stats
|
||||
|
||||
click.echo(f" Total records: {stats['total_records']}")
|
||||
click.echo(f" Migrated: {stats['migrated_records']}")
|
||||
click.echo(f" Failed: {stats['failed_records']}")
|
||||
if stats.get("duration"):
|
||||
click.echo(f" Duration: {stats['duration']:.2f}s")
|
||||
click.echo()
|
||||
|
||||
# Summary
|
||||
total_migrated = sum(stats.get("migrated_records", 0) for stats in total_stats.values())
|
||||
total_failed = sum(stats.get("failed_records", 0) for stats in total_stats.values())
|
||||
|
||||
click.echo("Migration Summary:")
|
||||
click.echo(f" Total migrated: {total_migrated}")
|
||||
click.echo(f" Total failed: {total_failed}")
|
||||
|
||||
# Show errors if any
|
||||
all_errors = []
|
||||
for stats in total_stats.values():
|
||||
all_errors.extend(stats.get("errors", []))
|
||||
|
||||
if all_errors:
|
||||
click.echo(f" Errors ({len(all_errors)}):")
|
||||
for error in all_errors[:10]: # Show first 10 errors
|
||||
click.echo(f" - {error}")
|
||||
if len(all_errors) > 10:
|
||||
click.echo(f" ... and {len(all_errors) - 10} more errors")
|
||||
|
||||
if dry_run:
|
||||
click.echo("\nThis was a dry run. No data was actually migrated.")
|
||||
else:
|
||||
click.echo(f"\nMigration {'completed successfully' if total_failed == 0 else 'completed with errors'}!")
|
||||
|
||||
except Exception as e:
|
||||
click.echo(f"Error: Migration failed: {str(e)}", err=True)
|
||||
logger.exception("Migration failed")
|
||||
|
||||
|
||||
@elasticsearch.command()
|
||||
@click.option(
|
||||
"--tenant-id",
|
||||
required=True,
|
||||
help="Tenant ID to validate",
|
||||
)
|
||||
@click.option(
|
||||
"--sample-size",
|
||||
type=int,
|
||||
default=100,
|
||||
help="Number of records to sample for validation",
|
||||
)
|
||||
def validate(tenant_id: str, sample_size: int):
|
||||
"""
|
||||
Validate migrated data by comparing samples from PostgreSQL and Elasticsearch.
|
||||
"""
|
||||
from extensions.ext_elasticsearch import elasticsearch as es_extension
|
||||
from services.elasticsearch_migration_service import ElasticsearchMigrationService
|
||||
|
||||
if not es_extension.is_available():
|
||||
click.echo("Error: Elasticsearch is not available. Please check your configuration.", err=True)
|
||||
return
|
||||
|
||||
migration_service = ElasticsearchMigrationService()
|
||||
|
||||
click.echo(f"Validating migration for tenant: {tenant_id}")
|
||||
click.echo(f"Sample size: {sample_size}")
|
||||
click.echo()
|
||||
|
||||
try:
|
||||
results = migration_service.validate_migration(tenant_id, sample_size)
|
||||
|
||||
click.echo("Validation Results:")
|
||||
|
||||
for data_type, stats in results.items():
|
||||
if data_type == "errors":
|
||||
continue
|
||||
|
||||
click.echo(f"\n{data_type.replace('_', ' ').title()}:")
|
||||
click.echo(f" Total sampled: {stats['total']}")
|
||||
click.echo(f" Matched: {stats['matched']}")
|
||||
click.echo(f" Mismatched: {stats['mismatched']}")
|
||||
click.echo(f" Missing in ES: {stats['missing']}")
|
||||
|
||||
if stats['total'] > 0:
|
||||
accuracy = (stats['matched'] / stats['total']) * 100
|
||||
click.echo(f" Accuracy: {accuracy:.1f}%")
|
||||
|
||||
if results["errors"]:
|
||||
click.echo(f"\nValidation Errors ({len(results['errors'])}):")
|
||||
for error in results["errors"][:10]:
|
||||
click.echo(f" - {error}")
|
||||
if len(results["errors"]) > 10:
|
||||
click.echo(f" ... and {len(results['errors']) - 10} more errors")
|
||||
|
||||
except Exception as e:
|
||||
click.echo(f"Error: Validation failed: {str(e)}", err=True)
|
||||
logger.exception("Validation failed")
|
||||
|
||||
|
||||
@elasticsearch.command()
|
||||
def status():
|
||||
"""
|
||||
Check Elasticsearch connection and index status.
|
||||
"""
|
||||
from extensions.ext_elasticsearch import elasticsearch as es_extension
|
||||
|
||||
if not es_extension.is_available():
|
||||
click.echo("Error: Elasticsearch is not available. Please check your configuration.", err=True)
|
||||
return
|
||||
|
||||
try:
|
||||
es_client = es_extension.client
|
||||
|
||||
# Cluster health
|
||||
health = es_client.cluster.health()
|
||||
click.echo("Elasticsearch Cluster Status:")
|
||||
click.echo(f" Status: {health['status']}")
|
||||
click.echo(f" Nodes: {health['number_of_nodes']}")
|
||||
click.echo(f" Data nodes: {health['number_of_data_nodes']}")
|
||||
click.echo()
|
||||
|
||||
# Index information
|
||||
index_pattern = "dify-*"
|
||||
|
||||
try:
|
||||
indices = es_client.indices.get(index=index_pattern)
|
||||
|
||||
click.echo(f"Indices matching '{index_pattern}':")
|
||||
total_docs = 0
|
||||
total_size = 0
|
||||
|
||||
for index_name, index_info in indices.items():
|
||||
stats = es_client.indices.stats(index=index_name)
|
||||
docs = stats['indices'][index_name]['total']['docs']['count']
|
||||
size_bytes = stats['indices'][index_name]['total']['store']['size_in_bytes']
|
||||
size_mb = size_bytes / (1024 * 1024)
|
||||
|
||||
total_docs += docs
|
||||
total_size += size_mb
|
||||
|
||||
click.echo(f" {index_name}: {docs:,} docs, {size_mb:.1f} MB")
|
||||
|
||||
click.echo(f"\nTotal: {total_docs:,} documents, {total_size:.1f} MB")
|
||||
|
||||
except Exception as e:
|
||||
if "index_not_found_exception" in str(e):
|
||||
click.echo(f"No indices found matching pattern '{index_pattern}'")
|
||||
else:
|
||||
raise
|
||||
|
||||
except Exception as e:
|
||||
click.echo(f"Error: Failed to get Elasticsearch status: {str(e)}", err=True)
|
||||
logger.exception("Status check failed")
|
||||
|
||||
@ -150,7 +150,7 @@ class CodeExecutionSandboxConfig(BaseSettings):
|
||||
|
||||
CODE_MAX_STRING_LENGTH: PositiveInt = Field(
|
||||
description="Maximum allowed length for strings in code execution",
|
||||
default=400_000,
|
||||
default=80000,
|
||||
)
|
||||
|
||||
CODE_MAX_STRING_ARRAY_LENGTH: PositiveInt = Field(
|
||||
@ -582,11 +582,6 @@ class WorkflowConfig(BaseSettings):
|
||||
default=200 * 1024,
|
||||
)
|
||||
|
||||
TEMPLATE_TRANSFORM_MAX_LENGTH: PositiveInt = Field(
|
||||
description="Maximum number of characters allowed in Template Transform node output",
|
||||
default=400_000,
|
||||
)
|
||||
|
||||
# GraphEngine Worker Pool Configuration
|
||||
GRAPH_ENGINE_MIN_WORKERS: PositiveInt = Field(
|
||||
description="Minimum number of workers per GraphEngine instance",
|
||||
@ -659,67 +654,6 @@ class RepositoryConfig(BaseSettings):
|
||||
)
|
||||
|
||||
|
||||
class ElasticsearchConfig(BaseSettings):
|
||||
"""
|
||||
Configuration for Elasticsearch integration
|
||||
"""
|
||||
|
||||
ELASTICSEARCH_ENABLED: bool = Field(
|
||||
description="Enable Elasticsearch for workflow logs storage",
|
||||
default=False,
|
||||
)
|
||||
|
||||
ELASTICSEARCH_HOSTS: list[str] = Field(
|
||||
description="List of Elasticsearch hosts",
|
||||
default=["http://localhost:9200"],
|
||||
)
|
||||
|
||||
ELASTICSEARCH_USERNAME: str | None = Field(
|
||||
description="Elasticsearch username for authentication",
|
||||
default=None,
|
||||
)
|
||||
|
||||
ELASTICSEARCH_PASSWORD: str | None = Field(
|
||||
description="Elasticsearch password for authentication",
|
||||
default=None,
|
||||
)
|
||||
|
||||
ELASTICSEARCH_USE_SSL: bool = Field(
|
||||
description="Use SSL/TLS for Elasticsearch connections",
|
||||
default=False,
|
||||
)
|
||||
|
||||
ELASTICSEARCH_VERIFY_CERTS: bool = Field(
|
||||
description="Verify SSL certificates for Elasticsearch connections",
|
||||
default=True,
|
||||
)
|
||||
|
||||
ELASTICSEARCH_CA_CERTS: str | None = Field(
|
||||
description="Path to CA certificates file for Elasticsearch SSL verification",
|
||||
default=None,
|
||||
)
|
||||
|
||||
ELASTICSEARCH_TIMEOUT: int = Field(
|
||||
description="Elasticsearch request timeout in seconds",
|
||||
default=30,
|
||||
)
|
||||
|
||||
ELASTICSEARCH_MAX_RETRIES: int = Field(
|
||||
description="Maximum number of retries for Elasticsearch requests",
|
||||
default=3,
|
||||
)
|
||||
|
||||
ELASTICSEARCH_INDEX_PREFIX: str = Field(
|
||||
description="Prefix for Elasticsearch indices",
|
||||
default="dify",
|
||||
)
|
||||
|
||||
ELASTICSEARCH_RETENTION_DAYS: int = Field(
|
||||
description="Number of days to retain data in Elasticsearch",
|
||||
default=30,
|
||||
)
|
||||
|
||||
|
||||
class AuthConfig(BaseSettings):
|
||||
"""
|
||||
Configuration for authentication and OAuth
|
||||
@ -1169,7 +1103,6 @@ class FeatureConfig(
|
||||
AuthConfig, # Changed from OAuthConfig to AuthConfig
|
||||
BillingConfig,
|
||||
CodeExecutionSandboxConfig,
|
||||
ElasticsearchConfig,
|
||||
PluginConfig,
|
||||
MarketplaceConfig,
|
||||
DataSetConfig,
|
||||
|
||||
@ -1,5 +1,4 @@
|
||||
from configs import dify_config
|
||||
from libs.collection_utils import convert_to_lower_and_upper_set
|
||||
|
||||
HIDDEN_VALUE = "[__HIDDEN__]"
|
||||
UNKNOWN_VALUE = "[__UNKNOWN__]"
|
||||
@ -7,39 +6,24 @@ UUID_NIL = "00000000-0000-0000-0000-000000000000"
|
||||
|
||||
DEFAULT_FILE_NUMBER_LIMITS = 3
|
||||
|
||||
IMAGE_EXTENSIONS = convert_to_lower_and_upper_set({"jpg", "jpeg", "png", "webp", "gif", "svg"})
|
||||
IMAGE_EXTENSIONS = ["jpg", "jpeg", "png", "webp", "gif", "svg"]
|
||||
IMAGE_EXTENSIONS.extend([ext.upper() for ext in IMAGE_EXTENSIONS])
|
||||
|
||||
VIDEO_EXTENSIONS = convert_to_lower_and_upper_set({"mp4", "mov", "mpeg", "webm"})
|
||||
VIDEO_EXTENSIONS = ["mp4", "mov", "mpeg", "webm"]
|
||||
VIDEO_EXTENSIONS.extend([ext.upper() for ext in VIDEO_EXTENSIONS])
|
||||
|
||||
AUDIO_EXTENSIONS = convert_to_lower_and_upper_set({"mp3", "m4a", "wav", "amr", "mpga"})
|
||||
AUDIO_EXTENSIONS = ["mp3", "m4a", "wav", "amr", "mpga"]
|
||||
AUDIO_EXTENSIONS.extend([ext.upper() for ext in AUDIO_EXTENSIONS])
|
||||
|
||||
_doc_extensions: set[str]
|
||||
|
||||
_doc_extensions: list[str]
|
||||
if dify_config.ETL_TYPE == "Unstructured":
|
||||
_doc_extensions = {
|
||||
"txt",
|
||||
"markdown",
|
||||
"md",
|
||||
"mdx",
|
||||
"pdf",
|
||||
"html",
|
||||
"htm",
|
||||
"xlsx",
|
||||
"xls",
|
||||
"vtt",
|
||||
"properties",
|
||||
"doc",
|
||||
"docx",
|
||||
"csv",
|
||||
"eml",
|
||||
"msg",
|
||||
"pptx",
|
||||
"xml",
|
||||
"epub",
|
||||
}
|
||||
_doc_extensions = ["txt", "markdown", "md", "mdx", "pdf", "html", "htm", "xlsx", "xls", "vtt", "properties"]
|
||||
_doc_extensions.extend(("doc", "docx", "csv", "eml", "msg", "pptx", "xml", "epub"))
|
||||
if dify_config.UNSTRUCTURED_API_URL:
|
||||
_doc_extensions.add("ppt")
|
||||
_doc_extensions.append("ppt")
|
||||
else:
|
||||
_doc_extensions = {
|
||||
_doc_extensions = [
|
||||
"txt",
|
||||
"markdown",
|
||||
"md",
|
||||
@ -53,5 +37,5 @@ else:
|
||||
"csv",
|
||||
"vtt",
|
||||
"properties",
|
||||
}
|
||||
DOCUMENT_EXTENSIONS: set[str] = convert_to_lower_and_upper_set(_doc_extensions)
|
||||
]
|
||||
DOCUMENT_EXTENSIONS = _doc_extensions + [ext.upper() for ext in _doc_extensions]
|
||||
|
||||
@ -19,7 +19,6 @@ from core.ops.ops_trace_manager import OpsTraceManager
|
||||
from extensions.ext_database import db
|
||||
from fields.app_fields import app_detail_fields, app_detail_fields_with_site, app_pagination_fields
|
||||
from libs.login import login_required
|
||||
from libs.validators import validate_description_length
|
||||
from models import Account, App
|
||||
from services.app_dsl_service import AppDslService, ImportMode
|
||||
from services.app_service import AppService
|
||||
@ -29,6 +28,12 @@ from services.feature_service import FeatureService
|
||||
ALLOW_CREATE_APP_MODES = ["chat", "agent-chat", "advanced-chat", "workflow", "completion"]
|
||||
|
||||
|
||||
def _validate_description_length(description):
|
||||
if description and len(description) > 400:
|
||||
raise ValueError("Description cannot exceed 400 characters.")
|
||||
return description
|
||||
|
||||
|
||||
@console_ns.route("/apps")
|
||||
class AppListApi(Resource):
|
||||
@api.doc("list_apps")
|
||||
@ -133,7 +138,7 @@ class AppListApi(Resource):
|
||||
"""Create app"""
|
||||
parser = reqparse.RequestParser()
|
||||
parser.add_argument("name", type=str, required=True, location="json")
|
||||
parser.add_argument("description", type=validate_description_length, location="json")
|
||||
parser.add_argument("description", type=_validate_description_length, location="json")
|
||||
parser.add_argument("mode", type=str, choices=ALLOW_CREATE_APP_MODES, location="json")
|
||||
parser.add_argument("icon_type", type=str, location="json")
|
||||
parser.add_argument("icon", type=str, location="json")
|
||||
@ -214,7 +219,7 @@ class AppApi(Resource):
|
||||
|
||||
parser = reqparse.RequestParser()
|
||||
parser.add_argument("name", type=str, required=True, nullable=False, location="json")
|
||||
parser.add_argument("description", type=validate_description_length, location="json")
|
||||
parser.add_argument("description", type=_validate_description_length, location="json")
|
||||
parser.add_argument("icon_type", type=str, location="json")
|
||||
parser.add_argument("icon", type=str, location="json")
|
||||
parser.add_argument("icon_background", type=str, location="json")
|
||||
@ -292,7 +297,7 @@ class AppCopyApi(Resource):
|
||||
|
||||
parser = reqparse.RequestParser()
|
||||
parser.add_argument("name", type=str, location="json")
|
||||
parser.add_argument("description", type=validate_description_length, location="json")
|
||||
parser.add_argument("description", type=_validate_description_length, location="json")
|
||||
parser.add_argument("icon_type", type=str, location="json")
|
||||
parser.add_argument("icon", type=str, location="json")
|
||||
parser.add_argument("icon_background", type=str, location="json")
|
||||
|
||||
@ -31,7 +31,6 @@ from fields.app_fields import related_app_list
|
||||
from fields.dataset_fields import dataset_detail_fields, dataset_query_detail_fields
|
||||
from fields.document_fields import document_status_fields
|
||||
from libs.login import login_required
|
||||
from libs.validators import validate_description_length
|
||||
from models import ApiToken, Dataset, Document, DocumentSegment, UploadFile
|
||||
from models.account import Account
|
||||
from models.dataset import DatasetPermissionEnum
|
||||
@ -45,6 +44,12 @@ def _validate_name(name: str) -> str:
|
||||
return name
|
||||
|
||||
|
||||
def _validate_description_length(description):
|
||||
if description and len(description) > 400:
|
||||
raise ValueError("Description cannot exceed 400 characters.")
|
||||
return description
|
||||
|
||||
|
||||
@console_ns.route("/datasets")
|
||||
class DatasetListApi(Resource):
|
||||
@api.doc("get_datasets")
|
||||
@ -144,7 +149,7 @@ class DatasetListApi(Resource):
|
||||
)
|
||||
parser.add_argument(
|
||||
"description",
|
||||
type=validate_description_length,
|
||||
type=_validate_description_length,
|
||||
nullable=True,
|
||||
required=False,
|
||||
default="",
|
||||
@ -285,7 +290,7 @@ class DatasetApi(Resource):
|
||||
help="type is required. Name must be between 1 to 40 characters.",
|
||||
type=_validate_name,
|
||||
)
|
||||
parser.add_argument("description", location="json", store_missing=False, type=validate_description_length)
|
||||
parser.add_argument("description", location="json", store_missing=False, type=_validate_description_length)
|
||||
parser.add_argument(
|
||||
"indexing_technique",
|
||||
type=str,
|
||||
|
||||
@ -1,3 +1,4 @@
|
||||
from fastapi.encoders import jsonable_encoder
|
||||
from flask import make_response, redirect, request
|
||||
from flask_login import current_user
|
||||
from flask_restx import Resource, reqparse
|
||||
@ -10,7 +11,6 @@ from controllers.console.wraps import (
|
||||
setup_required,
|
||||
)
|
||||
from core.model_runtime.errors.validate import CredentialsValidateFailedError
|
||||
from core.model_runtime.utils.encoders import jsonable_encoder
|
||||
from core.plugin.impl.oauth import OAuthHandler
|
||||
from libs.helper import StrLen
|
||||
from libs.login import login_required
|
||||
|
||||
@ -17,7 +17,6 @@ from core.provider_manager import ProviderManager
|
||||
from fields.dataset_fields import dataset_detail_fields
|
||||
from fields.tag_fields import build_dataset_tag_fields
|
||||
from libs.login import current_user
|
||||
from libs.validators import validate_description_length
|
||||
from models.account import Account
|
||||
from models.dataset import Dataset, DatasetPermissionEnum
|
||||
from models.provider_ids import ModelProviderID
|
||||
@ -32,6 +31,12 @@ def _validate_name(name):
|
||||
return name
|
||||
|
||||
|
||||
def _validate_description_length(description):
|
||||
if description and len(description) > 400:
|
||||
raise ValueError("Description cannot exceed 400 characters.")
|
||||
return description
|
||||
|
||||
|
||||
# Define parsers for dataset operations
|
||||
dataset_create_parser = reqparse.RequestParser()
|
||||
dataset_create_parser.add_argument(
|
||||
@ -43,7 +48,7 @@ dataset_create_parser.add_argument(
|
||||
)
|
||||
dataset_create_parser.add_argument(
|
||||
"description",
|
||||
type=validate_description_length,
|
||||
type=_validate_description_length,
|
||||
nullable=True,
|
||||
required=False,
|
||||
default="",
|
||||
@ -96,7 +101,7 @@ dataset_update_parser.add_argument(
|
||||
type=_validate_name,
|
||||
)
|
||||
dataset_update_parser.add_argument(
|
||||
"description", location="json", store_missing=False, type=validate_description_length
|
||||
"description", location="json", store_missing=False, type=_validate_description_length
|
||||
)
|
||||
dataset_update_parser.add_argument(
|
||||
"indexing_technique",
|
||||
|
||||
@ -1,5 +1,4 @@
|
||||
import uuid
|
||||
from typing import Literal, cast
|
||||
|
||||
from core.app.app_config.entities import (
|
||||
DatasetEntity,
|
||||
@ -75,9 +74,6 @@ class DatasetConfigManager:
|
||||
return None
|
||||
query_variable = config.get("dataset_query_variable")
|
||||
|
||||
metadata_model_config_dict = dataset_configs.get("metadata_model_config")
|
||||
metadata_filtering_conditions_dict = dataset_configs.get("metadata_filtering_conditions")
|
||||
|
||||
if dataset_configs["retrieval_model"] == "single":
|
||||
return DatasetEntity(
|
||||
dataset_ids=dataset_ids,
|
||||
@ -86,23 +82,18 @@ class DatasetConfigManager:
|
||||
retrieve_strategy=DatasetRetrieveConfigEntity.RetrieveStrategy.value_of(
|
||||
dataset_configs["retrieval_model"]
|
||||
),
|
||||
metadata_filtering_mode=cast(
|
||||
Literal["disabled", "automatic", "manual"],
|
||||
dataset_configs.get("metadata_filtering_mode", "disabled"),
|
||||
),
|
||||
metadata_model_config=ModelConfig(**metadata_model_config_dict)
|
||||
if isinstance(metadata_model_config_dict, dict)
|
||||
metadata_filtering_mode=dataset_configs.get("metadata_filtering_mode", "disabled"),
|
||||
metadata_model_config=ModelConfig(**dataset_configs.get("metadata_model_config"))
|
||||
if dataset_configs.get("metadata_model_config")
|
||||
else None,
|
||||
metadata_filtering_conditions=MetadataFilteringCondition(**metadata_filtering_conditions_dict)
|
||||
if isinstance(metadata_filtering_conditions_dict, dict)
|
||||
metadata_filtering_conditions=MetadataFilteringCondition(
|
||||
**dataset_configs.get("metadata_filtering_conditions", {})
|
||||
)
|
||||
if dataset_configs.get("metadata_filtering_conditions")
|
||||
else None,
|
||||
),
|
||||
)
|
||||
else:
|
||||
score_threshold_val = dataset_configs.get("score_threshold")
|
||||
reranking_model_val = dataset_configs.get("reranking_model")
|
||||
weights_val = dataset_configs.get("weights")
|
||||
|
||||
return DatasetEntity(
|
||||
dataset_ids=dataset_ids,
|
||||
retrieve_config=DatasetRetrieveConfigEntity(
|
||||
@ -110,23 +101,22 @@ class DatasetConfigManager:
|
||||
retrieve_strategy=DatasetRetrieveConfigEntity.RetrieveStrategy.value_of(
|
||||
dataset_configs["retrieval_model"]
|
||||
),
|
||||
top_k=int(dataset_configs.get("top_k", 4)),
|
||||
score_threshold=float(score_threshold_val)
|
||||
if dataset_configs.get("score_threshold_enabled", False) and score_threshold_val is not None
|
||||
top_k=dataset_configs.get("top_k", 4),
|
||||
score_threshold=dataset_configs.get("score_threshold")
|
||||
if dataset_configs.get("score_threshold_enabled", False)
|
||||
else None,
|
||||
reranking_model=reranking_model_val if isinstance(reranking_model_val, dict) else None,
|
||||
weights=weights_val if isinstance(weights_val, dict) else None,
|
||||
reranking_enabled=bool(dataset_configs.get("reranking_enabled", True)),
|
||||
reranking_model=dataset_configs.get("reranking_model"),
|
||||
weights=dataset_configs.get("weights"),
|
||||
reranking_enabled=dataset_configs.get("reranking_enabled", True),
|
||||
rerank_mode=dataset_configs.get("reranking_mode", "reranking_model"),
|
||||
metadata_filtering_mode=cast(
|
||||
Literal["disabled", "automatic", "manual"],
|
||||
dataset_configs.get("metadata_filtering_mode", "disabled"),
|
||||
),
|
||||
metadata_model_config=ModelConfig(**metadata_model_config_dict)
|
||||
if isinstance(metadata_model_config_dict, dict)
|
||||
metadata_filtering_mode=dataset_configs.get("metadata_filtering_mode", "disabled"),
|
||||
metadata_model_config=ModelConfig(**dataset_configs.get("metadata_model_config"))
|
||||
if dataset_configs.get("metadata_model_config")
|
||||
else None,
|
||||
metadata_filtering_conditions=MetadataFilteringCondition(**metadata_filtering_conditions_dict)
|
||||
if isinstance(metadata_filtering_conditions_dict, dict)
|
||||
metadata_filtering_conditions=MetadataFilteringCondition(
|
||||
**dataset_configs.get("metadata_filtering_conditions", {})
|
||||
)
|
||||
if dataset_configs.get("metadata_filtering_conditions")
|
||||
else None,
|
||||
),
|
||||
)
|
||||
@ -144,17 +134,18 @@ class DatasetConfigManager:
|
||||
config = cls.extract_dataset_config_for_legacy_compatibility(tenant_id, app_mode, config)
|
||||
|
||||
# dataset_configs
|
||||
if "dataset_configs" not in config or not config.get("dataset_configs"):
|
||||
config["dataset_configs"] = {}
|
||||
config["dataset_configs"]["retrieval_model"] = config["dataset_configs"].get("retrieval_model", "single")
|
||||
if not config.get("dataset_configs"):
|
||||
config["dataset_configs"] = {"retrieval_model": "single"}
|
||||
|
||||
if not isinstance(config["dataset_configs"], dict):
|
||||
raise ValueError("dataset_configs must be of object type")
|
||||
|
||||
if "datasets" not in config["dataset_configs"] or not config["dataset_configs"].get("datasets"):
|
||||
if not config["dataset_configs"].get("datasets"):
|
||||
config["dataset_configs"]["datasets"] = {"strategy": "router", "datasets": []}
|
||||
|
||||
need_manual_query_datasets = config.get("dataset_configs", {}).get("datasets", {}).get("datasets")
|
||||
need_manual_query_datasets = config.get("dataset_configs") and config["dataset_configs"].get(
|
||||
"datasets", {}
|
||||
).get("datasets")
|
||||
|
||||
if need_manual_query_datasets and app_mode == AppMode.COMPLETION:
|
||||
# Only check when mode is completion
|
||||
@ -175,8 +166,8 @@ class DatasetConfigManager:
|
||||
:param config: app model config args
|
||||
"""
|
||||
# Extract dataset config for legacy compatibility
|
||||
if "agent_mode" not in config or not config.get("agent_mode"):
|
||||
config["agent_mode"] = {}
|
||||
if not config.get("agent_mode"):
|
||||
config["agent_mode"] = {"enabled": False, "tools": []}
|
||||
|
||||
if not isinstance(config["agent_mode"], dict):
|
||||
raise ValueError("agent_mode must be of object type")
|
||||
@ -189,22 +180,19 @@ class DatasetConfigManager:
|
||||
raise ValueError("enabled in agent_mode must be of boolean type")
|
||||
|
||||
# tools
|
||||
if "tools" not in config["agent_mode"] or not config["agent_mode"].get("tools"):
|
||||
if not config["agent_mode"].get("tools"):
|
||||
config["agent_mode"]["tools"] = []
|
||||
|
||||
if not isinstance(config["agent_mode"]["tools"], list):
|
||||
raise ValueError("tools in agent_mode must be a list of objects")
|
||||
|
||||
# strategy
|
||||
if "strategy" not in config["agent_mode"] or not config["agent_mode"].get("strategy"):
|
||||
if not config["agent_mode"].get("strategy"):
|
||||
config["agent_mode"]["strategy"] = PlanningStrategy.ROUTER.value
|
||||
|
||||
has_datasets = False
|
||||
if config.get("agent_mode", {}).get("strategy") in {
|
||||
PlanningStrategy.ROUTER.value,
|
||||
PlanningStrategy.REACT_ROUTER.value,
|
||||
}:
|
||||
for tool in config.get("agent_mode", {}).get("tools", []):
|
||||
if config["agent_mode"]["strategy"] in {PlanningStrategy.ROUTER.value, PlanningStrategy.REACT_ROUTER.value}:
|
||||
for tool in config["agent_mode"]["tools"]:
|
||||
key = list(tool.keys())[0]
|
||||
if key == "dataset":
|
||||
# old style, use tool name as key
|
||||
@ -229,7 +217,7 @@ class DatasetConfigManager:
|
||||
|
||||
has_datasets = True
|
||||
|
||||
need_manual_query_datasets = has_datasets and config.get("agent_mode", {}).get("enabled")
|
||||
need_manual_query_datasets = has_datasets and config["agent_mode"]["enabled"]
|
||||
|
||||
if need_manual_query_datasets and app_mode == AppMode.COMPLETION:
|
||||
# Only check when mode is completion
|
||||
|
||||
@ -107,6 +107,7 @@ class MessageCycleManager:
|
||||
if dify_config.DEBUG:
|
||||
logger.exception("generate conversation name failed, conversation_id: %s", conversation_id)
|
||||
|
||||
db.session.merge(conversation)
|
||||
db.session.commit()
|
||||
db.session.close()
|
||||
|
||||
|
||||
@ -1,6 +1,7 @@
|
||||
from typing import TYPE_CHECKING, Any, Optional
|
||||
|
||||
from pydantic import BaseModel, Field
|
||||
from openai import BaseModel
|
||||
from pydantic import Field
|
||||
|
||||
# Import InvokeFrom locally to avoid circular import
|
||||
from core.app.entities.app_invoke_entities import InvokeFrom
|
||||
|
||||
@ -1,238 +0,0 @@
|
||||
"""
|
||||
Elasticsearch implementation of the WorkflowExecutionRepository.
|
||||
|
||||
This implementation stores workflow execution data in Elasticsearch for better
|
||||
performance and scalability compared to PostgreSQL storage.
|
||||
"""
|
||||
|
||||
import logging
|
||||
from datetime import datetime
|
||||
from typing import Any, Optional, Union
|
||||
|
||||
from sqlalchemy.engine import Engine
|
||||
from sqlalchemy.orm import sessionmaker
|
||||
|
||||
from core.workflow.entities import WorkflowExecution
|
||||
from core.workflow.repositories.workflow_execution_repository import WorkflowExecutionRepository
|
||||
from libs.helper import extract_tenant_id
|
||||
from models import Account, CreatorUserRole, EndUser
|
||||
from models.enums import WorkflowRunTriggeredFrom
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class ElasticsearchWorkflowExecutionRepository(WorkflowExecutionRepository):
|
||||
"""
|
||||
Elasticsearch implementation of the WorkflowExecutionRepository interface.
|
||||
|
||||
This implementation provides:
|
||||
- High-performance workflow execution storage
|
||||
- Time-series data optimization with date-based index rotation
|
||||
- Multi-tenant data isolation
|
||||
- Advanced search and analytics capabilities
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
session_factory: Union[sessionmaker, Engine],
|
||||
user: Union[Account, EndUser],
|
||||
app_id: str,
|
||||
triggered_from: WorkflowRunTriggeredFrom,
|
||||
index_prefix: str = "dify-workflow-executions",
|
||||
):
|
||||
"""
|
||||
Initialize the repository with Elasticsearch client and context information.
|
||||
|
||||
Args:
|
||||
session_factory: SQLAlchemy sessionmaker or engine (for compatibility with factory pattern)
|
||||
user: Account or EndUser object containing tenant_id, user ID, and role information
|
||||
app_id: App ID for filtering by application
|
||||
triggered_from: Source of the execution trigger
|
||||
index_prefix: Prefix for Elasticsearch indices
|
||||
"""
|
||||
# Get Elasticsearch client from global extension
|
||||
from extensions.ext_elasticsearch import elasticsearch as es_extension
|
||||
|
||||
self._es_client = es_extension.client
|
||||
if not self._es_client:
|
||||
raise ValueError("Elasticsearch client is not available. Please check your configuration.")
|
||||
|
||||
self._index_prefix = index_prefix
|
||||
|
||||
# Extract tenant_id from user
|
||||
tenant_id = extract_tenant_id(user)
|
||||
if not tenant_id:
|
||||
raise ValueError("User must have a tenant_id or current_tenant_id")
|
||||
self._tenant_id = tenant_id
|
||||
|
||||
# Store app context
|
||||
self._app_id = app_id
|
||||
|
||||
# Extract user context
|
||||
self._triggered_from = triggered_from
|
||||
self._creator_user_id = user.id
|
||||
|
||||
# Determine user role based on user type
|
||||
self._creator_user_role = CreatorUserRole.ACCOUNT if isinstance(user, Account) else CreatorUserRole.END_USER
|
||||
|
||||
# Ensure index template exists
|
||||
self._ensure_index_template()
|
||||
|
||||
def _get_index_name(self, date: Optional[datetime] = None) -> str:
|
||||
"""
|
||||
Generate index name with date-based rotation for better performance.
|
||||
|
||||
Args:
|
||||
date: Date for index name generation, defaults to current date
|
||||
|
||||
Returns:
|
||||
Index name in format: {prefix}-{tenant_id}-{YYYY.MM}
|
||||
"""
|
||||
if date is None:
|
||||
date = datetime.utcnow()
|
||||
|
||||
return f"{self._index_prefix}-{self._tenant_id}-{date.strftime('%Y.%m')}"
|
||||
|
||||
def _ensure_index_template(self):
|
||||
"""
|
||||
Ensure the index template exists for proper mapping and settings.
|
||||
"""
|
||||
template_name = f"{self._index_prefix}-template"
|
||||
template_body = {
|
||||
"index_patterns": [f"{self._index_prefix}-*"],
|
||||
"template": {
|
||||
"settings": {
|
||||
"number_of_shards": 1,
|
||||
"number_of_replicas": 0,
|
||||
"index.refresh_interval": "5s",
|
||||
"index.mapping.total_fields.limit": 2000,
|
||||
},
|
||||
"mappings": {
|
||||
"properties": {
|
||||
"id": {"type": "keyword"},
|
||||
"tenant_id": {"type": "keyword"},
|
||||
"app_id": {"type": "keyword"},
|
||||
"workflow_id": {"type": "keyword"},
|
||||
"workflow_version": {"type": "keyword"},
|
||||
"workflow_type": {"type": "keyword"},
|
||||
"triggered_from": {"type": "keyword"},
|
||||
"inputs": {"type": "object", "enabled": False},
|
||||
"outputs": {"type": "object", "enabled": False},
|
||||
"status": {"type": "keyword"},
|
||||
"error_message": {"type": "text"},
|
||||
"elapsed_time": {"type": "float"},
|
||||
"total_tokens": {"type": "long"},
|
||||
"total_steps": {"type": "integer"},
|
||||
"exceptions_count": {"type": "integer"},
|
||||
"created_by_role": {"type": "keyword"},
|
||||
"created_by": {"type": "keyword"},
|
||||
"started_at": {"type": "date"},
|
||||
"finished_at": {"type": "date"},
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
try:
|
||||
self._es_client.indices.put_index_template(
|
||||
name=template_name,
|
||||
body=template_body
|
||||
)
|
||||
logger.info("Index template %s created/updated successfully", template_name)
|
||||
except Exception as e:
|
||||
logger.error("Failed to create index template %s: %s", template_name, e)
|
||||
raise
|
||||
|
||||
def _serialize_complex_data(self, data: Any) -> Any:
|
||||
"""
|
||||
Serialize complex data structures to JSON-serializable format.
|
||||
|
||||
Args:
|
||||
data: Data to serialize
|
||||
|
||||
Returns:
|
||||
JSON-serializable data
|
||||
"""
|
||||
if data is None:
|
||||
return None
|
||||
|
||||
# Use Dify's existing JSON encoder for complex objects
|
||||
from core.model_runtime.utils.encoders import jsonable_encoder
|
||||
|
||||
try:
|
||||
return jsonable_encoder(data)
|
||||
except Exception as e:
|
||||
logger.warning("Failed to serialize complex data, using string representation: %s", e)
|
||||
return str(data)
|
||||
|
||||
def _to_workflow_run_document(self, execution: WorkflowExecution) -> dict[str, Any]:
|
||||
"""
|
||||
Convert WorkflowExecution domain entity to WorkflowRun-compatible document.
|
||||
This follows the same logic as SQLAlchemy implementation.
|
||||
|
||||
Args:
|
||||
execution: The domain entity to convert
|
||||
|
||||
Returns:
|
||||
Dictionary representing the WorkflowRun document for Elasticsearch
|
||||
"""
|
||||
# Calculate elapsed time (same logic as SQL implementation)
|
||||
elapsed_time = 0.0
|
||||
if execution.finished_at:
|
||||
elapsed_time = (execution.finished_at - execution.started_at).total_seconds()
|
||||
|
||||
doc = {
|
||||
"id": execution.id_,
|
||||
"tenant_id": self._tenant_id,
|
||||
"app_id": self._app_id,
|
||||
"workflow_id": execution.workflow_id,
|
||||
"type": execution.workflow_type.value,
|
||||
"triggered_from": self._triggered_from.value,
|
||||
"version": execution.workflow_version,
|
||||
"graph": self._serialize_complex_data(execution.graph),
|
||||
"inputs": self._serialize_complex_data(execution.inputs),
|
||||
"status": execution.status.value,
|
||||
"outputs": self._serialize_complex_data(execution.outputs),
|
||||
"error": execution.error_message or None,
|
||||
"elapsed_time": elapsed_time,
|
||||
"total_tokens": execution.total_tokens,
|
||||
"total_steps": execution.total_steps,
|
||||
"created_by_role": self._creator_user_role.value,
|
||||
"created_by": self._creator_user_id,
|
||||
"created_at": execution.started_at.isoformat() if execution.started_at else None,
|
||||
"finished_at": execution.finished_at.isoformat() if execution.finished_at else None,
|
||||
"exceptions_count": execution.exceptions_count,
|
||||
}
|
||||
|
||||
# Remove None values to reduce storage size
|
||||
return {k: v for k, v in doc.items() if v is not None}
|
||||
|
||||
def save(self, execution: WorkflowExecution) -> None:
|
||||
"""
|
||||
Save or update a WorkflowExecution instance to Elasticsearch.
|
||||
|
||||
Following the SQL implementation pattern, this saves the WorkflowExecution
|
||||
as WorkflowRun-compatible data that APIs can consume.
|
||||
|
||||
Args:
|
||||
execution: The WorkflowExecution instance to save or update
|
||||
"""
|
||||
try:
|
||||
# Convert to WorkflowRun-compatible document (same as SQL implementation)
|
||||
run_doc = self._to_workflow_run_document(execution)
|
||||
|
||||
# Save to workflow-runs index (this is what APIs query)
|
||||
run_index = f"dify-workflow-runs-{self._tenant_id}-{execution.started_at.strftime('%Y.%m')}"
|
||||
|
||||
self._es_client.index(
|
||||
index=run_index,
|
||||
id=execution.id_,
|
||||
body=run_doc,
|
||||
refresh="wait_for" # Ensure document is searchable immediately
|
||||
)
|
||||
|
||||
logger.debug(f"Saved workflow execution {execution.id_} as WorkflowRun to index {run_index}")
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to save workflow execution {execution.id_}: {e}")
|
||||
raise
|
||||
@ -1,403 +0,0 @@
|
||||
"""
|
||||
Elasticsearch implementation of the WorkflowNodeExecutionRepository.
|
||||
|
||||
This implementation stores workflow node execution logs in Elasticsearch for better
|
||||
performance and scalability compared to PostgreSQL storage.
|
||||
"""
|
||||
|
||||
import logging
|
||||
from collections.abc import Sequence
|
||||
from datetime import datetime
|
||||
from typing import Any, Optional, Union
|
||||
|
||||
from elasticsearch.exceptions import NotFoundError
|
||||
from sqlalchemy.engine import Engine
|
||||
from sqlalchemy.orm import sessionmaker
|
||||
|
||||
from core.workflow.entities.workflow_node_execution import WorkflowNodeExecution
|
||||
from core.workflow.enums import WorkflowNodeExecutionStatus
|
||||
from core.workflow.repositories.workflow_node_execution_repository import (
|
||||
OrderConfig,
|
||||
WorkflowNodeExecutionRepository,
|
||||
)
|
||||
from libs.helper import extract_tenant_id
|
||||
from models import Account, CreatorUserRole, EndUser
|
||||
from models.workflow import WorkflowNodeExecutionTriggeredFrom
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class ElasticsearchWorkflowNodeExecutionRepository(WorkflowNodeExecutionRepository):
|
||||
"""
|
||||
Elasticsearch implementation of the WorkflowNodeExecutionRepository interface.
|
||||
|
||||
This implementation provides:
|
||||
- High-performance log storage and retrieval
|
||||
- Full-text search capabilities
|
||||
- Time-series data optimization
|
||||
- Automatic index management with date-based rotation
|
||||
- Multi-tenancy support through index patterns
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
session_factory: Union[sessionmaker, Engine],
|
||||
user: Union[Account, EndUser],
|
||||
app_id: str | None,
|
||||
triggered_from: WorkflowNodeExecutionTriggeredFrom | None,
|
||||
index_prefix: str = "dify-workflow-node-executions",
|
||||
):
|
||||
"""
|
||||
Initialize the repository with Elasticsearch client and context information.
|
||||
|
||||
Args:
|
||||
session_factory: SQLAlchemy sessionmaker or engine (for compatibility with factory pattern)
|
||||
user: Account or EndUser object containing tenant_id, user ID, and role information
|
||||
app_id: App ID for filtering by application (can be None)
|
||||
triggered_from: Source of the execution trigger (SINGLE_STEP or WORKFLOW_RUN)
|
||||
index_prefix: Prefix for Elasticsearch indices
|
||||
"""
|
||||
# Get Elasticsearch client from global extension
|
||||
from extensions.ext_elasticsearch import elasticsearch as es_extension
|
||||
|
||||
self._es_client = es_extension.client
|
||||
if not self._es_client:
|
||||
raise ValueError("Elasticsearch client is not available. Please check your configuration.")
|
||||
|
||||
self._index_prefix = index_prefix
|
||||
|
||||
# Extract tenant_id from user
|
||||
tenant_id = extract_tenant_id(user)
|
||||
if not tenant_id:
|
||||
raise ValueError("User must have a tenant_id or current_tenant_id")
|
||||
self._tenant_id = tenant_id
|
||||
|
||||
# Store app context
|
||||
self._app_id = app_id
|
||||
|
||||
# Extract user context
|
||||
self._triggered_from = triggered_from
|
||||
self._creator_user_id = user.id
|
||||
|
||||
# Determine user role based on user type
|
||||
self._creator_user_role = CreatorUserRole.ACCOUNT if isinstance(user, Account) else CreatorUserRole.END_USER
|
||||
|
||||
# In-memory cache for workflow node executions
|
||||
self._execution_cache: dict[str, WorkflowNodeExecution] = {}
|
||||
|
||||
# Ensure index template exists
|
||||
self._ensure_index_template()
|
||||
|
||||
def _get_index_name(self, date: Optional[datetime] = None) -> str:
|
||||
"""
|
||||
Generate index name with date-based rotation for better performance.
|
||||
|
||||
Args:
|
||||
date: Date for index name generation, defaults to current date
|
||||
|
||||
Returns:
|
||||
Index name in format: {prefix}-{tenant_id}-{YYYY.MM}
|
||||
"""
|
||||
if date is None:
|
||||
date = datetime.utcnow()
|
||||
|
||||
return f"{self._index_prefix}-{self._tenant_id}-{date.strftime('%Y.%m')}"
|
||||
|
||||
def _ensure_index_template(self):
|
||||
"""
|
||||
Ensure the index template exists for proper mapping and settings.
|
||||
"""
|
||||
template_name = f"{self._index_prefix}-template"
|
||||
template_body = {
|
||||
"index_patterns": [f"{self._index_prefix}-*"],
|
||||
"template": {
|
||||
"settings": {
|
||||
"number_of_shards": 1,
|
||||
"number_of_replicas": 0,
|
||||
"index.refresh_interval": "5s",
|
||||
"index.mapping.total_fields.limit": 2000,
|
||||
},
|
||||
"mappings": {
|
||||
"properties": {
|
||||
"id": {"type": "keyword"},
|
||||
"tenant_id": {"type": "keyword"},
|
||||
"app_id": {"type": "keyword"},
|
||||
"workflow_id": {"type": "keyword"},
|
||||
"workflow_execution_id": {"type": "keyword"},
|
||||
"node_execution_id": {"type": "keyword"},
|
||||
"triggered_from": {"type": "keyword"},
|
||||
"index": {"type": "integer"},
|
||||
"predecessor_node_id": {"type": "keyword"},
|
||||
"node_id": {"type": "keyword"},
|
||||
"node_type": {"type": "keyword"},
|
||||
"title": {"type": "text", "fields": {"keyword": {"type": "keyword"}}},
|
||||
"inputs": {"type": "object", "enabled": False},
|
||||
"process_data": {"type": "object", "enabled": False},
|
||||
"outputs": {"type": "object", "enabled": False},
|
||||
"status": {"type": "keyword"},
|
||||
"error": {"type": "text"},
|
||||
"elapsed_time": {"type": "float"},
|
||||
"metadata": {"type": "object", "enabled": False},
|
||||
"created_at": {"type": "date"},
|
||||
"finished_at": {"type": "date"},
|
||||
"created_by_role": {"type": "keyword"},
|
||||
"created_by": {"type": "keyword"},
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
try:
|
||||
self._es_client.indices.put_index_template(
|
||||
name=template_name,
|
||||
body=template_body
|
||||
)
|
||||
logger.info("Index template %s created/updated successfully", template_name)
|
||||
except Exception as e:
|
||||
logger.error("Failed to create index template %s: %s", template_name, e)
|
||||
raise
|
||||
|
||||
def _serialize_complex_data(self, data: Any) -> Any:
|
||||
"""
|
||||
Serialize complex data structures to JSON-serializable format.
|
||||
|
||||
Args:
|
||||
data: Data to serialize
|
||||
|
||||
Returns:
|
||||
JSON-serializable data
|
||||
"""
|
||||
if data is None:
|
||||
return None
|
||||
|
||||
# Use Dify's existing JSON encoder for complex objects
|
||||
from core.model_runtime.utils.encoders import jsonable_encoder
|
||||
|
||||
try:
|
||||
return jsonable_encoder(data)
|
||||
except Exception as e:
|
||||
logger.warning("Failed to serialize complex data, using string representation: %s", e)
|
||||
return str(data)
|
||||
|
||||
def _to_es_document(self, execution: WorkflowNodeExecution) -> dict[str, Any]:
|
||||
"""
|
||||
Convert WorkflowNodeExecution domain entity to Elasticsearch document.
|
||||
|
||||
Args:
|
||||
execution: The domain entity to convert
|
||||
|
||||
Returns:
|
||||
Dictionary representing the Elasticsearch document
|
||||
"""
|
||||
doc = {
|
||||
"id": execution.id,
|
||||
"tenant_id": self._tenant_id,
|
||||
"app_id": self._app_id,
|
||||
"workflow_id": execution.workflow_id,
|
||||
"workflow_execution_id": execution.workflow_execution_id,
|
||||
"node_execution_id": execution.node_execution_id,
|
||||
"triggered_from": self._triggered_from.value if self._triggered_from else None,
|
||||
"index": execution.index,
|
||||
"predecessor_node_id": execution.predecessor_node_id,
|
||||
"node_id": execution.node_id,
|
||||
"node_type": execution.node_type.value,
|
||||
"title": execution.title,
|
||||
"inputs": self._serialize_complex_data(execution.inputs),
|
||||
"process_data": self._serialize_complex_data(execution.process_data),
|
||||
"outputs": self._serialize_complex_data(execution.outputs),
|
||||
"status": execution.status.value,
|
||||
"error": execution.error,
|
||||
"elapsed_time": execution.elapsed_time,
|
||||
"metadata": self._serialize_complex_data(execution.metadata),
|
||||
"created_at": execution.created_at.isoformat() if execution.created_at else None,
|
||||
"finished_at": execution.finished_at.isoformat() if execution.finished_at else None,
|
||||
"created_by_role": self._creator_user_role.value,
|
||||
"created_by": self._creator_user_id,
|
||||
}
|
||||
|
||||
# Remove None values to reduce storage size
|
||||
return {k: v for k, v in doc.items() if v is not None}
|
||||
|
||||
def _from_es_document(self, doc: dict[str, Any]) -> WorkflowNodeExecution:
|
||||
"""
|
||||
Convert Elasticsearch document to WorkflowNodeExecution domain entity.
|
||||
|
||||
Args:
|
||||
doc: Elasticsearch document
|
||||
|
||||
Returns:
|
||||
WorkflowNodeExecution domain entity
|
||||
"""
|
||||
from core.workflow.enums import NodeType
|
||||
|
||||
source = doc.get("_source", doc)
|
||||
|
||||
return WorkflowNodeExecution(
|
||||
id=source["id"],
|
||||
node_execution_id=source.get("node_execution_id"),
|
||||
workflow_id=source["workflow_id"],
|
||||
workflow_execution_id=source.get("workflow_execution_id"),
|
||||
index=source["index"],
|
||||
predecessor_node_id=source.get("predecessor_node_id"),
|
||||
node_id=source["node_id"],
|
||||
node_type=NodeType(source["node_type"]),
|
||||
title=source["title"],
|
||||
inputs=source.get("inputs"),
|
||||
process_data=source.get("process_data"),
|
||||
outputs=source.get("outputs"),
|
||||
status=WorkflowNodeExecutionStatus(source["status"]),
|
||||
error=source.get("error"),
|
||||
elapsed_time=source.get("elapsed_time", 0.0),
|
||||
metadata=source.get("metadata", {}),
|
||||
created_at=datetime.fromisoformat(source["created_at"]) if source.get("created_at") else None,
|
||||
finished_at=datetime.fromisoformat(source["finished_at"]) if source.get("finished_at") else None,
|
||||
)
|
||||
|
||||
def save(self, execution: WorkflowNodeExecution) -> None:
|
||||
"""
|
||||
Save or update a NodeExecution domain entity to Elasticsearch.
|
||||
|
||||
Args:
|
||||
execution: The NodeExecution domain entity to persist
|
||||
"""
|
||||
try:
|
||||
index_name = self._get_index_name(execution.created_at)
|
||||
doc = self._to_es_document(execution)
|
||||
|
||||
# Use upsert to handle both create and update operations
|
||||
self._es_client.index(
|
||||
index=index_name,
|
||||
id=execution.id,
|
||||
body=doc,
|
||||
refresh="wait_for" # Ensure document is searchable immediately
|
||||
)
|
||||
|
||||
# Update cache
|
||||
self._execution_cache[execution.id] = execution
|
||||
|
||||
logger.debug(f"Saved workflow node execution {execution.id} to index {index_name}")
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to save workflow node execution {execution.id}: {e}")
|
||||
raise
|
||||
|
||||
def save_execution_data(self, execution: WorkflowNodeExecution) -> None:
|
||||
"""
|
||||
Save or update the inputs, process_data, or outputs for a node execution.
|
||||
|
||||
Args:
|
||||
execution: The NodeExecution with updated data
|
||||
"""
|
||||
try:
|
||||
index_name = self._get_index_name(execution.created_at)
|
||||
|
||||
# Prepare partial update document
|
||||
update_doc = {}
|
||||
if execution.inputs is not None:
|
||||
update_doc["inputs"] = execution.inputs
|
||||
if execution.process_data is not None:
|
||||
update_doc["process_data"] = execution.process_data
|
||||
if execution.outputs is not None:
|
||||
update_doc["outputs"] = execution.outputs
|
||||
|
||||
if update_doc:
|
||||
# Serialize complex data in update document
|
||||
serialized_update_doc = {}
|
||||
for key, value in update_doc.items():
|
||||
serialized_update_doc[key] = self._serialize_complex_data(value)
|
||||
|
||||
self._es_client.update(
|
||||
index=index_name,
|
||||
id=execution.id,
|
||||
body={"doc": serialized_update_doc},
|
||||
refresh="wait_for"
|
||||
)
|
||||
|
||||
# Update cache
|
||||
if execution.id in self._execution_cache:
|
||||
cached_execution = self._execution_cache[execution.id]
|
||||
if execution.inputs is not None:
|
||||
cached_execution.inputs = execution.inputs
|
||||
if execution.process_data is not None:
|
||||
cached_execution.process_data = execution.process_data
|
||||
if execution.outputs is not None:
|
||||
cached_execution.outputs = execution.outputs
|
||||
|
||||
logger.debug(f"Updated execution data for {execution.id}")
|
||||
|
||||
except NotFoundError:
|
||||
# Document doesn't exist, create it
|
||||
self.save(execution)
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to update execution data for {execution.id}: {e}")
|
||||
raise
|
||||
|
||||
def get_by_workflow_run(
|
||||
self,
|
||||
workflow_run_id: str,
|
||||
order_config: OrderConfig | None = None,
|
||||
) -> Sequence[WorkflowNodeExecution]:
|
||||
"""
|
||||
Retrieve all NodeExecution instances for a specific workflow run.
|
||||
|
||||
Args:
|
||||
workflow_run_id: The workflow run ID
|
||||
order_config: Optional configuration for ordering results
|
||||
|
||||
Returns:
|
||||
A list of NodeExecution instances
|
||||
"""
|
||||
try:
|
||||
# Build query
|
||||
query = {
|
||||
"bool": {
|
||||
"must": [
|
||||
{"term": {"tenant_id": self._tenant_id}},
|
||||
{"term": {"workflow_execution_id": workflow_run_id}},
|
||||
]
|
||||
}
|
||||
}
|
||||
|
||||
if self._app_id:
|
||||
query["bool"]["must"].append({"term": {"app_id": self._app_id}})
|
||||
|
||||
if self._triggered_from:
|
||||
query["bool"]["must"].append({"term": {"triggered_from": self._triggered_from.value}})
|
||||
|
||||
# Build sort configuration
|
||||
sort_config = []
|
||||
if order_config and order_config.order_by:
|
||||
for field in order_config.order_by:
|
||||
direction = "desc" if order_config.order_direction == "desc" else "asc"
|
||||
sort_config.append({field: {"order": direction}})
|
||||
else:
|
||||
# Default sort by index and created_at
|
||||
sort_config = [
|
||||
{"index": {"order": "asc"}},
|
||||
{"created_at": {"order": "asc"}}
|
||||
]
|
||||
|
||||
# Search across all indices for this tenant
|
||||
index_pattern = f"{self._index_prefix}-{self._tenant_id}-*"
|
||||
|
||||
response = self._es_client.search(
|
||||
index=index_pattern,
|
||||
body={
|
||||
"query": query,
|
||||
"sort": sort_config,
|
||||
"size": 10000, # Adjust based on expected max executions per workflow
|
||||
}
|
||||
)
|
||||
|
||||
executions = []
|
||||
for hit in response["hits"]["hits"]:
|
||||
execution = self._from_es_document(hit)
|
||||
executions.append(execution)
|
||||
# Update cache
|
||||
self._execution_cache[execution.id] = execution
|
||||
|
||||
return executions
|
||||
|
||||
except Exception as e:
|
||||
logger.error("Failed to retrieve executions for workflow run %s: %s", workflow_run_id, e)
|
||||
raise
|
||||
@ -1,6 +1,7 @@
|
||||
from typing import Any
|
||||
|
||||
from pydantic import BaseModel, Field
|
||||
from openai import BaseModel
|
||||
from pydantic import Field
|
||||
|
||||
from core.app.entities.app_invoke_entities import InvokeFrom
|
||||
from core.tools.entities.tool_entities import CredentialType, ToolInvokeFrom
|
||||
|
||||
@ -1,121 +0,0 @@
|
||||
"""
|
||||
Adapter for converting WorkflowExecution domain entities to WorkflowRun database models.
|
||||
|
||||
This adapter bridges the gap between the core domain model (WorkflowExecution)
|
||||
and the database model (WorkflowRun) that APIs expect.
|
||||
"""
|
||||
|
||||
import json
|
||||
import logging
|
||||
|
||||
from core.workflow.entities import WorkflowExecution
|
||||
from core.workflow.enums import WorkflowExecutionStatus
|
||||
from models.workflow import WorkflowRun
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class WorkflowExecutionToRunAdapter:
|
||||
"""
|
||||
Adapter for converting WorkflowExecution domain entities to WorkflowRun database models.
|
||||
|
||||
This adapter ensures that API endpoints that expect WorkflowRun data can work
|
||||
with WorkflowExecution entities stored in Elasticsearch.
|
||||
"""
|
||||
|
||||
@staticmethod
|
||||
def to_workflow_run(
|
||||
execution: WorkflowExecution,
|
||||
tenant_id: str,
|
||||
app_id: str,
|
||||
triggered_from: str,
|
||||
created_by_role: str,
|
||||
created_by: str,
|
||||
) -> WorkflowRun:
|
||||
"""
|
||||
Convert a WorkflowExecution domain entity to a WorkflowRun database model.
|
||||
|
||||
Args:
|
||||
execution: The WorkflowExecution domain entity
|
||||
tenant_id: Tenant identifier
|
||||
app_id: Application identifier
|
||||
triggered_from: Source of the execution trigger
|
||||
created_by_role: Role of the user who created the execution
|
||||
created_by: ID of the user who created the execution
|
||||
|
||||
Returns:
|
||||
WorkflowRun database model instance
|
||||
"""
|
||||
# Map WorkflowExecutionStatus to string
|
||||
status_mapping = {
|
||||
WorkflowExecutionStatus.RUNNING: "running",
|
||||
WorkflowExecutionStatus.SUCCEEDED: "succeeded",
|
||||
WorkflowExecutionStatus.FAILED: "failed",
|
||||
WorkflowExecutionStatus.STOPPED: "stopped",
|
||||
WorkflowExecutionStatus.PARTIAL_SUCCEEDED: "partial-succeeded",
|
||||
}
|
||||
|
||||
workflow_run = WorkflowRun()
|
||||
workflow_run.id = execution.id_
|
||||
workflow_run.tenant_id = tenant_id
|
||||
workflow_run.app_id = app_id
|
||||
workflow_run.workflow_id = execution.workflow_id
|
||||
workflow_run.type = execution.workflow_type.value
|
||||
workflow_run.triggered_from = triggered_from
|
||||
workflow_run.version = execution.workflow_version
|
||||
workflow_run.graph = json.dumps(execution.graph) if execution.graph else None
|
||||
workflow_run.inputs = json.dumps(execution.inputs) if execution.inputs else None
|
||||
workflow_run.status = status_mapping.get(execution.status, "running")
|
||||
workflow_run.outputs = json.dumps(execution.outputs) if execution.outputs else None
|
||||
workflow_run.error = execution.error_message
|
||||
workflow_run.elapsed_time = execution.elapsed_time
|
||||
workflow_run.total_tokens = execution.total_tokens
|
||||
workflow_run.total_steps = execution.total_steps
|
||||
workflow_run.created_by_role = created_by_role
|
||||
workflow_run.created_by = created_by
|
||||
workflow_run.created_at = execution.started_at
|
||||
workflow_run.finished_at = execution.finished_at
|
||||
workflow_run.exceptions_count = execution.exceptions_count
|
||||
|
||||
return workflow_run
|
||||
|
||||
@staticmethod
|
||||
def from_workflow_run(workflow_run: WorkflowRun) -> WorkflowExecution:
|
||||
"""
|
||||
Convert a WorkflowRun database model to a WorkflowExecution domain entity.
|
||||
|
||||
Args:
|
||||
workflow_run: The WorkflowRun database model
|
||||
|
||||
Returns:
|
||||
WorkflowExecution domain entity
|
||||
"""
|
||||
from core.workflow.enums import WorkflowType
|
||||
|
||||
# Map string status to WorkflowExecutionStatus
|
||||
status_mapping = {
|
||||
"running": WorkflowExecutionStatus.RUNNING,
|
||||
"succeeded": WorkflowExecutionStatus.SUCCEEDED,
|
||||
"failed": WorkflowExecutionStatus.FAILED,
|
||||
"stopped": WorkflowExecutionStatus.STOPPED,
|
||||
"partial-succeeded": WorkflowExecutionStatus.PARTIAL_SUCCEEDED,
|
||||
}
|
||||
|
||||
execution = WorkflowExecution(
|
||||
id_=workflow_run.id,
|
||||
workflow_id=workflow_run.workflow_id,
|
||||
workflow_version=workflow_run.version,
|
||||
workflow_type=WorkflowType(workflow_run.type),
|
||||
graph=workflow_run.graph_dict,
|
||||
inputs=workflow_run.inputs_dict,
|
||||
outputs=workflow_run.outputs_dict,
|
||||
status=status_mapping.get(workflow_run.status, WorkflowExecutionStatus.RUNNING),
|
||||
error_message=workflow_run.error or "",
|
||||
total_tokens=workflow_run.total_tokens,
|
||||
total_steps=workflow_run.total_steps,
|
||||
exceptions_count=workflow_run.exceptions_count,
|
||||
started_at=workflow_run.created_at,
|
||||
finished_at=workflow_run.finished_at,
|
||||
)
|
||||
|
||||
return execution
|
||||
@ -1,7 +1,7 @@
|
||||
import os
|
||||
from collections.abc import Mapping, Sequence
|
||||
from typing import Any
|
||||
|
||||
from configs import dify_config
|
||||
from core.helper.code_executor.code_executor import CodeExecutionError, CodeExecutor, CodeLanguage
|
||||
from core.workflow.enums import ErrorStrategy, NodeType, WorkflowNodeExecutionStatus
|
||||
from core.workflow.node_events import NodeRunResult
|
||||
@ -9,7 +9,7 @@ from core.workflow.nodes.base.entities import BaseNodeData, RetryConfig
|
||||
from core.workflow.nodes.base.node import Node
|
||||
from core.workflow.nodes.template_transform.entities import TemplateTransformNodeData
|
||||
|
||||
MAX_TEMPLATE_TRANSFORM_OUTPUT_LENGTH = dify_config.TEMPLATE_TRANSFORM_MAX_LENGTH
|
||||
MAX_TEMPLATE_TRANSFORM_OUTPUT_LENGTH = int(os.environ.get("TEMPLATE_TRANSFORM_MAX_LENGTH", "80000"))
|
||||
|
||||
|
||||
class TemplateTransformNode(Node):
|
||||
|
||||
@ -1,129 +0,0 @@
|
||||
# 完整的 Elasticsearch 配置指南
|
||||
|
||||
## 🔧 **问题修复总结**
|
||||
|
||||
我已经修复了以下问题：
|
||||
|
||||
### 1. **构造函数参数不匹配**
|
||||
- **错误**: `ElasticsearchWorkflowExecutionRepository.__init__() got an unexpected keyword argument 'session_factory'`
|
||||
- **修复**: 修改构造函数接受 `session_factory` 参数，从全局扩展获取 Elasticsearch 客户端
|
||||
|
||||
### 2. **导入错误**
|
||||
- **错误**: `name 'sessionmaker' is not defined`
|
||||
- **修复**: 添加必要的 SQLAlchemy 导入
|
||||
|
||||
### 3. **SSL/HTTPS 配置**
|
||||
- **错误**: `received plaintext http traffic on an https channel`
|
||||
- **修复**: 使用 HTTPS 连接和正确的认证信息
|
||||
|
||||
### 4. **实体属性不匹配**
|
||||
- **错误**: `'WorkflowExecution' object has no attribute 'created_at'` 和 `'WorkflowExecution' object has no attribute 'id'`
|
||||
- **修复**: 使用正确的属性名：
|
||||
- `id_` 而不是 `id`
|
||||
- `started_at` 而不是 `created_at`
|
||||
- `error_message` 而不是 `error`
|
||||
|
||||
## 📋 **完整的 .env 配置**
|
||||
|
||||
请将以下配置添加到您的 `dify/api/.env` 文件：
|
||||
|
||||
```bash
|
||||
# ====================================
|
||||
# Elasticsearch 配置
|
||||
# ====================================
|
||||
|
||||
# 启用 Elasticsearch
|
||||
ELASTICSEARCH_ENABLED=true
|
||||
|
||||
# 连接设置（注意使用 HTTPS）
|
||||
ELASTICSEARCH_HOSTS=["https://localhost:9200"]
|
||||
ELASTICSEARCH_USERNAME=elastic
|
||||
ELASTICSEARCH_PASSWORD=2gYvv6+O36PGwaVD6yzE
|
||||
|
||||
# SSL 设置
|
||||
ELASTICSEARCH_USE_SSL=true
|
||||
ELASTICSEARCH_VERIFY_CERTS=false
|
||||
|
||||
# 性能设置
|
||||
ELASTICSEARCH_TIMEOUT=30
|
||||
ELASTICSEARCH_MAX_RETRIES=3
|
||||
ELASTICSEARCH_INDEX_PREFIX=dify
|
||||
ELASTICSEARCH_RETENTION_DAYS=30
|
||||
|
||||
# ====================================
|
||||
# Repository Factory 配置
|
||||
# 切换到 Elasticsearch 实现
|
||||
# ====================================
|
||||
|
||||
# 核心工作流 repositories
|
||||
CORE_WORKFLOW_EXECUTION_REPOSITORY=core.repositories.elasticsearch_workflow_execution_repository.ElasticsearchWorkflowExecutionRepository
|
||||
CORE_WORKFLOW_NODE_EXECUTION_REPOSITORY=core.repositories.elasticsearch_workflow_node_execution_repository.ElasticsearchWorkflowNodeExecutionRepository
|
||||
|
||||
# API 服务层 repositories
|
||||
API_WORKFLOW_RUN_REPOSITORY=repositories.elasticsearch_api_workflow_run_repository.ElasticsearchAPIWorkflowRunRepository
|
||||
```
|
||||
|
||||
## 🚀 **使用步骤**
|
||||
|
||||
### 1. 配置环境变量
|
||||
将上述配置复制到您的 `.env` 文件中
|
||||
|
||||
### 2. 重启应用
|
||||
重启 Dify API 服务以加载新配置
|
||||
|
||||
### 3. 测试连接
|
||||
```bash
|
||||
flask elasticsearch status
|
||||
```
|
||||
|
||||
### 4. 执行迁移
|
||||
```bash
|
||||
# 干运行测试
|
||||
flask elasticsearch migrate --dry-run
|
||||
|
||||
# 实际迁移（替换为您的实际 tenant_id）
|
||||
flask elasticsearch migrate --tenant-id your-tenant-id
|
||||
|
||||
# 验证迁移结果
|
||||
flask elasticsearch validate --tenant-id your-tenant-id
|
||||
```
|
||||
|
||||
## 📊 **四个日志表的处理方式**
|
||||
|
||||
| 表名 | Repository 配置 | 实现类 |
|
||||
|------|----------------|--------|
|
||||
| `workflow_runs` | `API_WORKFLOW_RUN_REPOSITORY` | `ElasticsearchAPIWorkflowRunRepository` |
|
||||
| `workflow_node_executions` | `CORE_WORKFLOW_NODE_EXECUTION_REPOSITORY` | `ElasticsearchWorkflowNodeExecutionRepository` |
|
||||
| `workflow_app_logs` | 不使用 factory | `ElasticsearchWorkflowAppLogRepository` |
|
||||
| `workflow_node_execution_offload` | 集成处理 | 在 node executions 中自动处理 |
|
||||
|
||||
## ✅ **验证配置正确性**
|
||||
|
||||
配置完成后，您可以通过以下方式验证：
|
||||
|
||||
1. **检查应用启动**: 应用应该能正常启动，无错误日志
|
||||
2. **测试 Elasticsearch 连接**: `flask elasticsearch status` 应该显示集群状态
|
||||
3. **测试工作流执行**: 在 Dify 界面中执行工作流，检查是否有错误
|
||||
|
||||
## 🔄 **回滚方案**
|
||||
|
||||
如果需要回滚到 PostgreSQL，只需注释掉或删除 Repository 配置：
|
||||
|
||||
```bash
|
||||
# 注释掉这些行以回滚到 PostgreSQL
|
||||
# CORE_WORKFLOW_EXECUTION_REPOSITORY=core.repositories.elasticsearch_workflow_execution_repository.ElasticsearchWorkflowExecutionRepository
|
||||
# CORE_WORKFLOW_NODE_EXECUTION_REPOSITORY=core.repositories.elasticsearch_workflow_node_execution_repository.ElasticsearchWorkflowNodeExecutionRepository
|
||||
# API_WORKFLOW_RUN_REPOSITORY=repositories.elasticsearch_api_workflow_run_repository.ElasticsearchAPIWorkflowRunRepository
|
||||
```
|
||||
|
||||
## 🎯 **关键优势**
|
||||
|
||||
切换到 Elasticsearch 后，您将获得：
|
||||
|
||||
1. **更好的性能**: 专为日志数据优化的存储引擎
|
||||
2. **全文搜索**: 支持复杂的日志搜索和分析
|
||||
3. **时间序列优化**: 自动索引轮转和数据生命周期管理
|
||||
4. **水平扩展**: 支持集群扩展处理大量数据
|
||||
5. **实时分析**: 近实时的数据查询和聚合分析
|
||||
|
||||
现在所有的错误都已经修复，您可以安全地使用 Elasticsearch 作为工作流日志的存储后端了！
|
||||
@ -1,86 +0,0 @@
|
||||
# Elasticsearch 错误修复总结
|
||||
|
||||
## 🔍 **遇到的错误和修复方案**
|
||||
|
||||
### 错误 1: 命令未找到
|
||||
**错误**: `No such command 'elasticsearch'`
|
||||
**原因**: CLI 命令没有正确注册
|
||||
**修复**: 将命令添加到 `commands.py` 并在 `ext_commands.py` 中注册
|
||||
|
||||
### 错误 2: SSL/HTTPS 配置问题
|
||||
**错误**: `received plaintext http traffic on an https channel`
|
||||
**原因**: Elasticsearch 启用了 HTTPS，但客户端使用 HTTP
|
||||
**修复**: 使用 HTTPS 连接和正确的认证信息
|
||||
|
||||
### 错误 3: 构造函数参数不匹配
|
||||
**错误**: `ElasticsearchWorkflowExecutionRepository.__init__() got an unexpected keyword argument 'session_factory'`
|
||||
**原因**: Factory 传递的参数与 Elasticsearch repository 构造函数不匹配
|
||||
**修复**: 修改构造函数接受 `session_factory` 参数，从全局扩展获取 ES 客户端
|
||||
|
||||
### 错误 4: 导入错误
|
||||
**错误**: `name 'sessionmaker' is not defined`
|
||||
**原因**: 类型注解中使用了未导入的类型
|
||||
**修复**: 添加必要的 SQLAlchemy 导入
|
||||
|
||||
### 错误 5: 实体属性不匹配
|
||||
**错误**: `'WorkflowExecution' object has no attribute 'created_at'` 和 `'id'`
|
||||
**原因**: WorkflowExecution 实体使用不同的属性名
|
||||
**修复**: 使用正确的属性名：
|
||||
- `id_` 而不是 `id`
|
||||
- `started_at` 而不是 `created_at`
|
||||
- `error_message` 而不是 `error`
|
||||
|
||||
### 错误 6: JSON 序列化问题
|
||||
**错误**: `Unable to serialize ArrayFileSegment`
|
||||
**原因**: Elasticsearch 无法序列化 Dify 的自定义 Segment 对象
|
||||
**修复**: 添加 `_serialize_complex_data()` 方法，使用 `jsonable_encoder` 处理复杂对象
|
||||
|
||||
## ✅ **最终解决方案**
|
||||
|
||||
### 完整的 .env 配置
|
||||
```bash
|
||||
# Elasticsearch 配置
|
||||
ELASTICSEARCH_ENABLED=true
|
||||
ELASTICSEARCH_HOSTS=["https://localhost:9200"]
|
||||
ELASTICSEARCH_USERNAME=elastic
|
||||
ELASTICSEARCH_PASSWORD=2gYvv6+O36PGwaVD6yzE
|
||||
ELASTICSEARCH_USE_SSL=true
|
||||
ELASTICSEARCH_VERIFY_CERTS=false
|
||||
ELASTICSEARCH_TIMEOUT=30
|
||||
ELASTICSEARCH_MAX_RETRIES=3
|
||||
ELASTICSEARCH_INDEX_PREFIX=dify
|
||||
ELASTICSEARCH_RETENTION_DAYS=30
|
||||
|
||||
# Repository Factory 配置
|
||||
CORE_WORKFLOW_EXECUTION_REPOSITORY=core.repositories.elasticsearch_workflow_execution_repository.ElasticsearchWorkflowExecutionRepository
|
||||
CORE_WORKFLOW_NODE_EXECUTION_REPOSITORY=core.repositories.elasticsearch_workflow_node_execution_repository.ElasticsearchWorkflowNodeExecutionRepository
|
||||
API_WORKFLOW_RUN_REPOSITORY=repositories.elasticsearch_api_workflow_run_repository.ElasticsearchAPIWorkflowRunRepository
|
||||
```
|
||||
|
||||
### 关键修复点
|
||||
1. **序列化处理**: 所有复杂对象都通过 `jsonable_encoder` 序列化
|
||||
2. **属性映射**: 正确映射 WorkflowExecution 实体属性
|
||||
3. **构造函数兼容**: 与现有 factory 模式完全兼容
|
||||
4. **错误处理**: 完善的错误处理和日志记录
|
||||
|
||||
## 🚀 **使用步骤**
|
||||
|
||||
1. **配置环境**: 将上述配置添加到 `.env` 文件
|
||||
2. **重启应用**: 重启 Dify API 服务
|
||||
3. **测试功能**: 执行工作流，检查是否正常工作
|
||||
4. **查看日志**: 检查 Elasticsearch 中的日志数据
|
||||
|
||||
## 📊 **验证方法**
|
||||
|
||||
```bash
|
||||
# 检查 Elasticsearch 状态
|
||||
flask elasticsearch status
|
||||
|
||||
# 查看索引和数据
|
||||
curl -k -u elastic:2gYvv6+O36PGwaVD6yzE -X GET "https://localhost:9200/_cat/indices/dify-*?v"
|
||||
|
||||
# 查看具体数据
|
||||
curl -k -u elastic:2gYvv6+O36PGwaVD6yzE -X GET "https://localhost:9200/dify-*/_search?pretty&size=1"
|
||||
```
|
||||
|
||||
现在所有错误都已修复，Elasticsearch 集成应该可以正常工作了！
|
||||
@ -1,66 +0,0 @@
|
||||
# Elasticsearch Factory 配置指南
|
||||
|
||||
## 配置您的 .env 文件
|
||||
|
||||
请在您的 `dify/api/.env` 文件中添加以下配置：
|
||||
|
||||
### 1. Elasticsearch 连接配置
|
||||
|
||||
```bash
|
||||
# 启用 Elasticsearch
|
||||
ELASTICSEARCH_ENABLED=true
|
||||
|
||||
# 连接设置（使用 HTTPS 和认证）
|
||||
ELASTICSEARCH_HOSTS=["https://localhost:9200"]
|
||||
ELASTICSEARCH_USERNAME=elastic
|
||||
ELASTICSEARCH_PASSWORD=2gYvv6+O36PGwaVD6yzE
|
||||
|
||||
# SSL 设置
|
||||
ELASTICSEARCH_USE_SSL=true
|
||||
ELASTICSEARCH_VERIFY_CERTS=false
|
||||
|
||||
# 性能设置
|
||||
ELASTICSEARCH_TIMEOUT=30
|
||||
ELASTICSEARCH_MAX_RETRIES=3
|
||||
ELASTICSEARCH_INDEX_PREFIX=dify
|
||||
ELASTICSEARCH_RETENTION_DAYS=30
|
||||
```
|
||||
|
||||
### 2. Factory 模式配置 - 切换到 Elasticsearch 实现
|
||||
|
||||
```bash
|
||||
# 核心工作流 repositories
|
||||
CORE_WORKFLOW_EXECUTION_REPOSITORY=core.repositories.elasticsearch_workflow_execution_repository.ElasticsearchWorkflowExecutionRepository
|
||||
CORE_WORKFLOW_NODE_EXECUTION_REPOSITORY=core.repositories.elasticsearch_workflow_node_execution_repository.ElasticsearchWorkflowNodeExecutionRepository
|
||||
|
||||
# API 服务层 repositories
|
||||
API_WORKFLOW_RUN_REPOSITORY=repositories.elasticsearch_api_workflow_run_repository.ElasticsearchAPIWorkflowRunRepository
|
||||
```
|
||||
|
||||
## 测试配置
|
||||
|
||||
配置完成后，重启应用并测试：
|
||||
|
||||
```bash
|
||||
# 检查连接状态
|
||||
flask elasticsearch status
|
||||
|
||||
# 测试迁移（干运行）
|
||||
flask elasticsearch migrate --dry-run
|
||||
```
|
||||
|
||||
## 四个日志表的 Repository 映射
|
||||
|
||||
| 日志表 | Repository 配置 | 说明 |
|
||||
|--------|----------------|------|
|
||||
| `workflow_runs` | `API_WORKFLOW_RUN_REPOSITORY` | API 服务层使用 |
|
||||
| `workflow_node_executions` | `CORE_WORKFLOW_NODE_EXECUTION_REPOSITORY` | 核心工作流使用 |
|
||||
| `workflow_app_logs` | 直接使用服务 | 不通过 factory 模式 |
|
||||
| `workflow_node_execution_offload` | 集成在 node_executions 中 | 大数据卸载处理 |
|
||||
|
||||
## 注意事项
|
||||
|
||||
1. **密码安全**: 请使用您自己的安全密码替换示例密码
|
||||
2. **渐进迁移**: 建议先在测试环境验证
|
||||
3. **数据备份**: 切换前请确保有完整备份
|
||||
4. **监控**: 切换后密切监控应用性能
|
||||
@ -1,33 +0,0 @@
|
||||
# ====================================
|
||||
# Elasticsearch 最终配置
|
||||
# 请将以下内容添加到您的 dify/api/.env 文件
|
||||
# ====================================
|
||||
|
||||
# Elasticsearch 连接配置
|
||||
ELASTICSEARCH_ENABLED=true
|
||||
ELASTICSEARCH_HOSTS=["https://localhost:9200"]
|
||||
ELASTICSEARCH_USERNAME=elastic
|
||||
ELASTICSEARCH_PASSWORD=2gYvv6+O36PGwaVD6yzE
|
||||
ELASTICSEARCH_USE_SSL=true
|
||||
ELASTICSEARCH_VERIFY_CERTS=false
|
||||
ELASTICSEARCH_TIMEOUT=30
|
||||
ELASTICSEARCH_MAX_RETRIES=3
|
||||
ELASTICSEARCH_INDEX_PREFIX=dify
|
||||
ELASTICSEARCH_RETENTION_DAYS=30
|
||||
|
||||
# Factory 模式配置 - 选择 Elasticsearch 实现
|
||||
CORE_WORKFLOW_EXECUTION_REPOSITORY=core.repositories.elasticsearch_workflow_execution_repository.ElasticsearchWorkflowExecutionRepository
|
||||
CORE_WORKFLOW_NODE_EXECUTION_REPOSITORY=core.repositories.elasticsearch_workflow_node_execution_repository.ElasticsearchWorkflowNodeExecutionRepository
|
||||
API_WORKFLOW_RUN_REPOSITORY=repositories.elasticsearch_api_workflow_run_repository.ElasticsearchAPIWorkflowRunRepository
|
||||
|
||||
# ====================================
|
||||
# 修复的问题总结：
|
||||
# ====================================
|
||||
# 1. SSL/HTTPS 配置：使用 HTTPS 和正确认证
|
||||
# 2. 构造函数兼容：修改为接受 session_factory 参数
|
||||
# 3. 导入修复：添加必要的 SQLAlchemy 导入
|
||||
# 4. 实体属性：使用正确的 WorkflowExecution 属性名
|
||||
# - id_ (不是 id)
|
||||
# - started_at (不是 created_at)
|
||||
# - error_message (不是 error)
|
||||
# ====================================
|
||||
@ -1,204 +0,0 @@
|
||||
# Elasticsearch Implementation Summary
|
||||
|
||||
## 概述
|
||||
|
||||
基于您的需求，我已经为 Dify 设计并实现了完整的 Elasticsearch 日志存储方案，用于替代 PostgreSQL 存储四个日志表的数据。这个方案遵循了 Dify 现有的 Repository 模式和 Factory 模式，提供了高性能、可扩展的日志存储解决方案。
|
||||
|
||||
## 实现的组件
|
||||
|
||||
### 1. 核心 Repository 实现
|
||||
|
||||
#### `ElasticsearchWorkflowNodeExecutionRepository`
|
||||
- **位置**: `dify/api/core/repositories/elasticsearch_workflow_node_execution_repository.py`
|
||||
- **功能**: 实现 `WorkflowNodeExecutionRepository` 接口
|
||||
- **特性**:
|
||||
- 时间序列索引优化（按月分割）
|
||||
- 多租户数据隔离
|
||||
- 大数据自动截断和存储
|
||||
- 内存缓存提升性能
|
||||
- 自动索引模板管理
|
||||
|
||||
#### `ElasticsearchWorkflowExecutionRepository`
|
||||
- **位置**: `dify/api/core/repositories/elasticsearch_workflow_execution_repository.py`
|
||||
- **功能**: 实现 `WorkflowExecutionRepository` 接口
|
||||
- **特性**:
|
||||
- 工作流执行数据的 ES 存储
|
||||
- 支持按 ID 查询和删除
|
||||
- 时间序列索引管理
|
||||
|
||||
### 2. API 层 Repository 实现
|
||||
|
||||
#### `ElasticsearchAPIWorkflowRunRepository`
|
||||
- **位置**: `dify/api/repositories/elasticsearch_api_workflow_run_repository.py`
|
||||
- **功能**: 实现 `APIWorkflowRunRepository` 接口
|
||||
- **特性**:
|
||||
- 分页查询支持
|
||||
- 游标分页优化
|
||||
- 批量删除操作
|
||||
- 高级搜索功能（全文搜索）
|
||||
- 过期数据清理
|
||||
|
||||
#### `ElasticsearchWorkflowAppLogRepository`
|
||||
- **位置**: `dify/api/repositories/elasticsearch_workflow_app_log_repository.py`
|
||||
- **功能**: WorkflowAppLog 的 ES 存储实现
|
||||
- **特性**:
|
||||
- 应用日志的高效存储
|
||||
- 多维度过滤查询
|
||||
- 时间范围查询优化
|
||||
|
||||
### 3. 扩展和配置
|
||||
|
||||
#### `ElasticsearchExtension`
|
||||
- **位置**: `dify/api/extensions/ext_elasticsearch.py`
|
||||
- **功能**: Flask 应用的 ES 扩展
|
||||
- **特性**:
|
||||
- 集中化的 ES 客户端管理
|
||||
- 连接健康检查
|
||||
- SSL/认证支持
|
||||
- 配置化连接参数
|
||||
|
||||
#### 配置集成
|
||||
- **位置**: `dify/api/configs/feature/__init__.py`
|
||||
- **新增**: `ElasticsearchConfig` 类
|
||||
- **配置项**:
|
||||
- ES 连接参数
|
||||
- 认证配置
|
||||
- SSL 设置
|
||||
- 性能参数
|
||||
- 索引前缀和保留策略
|
||||
|
||||
### 4. 数据迁移服务
|
||||
|
||||
#### `ElasticsearchMigrationService`
|
||||
- **位置**: `dify/api/services/elasticsearch_migration_service.py`
|
||||
- **功能**: 完整的数据迁移解决方案
|
||||
- **特性**:
|
||||
- 批量数据迁移
|
||||
- 进度跟踪
|
||||
- 数据验证
|
||||
- 回滚支持
|
||||
- 性能监控
|
||||
|
||||
#### CLI 迁移工具
|
||||
- **位置**: `dify/api/commands/migrate_to_elasticsearch.py`
|
||||
- **功能**: 命令行迁移工具
|
||||
- **命令**:
|
||||
- `flask elasticsearch migrate` - 数据迁移
|
||||
- `flask elasticsearch validate` - 数据验证
|
||||
- `flask elasticsearch cleanup-pg` - PG 数据清理
|
||||
- `flask elasticsearch status` - 状态检查
|
||||
|
||||
## 架构设计特点
|
||||
|
||||
### 1. 遵循现有模式
|
||||
- **Repository 模式**: 完全兼容现有的 Repository 接口
|
||||
- **Factory 模式**: 通过配置切换不同实现
|
||||
- **依赖注入**: 支持 sessionmaker 和 ES client 注入
|
||||
- **多租户**: 保持现有的多租户隔离机制
|
||||
|
||||
### 2. 性能优化
|
||||
- **时间序列索引**: 按月分割索引，提升查询性能
|
||||
- **数据截断**: 大数据自动截断，避免 ES 性能问题
|
||||
- **批量操作**: 支持批量写入和删除
|
||||
- **缓存机制**: 内存缓存减少重复查询
|
||||
|
||||
### 3. 可扩展性
|
||||
- **水平扩展**: ES 集群支持水平扩展
|
||||
- **索引轮转**: 自动索引轮转和清理
|
||||
- **配置化**: 所有参数可通过配置调整
|
||||
- **插件化**: 可以轻松添加新的数据类型支持
|
||||
|
||||
### 4. 数据安全
|
||||
- **多租户隔离**: 每个租户独立的索引模式
|
||||
- **数据验证**: 迁移后的数据完整性验证
|
||||
- **备份恢复**: 支持数据备份和恢复策略
|
||||
- **渐进迁移**: 支持增量迁移，降低风险
|
||||
|
||||
## 使用方式
|
||||
|
||||
### 1. 配置切换
|
||||
|
||||
通过环境变量切换到 Elasticsearch：
|
||||
|
||||
```bash
|
||||
# 启用 Elasticsearch
|
||||
ELASTICSEARCH_ENABLED=true
|
||||
ELASTICSEARCH_HOSTS=["http://localhost:9200"]
|
||||
|
||||
# 切换 Repository 实现
|
||||
CORE_WORKFLOW_NODE_EXECUTION_REPOSITORY=core.repositories.elasticsearch_workflow_node_execution_repository.ElasticsearchWorkflowNodeExecutionRepository
|
||||
API_WORKFLOW_RUN_REPOSITORY=repositories.elasticsearch_api_workflow_run_repository.ElasticsearchAPIWorkflowRunRepository
|
||||
```
|
||||
|
||||
### 2. 数据迁移
|
||||
|
||||
```bash
|
||||
# 干运行测试
|
||||
flask elasticsearch migrate --dry-run
|
||||
|
||||
# 实际迁移
|
||||
flask elasticsearch migrate --tenant-id tenant-123
|
||||
|
||||
# 验证迁移
|
||||
flask elasticsearch validate --tenant-id tenant-123
|
||||
```
|
||||
|
||||
### 3. 代码使用
|
||||
|
||||
现有代码无需修改，Repository 接口保持不变：
|
||||
|
||||
```python
|
||||
# 现有代码继续工作
|
||||
from repositories.factory import DifyAPIRepositoryFactory
|
||||
|
||||
session_maker = sessionmaker(bind=db.engine)
|
||||
repo = DifyAPIRepositoryFactory.create_api_workflow_run_repository(session_maker)
|
||||
|
||||
# 自动使用 Elasticsearch 实现
|
||||
runs = repo.get_paginated_workflow_runs(tenant_id, app_id, "debugging")
|
||||
```
|
||||
|
||||
## 优势总结
|
||||
|
||||
### 1. 性能提升
|
||||
- **查询性能**: ES 针对日志查询优化，性能显著提升
|
||||
- **存储效率**: 时间序列数据压缩，存储空间更小
|
||||
- **并发处理**: ES 支持高并发读写操作
|
||||
|
||||
### 2. 功能增强
|
||||
- **全文搜索**: 支持日志内容的全文搜索
|
||||
- **聚合分析**: 支持复杂的数据分析和统计
|
||||
- **实时查询**: 近实时的数据查询能力
|
||||
|
||||
### 3. 运维友好
|
||||
- **自动管理**: 索引自动轮转和清理
|
||||
- **监控完善**: 丰富的监控和告警机制
|
||||
- **扩展简单**: 水平扩展容易实现
|
||||
|
||||
### 4. 兼容性好
|
||||
- **无缝切换**: 现有代码无需修改
|
||||
- **渐进迁移**: 支持逐步迁移，降低风险
|
||||
- **回滚支持**: 可以随时回滚到 PostgreSQL
|
||||
|
||||
## 部署建议
|
||||
|
||||
### 1. 测试环境
|
||||
1. 部署 Elasticsearch 集群
|
||||
2. 配置 Dify 连接 ES
|
||||
3. 执行小规模数据迁移测试
|
||||
4. 验证功能和性能
|
||||
|
||||
### 2. 生产环境
|
||||
1. 规划 ES 集群容量
|
||||
2. 配置监控和告警
|
||||
3. 执行渐进式迁移
|
||||
4. 监控性能和稳定性
|
||||
5. 逐步清理 PostgreSQL 数据
|
||||
|
||||
### 3. 监控要点
|
||||
- ES 集群健康状态
|
||||
- 索引大小和文档数量
|
||||
- 查询性能指标
|
||||
- 迁移进度和错误率
|
||||
|
||||
这个实现方案完全符合 Dify 的架构设计原则，提供了高性能、可扩展的日志存储解决方案，同时保持了良好的向后兼容性和运维友好性。
|
||||
@ -1,297 +0,0 @@
|
||||
# Elasticsearch Migration Guide
|
||||
|
||||
This guide explains how to migrate workflow log data from PostgreSQL to Elasticsearch for better performance and scalability.
|
||||
|
||||
## Overview
|
||||
|
||||
The Elasticsearch integration provides:
|
||||
|
||||
- **High-performance log storage**: Better suited for time-series log data
|
||||
- **Advanced search capabilities**: Full-text search and complex queries
|
||||
- **Scalability**: Horizontal scaling for large datasets
|
||||
- **Time-series optimization**: Date-based index rotation for efficient storage
|
||||
- **Multi-tenant isolation**: Separate indices per tenant for data isolation
|
||||
|
||||
## Architecture
|
||||
|
||||
The migration involves four main log tables:
|
||||
|
||||
1. **workflow_runs**: Core workflow execution records
|
||||
2. **workflow_app_logs**: Application-level workflow logs
|
||||
3. **workflow_node_executions**: Individual node execution records
|
||||
4. **workflow_node_execution_offload**: Large data offloaded to storage
|
||||
|
||||
## Configuration
|
||||
|
||||
### Environment Variables
|
||||
|
||||
Add the following to your `.env` file:
|
||||
|
||||
```bash
|
||||
# Enable Elasticsearch
|
||||
ELASTICSEARCH_ENABLED=true
|
||||
|
||||
# Elasticsearch connection
|
||||
ELASTICSEARCH_HOSTS=["http://localhost:9200"]
|
||||
ELASTICSEARCH_USERNAME=elastic
|
||||
ELASTICSEARCH_PASSWORD=your_password
|
||||
|
||||
# SSL configuration (optional)
|
||||
ELASTICSEARCH_USE_SSL=false
|
||||
ELASTICSEARCH_VERIFY_CERTS=true
|
||||
ELASTICSEARCH_CA_CERTS=/path/to/ca.crt
|
||||
|
||||
# Performance settings
|
||||
ELASTICSEARCH_TIMEOUT=30
|
||||
ELASTICSEARCH_MAX_RETRIES=3
|
||||
ELASTICSEARCH_INDEX_PREFIX=dify
|
||||
ELASTICSEARCH_RETENTION_DAYS=30
|
||||
```
|
||||
|
||||
### Repository Configuration
|
||||
|
||||
Update your configuration to use Elasticsearch repositories:
|
||||
|
||||
```bash
|
||||
# Core repositories
|
||||
CORE_WORKFLOW_EXECUTION_REPOSITORY=core.repositories.elasticsearch_workflow_execution_repository.ElasticsearchWorkflowExecutionRepository
|
||||
CORE_WORKFLOW_NODE_EXECUTION_REPOSITORY=core.repositories.elasticsearch_workflow_node_execution_repository.ElasticsearchWorkflowNodeExecutionRepository
|
||||
|
||||
# API repositories
|
||||
API_WORKFLOW_RUN_REPOSITORY=repositories.elasticsearch_api_workflow_run_repository.ElasticsearchAPIWorkflowRunRepository
|
||||
```
|
||||
|
||||
## Migration Process
|
||||
|
||||
### 1. Setup Elasticsearch
|
||||
|
||||
First, ensure Elasticsearch is running and accessible:
|
||||
|
||||
```bash
|
||||
# Check Elasticsearch status
|
||||
curl -X GET "localhost:9200/_cluster/health?pretty"
|
||||
```
|
||||
|
||||
### 2. Test Configuration
|
||||
|
||||
Verify your Dify configuration:
|
||||
|
||||
```bash
|
||||
# Check Elasticsearch connection
|
||||
flask elasticsearch status
|
||||
```
|
||||
|
||||
### 3. Dry Run Migration
|
||||
|
||||
Perform a dry run to estimate migration scope:
|
||||
|
||||
```bash
|
||||
# Dry run for all data
|
||||
flask elasticsearch migrate --dry-run
|
||||
|
||||
# Dry run for specific tenant
|
||||
flask elasticsearch migrate --tenant-id tenant-123 --dry-run
|
||||
|
||||
# Dry run for date range
|
||||
flask elasticsearch migrate --start-date 2024-01-01 --end-date 2024-01-31 --dry-run
|
||||
```
|
||||
|
||||
### 4. Incremental Migration
|
||||
|
||||
Start with recent data and work backwards:
|
||||
|
||||
```bash
|
||||
# Migrate last 7 days
|
||||
flask elasticsearch migrate --start-date $(date -d '7 days ago' +%Y-%m-%d)
|
||||
|
||||
# Migrate specific data types
|
||||
flask elasticsearch migrate --data-type workflow_runs
|
||||
flask elasticsearch migrate --data-type app_logs
|
||||
flask elasticsearch migrate --data-type node_executions
|
||||
```
|
||||
|
||||
### 5. Full Migration
|
||||
|
||||
Migrate all historical data:
|
||||
|
||||
```bash
|
||||
# Migrate all data (use appropriate batch size)
|
||||
flask elasticsearch migrate --batch-size 500
|
||||
|
||||
# Migrate specific tenant
|
||||
flask elasticsearch migrate --tenant-id tenant-123
|
||||
```
|
||||
|
||||
### 6. Validation
|
||||
|
||||
Validate the migrated data:
|
||||
|
||||
```bash
|
||||
# Validate migration for tenant
|
||||
flask elasticsearch validate --tenant-id tenant-123 --sample-size 1000
|
||||
```
|
||||
|
||||
### 7. Switch Configuration
|
||||
|
||||
Once validation passes, update your configuration to use Elasticsearch repositories and restart the application.
|
||||
|
||||
### 8. Cleanup (Optional)
|
||||
|
||||
After successful migration and validation, clean up old PostgreSQL data:
|
||||
|
||||
```bash
|
||||
# Dry run cleanup
|
||||
flask elasticsearch cleanup-pg --tenant-id tenant-123 --before-date 2024-01-01 --dry-run
|
||||
|
||||
# Actual cleanup (CAUTION: This cannot be undone)
|
||||
flask elasticsearch cleanup-pg --tenant-id tenant-123 --before-date 2024-01-01
|
||||
```
|
||||
|
||||
## Index Management
|
||||
|
||||
### Index Structure
|
||||
|
||||
Elasticsearch indices are organized as:
|
||||
- `dify-workflow-runs-{tenant_id}-{YYYY.MM}`
|
||||
- `dify-workflow-app-logs-{tenant_id}-{YYYY.MM}`
|
||||
- `dify-workflow-node-executions-{tenant_id}-{YYYY.MM}`
|
||||
|
||||
### Retention Policy
|
||||
|
||||
Configure automatic cleanup of old indices:
|
||||
|
||||
```python
|
||||
# In your scheduled tasks
|
||||
from services.elasticsearch_migration_service import ElasticsearchMigrationService
|
||||
|
||||
migration_service = ElasticsearchMigrationService()
|
||||
|
||||
# Clean up indices older than 30 days
|
||||
for tenant_id in get_all_tenant_ids():
|
||||
migration_service._workflow_run_repo.cleanup_old_indices(tenant_id, retention_days=30)
|
||||
migration_service._app_log_repo.cleanup_old_indices(tenant_id, retention_days=30)
|
||||
```
|
||||
|
||||
## Performance Tuning
|
||||
|
||||
### Elasticsearch Settings
|
||||
|
||||
Optimize Elasticsearch for log data:
|
||||
|
||||
```json
|
||||
{
|
||||
"settings": {
|
||||
"number_of_shards": 1,
|
||||
"number_of_replicas": 0,
|
||||
"index.refresh_interval": "30s",
|
||||
"index.mapping.total_fields.limit": 2000
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Batch Processing
|
||||
|
||||
Adjust batch sizes based on your system:
|
||||
|
||||
```bash
|
||||
# Smaller batches for limited memory
|
||||
flask elasticsearch migrate --batch-size 100
|
||||
|
||||
# Larger batches for high-performance systems
|
||||
flask elasticsearch migrate --batch-size 5000
|
||||
```
|
||||
|
||||
## Monitoring
|
||||
|
||||
### Check Migration Progress
|
||||
|
||||
```bash
|
||||
# Monitor Elasticsearch status
|
||||
flask elasticsearch status
|
||||
|
||||
# Check specific tenant indices
|
||||
flask elasticsearch status --tenant-id tenant-123
|
||||
```
|
||||
|
||||
### Query Performance
|
||||
|
||||
Monitor query performance in your application logs and Elasticsearch slow query logs.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Common Issues
|
||||
|
||||
1. **Connection Timeout**
|
||||
- Increase `ELASTICSEARCH_TIMEOUT`
|
||||
- Check network connectivity
|
||||
- Verify Elasticsearch is running
|
||||
|
||||
2. **Memory Issues**
|
||||
- Reduce batch size
|
||||
- Increase JVM heap size for Elasticsearch
|
||||
- Process data in smaller date ranges
|
||||
|
||||
3. **Index Template Conflicts**
|
||||
- Delete existing templates: `DELETE _index_template/dify-*-template`
|
||||
- Restart migration
|
||||
|
||||
4. **Data Validation Failures**
|
||||
- Check Elasticsearch logs for indexing errors
|
||||
- Verify data integrity in PostgreSQL
|
||||
- Re-run migration for failed records
|
||||
|
||||
### Recovery
|
||||
|
||||
If migration fails:
|
||||
|
||||
1. Check logs for specific errors
|
||||
2. Fix configuration issues
|
||||
3. Resume migration from last successful point
|
||||
4. Use date ranges to process data incrementally
|
||||
|
||||
## Best Practices
|
||||
|
||||
1. **Test First**: Always run dry runs and validate on staging
|
||||
2. **Incremental Migration**: Start with recent data, migrate incrementally
|
||||
3. **Monitor Resources**: Watch CPU, memory, and disk usage during migration
|
||||
4. **Backup**: Ensure PostgreSQL backups before cleanup
|
||||
5. **Gradual Rollout**: Switch tenants to Elasticsearch gradually
|
||||
6. **Index Lifecycle**: Implement proper index rotation and cleanup
|
||||
|
||||
## Example Migration Script
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
|
||||
# Complete migration workflow
|
||||
TENANT_ID="tenant-123"
|
||||
START_DATE="2024-01-01"
|
||||
|
||||
echo "Starting Elasticsearch migration for $TENANT_ID"
|
||||
|
||||
# 1. Dry run
|
||||
echo "Performing dry run..."
|
||||
flask elasticsearch migrate --tenant-id $TENANT_ID --start-date $START_DATE --dry-run
|
||||
|
||||
# 2. Migrate data
|
||||
echo "Migrating data..."
|
||||
flask elasticsearch migrate --tenant-id $TENANT_ID --start-date $START_DATE --batch-size 1000
|
||||
|
||||
# 3. Validate
|
||||
echo "Validating migration..."
|
||||
flask elasticsearch validate --tenant-id $TENANT_ID --sample-size 500
|
||||
|
||||
# 4. Check status
|
||||
echo "Checking status..."
|
||||
flask elasticsearch status --tenant-id $TENANT_ID
|
||||
|
||||
echo "Migration completed for $TENANT_ID"
|
||||
```
|
||||
|
||||
## Support
|
||||
|
||||
For issues or questions:
|
||||
1. Check application logs for detailed error messages
|
||||
2. Review Elasticsearch cluster logs
|
||||
3. Verify configuration settings
|
||||
4. Test with smaller datasets first
|
||||
@ -1,91 +0,0 @@
|
||||
# WorkflowRun API 数据问题修复总结
|
||||
|
||||
## 🎯 **问题解决状态**
|
||||
|
||||
✅ **已修复**: API 现在应该能返回多条 WorkflowRun 数据
|
||||
|
||||
## 🔍 **问题根源分析**
|
||||
|
||||
通过参考 SQL 实现，我发现了关键问题：
|
||||
|
||||
### SQL 实现的逻辑
|
||||
```python
|
||||
# SQLAlchemyWorkflowExecutionRepository.save()
|
||||
def save(self, execution: WorkflowExecution):
|
||||
# 1. 将 WorkflowExecution 转换为 WorkflowRun 数据库模型
|
||||
db_model = self._to_db_model(execution)
|
||||
|
||||
# 2. 保存到 workflow_runs 表
|
||||
session.merge(db_model)
|
||||
session.commit()
|
||||
```
|
||||
|
||||
### 我们的 Elasticsearch 实现
|
||||
```python
|
||||
# ElasticsearchWorkflowExecutionRepository.save()
|
||||
def save(self, execution: WorkflowExecution):
|
||||
# 1. 将 WorkflowExecution 转换为 WorkflowRun 格式的文档
|
||||
run_doc = self._to_workflow_run_document(execution)
|
||||
|
||||
# 2. 保存到 dify-workflow-runs-* 索引
|
||||
self._es_client.index(index=run_index, id=execution.id_, body=run_doc)
|
||||
```
|
||||
|
||||
## ✅ **修复的关键点**
|
||||
|
||||
### 1. **数据格式对齐**
|
||||
- 完全按照 SQL 实现的 `_to_db_model()` 逻辑
|
||||
- 确保字段名和数据类型与 `WorkflowRun` 模型一致
|
||||
- 正确计算 `elapsed_time`
|
||||
|
||||
### 2. **复杂对象序列化**
|
||||
- 使用 `jsonable_encoder` 处理 `ArrayFileSegment` 等复杂对象
|
||||
- 避免 JSON 序列化错误
|
||||
|
||||
### 3. **查询类型匹配**
|
||||
- API 查询 `debugging` 类型的记录
|
||||
- 这与实际保存的数据类型一致
|
||||
|
||||
## 📊 **当前数据状态**
|
||||
|
||||
### Elasticsearch 中的数据
|
||||
- **您的应用**: 2条 `debugging` 类型的 WorkflowRun 记录
|
||||
- **最新记录**: 2025-10-10 执行成功
|
||||
- **数据完整**: 包含完整的 inputs, outputs, graph 等信息
|
||||
|
||||
### API 查询结果
|
||||
现在 `/console/api/apps/{app_id}/advanced-chat/workflow-runs` 应该返回这2条记录
|
||||
|
||||
## 🚀 **验证步骤**
|
||||
|
||||
1. **重启应用** (如果还没有重启)
|
||||
2. **访问 API**: 检查是否返回多条记录
|
||||
3. **执行新工作流**: 在前端执行新的对话，应该会增加新记录
|
||||
4. **检查数据**: 新记录应该立即出现在 API 响应中
|
||||
|
||||
## 📋 **数据流程确认**
|
||||
|
||||
```
|
||||
前端执行工作流
|
||||
↓
|
||||
WorkflowCycleManager (debugging 模式)
|
||||
↓
|
||||
ElasticsearchWorkflowExecutionRepository.save()
|
||||
↓
|
||||
转换为 WorkflowRun 格式并保存到 ES
|
||||
↓
|
||||
API 查询 debugging 类型的记录
|
||||
↓
|
||||
返回完整的工作流运行列表 ✅
|
||||
```
|
||||
|
||||
## 🎉 **结论**
|
||||
|
||||
问题已经解决！您的 Elasticsearch 集成现在：
|
||||
|
||||
1. ✅ **正确保存数据**: 按照 SQL 实现的逻辑保存 WorkflowRun 数据
|
||||
2. ✅ **处理复杂对象**: 正确序列化 ArrayFileSegment 等复杂类型
|
||||
3. ✅ **查询逻辑正确**: API 查询正确的数据类型
|
||||
4. ✅ **数据完整性**: 包含所有必要的字段和元数据
|
||||
|
||||
现在 API 应该能返回您执行的所有工作流记录了！
|
||||
@ -1,109 +0,0 @@
|
||||
# WorkflowRun API 数据问题分析和解决方案
|
||||
|
||||
## 🔍 **问题分析**
|
||||
|
||||
您遇到的问题是：`/console/api/apps/{app_id}/advanced-chat/workflow-runs` API 只返回一条数据，但实际执行了多次工作流。
|
||||
|
||||
### 根本原因
|
||||
|
||||
1. **数据存储分离**:
|
||||
- `WorkflowExecution` (域模型) → 存储在 `dify-workflow-executions-*` 索引
|
||||
- `WorkflowRun` (数据库模型) → 存储在 `dify-workflow-runs-*` 索引
|
||||
- API 查询的是 `WorkflowRun` 数据
|
||||
|
||||
2. **查询类型过滤**:
|
||||
- API 只查询 `triggered_from == debugging` 的记录
|
||||
- 但前端执行的工作流可能是 `app-run` 类型
|
||||
|
||||
3. **数据同步缺失**:
|
||||
- 系统创建了 `WorkflowExecution` 记录（65条）
|
||||
- 但没有创建对应的 `WorkflowRun` 记录
|
||||
|
||||
## ✅ **解决方案**
|
||||
|
||||
### 1. 修改 WorkflowExecutionRepository
|
||||
我已经修改了 `ElasticsearchWorkflowExecutionRepository.save()` 方法，现在它会：
|
||||
- 保存 `WorkflowExecution` 数据到 `workflow-executions` 索引
|
||||
- 同时保存对应的 `WorkflowRun` 数据到 `workflow-runs` 索引
|
||||
|
||||
### 2. 修改查询逻辑
|
||||
修改了 `WorkflowRunService.get_paginate_advanced_chat_workflow_runs()` 方法：
|
||||
- 从查询 `debugging` 类型改为查询 `app-run` 类型
|
||||
- 这样可以返回用户在前端执行的工作流记录
|
||||
|
||||
## 🚀 **测试步骤**
|
||||
|
||||
### 1. 重启应用
|
||||
使用新的配置重启 Dify API 服务
|
||||
|
||||
### 2. 执行新的工作流
|
||||
在前端执行一个新的工作流对话
|
||||
|
||||
### 3. 检查数据
|
||||
```bash
|
||||
# 检查 Elasticsearch 中的数据
|
||||
curl -k -u elastic:2gYvv6+O36PGwaVD6yzE -X GET "https://localhost:9200/dify-workflow-runs-*/_search?pretty&size=1"
|
||||
|
||||
# 检查 triggered_from 统计
|
||||
curl -k -u elastic:2gYvv6+O36PGwaVD6yzE -X GET "https://localhost:9200/dify-workflow-runs-*/_search?pretty" -H 'Content-Type: application/json' -d '{
|
||||
"size": 0,
|
||||
"aggs": {
|
||||
"triggered_from_stats": {
|
||||
"terms": {
|
||||
"field": "triggered_from"
|
||||
}
|
||||
}
|
||||
}
|
||||
}'
|
||||
```
|
||||
|
||||
### 4. 测试 API
|
||||
访问 `http://localhost:5001/console/api/apps/2b517b83-ecd1-4097-83e4-48bc626fd0af/advanced-chat/workflow-runs`
|
||||
|
||||
## 📊 **数据流程图**
|
||||
|
||||
```
|
||||
前端执行工作流
|
||||
↓
|
||||
WorkflowCycleManager.handle_workflow_run_start()
|
||||
↓
|
||||
WorkflowExecutionRepository.save(WorkflowExecution)
|
||||
↓
|
||||
ElasticsearchWorkflowExecutionRepository.save()
|
||||
↓
|
||||
保存到两个索引：
|
||||
├── dify-workflow-executions-* (WorkflowExecution 数据)
|
||||
└── dify-workflow-runs-* (WorkflowRun 数据)
|
||||
↓
|
||||
API 查询 workflow-runs 索引
|
||||
↓
|
||||
返回完整的工作流运行列表
|
||||
```
|
||||
|
||||
## 🔧 **配置要求**
|
||||
|
||||
确保您的 `.env` 文件包含：
|
||||
|
||||
```bash
|
||||
# Elasticsearch 配置
|
||||
ELASTICSEARCH_ENABLED=true
|
||||
ELASTICSEARCH_HOSTS=["https://localhost:9200"]
|
||||
ELASTICSEARCH_USERNAME=elastic
|
||||
ELASTICSEARCH_PASSWORD=2gYvv6+O36PGwaVD6yzE
|
||||
ELASTICSEARCH_USE_SSL=true
|
||||
ELASTICSEARCH_VERIFY_CERTS=false
|
||||
|
||||
# Repository 配置
|
||||
CORE_WORKFLOW_EXECUTION_REPOSITORY=core.repositories.elasticsearch_workflow_execution_repository.ElasticsearchWorkflowExecutionRepository
|
||||
CORE_WORKFLOW_NODE_EXECUTION_REPOSITORY=core.repositories.elasticsearch_workflow_node_execution_repository.ElasticsearchWorkflowNodeExecutionRepository
|
||||
API_WORKFLOW_RUN_REPOSITORY=repositories.elasticsearch_api_workflow_run_repository.ElasticsearchAPIWorkflowRunRepository
|
||||
```
|
||||
|
||||
## 🎯 **预期结果**
|
||||
|
||||
修复后，您应该能够：
|
||||
1. 在前端执行多次工作流
|
||||
2. API 返回所有执行的工作流记录
|
||||
3. 数据同时存储在两个索引中，保持一致性
|
||||
|
||||
现在重启应用并测试新的工作流执行，应该可以看到完整的运行历史了！
|
||||
@ -10,14 +10,14 @@ from dify_app import DifyApp
|
||||
|
||||
def init_app(app: DifyApp):
|
||||
@app.after_request
|
||||
def after_request(response): # pyright: ignore[reportUnusedFunction]
|
||||
def after_request(response):
|
||||
"""Add Version headers to the response."""
|
||||
response.headers.add("X-Version", dify_config.project.version)
|
||||
response.headers.add("X-Env", dify_config.DEPLOY_ENV)
|
||||
return response
|
||||
|
||||
@app.route("/health")
|
||||
def health(): # pyright: ignore[reportUnusedFunction]
|
||||
def health():
|
||||
return Response(
|
||||
json.dumps({"pid": os.getpid(), "status": "ok", "version": dify_config.project.version}),
|
||||
status=200,
|
||||
@ -25,7 +25,7 @@ def init_app(app: DifyApp):
|
||||
)
|
||||
|
||||
@app.route("/threads")
|
||||
def threads(): # pyright: ignore[reportUnusedFunction]
|
||||
def threads():
|
||||
num_threads = threading.active_count()
|
||||
threads = threading.enumerate()
|
||||
|
||||
@ -50,7 +50,7 @@ def init_app(app: DifyApp):
|
||||
}
|
||||
|
||||
@app.route("/db-pool-stat")
|
||||
def pool_stat(): # pyright: ignore[reportUnusedFunction]
|
||||
def pool_stat():
|
||||
from extensions.ext_database import db
|
||||
|
||||
engine = db.engine
|
||||
|
||||
@ -9,7 +9,6 @@ def init_app(app: DifyApp):
|
||||
clear_orphaned_file_records,
|
||||
convert_to_agent_apps,
|
||||
create_tenant,
|
||||
elasticsearch,
|
||||
extract_plugins,
|
||||
extract_unique_plugins,
|
||||
fix_app_site_missing,
|
||||
@ -43,7 +42,6 @@ def init_app(app: DifyApp):
|
||||
extract_plugins,
|
||||
extract_unique_plugins,
|
||||
install_plugins,
|
||||
elasticsearch,
|
||||
old_metadata_migration,
|
||||
clear_free_plan_tenant_expired_logs,
|
||||
clear_orphaned_file_records,
|
||||
|
||||
@ -10,7 +10,7 @@ from models.engine import db
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# Global flag to avoid duplicate registration of event listener
|
||||
_gevent_compatibility_setup: bool = False
|
||||
_GEVENT_COMPATIBILITY_SETUP: bool = False
|
||||
|
||||
|
||||
def _safe_rollback(connection):
|
||||
@ -26,14 +26,14 @@ def _safe_rollback(connection):
|
||||
|
||||
|
||||
def _setup_gevent_compatibility():
|
||||
global _gevent_compatibility_setup # pylint: disable=global-statement
|
||||
global _GEVENT_COMPATIBILITY_SETUP # pylint: disable=global-statement
|
||||
|
||||
# Avoid duplicate registration
|
||||
if _gevent_compatibility_setup:
|
||||
if _GEVENT_COMPATIBILITY_SETUP:
|
||||
return
|
||||
|
||||
@event.listens_for(Pool, "reset")
|
||||
def _safe_reset(dbapi_connection, connection_record, reset_state): # pyright: ignore[reportUnusedFunction]
|
||||
def _safe_reset(dbapi_connection, connection_record, reset_state): # pylint: disable=unused-argument
|
||||
if reset_state.terminate_only:
|
||||
return
|
||||
|
||||
@ -47,7 +47,7 @@ def _setup_gevent_compatibility():
|
||||
except (AttributeError, ImportError):
|
||||
_safe_rollback(dbapi_connection)
|
||||
|
||||
_gevent_compatibility_setup = True
|
||||
_GEVENT_COMPATIBILITY_SETUP = True
|
||||
|
||||
|
||||
def init_app(app: DifyApp):
|
||||
|
||||
@ -1,119 +0,0 @@
|
||||
"""
|
||||
Elasticsearch extension for Dify.
|
||||
|
||||
This module provides Elasticsearch client configuration and initialization
|
||||
for storing workflow logs and execution data.
|
||||
"""
|
||||
|
||||
import logging
|
||||
from typing import Optional
|
||||
|
||||
from elasticsearch import Elasticsearch
|
||||
from flask import Flask
|
||||
|
||||
from configs import dify_config
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class ElasticsearchExtension:
|
||||
"""
|
||||
Elasticsearch extension for Flask application.
|
||||
|
||||
Provides centralized Elasticsearch client management with proper
|
||||
configuration and connection handling.
|
||||
"""
|
||||
|
||||
def __init__(self):
|
||||
self._client: Optional[Elasticsearch] = None
|
||||
|
||||
def init_app(self, app: Flask) -> None:
|
||||
"""
|
||||
Initialize Elasticsearch extension with Flask app.
|
||||
|
||||
Args:
|
||||
app: Flask application instance
|
||||
"""
|
||||
# Only initialize if Elasticsearch is enabled
|
||||
if not dify_config.ELASTICSEARCH_ENABLED:
|
||||
logger.info("Elasticsearch is disabled, skipping initialization")
|
||||
return
|
||||
|
||||
try:
|
||||
# Create Elasticsearch client with configuration
|
||||
client_config = {
|
||||
"hosts": dify_config.ELASTICSEARCH_HOSTS,
|
||||
"timeout": dify_config.ELASTICSEARCH_TIMEOUT,
|
||||
"max_retries": dify_config.ELASTICSEARCH_MAX_RETRIES,
|
||||
"retry_on_timeout": True,
|
||||
}
|
||||
|
||||
# Add authentication if configured
|
||||
if dify_config.ELASTICSEARCH_USERNAME and dify_config.ELASTICSEARCH_PASSWORD:
|
||||
client_config["http_auth"] = (
|
||||
dify_config.ELASTICSEARCH_USERNAME,
|
||||
dify_config.ELASTICSEARCH_PASSWORD,
|
||||
)
|
||||
|
||||
# Add SSL configuration if enabled
|
||||
if dify_config.ELASTICSEARCH_USE_SSL:
|
||||
client_config["verify_certs"] = dify_config.ELASTICSEARCH_VERIFY_CERTS
|
||||
|
||||
if dify_config.ELASTICSEARCH_CA_CERTS:
|
||||
client_config["ca_certs"] = dify_config.ELASTICSEARCH_CA_CERTS
|
||||
|
||||
self._client = Elasticsearch(**client_config)
|
||||
|
||||
# Test connection
|
||||
if self._client.ping():
|
||||
logger.info("Elasticsearch connection established successfully")
|
||||
else:
|
||||
logger.error("Failed to connect to Elasticsearch")
|
||||
self._client = None
|
||||
|
||||
except Exception as e:
|
||||
logger.error("Failed to initialize Elasticsearch client: %s", e)
|
||||
self._client = None
|
||||
|
||||
# Store client in app context
|
||||
app.elasticsearch = self._client
|
||||
|
||||
@property
|
||||
def client(self) -> Optional[Elasticsearch]:
|
||||
"""
|
||||
Get the Elasticsearch client instance.
|
||||
|
||||
Returns:
|
||||
Elasticsearch client if available, None otherwise
|
||||
"""
|
||||
return self._client
|
||||
|
||||
def is_available(self) -> bool:
|
||||
"""
|
||||
Check if Elasticsearch is available and connected.
|
||||
|
||||
Returns:
|
||||
True if Elasticsearch is available, False otherwise
|
||||
"""
|
||||
if not self._client:
|
||||
return False
|
||||
|
||||
try:
|
||||
return self._client.ping()
|
||||
except Exception:
|
||||
return False
|
||||
|
||||
|
||||
# Global Elasticsearch extension instance
|
||||
elasticsearch = ElasticsearchExtension()
|
||||
|
||||
|
||||
def init_app(app):
|
||||
"""Initialize Elasticsearch extension with Flask app."""
|
||||
elasticsearch.init_app(app)
|
||||
|
||||
|
||||
def is_enabled():
|
||||
"""Check if Elasticsearch extension is enabled."""
|
||||
from configs import dify_config
|
||||
return dify_config.ELASTICSEARCH_ENABLED
|
||||
@ -2,4 +2,4 @@ from dify_app import DifyApp
|
||||
|
||||
|
||||
def init_app(app: DifyApp):
|
||||
from events import event_handlers # noqa: F401 # pyright: ignore[reportUnusedImport]
|
||||
from events import event_handlers # noqa: F401
|
||||
|
||||
@ -136,7 +136,6 @@ def init_app(app: DifyApp):
|
||||
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter as HTTPSpanExporter
|
||||
from opentelemetry.instrumentation.celery import CeleryInstrumentor
|
||||
from opentelemetry.instrumentation.flask import FlaskInstrumentor
|
||||
from opentelemetry.instrumentation.httpx import HTTPXClientInstrumentor
|
||||
from opentelemetry.instrumentation.redis import RedisInstrumentor
|
||||
from opentelemetry.instrumentation.requests import RequestsInstrumentor
|
||||
from opentelemetry.instrumentation.sqlalchemy import SQLAlchemyInstrumentor
|
||||
@ -239,7 +238,6 @@ def init_app(app: DifyApp):
|
||||
init_sqlalchemy_instrumentor(app)
|
||||
RedisInstrumentor().instrument()
|
||||
RequestsInstrumentor().instrument()
|
||||
HTTPXClientInstrumentor().instrument()
|
||||
atexit.register(shutdown_tracer)
|
||||
|
||||
|
||||
|
||||
@ -4,6 +4,7 @@ from dify_app import DifyApp
|
||||
|
||||
def init_app(app: DifyApp):
|
||||
if dify_config.SENTRY_DSN:
|
||||
import openai
|
||||
import sentry_sdk
|
||||
from langfuse import parse_error # type: ignore
|
||||
from sentry_sdk.integrations.celery import CeleryIntegration
|
||||
@ -27,6 +28,7 @@ def init_app(app: DifyApp):
|
||||
HTTPException,
|
||||
ValueError,
|
||||
FileNotFoundError,
|
||||
openai.APIStatusError,
|
||||
InvokeRateLimitError,
|
||||
parse_error.defaultErrorResponse,
|
||||
],
|
||||
|
||||
@ -33,9 +33,7 @@ class AliyunOssStorage(BaseStorage):
|
||||
|
||||
def load_once(self, filename: str) -> bytes:
|
||||
obj = self.client.get_object(self.__wrapper_folder_filename(filename))
|
||||
data = obj.read()
|
||||
if not isinstance(data, bytes):
|
||||
return b""
|
||||
data: bytes = obj.read()
|
||||
return data
|
||||
|
||||
def load_stream(self, filename: str) -> Generator:
|
||||
|
||||
@ -39,10 +39,10 @@ class AwsS3Storage(BaseStorage):
|
||||
self.client.head_bucket(Bucket=self.bucket_name)
|
||||
except ClientError as e:
|
||||
# if bucket not exists, create it
|
||||
if e.response.get("Error", {}).get("Code") == "404":
|
||||
if e.response["Error"]["Code"] == "404":
|
||||
self.client.create_bucket(Bucket=self.bucket_name)
|
||||
# if bucket is not accessible, pass, maybe the bucket is existing but not accessible
|
||||
elif e.response.get("Error", {}).get("Code") == "403":
|
||||
elif e.response["Error"]["Code"] == "403":
|
||||
pass
|
||||
else:
|
||||
# other error, raise exception
|
||||
@ -55,7 +55,7 @@ class AwsS3Storage(BaseStorage):
|
||||
try:
|
||||
data: bytes = self.client.get_object(Bucket=self.bucket_name, Key=filename)["Body"].read()
|
||||
except ClientError as ex:
|
||||
if ex.response.get("Error", {}).get("Code") == "NoSuchKey":
|
||||
if ex.response["Error"]["Code"] == "NoSuchKey":
|
||||
raise FileNotFoundError("File not found")
|
||||
else:
|
||||
raise
|
||||
@ -66,7 +66,7 @@ class AwsS3Storage(BaseStorage):
|
||||
response = self.client.get_object(Bucket=self.bucket_name, Key=filename)
|
||||
yield from response["Body"].iter_chunks()
|
||||
except ClientError as ex:
|
||||
if ex.response.get("Error", {}).get("Code") == "NoSuchKey":
|
||||
if ex.response["Error"]["Code"] == "NoSuchKey":
|
||||
raise FileNotFoundError("file not found")
|
||||
elif "reached max retries" in str(ex):
|
||||
raise ValueError("please do not request the same file too frequently")
|
||||
|
||||
@ -27,38 +27,24 @@ class AzureBlobStorage(BaseStorage):
|
||||
self.credential = None
|
||||
|
||||
def save(self, filename, data):
|
||||
if not self.bucket_name:
|
||||
return
|
||||
|
||||
client = self._sync_client()
|
||||
blob_container = client.get_container_client(container=self.bucket_name)
|
||||
blob_container.upload_blob(filename, data)
|
||||
|
||||
def load_once(self, filename: str) -> bytes:
|
||||
if not self.bucket_name:
|
||||
raise FileNotFoundError("Azure bucket name is not configured.")
|
||||
|
||||
client = self._sync_client()
|
||||
blob = client.get_container_client(container=self.bucket_name)
|
||||
blob = blob.get_blob_client(blob=filename)
|
||||
data = blob.download_blob().readall()
|
||||
if not isinstance(data, bytes):
|
||||
raise TypeError(f"Expected bytes from blob.readall(), got {type(data).__name__}")
|
||||
data: bytes = blob.download_blob().readall()
|
||||
return data
|
||||
|
||||
def load_stream(self, filename: str) -> Generator:
|
||||
if not self.bucket_name:
|
||||
raise FileNotFoundError("Azure bucket name is not configured.")
|
||||
|
||||
client = self._sync_client()
|
||||
blob = client.get_blob_client(container=self.bucket_name, blob=filename)
|
||||
blob_data = blob.download_blob()
|
||||
yield from blob_data.chunks()
|
||||
|
||||
def download(self, filename, target_filepath):
|
||||
if not self.bucket_name:
|
||||
return
|
||||
|
||||
client = self._sync_client()
|
||||
|
||||
blob = client.get_blob_client(container=self.bucket_name, blob=filename)
|
||||
@ -67,18 +53,12 @@ class AzureBlobStorage(BaseStorage):
|
||||
blob_data.readinto(my_blob)
|
||||
|
||||
def exists(self, filename):
|
||||
if not self.bucket_name:
|
||||
return False
|
||||
|
||||
client = self._sync_client()
|
||||
|
||||
blob = client.get_blob_client(container=self.bucket_name, blob=filename)
|
||||
return blob.exists()
|
||||
|
||||
def delete(self, filename):
|
||||
if not self.bucket_name:
|
||||
return
|
||||
|
||||
client = self._sync_client()
|
||||
|
||||
blob_container = client.get_container_client(container=self.bucket_name)
|
||||
|
||||
@ -430,7 +430,7 @@ class ClickZettaVolumeStorage(BaseStorage):
|
||||
|
||||
rows = self._execute_sql(sql, fetch=True)
|
||||
|
||||
exists = len(rows) > 0 if rows else False
|
||||
exists = len(rows) > 0
|
||||
logger.debug("File %s exists check: %s", filename, exists)
|
||||
return exists
|
||||
except Exception as e:
|
||||
@ -509,17 +509,16 @@ class ClickZettaVolumeStorage(BaseStorage):
|
||||
rows = self._execute_sql(sql, fetch=True)
|
||||
|
||||
result = []
|
||||
if rows:
|
||||
for row in rows:
|
||||
file_path = row[0] # relative_path column
|
||||
for row in rows:
|
||||
file_path = row[0] # relative_path column
|
||||
|
||||
# For User Volume, remove dify prefix from results
|
||||
dify_prefix_with_slash = f"{self._config.dify_prefix}/"
|
||||
if volume_prefix == "USER VOLUME" and file_path.startswith(dify_prefix_with_slash):
|
||||
file_path = file_path[len(dify_prefix_with_slash) :] # Remove prefix
|
||||
# For User Volume, remove dify prefix from results
|
||||
dify_prefix_with_slash = f"{self._config.dify_prefix}/"
|
||||
if volume_prefix == "USER VOLUME" and file_path.startswith(dify_prefix_with_slash):
|
||||
file_path = file_path[len(dify_prefix_with_slash) :] # Remove prefix
|
||||
|
||||
if files and not file_path.endswith("/") or directories and file_path.endswith("/"):
|
||||
result.append(file_path)
|
||||
if files and not file_path.endswith("/") or directories and file_path.endswith("/"):
|
||||
result.append(file_path)
|
||||
|
||||
logger.debug("Scanned %d items in path %s", len(result), path)
|
||||
return result
|
||||
|
||||
@ -439,11 +439,6 @@ class VolumePermissionManager:
|
||||
self._permission_cache.clear()
|
||||
logger.debug("Permission cache cleared")
|
||||
|
||||
@property
|
||||
def volume_type(self) -> str | None:
|
||||
"""Get the volume type."""
|
||||
return self._volume_type
|
||||
|
||||
def get_permission_summary(self, dataset_id: str | None = None) -> dict[str, bool]:
|
||||
"""Get permission summary
|
||||
|
||||
@ -637,13 +632,13 @@ def check_volume_permission(permission_manager: VolumePermissionManager, operati
|
||||
VolumePermissionError: If no permission
|
||||
"""
|
||||
if not permission_manager.validate_operation(operation, dataset_id):
|
||||
error_message = f"Permission denied for operation '{operation}' on {permission_manager.volume_type} volume"
|
||||
error_message = f"Permission denied for operation '{operation}' on {permission_manager._volume_type} volume"
|
||||
if dataset_id:
|
||||
error_message += f" (dataset: {dataset_id})"
|
||||
|
||||
raise VolumePermissionError(
|
||||
error_message,
|
||||
operation=operation,
|
||||
volume_type=permission_manager.volume_type or "unknown",
|
||||
volume_type=permission_manager._volume_type or "unknown",
|
||||
dataset_id=dataset_id,
|
||||
)
|
||||
|
||||
@ -35,16 +35,12 @@ class GoogleCloudStorage(BaseStorage):
|
||||
def load_once(self, filename: str) -> bytes:
|
||||
bucket = self.client.get_bucket(self.bucket_name)
|
||||
blob = bucket.get_blob(filename)
|
||||
if blob is None:
|
||||
raise FileNotFoundError("File not found")
|
||||
data: bytes = blob.download_as_bytes()
|
||||
return data
|
||||
|
||||
def load_stream(self, filename: str) -> Generator:
|
||||
bucket = self.client.get_bucket(self.bucket_name)
|
||||
blob = bucket.get_blob(filename)
|
||||
if blob is None:
|
||||
raise FileNotFoundError("File not found")
|
||||
with blob.open(mode="rb") as blob_stream:
|
||||
while chunk := blob_stream.read(4096):
|
||||
yield chunk
|
||||
@ -52,8 +48,6 @@ class GoogleCloudStorage(BaseStorage):
|
||||
def download(self, filename, target_filepath):
|
||||
bucket = self.client.get_bucket(self.bucket_name)
|
||||
blob = bucket.get_blob(filename)
|
||||
if blob is None:
|
||||
raise FileNotFoundError("File not found")
|
||||
blob.download_to_filename(target_filepath)
|
||||
|
||||
def exists(self, filename):
|
||||
|
||||
@ -45,7 +45,7 @@ class HuaweiObsStorage(BaseStorage):
|
||||
|
||||
def _get_meta(self, filename):
|
||||
res = self.client.getObjectMetadata(bucketName=self.bucket_name, objectKey=filename)
|
||||
if res and res.status and res.status < 300:
|
||||
if res.status < 300:
|
||||
return res
|
||||
else:
|
||||
return None
|
||||
|
||||
@ -3,9 +3,9 @@ import os
|
||||
from collections.abc import Generator
|
||||
from pathlib import Path
|
||||
|
||||
import opendal
|
||||
from dotenv import dotenv_values
|
||||
from opendal import Operator
|
||||
from opendal.layers import RetryLayer
|
||||
|
||||
from extensions.storage.base_storage import BaseStorage
|
||||
|
||||
@ -35,7 +35,7 @@ class OpenDALStorage(BaseStorage):
|
||||
root = kwargs.get("root", "storage")
|
||||
Path(root).mkdir(parents=True, exist_ok=True)
|
||||
|
||||
retry_layer = opendal.layers.RetryLayer(max_times=3, factor=2.0, jitter=True)
|
||||
retry_layer = RetryLayer(max_times=3, factor=2.0, jitter=True)
|
||||
self.op = Operator(scheme=scheme, **kwargs).layer(retry_layer)
|
||||
logger.debug("opendal operator created with scheme %s", scheme)
|
||||
logger.debug("added retry layer to opendal operator")
|
||||
|
||||
@ -29,7 +29,7 @@ class OracleOCIStorage(BaseStorage):
|
||||
try:
|
||||
data: bytes = self.client.get_object(Bucket=self.bucket_name, Key=filename)["Body"].read()
|
||||
except ClientError as ex:
|
||||
if ex.response.get("Error", {}).get("Code") == "NoSuchKey":
|
||||
if ex.response["Error"]["Code"] == "NoSuchKey":
|
||||
raise FileNotFoundError("File not found")
|
||||
else:
|
||||
raise
|
||||
@ -40,7 +40,7 @@ class OracleOCIStorage(BaseStorage):
|
||||
response = self.client.get_object(Bucket=self.bucket_name, Key=filename)
|
||||
yield from response["Body"].iter_chunks()
|
||||
except ClientError as ex:
|
||||
if ex.response.get("Error", {}).get("Code") == "NoSuchKey":
|
||||
if ex.response["Error"]["Code"] == "NoSuchKey":
|
||||
raise FileNotFoundError("File not found")
|
||||
else:
|
||||
raise
|
||||
|
||||
@ -46,13 +46,13 @@ class SupabaseStorage(BaseStorage):
|
||||
Path(target_filepath).write_bytes(result)
|
||||
|
||||
def exists(self, filename):
|
||||
result = self.client.storage.from_(self.bucket_name).list(path=filename)
|
||||
if len(result) > 0:
|
||||
result = self.client.storage.from_(self.bucket_name).list(filename)
|
||||
if result.count() > 0:
|
||||
return True
|
||||
return False
|
||||
|
||||
def delete(self, filename):
|
||||
self.client.storage.from_(self.bucket_name).remove([filename])
|
||||
self.client.storage.from_(self.bucket_name).remove(filename)
|
||||
|
||||
def bucket_exists(self):
|
||||
buckets = self.client.storage.list_buckets()
|
||||
|
||||
@ -11,14 +11,6 @@ class VolcengineTosStorage(BaseStorage):
|
||||
|
||||
def __init__(self):
|
||||
super().__init__()
|
||||
if not dify_config.VOLCENGINE_TOS_ACCESS_KEY:
|
||||
raise ValueError("VOLCENGINE_TOS_ACCESS_KEY is not set")
|
||||
if not dify_config.VOLCENGINE_TOS_SECRET_KEY:
|
||||
raise ValueError("VOLCENGINE_TOS_SECRET_KEY is not set")
|
||||
if not dify_config.VOLCENGINE_TOS_ENDPOINT:
|
||||
raise ValueError("VOLCENGINE_TOS_ENDPOINT is not set")
|
||||
if not dify_config.VOLCENGINE_TOS_REGION:
|
||||
raise ValueError("VOLCENGINE_TOS_REGION is not set")
|
||||
self.bucket_name = dify_config.VOLCENGINE_TOS_BUCKET_NAME
|
||||
self.client = tos.TosClientV2(
|
||||
ak=dify_config.VOLCENGINE_TOS_ACCESS_KEY,
|
||||
@ -28,39 +20,27 @@ class VolcengineTosStorage(BaseStorage):
|
||||
)
|
||||
|
||||
def save(self, filename, data):
|
||||
if not self.bucket_name:
|
||||
raise ValueError("VOLCENGINE_TOS_BUCKET_NAME is not set")
|
||||
self.client.put_object(bucket=self.bucket_name, key=filename, content=data)
|
||||
|
||||
def load_once(self, filename: str) -> bytes:
|
||||
if not self.bucket_name:
|
||||
raise FileNotFoundError("VOLCENGINE_TOS_BUCKET_NAME is not set")
|
||||
data = self.client.get_object(bucket=self.bucket_name, key=filename).read()
|
||||
if not isinstance(data, bytes):
|
||||
raise TypeError(f"Expected bytes, got {type(data).__name__}")
|
||||
return data
|
||||
|
||||
def load_stream(self, filename: str) -> Generator:
|
||||
if not self.bucket_name:
|
||||
raise FileNotFoundError("VOLCENGINE_TOS_BUCKET_NAME is not set")
|
||||
response = self.client.get_object(bucket=self.bucket_name, key=filename)
|
||||
while chunk := response.read(4096):
|
||||
yield chunk
|
||||
|
||||
def download(self, filename, target_filepath):
|
||||
if not self.bucket_name:
|
||||
raise ValueError("VOLCENGINE_TOS_BUCKET_NAME is not set")
|
||||
self.client.get_object_to_file(bucket=self.bucket_name, key=filename, file_path=target_filepath)
|
||||
|
||||
def exists(self, filename):
|
||||
if not self.bucket_name:
|
||||
return False
|
||||
res = self.client.head_object(bucket=self.bucket_name, key=filename)
|
||||
if res.status_code != 200:
|
||||
return False
|
||||
return True
|
||||
|
||||
def delete(self, filename):
|
||||
if not self.bucket_name:
|
||||
return
|
||||
self.client.delete_object(bucket=self.bucket_name, key=filename)
|
||||
|
||||
@ -1,14 +0,0 @@
|
||||
def convert_to_lower_and_upper_set(inputs: list[str] | set[str]) -> set[str]:
|
||||
"""
|
||||
Convert a list or set of strings to a set containing both lower and upper case versions of each string.
|
||||
|
||||
Args:
|
||||
inputs (list[str] | set[str]): A list or set of strings to be converted.
|
||||
|
||||
Returns:
|
||||
set[str]: A set containing both lower and upper case versions of each string.
|
||||
"""
|
||||
if not inputs:
|
||||
return set()
|
||||
else:
|
||||
return {case for s in inputs if s for case in (s.lower(), s.upper())}
|
||||
@ -1,5 +0,0 @@
|
||||
def validate_description_length(description: str | None) -> str | None:
|
||||
"""Validate description length."""
|
||||
if description and len(description) > 400:
|
||||
raise ValueError("Description cannot exceed 400 characters.")
|
||||
return description
|
||||
@ -5,6 +5,7 @@ requires-python = ">=3.11,<3.13"
|
||||
|
||||
dependencies = [
|
||||
"arize-phoenix-otel~=0.9.2",
|
||||
"authlib==1.6.4",
|
||||
"azure-identity==1.16.1",
|
||||
"beautifulsoup4==4.12.2",
|
||||
"boto3==1.35.99",
|
||||
@ -33,8 +34,10 @@ dependencies = [
|
||||
"json-repair>=0.41.1",
|
||||
"langfuse~=2.51.3",
|
||||
"langsmith~=0.1.77",
|
||||
"mailchimp-transactional~=1.0.50",
|
||||
"markdown~=3.5.1",
|
||||
"numpy~=1.26.4",
|
||||
"openai~=1.61.0",
|
||||
"openpyxl~=3.1.5",
|
||||
"opik~=1.7.25",
|
||||
"opentelemetry-api==1.27.0",
|
||||
@ -46,7 +49,6 @@ dependencies = [
|
||||
"opentelemetry-instrumentation==0.48b0",
|
||||
"opentelemetry-instrumentation-celery==0.48b0",
|
||||
"opentelemetry-instrumentation-flask==0.48b0",
|
||||
"opentelemetry-instrumentation-httpx==0.48b0",
|
||||
"opentelemetry-instrumentation-redis==0.48b0",
|
||||
"opentelemetry-instrumentation-requests==0.48b0",
|
||||
"opentelemetry-instrumentation-sqlalchemy==0.48b0",
|
||||
@ -58,6 +60,7 @@ dependencies = [
|
||||
"opentelemetry-semantic-conventions==0.48b0",
|
||||
"opentelemetry-util-http==0.48b0",
|
||||
"pandas[excel,output-formatting,performance]~=2.2.2",
|
||||
"pandoc~=2.4",
|
||||
"psycogreen~=1.0.2",
|
||||
"psycopg2-binary~=2.9.6",
|
||||
"pycryptodome==3.19.1",
|
||||
@ -175,10 +178,10 @@ dev = [
|
||||
# Required for storage clients
|
||||
############################################################
|
||||
storage = [
|
||||
"azure-storage-blob==12.26.0",
|
||||
"azure-storage-blob==12.13.0",
|
||||
"bce-python-sdk~=0.9.23",
|
||||
"cos-python-sdk-v5==1.9.38",
|
||||
"esdk-obs-python==3.25.8",
|
||||
"esdk-obs-python==3.24.6.1",
|
||||
"google-cloud-storage==2.16.0",
|
||||
"opendal~=0.46.0",
|
||||
"oss2==2.18.5",
|
||||
|
||||
@ -1,10 +1,12 @@
|
||||
{
|
||||
"include": ["."],
|
||||
"exclude": [
|
||||
"tests/",
|
||||
".venv",
|
||||
"tests/",
|
||||
"migrations/",
|
||||
"core/rag"
|
||||
"core/rag",
|
||||
"extensions",
|
||||
"core/app/app_config/easy_ui_based_app/dataset"
|
||||
],
|
||||
"typeCheckingMode": "strict",
|
||||
"allowedUntypedLibraries": [
|
||||
@ -12,7 +14,6 @@
|
||||
"flask_login",
|
||||
"opentelemetry.instrumentation.celery",
|
||||
"opentelemetry.instrumentation.flask",
|
||||
"opentelemetry.instrumentation.httpx",
|
||||
"opentelemetry.instrumentation.requests",
|
||||
"opentelemetry.instrumentation.sqlalchemy",
|
||||
"opentelemetry.instrumentation.redis"
|
||||
@ -24,6 +25,7 @@
|
||||
"reportUnknownLambdaType": "hint",
|
||||
"reportMissingParameterType": "hint",
|
||||
"reportMissingTypeArgument": "hint",
|
||||
"reportUnnecessaryContains": "hint",
|
||||
"reportUnnecessaryComparison": "hint",
|
||||
"reportUnnecessaryCast": "hint",
|
||||
"reportUnnecessaryIsInstance": "hint",
|
||||
|
||||
@ -7,7 +7,7 @@ env =
|
||||
CHATGLM_API_BASE = http://a.abc.com:11451
|
||||
CODE_EXECUTION_API_KEY = dify-sandbox
|
||||
CODE_EXECUTION_ENDPOINT = http://127.0.0.1:8194
|
||||
CODE_MAX_STRING_LENGTH = 400000
|
||||
CODE_MAX_STRING_LENGTH = 80000
|
||||
PLUGIN_DAEMON_KEY=lYkiYYT6owG+71oLerGzA7GXCgOT++6ovaezWAjpCjf+Sjc3ZtU+qUEi
|
||||
PLUGIN_DAEMON_URL=http://127.0.0.1:5002
|
||||
PLUGIN_MAX_PACKAGE_SIZE=15728640
|
||||
|
||||
@ -1,567 +0,0 @@
|
||||
"""
|
||||
Elasticsearch API WorkflowRun Repository Implementation
|
||||
|
||||
This module provides the Elasticsearch-based implementation of the APIWorkflowRunRepository
|
||||
protocol. It handles service-layer WorkflowRun database operations using Elasticsearch
|
||||
for better performance and scalability.
|
||||
|
||||
Key Features:
|
||||
- High-performance log storage and retrieval in Elasticsearch
|
||||
- Time-series data optimization with date-based index rotation
|
||||
- Full-text search capabilities for workflow run data
|
||||
- Multi-tenant data isolation through index patterns
|
||||
- Efficient pagination and filtering
|
||||
"""
|
||||
|
||||
import logging
|
||||
from collections.abc import Sequence
|
||||
from datetime import datetime, timedelta
|
||||
from typing import Any, Optional
|
||||
|
||||
from sqlalchemy.orm import sessionmaker
|
||||
|
||||
from libs.infinite_scroll_pagination import InfiniteScrollPagination
|
||||
from models.workflow import WorkflowRun
|
||||
from repositories.api_workflow_run_repository import APIWorkflowRunRepository
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class ElasticsearchAPIWorkflowRunRepository(APIWorkflowRunRepository):
|
||||
"""
|
||||
Elasticsearch implementation of APIWorkflowRunRepository.
|
||||
|
||||
Provides service-layer WorkflowRun operations using Elasticsearch for
|
||||
improved performance and scalability. Supports time-series optimization
|
||||
with automatic index rotation and multi-tenant data isolation.
|
||||
|
||||
Args:
|
||||
es_client: Elasticsearch client instance
|
||||
index_prefix: Prefix for Elasticsearch indices
|
||||
"""
|
||||
|
||||
def __init__(self, session_maker: sessionmaker, index_prefix: str = "dify-workflow-runs"):
|
||||
"""
|
||||
Initialize the repository with Elasticsearch client.
|
||||
|
||||
Args:
|
||||
session_maker: SQLAlchemy sessionmaker (for compatibility with factory pattern)
|
||||
index_prefix: Prefix for Elasticsearch indices
|
||||
"""
|
||||
# Get Elasticsearch client from global extension
|
||||
from extensions.ext_elasticsearch import elasticsearch as es_extension
|
||||
|
||||
self._es_client = es_extension.client
|
||||
if not self._es_client:
|
||||
raise ValueError("Elasticsearch client is not available. Please check your configuration.")
|
||||
|
||||
self._index_prefix = index_prefix
|
||||
|
||||
# Ensure index template exists
|
||||
self._ensure_index_template()
|
||||
|
||||
def _get_index_name(self, tenant_id: str, date: Optional[datetime] = None) -> str:
|
||||
"""
|
||||
Generate index name with date-based rotation for better performance.
|
||||
|
||||
Args:
|
||||
tenant_id: Tenant identifier for multi-tenant isolation
|
||||
date: Date for index name generation, defaults to current date
|
||||
|
||||
Returns:
|
||||
Index name in format: {prefix}-{tenant_id}-{YYYY.MM}
|
||||
"""
|
||||
if date is None:
|
||||
date = datetime.utcnow()
|
||||
|
||||
return f"{self._index_prefix}-{tenant_id}-{date.strftime('%Y.%m')}"
|
||||
|
||||
def _ensure_index_template(self):
|
||||
"""
|
||||
Ensure the index template exists for proper mapping and settings.
|
||||
"""
|
||||
template_name = f"{self._index_prefix}-template"
|
||||
template_body = {
|
||||
"index_patterns": [f"{self._index_prefix}-*"],
|
||||
"template": {
|
||||
"settings": {
|
||||
"number_of_shards": 1,
|
||||
"number_of_replicas": 0,
|
||||
"index.refresh_interval": "5s",
|
||||
"index.mapping.total_fields.limit": 2000,
|
||||
},
|
||||
"mappings": {
|
||||
"properties": {
|
||||
"id": {"type": "keyword"},
|
||||
"tenant_id": {"type": "keyword"},
|
||||
"app_id": {"type": "keyword"},
|
||||
"workflow_id": {"type": "keyword"},
|
||||
"type": {"type": "keyword"},
|
||||
"triggered_from": {"type": "keyword"},
|
||||
"version": {"type": "keyword"},
|
||||
"graph": {"type": "object", "enabled": False},
|
||||
"inputs": {"type": "object", "enabled": False},
|
||||
"status": {"type": "keyword"},
|
||||
"outputs": {"type": "object", "enabled": False},
|
||||
"error": {"type": "text"},
|
||||
"elapsed_time": {"type": "float"},
|
||||
"total_tokens": {"type": "long"},
|
||||
"total_steps": {"type": "integer"},
|
||||
"created_by_role": {"type": "keyword"},
|
||||
"created_by": {"type": "keyword"},
|
||||
"created_at": {"type": "date"},
|
||||
"finished_at": {"type": "date"},
|
||||
"exceptions_count": {"type": "integer"},
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
try:
|
||||
self._es_client.indices.put_index_template(
|
||||
name=template_name,
|
||||
body=template_body
|
||||
)
|
||||
logger.info("Index template %s created/updated successfully", template_name)
|
||||
except Exception as e:
|
||||
logger.error("Failed to create index template %s: %s", template_name, e)
|
||||
raise
|
||||
|
||||
def _to_es_document(self, workflow_run: WorkflowRun) -> dict[str, Any]:
|
||||
"""
|
||||
Convert WorkflowRun model to Elasticsearch document.
|
||||
|
||||
Args:
|
||||
workflow_run: The WorkflowRun model to convert
|
||||
|
||||
Returns:
|
||||
Dictionary representing the Elasticsearch document
|
||||
"""
|
||||
doc = {
|
||||
"id": workflow_run.id,
|
||||
"tenant_id": workflow_run.tenant_id,
|
||||
"app_id": workflow_run.app_id,
|
||||
"workflow_id": workflow_run.workflow_id,
|
||||
"type": workflow_run.type,
|
||||
"triggered_from": workflow_run.triggered_from,
|
||||
"version": workflow_run.version,
|
||||
"graph": workflow_run.graph_dict,
|
||||
"inputs": workflow_run.inputs_dict,
|
||||
"status": workflow_run.status,
|
||||
"outputs": workflow_run.outputs_dict,
|
||||
"error": workflow_run.error,
|
||||
"elapsed_time": workflow_run.elapsed_time,
|
||||
"total_tokens": workflow_run.total_tokens,
|
||||
"total_steps": workflow_run.total_steps,
|
||||
"created_by_role": workflow_run.created_by_role,
|
||||
"created_by": workflow_run.created_by,
|
||||
"created_at": workflow_run.created_at.isoformat() if workflow_run.created_at else None,
|
||||
"finished_at": workflow_run.finished_at.isoformat() if workflow_run.finished_at else None,
|
||||
"exceptions_count": workflow_run.exceptions_count,
|
||||
}
|
||||
|
||||
# Remove None values to reduce storage size
|
||||
return {k: v for k, v in doc.items() if v is not None}
|
||||
|
||||
def _from_es_document(self, doc: dict[str, Any]) -> WorkflowRun:
|
||||
"""
|
||||
Convert Elasticsearch document to WorkflowRun model.
|
||||
|
||||
Args:
|
||||
doc: Elasticsearch document
|
||||
|
||||
Returns:
|
||||
WorkflowRun model instance
|
||||
"""
|
||||
source = doc.get("_source", doc)
|
||||
|
||||
return WorkflowRun.from_dict({
|
||||
"id": source["id"],
|
||||
"tenant_id": source["tenant_id"],
|
||||
"app_id": source["app_id"],
|
||||
"workflow_id": source["workflow_id"],
|
||||
"type": source["type"],
|
||||
"triggered_from": source["triggered_from"],
|
||||
"version": source["version"],
|
||||
"graph": source.get("graph", {}),
|
||||
"inputs": source.get("inputs", {}),
|
||||
"status": source["status"],
|
||||
"outputs": source.get("outputs", {}),
|
||||
"error": source.get("error"),
|
||||
"elapsed_time": source.get("elapsed_time", 0.0),
|
||||
"total_tokens": source.get("total_tokens", 0),
|
||||
"total_steps": source.get("total_steps", 0),
|
||||
"created_by_role": source["created_by_role"],
|
||||
"created_by": source["created_by"],
|
||||
"created_at": datetime.fromisoformat(source["created_at"]) if source.get("created_at") else None,
|
||||
"finished_at": datetime.fromisoformat(source["finished_at"]) if source.get("finished_at") else None,
|
||||
"exceptions_count": source.get("exceptions_count", 0),
|
||||
})
|
||||
|
||||
def save(self, workflow_run: WorkflowRun) -> None:
|
||||
"""
|
||||
Save or update a WorkflowRun to Elasticsearch.
|
||||
|
||||
Args:
|
||||
workflow_run: The WorkflowRun to save
|
||||
"""
|
||||
try:
|
||||
index_name = self._get_index_name(workflow_run.tenant_id, workflow_run.created_at)
|
||||
doc = self._to_es_document(workflow_run)
|
||||
|
||||
self._es_client.index(
|
||||
index=index_name,
|
||||
id=workflow_run.id,
|
||||
body=doc,
|
||||
refresh="wait_for"
|
||||
)
|
||||
|
||||
logger.debug(f"Saved workflow run {workflow_run.id} to index {index_name}")
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to save workflow run {workflow_run.id}: {e}")
|
||||
raise
|
||||
|
||||
def get_paginated_workflow_runs(
|
||||
self,
|
||||
tenant_id: str,
|
||||
app_id: str,
|
||||
triggered_from: str,
|
||||
limit: int = 20,
|
||||
last_id: str | None = None,
|
||||
) -> InfiniteScrollPagination:
|
||||
"""
|
||||
Get paginated workflow runs with filtering using Elasticsearch.
|
||||
|
||||
Implements cursor-based pagination using created_at timestamps for
|
||||
efficient handling of large datasets.
|
||||
"""
|
||||
try:
|
||||
# Build query
|
||||
query = {
|
||||
"bool": {
|
||||
"must": [
|
||||
{"term": {"tenant_id": tenant_id}},
|
||||
{"term": {"app_id": app_id}},
|
||||
{"term": {"triggered_from": triggered_from}},
|
||||
]
|
||||
}
|
||||
}
|
||||
|
||||
# Handle cursor-based pagination
|
||||
sort_config = [{"created_at": {"order": "desc"}}]
|
||||
|
||||
if last_id:
|
||||
# Get the last workflow run for cursor-based pagination
|
||||
last_run = self.get_workflow_run_by_id(tenant_id, app_id, last_id)
|
||||
if not last_run:
|
||||
raise ValueError("Last workflow run not exists")
|
||||
|
||||
# Add range query for pagination
|
||||
query["bool"]["must"].append({
|
||||
"range": {
|
||||
"created_at": {
|
||||
"lt": last_run.created_at.isoformat()
|
||||
}
|
||||
}
|
||||
})
|
||||
|
||||
# Search across all indices for this tenant
|
||||
index_pattern = f"{self._index_prefix}-{tenant_id}-*"
|
||||
|
||||
response = self._es_client.search(
|
||||
index=index_pattern,
|
||||
body={
|
||||
"query": query,
|
||||
"sort": sort_config,
|
||||
"size": limit + 1, # Get one extra to check if there are more
|
||||
}
|
||||
)
|
||||
|
||||
# Convert results
|
||||
workflow_runs = []
|
||||
for hit in response["hits"]["hits"]:
|
||||
workflow_run = self._from_es_document(hit)
|
||||
workflow_runs.append(workflow_run)
|
||||
|
||||
# Check if there are more records for pagination
|
||||
has_more = len(workflow_runs) > limit
|
||||
if has_more:
|
||||
workflow_runs = workflow_runs[:-1]
|
||||
|
||||
return InfiniteScrollPagination(data=workflow_runs, limit=limit, has_more=has_more)
|
||||
|
||||
except Exception as e:
|
||||
logger.error("Failed to get paginated workflow runs: %s", e)
|
||||
raise
|
||||
|
||||
def get_workflow_run_by_id(
|
||||
self,
|
||||
tenant_id: str,
|
||||
app_id: str,
|
||||
run_id: str,
|
||||
) -> WorkflowRun | None:
|
||||
"""
|
||||
Get a specific workflow run by ID with tenant and app isolation.
|
||||
"""
|
||||
try:
|
||||
query = {
|
||||
"bool": {
|
||||
"must": [
|
||||
{"term": {"id": run_id}},
|
||||
{"term": {"tenant_id": tenant_id}},
|
||||
{"term": {"app_id": app_id}},
|
||||
]
|
||||
}
|
||||
}
|
||||
|
||||
index_pattern = f"{self._index_prefix}-{tenant_id}-*"
|
||||
|
||||
response = self._es_client.search(
|
||||
index=index_pattern,
|
||||
body={
|
||||
"query": query,
|
||||
"size": 1
|
||||
}
|
||||
)
|
||||
|
||||
if response["hits"]["total"]["value"] > 0:
|
||||
hit = response["hits"]["hits"][0]
|
||||
return self._from_es_document(hit)
|
||||
|
||||
return None
|
||||
|
||||
except Exception as e:
|
||||
logger.error("Failed to get workflow run %s: %s", run_id, e)
|
||||
raise
|
||||
|
||||
def get_expired_runs_batch(
|
||||
self,
|
||||
tenant_id: str,
|
||||
before_date: datetime,
|
||||
batch_size: int = 1000,
|
||||
) -> Sequence[WorkflowRun]:
|
||||
"""
|
||||
Get a batch of expired workflow runs for cleanup operations.
|
||||
"""
|
||||
try:
|
||||
query = {
|
||||
"bool": {
|
||||
"must": [
|
||||
{"term": {"tenant_id": tenant_id}},
|
||||
{"range": {"created_at": {"lt": before_date.isoformat()}}},
|
||||
]
|
||||
}
|
||||
}
|
||||
|
||||
index_pattern = f"{self._index_prefix}-{tenant_id}-*"
|
||||
|
||||
response = self._es_client.search(
|
||||
index=index_pattern,
|
||||
body={
|
||||
"query": query,
|
||||
"sort": [{"created_at": {"order": "asc"}}],
|
||||
"size": batch_size
|
||||
}
|
||||
)
|
||||
|
||||
workflow_runs = []
|
||||
for hit in response["hits"]["hits"]:
|
||||
workflow_run = self._from_es_document(hit)
|
||||
workflow_runs.append(workflow_run)
|
||||
|
||||
return workflow_runs
|
||||
|
||||
except Exception as e:
|
||||
logger.error("Failed to get expired runs batch: %s", e)
|
||||
raise
|
||||
|
||||
def delete_runs_by_ids(
|
||||
self,
|
||||
run_ids: Sequence[str],
|
||||
) -> int:
|
||||
"""
|
||||
Delete workflow runs by their IDs using bulk deletion.
|
||||
"""
|
||||
if not run_ids:
|
||||
return 0
|
||||
|
||||
try:
|
||||
query = {
|
||||
"terms": {"id": list(run_ids)}
|
||||
}
|
||||
|
||||
# We need to search across all indices since we don't know the tenant_id
|
||||
# In practice, you might want to pass tenant_id as a parameter
|
||||
index_pattern = f"{self._index_prefix}-*"
|
||||
|
||||
response = self._es_client.delete_by_query(
|
||||
index=index_pattern,
|
||||
body={"query": query},
|
||||
refresh=True
|
||||
)
|
||||
|
||||
deleted_count = response.get("deleted", 0)
|
||||
logger.info("Deleted %s workflow runs by IDs", deleted_count)
|
||||
return deleted_count
|
||||
|
||||
except Exception as e:
|
||||
logger.error("Failed to delete workflow runs by IDs: %s", e)
|
||||
raise
|
||||
|
||||
def delete_runs_by_app(
|
||||
self,
|
||||
tenant_id: str,
|
||||
app_id: str,
|
||||
batch_size: int = 1000,
|
||||
) -> int:
|
||||
"""
|
||||
Delete all workflow runs for a specific app in batches.
|
||||
"""
|
||||
try:
|
||||
query = {
|
||||
"bool": {
|
||||
"must": [
|
||||
{"term": {"tenant_id": tenant_id}},
|
||||
{"term": {"app_id": app_id}},
|
||||
]
|
||||
}
|
||||
}
|
||||
|
||||
index_pattern = f"{self._index_prefix}-{tenant_id}-*"
|
||||
|
||||
response = self._es_client.delete_by_query(
|
||||
index=index_pattern,
|
||||
body={"query": query},
|
||||
refresh=True,
|
||||
wait_for_completion=True
|
||||
)
|
||||
|
||||
deleted_count = response.get("deleted", 0)
|
||||
logger.info("Deleted %s workflow runs for app %s", deleted_count, app_id)
|
||||
return deleted_count
|
||||
|
||||
except Exception as e:
|
||||
logger.error("Failed to delete workflow runs for app %s: %s", app_id, e)
|
||||
raise
|
||||
|
||||
def cleanup_old_indices(self, tenant_id: str, retention_days: int = 30) -> None:
|
||||
"""
|
||||
Clean up old indices based on retention policy.
|
||||
|
||||
Args:
|
||||
tenant_id: Tenant identifier
|
||||
retention_days: Number of days to retain data
|
||||
"""
|
||||
try:
|
||||
cutoff_date = datetime.utcnow() - timedelta(days=retention_days)
|
||||
cutoff_month = cutoff_date.strftime('%Y.%m')
|
||||
|
||||
# Get all indices matching our pattern
|
||||
index_pattern = f"{self._index_prefix}-{tenant_id}-*"
|
||||
indices = self._es_client.indices.get(index=index_pattern)
|
||||
|
||||
indices_to_delete = []
|
||||
for index_name in indices.keys():
|
||||
# Extract date from index name
|
||||
try:
|
||||
date_part = index_name.split('-')[-1] # Get YYYY.MM part
|
||||
if date_part < cutoff_month:
|
||||
indices_to_delete.append(index_name)
|
||||
except (IndexError, ValueError):
|
||||
continue
|
||||
|
||||
if indices_to_delete:
|
||||
self._es_client.indices.delete(index=','.join(indices_to_delete))
|
||||
logger.info("Deleted old indices: %s", indices_to_delete)
|
||||
|
||||
except Exception as e:
|
||||
logger.error("Failed to cleanup old indices: %s", e)
|
||||
raise
|
||||
|
||||
def search_workflow_runs(
|
||||
self,
|
||||
tenant_id: str,
|
||||
app_id: str | None = None,
|
||||
keyword: str | None = None,
|
||||
status: str | None = None,
|
||||
created_at_after: datetime | None = None,
|
||||
created_at_before: datetime | None = None,
|
||||
limit: int = 20,
|
||||
offset: int = 0,
|
||||
) -> dict[str, Any]:
|
||||
"""
|
||||
Advanced search for workflow runs with full-text search capabilities.
|
||||
|
||||
Args:
|
||||
tenant_id: Tenant identifier
|
||||
app_id: Optional app filter
|
||||
keyword: Search keyword for full-text search
|
||||
status: Status filter
|
||||
created_at_after: Filter runs created after this date
|
||||
created_at_before: Filter runs created before this date
|
||||
limit: Maximum number of results
|
||||
offset: Offset for pagination
|
||||
|
||||
Returns:
|
||||
Dictionary with search results and metadata
|
||||
"""
|
||||
try:
|
||||
# Build query
|
||||
must_clauses = [{"term": {"tenant_id": tenant_id}}]
|
||||
|
||||
if app_id:
|
||||
must_clauses.append({"term": {"app_id": app_id}})
|
||||
|
||||
if status:
|
||||
must_clauses.append({"term": {"status": status}})
|
||||
|
||||
# Date range filter
|
||||
if created_at_after or created_at_before:
|
||||
range_query = {}
|
||||
if created_at_after:
|
||||
range_query["gte"] = created_at_after.isoformat()
|
||||
if created_at_before:
|
||||
range_query["lte"] = created_at_before.isoformat()
|
||||
must_clauses.append({"range": {"created_at": range_query}})
|
||||
|
||||
query = {"bool": {"must": must_clauses}}
|
||||
|
||||
# Add full-text search if keyword provided
|
||||
if keyword:
|
||||
query["bool"]["should"] = [
|
||||
{"match": {"inputs": keyword}},
|
||||
{"match": {"outputs": keyword}},
|
||||
{"match": {"error": keyword}},
|
||||
]
|
||||
query["bool"]["minimum_should_match"] = 1
|
||||
|
||||
index_pattern = f"{self._index_prefix}-{tenant_id}-*"
|
||||
|
||||
response = self._es_client.search(
|
||||
index=index_pattern,
|
||||
body={
|
||||
"query": query,
|
||||
"sort": [{"created_at": {"order": "desc"}}],
|
||||
"size": limit,
|
||||
"from": offset
|
||||
}
|
||||
)
|
||||
|
||||
# Convert results
|
||||
workflow_runs = []
|
||||
for hit in response["hits"]["hits"]:
|
||||
workflow_run = self._from_es_document(hit)
|
||||
workflow_runs.append(workflow_run)
|
||||
|
||||
return {
|
||||
"data": workflow_runs,
|
||||
"total": response["hits"]["total"]["value"],
|
||||
"limit": limit,
|
||||
"offset": offset,
|
||||
"has_more": response["hits"]["total"]["value"] > offset + limit
|
||||
}
|
||||
|
||||
except Exception as e:
|
||||
logger.error("Failed to search workflow runs: %s", e)
|
||||
raise
|
||||
@ -1,393 +0,0 @@
|
||||
"""
|
||||
Elasticsearch WorkflowAppLog Repository Implementation
|
||||
|
||||
This module provides Elasticsearch-based storage for WorkflowAppLog entities,
|
||||
offering better performance and scalability for log data management.
|
||||
"""
|
||||
|
||||
import logging
|
||||
from datetime import datetime, timedelta
|
||||
from typing import Any, Optional
|
||||
|
||||
from elasticsearch import Elasticsearch
|
||||
|
||||
from models.workflow import WorkflowAppLog
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class ElasticsearchWorkflowAppLogRepository:
|
||||
"""
|
||||
Elasticsearch implementation for WorkflowAppLog storage and retrieval.
|
||||
|
||||
This repository provides:
|
||||
- High-performance log storage in Elasticsearch
|
||||
- Time-series optimization with date-based index rotation
|
||||
- Multi-tenant data isolation
|
||||
- Advanced search and filtering capabilities
|
||||
"""
|
||||
|
||||
def __init__(self, es_client: Elasticsearch, index_prefix: str = "dify-workflow-app-logs"):
|
||||
"""
|
||||
Initialize the repository with Elasticsearch client.
|
||||
|
||||
Args:
|
||||
es_client: Elasticsearch client instance
|
||||
index_prefix: Prefix for Elasticsearch indices
|
||||
"""
|
||||
self._es_client = es_client
|
||||
self._index_prefix = index_prefix
|
||||
|
||||
# Ensure index template exists
|
||||
self._ensure_index_template()
|
||||
|
||||
def _get_index_name(self, tenant_id: str, date: Optional[datetime] = None) -> str:
|
||||
"""
|
||||
Generate index name with date-based rotation.
|
||||
|
||||
Args:
|
||||
tenant_id: Tenant identifier for multi-tenant isolation
|
||||
date: Date for index name generation, defaults to current date
|
||||
|
||||
Returns:
|
||||
Index name in format: {prefix}-{tenant_id}-{YYYY.MM}
|
||||
"""
|
||||
if date is None:
|
||||
date = datetime.utcnow()
|
||||
|
||||
return f"{self._index_prefix}-{tenant_id}-{date.strftime('%Y.%m')}"
|
||||
|
||||
def _ensure_index_template(self):
|
||||
"""
|
||||
Ensure the index template exists for proper mapping and settings.
|
||||
"""
|
||||
template_name = f"{self._index_prefix}-template"
|
||||
template_body = {
|
||||
"index_patterns": [f"{self._index_prefix}-*"],
|
||||
"template": {
|
||||
"settings": {
|
||||
"number_of_shards": 1,
|
||||
"number_of_replicas": 0,
|
||||
"index.refresh_interval": "5s",
|
||||
},
|
||||
"mappings": {
|
||||
"properties": {
|
||||
"id": {"type": "keyword"},
|
||||
"tenant_id": {"type": "keyword"},
|
||||
"app_id": {"type": "keyword"},
|
||||
"workflow_id": {"type": "keyword"},
|
||||
"workflow_run_id": {"type": "keyword"},
|
||||
"created_from": {"type": "keyword"},
|
||||
"created_by_role": {"type": "keyword"},
|
||||
"created_by": {"type": "keyword"},
|
||||
"created_at": {"type": "date"},
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
try:
|
||||
self._es_client.indices.put_index_template(
|
||||
name=template_name,
|
||||
body=template_body
|
||||
)
|
||||
logger.info("Index template %s created/updated successfully", template_name)
|
||||
except Exception as e:
|
||||
logger.error("Failed to create index template %s: %s", template_name, e)
|
||||
raise
|
||||
|
||||
def _to_es_document(self, app_log: WorkflowAppLog) -> dict[str, Any]:
|
||||
"""
|
||||
Convert WorkflowAppLog model to Elasticsearch document.
|
||||
|
||||
Args:
|
||||
app_log: The WorkflowAppLog model to convert
|
||||
|
||||
Returns:
|
||||
Dictionary representing the Elasticsearch document
|
||||
"""
|
||||
return {
|
||||
"id": app_log.id,
|
||||
"tenant_id": app_log.tenant_id,
|
||||
"app_id": app_log.app_id,
|
||||
"workflow_id": app_log.workflow_id,
|
||||
"workflow_run_id": app_log.workflow_run_id,
|
||||
"created_from": app_log.created_from,
|
||||
"created_by_role": app_log.created_by_role,
|
||||
"created_by": app_log.created_by,
|
||||
"created_at": app_log.created_at.isoformat() if app_log.created_at else None,
|
||||
}
|
||||
|
||||
def _from_es_document(self, doc: dict[str, Any]) -> WorkflowAppLog:
|
||||
"""
|
||||
Convert Elasticsearch document to WorkflowAppLog model.
|
||||
|
||||
Args:
|
||||
doc: Elasticsearch document
|
||||
|
||||
Returns:
|
||||
WorkflowAppLog model instance
|
||||
"""
|
||||
source = doc.get("_source", doc)
|
||||
|
||||
app_log = WorkflowAppLog()
|
||||
app_log.id = source["id"]
|
||||
app_log.tenant_id = source["tenant_id"]
|
||||
app_log.app_id = source["app_id"]
|
||||
app_log.workflow_id = source["workflow_id"]
|
||||
app_log.workflow_run_id = source["workflow_run_id"]
|
||||
app_log.created_from = source["created_from"]
|
||||
app_log.created_by_role = source["created_by_role"]
|
||||
app_log.created_by = source["created_by"]
|
||||
app_log.created_at = datetime.fromisoformat(source["created_at"]) if source.get("created_at") else None
|
||||
|
||||
return app_log
|
||||
|
||||
def save(self, app_log: WorkflowAppLog) -> None:
|
||||
"""
|
||||
Save a WorkflowAppLog to Elasticsearch.
|
||||
|
||||
Args:
|
||||
app_log: The WorkflowAppLog to save
|
||||
"""
|
||||
try:
|
||||
index_name = self._get_index_name(app_log.tenant_id, app_log.created_at)
|
||||
doc = self._to_es_document(app_log)
|
||||
|
||||
self._es_client.index(
|
||||
index=index_name,
|
||||
id=app_log.id,
|
||||
body=doc,
|
||||
refresh="wait_for"
|
||||
)
|
||||
|
||||
logger.debug(f"Saved workflow app log {app_log.id} to index {index_name}")
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to save workflow app log {app_log.id}: {e}")
|
||||
raise
|
||||
|
||||
def get_by_id(self, tenant_id: str, log_id: str) -> Optional[WorkflowAppLog]:
|
||||
"""
|
||||
Get a WorkflowAppLog by ID.
|
||||
|
||||
Args:
|
||||
tenant_id: Tenant identifier
|
||||
log_id: Log ID
|
||||
|
||||
Returns:
|
||||
WorkflowAppLog if found, None otherwise
|
||||
"""
|
||||
try:
|
||||
query = {
|
||||
"bool": {
|
||||
"must": [
|
||||
{"term": {"id": log_id}},
|
||||
{"term": {"tenant_id": tenant_id}},
|
||||
]
|
||||
}
|
||||
}
|
||||
|
||||
index_pattern = f"{self._index_prefix}-{tenant_id}-*"
|
||||
|
||||
response = self._es_client.search(
|
||||
index=index_pattern,
|
||||
body={
|
||||
"query": query,
|
||||
"size": 1
|
||||
}
|
||||
)
|
||||
|
||||
if response["hits"]["total"]["value"] > 0:
|
||||
hit = response["hits"]["hits"][0]
|
||||
return self._from_es_document(hit)
|
||||
|
||||
return None
|
||||
|
||||
except Exception as e:
|
||||
logger.error("Failed to get workflow app log %s: %s", log_id, e)
|
||||
raise
|
||||
|
||||
def get_paginated_logs(
|
||||
self,
|
||||
tenant_id: str,
|
||||
app_id: str,
|
||||
created_at_after: Optional[datetime] = None,
|
||||
created_at_before: Optional[datetime] = None,
|
||||
created_from: Optional[str] = None,
|
||||
limit: int = 20,
|
||||
offset: int = 0,
|
||||
) -> dict[str, Any]:
|
||||
"""
|
||||
Get paginated workflow app logs with filtering.
|
||||
|
||||
Args:
|
||||
tenant_id: Tenant identifier
|
||||
app_id: App identifier
|
||||
created_at_after: Filter logs created after this date
|
||||
created_at_before: Filter logs created before this date
|
||||
created_from: Filter by creation source
|
||||
limit: Maximum number of results
|
||||
offset: Offset for pagination
|
||||
|
||||
Returns:
|
||||
Dictionary with paginated results
|
||||
"""
|
||||
try:
|
||||
# Build query
|
||||
must_clauses = [
|
||||
{"term": {"tenant_id": tenant_id}},
|
||||
{"term": {"app_id": app_id}},
|
||||
]
|
||||
|
||||
if created_from:
|
||||
must_clauses.append({"term": {"created_from": created_from}})
|
||||
|
||||
# Date range filter
|
||||
if created_at_after or created_at_before:
|
||||
range_query = {}
|
||||
if created_at_after:
|
||||
range_query["gte"] = created_at_after.isoformat()
|
||||
if created_at_before:
|
||||
range_query["lte"] = created_at_before.isoformat()
|
||||
must_clauses.append({"range": {"created_at": range_query}})
|
||||
|
||||
query = {"bool": {"must": must_clauses}}
|
||||
|
||||
index_pattern = f"{self._index_prefix}-{tenant_id}-*"
|
||||
|
||||
response = self._es_client.search(
|
||||
index=index_pattern,
|
||||
body={
|
||||
"query": query,
|
||||
"sort": [{"created_at": {"order": "desc"}}],
|
||||
"size": limit,
|
||||
"from": offset
|
||||
}
|
||||
)
|
||||
|
||||
# Convert results
|
||||
app_logs = []
|
||||
for hit in response["hits"]["hits"]:
|
||||
app_log = self._from_es_document(hit)
|
||||
app_logs.append(app_log)
|
||||
|
||||
return {
|
||||
"data": app_logs,
|
||||
"total": response["hits"]["total"]["value"],
|
||||
"limit": limit,
|
||||
"offset": offset,
|
||||
"has_more": response["hits"]["total"]["value"] > offset + limit
|
||||
}
|
||||
|
||||
except Exception as e:
|
||||
logger.error("Failed to get paginated workflow app logs: %s", e)
|
||||
raise
|
||||
|
||||
def delete_by_app(self, tenant_id: str, app_id: str) -> int:
|
||||
"""
|
||||
Delete all workflow app logs for a specific app.
|
||||
|
||||
Args:
|
||||
tenant_id: Tenant identifier
|
||||
app_id: App identifier
|
||||
|
||||
Returns:
|
||||
Number of deleted documents
|
||||
"""
|
||||
try:
|
||||
query = {
|
||||
"bool": {
|
||||
"must": [
|
||||
{"term": {"tenant_id": tenant_id}},
|
||||
{"term": {"app_id": app_id}},
|
||||
]
|
||||
}
|
||||
}
|
||||
|
||||
index_pattern = f"{self._index_prefix}-{tenant_id}-*"
|
||||
|
||||
response = self._es_client.delete_by_query(
|
||||
index=index_pattern,
|
||||
body={"query": query},
|
||||
refresh=True
|
||||
)
|
||||
|
||||
deleted_count = response.get("deleted", 0)
|
||||
logger.info("Deleted %s workflow app logs for app %s", deleted_count, app_id)
|
||||
return deleted_count
|
||||
|
||||
except Exception as e:
|
||||
logger.error("Failed to delete workflow app logs for app %s: %s", app_id, e)
|
||||
raise
|
||||
|
||||
def delete_expired_logs(self, tenant_id: str, before_date: datetime) -> int:
|
||||
"""
|
||||
Delete expired workflow app logs.
|
||||
|
||||
Args:
|
||||
tenant_id: Tenant identifier
|
||||
before_date: Delete logs created before this date
|
||||
|
||||
Returns:
|
||||
Number of deleted documents
|
||||
"""
|
||||
try:
|
||||
query = {
|
||||
"bool": {
|
||||
"must": [
|
||||
{"term": {"tenant_id": tenant_id}},
|
||||
{"range": {"created_at": {"lt": before_date.isoformat()}}},
|
||||
]
|
||||
}
|
||||
}
|
||||
|
||||
index_pattern = f"{self._index_prefix}-{tenant_id}-*"
|
||||
|
||||
response = self._es_client.delete_by_query(
|
||||
index=index_pattern,
|
||||
body={"query": query},
|
||||
refresh=True
|
||||
)
|
||||
|
||||
deleted_count = response.get("deleted", 0)
|
||||
logger.info("Deleted %s expired workflow app logs for tenant %s", deleted_count, tenant_id)
|
||||
return deleted_count
|
||||
|
||||
except Exception as e:
|
||||
logger.error("Failed to delete expired workflow app logs: %s", e)
|
||||
raise
|
||||
|
||||
def cleanup_old_indices(self, tenant_id: str, retention_days: int = 30) -> None:
|
||||
"""
|
||||
Clean up old indices based on retention policy.
|
||||
|
||||
Args:
|
||||
tenant_id: Tenant identifier
|
||||
retention_days: Number of days to retain data
|
||||
"""
|
||||
try:
|
||||
cutoff_date = datetime.utcnow() - timedelta(days=retention_days)
|
||||
cutoff_month = cutoff_date.strftime('%Y.%m')
|
||||
|
||||
# Get all indices matching our pattern
|
||||
index_pattern = f"{self._index_prefix}-{tenant_id}-*"
|
||||
indices = self._es_client.indices.get(index=index_pattern)
|
||||
|
||||
indices_to_delete = []
|
||||
for index_name in indices.keys():
|
||||
# Extract date from index name
|
||||
try:
|
||||
date_part = index_name.split('-')[-1] # Get YYYY.MM part
|
||||
if date_part < cutoff_month:
|
||||
indices_to_delete.append(index_name)
|
||||
except (IndexError, ValueError):
|
||||
continue
|
||||
|
||||
if indices_to_delete:
|
||||
self._es_client.indices.delete(index=','.join(indices_to_delete))
|
||||
logger.info("Deleted old indices: %s", indices_to_delete)
|
||||
|
||||
except Exception as e:
|
||||
logger.error("Failed to cleanup old indices: %s", e)
|
||||
raise
|
||||
@ -2,6 +2,8 @@ import uuid
|
||||
from collections.abc import Generator, Mapping
|
||||
from typing import Any, Union
|
||||
|
||||
from openai._exceptions import RateLimitError
|
||||
|
||||
from configs import dify_config
|
||||
from core.app.apps.advanced_chat.app_generator import AdvancedChatAppGenerator
|
||||
from core.app.apps.agent_chat.app_generator import AgentChatAppGenerator
|
||||
@ -120,6 +122,8 @@ class AppGenerateService:
|
||||
)
|
||||
else:
|
||||
raise ValueError(f"Invalid app mode {app_model.mode}")
|
||||
except RateLimitError as e:
|
||||
raise InvokeRateLimitError(str(e))
|
||||
except Exception:
|
||||
rate_limit.exit(request_id)
|
||||
raise
|
||||
|
||||
@ -1,631 +0,0 @@
|
||||
"""
|
||||
Elasticsearch Migration Service
|
||||
|
||||
This service provides tools for migrating workflow log data from PostgreSQL
|
||||
to Elasticsearch, including data validation, progress tracking, and rollback capabilities.
|
||||
"""
|
||||
|
||||
import json
|
||||
import logging
|
||||
from datetime import datetime
|
||||
from typing import Any, Optional
|
||||
|
||||
from elasticsearch import Elasticsearch
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.orm import sessionmaker
|
||||
|
||||
from extensions.ext_database import db
|
||||
from extensions.ext_elasticsearch import elasticsearch
|
||||
from models.workflow import (
|
||||
WorkflowAppLog,
|
||||
WorkflowNodeExecutionModel,
|
||||
WorkflowNodeExecutionOffload,
|
||||
WorkflowRun,
|
||||
)
|
||||
from repositories.elasticsearch_api_workflow_run_repository import ElasticsearchAPIWorkflowRunRepository
|
||||
from repositories.elasticsearch_workflow_app_log_repository import ElasticsearchWorkflowAppLogRepository
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class ElasticsearchMigrationService:
|
||||
"""
|
||||
Service for migrating workflow log data from PostgreSQL to Elasticsearch.
|
||||
|
||||
Provides comprehensive migration capabilities including:
|
||||
- Batch processing for large datasets
|
||||
- Progress tracking and resumption
|
||||
- Data validation and integrity checks
|
||||
- Rollback capabilities
|
||||
- Performance monitoring
|
||||
"""
|
||||
|
||||
def __init__(self, es_client: Optional[Elasticsearch] = None, batch_size: int = 1000):
|
||||
"""
|
||||
Initialize the migration service.
|
||||
|
||||
Args:
|
||||
es_client: Elasticsearch client instance (uses global client if None)
|
||||
batch_size: Number of records to process in each batch
|
||||
"""
|
||||
self._es_client = es_client or elasticsearch.client
|
||||
if not self._es_client:
|
||||
raise ValueError("Elasticsearch client is not available")
|
||||
|
||||
self._batch_size = batch_size
|
||||
self._session_maker = sessionmaker(bind=db.engine, expire_on_commit=False)
|
||||
|
||||
# Initialize repositories
|
||||
self._workflow_run_repo = ElasticsearchAPIWorkflowRunRepository(self._es_client)
|
||||
self._app_log_repo = ElasticsearchWorkflowAppLogRepository(self._es_client)
|
||||
|
||||
def migrate_workflow_runs(
|
||||
self,
|
||||
tenant_id: Optional[str] = None,
|
||||
start_date: Optional[datetime] = None,
|
||||
end_date: Optional[datetime] = None,
|
||||
dry_run: bool = False,
|
||||
) -> dict[str, Any]:
|
||||
"""
|
||||
Migrate WorkflowRun data from PostgreSQL to Elasticsearch.
|
||||
|
||||
Args:
|
||||
tenant_id: Optional tenant filter for migration
|
||||
start_date: Optional start date filter
|
||||
end_date: Optional end date filter
|
||||
dry_run: If True, only count records without migrating
|
||||
|
||||
Returns:
|
||||
Migration statistics and results
|
||||
"""
|
||||
logger.info("Starting WorkflowRun migration to Elasticsearch")
|
||||
|
||||
stats = {
|
||||
"total_records": 0,
|
||||
"migrated_records": 0,
|
||||
"failed_records": 0,
|
||||
"start_time": datetime.utcnow(),
|
||||
"errors": [],
|
||||
}
|
||||
|
||||
try:
|
||||
with self._session_maker() as session:
|
||||
# Build query
|
||||
query = select(WorkflowRun)
|
||||
|
||||
if tenant_id:
|
||||
query = query.where(WorkflowRun.tenant_id == tenant_id)
|
||||
|
||||
if start_date:
|
||||
query = query.where(WorkflowRun.created_at >= start_date)
|
||||
|
||||
if end_date:
|
||||
query = query.where(WorkflowRun.created_at <= end_date)
|
||||
|
||||
# Get total count
|
||||
count_query = select(db.func.count()).select_from(query.subquery())
|
||||
stats["total_records"] = session.scalar(count_query) or 0
|
||||
|
||||
if dry_run:
|
||||
logger.info(f"Dry run: Found {stats['total_records']} WorkflowRun records to migrate")
|
||||
return stats
|
||||
|
||||
# Process in batches
|
||||
offset = 0
|
||||
while offset < stats["total_records"]:
|
||||
batch_query = query.offset(offset).limit(self._batch_size)
|
||||
workflow_runs = session.scalars(batch_query).all()
|
||||
|
||||
if not workflow_runs:
|
||||
break
|
||||
|
||||
# Migrate batch
|
||||
for workflow_run in workflow_runs:
|
||||
try:
|
||||
self._workflow_run_repo.save(workflow_run)
|
||||
stats["migrated_records"] += 1
|
||||
|
||||
if stats["migrated_records"] % 100 == 0:
|
||||
logger.info(f"Migrated {stats['migrated_records']}/{stats['total_records']} WorkflowRuns")
|
||||
|
||||
except Exception as e:
|
||||
error_msg = f"Failed to migrate WorkflowRun {workflow_run.id}: {str(e)}"
|
||||
logger.error(error_msg)
|
||||
stats["errors"].append(error_msg)
|
||||
stats["failed_records"] += 1
|
||||
|
||||
offset += self._batch_size
|
||||
|
||||
except Exception as e:
|
||||
error_msg = f"Migration failed: {str(e)}"
|
||||
logger.error(error_msg)
|
||||
stats["errors"].append(error_msg)
|
||||
raise
|
||||
|
||||
stats["end_time"] = datetime.utcnow()
|
||||
stats["duration"] = (stats["end_time"] - stats["start_time"]).total_seconds()
|
||||
|
||||
logger.info(f"WorkflowRun migration completed: {stats['migrated_records']} migrated, "
|
||||
f"{stats['failed_records']} failed in {stats['duration']:.2f}s")
|
||||
|
||||
return stats
|
||||
|
||||
def migrate_workflow_app_logs(
|
||||
self,
|
||||
tenant_id: Optional[str] = None,
|
||||
start_date: Optional[datetime] = None,
|
||||
end_date: Optional[datetime] = None,
|
||||
dry_run: bool = False,
|
||||
) -> dict[str, Any]:
|
||||
"""
|
||||
Migrate WorkflowAppLog data from PostgreSQL to Elasticsearch.
|
||||
|
||||
Args:
|
||||
tenant_id: Optional tenant filter for migration
|
||||
start_date: Optional start date filter
|
||||
end_date: Optional end date filter
|
||||
dry_run: If True, only count records without migrating
|
||||
|
||||
Returns:
|
||||
Migration statistics and results
|
||||
"""
|
||||
logger.info("Starting WorkflowAppLog migration to Elasticsearch")
|
||||
|
||||
stats = {
|
||||
"total_records": 0,
|
||||
"migrated_records": 0,
|
||||
"failed_records": 0,
|
||||
"start_time": datetime.utcnow(),
|
||||
"errors": [],
|
||||
}
|
||||
|
||||
try:
|
||||
with self._session_maker() as session:
|
||||
# Build query
|
||||
query = select(WorkflowAppLog)
|
||||
|
||||
if tenant_id:
|
||||
query = query.where(WorkflowAppLog.tenant_id == tenant_id)
|
||||
|
||||
if start_date:
|
||||
query = query.where(WorkflowAppLog.created_at >= start_date)
|
||||
|
||||
if end_date:
|
||||
query = query.where(WorkflowAppLog.created_at <= end_date)
|
||||
|
||||
# Get total count
|
||||
count_query = select(db.func.count()).select_from(query.subquery())
|
||||
stats["total_records"] = session.scalar(count_query) or 0
|
||||
|
||||
if dry_run:
|
||||
logger.info(f"Dry run: Found {stats['total_records']} WorkflowAppLog records to migrate")
|
||||
return stats
|
||||
|
||||
# Process in batches
|
||||
offset = 0
|
||||
while offset < stats["total_records"]:
|
||||
batch_query = query.offset(offset).limit(self._batch_size)
|
||||
app_logs = session.scalars(batch_query).all()
|
||||
|
||||
if not app_logs:
|
||||
break
|
||||
|
||||
# Migrate batch
|
||||
for app_log in app_logs:
|
||||
try:
|
||||
self._app_log_repo.save(app_log)
|
||||
stats["migrated_records"] += 1
|
||||
|
||||
if stats["migrated_records"] % 100 == 0:
|
||||
logger.info(f"Migrated {stats['migrated_records']}/{stats['total_records']} WorkflowAppLogs")
|
||||
|
||||
except Exception as e:
|
||||
error_msg = f"Failed to migrate WorkflowAppLog {app_log.id}: {str(e)}"
|
||||
logger.error(error_msg)
|
||||
stats["errors"].append(error_msg)
|
||||
stats["failed_records"] += 1
|
||||
|
||||
offset += self._batch_size
|
||||
|
||||
except Exception as e:
|
||||
error_msg = f"Migration failed: {str(e)}"
|
||||
logger.error(error_msg)
|
||||
stats["errors"].append(error_msg)
|
||||
raise
|
||||
|
||||
stats["end_time"] = datetime.utcnow()
|
||||
stats["duration"] = (stats["end_time"] - stats["start_time"]).total_seconds()
|
||||
|
||||
logger.info(f"WorkflowAppLog migration completed: {stats['migrated_records']} migrated, "
|
||||
f"{stats['failed_records']} failed in {stats['duration']:.2f}s")
|
||||
|
||||
return stats
|
||||
|
||||
def migrate_workflow_node_executions(
|
||||
self,
|
||||
tenant_id: Optional[str] = None,
|
||||
start_date: Optional[datetime] = None,
|
||||
end_date: Optional[datetime] = None,
|
||||
dry_run: bool = False,
|
||||
) -> dict[str, Any]:
|
||||
"""
|
||||
Migrate WorkflowNodeExecution data from PostgreSQL to Elasticsearch.
|
||||
|
||||
Note: This requires the Elasticsearch WorkflowNodeExecution repository
|
||||
to be properly configured and initialized.
|
||||
|
||||
Args:
|
||||
tenant_id: Optional tenant filter for migration
|
||||
start_date: Optional start date filter
|
||||
end_date: Optional end date filter
|
||||
dry_run: If True, only count records without migrating
|
||||
|
||||
Returns:
|
||||
Migration statistics and results
|
||||
"""
|
||||
logger.info("Starting WorkflowNodeExecution migration to Elasticsearch")
|
||||
|
||||
stats = {
|
||||
"total_records": 0,
|
||||
"migrated_records": 0,
|
||||
"failed_records": 0,
|
||||
"start_time": datetime.utcnow(),
|
||||
"errors": [],
|
||||
}
|
||||
|
||||
try:
|
||||
with self._session_maker() as session:
|
||||
# Build query with offload data preloaded
|
||||
query = WorkflowNodeExecutionModel.preload_offload_data_and_files(
|
||||
select(WorkflowNodeExecutionModel)
|
||||
)
|
||||
|
||||
if tenant_id:
|
||||
query = query.where(WorkflowNodeExecutionModel.tenant_id == tenant_id)
|
||||
|
||||
if start_date:
|
||||
query = query.where(WorkflowNodeExecutionModel.created_at >= start_date)
|
||||
|
||||
if end_date:
|
||||
query = query.where(WorkflowNodeExecutionModel.created_at <= end_date)
|
||||
|
||||
# Get total count
|
||||
count_query = select(db.func.count()).select_from(
|
||||
select(WorkflowNodeExecutionModel).where(
|
||||
*([WorkflowNodeExecutionModel.tenant_id == tenant_id] if tenant_id else []),
|
||||
*([WorkflowNodeExecutionModel.created_at >= start_date] if start_date else []),
|
||||
*([WorkflowNodeExecutionModel.created_at <= end_date] if end_date else []),
|
||||
).subquery()
|
||||
)
|
||||
stats["total_records"] = session.scalar(count_query) or 0
|
||||
|
||||
if dry_run:
|
||||
logger.info(f"Dry run: Found {stats['total_records']} WorkflowNodeExecution records to migrate")
|
||||
return stats
|
||||
|
||||
# Process in batches
|
||||
offset = 0
|
||||
while offset < stats["total_records"]:
|
||||
batch_query = query.offset(offset).limit(self._batch_size)
|
||||
node_executions = session.scalars(batch_query).all()
|
||||
|
||||
if not node_executions:
|
||||
break
|
||||
|
||||
# Migrate batch
|
||||
for node_execution in node_executions:
|
||||
try:
|
||||
# Convert to Elasticsearch document format
|
||||
doc = self._convert_node_execution_to_es_doc(node_execution)
|
||||
|
||||
# Save to Elasticsearch
|
||||
index_name = f"dify-workflow-node-executions-{tenant_id or node_execution.tenant_id}-{node_execution.created_at.strftime('%Y.%m')}"
|
||||
self._es_client.index(
|
||||
index=index_name,
|
||||
id=node_execution.id,
|
||||
body=doc,
|
||||
refresh="wait_for"
|
||||
)
|
||||
|
||||
stats["migrated_records"] += 1
|
||||
|
||||
if stats["migrated_records"] % 100 == 0:
|
||||
logger.info(f"Migrated {stats['migrated_records']}/{stats['total_records']} WorkflowNodeExecutions")
|
||||
|
||||
except Exception as e:
|
||||
error_msg = f"Failed to migrate WorkflowNodeExecution {node_execution.id}: {str(e)}"
|
||||
logger.error(error_msg)
|
||||
stats["errors"].append(error_msg)
|
||||
stats["failed_records"] += 1
|
||||
|
||||
offset += self._batch_size
|
||||
|
||||
except Exception as e:
|
||||
error_msg = f"Migration failed: {str(e)}"
|
||||
logger.error(error_msg)
|
||||
stats["errors"].append(error_msg)
|
||||
raise
|
||||
|
||||
stats["end_time"] = datetime.utcnow()
|
||||
stats["duration"] = (stats["end_time"] - stats["start_time"]).total_seconds()
|
||||
|
||||
logger.info(f"WorkflowNodeExecution migration completed: {stats['migrated_records']} migrated, "
|
||||
f"{stats['failed_records']} failed in {stats['duration']:.2f}s")
|
||||
|
||||
return stats
|
||||
|
||||
def _convert_node_execution_to_es_doc(self, node_execution: WorkflowNodeExecutionModel) -> dict[str, Any]:
|
||||
"""
|
||||
Convert WorkflowNodeExecutionModel to Elasticsearch document format.
|
||||
|
||||
Args:
|
||||
node_execution: The database model to convert
|
||||
|
||||
Returns:
|
||||
Dictionary representing the Elasticsearch document
|
||||
"""
|
||||
# Load full data if offloaded
|
||||
inputs = node_execution.inputs_dict
|
||||
outputs = node_execution.outputs_dict
|
||||
process_data = node_execution.process_data_dict
|
||||
|
||||
# If data is offloaded, load from storage
|
||||
if node_execution.offload_data:
|
||||
from extensions.ext_storage import storage
|
||||
|
||||
for offload in node_execution.offload_data:
|
||||
if offload.file:
|
||||
content = storage.load(offload.file.key)
|
||||
data = json.loads(content)
|
||||
|
||||
if offload.type_.value == "inputs":
|
||||
inputs = data
|
||||
elif offload.type_.value == "outputs":
|
||||
outputs = data
|
||||
elif offload.type_.value == "process_data":
|
||||
process_data = data
|
||||
|
||||
doc = {
|
||||
"id": node_execution.id,
|
||||
"tenant_id": node_execution.tenant_id,
|
||||
"app_id": node_execution.app_id,
|
||||
"workflow_id": node_execution.workflow_id,
|
||||
"workflow_execution_id": node_execution.workflow_run_id,
|
||||
"node_execution_id": node_execution.node_execution_id,
|
||||
"triggered_from": node_execution.triggered_from,
|
||||
"index": node_execution.index,
|
||||
"predecessor_node_id": node_execution.predecessor_node_id,
|
||||
"node_id": node_execution.node_id,
|
||||
"node_type": node_execution.node_type,
|
||||
"title": node_execution.title,
|
||||
"inputs": inputs,
|
||||
"process_data": process_data,
|
||||
"outputs": outputs,
|
||||
"status": node_execution.status,
|
||||
"error": node_execution.error,
|
||||
"elapsed_time": node_execution.elapsed_time,
|
||||
"metadata": node_execution.execution_metadata_dict,
|
||||
"created_at": node_execution.created_at.isoformat() if node_execution.created_at else None,
|
||||
"finished_at": node_execution.finished_at.isoformat() if node_execution.finished_at else None,
|
||||
"created_by_role": node_execution.created_by_role,
|
||||
"created_by": node_execution.created_by,
|
||||
}
|
||||
|
||||
# Remove None values to reduce storage size
|
||||
return {k: v for k, v in doc.items() if v is not None}
|
||||
|
||||
def validate_migration(self, tenant_id: str, sample_size: int = 100) -> dict[str, Any]:
|
||||
"""
|
||||
Validate migrated data by comparing samples from PostgreSQL and Elasticsearch.
|
||||
|
||||
Args:
|
||||
tenant_id: Tenant ID to validate
|
||||
sample_size: Number of records to sample for validation
|
||||
|
||||
Returns:
|
||||
Validation results and statistics
|
||||
"""
|
||||
logger.info("Starting migration validation for tenant %s", tenant_id)
|
||||
|
||||
validation_results = {
|
||||
"workflow_runs": {"total": 0, "matched": 0, "mismatched": 0, "missing": 0},
|
||||
"app_logs": {"total": 0, "matched": 0, "mismatched": 0, "missing": 0},
|
||||
"node_executions": {"total": 0, "matched": 0, "mismatched": 0, "missing": 0},
|
||||
"errors": [],
|
||||
}
|
||||
|
||||
try:
|
||||
with self._session_maker() as session:
|
||||
# Validate WorkflowRuns
|
||||
workflow_runs = session.scalars(
|
||||
select(WorkflowRun)
|
||||
.where(WorkflowRun.tenant_id == tenant_id)
|
||||
.limit(sample_size)
|
||||
).all()
|
||||
|
||||
validation_results["workflow_runs"]["total"] = len(workflow_runs)
|
||||
|
||||
for workflow_run in workflow_runs:
|
||||
try:
|
||||
es_run = self._workflow_run_repo.get_workflow_run_by_id(
|
||||
tenant_id, workflow_run.app_id, workflow_run.id
|
||||
)
|
||||
|
||||
if es_run:
|
||||
if self._compare_workflow_runs(workflow_run, es_run):
|
||||
validation_results["workflow_runs"]["matched"] += 1
|
||||
else:
|
||||
validation_results["workflow_runs"]["mismatched"] += 1
|
||||
else:
|
||||
validation_results["workflow_runs"]["missing"] += 1
|
||||
|
||||
except Exception as e:
|
||||
validation_results["errors"].append(f"Error validating WorkflowRun {workflow_run.id}: {str(e)}")
|
||||
|
||||
# Validate WorkflowAppLogs
|
||||
app_logs = session.scalars(
|
||||
select(WorkflowAppLog)
|
||||
.where(WorkflowAppLog.tenant_id == tenant_id)
|
||||
.limit(sample_size)
|
||||
).all()
|
||||
|
||||
validation_results["app_logs"]["total"] = len(app_logs)
|
||||
|
||||
for app_log in app_logs:
|
||||
try:
|
||||
es_log = self._app_log_repo.get_by_id(tenant_id, app_log.id)
|
||||
|
||||
if es_log:
|
||||
if self._compare_app_logs(app_log, es_log):
|
||||
validation_results["app_logs"]["matched"] += 1
|
||||
else:
|
||||
validation_results["app_logs"]["mismatched"] += 1
|
||||
else:
|
||||
validation_results["app_logs"]["missing"] += 1
|
||||
|
||||
except Exception as e:
|
||||
validation_results["errors"].append(f"Error validating WorkflowAppLog {app_log.id}: {str(e)}")
|
||||
|
||||
except Exception as e:
|
||||
error_msg = f"Validation failed: {str(e)}"
|
||||
logger.error(error_msg)
|
||||
validation_results["errors"].append(error_msg)
|
||||
|
||||
logger.info("Migration validation completed for tenant %s", tenant_id)
|
||||
return validation_results
|
||||
|
||||
def _compare_workflow_runs(self, pg_run: WorkflowRun, es_run: WorkflowRun) -> bool:
|
||||
"""Compare WorkflowRun records from PostgreSQL and Elasticsearch."""
|
||||
return (
|
||||
pg_run.id == es_run.id
|
||||
and pg_run.status == es_run.status
|
||||
and pg_run.elapsed_time == es_run.elapsed_time
|
||||
and pg_run.total_tokens == es_run.total_tokens
|
||||
)
|
||||
|
||||
def _compare_app_logs(self, pg_log: WorkflowAppLog, es_log: WorkflowAppLog) -> bool:
|
||||
"""Compare WorkflowAppLog records from PostgreSQL and Elasticsearch."""
|
||||
return (
|
||||
pg_log.id == es_log.id
|
||||
and pg_log.workflow_run_id == es_log.workflow_run_id
|
||||
and pg_log.created_from == es_log.created_from
|
||||
)
|
||||
|
||||
def cleanup_old_pg_data(
|
||||
self,
|
||||
tenant_id: str,
|
||||
before_date: datetime,
|
||||
dry_run: bool = True,
|
||||
) -> dict[str, Any]:
|
||||
"""
|
||||
Clean up old PostgreSQL data after successful migration to Elasticsearch.
|
||||
|
||||
Args:
|
||||
tenant_id: Tenant ID to clean up
|
||||
before_date: Delete records created before this date
|
||||
dry_run: If True, only count records without deleting
|
||||
|
||||
Returns:
|
||||
Cleanup statistics
|
||||
"""
|
||||
logger.info("Starting PostgreSQL data cleanup for tenant %s", tenant_id)
|
||||
|
||||
stats = {
|
||||
"workflow_runs_deleted": 0,
|
||||
"app_logs_deleted": 0,
|
||||
"node_executions_deleted": 0,
|
||||
"offload_records_deleted": 0,
|
||||
"start_time": datetime.utcnow(),
|
||||
}
|
||||
|
||||
try:
|
||||
with self._session_maker() as session:
|
||||
if not dry_run:
|
||||
# Delete WorkflowNodeExecutionOffload records
|
||||
offload_count = session.query(WorkflowNodeExecutionOffload).filter(
|
||||
WorkflowNodeExecutionOffload.tenant_id == tenant_id,
|
||||
WorkflowNodeExecutionOffload.created_at < before_date,
|
||||
).count()
|
||||
|
||||
session.query(WorkflowNodeExecutionOffload).filter(
|
||||
WorkflowNodeExecutionOffload.tenant_id == tenant_id,
|
||||
WorkflowNodeExecutionOffload.created_at < before_date,
|
||||
).delete()
|
||||
|
||||
stats["offload_records_deleted"] = offload_count
|
||||
|
||||
# Delete WorkflowNodeExecution records
|
||||
node_exec_count = session.query(WorkflowNodeExecutionModel).filter(
|
||||
WorkflowNodeExecutionModel.tenant_id == tenant_id,
|
||||
WorkflowNodeExecutionModel.created_at < before_date,
|
||||
).count()
|
||||
|
||||
session.query(WorkflowNodeExecutionModel).filter(
|
||||
WorkflowNodeExecutionModel.tenant_id == tenant_id,
|
||||
WorkflowNodeExecutionModel.created_at < before_date,
|
||||
).delete()
|
||||
|
||||
stats["node_executions_deleted"] = node_exec_count
|
||||
|
||||
# Delete WorkflowAppLog records
|
||||
app_log_count = session.query(WorkflowAppLog).filter(
|
||||
WorkflowAppLog.tenant_id == tenant_id,
|
||||
WorkflowAppLog.created_at < before_date,
|
||||
).count()
|
||||
|
||||
session.query(WorkflowAppLog).filter(
|
||||
WorkflowAppLog.tenant_id == tenant_id,
|
||||
WorkflowAppLog.created_at < before_date,
|
||||
).delete()
|
||||
|
||||
stats["app_logs_deleted"] = app_log_count
|
||||
|
||||
# Delete WorkflowRun records
|
||||
workflow_run_count = session.query(WorkflowRun).filter(
|
||||
WorkflowRun.tenant_id == tenant_id,
|
||||
WorkflowRun.created_at < before_date,
|
||||
).count()
|
||||
|
||||
session.query(WorkflowRun).filter(
|
||||
WorkflowRun.tenant_id == tenant_id,
|
||||
WorkflowRun.created_at < before_date,
|
||||
).delete()
|
||||
|
||||
stats["workflow_runs_deleted"] = workflow_run_count
|
||||
|
||||
session.commit()
|
||||
else:
|
||||
# Dry run - just count records
|
||||
stats["workflow_runs_deleted"] = session.query(WorkflowRun).filter(
|
||||
WorkflowRun.tenant_id == tenant_id,
|
||||
WorkflowRun.created_at < before_date,
|
||||
).count()
|
||||
|
||||
stats["app_logs_deleted"] = session.query(WorkflowAppLog).filter(
|
||||
WorkflowAppLog.tenant_id == tenant_id,
|
||||
WorkflowAppLog.created_at < before_date,
|
||||
).count()
|
||||
|
||||
stats["node_executions_deleted"] = session.query(WorkflowNodeExecutionModel).filter(
|
||||
WorkflowNodeExecutionModel.tenant_id == tenant_id,
|
||||
WorkflowNodeExecutionModel.created_at < before_date,
|
||||
).count()
|
||||
|
||||
stats["offload_records_deleted"] = session.query(WorkflowNodeExecutionOffload).filter(
|
||||
WorkflowNodeExecutionOffload.tenant_id == tenant_id,
|
||||
WorkflowNodeExecutionOffload.created_at < before_date,
|
||||
).count()
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Cleanup failed: {str(e)}")
|
||||
raise
|
||||
|
||||
stats["end_time"] = datetime.utcnow()
|
||||
stats["duration"] = (stats["end_time"] - stats["start_time"]).total_seconds()
|
||||
|
||||
action = "Would delete" if dry_run else "Deleted"
|
||||
logger.info(f"PostgreSQL cleanup completed: {action} {stats['workflow_runs_deleted']} WorkflowRuns, "
|
||||
f"{stats['app_logs_deleted']} AppLogs, {stats['node_executions_deleted']} NodeExecutions, "
|
||||
f"{stats['offload_records_deleted']} OffloadRecords in {stats['duration']:.2f}s")
|
||||
|
||||
return stats
|
||||
@ -149,7 +149,8 @@ class RagPipelineTransformService:
|
||||
file_extensions = node.get("data", {}).get("fileExtensions", [])
|
||||
if not file_extensions:
|
||||
return node
|
||||
node["data"]["fileExtensions"] = [ext.lower() for ext in file_extensions if ext in DOCUMENT_EXTENSIONS]
|
||||
file_extensions = [file_extension.lower() for file_extension in file_extensions]
|
||||
node["data"]["fileExtensions"] = DOCUMENT_EXTENSIONS
|
||||
return node
|
||||
|
||||
def _deal_knowledge_index(
|
||||
|
||||
@ -349,10 +349,14 @@ class BuiltinToolManageService:
|
||||
provider_controller = ToolManager.get_builtin_provider(default_provider.provider, tenant_id)
|
||||
|
||||
credentials: list[ToolProviderCredentialApiEntity] = []
|
||||
encrypters = {}
|
||||
for provider in providers:
|
||||
encrypter, _ = BuiltinToolManageService.create_tool_encrypter(
|
||||
tenant_id, provider, provider.provider, provider_controller
|
||||
)
|
||||
credential_type = provider.credential_type
|
||||
if credential_type not in encrypters:
|
||||
encrypters[credential_type] = BuiltinToolManageService.create_tool_encrypter(
|
||||
tenant_id, provider, provider.provider, provider_controller
|
||||
)[0]
|
||||
encrypter = encrypters[credential_type]
|
||||
decrypt_credential = encrypter.mask_tool_credentials(encrypter.decrypt(provider.credentials))
|
||||
credential_entity = ToolTransformService.convert_builtin_provider_to_credential_entity(
|
||||
provider=provider,
|
||||
|
||||
@ -79,6 +79,7 @@ class WorkflowConverter:
|
||||
new_app.updated_by = account.id
|
||||
db.session.add(new_app)
|
||||
db.session.flush()
|
||||
db.session.commit()
|
||||
|
||||
workflow.app_id = new_app.id
|
||||
db.session.commit()
|
||||
|
||||
@ -29,10 +29,23 @@ def priority_rag_pipeline_run_task(
|
||||
tenant_id: str,
|
||||
):
|
||||
"""
|
||||
Async Run rag pipeline task using high priority queue.
|
||||
|
||||
:param rag_pipeline_invoke_entities_file_id: File ID containing serialized RAG pipeline invoke entities
|
||||
:param tenant_id: Tenant ID for the pipeline execution
|
||||
Async Run rag pipeline
|
||||
:param rag_pipeline_invoke_entities: Rag pipeline invoke entities
|
||||
rag_pipeline_invoke_entities include:
|
||||
:param pipeline_id: Pipeline ID
|
||||
:param user_id: User ID
|
||||
:param tenant_id: Tenant ID
|
||||
:param workflow_id: Workflow ID
|
||||
:param invoke_from: Invoke source (debugger, published, etc.)
|
||||
:param streaming: Whether to stream results
|
||||
:param datasource_type: Type of datasource
|
||||
:param datasource_info: Datasource information dict
|
||||
:param batch: Batch identifier
|
||||
:param document_id: Document ID (optional)
|
||||
:param start_node_id: Starting node ID
|
||||
:param inputs: Input parameters dict
|
||||
:param workflow_execution_id: Workflow execution ID
|
||||
:param workflow_thread_pool_id: Thread pool ID for workflow execution
|
||||
"""
|
||||
# run with threading, thread pool size is 10
|
||||
|
||||
|
||||
@ -30,10 +30,23 @@ def rag_pipeline_run_task(
|
||||
tenant_id: str,
|
||||
):
|
||||
"""
|
||||
Async Run rag pipeline task using regular priority queue.
|
||||
|
||||
:param rag_pipeline_invoke_entities_file_id: File ID containing serialized RAG pipeline invoke entities
|
||||
:param tenant_id: Tenant ID for the pipeline execution
|
||||
Async Run rag pipeline
|
||||
:param rag_pipeline_invoke_entities: Rag pipeline invoke entities
|
||||
rag_pipeline_invoke_entities include:
|
||||
:param pipeline_id: Pipeline ID
|
||||
:param user_id: User ID
|
||||
:param tenant_id: Tenant ID
|
||||
:param workflow_id: Workflow ID
|
||||
:param invoke_from: Invoke source (debugger, published, etc.)
|
||||
:param streaming: Whether to stream results
|
||||
:param datasource_type: Type of datasource
|
||||
:param datasource_info: Datasource information dict
|
||||
:param batch: Batch identifier
|
||||
:param document_id: Document ID (optional)
|
||||
:param start_node_id: Starting node ID
|
||||
:param inputs: Input parameters dict
|
||||
:param workflow_execution_id: Workflow execution ID
|
||||
:param workflow_thread_pool_id: Thread pool ID for workflow execution
|
||||
"""
|
||||
# run with threading, thread pool size is 10
|
||||
|
||||
|
||||
@ -5,10 +5,15 @@ These tasks provide asynchronous storage capabilities for workflow execution dat
|
||||
improving performance by offloading storage operations to background workers.
|
||||
"""
|
||||
|
||||
import logging
|
||||
|
||||
from celery import shared_task # type: ignore[import-untyped]
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from extensions.ext_database import db
|
||||
|
||||
_logger = logging.getLogger(__name__)
|
||||
|
||||
from services.workflow_draft_variable_service import DraftVarFileDeletion, WorkflowDraftVariableService
|
||||
|
||||
|
||||
|
||||
@ -11,8 +11,8 @@ from controllers.console.app import completion as completion_api
|
||||
from controllers.console.app import message as message_api
|
||||
from controllers.console.app import wraps
|
||||
from libs.datetime_utils import naive_utc_now
|
||||
from models import App, Tenant
|
||||
from models.account import Account, TenantAccountJoin, TenantAccountRole
|
||||
from models import Account, App, Tenant
|
||||
from models.account import TenantAccountRole
|
||||
from models.model import AppMode
|
||||
from services.app_generate_service import AppGenerateService
|
||||
|
||||
@ -31,8 +31,9 @@ class TestChatMessageApiPermissions:
|
||||
return app
|
||||
|
||||
@pytest.fixture
|
||||
def mock_account(self, monkeypatch: pytest.MonkeyPatch):
|
||||
def mock_account(self):
|
||||
"""Create a mock Account for testing."""
|
||||
|
||||
account = Account()
|
||||
account.id = str(uuid.uuid4())
|
||||
account.name = "Test User"
|
||||
@ -41,24 +42,12 @@ class TestChatMessageApiPermissions:
|
||||
account.created_at = naive_utc_now()
|
||||
account.updated_at = naive_utc_now()
|
||||
|
||||
# Create mock tenant
|
||||
tenant = Tenant()
|
||||
tenant.id = str(uuid.uuid4())
|
||||
tenant.name = "Test Tenant"
|
||||
|
||||
mock_session_instance = mock.Mock()
|
||||
|
||||
mock_tenant_join = TenantAccountJoin(role=TenantAccountRole.OWNER)
|
||||
monkeypatch.setattr(mock_session_instance, "scalar", mock.Mock(return_value=mock_tenant_join))
|
||||
|
||||
mock_scalars_result = mock.Mock()
|
||||
mock_scalars_result.one.return_value = tenant
|
||||
monkeypatch.setattr(mock_session_instance, "scalars", mock.Mock(return_value=mock_scalars_result))
|
||||
|
||||
mock_session_context = mock.Mock()
|
||||
mock_session_context.__enter__.return_value = mock_session_instance
|
||||
monkeypatch.setattr("models.account.Session", lambda _, expire_on_commit: mock_session_context)
|
||||
|
||||
account.current_tenant = tenant
|
||||
account._current_tenant = tenant
|
||||
return account
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
|
||||
@ -18,87 +18,124 @@ class TestAppDescriptionValidationUnit:
|
||||
"""Unit tests for description validation function"""
|
||||
|
||||
def test_validate_description_length_function(self):
|
||||
"""Test the validate_description_length function directly"""
|
||||
from libs.validators import validate_description_length
|
||||
"""Test the _validate_description_length function directly"""
|
||||
from controllers.console.app.app import _validate_description_length
|
||||
|
||||
# Test valid descriptions
|
||||
assert validate_description_length("") == ""
|
||||
assert validate_description_length("x" * 400) == "x" * 400
|
||||
assert validate_description_length(None) is None
|
||||
assert _validate_description_length("") == ""
|
||||
assert _validate_description_length("x" * 400) == "x" * 400
|
||||
assert _validate_description_length(None) is None
|
||||
|
||||
# Test invalid descriptions
|
||||
with pytest.raises(ValueError) as exc_info:
|
||||
validate_description_length("x" * 401)
|
||||
_validate_description_length("x" * 401)
|
||||
assert "Description cannot exceed 400 characters." in str(exc_info.value)
|
||||
|
||||
with pytest.raises(ValueError) as exc_info:
|
||||
validate_description_length("x" * 500)
|
||||
_validate_description_length("x" * 500)
|
||||
assert "Description cannot exceed 400 characters." in str(exc_info.value)
|
||||
|
||||
with pytest.raises(ValueError) as exc_info:
|
||||
validate_description_length("x" * 1000)
|
||||
_validate_description_length("x" * 1000)
|
||||
assert "Description cannot exceed 400 characters." in str(exc_info.value)
|
||||
|
||||
def test_validation_consistency_with_dataset(self):
|
||||
"""Test that App and Dataset validation functions are consistent"""
|
||||
from controllers.console.app.app import _validate_description_length as app_validate
|
||||
from controllers.console.datasets.datasets import _validate_description_length as dataset_validate
|
||||
from controllers.service_api.dataset.dataset import _validate_description_length as service_dataset_validate
|
||||
|
||||
# Test same valid inputs
|
||||
valid_desc = "x" * 400
|
||||
assert app_validate(valid_desc) == dataset_validate(valid_desc) == service_dataset_validate(valid_desc)
|
||||
assert app_validate("") == dataset_validate("") == service_dataset_validate("")
|
||||
assert app_validate(None) == dataset_validate(None) == service_dataset_validate(None)
|
||||
|
||||
# Test same invalid inputs produce same error
|
||||
invalid_desc = "x" * 401
|
||||
|
||||
app_error = None
|
||||
dataset_error = None
|
||||
service_dataset_error = None
|
||||
|
||||
try:
|
||||
app_validate(invalid_desc)
|
||||
except ValueError as e:
|
||||
app_error = str(e)
|
||||
|
||||
try:
|
||||
dataset_validate(invalid_desc)
|
||||
except ValueError as e:
|
||||
dataset_error = str(e)
|
||||
|
||||
try:
|
||||
service_dataset_validate(invalid_desc)
|
||||
except ValueError as e:
|
||||
service_dataset_error = str(e)
|
||||
|
||||
assert app_error == dataset_error == service_dataset_error
|
||||
assert app_error == "Description cannot exceed 400 characters."
|
||||
|
||||
def test_boundary_values(self):
|
||||
"""Test boundary values for description validation"""
|
||||
from libs.validators import validate_description_length
|
||||
from controllers.console.app.app import _validate_description_length
|
||||
|
||||
# Test exact boundary
|
||||
exactly_400 = "x" * 400
|
||||
assert validate_description_length(exactly_400) == exactly_400
|
||||
assert _validate_description_length(exactly_400) == exactly_400
|
||||
|
||||
# Test just over boundary
|
||||
just_over_400 = "x" * 401
|
||||
with pytest.raises(ValueError):
|
||||
validate_description_length(just_over_400)
|
||||
_validate_description_length(just_over_400)
|
||||
|
||||
# Test just under boundary
|
||||
just_under_400 = "x" * 399
|
||||
assert validate_description_length(just_under_400) == just_under_400
|
||||
assert _validate_description_length(just_under_400) == just_under_400
|
||||
|
||||
def test_edge_cases(self):
|
||||
"""Test edge cases for description validation"""
|
||||
from libs.validators import validate_description_length
|
||||
from controllers.console.app.app import _validate_description_length
|
||||
|
||||
# Test None input
|
||||
assert validate_description_length(None) is None
|
||||
assert _validate_description_length(None) is None
|
||||
|
||||
# Test empty string
|
||||
assert validate_description_length("") == ""
|
||||
assert _validate_description_length("") == ""
|
||||
|
||||
# Test single character
|
||||
assert validate_description_length("a") == "a"
|
||||
assert _validate_description_length("a") == "a"
|
||||
|
||||
# Test unicode characters
|
||||
unicode_desc = "测试" * 200 # 400 characters in Chinese
|
||||
assert validate_description_length(unicode_desc) == unicode_desc
|
||||
assert _validate_description_length(unicode_desc) == unicode_desc
|
||||
|
||||
# Test unicode over limit
|
||||
unicode_over = "测试" * 201 # 402 characters
|
||||
with pytest.raises(ValueError):
|
||||
validate_description_length(unicode_over)
|
||||
_validate_description_length(unicode_over)
|
||||
|
||||
def test_whitespace_handling(self):
|
||||
"""Test how validation handles whitespace"""
|
||||
from libs.validators import validate_description_length
|
||||
from controllers.console.app.app import _validate_description_length
|
||||
|
||||
# Test description with spaces
|
||||
spaces_400 = " " * 400
|
||||
assert validate_description_length(spaces_400) == spaces_400
|
||||
assert _validate_description_length(spaces_400) == spaces_400
|
||||
|
||||
# Test description with spaces over limit
|
||||
spaces_401 = " " * 401
|
||||
with pytest.raises(ValueError):
|
||||
validate_description_length(spaces_401)
|
||||
_validate_description_length(spaces_401)
|
||||
|
||||
# Test mixed content
|
||||
mixed_400 = "a" * 200 + " " * 200
|
||||
assert validate_description_length(mixed_400) == mixed_400
|
||||
assert _validate_description_length(mixed_400) == mixed_400
|
||||
|
||||
# Test mixed over limit
|
||||
mixed_401 = "a" * 200 + " " * 201
|
||||
with pytest.raises(ValueError):
|
||||
validate_description_length(mixed_401)
|
||||
_validate_description_length(mixed_401)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
|
||||
@ -9,8 +9,8 @@ from flask.testing import FlaskClient
|
||||
from controllers.console.app import model_config as model_config_api
|
||||
from controllers.console.app import wraps
|
||||
from libs.datetime_utils import naive_utc_now
|
||||
from models import App, Tenant
|
||||
from models.account import Account, TenantAccountJoin, TenantAccountRole
|
||||
from models import Account, App, Tenant
|
||||
from models.account import TenantAccountRole
|
||||
from models.model import AppMode
|
||||
from services.app_model_config_service import AppModelConfigService
|
||||
|
||||
@ -30,8 +30,9 @@ class TestModelConfigResourcePermissions:
|
||||
return app
|
||||
|
||||
@pytest.fixture
|
||||
def mock_account(self, monkeypatch: pytest.MonkeyPatch):
|
||||
def mock_account(self):
|
||||
"""Create a mock Account for testing."""
|
||||
|
||||
account = Account()
|
||||
account.id = str(uuid.uuid4())
|
||||
account.name = "Test User"
|
||||
@ -40,24 +41,12 @@ class TestModelConfigResourcePermissions:
|
||||
account.created_at = naive_utc_now()
|
||||
account.updated_at = naive_utc_now()
|
||||
|
||||
# Create mock tenant
|
||||
tenant = Tenant()
|
||||
tenant.id = str(uuid.uuid4())
|
||||
tenant.name = "Test Tenant"
|
||||
|
||||
mock_session_instance = mock.Mock()
|
||||
|
||||
mock_tenant_join = TenantAccountJoin(role=TenantAccountRole.OWNER)
|
||||
monkeypatch.setattr(mock_session_instance, "scalar", mock.Mock(return_value=mock_tenant_join))
|
||||
|
||||
mock_scalars_result = mock.Mock()
|
||||
mock_scalars_result.one.return_value = tenant
|
||||
monkeypatch.setattr(mock_session_instance, "scalars", mock.Mock(return_value=mock_scalars_result))
|
||||
|
||||
mock_session_context = mock.Mock()
|
||||
mock_session_context.__enter__.return_value = mock_session_instance
|
||||
monkeypatch.setattr("models.account.Session", lambda _, expire_on_commit: mock_session_context)
|
||||
|
||||
account.current_tenant = tenant
|
||||
account._current_tenant = tenant
|
||||
return account
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
|
||||
175
api/tests/integration_tests/ssrf_proxy/TEST_CASES_README.md
Normal file
175
api/tests/integration_tests/ssrf_proxy/TEST_CASES_README.md
Normal file
@ -0,0 +1,175 @@
|
||||
# SSRF Proxy Test Cases
|
||||
|
||||
## Overview
|
||||
|
||||
The SSRF proxy test suite uses YAML files to define test cases, making them easier to maintain and extend without modifying code. These tests validate the SSRF proxy configuration in `docker/ssrf_proxy/`.
|
||||
|
||||
## Location
|
||||
|
||||
These tests are located in `api/tests/integration_tests/ssrf_proxy/` because they require the Python environment from the API project.
|
||||
|
||||
## Usage
|
||||
|
||||
### Basic Testing
|
||||
|
||||
From the `api/` directory:
|
||||
|
||||
```bash
|
||||
uv run python tests/integration_tests/ssrf_proxy/test_ssrf_proxy.py
|
||||
```
|
||||
|
||||
Or from the repository root:
|
||||
|
||||
```bash
|
||||
cd api && uv run python tests/integration_tests/ssrf_proxy/test_ssrf_proxy.py
|
||||
```
|
||||
|
||||
### List Available Tests
|
||||
|
||||
View all test cases without running them:
|
||||
|
||||
```bash
|
||||
uv run python tests/integration_tests/ssrf_proxy/test_ssrf_proxy.py --list-tests
|
||||
```
|
||||
|
||||
### Use Custom Test File
|
||||
|
||||
Run tests from a specific YAML file:
|
||||
|
||||
```bash
|
||||
uv run python tests/integration_tests/ssrf_proxy/test_ssrf_proxy.py --test-file test_cases_extended.yaml
|
||||
```
|
||||
|
||||
### Development Mode Testing
|
||||
|
||||
**WARNING: Development mode DISABLES all SSRF protections! Only use in development environments!**
|
||||
|
||||
Test the development mode configuration (used by docker-compose.middleware.yaml):
|
||||
|
||||
```bash
|
||||
uv run python tests/integration_tests/ssrf_proxy/test_ssrf_proxy.py --dev-mode
|
||||
```
|
||||
|
||||
Development mode:
|
||||
|
||||
- Mounts `conf.d.dev/` configuration that allows ALL requests
|
||||
- Uses `test_cases_dev_mode.yaml` by default (all tests expect ALLOW)
|
||||
- Verifies that private networks, cloud metadata, and non-standard ports are accessible
|
||||
- Should NEVER be used in production environments
|
||||
|
||||
### Command Line Options
|
||||
|
||||
- `--host HOST`: Proxy host (default: localhost)
|
||||
- `--port PORT`: Proxy port (default: 3128)
|
||||
- `--no-container`: Don't start container (assume proxy is already running)
|
||||
- `--save-results`: Save test results to JSON file
|
||||
- `--test-file FILE`: Path to YAML file containing test cases
|
||||
- `--list-tests`: List all test cases without running them
|
||||
- `--dev-mode`: Run in development mode (DISABLES all SSRF protections - DO NOT use in production!)
|
||||
|
||||
## YAML Test Case Format
|
||||
|
||||
Test cases are organized by categories in YAML files:
|
||||
|
||||
```yaml
|
||||
test_categories:
|
||||
category_key:
|
||||
name: "Category Display Name"
|
||||
description: "Category description"
|
||||
test_cases:
|
||||
- name: "Test Case Name"
|
||||
url: "http://example.com"
|
||||
expected_blocked: false # true if should be blocked, false if allowed
|
||||
description: "Optional test description"
|
||||
```
|
||||
|
||||
## Available Test Files
|
||||
|
||||
1. **test_cases.yaml** - Standard test suite with essential test cases (default)
|
||||
1. **test_cases_extended.yaml** - Extended test suite with additional edge cases and scenarios
|
||||
1. **test_cases_dev_mode.yaml** - Development mode test suite (all requests should be allowed)
|
||||
|
||||
All files are located in `api/tests/integration_tests/ssrf_proxy/`
|
||||
|
||||
## Categories
|
||||
|
||||
### Standard Categories
|
||||
|
||||
- **Private Networks**: Tests for blocking private IP ranges and loopback addresses
|
||||
- **Cloud Metadata**: Tests for blocking cloud provider metadata endpoints
|
||||
- **Public Internet**: Tests for allowing legitimate public internet access
|
||||
- **Port Restrictions**: Tests for port-based access control
|
||||
|
||||
### Extended Categories (in test_cases_extended.yaml)
|
||||
|
||||
- **IPv6 Tests**: Tests for IPv6 address handling
|
||||
- **Special Cases**: Edge cases like decimal/octal/hex IP notation
|
||||
|
||||
## Adding New Test Cases
|
||||
|
||||
1. Edit the YAML file (or create a new one)
|
||||
1. Add test cases under appropriate categories
|
||||
1. Run with `--test-file` option if using a custom file
|
||||
|
||||
Example:
|
||||
|
||||
```yaml
|
||||
test_categories:
|
||||
custom_tests:
|
||||
name: "Custom Tests"
|
||||
description: "My custom test cases"
|
||||
test_cases:
|
||||
- name: "Custom Test 1"
|
||||
url: "http://test.example.com"
|
||||
expected_blocked: false
|
||||
description: "Testing custom domain"
|
||||
```
|
||||
|
||||
## What Gets Tested
|
||||
|
||||
The tests validate the SSRF proxy configuration files in `docker/ssrf_proxy/`:
|
||||
|
||||
- `squid.conf.template` - Squid proxy configuration
|
||||
- `docker-entrypoint.sh` - Container initialization script
|
||||
- `conf.d/` - Additional configuration files (if present)
|
||||
- `conf.d.dev/` - Development mode configuration (when using --dev-mode)
|
||||
|
||||
## Development Mode Configuration
|
||||
|
||||
Development mode provides a zero-configuration environment for local development:
|
||||
|
||||
- Mounts `conf.d.dev/` instead of `conf.d/`
|
||||
- Allows ALL requests including private networks and cloud metadata
|
||||
- Enables access to any port
|
||||
- Disables all SSRF protections
|
||||
|
||||
### Using Development Mode with Docker Compose
|
||||
|
||||
From the main Dify repository root:
|
||||
|
||||
```bash
|
||||
# Use the development overlay
|
||||
docker-compose -f docker-compose.middleware.yaml -f docker/ssrf_proxy/docker-compose.dev.yaml up ssrf_proxy
|
||||
```
|
||||
|
||||
Or manually mount the development configuration:
|
||||
|
||||
```bash
|
||||
docker run -d \
|
||||
--name ssrf-proxy-dev \
|
||||
-p 3128:3128 \
|
||||
-v ./docker/ssrf_proxy/conf.d.dev:/etc/squid/conf.d:ro \
|
||||
# ... other volumes
|
||||
ubuntu/squid:latest
|
||||
```
|
||||
|
||||
**CRITICAL**: Never use this configuration in production!
|
||||
|
||||
## Benefits
|
||||
|
||||
- **Maintainability**: Test cases can be updated without code changes
|
||||
- **Extensibility**: Easy to add new test cases or categories
|
||||
- **Clarity**: YAML format is human-readable and self-documenting
|
||||
- **Flexibility**: Multiple test files for different scenarios
|
||||
- **Fallback**: Code includes default test cases if YAML loading fails
|
||||
- **Integration**: Properly integrated with the API project's Python environment
|
||||
1
api/tests/integration_tests/ssrf_proxy/__init__.py
Normal file
1
api/tests/integration_tests/ssrf_proxy/__init__.py
Normal file
@ -0,0 +1 @@
|
||||
"""SSRF Proxy Integration Tests"""
|
||||
129
api/tests/integration_tests/ssrf_proxy/test_cases.yaml
Normal file
129
api/tests/integration_tests/ssrf_proxy/test_cases.yaml
Normal file
@ -0,0 +1,129 @@
|
||||
# SSRF Proxy Test Cases Configuration
|
||||
# This file defines all test cases for the SSRF proxy
|
||||
# Each test case validates whether the proxy correctly blocks or allows requests
|
||||
|
||||
test_categories:
|
||||
private_networks:
|
||||
name: "Private Networks"
|
||||
description: "Tests for blocking private IP ranges and loopback addresses"
|
||||
test_cases:
|
||||
- name: "Loopback (127.0.0.1)"
|
||||
url: "http://127.0.0.1"
|
||||
expected_blocked: true
|
||||
description: "IPv4 loopback address"
|
||||
|
||||
- name: "Localhost"
|
||||
url: "http://localhost"
|
||||
expected_blocked: true
|
||||
description: "Localhost hostname"
|
||||
|
||||
- name: "Private 10.x.x.x"
|
||||
url: "http://10.0.0.1"
|
||||
expected_blocked: true
|
||||
description: "RFC 1918 private network"
|
||||
|
||||
- name: "Private 172.16.x.x"
|
||||
url: "http://172.16.0.1"
|
||||
expected_blocked: true
|
||||
description: "RFC 1918 private network"
|
||||
|
||||
- name: "Private 192.168.x.x"
|
||||
url: "http://192.168.1.1"
|
||||
expected_blocked: true
|
||||
description: "RFC 1918 private network"
|
||||
|
||||
- name: "Link-local"
|
||||
url: "http://169.254.1.1"
|
||||
expected_blocked: true
|
||||
description: "Link-local address"
|
||||
|
||||
- name: "This network"
|
||||
url: "http://0.0.0.0"
|
||||
expected_blocked: true
|
||||
description: "'This' network address"
|
||||
|
||||
cloud_metadata:
|
||||
name: "Cloud Metadata"
|
||||
description: "Tests for blocking cloud provider metadata endpoints"
|
||||
test_cases:
|
||||
- name: "AWS Metadata"
|
||||
url: "http://169.254.169.254/latest/meta-data/"
|
||||
expected_blocked: true
|
||||
description: "AWS EC2 metadata endpoint"
|
||||
|
||||
- name: "Azure Metadata"
|
||||
url: "http://169.254.169.254/metadata/instance"
|
||||
expected_blocked: true
|
||||
description: "Azure metadata endpoint"
|
||||
|
||||
# Note: metadata.google.internal is not included as it may resolve to public IPs
|
||||
|
||||
public_internet:
|
||||
name: "Public Internet"
|
||||
description: "Tests for allowing legitimate public internet access"
|
||||
test_cases:
|
||||
- name: "Example.com"
|
||||
url: "http://example.com"
|
||||
expected_blocked: false
|
||||
description: "Public website"
|
||||
|
||||
- name: "Google HTTPS"
|
||||
url: "https://www.google.com"
|
||||
expected_blocked: false
|
||||
description: "HTTPS public website"
|
||||
|
||||
- name: "HTTPBin API"
|
||||
url: "http://httpbin.org/get"
|
||||
expected_blocked: false
|
||||
description: "Public API endpoint"
|
||||
|
||||
- name: "GitHub API"
|
||||
url: "https://api.github.com"
|
||||
expected_blocked: false
|
||||
description: "Public API over HTTPS"
|
||||
|
||||
port_restrictions:
|
||||
name: "Port Restrictions"
|
||||
description: "Tests for port-based access control"
|
||||
test_cases:
|
||||
- name: "HTTP Port 80"
|
||||
url: "http://example.com:80"
|
||||
expected_blocked: false
|
||||
description: "Standard HTTP port"
|
||||
|
||||
- name: "HTTPS Port 443"
|
||||
url: "http://example.com:443"
|
||||
expected_blocked: false
|
||||
description: "Standard HTTPS port"
|
||||
|
||||
- name: "Port 8080"
|
||||
url: "http://example.com:8080"
|
||||
expected_blocked: true
|
||||
description: "Non-standard port"
|
||||
|
||||
- name: "Port 3000"
|
||||
url: "http://example.com:3000"
|
||||
expected_blocked: true
|
||||
description: "Development port"
|
||||
|
||||
- name: "SSH Port 22"
|
||||
url: "http://example.com:22"
|
||||
expected_blocked: true
|
||||
description: "SSH port"
|
||||
|
||||
- name: "MySQL Port 3306"
|
||||
url: "http://example.com:3306"
|
||||
expected_blocked: true
|
||||
description: "Database port"
|
||||
|
||||
# Additional test configurations can be added here
|
||||
# For example:
|
||||
#
|
||||
# ipv6_tests:
|
||||
# name: "IPv6 Tests"
|
||||
# description: "Tests for IPv6 address handling"
|
||||
# test_cases:
|
||||
# - name: "IPv6 Loopback"
|
||||
# url: "http://[::1]"
|
||||
# expected_blocked: true
|
||||
# description: "IPv6 loopback address"
|
||||
168
api/tests/integration_tests/ssrf_proxy/test_cases_dev_mode.yaml
Normal file
168
api/tests/integration_tests/ssrf_proxy/test_cases_dev_mode.yaml
Normal file
@ -0,0 +1,168 @@
|
||||
# Development Mode Test Cases for SSRF Proxy
|
||||
# These test cases verify that development mode correctly disables all SSRF protections
|
||||
# WARNING: All requests should be ALLOWED in development mode
|
||||
|
||||
test_categories:
|
||||
private_networks:
|
||||
name: "Private Networks (Dev Mode)"
|
||||
description: "In dev mode, private networks should be ALLOWED"
|
||||
test_cases:
|
||||
- name: "Loopback (127.0.0.1)"
|
||||
url: "http://127.0.0.1"
|
||||
expected_blocked: false # ALLOWED in dev mode
|
||||
description: "IPv4 loopback - normally blocked, allowed in dev mode"
|
||||
|
||||
- name: "Localhost"
|
||||
url: "http://localhost"
|
||||
expected_blocked: false # ALLOWED in dev mode
|
||||
description: "Localhost hostname - normally blocked, allowed in dev mode"
|
||||
|
||||
- name: "Private 10.x.x.x"
|
||||
url: "http://10.0.0.1"
|
||||
expected_blocked: false # ALLOWED in dev mode
|
||||
description: "RFC 1918 private network - normally blocked, allowed in dev mode"
|
||||
|
||||
- name: "Private 172.16.x.x"
|
||||
url: "http://172.16.0.1"
|
||||
expected_blocked: false # ALLOWED in dev mode
|
||||
description: "RFC 1918 private network - normally blocked, allowed in dev mode"
|
||||
|
||||
- name: "Private 192.168.x.x"
|
||||
url: "http://192.168.1.1"
|
||||
expected_blocked: false # ALLOWED in dev mode
|
||||
description: "RFC 1918 private network - normally blocked, allowed in dev mode"
|
||||
|
||||
- name: "Link-local"
|
||||
url: "http://169.254.1.1"
|
||||
expected_blocked: false # ALLOWED in dev mode
|
||||
description: "Link-local address - normally blocked, allowed in dev mode"
|
||||
|
||||
- name: "This network"
|
||||
url: "http://0.0.0.0"
|
||||
expected_blocked: false # ALLOWED in dev mode
|
||||
description: "'This' network address - normally blocked, allowed in dev mode"
|
||||
|
||||
cloud_metadata:
|
||||
name: "Cloud Metadata (Dev Mode)"
|
||||
description: "In dev mode, cloud metadata endpoints should be ALLOWED"
|
||||
test_cases:
|
||||
- name: "AWS Metadata"
|
||||
url: "http://169.254.169.254/latest/meta-data/"
|
||||
expected_blocked: false # ALLOWED in dev mode
|
||||
description: "AWS EC2 metadata - normally blocked, allowed in dev mode"
|
||||
|
||||
- name: "Azure Metadata"
|
||||
url: "http://169.254.169.254/metadata/instance"
|
||||
expected_blocked: false # ALLOWED in dev mode
|
||||
description: "Azure metadata - normally blocked, allowed in dev mode"
|
||||
|
||||
non_standard_ports:
|
||||
name: "Non-Standard Ports (Dev Mode)"
|
||||
description: "In dev mode, all ports should be ALLOWED"
|
||||
test_cases:
|
||||
- name: "Port 8080"
|
||||
url: "http://example.com:8080"
|
||||
expected_blocked: false # ALLOWED in dev mode
|
||||
description: "Alternative HTTP port - normally blocked, allowed in dev mode"
|
||||
|
||||
- name: "Port 3000"
|
||||
url: "http://example.com:3000"
|
||||
expected_blocked: false # ALLOWED in dev mode
|
||||
description: "Node.js development port - normally blocked, allowed in dev mode"
|
||||
|
||||
- name: "SSH Port 22"
|
||||
url: "http://example.com:22"
|
||||
expected_blocked: false # ALLOWED in dev mode
|
||||
description: "SSH port - normally blocked, allowed in dev mode"
|
||||
|
||||
- name: "Database Port 3306"
|
||||
url: "http://example.com:3306"
|
||||
expected_blocked: false # ALLOWED in dev mode
|
||||
description: "MySQL port - normally blocked, allowed in dev mode"
|
||||
|
||||
- name: "Database Port 5432"
|
||||
url: "http://example.com:5432"
|
||||
expected_blocked: false # ALLOWED in dev mode
|
||||
description: "PostgreSQL port - normally blocked, allowed in dev mode"
|
||||
|
||||
- name: "Redis Port 6379"
|
||||
url: "http://example.com:6379"
|
||||
expected_blocked: false # ALLOWED in dev mode
|
||||
description: "Redis port - normally blocked, allowed in dev mode"
|
||||
|
||||
- name: "MongoDB Port 27017"
|
||||
url: "http://example.com:27017"
|
||||
expected_blocked: false # ALLOWED in dev mode
|
||||
description: "MongoDB port - normally blocked, allowed in dev mode"
|
||||
|
||||
- name: "High Port 12345"
|
||||
url: "http://example.com:12345"
|
||||
expected_blocked: false # ALLOWED in dev mode
|
||||
description: "Random high port - normally blocked, allowed in dev mode"
|
||||
|
||||
localhost_ports:
|
||||
name: "Localhost with Various Ports (Dev Mode)"
|
||||
description: "In dev mode, localhost with any port should be ALLOWED"
|
||||
test_cases:
|
||||
- name: "Localhost:8080"
|
||||
url: "http://localhost:8080"
|
||||
expected_blocked: false # ALLOWED in dev mode
|
||||
description: "Localhost with port 8080 - normally blocked, allowed in dev mode"
|
||||
|
||||
- name: "Localhost:3000"
|
||||
url: "http://localhost:3000"
|
||||
expected_blocked: false # ALLOWED in dev mode
|
||||
description: "Localhost with port 3000 - normally blocked, allowed in dev mode"
|
||||
|
||||
- name: "127.0.0.1:9200"
|
||||
url: "http://127.0.0.1:9200"
|
||||
expected_blocked: false # ALLOWED in dev mode
|
||||
description: "Loopback with Elasticsearch port - normally blocked, allowed in dev mode"
|
||||
|
||||
- name: "127.0.0.1:5001"
|
||||
url: "http://127.0.0.1:5001"
|
||||
expected_blocked: false # ALLOWED in dev mode
|
||||
description: "Loopback with API port - normally blocked, allowed in dev mode"
|
||||
|
||||
public_internet:
|
||||
name: "Public Internet (Dev Mode)"
|
||||
description: "Public internet should still work in dev mode"
|
||||
test_cases:
|
||||
- name: "Example.com"
|
||||
url: "http://example.com"
|
||||
expected_blocked: false
|
||||
description: "Public website - always allowed"
|
||||
|
||||
- name: "Google HTTPS"
|
||||
url: "https://www.google.com"
|
||||
expected_blocked: false
|
||||
description: "HTTPS public website - always allowed"
|
||||
|
||||
- name: "GitHub API"
|
||||
url: "https://api.github.com"
|
||||
expected_blocked: false
|
||||
description: "Public API over HTTPS - always allowed"
|
||||
|
||||
special_cases:
|
||||
name: "Special Cases (Dev Mode)"
|
||||
description: "Edge cases that should all be allowed in dev mode"
|
||||
test_cases:
|
||||
- name: "Decimal IP notation"
|
||||
url: "http://2130706433"
|
||||
expected_blocked: false # ALLOWED in dev mode
|
||||
description: "127.0.0.1 in decimal - normally blocked, allowed in dev mode"
|
||||
|
||||
- name: "Private network in subdomain"
|
||||
url: "http://192-168-1-1.example.com"
|
||||
expected_blocked: false
|
||||
description: "Domain that looks like private IP - always allowed as it resolves externally"
|
||||
|
||||
- name: "IPv6 Loopback"
|
||||
url: "http://[::1]"
|
||||
expected_blocked: false # ALLOWED in dev mode
|
||||
description: "IPv6 loopback - normally blocked, allowed in dev mode"
|
||||
|
||||
- name: "IPv6 Link-local"
|
||||
url: "http://[fe80::1]"
|
||||
expected_blocked: false # ALLOWED in dev mode
|
||||
description: "IPv6 link-local - normally blocked, allowed in dev mode"
|
||||
219
api/tests/integration_tests/ssrf_proxy/test_cases_extended.yaml
Normal file
219
api/tests/integration_tests/ssrf_proxy/test_cases_extended.yaml
Normal file
@ -0,0 +1,219 @@
|
||||
# Extended SSRF Proxy Test Cases Configuration
|
||||
# This file contains additional test cases for comprehensive testing
|
||||
# Use with: python test_ssrf_proxy.py --test-file test_cases_extended.yaml
|
||||
|
||||
test_categories:
|
||||
# Standard test cases
|
||||
private_networks:
|
||||
name: "Private Networks"
|
||||
description: "Tests for blocking private IP ranges and loopback addresses"
|
||||
test_cases:
|
||||
- name: "Loopback (127.0.0.1)"
|
||||
url: "http://127.0.0.1"
|
||||
expected_blocked: true
|
||||
description: "IPv4 loopback address"
|
||||
|
||||
- name: "Localhost"
|
||||
url: "http://localhost"
|
||||
expected_blocked: true
|
||||
description: "Localhost hostname"
|
||||
|
||||
- name: "Private 10.x.x.x"
|
||||
url: "http://10.0.0.1"
|
||||
expected_blocked: true
|
||||
description: "RFC 1918 private network"
|
||||
|
||||
- name: "Private 172.16.x.x"
|
||||
url: "http://172.16.0.1"
|
||||
expected_blocked: true
|
||||
description: "RFC 1918 private network"
|
||||
|
||||
- name: "Private 192.168.x.x"
|
||||
url: "http://192.168.1.1"
|
||||
expected_blocked: true
|
||||
description: "RFC 1918 private network"
|
||||
|
||||
- name: "Link-local"
|
||||
url: "http://169.254.1.1"
|
||||
expected_blocked: true
|
||||
description: "Link-local address"
|
||||
|
||||
- name: "This network"
|
||||
url: "http://0.0.0.0"
|
||||
expected_blocked: true
|
||||
description: "'This' network address"
|
||||
|
||||
cloud_metadata:
|
||||
name: "Cloud Metadata"
|
||||
description: "Tests for blocking cloud provider metadata endpoints"
|
||||
test_cases:
|
||||
- name: "AWS Metadata"
|
||||
url: "http://169.254.169.254/latest/meta-data/"
|
||||
expected_blocked: true
|
||||
description: "AWS EC2 metadata endpoint"
|
||||
|
||||
- name: "Azure Metadata"
|
||||
url: "http://169.254.169.254/metadata/instance"
|
||||
expected_blocked: true
|
||||
description: "Azure metadata endpoint"
|
||||
|
||||
- name: "DigitalOcean Metadata"
|
||||
url: "http://169.254.169.254/metadata/v1"
|
||||
expected_blocked: true
|
||||
description: "DigitalOcean metadata endpoint"
|
||||
|
||||
- name: "Oracle Cloud Metadata"
|
||||
url: "http://169.254.169.254/opc/v1"
|
||||
expected_blocked: true
|
||||
description: "Oracle Cloud metadata endpoint"
|
||||
|
||||
public_internet:
|
||||
name: "Public Internet"
|
||||
description: "Tests for allowing legitimate public internet access"
|
||||
test_cases:
|
||||
- name: "Example.com"
|
||||
url: "http://example.com"
|
||||
expected_blocked: false
|
||||
description: "Public website"
|
||||
|
||||
- name: "Google HTTPS"
|
||||
url: "https://www.google.com"
|
||||
expected_blocked: false
|
||||
description: "HTTPS public website"
|
||||
|
||||
- name: "HTTPBin API"
|
||||
url: "http://httpbin.org/get"
|
||||
expected_blocked: false
|
||||
description: "Public API endpoint"
|
||||
|
||||
- name: "GitHub API"
|
||||
url: "https://api.github.com"
|
||||
expected_blocked: false
|
||||
description: "Public API over HTTPS"
|
||||
|
||||
- name: "OpenAI API"
|
||||
url: "https://api.openai.com"
|
||||
expected_blocked: false
|
||||
description: "OpenAI API endpoint"
|
||||
|
||||
- name: "Anthropic API"
|
||||
url: "https://api.anthropic.com"
|
||||
expected_blocked: false
|
||||
description: "Anthropic API endpoint"
|
||||
|
||||
port_restrictions:
|
||||
name: "Port Restrictions"
|
||||
description: "Tests for port-based access control"
|
||||
test_cases:
|
||||
- name: "HTTP Port 80"
|
||||
url: "http://example.com:80"
|
||||
expected_blocked: false
|
||||
description: "Standard HTTP port"
|
||||
|
||||
- name: "HTTPS Port 443"
|
||||
url: "http://example.com:443"
|
||||
expected_blocked: false
|
||||
description: "Standard HTTPS port"
|
||||
|
||||
- name: "Port 8080"
|
||||
url: "http://example.com:8080"
|
||||
expected_blocked: true
|
||||
description: "Alternative HTTP port"
|
||||
|
||||
- name: "Port 3000"
|
||||
url: "http://example.com:3000"
|
||||
expected_blocked: true
|
||||
description: "Node.js development port"
|
||||
|
||||
- name: "SSH Port 22"
|
||||
url: "http://example.com:22"
|
||||
expected_blocked: true
|
||||
description: "SSH port"
|
||||
|
||||
- name: "Telnet Port 23"
|
||||
url: "http://example.com:23"
|
||||
expected_blocked: true
|
||||
description: "Telnet port"
|
||||
|
||||
- name: "SMTP Port 25"
|
||||
url: "http://example.com:25"
|
||||
expected_blocked: true
|
||||
description: "SMTP mail port"
|
||||
|
||||
- name: "MySQL Port 3306"
|
||||
url: "http://example.com:3306"
|
||||
expected_blocked: true
|
||||
description: "MySQL database port"
|
||||
|
||||
- name: "PostgreSQL Port 5432"
|
||||
url: "http://example.com:5432"
|
||||
expected_blocked: true
|
||||
description: "PostgreSQL database port"
|
||||
|
||||
- name: "Redis Port 6379"
|
||||
url: "http://example.com:6379"
|
||||
expected_blocked: true
|
||||
description: "Redis port"
|
||||
|
||||
- name: "MongoDB Port 27017"
|
||||
url: "http://example.com:27017"
|
||||
expected_blocked: true
|
||||
description: "MongoDB port"
|
||||
|
||||
ipv6_tests:
|
||||
name: "IPv6 Tests"
|
||||
description: "Tests for IPv6 address handling"
|
||||
test_cases:
|
||||
- name: "IPv6 Loopback"
|
||||
url: "http://[::1]"
|
||||
expected_blocked: true
|
||||
description: "IPv6 loopback address"
|
||||
|
||||
- name: "IPv6 All zeros"
|
||||
url: "http://[::]"
|
||||
expected_blocked: true
|
||||
description: "IPv6 all zeros address"
|
||||
|
||||
- name: "IPv6 Link-local"
|
||||
url: "http://[fe80::1]"
|
||||
expected_blocked: true
|
||||
description: "IPv6 link-local address"
|
||||
|
||||
- name: "IPv6 Unique local"
|
||||
url: "http://[fc00::1]"
|
||||
expected_blocked: true
|
||||
description: "IPv6 unique local address"
|
||||
|
||||
special_cases:
|
||||
name: "Special Cases"
|
||||
description: "Edge cases and special scenarios"
|
||||
test_cases:
|
||||
- name: "Decimal IP notation"
|
||||
url: "http://2130706433"
|
||||
expected_blocked: true
|
||||
description: "127.0.0.1 in decimal notation"
|
||||
|
||||
- name: "Octal IP notation"
|
||||
url: "http://0177.0.0.1"
|
||||
expected_blocked: true
|
||||
description: "127.0.0.1 with octal notation"
|
||||
|
||||
- name: "Hex IP notation"
|
||||
url: "http://0x7f.0.0.1"
|
||||
expected_blocked: true
|
||||
description: "127.0.0.1 with hex notation"
|
||||
|
||||
- name: "Mixed notation"
|
||||
url: "http://0x7f.0.0.0x1"
|
||||
expected_blocked: true
|
||||
description: "127.0.0.1 with mixed hex notation"
|
||||
|
||||
- name: "Localhost with port"
|
||||
url: "http://localhost:8080"
|
||||
expected_blocked: true
|
||||
description: "Localhost with non-standard port"
|
||||
|
||||
- name: "Domain with private IP"
|
||||
url: "http://192-168-1-1.example.com"
|
||||
expected_blocked: false
|
||||
description: "Domain that looks like private IP (should resolve)"
|
||||
482
api/tests/integration_tests/ssrf_proxy/test_ssrf_proxy.py
Executable file
482
api/tests/integration_tests/ssrf_proxy/test_ssrf_proxy.py
Executable file
@ -0,0 +1,482 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
SSRF Proxy Test Suite
|
||||
|
||||
This script tests the SSRF proxy configuration to ensure it blocks
|
||||
private networks while allowing public internet access.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
from dataclasses import dataclass
|
||||
from enum import Enum
|
||||
from typing import final
|
||||
|
||||
import yaml
|
||||
|
||||
|
||||
# Color codes for terminal output
|
||||
class Colors:
|
||||
RED: str = "\033[0;31m"
|
||||
GREEN: str = "\033[0;32m"
|
||||
YELLOW: str = "\033[1;33m"
|
||||
BLUE: str = "\033[0;34m"
|
||||
NC: str = "\033[0m" # No Color
|
||||
|
||||
|
||||
class TestResult(Enum):
|
||||
PASSED = "passed"
|
||||
FAILED = "failed"
|
||||
SKIPPED = "skipped"
|
||||
|
||||
|
||||
@dataclass
|
||||
class TestCase:
|
||||
name: str
|
||||
url: str
|
||||
expected_blocked: bool
|
||||
category: str
|
||||
description: str = ""
|
||||
|
||||
|
||||
@final
|
||||
class SSRFProxyTester:
|
||||
def __init__(
|
||||
self,
|
||||
proxy_host: str = "localhost",
|
||||
proxy_port: int = 3128,
|
||||
test_file: str | None = None,
|
||||
dev_mode: bool = False,
|
||||
):
|
||||
self.proxy_host = proxy_host
|
||||
self.proxy_port = proxy_port
|
||||
self.proxy_url = f"http://{proxy_host}:{proxy_port}"
|
||||
self.container_name = "ssrf-proxy-test-dev" if dev_mode else "ssrf-proxy-test"
|
||||
self.image = "ubuntu/squid:latest"
|
||||
self.results: list[dict[str, object]] = []
|
||||
self.dev_mode = dev_mode
|
||||
# Use dev mode test cases by default when in dev mode
|
||||
if dev_mode and test_file is None:
|
||||
self.test_file = "test_cases_dev_mode.yaml"
|
||||
else:
|
||||
self.test_file = test_file or "test_cases.yaml"
|
||||
|
||||
def start_proxy_container(self) -> bool:
|
||||
"""Start the SSRF proxy container"""
|
||||
mode_str = " (DEVELOPMENT MODE)" if self.dev_mode else ""
|
||||
print(f"{Colors.YELLOW}Starting SSRF proxy container{mode_str}...{Colors.NC}")
|
||||
if self.dev_mode:
|
||||
print(f"{Colors.RED}WARNING: Development mode DISABLES all SSRF protections!{Colors.NC}")
|
||||
|
||||
# Stop and remove existing container if exists
|
||||
_ = subprocess.run(["docker", "stop", self.container_name], capture_output=True, text=True)
|
||||
_ = subprocess.run(["docker", "rm", self.container_name], capture_output=True, text=True)
|
||||
|
||||
# Get directories for mounting config files
|
||||
script_dir = os.path.dirname(os.path.abspath(__file__))
|
||||
# Docker config files are in docker/ssrf_proxy relative to project root
|
||||
project_root = os.path.abspath(os.path.join(script_dir, "..", "..", "..", ".."))
|
||||
docker_config_dir = os.path.join(project_root, "docker", "ssrf_proxy")
|
||||
|
||||
# Choose configuration template based on mode
|
||||
if self.dev_mode:
|
||||
config_template = "squid.conf.dev.template"
|
||||
else:
|
||||
config_template = "squid.conf.template"
|
||||
|
||||
# Start container
|
||||
cmd = [
|
||||
"docker",
|
||||
"run",
|
||||
"-d",
|
||||
"--name",
|
||||
self.container_name,
|
||||
"-p",
|
||||
f"{self.proxy_port}:{self.proxy_port}",
|
||||
"-p",
|
||||
"8194:8194",
|
||||
"-v",
|
||||
f"{docker_config_dir}/{config_template}:/etc/squid/squid.conf.template:ro",
|
||||
"-v",
|
||||
f"{docker_config_dir}/docker-entrypoint.sh:/docker-entrypoint-mount.sh:ro",
|
||||
"-e",
|
||||
f"HTTP_PORT={self.proxy_port}",
|
||||
"-e",
|
||||
"COREDUMP_DIR=/var/spool/squid",
|
||||
"-e",
|
||||
"REVERSE_PROXY_PORT=8194",
|
||||
"-e",
|
||||
"SANDBOX_HOST=sandbox",
|
||||
"-e",
|
||||
"SANDBOX_PORT=8194",
|
||||
"--entrypoint",
|
||||
"sh",
|
||||
self.image,
|
||||
"-c",
|
||||
"cp /docker-entrypoint-mount.sh /docker-entrypoint.sh && sed -i 's/\\r$//' /docker-entrypoint.sh && chmod +x /docker-entrypoint.sh && /docker-entrypoint.sh", # noqa: E501
|
||||
]
|
||||
|
||||
# Mount configuration directory (only in normal mode)
|
||||
# In dev mode, the dev template already allows everything
|
||||
if not self.dev_mode:
|
||||
# Normal mode: mount regular conf.d if it exists
|
||||
conf_d_path = f"{docker_config_dir}/conf.d"
|
||||
if os.path.exists(conf_d_path) and os.listdir(conf_d_path):
|
||||
cmd.insert(-3, "-v")
|
||||
cmd.insert(-3, f"{conf_d_path}:/etc/squid/conf.d:ro")
|
||||
else:
|
||||
print(f"{Colors.YELLOW}Using development mode configuration (all SSRF protections disabled){Colors.NC}")
|
||||
|
||||
result = subprocess.run(cmd, capture_output=True, text=True)
|
||||
|
||||
if result.returncode != 0:
|
||||
print(f"{Colors.RED}Failed to start container: {result.stderr}{Colors.NC}")
|
||||
return False
|
||||
|
||||
# Wait for proxy to start
|
||||
print(f"{Colors.YELLOW}Waiting for proxy to start...{Colors.NC}")
|
||||
time.sleep(5)
|
||||
|
||||
# Check if container is running
|
||||
result = subprocess.run(
|
||||
["docker", "ps", "--filter", f"name={self.container_name}"],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
)
|
||||
|
||||
if self.container_name not in result.stdout:
|
||||
print(f"{Colors.RED}Container failed to start!{Colors.NC}")
|
||||
logs = subprocess.run(["docker", "logs", self.container_name], capture_output=True, text=True)
|
||||
print(logs.stdout)
|
||||
return False
|
||||
|
||||
print(f"{Colors.GREEN}Proxy started successfully!{Colors.NC}\n")
|
||||
return True
|
||||
|
||||
def stop_proxy_container(self):
|
||||
"""Stop and remove the proxy container"""
|
||||
_ = subprocess.run(["docker", "stop", self.container_name], capture_output=True, text=True)
|
||||
_ = subprocess.run(["docker", "rm", self.container_name], capture_output=True, text=True)
|
||||
|
||||
def test_url(self, test_case: TestCase) -> TestResult:
|
||||
"""Test a single URL through the proxy"""
|
||||
# Configure proxy for urllib
|
||||
proxy_handler = urllib.request.ProxyHandler({"http": self.proxy_url, "https": self.proxy_url})
|
||||
opener = urllib.request.build_opener(proxy_handler)
|
||||
|
||||
try:
|
||||
# Make request through proxy
|
||||
request = urllib.request.Request(test_case.url)
|
||||
with opener.open(request, timeout=5):
|
||||
# If we got a response, the request was allowed
|
||||
is_blocked = False
|
||||
|
||||
except urllib.error.HTTPError as e:
|
||||
# HTTP errors like 403 from proxy mean blocked
|
||||
if e.code in [403, 407]:
|
||||
is_blocked = True
|
||||
else:
|
||||
# Other HTTP errors mean the request went through
|
||||
is_blocked = False
|
||||
except (urllib.error.URLError, OSError, TimeoutError) as e:
|
||||
# In dev mode, connection errors to 169.254.x.x addresses are expected
|
||||
# These addresses don't exist locally, so timeout is normal
|
||||
# The proxy allowed the request, but the destination is unreachable
|
||||
if self.dev_mode and "169.254" in test_case.url:
|
||||
# In dev mode, if we're testing 169.254.x.x addresses,
|
||||
# a timeout means the proxy allowed it (not blocked)
|
||||
is_blocked = False
|
||||
else:
|
||||
# In normal mode, or for other addresses, connection errors mean blocked
|
||||
is_blocked = True
|
||||
except Exception as e:
|
||||
# Unexpected error
|
||||
print(f"{Colors.YELLOW}Warning: Unexpected error testing {test_case.url}: {e}{Colors.NC}")
|
||||
return TestResult.SKIPPED
|
||||
|
||||
# Check if result matches expectation
|
||||
if is_blocked == test_case.expected_blocked:
|
||||
return TestResult.PASSED
|
||||
else:
|
||||
return TestResult.FAILED
|
||||
|
||||
def run_test(self, test_case: TestCase):
|
||||
"""Run a single test and record result"""
|
||||
result = self.test_url(test_case)
|
||||
|
||||
# Print result
|
||||
if result == TestResult.PASSED:
|
||||
symbol = f"{Colors.GREEN}✓{Colors.NC}"
|
||||
elif result == TestResult.FAILED:
|
||||
symbol = f"{Colors.RED}✗{Colors.NC}"
|
||||
else:
|
||||
symbol = f"{Colors.YELLOW}⊘{Colors.NC}"
|
||||
|
||||
status = "blocked" if test_case.expected_blocked else "allowed"
|
||||
print(f" {symbol} {test_case.name} (should be {status})")
|
||||
|
||||
# Record result
|
||||
self.results.append(
|
||||
{
|
||||
"name": test_case.name,
|
||||
"category": test_case.category,
|
||||
"url": test_case.url,
|
||||
"expected_blocked": test_case.expected_blocked,
|
||||
"result": result.value,
|
||||
"description": test_case.description,
|
||||
}
|
||||
)
|
||||
|
||||
def run_all_tests(self):
|
||||
"""Run all test cases"""
|
||||
test_cases = self.get_test_cases()
|
||||
|
||||
print("=" * 50)
|
||||
if self.dev_mode:
|
||||
print(" SSRF Proxy Test Suite (DEV MODE)")
|
||||
print("=" * 50)
|
||||
print(f"{Colors.RED}WARNING: Testing with SSRF protections DISABLED!{Colors.NC}")
|
||||
print(f"{Colors.YELLOW}All requests should be ALLOWED in dev mode.{Colors.NC}")
|
||||
else:
|
||||
print(" SSRF Proxy Test Suite")
|
||||
print("=" * 50)
|
||||
|
||||
# Group tests by category
|
||||
categories: dict[str, list[TestCase]] = {}
|
||||
for test in test_cases:
|
||||
if test.category not in categories:
|
||||
categories[test.category] = []
|
||||
categories[test.category].append(test)
|
||||
|
||||
# Run tests by category
|
||||
for category, tests in categories.items():
|
||||
print(f"\n{Colors.YELLOW}{category}:{Colors.NC}")
|
||||
for test in tests:
|
||||
self.run_test(test)
|
||||
|
||||
def load_test_cases_from_yaml(self, yaml_file: str = "test_cases.yaml") -> list[TestCase]:
|
||||
"""Load test cases from YAML configuration file"""
|
||||
try:
|
||||
# Try to load from YAML file
|
||||
yaml_path = os.path.join(os.path.dirname(os.path.abspath(__file__)), yaml_file)
|
||||
|
||||
with open(yaml_path) as f:
|
||||
config = yaml.safe_load(f) # pyright: ignore[reportAny]
|
||||
|
||||
test_cases: list[TestCase] = []
|
||||
|
||||
# Parse test categories and cases from YAML
|
||||
test_categories = config.get("test_categories", {}) # pyright: ignore[reportAny]
|
||||
for category_key, category_data in test_categories.items(): # pyright: ignore[reportAny]
|
||||
category_name: str = str(category_data.get("name", category_key)) # pyright: ignore[reportAny]
|
||||
|
||||
test_cases_list = category_data.get("test_cases", []) # pyright: ignore[reportAny]
|
||||
for test_data in test_cases_list: # pyright: ignore[reportAny]
|
||||
test_case = TestCase(
|
||||
name=str(test_data["name"]), # pyright: ignore[reportAny]
|
||||
url=str(test_data["url"]), # pyright: ignore[reportAny]
|
||||
expected_blocked=bool(test_data["expected_blocked"]), # pyright: ignore[reportAny]
|
||||
category=category_name,
|
||||
description=str(test_data.get("description", "")), # pyright: ignore[reportAny]
|
||||
)
|
||||
test_cases.append(test_case)
|
||||
|
||||
if test_cases:
|
||||
print(f"{Colors.BLUE}Loaded {len(test_cases)} test cases from {yaml_file}{Colors.NC}")
|
||||
return test_cases
|
||||
else:
|
||||
print(f"{Colors.YELLOW}No test cases found in {yaml_file}, using defaults{Colors.NC}")
|
||||
return self.get_default_test_cases()
|
||||
|
||||
except FileNotFoundError:
|
||||
print(f"{Colors.YELLOW}Test case file {yaml_file} not found, using defaults{Colors.NC}")
|
||||
return self.get_default_test_cases()
|
||||
except yaml.YAMLError as e:
|
||||
print(f"{Colors.YELLOW}Error parsing {yaml_file}: {e}, using defaults{Colors.NC}")
|
||||
return self.get_default_test_cases()
|
||||
except Exception as e:
|
||||
print(f"{Colors.YELLOW}Unexpected error loading {yaml_file}: {e}, using defaults{Colors.NC}")
|
||||
return self.get_default_test_cases()
|
||||
|
||||
def get_default_test_cases(self) -> list[TestCase]:
|
||||
"""Fallback test cases if YAML loading fails"""
|
||||
return [
|
||||
# Essential test cases as fallback
|
||||
TestCase("Loopback", "http://127.0.0.1", True, "Private Networks", "IPv4 loopback"),
|
||||
TestCase("Private Network", "http://192.168.1.1", True, "Private Networks", "RFC 1918"),
|
||||
TestCase("AWS Metadata", "http://169.254.169.254", True, "Cloud Metadata", "AWS metadata"),
|
||||
TestCase("Public Site", "http://example.com", False, "Public Internet", "Public website"),
|
||||
TestCase("Port 8080", "http://example.com:8080", True, "Port Restrictions", "Non-standard port"),
|
||||
]
|
||||
|
||||
def get_test_cases(self) -> list[TestCase]:
|
||||
"""Get all test cases from YAML or defaults"""
|
||||
return self.load_test_cases_from_yaml(self.test_file)
|
||||
|
||||
def print_summary(self):
|
||||
"""Print test results summary"""
|
||||
passed = sum(1 for r in self.results if r["result"] == "passed")
|
||||
failed = sum(1 for r in self.results if r["result"] == "failed")
|
||||
skipped = sum(1 for r in self.results if r["result"] == "skipped")
|
||||
|
||||
print("\n" + "=" * 50)
|
||||
print(" Test Summary")
|
||||
print("=" * 50)
|
||||
print(f"Tests Passed: {Colors.GREEN}{passed}{Colors.NC}")
|
||||
print(f"Tests Failed: {Colors.RED}{failed}{Colors.NC}")
|
||||
if skipped > 0:
|
||||
print(f"Tests Skipped: {Colors.YELLOW}{skipped}{Colors.NC}")
|
||||
|
||||
if failed == 0:
|
||||
if hasattr(self, "dev_mode") and self.dev_mode:
|
||||
print(f"\n{Colors.GREEN}✓ All tests passed! Development mode is working correctly.{Colors.NC}")
|
||||
print(
|
||||
f"{Colors.YELLOW}Remember: Dev mode DISABLES all SSRF protections - "
|
||||
f"use only for development!{Colors.NC}"
|
||||
)
|
||||
else:
|
||||
print(f"\n{Colors.GREEN}✓ All tests passed! SSRF proxy is configured correctly.{Colors.NC}")
|
||||
else:
|
||||
if hasattr(self, "dev_mode") and self.dev_mode:
|
||||
print(f"\n{Colors.RED}✗ Some tests failed. Dev mode should allow ALL requests!{Colors.NC}")
|
||||
else:
|
||||
print(f"\n{Colors.RED}✗ Some tests failed. Please review the configuration.{Colors.NC}")
|
||||
print("\nFailed tests:")
|
||||
for r in self.results:
|
||||
if r["result"] == "failed":
|
||||
status = "should be blocked" if r["expected_blocked"] else "should be allowed"
|
||||
print(f" - {r['name']} ({status}): {r['url']}")
|
||||
|
||||
return failed == 0
|
||||
|
||||
def save_results(self, filename: str = "test_results.json"):
|
||||
"""Save test results to JSON file"""
|
||||
with open(filename, "w") as f:
|
||||
json.dump(
|
||||
{
|
||||
"timestamp": time.strftime("%Y-%m-%d %H:%M:%S"),
|
||||
"proxy_url": self.proxy_url,
|
||||
"results": self.results,
|
||||
},
|
||||
f,
|
||||
indent=2,
|
||||
)
|
||||
print(f"\nResults saved to {filename}")
|
||||
|
||||
|
||||
def main():
|
||||
@dataclass
|
||||
class Args:
|
||||
host: str = "localhost"
|
||||
port: int = 3128
|
||||
no_container: bool = False
|
||||
save_results: bool = False
|
||||
test_file: str | None = None
|
||||
list_tests: bool = False
|
||||
dev_mode: bool = False
|
||||
|
||||
def parse_args() -> Args:
|
||||
parser = argparse.ArgumentParser(description="Test SSRF Proxy Configuration")
|
||||
_ = parser.add_argument("--host", type=str, default="localhost", help="Proxy host (default: localhost)")
|
||||
_ = parser.add_argument("--port", type=int, default=3128, help="Proxy port (default: 3128)")
|
||||
_ = parser.add_argument(
|
||||
"--no-container",
|
||||
action="store_true",
|
||||
help="Don't start container (assume proxy is already running)",
|
||||
)
|
||||
_ = parser.add_argument("--save-results", action="store_true", help="Save test results to JSON file")
|
||||
_ = parser.add_argument(
|
||||
"--test-file", type=str, help="Path to YAML file containing test cases (default: test_cases.yaml)"
|
||||
)
|
||||
_ = parser.add_argument("--list-tests", action="store_true", help="List all test cases without running them")
|
||||
_ = parser.add_argument(
|
||||
"--dev-mode",
|
||||
action="store_true",
|
||||
help="Run in development mode (DISABLES all SSRF protections - DO NOT use in production!)",
|
||||
)
|
||||
|
||||
# Parse arguments - argparse.Namespace has Any-typed attributes
|
||||
# This is a known limitation of argparse in Python's type system
|
||||
namespace = parser.parse_args()
|
||||
|
||||
# Convert namespace attributes to properly typed values
|
||||
# argparse guarantees these attributes exist with the correct types
|
||||
# based on our argument definitions, but the type system cannot verify this
|
||||
return Args(
|
||||
host=str(namespace.host), # pyright: ignore[reportAny]
|
||||
port=int(namespace.port), # pyright: ignore[reportAny]
|
||||
no_container=bool(namespace.no_container), # pyright: ignore[reportAny]
|
||||
save_results=bool(namespace.save_results), # pyright: ignore[reportAny]
|
||||
test_file=namespace.test_file or None, # pyright: ignore[reportAny]
|
||||
list_tests=bool(namespace.list_tests), # pyright: ignore[reportAny]
|
||||
dev_mode=bool(namespace.dev_mode), # pyright: ignore[reportAny]
|
||||
)
|
||||
|
||||
args = parse_args()
|
||||
|
||||
tester = SSRFProxyTester(args.host, args.port, args.test_file, args.dev_mode)
|
||||
|
||||
# If --list-tests flag is set, just list the tests and exit
|
||||
if args.list_tests:
|
||||
test_cases = tester.get_test_cases()
|
||||
mode_str = " (DEVELOPMENT MODE)" if args.dev_mode else ""
|
||||
print("\n" + "=" * 50)
|
||||
print(f" Available Test Cases{mode_str}")
|
||||
print("=" * 50)
|
||||
if args.dev_mode:
|
||||
print(f"\n{Colors.RED}WARNING: Dev mode test cases expect ALL requests to be ALLOWED!{Colors.NC}")
|
||||
|
||||
# Group by category for display
|
||||
categories: dict[str, list[TestCase]] = {}
|
||||
for test in test_cases:
|
||||
if test.category not in categories:
|
||||
categories[test.category] = []
|
||||
categories[test.category].append(test)
|
||||
|
||||
for category, tests in categories.items():
|
||||
print(f"\n{Colors.YELLOW}{category}:{Colors.NC}")
|
||||
for test in tests:
|
||||
blocked_status = "BLOCK" if test.expected_blocked else "ALLOW"
|
||||
color = Colors.RED if test.expected_blocked else Colors.GREEN
|
||||
print(f" {color}[{blocked_status}]{Colors.NC} {test.name}")
|
||||
if test.description:
|
||||
print(f" {test.description}")
|
||||
print(f" URL: {test.url}")
|
||||
|
||||
print(f"\nTotal: {len(test_cases)} test cases")
|
||||
sys.exit(0)
|
||||
|
||||
try:
|
||||
# Start container unless --no-container flag is set
|
||||
if not args.no_container:
|
||||
if not tester.start_proxy_container():
|
||||
sys.exit(1)
|
||||
|
||||
# Run tests
|
||||
tester.run_all_tests()
|
||||
|
||||
# Print summary
|
||||
success = tester.print_summary()
|
||||
|
||||
# Save results if requested
|
||||
if args.save_results:
|
||||
tester.save_results()
|
||||
|
||||
# Exit with appropriate code
|
||||
sys.exit(0 if success else 1)
|
||||
|
||||
finally:
|
||||
# Cleanup
|
||||
if not args.no_container:
|
||||
print(f"\n{Colors.YELLOW}Cleaning up...{Colors.NC}")
|
||||
tester.stop_proxy_container()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@ -1,9 +1,9 @@
|
||||
import time
|
||||
import uuid
|
||||
from os import getenv
|
||||
|
||||
import pytest
|
||||
|
||||
from configs import dify_config
|
||||
from core.app.entities.app_invoke_entities import InvokeFrom
|
||||
from core.workflow.entities import GraphInitParams, GraphRuntimeState, VariablePool
|
||||
from core.workflow.enums import WorkflowNodeExecutionStatus
|
||||
@ -15,7 +15,7 @@ from core.workflow.system_variable import SystemVariable
|
||||
from models.enums import UserFrom
|
||||
from tests.integration_tests.workflow.nodes.__mock.code_executor import setup_code_executor_mock
|
||||
|
||||
CODE_MAX_STRING_LENGTH = dify_config.CODE_MAX_STRING_LENGTH
|
||||
CODE_MAX_STRING_LENGTH = int(getenv("CODE_MAX_STRING_LENGTH", "10000"))
|
||||
|
||||
|
||||
def init_code_node(code_config: dict):
|
||||
|
||||
@ -18,7 +18,6 @@ from flask.testing import FlaskClient
|
||||
from sqlalchemy import Engine, text
|
||||
from sqlalchemy.orm import Session
|
||||
from testcontainers.core.container import DockerContainer
|
||||
from testcontainers.core.network import Network
|
||||
from testcontainers.core.waiting_utils import wait_for_logs
|
||||
from testcontainers.postgres import PostgresContainer
|
||||
from testcontainers.redis import RedisContainer
|
||||
@ -42,7 +41,6 @@ class DifyTestContainers:
|
||||
|
||||
def __init__(self):
|
||||
"""Initialize container management with default configurations."""
|
||||
self.network: Network | None = None
|
||||
self.postgres: PostgresContainer | None = None
|
||||
self.redis: RedisContainer | None = None
|
||||
self.dify_sandbox: DockerContainer | None = None
|
||||
@ -64,18 +62,12 @@ class DifyTestContainers:
|
||||
|
||||
logger.info("Starting test containers for Dify integration tests...")
|
||||
|
||||
# Create Docker network for container communication
|
||||
logger.info("Creating Docker network for container communication...")
|
||||
self.network = Network()
|
||||
self.network.create()
|
||||
logger.info("Docker network created successfully with name: %s", self.network.name)
|
||||
|
||||
# Start PostgreSQL container for main application database
|
||||
# PostgreSQL is used for storing user data, workflows, and application state
|
||||
logger.info("Initializing PostgreSQL container...")
|
||||
self.postgres = PostgresContainer(
|
||||
image="postgres:14-alpine",
|
||||
).with_network(self.network)
|
||||
)
|
||||
self.postgres.start()
|
||||
db_host = self.postgres.get_container_host_ip()
|
||||
db_port = self.postgres.get_exposed_port(5432)
|
||||
@ -145,7 +137,7 @@ class DifyTestContainers:
|
||||
# Start Redis container for caching and session management
|
||||
# Redis is used for storing session data, cache entries, and temporary data
|
||||
logger.info("Initializing Redis container...")
|
||||
self.redis = RedisContainer(image="redis:6-alpine", port=6379).with_network(self.network)
|
||||
self.redis = RedisContainer(image="redis:6-alpine", port=6379)
|
||||
self.redis.start()
|
||||
redis_host = self.redis.get_container_host_ip()
|
||||
redis_port = self.redis.get_exposed_port(6379)
|
||||
@ -161,7 +153,7 @@ class DifyTestContainers:
|
||||
# Start Dify Sandbox container for code execution environment
|
||||
# Dify Sandbox provides a secure environment for executing user code
|
||||
logger.info("Initializing Dify Sandbox container...")
|
||||
self.dify_sandbox = DockerContainer(image="langgenius/dify-sandbox:latest").with_network(self.network)
|
||||
self.dify_sandbox = DockerContainer(image="langgenius/dify-sandbox:latest")
|
||||
self.dify_sandbox.with_exposed_ports(8194)
|
||||
self.dify_sandbox.env = {
|
||||
"API_KEY": "test_api_key",
|
||||
@ -181,28 +173,22 @@ class DifyTestContainers:
|
||||
# Start Dify Plugin Daemon container for plugin management
|
||||
# Dify Plugin Daemon provides plugin lifecycle management and execution
|
||||
logger.info("Initializing Dify Plugin Daemon container...")
|
||||
self.dify_plugin_daemon = DockerContainer(image="langgenius/dify-plugin-daemon:0.3.0-local").with_network(
|
||||
self.network
|
||||
)
|
||||
self.dify_plugin_daemon = DockerContainer(image="langgenius/dify-plugin-daemon:0.3.0-local")
|
||||
self.dify_plugin_daemon.with_exposed_ports(5002)
|
||||
# Get container internal network addresses
|
||||
postgres_container_name = self.postgres.get_wrapped_container().name
|
||||
redis_container_name = self.redis.get_wrapped_container().name
|
||||
|
||||
self.dify_plugin_daemon.env = {
|
||||
"DB_HOST": postgres_container_name, # Use container name for internal network communication
|
||||
"DB_PORT": "5432", # Use internal port
|
||||
"DB_HOST": db_host,
|
||||
"DB_PORT": str(db_port),
|
||||
"DB_USERNAME": self.postgres.username,
|
||||
"DB_PASSWORD": self.postgres.password,
|
||||
"DB_DATABASE": "dify_plugin",
|
||||
"REDIS_HOST": redis_container_name, # Use container name for internal network communication
|
||||
"REDIS_PORT": "6379", # Use internal port
|
||||
"REDIS_HOST": redis_host,
|
||||
"REDIS_PORT": str(redis_port),
|
||||
"REDIS_PASSWORD": "",
|
||||
"SERVER_PORT": "5002",
|
||||
"SERVER_KEY": "test_plugin_daemon_key",
|
||||
"MAX_PLUGIN_PACKAGE_SIZE": "52428800",
|
||||
"PPROF_ENABLED": "false",
|
||||
"DIFY_INNER_API_URL": f"http://{postgres_container_name}:5001",
|
||||
"DIFY_INNER_API_URL": f"http://{db_host}:5001",
|
||||
"DIFY_INNER_API_KEY": "test_inner_api_key",
|
||||
"PLUGIN_REMOTE_INSTALLING_HOST": "0.0.0.0",
|
||||
"PLUGIN_REMOTE_INSTALLING_PORT": "5003",
|
||||
@ -267,15 +253,6 @@ class DifyTestContainers:
|
||||
# Log error but don't fail the test cleanup
|
||||
logger.warning("Failed to stop container %s: %s", container, e)
|
||||
|
||||
# Stop and remove the network
|
||||
if self.network:
|
||||
try:
|
||||
logger.info("Removing Docker network...")
|
||||
self.network.remove()
|
||||
logger.info("Successfully removed Docker network")
|
||||
except Exception as e:
|
||||
logger.warning("Failed to remove Docker network: %s", e)
|
||||
|
||||
self._containers_started = False
|
||||
logger.info("All test containers stopped and cleaned up successfully")
|
||||
|
||||
|
||||
@ -3,6 +3,7 @@ from unittest.mock import MagicMock, patch
|
||||
|
||||
import pytest
|
||||
from faker import Faker
|
||||
from openai._exceptions import RateLimitError
|
||||
|
||||
from core.app.entities.app_invoke_entities import InvokeFrom
|
||||
from models.model import EndUser
|
||||
@ -483,6 +484,36 @@ class TestAppGenerateService:
|
||||
# Verify error message
|
||||
assert "Rate limit exceeded" in str(exc_info.value)
|
||||
|
||||
def test_generate_with_rate_limit_error_from_openai(
|
||||
self, db_session_with_containers, mock_external_service_dependencies
|
||||
):
|
||||
"""
|
||||
Test generation when OpenAI rate limit error occurs.
|
||||
"""
|
||||
fake = Faker()
|
||||
app, account = self._create_test_app_and_account(
|
||||
db_session_with_containers, mock_external_service_dependencies, mode="completion"
|
||||
)
|
||||
|
||||
# Setup completion generator to raise RateLimitError
|
||||
mock_response = MagicMock()
|
||||
mock_response.request = MagicMock()
|
||||
mock_external_service_dependencies["completion_generator"].return_value.generate.side_effect = RateLimitError(
|
||||
"Rate limit exceeded", response=mock_response, body=None
|
||||
)
|
||||
|
||||
# Setup test arguments
|
||||
args = {"inputs": {"query": fake.text(max_nb_chars=50)}, "response_mode": "streaming"}
|
||||
|
||||
# Execute the method under test and expect rate limit error
|
||||
with pytest.raises(InvokeRateLimitError) as exc_info:
|
||||
AppGenerateService.generate(
|
||||
app_model=app, user=account, args=args, invoke_from=InvokeFrom.SERVICE_API, streaming=True
|
||||
)
|
||||
|
||||
# Verify error message
|
||||
assert "Rate limit exceeded" in str(exc_info.value)
|
||||
|
||||
def test_generate_with_invalid_app_mode(self, db_session_with_containers, mock_external_service_dependencies):
|
||||
"""
|
||||
Test generation with invalid app mode.
|
||||
|
||||
@ -784,6 +784,133 @@ class TestCleanDatasetTask:
|
||||
print(f"Total cleanup time: {cleanup_duration:.3f} seconds")
|
||||
print(f"Average time per document: {cleanup_duration / len(documents):.3f} seconds")
|
||||
|
||||
def test_clean_dataset_task_concurrent_cleanup_scenarios(
|
||||
self, db_session_with_containers, mock_external_service_dependencies
|
||||
):
|
||||
"""
|
||||
Test dataset cleanup with concurrent cleanup scenarios and race conditions.
|
||||
|
||||
This test verifies that the task can properly:
|
||||
1. Handle multiple cleanup operations on the same dataset
|
||||
2. Prevent data corruption during concurrent access
|
||||
3. Maintain data consistency across multiple cleanup attempts
|
||||
4. Handle race conditions gracefully
|
||||
5. Ensure idempotent cleanup operations
|
||||
"""
|
||||
# Create test data
|
||||
account, tenant = self._create_test_account_and_tenant(db_session_with_containers)
|
||||
dataset = self._create_test_dataset(db_session_with_containers, account, tenant)
|
||||
document = self._create_test_document(db_session_with_containers, account, tenant, dataset)
|
||||
segment = self._create_test_segment(db_session_with_containers, account, tenant, dataset, document)
|
||||
upload_file = self._create_test_upload_file(db_session_with_containers, account, tenant)
|
||||
|
||||
# Update document with file reference
|
||||
import json
|
||||
|
||||
document.data_source_info = json.dumps({"upload_file_id": upload_file.id})
|
||||
from extensions.ext_database import db
|
||||
|
||||
db.session.commit()
|
||||
|
||||
# Save IDs for verification
|
||||
dataset_id = dataset.id
|
||||
tenant_id = tenant.id
|
||||
upload_file_id = upload_file.id
|
||||
|
||||
# Mock storage to simulate slow operations
|
||||
mock_storage = mock_external_service_dependencies["storage"]
|
||||
original_delete = mock_storage.delete
|
||||
|
||||
def slow_delete(key):
|
||||
import time
|
||||
|
||||
time.sleep(0.1) # Simulate slow storage operation
|
||||
return original_delete(key)
|
||||
|
||||
mock_storage.delete.side_effect = slow_delete
|
||||
|
||||
# Execute multiple cleanup operations concurrently
|
||||
import threading
|
||||
|
||||
cleanup_results = []
|
||||
cleanup_errors = []
|
||||
|
||||
def run_cleanup():
|
||||
try:
|
||||
clean_dataset_task(
|
||||
dataset_id=dataset_id,
|
||||
tenant_id=tenant_id,
|
||||
indexing_technique="high_quality",
|
||||
index_struct='{"type": "paragraph"}',
|
||||
collection_binding_id=str(uuid.uuid4()),
|
||||
doc_form="paragraph_index",
|
||||
)
|
||||
cleanup_results.append("success")
|
||||
except Exception as e:
|
||||
cleanup_errors.append(str(e))
|
||||
|
||||
# Start multiple cleanup threads
|
||||
threads = []
|
||||
for i in range(3):
|
||||
thread = threading.Thread(target=run_cleanup)
|
||||
threads.append(thread)
|
||||
thread.start()
|
||||
|
||||
# Wait for all threads to complete
|
||||
for thread in threads:
|
||||
thread.join()
|
||||
|
||||
# Verify results
|
||||
# Check that all documents were deleted (only once)
|
||||
remaining_documents = db.session.query(Document).filter_by(dataset_id=dataset_id).all()
|
||||
assert len(remaining_documents) == 0
|
||||
|
||||
# Check that all segments were deleted (only once)
|
||||
remaining_segments = db.session.query(DocumentSegment).filter_by(dataset_id=dataset_id).all()
|
||||
assert len(remaining_segments) == 0
|
||||
|
||||
# Check that upload file was deleted (only once)
|
||||
# Note: In concurrent scenarios, the first thread deletes documents and segments,
|
||||
# subsequent threads may not find the related data to clean up upload files
|
||||
# This demonstrates the idempotent nature of the cleanup process
|
||||
remaining_files = db.session.query(UploadFile).filter_by(id=upload_file_id).all()
|
||||
# The upload file should be deleted by the first successful cleanup operation
|
||||
# However, in concurrent scenarios, this may not always happen due to race conditions
|
||||
# This test demonstrates the idempotent nature of the cleanup process
|
||||
if len(remaining_files) > 0:
|
||||
print(f"Warning: Upload file {upload_file_id} was not deleted in concurrent scenario")
|
||||
print("This is expected behavior demonstrating the idempotent nature of cleanup")
|
||||
# We don't assert here as the behavior depends on timing and race conditions
|
||||
|
||||
# Verify that storage.delete was called (may be called multiple times in concurrent scenarios)
|
||||
# In concurrent scenarios, storage operations may be called multiple times due to race conditions
|
||||
assert mock_storage.delete.call_count > 0
|
||||
|
||||
# Verify that index processor was called (may be called multiple times in concurrent scenarios)
|
||||
mock_index_processor = mock_external_service_dependencies["index_processor"]
|
||||
assert mock_index_processor.clean.call_count > 0
|
||||
|
||||
# Check cleanup results
|
||||
assert len(cleanup_results) == 3, "All cleanup operations should complete"
|
||||
assert len(cleanup_errors) == 0, "No cleanup errors should occur"
|
||||
|
||||
# Verify idempotency by running cleanup again on the same dataset
|
||||
# This should not perform any additional operations since data is already cleaned
|
||||
clean_dataset_task(
|
||||
dataset_id=dataset_id,
|
||||
tenant_id=tenant_id,
|
||||
indexing_technique="high_quality",
|
||||
index_struct='{"type": "paragraph"}',
|
||||
collection_binding_id=str(uuid.uuid4()),
|
||||
doc_form="paragraph_index",
|
||||
)
|
||||
|
||||
# Verify that no additional storage operations were performed
|
||||
# Note: In concurrent scenarios, the exact count may vary due to race conditions
|
||||
print(f"Final storage delete calls: {mock_storage.delete.call_count}")
|
||||
print(f"Final index processor calls: {mock_index_processor.clean.call_count}")
|
||||
print("Note: Multiple calls in concurrent scenarios are expected due to race conditions")
|
||||
|
||||
def test_clean_dataset_task_storage_exception_handling(
|
||||
self, db_session_with_containers, mock_external_service_dependencies
|
||||
):
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user