bump version to 0.5.3 (#2306)

Co-authored-by: luowei <glpat-EjySCyNjWiLqAED-YmwM>
Co-authored-by: crazywoola <427733928@qq.com>
Co-authored-by: crazywoola <100913391+crazywoola@users.noreply.github.com>

CONTRIBUTING_CN.md

@@ -1,57 +1,155 @@
-# 贡献
+所以你想为 Dify 做贡献 - 这太棒了，我们迫不及待地想看到你的贡献。作为一家人员和资金有限的初创公司，我们有着雄心勃勃的目标，希望设计出最直观的工作流程来构建和管理 LLM 应用程序。社区的任何帮助都是宝贵的。
 
-感谢您对 [Dify](https://dify.ai) 的兴趣，并希望您能够做出贡献！在开始之前，请先阅读[行为准则](https://github.com/langgenius/.github/blob/main/CODE_OF_CONDUCT.md)并查看[现有问题](https://github.com/langgenius/dify/issues)。
-本文档介绍了如何设置开发环境以构建和测试 [Dify](https://dify.ai)。
+考虑到我们的现状，我们需要灵活快速地交付，但我们也希望确保像你这样的贡献者在贡献过程中获得尽可能顺畅的体验。我们为此编写了这份贡献指南，旨在让你熟悉代码库和我们与贡献者的合作方式，以便你能快速进入有趣的部分。
 
-### 安装依赖项
+这份指南，就像 Dify 本身一样，是一个不断改进的工作。如果有时它落后于实际项目，我们非常感谢你的理解，并欢迎任何反馈以供我们改进。
 
-您需要在计算机上安装和配置以下依赖项才能构建 [Dify](https://dify.ai)：
+在许可方面，请花一分钟阅读我们简短的[许可证和贡献者协议](./license)。社区还遵守[行为准则](https://github.com/langgenius/.github/blob/main/CODE_OF_CONDUCT.md)。
 
-- [Git](http://git-scm.com/)
-- [Docker](https://www.docker.com/)
-- [Docker Compose](https://docs.docker.com/compose/install/)
-- [Node.js v18.x (LTS)](http://nodejs.org)
-- [npm](https://www.npmjs.com/) 版本 8.x.x 或 [Yarn](https://yarnpkg.com/)
-- [Python](https://www.python.org/) 版本 3.10.x
+## 在开始之前
 
-## 本地开发
+[查找](https://github.com/langgenius/dify/issues?q=is:issue+is:closed)现有问题，或[创建](https://github.com/langgenius/dify/issues/new/choose)一个新问题。我们将问题分为两类：
 
-要设置一个可工作的开发环境，只需 fork 项目的 git 存储库，并使用适当的软件包管理器安装后端和前端依赖项，然后创建并运行 docker-compose。
+### 功能请求：
 
-### Fork存储库
+* 如果您要提出新的功能请求，请解释所提议的功能的目标，并尽可能提供详细的上下文。[@perzeusss](https://github.com/perzeuss)制作了一个很好的[功能请求助手](https://udify.app/chat/MK2kVSnw1gakVwMX)，可以帮助您起草需求。随时尝试一下。
 
-您需要 fork [Git 仓库](https://github.com/langgenius/dify)。
+* 如果您想从现有问题中选择一个，请在其下方留下评论表示您的意愿。
 
-### 克隆存储库
+相关方向的团队成员将参与其中。如果一切顺利，他们将批准您开始编码。在此之前，请不要开始工作，以免我们提出更改导致您的工作付诸东流。
 
-克隆您在 GitHub 上 fork 的仓库：
+根据所提议的功能所属的领域不同，您可能需要与不同的团队成员交流。以下是我们团队成员目前正在从事的各个领域的概述：
+
+  | Member                                                       | Scope                                                |
+  | ------------------------------------------------------------ | ---------------------------------------------------- |
+  | [@yeuoly](https://github.com/Yeuoly)                         | Architecting Agents                                  |
+  | [@jyong](https://github.com/JohnJyong)                       | RAG pipeline design                                  |
+  | [@GarfieldDai](https://github.com/GarfieldDai)               | Building workflow orchestrations                     |
+  | [@iamjoel](https://github.com/iamjoel) & [@zxhlyh](https://github.com/zxhlyh) | Making our frontend a breeze to use                  |
+  | [@guchenhe](https://github.com/guchenhe) & [@crazywoola](https://github.com/crazywoola) | Developer experience, points of contact for anything |
+  | [@takatost](https://github.com/takatost)                     | Overall product direction and architecture           |
+
+  How we prioritize:
+
+  | Feature Type                                                 | Priority        |
+  | ------------------------------------------------------------ | --------------- |
+  | High-Priority Features as being labeled by a team member     | High Priority   |
+  | Popular feature requests from our [community feedback board](https://feedback.dify.ai/) | Medium Priority |
+  | Non-core features and minor enhancements                     | Low Priority    |
+  | Valuable but not immediate                                   | Future-Feature  |
+
+### 其他任何事情（例如bug报告、性能优化、拼写错误更正）：
+* 立即开始编码。
+
+  How we prioritize:
+
+  | Issue Type                                                   | Priority        |
+  | ------------------------------------------------------------ | --------------- |
+  | Bugs in core functions (cannot login, applications not working, security loopholes) | Critical        |
+  | Non-critical bugs, performance boosts                        | Medium Priority |
+  | Minor fixes (typos, confusing but working UI)                | Low Priority    |
+
+
+## 安装
+
+以下是设置Dify进行开发的步骤：
+
+### 1. Fork该仓库
+
+### 2. 克隆仓库
+
+从终端克隆fork的仓库：
 
 ```
 git clone git@github.com:<github_username>/dify.git
 ```
 
-### 安装后端
+### 3. 验证依赖项
 
-要了解如何安装后端应用程序，请参阅[后端 README](api/README.md)。
+Dify 依赖以下工具和库：
 
-### 安装前端
+- [Docker](https://www.docker.com/)
+- [Docker Compose](https://docs.docker.com/compose/install/)
+- [Node.js v18.x (LTS)](http://nodejs.org)
+- [npm](https://www.npmjs.com/) version 8.x.x or [Yarn](https://yarnpkg.com/)
+- [Python](https://www.python.org/) version 3.10.x
 
-要了解如何安装前端应用程序，请参阅[前端 README](web/README.md)。
+### 4. 安装
 
-### 在浏览器中访问 Dify
+Dify由后端和前端组成。通过`cd api/`导航到后端目录，然后按照[后端README](api/README.md)进行安装。在另一个终端中，通过`cd web/`导航到前端目录，然后按照[前端README](web/README.md)进行安装。
 
-最后，您现在可以访问 [http://localhost:3000](http://localhost:3000) 在本地环境中查看 [Dify](https://dify.ai)。
+查看[安装常见问题解答](https://docs.dify.ai/getting-started/faq/install-faq)以获取常见问题列表和故障排除步骤。
 
-## 创建拉取请求
+### 5. 在浏览器中访问Dify
 
-在进行更改后，打开一个拉取请求（PR）。提交拉取请求后，Dify 团队/社区的其他人将与您一起审查它。
+为了验证您的设置，打开浏览器并访问[http://localhost:3000](http://localhost:3000)（默认或您自定义的URL和端口）。现在您应该看到Dify正在运行。
 
-如果遇到问题，比如合并冲突或不知道如何打开拉取请求，请查看 GitHub 的[拉取请求教程](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests)，了解如何解决合并冲突和其他问题。一旦您的 PR 被合并，您将自豪地被列为[贡献者表](https://github.com/langgenius/dify/graphs/contributors)中的一员。
+## 开发
 
-## 社区渠道
+如果您要添加模型提供程序，请参考[此指南](https://github.com/langgenius/dify/blob/main/api/core/model_runtime/README.md)。
 
-遇到困难了吗？有任何问题吗? 加入 [Discord Community Server](https://discord.gg/AhzKf7dNgk)，我们将为您提供帮助。
+如果您要向Agent或Workflow添加工具提供程序，请参考[此指南](./api/core/tools/README.md)。
 
-### 多语言支持
+为了帮助您快速了解您的贡献在哪个部分，以下是Dify后端和前端的简要注释大纲：
 
-需要参与贡献翻译内容，请参阅[前端多语言翻译 README](web/i18n/README_CN.md)。
+### 后端
+
+Dify的后端使用Python编写，使用[Flask](https://flask.palletsprojects.com/en/3.0.x/)框架。它使用[SQLAlchemy](https://www.sqlalchemy.org/)作为ORM，使用[Celery](https://docs.celeryq.dev/en/stable/getting-started/introduction.html)作为任务队列。授权逻辑通过Flask-login进行处理。
+
+```
+[api/]
+├── constants             // Constant settings used throughout code base.
+├── controllers           // API route definitions and request handling logic.           
+├── core                  // Core application orchestration, model integrations, and tools.
+├── docker                // Docker & containerization related configurations.
+├── events                // Event handling and processing
+├── extensions            // Extensions with 3rd party frameworks/platforms.
+├── fields                // field definitions for serialization/marshalling.
+├── libs                  // Reusable libraries and helpers.
+├── migrations            // Scripts for database migration.
+├── models                // Database models & schema definitions.
+├── services              // Specifies business logic.
+├── storage               // Private key storage.      
+├── tasks                 // Handling of async tasks and background jobs.
+└── tests
+```
+
+### 前端
+
+该网站使用基于Typescript的[Next.js](https://nextjs.org/)模板进行引导，并使用[Tailwind CSS](https://tailwindcss.com/)进行样式设计。[React-i18next](https://react.i18next.com/)用于国际化。
+
+```
+[web/]
+├── app                   // layouts, pages, and components
+│   ├── (commonLayout)    // common layout used throughout the app
+│   ├── (shareLayout)     // layouts specifically shared across token-specific sessions 
+│   ├── activate          // activate page
+│   ├── components        // shared by pages and layouts
+│   ├── install           // install page
+│   ├── signin            // signin page
+│   └── styles            // globally shared styles
+├── assets                // Static assets
+├── bin                   // scripts ran at build step
+├── config                // adjustable settings and options 
+├── context               // shared contexts used by different portions of the app
+├── dictionaries          // Language-specific translate files 
+├── docker                // container configurations
+├── hooks                 // Reusable hooks
+├── i18n                  // Internationalization configuration
+├── models                // describes data models & shapes of API responses
+├── public                // meta assets like favicon
+├── service               // specifies shapes of API actions
+├── test                  
+├── types                 // descriptions of function params and return values
+└── utils                 // Shared utility functions
+```
+
+## 提交你的 PR
+
+最后，是时候向我们的仓库提交一个拉取请求（PR）了。对于重要的功能，我们首先将它们合并到 `deploy/dev` 分支进行测试，然后再合并到 `main` 分支。如果你遇到合并冲突或者不知道如何提交拉取请求的问题，请查看 [GitHub 的拉取请求教程](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests)。
+
+就是这样！一旦你的 PR 被合并，你将成为我们 [README](https://github.com/langgenius/dify/blob/main/README.md) 中的贡献者。
+
+## 获取帮助
+
+如果你在贡献过程中遇到困难或者有任何问题，可以通过相关的 GitHub 问题提出你的疑问，或者加入我们的 [Discord](https://discord.gg/AhzKf7dNgk) 进行快速交流。

CONTRIBUTING_JA.md

@@ -1,55 +0,0 @@
-# コントリビュート
-
-[Dify](https://dify.ai) に興味を持ち、貢献したいと思うようになったことに感謝します！始める前に、
-[行動規範](https://github.com/langgenius/.github/blob/main/CODE_OF_CONDUCT.md)を読み、
-[既存の問題](https://github.com/langgenius/langgenius-gateway/issues)をチェックしてください。
-本ドキュメントは、[Dify](https://dify.ai) をビルドしてテストするための開発環境の構築方法を説明するものです。
-
-### 依存関係のインストール
-
-[Dify](https://dify.ai)をビルドするには、お使いのマシンに以下の依存関係をインストールし、設定する必要があります:
-
-- [Git](http://git-scm.com/)
-- [Docker](https://www.docker.com/)
-- [Docker Compose](https://docs.docker.com/compose/install/)
-- [Node.js v18.x (LTS)](http://nodejs.org)
-- [npm](https://www.npmjs.com/) バージョン 8.x.x もしくは [Yarn](https://yarnpkg.com/)
-- [Python](https://www.python.org/) バージョン 3.10.x
-
-## ローカル開発
-
-開発環境を構築するには、プロジェクトの git リポジトリをフォークし、適切なパッケージマネージャを使用してバックエンドとフロントエンドの依存関係をインストールし、docker-compose スタックを実行するように作成します。
-
-### リポジトリのフォーク
-
-[リポジトリ](https://github.com/langgenius/dify) をフォークする必要があります。
-
-### リポジトリのクローン
-
-GitHub でフォークしたリポジトリのクローンを作成する:
-
-```
-git clone git@github.com:<github_username>/dify.git
-```
-
-### バックエンドのインストール
-
-バックエンドアプリケーションのインストール方法については、[Backend README](api/README.md) を参照してください。
-
-### フロントエンドのインストール
-
-フロントエンドアプリケーションのインストール方法については、[Frontend README](web/README.md) を参照してください。
-
-### ブラウザで dify にアクセス
-
-[Dify](https://dify.ai) をローカル環境で見ることができるようになりました [http://localhost:3000](http://localhost:3000)。
-
-## プルリクエストの作成
-
-変更後、プルリクエスト (PR) をオープンしてください。プルリクエストを提出すると、Dify チーム/コミュニティの他の人があなたと一緒にそれをレビューします。
-
-マージコンフリクトなどの問題が発生したり、プルリクエストの開き方がわからなくなったりしませんでしたか？ [GitHub's pull request tutorial](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests) で、マージコンフリクトやその他の問題を解決する方法をチェックしてみてください。あなたの PR がマージされると、[コントリビュータチャート](https://github.com/langgenius/langgenius-gateway/graphs/contributors)にコントリビュータとして誇らしげに掲載されます。
-
-## コミュニティチャンネル
-
-お困りですか？何か質問がありますか？ [Discord Community サーバ](https://discord.gg/j3XRWSPBf7) に参加してください。私たちがお手伝いします！

README.md

@@ -21,6 +21,11 @@
         <img alt="Docker Pulls" src="https://img.shields.io/docker/pulls/langgenius/dify-web"></a>
 </p>
 
+<p align="center">
+   <a href="https://dify.ai/blog/dify-ai-unveils-ai-agent-creating-gpts-and-assistants-with-various-llms" target="_blank">
+   Dify.AI Unveils AI Agent: Creating GPTs and Assistants with Various LLMs
+  </a>
+</p>
 
 **Dify** is an LLM application development platform that has helped built over **100,000** applications. It integrates BaaS and LLMOps, covering the essential tech stack for building generative AI-native applications, including a built-in RAG engine. Dify allows you to **deploy your own version of Assistants API and GPTs, based on any LLMs.**
 
@@ -55,7 +60,8 @@ You can try out [Dify.AI Cloud](https://dify.ai) now. It provides all the capabi
 
 **3. RAG Engine**: Includes various RAG capabilities based on full-text indexing or vector database embeddings, allowing direct upload of PDFs, TXTs, and other text formats.
 
-**4. Agents**: A Function Calling based Agent framework that allows users to configure what they see is what they get. Dify includes basic plugin capabilities like Google Search.
+**4. AI Agent**: Based on Function Calling and ReAct, the Agent inference framework allows users to customize tools, what you see is what you get. Dify provides more than a dozen built-in tool calling capabilities, such as Google Search, DELL·E, Stable Diffusion, WolframAlpha, etc.
+
 
 **5. Continuous Operations**: Monitor and analyze application logs and performance, continuously improving Prompts, datasets, or models using production data.
 

README_CN.md

@@ -21,6 +21,12 @@
         <img alt="Docker Pulls" src="https://img.shields.io/docker/pulls/langgenius/dify-web"></a>
 </p>
 
+<p align="center">
+   <a href="https://mp.weixin.qq.com/s/TnyfIuH-tPi9o1KNjwVArw" target="_blank">
+   Dify 发布 AI Agent 能力：基于不同的大型语言模型构建 GPTs 和 Assistants
+  </a>
+</p>
+
 Dify 是一个 LLM 应用开发平台，已经有超过 10 万个应用基于 Dify.AI 构建。它融合了 Backend as Service 和 LLMOps 的理念，涵盖了构建生成式 AI 原生应用所需的核心技术栈，包括一个内置 RAG 引擎。使用 Dify，你可以基于任何模型自部署类似 Assistants API 和 GPTs 的能力。
 
 ![](./images/demo.png)
@@ -53,7 +59,7 @@ Dify 具有模型中立性，相较 LangChain 等硬编码开发库 Dify 是一
 
 **3. RAG引擎**：包括各种基于全文索引或向量数据库嵌入的 RAG 能力，允许直接上传 PDF、TXT 等各种文本格式。
 
-**4. Agent**：基于函数调用的 Agent框架，允许用户自定义配置，所见即所得。Dify 提供了基本的插件能力，如谷歌搜索。
+**4. AI Agent**：基于 Function Calling 和 ReAct 的 Agent 推理框架，允许用户自定义工具，所见即所得。Dify 提供了十多种内置工具调用能力，如谷歌搜索、DELL·E、Stable Diffusion、WolframAlpha 等。
 
 **5. 持续运营**：监控和分析应用日志和性能，使用生产数据持续改进 Prompt、数据集或模型。
 

README_ES.md

@@ -21,6 +21,12 @@
         <img alt="Docker Pulls" src="https://img.shields.io/docker/pulls/langgenius/dify-web"></a>
 </p>
 
+<p align="center">
+   <a href="https://dify.ai/blog/dify-ai-unveils-ai-agent-creating-gpts-and-assistants-with-various-llms" target="_blank">
+   Dify.AI Unveils AI Agent: Creating GPTs and Assistants with Various LLMs
+  </a>
+</p>
+
 **Dify** es una plataforma de desarrollo de aplicaciones para modelos de lenguaje de gran tamaño (LLM) que ya ha visto la creación de más de **100,000** aplicaciones basadas en Dify.AI. Integra los conceptos de Backend como Servicio y LLMOps, cubriendo el conjunto de tecnologías esenciales requerido para construir aplicaciones nativas de inteligencia artificial generativa, incluyendo un motor RAG incorporado. Con Dify, **puedes auto-desplegar capacidades similares a las de Assistants API y GPTs basadas en cualquier LLM.**
 
 ![](./images/demo.png)
@@ -52,7 +58,7 @@ Dify se caracteriza por su neutralidad de modelo y es un conjunto tecnológico c
 
 **3. Motor RAG**: Incluye varias capacidades RAG basadas en indexación de texto completo o incrustaciones de base de datos vectoriales, permitiendo la carga directa de PDFs, TXTs y otros formatos de texto.
 
-**4. Agentes**: Un marco de Agentes basado en Llamadas de Función que permite a los usuarios configurar lo que ven es lo que obtienen. Dify incluye capacidades básicas de plugins como la Búsqueda de Google.
+**4. Agente de IA**: Basado en la llamada de funciones y ReAct, el marco de inferencia del Agente permite a los usuarios personalizar las herramientas, lo que ves es lo que obtienes. Dify proporciona más de una docena de capacidades de llamada de herramientas incorporadas, como Búsqueda de Google, DELL·E, Difusión Estable, WolframAlpha, etc.
 
 **5. Operaciones Continuas**: Monitorear y analizar registros de aplicaciones y rendimiento, mejorando continuamente Prompts, conjuntos de datos o modelos usando datos de producción.
 

README_FR.md

@@ -21,6 +21,13 @@
         <img alt="Docker Pulls" src="https://img.shields.io/docker/pulls/langgenius/dify-web"></a>
 </p>
 
+<p align="center">
+   <a href="https://dify.ai/blog/dify-ai-unveils-ai-agent-creating-gpts-and-assistants-with-various-llms" target="_blank">
+   Dify.AI Unveils AI Agent: Creating GPTs and Assistants with Various LLMs
+  </a>
+</p>
+
+
 **Dify** est une plateforme de développement d'applications LLM qui a déjà vu plus de **100,000** applications construites sur Dify.AI. Elle intègre les concepts de Backend as a Service et LLMOps, couvrant la pile technologique de base requise pour construire des applications natives d'IA générative, y compris un moteur RAG intégré. Avec Dify, **vous pouvez auto-déployer des capacités similaires aux API Assistants et GPT basées sur n'importe quels LLM.**
 
 ![](./images/demo.png)
@@ -52,7 +59,7 @@ Dify présente une neutralité de modèle et est une pile technologique complèt
 
 **3\. Moteur RAG**: Comprend diverses capacités RAG basées sur l'indexation de texte intégral ou les embeddings de base de données vectorielles, permettant le chargement direct de PDF, TXT et autres formats de texte.
 
-**4\. Agents**: Un framework d'agents basé sur l'appel de fonctions qui permet aux utilisateurs de configurer ce qu'ils voient est ce qu'ils obtiennent. Dify comprend des capacités de plug-in de base comme Google Search.
+**4\. AI Agent**: Basé sur l'appel de fonction et ReAct, le framework d'inférence de l'Agent permet aux utilisateurs de personnaliser les outils, ce que vous voyez est ce que vous obtenez. Dify propose plus d'une douzaine de capacités d'appel d'outils intégrées, telles que la recherche Google, DELL·E, Diffusion Stable, WolframAlpha, etc.
 
 **5\. Opérations continues**: Surveillez et analysez les journaux et les performances des applications, améliorez en continu les invites, les datasets ou les modèles à l'aide de données de production.  
 

README_JA.md

@@ -21,6 +21,13 @@
         <img alt="Docker Pulls" src="https://img.shields.io/docker/pulls/langgenius/dify-web"></a>
 </p>
 
+<p align="center">
+   <a href="https://dify.ai/blog/dify-ai-unveils-ai-agent-creating-gpts-and-assistants-with-various-llms" target="_blank">
+   Dify.AI Unveils AI Agent: Creating GPTs and Assistants with Various LLMs
+  </a>
+</p>
+
+
 "Difyは、既にDify.AI上で10万以上のアプリケーションが構築されているLLMアプリケーション開発プラットフォームです。バックエンド・アズ・ア・サービスとLLMOpsの概念を統合し、組み込みのRAGエンジンを含む、生成AIネイティブアプリケーションを構築するためのコアテックスタックをカバーしています。Difyを使用すると、どのLLMに基づいても、Assistants APIやGPTのような機能を自己デプロイすることができます。"
 
 Please note that translating complex technical terms can sometimes result in slight variations in meaning due to differences in language nuances.
@@ -54,7 +61,7 @@ Difyはモデルニュートラルであり、LangChainのようなハードコ
 
 **3\. RAGエンジン**: フルテキストインデックスまたはベクトルデータベース埋め込みに基づくさまざまなRAG機能を含み、PDF、TXT、その他のテキストフォーマットの直接アップロードを可能にします。
 
-**4\. エージェント**: ユーザーが sees what they get を設定できる関数呼び出しベースのエージェントフレームワーク。 Difyには、Google検索などの基本的なプラグイン機能が含まれています。
+**4. AIエージェント**: 関数呼び出しとReActに基づくAgent推論フレームワークにより、ユーザーはツールをカスタマイズすることができます。Difyは、Google検索、DELL·E、Stable Diffusion、WolframAlphaなど、十数種類の組み込みツール呼び出し機能を提供しています。
 
 **5\. 継続的運用**: アプリケーションログとパフォーマンスを監視および分析し、運用データを使用してプロンプト、データセット、またはモデルを継続的に改善します。
 

README_KL.md

@@ -52,7 +52,7 @@ Dify Daq rIn neutrality 'ej Hoch, LangChain tInHar HubwI'. maH Daqbe'law' Qawqar
 
 **3. RAG Engine**: RAG vaD tIqpu' lo'taH indexing qor neH vector database wa' embeddings wIj, PDFs, TXTs, 'ej ghojmoHmoH HIq qorlIj je upload.
 
-**4. jenSuvpu'**: jenbe' SuDqang naQ moDwu' jenSuvpu' porgh cha'logh choHvam. Dify Google Search Hur vItlhutlh plugin choH.
+**4. AI Agent**: Function Calling 'ej ReAct Daq Hurmey, Agent inference framework Hoch users customize tools, vaj 'oH QaQ. Dify Hoch loS ghaH 'ej wa'vatlh built-in tool calling capabilities, Google Search, DELL·E, Stable Diffusion, WolframAlpha, 'ej.
 
 **5. QaS muDHa'wI': cha'logh wa' pIq mI' logs 'ej quv yIn, vItlhutlh tIq 'e'wIj lo'taHmoHmoH Prompts, vItlhutlh, Hurmey ghaH production data jatlh.
 

api/Dockerfile

@@ -1,18 +1,20 @@
-# packages install stage
-FROM python:3.10-slim AS base
+# base image
+FROM python:3.10-slim-bookworm AS base
 
 LABEL maintainer="takatost@gmail.com"
 
+# install packages
+FROM base as packages
+
 RUN apt-get update \
-    && apt-get install -y --no-install-recommends gcc g++ python3-dev libc-dev libffi-dev
+    && apt-get install -y --no-install-recommends gcc g++ libc-dev libffi-dev libgmp-dev libmpfr-dev libmpc-dev
 
 COPY requirements.txt /requirements.txt
 
 RUN pip install --prefix=/pkg -r requirements.txt
 
-# build stage
-FROM python:3.10-slim AS builder
-
+# production stage
+FROM base AS production
 
 ENV FLASK_APP app.py
 ENV EDITION SELF_HOSTED
@@ -30,11 +32,11 @@ ENV TZ UTC
 WORKDIR /app/api
 
 RUN apt-get update \
-    && apt-get install -y --no-install-recommends bash curl wget vim nodejs ffmpeg \
+    && apt-get install -y --no-install-recommends curl wget vim nodejs ffmpeg libgmp-dev libmpfr-dev libmpc-dev \
     && apt-get autoremove \
     && rm -rf /var/lib/apt/lists/*
 
-COPY --from=base /pkg /usr/local
+COPY --from=packages /pkg /usr/local
 COPY . /app/api/
 
 COPY docker/entrypoint.sh /entrypoint.sh

api/app.py

@@ -30,7 +30,7 @@ from flask import Flask, Response, request
 from flask_cors import CORS
 from libs.passport import PassportService
 # DO NOT REMOVE BELOW
-from models import account, dataset, model, source, task, tool, web, tools
+from models import account, dataset, model, source, task, tool, tools, web
 from services.account_service import AccountService
 
 # DO NOT REMOVE ABOVE

api/commands.py

@@ -1,32 +1,20 @@
 import base64
-import datetime
 import json
-import math
-import random
 import secrets
-import string
-import threading
-import time
-import uuid
 
 import click
-import qdrant_client
-from constants.languages import user_input_form_template
 from core.embedding.cached_embedding import CacheEmbedding
-from core.index.index import IndexBuilder
 from core.model_manager import ModelManager
 from core.model_runtime.entities.model_entities import ModelType
 from extensions.ext_database import db
-from flask import Flask, current_app
+from flask import current_app
 from libs.helper import email as email_validate
 from libs.password import hash_password, password_pattern, valid_password
 from libs.rsa import generate_key_pair
-from models.account import InvitationCode, Tenant, TenantAccountJoin
-from models.dataset import Dataset, DatasetCollectionBinding, DatasetQuery, Document
-from models.model import Account, App, AppModelConfig, Message, MessageAnnotation, InstalledApp
-from models.provider import Provider, ProviderModel, ProviderQuotaType, ProviderType
-from qdrant_client.http.models import TextIndexParams, TextIndexType, TokenizerType
-from tqdm import tqdm
+from models.account import Tenant
+from models.dataset import Dataset
+from models.model import Account
+from models.provider import Provider, ProviderModel
 from werkzeug.exceptions import NotFound
 
 
@@ -35,15 +23,22 @@ from werkzeug.exceptions import NotFound
 @click.option('--new-password', prompt=True, help='the new password.')
 @click.option('--password-confirm', prompt=True, help='the new password confirm.')
 def reset_password(email, new_password, password_confirm):
+    """
+    Reset password of owner account
+    Only available in SELF_HOSTED mode
+    """
     if str(new_password).strip() != str(password_confirm).strip():
         click.echo(click.style('sorry. The two passwords do not match.', fg='red'))
         return
+
     account = db.session.query(Account). \
         filter(Account.email == email). \
         one_or_none()
+
     if not account:
         click.echo(click.style('sorry. the account: [{}] not exist .'.format(email), fg='red'))
         return
+
     try:
         valid_password(new_password)
     except:
@@ -69,15 +64,22 @@ def reset_password(email, new_password, password_confirm):
 @click.option('--new-email', prompt=True, help='the new email.')
 @click.option('--email-confirm', prompt=True, help='the new email confirm.')
 def reset_email(email, new_email, email_confirm):
+    """
+    Replace account email
+    :return:
+    """
     if str(new_email).strip() != str(email_confirm).strip():
         click.echo(click.style('Sorry, new email and confirm email do not match.', fg='red'))
         return
+
     account = db.session.query(Account). \
         filter(Account.email == email). \
         one_or_none()
+
     if not account:
         click.echo(click.style('sorry. the account: [{}] not exist .'.format(email), fg='red'))
         return
+
     try:
         email_validate(new_email)
     except:
@@ -97,6 +99,11 @@ def reset_email(email, new_email, email_confirm):
 @click.confirmation_option(prompt=click.style('Are you sure you want to reset encrypt key pair?'
                                               ' this operation cannot be rolled back!', fg='red'))
 def reset_encrypt_key_pair():
+    """
+    Reset the encrypted key pair of workspace for encrypt LLM credentials.
+    After the reset, all LLM credentials will become invalid, requiring re-entry.
+    Only support SELF_HOSTED mode.
+    """
     if current_app.config['EDITION'] != 'SELF_HOSTED':
         click.echo(click.style('Sorry, only support SELF_HOSTED mode.', fg='red'))
         return
@@ -116,201 +123,11 @@ def reset_encrypt_key_pair():
                            'the asymmetric key pair of workspace {} has been reset.'.format(tenant.id), fg='green'))
 
 
-@click.command('generate-invitation-codes', help='Generate invitation codes.')
-@click.option('--batch', help='The batch of invitation codes.')
-@click.option('--count', prompt=True, help='Invitation codes count.')
-def generate_invitation_codes(batch, count):
-    if not batch:
-        now = datetime.datetime.now()
-        batch = now.strftime('%Y%m%d%H%M%S')
-
-    if not count or int(count) <= 0:
-        click.echo(click.style('sorry. the count must be greater than 0.', fg='red'))
-        return
-
-    count = int(count)
-
-    click.echo('Start generate {} invitation codes for batch {}.'.format(count, batch))
-
-    codes = ''
-    for i in range(count):
-        code = generate_invitation_code()
-        invitation_code = InvitationCode(
-            code=code,
-            batch=batch
-        )
-        db.session.add(invitation_code)
-        click.echo(code)
-
-        codes += code + "\n"
-    db.session.commit()
-
-    filename = 'storage/invitation-codes-{}.txt'.format(batch)
-
-    with open(filename, 'w') as f:
-        f.write(codes)
-
-    click.echo(click.style(
-        'Congratulations! Generated {} invitation codes for batch {} and saved to the file \'{}\''.format(count, batch,
-                                                                                                          filename),
-        fg='green'))
-
-
-def generate_invitation_code():
-    code = generate_upper_string()
-    while db.session.query(InvitationCode).filter(InvitationCode.code == code).count() > 0:
-        code = generate_upper_string()
-
-    return code
-
-
-def generate_upper_string():
-    letters_digits = string.ascii_uppercase + string.digits
-    result = ""
-    for i in range(8):
-        result += random.choice(letters_digits)
-
-    return result
-
-
-@click.command('recreate-all-dataset-indexes', help='Recreate all dataset indexes.')
-def recreate_all_dataset_indexes():
-    click.echo(click.style('Start recreate all dataset indexes.', fg='green'))
-    recreate_count = 0
-
-    page = 1
-    while True:
-        try:
-            datasets = db.session.query(Dataset).filter(Dataset.indexing_technique == 'high_quality') \
-                .order_by(Dataset.created_at.desc()).paginate(page=page, per_page=50)
-        except NotFound:
-            break
-
-        page += 1
-        for dataset in datasets:
-            try:
-                click.echo('Recreating dataset index: {}'.format(dataset.id))
-                index = IndexBuilder.get_index(dataset, 'high_quality')
-                if index and index._is_origin():
-                    index.recreate_dataset(dataset)
-                    recreate_count += 1
-                else:
-                    click.echo('passed.')
-            except Exception as e:
-                click.echo(
-                    click.style('Recreate dataset index error: {} {}'.format(e.__class__.__name__, str(e)), fg='red'))
-                continue
-
-    click.echo(click.style('Congratulations! Recreate {} dataset indexes.'.format(recreate_count), fg='green'))
-
-
-@click.command('clean-unused-dataset-indexes', help='Clean unused dataset indexes.')
-def clean_unused_dataset_indexes():
-    click.echo(click.style('Start clean unused dataset indexes.', fg='green'))
-    clean_days = int(current_app.config.get('CLEAN_DAY_SETTING'))
-    start_at = time.perf_counter()
-    thirty_days_ago = datetime.datetime.now() - datetime.timedelta(days=clean_days)
-    page = 1
-    while True:
-        try:
-            datasets = db.session.query(Dataset).filter(Dataset.created_at < thirty_days_ago) \
-                .order_by(Dataset.created_at.desc()).paginate(page=page, per_page=50)
-        except NotFound:
-            break
-        page += 1
-        for dataset in datasets:
-            dataset_query = db.session.query(DatasetQuery).filter(
-                DatasetQuery.created_at > thirty_days_ago,
-                DatasetQuery.dataset_id == dataset.id
-            ).all()
-            if not dataset_query or len(dataset_query) == 0:
-                documents = db.session.query(Document).filter(
-                    Document.dataset_id == dataset.id,
-                    Document.indexing_status == 'completed',
-                    Document.enabled == True,
-                    Document.archived == False,
-                    Document.updated_at > thirty_days_ago
-                ).all()
-                if not documents or len(documents) == 0:
-                    try:
-                        # remove index
-                        vector_index = IndexBuilder.get_index(dataset, 'high_quality')
-                        kw_index = IndexBuilder.get_index(dataset, 'economy')
-                        # delete from vector index
-                        if vector_index:
-                            if dataset.collection_binding_id:
-                                vector_index.delete_by_group_id(dataset.id)
-                            else:
-                                if dataset.collection_binding_id:
-                                    vector_index.delete_by_group_id(dataset.id)
-                                else:
-                                    vector_index.delete()
-                        kw_index.delete()
-                        # update document
-                        update_params = {
-                            Document.enabled: False
-                        }
-
-                        Document.query.filter_by(dataset_id=dataset.id).update(update_params)
-                        db.session.commit()
-                        click.echo(click.style('Cleaned unused dataset {} from db success!'.format(dataset.id),
-                                               fg='green'))
-                    except Exception as e:
-                        click.echo(
-                            click.style('clean dataset index error: {} {}'.format(e.__class__.__name__, str(e)),
-                                        fg='red'))
-    end_at = time.perf_counter()
-    click.echo(click.style('Cleaned unused dataset from db success latency: {}'.format(end_at - start_at), fg='green'))
-
-
-@click.command('sync-anthropic-hosted-providers', help='Sync anthropic hosted providers.')
-def sync_anthropic_hosted_providers():
-    if not hosted_model_providers.anthropic:
-        click.echo(click.style('Anthropic hosted provider is not configured.', fg='red'))
-        return
-
-    click.echo(click.style('Start sync anthropic hosted providers.', fg='green'))
-    count = 0
-
-    new_quota_limit = hosted_model_providers.anthropic.quota_limit
-
-    page = 1
-    while True:
-        try:
-            providers = db.session.query(Provider).filter(
-                Provider.provider_name == 'anthropic',
-                Provider.provider_type == ProviderType.SYSTEM.value,
-                Provider.quota_type == ProviderQuotaType.TRIAL.value,
-                Provider.quota_limit != new_quota_limit
-            ).order_by(Provider.created_at.desc()).paginate(page=page, per_page=100)
-        except NotFound:
-            break
-
-        page += 1
-        for provider in providers:
-            try:
-                click.echo('Syncing tenant anthropic hosted provider: {}, origin: limit {}, used {}'
-                           .format(provider.tenant_id, provider.quota_limit, provider.quota_used))
-                original_quota_limit = provider.quota_limit
-                division = math.ceil(new_quota_limit / 1000)
-
-                provider.quota_limit = new_quota_limit if original_quota_limit == 1000 \
-                    else original_quota_limit * division
-                provider.quota_used = division * provider.quota_used
-                db.session.commit()
-
-                count += 1
-            except Exception as e:
-                click.echo(click.style(
-                    'Sync tenant anthropic hosted provider error: {} {}'.format(e.__class__.__name__, str(e)),
-                    fg='red'))
-                continue
-
-    click.echo(click.style('Congratulations! Synced {} anthropic hosted providers.'.format(count), fg='green'))
-
-
 @click.command('create-qdrant-indexes', help='Create qdrant indexes.')
 def create_qdrant_indexes():
+    """
+    Migrate other vector database datas to Qdrant.
+    """
     click.echo(click.style('Start create qdrant indexes.', fg='green'))
     create_count = 0
 
@@ -339,26 +156,7 @@ def create_qdrant_indexes():
 
                             )
                         except Exception:
-                            try:
-                                embedding_model = model_manager.get_default_model_instance(
-                                    tenant_id=dataset.tenant_id,
-                                    model_type=ModelType.TEXT_EMBEDDING,
-                                )
-                                dataset.embedding_model = embedding_model.model
-                                dataset.embedding_model_provider = embedding_model.provider
-                            except Exception:
-
-                                provider = Provider(
-                                    id='provider_id',
-                                    tenant_id=dataset.tenant_id,
-                                    provider_name='openai',
-                                    provider_type=ProviderType.SYSTEM.value,
-                                    encrypted_config=json.dumps({'openai_api_key': 'TEST'}),
-                                    is_valid=True,
-                                )
-                                model_provider = OpenAIProvider(provider=provider)
-                                embedding_model = OpenAIEmbedding(name="text-embedding-ada-002",
-                                                                  model_provider=model_provider)
+                            continue
                         embeddings = CacheEmbedding(embedding_model)
 
                         from core.index.vector_index.qdrant_vector_index import QdrantConfig, QdrantVectorIndex
@@ -393,380 +191,8 @@ def create_qdrant_indexes():
     click.echo(click.style('Congratulations! Create {} dataset indexes.'.format(create_count), fg='green'))
 
 
-@click.command('update-qdrant-indexes', help='Update qdrant indexes.')
-def update_qdrant_indexes():
-    click.echo(click.style('Start Update qdrant indexes.', fg='green'))
-    create_count = 0
-
-    page = 1
-    while True:
-        try:
-            datasets = db.session.query(Dataset).filter(Dataset.indexing_technique == 'high_quality') \
-                .order_by(Dataset.created_at.desc()).paginate(page=page, per_page=50)
-        except NotFound:
-            break
-
-        page += 1
-        for dataset in datasets:
-            if dataset.index_struct_dict:
-                if dataset.index_struct_dict['type'] != 'qdrant':
-                    try:
-                        click.echo('Update dataset qdrant index: {}'.format(dataset.id))
-                        try:
-                            embedding_model = ModelFactory.get_embedding_model(
-                                tenant_id=dataset.tenant_id,
-                                model_provider_name=dataset.embedding_model_provider,
-                                model_name=dataset.embedding_model
-                            )
-                        except Exception:
-                            provider = Provider(
-                                id='provider_id',
-                                tenant_id=dataset.tenant_id,
-                                provider_name='openai',
-                                provider_type=ProviderType.CUSTOM.value,
-                                encrypted_config=json.dumps({'openai_api_key': 'TEST'}),
-                                is_valid=True,
-                            )
-                            model_provider = OpenAIProvider(provider=provider)
-                            embedding_model = OpenAIEmbedding(name="text-embedding-ada-002",
-                                                              model_provider=model_provider)
-                        embeddings = CacheEmbedding(embedding_model)
-
-                        from core.index.vector_index.qdrant_vector_index import QdrantConfig, QdrantVectorIndex
-
-                        index = QdrantVectorIndex(
-                            dataset=dataset,
-                            config=QdrantConfig(
-                                endpoint=current_app.config.get('QDRANT_URL'),
-                                api_key=current_app.config.get('QDRANT_API_KEY'),
-                                root_path=current_app.root_path
-                            ),
-                            embeddings=embeddings
-                        )
-                        if index:
-                            index.update_qdrant_dataset(dataset)
-                            create_count += 1
-                        else:
-                            click.echo('passed.')
-                    except Exception as e:
-                        click.echo(
-                            click.style('Create dataset index error: {} {}'.format(e.__class__.__name__, str(e)),
-                                        fg='red'))
-                        continue
-
-    click.echo(click.style('Congratulations! Update {} dataset indexes.'.format(create_count), fg='green'))
-
-
-@click.command('normalization-collections', help='restore all collections in one')
-def normalization_collections():
-    click.echo(click.style('Start normalization collections.', fg='green'))
-    normalization_count = []
-    page = 1
-    while True:
-        try:
-            datasets = db.session.query(Dataset).filter(Dataset.indexing_technique == 'high_quality') \
-                .order_by(Dataset.created_at.desc()).paginate(page=page, per_page=100)
-        except NotFound:
-            break
-        datasets_result = datasets.items
-        page += 1
-        for i in range(0, len(datasets_result), 5):
-            threads = []
-            sub_datasets = datasets_result[i:i + 5]
-            for dataset in sub_datasets:
-                document_format_thread = threading.Thread(target=deal_dataset_vector, kwargs={
-                    'flask_app': current_app._get_current_object(),
-                    'dataset': dataset,
-                    'normalization_count': normalization_count
-                })
-                threads.append(document_format_thread)
-                document_format_thread.start()
-            for thread in threads:
-                thread.join()
-
-    click.echo(click.style('Congratulations! restore {} dataset indexes.'.format(len(normalization_count)), fg='green'))
-
-
-@click.command('add-qdrant-full-text-index', help='add qdrant full text index')
-def add_qdrant_full_text_index():
-    click.echo(click.style('Start add full text index.', fg='green'))
-    binds = db.session.query(DatasetCollectionBinding).all()
-    if binds and current_app.config['VECTOR_STORE'] == 'qdrant':
-        qdrant_url = current_app.config['QDRANT_URL']
-        qdrant_api_key = current_app.config['QDRANT_API_KEY']
-        client = qdrant_client.QdrantClient(
-            qdrant_url,
-            api_key=qdrant_api_key,  # For Qdrant Cloud, None for local instance
-        )
-        for bind in binds:
-            try:
-                text_index_params = TextIndexParams(
-                    type=TextIndexType.TEXT,
-                    tokenizer=TokenizerType.MULTILINGUAL,
-                    min_token_len=2,
-                    max_token_len=20,
-                    lowercase=True
-                )
-                client.create_payload_index(bind.collection_name, 'page_content',
-                                            field_schema=text_index_params)
-            except Exception as e:
-                click.echo(
-                    click.style('Create full text index error: {} {}'.format(e.__class__.__name__, str(e)),
-                                fg='red'))
-            click.echo(
-                click.style(
-                    'Congratulations! add collection {} full text index successful.'.format(bind.collection_name),
-                    fg='green'))
-
-
-def deal_dataset_vector(flask_app: Flask, dataset: Dataset, normalization_count: list):
-    with flask_app.app_context():
-        try:
-            click.echo('restore dataset index: {}'.format(dataset.id))
-            try:
-                embedding_model = ModelFactory.get_embedding_model(
-                    tenant_id=dataset.tenant_id,
-                    model_provider_name=dataset.embedding_model_provider,
-                    model_name=dataset.embedding_model
-                )
-            except Exception:
-                provider = Provider(
-                    id='provider_id',
-                    tenant_id=dataset.tenant_id,
-                    provider_name='openai',
-                    provider_type=ProviderType.CUSTOM.value,
-                    encrypted_config=json.dumps({'openai_api_key': 'TEST'}),
-                    is_valid=True,
-                )
-                model_provider = OpenAIProvider(provider=provider)
-                embedding_model = OpenAIEmbedding(name="text-embedding-ada-002",
-                                                  model_provider=model_provider)
-            embeddings = CacheEmbedding(embedding_model)
-            dataset_collection_binding = db.session.query(DatasetCollectionBinding). \
-                filter(DatasetCollectionBinding.provider_name == embedding_model.model_provider.provider_name,
-                       DatasetCollectionBinding.model_name == embedding_model.name). \
-                order_by(DatasetCollectionBinding.created_at). \
-                first()
-
-            if not dataset_collection_binding:
-                dataset_collection_binding = DatasetCollectionBinding(
-                    provider_name=embedding_model.model_provider.provider_name,
-                    model_name=embedding_model.name,
-                    collection_name="Vector_index_" + str(uuid.uuid4()).replace("-", "_") + '_Node'
-                )
-                db.session.add(dataset_collection_binding)
-                db.session.commit()
-
-            from core.index.vector_index.qdrant_vector_index import QdrantConfig, QdrantVectorIndex
-
-            index = QdrantVectorIndex(
-                dataset=dataset,
-                config=QdrantConfig(
-                    endpoint=current_app.config.get('QDRANT_URL'),
-                    api_key=current_app.config.get('QDRANT_API_KEY'),
-                    root_path=current_app.root_path
-                ),
-                embeddings=embeddings
-            )
-            if index:
-                # index.delete_by_group_id(dataset.id)
-                index.restore_dataset_in_one(dataset, dataset_collection_binding)
-            else:
-                click.echo('passed.')
-            normalization_count.append(1)
-        except Exception as e:
-            click.echo(
-                click.style('Create dataset index error: {} {}'.format(e.__class__.__name__, str(e)),
-                            fg='red'))
-
-
-@click.command('update_app_model_configs', help='Migrate data to support paragraph variable.')
-@click.option("--batch-size", default=500, help="Number of records to migrate in each batch.")
-def update_app_model_configs(batch_size):
-    pre_prompt_template = '{{default_input}}'
-
-    click.secho("Start migrate old data that the text generator can support paragraph variable.", fg='green')
-
-    total_records = db.session.query(AppModelConfig) \
-        .join(App, App.app_model_config_id == AppModelConfig.id) \
-        .filter(App.mode == 'completion') \
-        .count()
-
-    if total_records == 0:
-        click.secho("No data to migrate.", fg='green')
-        return
-
-    num_batches = (total_records + batch_size - 1) // batch_size
-
-    with tqdm(total=total_records, desc="Migrating Data") as pbar:
-        for i in range(num_batches):
-            offset = i * batch_size
-            limit = min(batch_size, total_records - offset)
-
-            click.secho(f"Fetching batch {i + 1}/{num_batches} from source database...", fg='green')
-
-            data_batch = db.session.query(AppModelConfig) \
-                .join(App, App.app_model_config_id == AppModelConfig.id) \
-                .filter(App.mode == 'completion') \
-                .order_by(App.created_at) \
-                .offset(offset).limit(limit).all()
-
-            if not data_batch:
-                click.secho("No more data to migrate.", fg='green')
-                break
-
-            try:
-                click.secho(f"Migrating {len(data_batch)} records...", fg='green')
-                for data in data_batch:
-                    # click.secho(f"Migrating data {data.id}, pre_prompt: {data.pre_prompt}, user_input_form: {data.user_input_form}", fg='green')
-
-                    if data.pre_prompt is None:
-                        data.pre_prompt = pre_prompt_template
-                    else:
-                        if pre_prompt_template in data.pre_prompt:
-                            continue
-                        data.pre_prompt += pre_prompt_template
-
-                    app_data = db.session.query(App) \
-                        .filter(App.id == data.app_id) \
-                        .one()
-
-                    account_data = db.session.query(Account) \
-                        .join(TenantAccountJoin, Account.id == TenantAccountJoin.account_id) \
-                        .filter(TenantAccountJoin.role == 'owner') \
-                        .filter(TenantAccountJoin.tenant_id == app_data.tenant_id) \
-                        .one_or_none()
-
-                    if not account_data:
-                        continue
-
-                    if data.user_input_form is None or data.user_input_form == 'null':
-                        data.user_input_form = json.dumps(user_input_form_template[account_data.interface_language])
-                    else:
-                        raw_json_data = json.loads(data.user_input_form)
-                        raw_json_data.append(user_input_form_template[account_data.interface_language][0])
-                        data.user_input_form = json.dumps(raw_json_data)
-
-                    # click.secho(f"Updated data {data.id}, pre_prompt: {data.pre_prompt}, user_input_form: {data.user_input_form}", fg='green')
-
-                db.session.commit()
-
-            except Exception as e:
-                click.secho(f"Error while migrating data: {e}, app_id: {data.app_id}, app_model_config_id: {data.id}",
-                            fg='red')
-                continue
-
-            click.secho(f"Successfully migrated batch {i + 1}/{num_batches}.", fg='green')
-
-            pbar.update(len(data_batch))
-
-
-@click.command('migrate_default_input_to_dataset_query_variable')
-@click.option("--batch-size", default=500, help="Number of records to migrate in each batch.")
-def migrate_default_input_to_dataset_query_variable(batch_size):
-    click.secho("Starting...", fg='green')
-
-    total_records = db.session.query(AppModelConfig) \
-        .join(App, App.app_model_config_id == AppModelConfig.id) \
-        .filter(App.mode == 'completion') \
-        .filter(AppModelConfig.dataset_query_variable == None) \
-        .count()
-
-    if total_records == 0:
-        click.secho("No data to migrate.", fg='green')
-        return
-
-    num_batches = (total_records + batch_size - 1) // batch_size
-
-    with tqdm(total=total_records, desc="Migrating Data") as pbar:
-        for i in range(num_batches):
-            offset = i * batch_size
-            limit = min(batch_size, total_records - offset)
-
-            click.secho(f"Fetching batch {i + 1}/{num_batches} from source database...", fg='green')
-
-            data_batch = db.session.query(AppModelConfig) \
-                .join(App, App.app_model_config_id == AppModelConfig.id) \
-                .filter(App.mode == 'completion') \
-                .filter(AppModelConfig.dataset_query_variable == None) \
-                .order_by(App.created_at) \
-                .offset(offset).limit(limit).all()
-
-            if not data_batch:
-                click.secho("No more data to migrate.", fg='green')
-                break
-
-            try:
-                click.secho(f"Migrating {len(data_batch)} records...", fg='green')
-                for data in data_batch:
-                    config = AppModelConfig.to_dict(data)
-
-                    tools = config["agent_mode"]["tools"]
-                    dataset_exists = "dataset" in str(tools)
-                    if not dataset_exists:
-                        continue
-
-                    user_input_form = config.get("user_input_form", [])
-                    for form in user_input_form:
-                        paragraph = form.get('paragraph')
-                        if paragraph \
-                                and paragraph.get('variable') == 'query':
-                            data.dataset_query_variable = 'query'
-                            break
-
-                        if paragraph \
-                                and paragraph.get('variable') == 'default_input':
-                            data.dataset_query_variable = 'default_input'
-                            break
-
-                db.session.commit()
-
-            except Exception as e:
-                click.secho(f"Error while migrating data: {e}, app_id: {data.app_id}, app_model_config_id: {data.id}",
-                            fg='red')
-                continue
-
-            click.secho(f"Successfully migrated batch {i + 1}/{num_batches}.", fg='green')
-
-            pbar.update(len(data_batch))
-
-
-@click.command('add-annotation-question-field-value', help='add annotation question value')
-def add_annotation_question_field_value():
-    click.echo(click.style('Start add annotation question value.', fg='green'))
-    message_annotations = db.session.query(MessageAnnotation).all()
-    message_annotation_deal_count = 0
-    if message_annotations:
-        for message_annotation in message_annotations:
-            try:
-                if message_annotation.message_id and not message_annotation.question:
-                    message = db.session.query(Message).filter(
-                        Message.id == message_annotation.message_id
-                    ).first()
-                    message_annotation.question = message.query
-                    db.session.add(message_annotation)
-                    db.session.commit()
-                    message_annotation_deal_count += 1
-            except Exception as e:
-                click.echo(
-                    click.style('Add annotation question value error: {} {}'.format(e.__class__.__name__, str(e)),
-                                fg='red'))
-            click.echo(
-                click.style(f'Congratulations! add annotation question value successful. Deal count {message_annotation_deal_count}', fg='green'))
-
-
 def register_commands(app):
     app.cli.add_command(reset_password)
     app.cli.add_command(reset_email)
-    app.cli.add_command(generate_invitation_codes)
     app.cli.add_command(reset_encrypt_key_pair)
-    app.cli.add_command(recreate_all_dataset_indexes)
-    app.cli.add_command(sync_anthropic_hosted_providers)
-    app.cli.add_command(clean_unused_dataset_indexes)
     app.cli.add_command(create_qdrant_indexes)
-    app.cli.add_command(update_qdrant_indexes)
-    app.cli.add_command(update_app_model_configs)
-    app.cli.add_command(normalization_collections)
-    app.cli.add_command(migrate_default_input_to_dataset_query_variable)
-    app.cli.add_command(add_qdrant_full_text_index)
-    app.cli.add_command(add_annotation_question_field_value)

api/config.py

@@ -40,17 +40,11 @@ DEFAULTS = {
     'HOSTED_OPENAI_QUOTA_LIMIT': 200,
     'HOSTED_OPENAI_TRIAL_ENABLED': 'False',
     'HOSTED_OPENAI_PAID_ENABLED': 'False',
-    'HOSTED_OPENAI_PAID_INCREASE_QUOTA': 1,
-    'HOSTED_OPENAI_PAID_MIN_QUANTITY': 1,
-    'HOSTED_OPENAI_PAID_MAX_QUANTITY': 1,
     'HOSTED_AZURE_OPENAI_ENABLED': 'False',
     'HOSTED_AZURE_OPENAI_QUOTA_LIMIT': 200,
     'HOSTED_ANTHROPIC_QUOTA_LIMIT': 600000,
     'HOSTED_ANTHROPIC_TRIAL_ENABLED': 'False',
     'HOSTED_ANTHROPIC_PAID_ENABLED': 'False',
-    'HOSTED_ANTHROPIC_PAID_INCREASE_QUOTA': 1,
-    'HOSTED_ANTHROPIC_PAID_MIN_QUANTITY': 1,
-    'HOSTED_ANTHROPIC_PAID_MAX_QUANTITY': 1,
     'HOSTED_MODERATION_ENABLED': 'False',
     'HOSTED_MODERATION_PROVIDERS': '',
     'CLEAN_DAY_SETTING': 30,
@@ -93,7 +87,7 @@ class Config:
         # ------------------------
         # General Configurations.
         # ------------------------
-        self.CURRENT_VERSION = "0.5.0"
+        self.CURRENT_VERSION = "0.5.3"
         self.COMMIT_SHA = get_env('COMMIT_SHA')
         self.EDITION = "SELF_HOSTED"
         self.DEPLOY_ENV = get_env('DEPLOY_ENV')
@@ -262,10 +256,6 @@ class Config:
         self.HOSTED_OPENAI_TRIAL_ENABLED = get_bool_env('HOSTED_OPENAI_TRIAL_ENABLED')
         self.HOSTED_OPENAI_QUOTA_LIMIT = int(get_env('HOSTED_OPENAI_QUOTA_LIMIT'))
         self.HOSTED_OPENAI_PAID_ENABLED = get_bool_env('HOSTED_OPENAI_PAID_ENABLED')
-        self.HOSTED_OPENAI_PAID_STRIPE_PRICE_ID = get_env('HOSTED_OPENAI_PAID_STRIPE_PRICE_ID')
-        self.HOSTED_OPENAI_PAID_INCREASE_QUOTA = int(get_env('HOSTED_OPENAI_PAID_INCREASE_QUOTA'))
-        self.HOSTED_OPENAI_PAID_MIN_QUANTITY = int(get_env('HOSTED_OPENAI_PAID_MIN_QUANTITY'))
-        self.HOSTED_OPENAI_PAID_MAX_QUANTITY = int(get_env('HOSTED_OPENAI_PAID_MAX_QUANTITY'))
 
         self.HOSTED_AZURE_OPENAI_ENABLED = get_bool_env('HOSTED_AZURE_OPENAI_ENABLED')
         self.HOSTED_AZURE_OPENAI_API_KEY = get_env('HOSTED_AZURE_OPENAI_API_KEY')
@@ -277,10 +267,6 @@ class Config:
         self.HOSTED_ANTHROPIC_TRIAL_ENABLED = get_bool_env('HOSTED_ANTHROPIC_TRIAL_ENABLED')
         self.HOSTED_ANTHROPIC_QUOTA_LIMIT = int(get_env('HOSTED_ANTHROPIC_QUOTA_LIMIT'))
         self.HOSTED_ANTHROPIC_PAID_ENABLED = get_bool_env('HOSTED_ANTHROPIC_PAID_ENABLED')
-        self.HOSTED_ANTHROPIC_PAID_STRIPE_PRICE_ID = get_env('HOSTED_ANTHROPIC_PAID_STRIPE_PRICE_ID')
-        self.HOSTED_ANTHROPIC_PAID_INCREASE_QUOTA = int(get_env('HOSTED_ANTHROPIC_PAID_INCREASE_QUOTA'))
-        self.HOSTED_ANTHROPIC_PAID_MIN_QUANTITY = int(get_env('HOSTED_ANTHROPIC_PAID_MIN_QUANTITY'))
-        self.HOSTED_ANTHROPIC_PAID_MAX_QUANTITY = int(get_env('HOSTED_ANTHROPIC_PAID_MAX_QUANTITY'))
 
         self.HOSTED_MINIMAX_ENABLED = get_bool_env('HOSTED_MINIMAX_ENABLED')
         self.HOSTED_SPARK_ENABLED = get_bool_env('HOSTED_SPARK_ENABLED')

api/constants/languages.py

@@ -1,5 +1,6 @@
 
 import json
+
 from models.model import AppModelConfig
 
 languages = ['en-US', 'zh-Hans', 'pt-BR', 'es-ES', 'fr-FR', 'de-DE', 'ja-JP', 'ko-KR', 'ru-RU', 'it-IT']

api/controllers/console/__init__.py

@@ -11,6 +11,7 @@ from .app import (advanced_prompt_template, annotation, app, audio, completion,
                   model_config, site, statistic)
 # Import auth controllers
 from .auth import activate, data_source_oauth, login, oauth
+# Import billing controllers
 from .billing import billing
 # Import datasets controllers
 from .datasets import data_source, datasets, datasets_document, datasets_segments, file, hit_testing

api/controllers/console/admin.py

@@ -1,12 +1,12 @@
 import os
 from functools import wraps
 
+from constants.languages import supported_language
 from controllers.console import api
 from controllers.console.wraps import only_edition_cloud
 from extensions.ext_database import db
 from flask import request
 from flask_restful import Resource, reqparse
-from constants.languages import supported_language
 from models.model import App, InstalledApp, RecommendedApp
 from werkzeug.exceptions import NotFound, Unauthorized
 

api/controllers/console/apikey.py

@@ -61,9 +61,7 @@ class BaseApiKeyListResource(Resource):
         resource_id = str(resource_id)
         _get_resource(resource_id, current_user.current_tenant_id,
                       self.resource_model)
-
-        # The role of the current user in the ta table must be admin or owner
-        if current_user.current_tenant.current_role not in ['admin', 'owner']:
+        if not current_user.is_admin_or_owner:
             raise Forbidden()
 
         current_key_count = db.session.query(ApiToken). \
@@ -102,7 +100,7 @@ class BaseApiKeyResource(Resource):
                       self.resource_model)
 
         # The role of the current user in the ta table must be admin or owner
-        if current_user.current_tenant.current_role not in ['admin', 'owner']:
+        if not current_user.is_admin_or_owner:
             raise Forbidden()
 
         key = db.session.query(ApiToken). \

api/controllers/console/app/annotation.py

@@ -21,7 +21,7 @@ class AnnotationReplyActionApi(Resource):
     @cloud_edition_billing_resource_check('annotation')
     def post(self, app_id, action):
         # The role of the current user in the ta table must be admin or owner
-        if current_user.current_tenant.current_role not in ['admin', 'owner']:
+        if not current_user.is_admin_or_owner:
             raise Forbidden()
 
         app_id = str(app_id)
@@ -45,7 +45,7 @@ class AppAnnotationSettingDetailApi(Resource):
     @account_initialization_required
     def get(self, app_id):
         # The role of the current user in the ta table must be admin or owner
-        if current_user.current_tenant.current_role not in ['admin', 'owner']:
+        if not current_user.is_admin_or_owner:
             raise Forbidden()
 
         app_id = str(app_id)
@@ -59,7 +59,7 @@ class AppAnnotationSettingUpdateApi(Resource):
     @account_initialization_required
     def post(self, app_id, annotation_setting_id):
         # The role of the current user in the ta table must be admin or owner
-        if current_user.current_tenant.current_role not in ['admin', 'owner']:
+        if not current_user.is_admin_or_owner:
             raise Forbidden()
 
         app_id = str(app_id)
@@ -80,7 +80,7 @@ class AnnotationReplyActionStatusApi(Resource):
     @cloud_edition_billing_resource_check('annotation')
     def get(self, app_id, job_id, action):
         # The role of the current user in the ta table must be admin or owner
-        if current_user.current_tenant.current_role not in ['admin', 'owner']:
+        if not current_user.is_admin_or_owner:
             raise Forbidden()
 
         job_id = str(job_id)
@@ -108,7 +108,7 @@ class AnnotationListApi(Resource):
     @account_initialization_required
     def get(self, app_id):
         # The role of the current user in the ta table must be admin or owner
-        if current_user.current_tenant.current_role not in ['admin', 'owner']:
+        if not current_user.is_admin_or_owner:
             raise Forbidden()
 
         page = request.args.get('page', default=1, type=int)
@@ -133,7 +133,7 @@ class AnnotationExportApi(Resource):
     @account_initialization_required
     def get(self, app_id):
         # The role of the current user in the ta table must be admin or owner
-        if current_user.current_tenant.current_role not in ['admin', 'owner']:
+        if not current_user.is_admin_or_owner:
             raise Forbidden()
 
         app_id = str(app_id)
@@ -152,7 +152,7 @@ class AnnotationCreateApi(Resource):
     @marshal_with(annotation_fields)
     def post(self, app_id):
         # The role of the current user in the ta table must be admin or owner
-        if current_user.current_tenant.current_role not in ['admin', 'owner']:
+        if not current_user.is_admin_or_owner:
             raise Forbidden()
 
         app_id = str(app_id)
@@ -172,7 +172,7 @@ class AnnotationUpdateDeleteApi(Resource):
     @marshal_with(annotation_fields)
     def post(self, app_id, annotation_id):
         # The role of the current user in the ta table must be admin or owner
-        if current_user.current_tenant.current_role not in ['admin', 'owner']:
+        if not current_user.is_admin_or_owner:
             raise Forbidden()
 
         app_id = str(app_id)
@@ -189,7 +189,7 @@ class AnnotationUpdateDeleteApi(Resource):
     @account_initialization_required
     def delete(self, app_id, annotation_id):
         # The role of the current user in the ta table must be admin or owner
-        if current_user.current_tenant.current_role not in ['admin', 'owner']:
+        if not current_user.is_admin_or_owner:
             raise Forbidden()
 
         app_id = str(app_id)
@@ -205,7 +205,7 @@ class AnnotationBatchImportApi(Resource):
     @cloud_edition_billing_resource_check('annotation')
     def post(self, app_id):
         # The role of the current user in the ta table must be admin or owner
-        if current_user.current_tenant.current_role not in ['admin', 'owner']:
+        if not current_user.is_admin_or_owner:
             raise Forbidden()
 
         app_id = str(app_id)
@@ -230,7 +230,7 @@ class AnnotationBatchImportStatusApi(Resource):
     @cloud_edition_billing_resource_check('annotation')
     def get(self, app_id, job_id):
         # The role of the current user in the ta table must be admin or owner
-        if current_user.current_tenant.current_role not in ['admin', 'owner']:
+        if not current_user.is_admin_or_owner:
             raise Forbidden()
 
         job_id = str(job_id)
@@ -257,7 +257,7 @@ class AnnotationHitHistoryListApi(Resource):
     @account_initialization_required
     def get(self, app_id, annotation_id):
         # The role of the current user in the table must be admin or owner
-        if current_user.current_tenant.current_role not in ['admin', 'owner']:
+        if not current_user.is_admin_or_owner:
             raise Forbidden()
 
         page = request.args.get('page', default=1, type=int)

api/controllers/console/app/app.py

@@ -3,8 +3,8 @@ import json
 import logging
 from datetime import datetime
 
-from constants.model_template import model_templates
 from constants.languages import demo_model_templates, languages
+from constants.model_template import model_templates
 from controllers.console import api
 from controllers.console.app.error import AppNotFoundError, ProviderNotInitializeError
 from controllers.console.setup import setup_required
@@ -26,6 +26,7 @@ from models.tools import ApiToolProvider
 from services.app_model_config_service import AppModelConfigService
 from werkzeug.exceptions import Forbidden
 
+
 def _get_app(app_id, tenant_id):
     app = db.session.query(App).filter(App.id == app_id, App.tenant_id == tenant_id).first()
     if not app:
@@ -88,7 +89,7 @@ class AppListApi(Resource):
         args = parser.parse_args()
 
         # The role of the current user in the ta table must be admin or owner
-        if current_user.current_tenant.current_role not in ['admin', 'owner']:
+        if not current_user.is_admin_or_owner:
             raise Forbidden()
 
         try:
@@ -107,20 +108,33 @@ class AppListApi(Resource):
             # validate config
             model_config_dict = args['model_config']
 
-            # get model provider
-            model_manager = ModelManager()
-            model_instance = model_manager.get_default_model_instance(
-                tenant_id=current_user.current_tenant_id,
-                model_type=ModelType.LLM
+            # Get provider configurations
+            provider_manager = ProviderManager()
+            provider_configurations = provider_manager.get_configurations(current_user.current_tenant_id)
+
+            # get available models from provider_configurations
+            available_models = provider_configurations.get_models(
+                model_type=ModelType.LLM,
+                only_active=True
             )
 
-            if not model_instance:
-                raise ProviderNotInitializeError(
-                    f"No Default System Reasoning Model available. Please configure "
-                    f"in the Settings -> Model Provider.")
-            else:
-                model_config_dict["model"]["provider"] = model_instance.provider
-                model_config_dict["model"]["name"] = model_instance.model
+            # check if model is available
+            available_models_names = [f'{model.provider.provider}.{model.model}' for model in available_models]
+            provider_model = f"{model_config_dict['model']['provider']}.{model_config_dict['model']['name']}"
+            if provider_model not in available_models_names:
+                model_manager = ModelManager()
+                model_instance = model_manager.get_default_model_instance(
+                    tenant_id=current_user.current_tenant_id,
+                    model_type=ModelType.LLM
+                )
+
+                if not model_instance:
+                    raise ProviderNotInitializeError(
+                        f"No Default System Reasoning Model available. Please configure "
+                        f"in the Settings -> Model Provider.")
+                else:
+                    model_config_dict["model"]["provider"] = model_instance.provider
+                    model_config_dict["model"]["name"] = model_instance.model
 
             model_configuration = AppModelConfigService.validate_configuration(
                 tenant_id=current_user.current_tenant_id,
@@ -237,7 +251,7 @@ class AppApi(Resource):
         """Delete app"""
         app_id = str(app_id)
 
-        if current_user.current_tenant.current_role not in ['admin', 'owner']:
+        if not current_user.is_admin_or_owner:
             raise Forbidden()
 
         app = _get_app(app_id, current_user.current_tenant_id)

api/controllers/console/app/audio.py

@@ -34,8 +34,7 @@ class ChatMessageAudioApi(Resource):
         try:
             response = AudioService.transcript_asr(
                 tenant_id=app_model.tenant_id,
-                file=file,
-                promot=app_model.app_model_config.pre_prompt
+                file=file
             )
 
             return response

api/controllers/console/app/message.py

@@ -157,7 +157,7 @@ class MessageAnnotationApi(Resource):
     @marshal_with(annotation_fields)
     def post(self, app_id):
         # The role of the current user in the ta table must be admin or owner
-        if current_user.current_tenant.current_role not in ['admin', 'owner']:
+        if not current_user.is_admin_or_owner:
             raise Forbidden()
 
         app_id = str(app_id)

api/controllers/console/app/site.py

@@ -1,4 +1,5 @@
 # -*- coding:utf-8 -*-
+from constants.languages import supported_language
 from controllers.console import api
 from controllers.console.app import _get_app
 from controllers.console.setup import setup_required
@@ -7,7 +8,6 @@ from extensions.ext_database import db
 from fields.app_fields import app_site_fields
 from flask_login import current_user
 from flask_restful import Resource, marshal_with, reqparse
-from constants.languages import supported_language
 from libs.login import login_required
 from models.model import Site
 from werkzeug.exceptions import Forbidden, NotFound
@@ -42,7 +42,7 @@ class AppSite(Resource):
         app_model = _get_app(app_id)
 
         # The role of the current user in the ta table must be admin or owner
-        if current_user.current_tenant.current_role not in ['admin', 'owner']:
+        if not current_user.is_admin_or_owner:
             raise Forbidden()
 
         site = db.session.query(Site). \
@@ -88,7 +88,7 @@ class AppSiteAccessTokenReset(Resource):
         app_model = _get_app(app_id)
 
         # The role of the current user in the ta table must be admin or owner
-        if current_user.current_tenant.current_role not in ['admin', 'owner']:
+        if not current_user.is_admin_or_owner:
             raise Forbidden()
 
         site = db.session.query(Site).filter(Site.app_id == app_model.id).first()

api/controllers/console/auth/activate.py

@@ -2,12 +2,12 @@ import base64
 import secrets
 from datetime import datetime
 
+from constants.languages import supported_language
 from controllers.console import api
 from controllers.console.error import AlreadyActivateError
 from extensions.ext_database import db
 from flask_restful import Resource, reqparse
 from libs.helper import email, str_len, timezone
-from constants.languages import supported_language
 from libs.password import hash_password, valid_password
 from models.account import AccountStatus, Tenant
 from services.account_service import RegisterService

api/controllers/console/auth/data_source_oauth.py

@@ -30,7 +30,7 @@ def get_oauth_providers():
 class OAuthDataSource(Resource):
     def get(self, provider: str):
         # The role of the current user in the table must be admin or owner
-        if current_user.current_tenant.current_role not in ['admin', 'owner']:
+        if not current_user.is_admin_or_owner:
             raise Forbidden()
         OAUTH_DATASOURCE_PROVIDERS = get_oauth_providers()
         with current_app.app_context():

api/controllers/console/billing/billing.py

@@ -20,7 +20,7 @@ class Subscription(Resource):
         parser.add_argument('interval', type=str, required=True, location='args', choices=['month', 'year'])
         args = parser.parse_args()
 
-        BillingService.is_tenant_owner(current_user)
+        BillingService.is_tenant_owner_or_admin(current_user)
 
         return BillingService.get_subscription(args['plan'],
                                                args['interval'],
@@ -35,8 +35,8 @@ class Invoices(Resource):
     @account_initialization_required
     @only_edition_cloud
     def get(self):
-        BillingService.is_tenant_owner(current_user)
-        return BillingService.get_invoices(current_user.email)
+        BillingService.is_tenant_owner_or_admin(current_user)
+        return BillingService.get_invoices(current_user.email, current_user.current_tenant_id)
 
 
 api.add_resource(Subscription, '/billing/subscription')

api/controllers/console/datasets/datasets.py

@@ -103,7 +103,7 @@ class DatasetListApi(Resource):
         args = parser.parse_args()
 
         # The role of the current user in the ta table must be admin or owner
-        if current_user.current_tenant.current_role not in ['admin', 'owner']:
+        if not current_user.is_admin_or_owner:
             raise Forbidden()
 
         try:
@@ -187,7 +187,7 @@ class DatasetApi(Resource):
         args = parser.parse_args()
 
         # The role of the current user in the ta table must be admin or owner
-        if current_user.current_tenant.current_role not in ['admin', 'owner']:
+        if not current_user.is_admin_or_owner:
             raise Forbidden()
 
         dataset = DatasetService.update_dataset(
@@ -205,7 +205,7 @@ class DatasetApi(Resource):
         dataset_id_str = str(dataset_id)
 
         # The role of the current user in the ta table must be admin or owner
-        if current_user.current_tenant.current_role not in ['admin', 'owner']:
+        if not current_user.is_admin_or_owner:
             raise Forbidden()
 
         if DatasetService.delete_dataset(dataset_id_str, current_user):
@@ -391,7 +391,7 @@ class DatasetApiKeyApi(Resource):
     @marshal_with(api_key_fields)
     def post(self):
         # The role of the current user in the ta table must be admin or owner
-        if current_user.current_tenant.current_role not in ['admin', 'owner']:
+        if not current_user.is_admin_or_owner:
             raise Forbidden()
 
         current_key_count = db.session.query(ApiToken). \
@@ -425,7 +425,7 @@ class DatasetApiDeleteApi(Resource):
         api_key_id = str(api_key_id)
 
         # The role of the current user in the ta table must be admin or owner
-        if current_user.current_tenant.current_role not in ['admin', 'owner']:
+        if not current_user.is_admin_or_owner:
             raise Forbidden()
 
         key = db.session.query(ApiToken). \

api/controllers/console/datasets/datasets_document.py

@@ -204,7 +204,7 @@ class DatasetDocumentListApi(Resource):
             raise NotFound('Dataset not found.')
 
         # The role of the current user in the ta table must be admin or owner
-        if current_user.current_tenant.current_role not in ['admin', 'owner']:
+        if not current_user.is_admin_or_owner:
             raise Forbidden()
 
         try:
@@ -256,7 +256,7 @@ class DatasetInitApi(Resource):
     @cloud_edition_billing_resource_check('vector_space')
     def post(self):
         # The role of the current user in the ta table must be admin or owner
-        if current_user.current_tenant.current_role not in ['admin', 'owner']:
+        if not current_user.is_admin_or_owner:
             raise Forbidden()
 
         parser = reqparse.RequestParser()
@@ -599,7 +599,7 @@ class DocumentProcessingApi(DocumentResource):
         document = self.get_document(dataset_id, document_id)
 
         # The role of the current user in the ta table must be admin or owner
-        if current_user.current_tenant.current_role not in ['admin', 'owner']:
+        if not current_user.is_admin_or_owner:
             raise Forbidden()
 
         if action == "pause":
@@ -663,7 +663,7 @@ class DocumentMetadataApi(DocumentResource):
         doc_metadata = req_data.get('doc_metadata')
 
         # The role of the current user in the ta table must be admin or owner
-        if current_user.current_tenant.current_role not in ['admin', 'owner']:
+        if not current_user.is_admin_or_owner:
             raise Forbidden()
 
         if doc_type is None or doc_metadata is None:
@@ -710,7 +710,7 @@ class DocumentStatusApi(DocumentResource):
         document = self.get_document(dataset_id, document_id)
 
         # The role of the current user in the ta table must be admin or owner
-        if current_user.current_tenant.current_role not in ['admin', 'owner']:
+        if not current_user.is_admin_or_owner:
             raise Forbidden()
 
         indexing_cache_key = 'document_{}_indexing'.format(document.id)

api/controllers/console/datasets/datasets_segments.py

@@ -123,7 +123,7 @@ class DatasetDocumentSegmentApi(Resource):
         # check user's model setting
         DatasetService.check_dataset_model_setting(dataset)
         # The role of the current user in the ta table must be admin or owner
-        if current_user.current_tenant.current_role not in ['admin', 'owner']:
+        if not current_user.is_admin_or_owner:
             raise Forbidden()
 
         try:
@@ -219,7 +219,7 @@ class DatasetDocumentSegmentAddApi(Resource):
         if not document:
             raise NotFound('Document not found.')
         # The role of the current user in the ta table must be admin or owner
-        if current_user.current_tenant.current_role not in ['admin', 'owner']:
+        if not current_user.is_admin_or_owner:
             raise Forbidden()
         # check embedding model setting
         if dataset.indexing_technique == 'high_quality':
@@ -298,7 +298,7 @@ class DatasetDocumentSegmentUpdateApi(Resource):
         if not segment:
             raise NotFound('Segment not found.')
         # The role of the current user in the ta table must be admin or owner
-        if current_user.current_tenant.current_role not in ['admin', 'owner']:
+        if not current_user.is_admin_or_owner:
             raise Forbidden()
         try:
             DatasetService.check_dataset_permission(dataset, current_user)
@@ -342,7 +342,7 @@ class DatasetDocumentSegmentUpdateApi(Resource):
         if not segment:
             raise NotFound('Segment not found.')
         # The role of the current user in the ta table must be admin or owner
-        if current_user.current_tenant.current_role not in ['admin', 'owner']:
+        if not current_user.is_admin_or_owner:
             raise Forbidden()
         try:
             DatasetService.check_dataset_permission(dataset, current_user)

api/controllers/console/datasets/file.py

@@ -9,7 +9,7 @@ from flask import current_app, request
 from flask_login import current_user
 from flask_restful import Resource, marshal_with
 from libs.login import login_required
-from services.file_service import FileService, ALLOWED_EXTENSIONS, UNSTRUSTURED_ALLOWED_EXTENSIONS
+from services.file_service import ALLOWED_EXTENSIONS, UNSTRUSTURED_ALLOWED_EXTENSIONS, FileService
 
 PREVIEW_WORDS_LIMIT = 3000
 

api/controllers/console/error.py

@@ -13,6 +13,16 @@ class NotSetupError(BaseHTTPException):
                   "Please proceed with the initialization and installation process first."
     code = 401
 
+class NotInitValidateError(BaseHTTPException):
+    error_code = 'not_init_validated'
+    description = "Init validation has not been completed yet. " \
+                  "Please proceed with the init validation process first."
+    code = 401
+
+class InitValidateFailedError(BaseHTTPException):
+    error_code = 'init_validate_failed'
+    description = "Init validation failed. Please check the password and try again."
+    code = 401
 
 class AccountNotLinkTenantError(BaseHTTPException):
     error_code = 'account_not_link_tenant'

api/controllers/console/explore/audio.py

@@ -32,6 +32,7 @@ class ChatAudioApi(InstalledAppResource):
             response = AudioService.transcript_asr(
                 tenant_id=app_model.tenant_id,
                 file=file,
+                end_user=None
             )
 
             return response

api/controllers/console/explore/message.py

@@ -17,9 +17,9 @@ from core.model_runtime.errors.invoke import InvokeError
 from fields.message_fields import message_infinite_scroll_pagination_fields
 from flask import Response, stream_with_context
 from flask_login import current_user
-from flask_restful import marshal_with, reqparse, fields
+from flask_restful import fields, marshal_with, reqparse
 from flask_restful.inputs import int_range
-from libs.helper import uuid_value, TimestampField
+from libs.helper import TimestampField, uuid_value
 from services.completion_service import CompletionService
 from services.errors.app import MoreLikeThisDisabledError
 from services.errors.conversation import ConversationNotExistsError

api/controllers/console/explore/parameter.py

@@ -3,12 +3,12 @@ import json
 
 from controllers.console import api
 from controllers.console.explore.wraps import InstalledAppResource
+from extensions.ext_database import db
 from flask import current_app
 from flask_restful import fields, marshal_with
-from models.model import InstalledApp, AppModelConfig
+from models.model import AppModelConfig, InstalledApp
 from models.tools import ApiToolProvider
 
-from extensions.ext_database import db
 
 class AppParameterApi(InstalledAppResource):
     """Resource for app variables."""

api/controllers/console/explore/recommended_app.py

@@ -1,4 +1,5 @@
 # -*- coding:utf-8 -*-
+from constants.languages import languages
 from controllers.console import api
 from controllers.console.app.error import AppNotFoundError
 from controllers.console.wraps import account_initialization_required
@@ -9,7 +10,6 @@ from libs.login import login_required
 from models.model import App, InstalledApp, RecommendedApp
 from services.account_service import TenantService
 from sqlalchemy import and_
-from constants.languages import languages
 
 app_fields = {
     'id': fields.String,

api/controllers/console/feature.py

@@ -3,10 +3,12 @@ from flask_restful import Resource
 from services.feature_service import FeatureService
 
 from . import api
+from .wraps import cloud_utm_record
 
 
 class FeatureApi(Resource):
 
+    @cloud_utm_record
     def get(self):
         return FeatureService.get_features(current_user.current_tenant_id).dict()
 

api/controllers/console/init_validate.py

@@ -0,0 +1,48 @@
+import os
+
+from flask import current_app, session
+from flask_restful import Resource, reqparse
+from libs.helper import str_len
+from models.model import DifySetup
+from services.account_service import TenantService
+
+from . import api
+from .error import AlreadySetupError, InitValidateFailedError
+from .wraps import only_edition_self_hosted
+
+
+class InitValidateAPI(Resource):
+
+    def get(self):
+        init_status = get_init_validate_status()
+        if init_status:
+            return { 'status': 'finished' }
+        return {'status': 'not_started' }
+
+    @only_edition_self_hosted
+    def post(self):
+        # is tenant created
+        tenant_count = TenantService.get_tenant_count()
+        if tenant_count > 0:
+            raise AlreadySetupError()
+
+        parser = reqparse.RequestParser()
+        parser.add_argument('password', type=str_len(30),
+                            required=True, location='json')
+        input_password = parser.parse_args()['password']
+
+        if input_password != os.environ.get('INIT_PASSWORD'):
+            session['is_init_validated'] = False
+            raise InitValidateFailedError()
+            
+        session['is_init_validated'] = True
+        return {'result': 'success'}, 201
+
+def get_init_validate_status():
+    if current_app.config['EDITION'] == 'SELF_HOSTED':
+        if os.environ.get('INIT_PASSWORD'):
+            return session.get('is_init_validated') or DifySetup.query.first()
+    
+    return True
+
+api.add_resource(InitValidateAPI, '/init')

api/controllers/console/setup.py

@@ -10,7 +10,8 @@ from models.model import DifySetup
 from services.account_service import AccountService, RegisterService, TenantService
 
 from . import api
-from .error import AlreadySetupError, NotSetupError
+from .error import AlreadySetupError, NotInitValidateError, NotSetupError
+from .init_validate import get_init_validate_status
 from .wraps import only_edition_self_hosted
 
 
@@ -24,7 +25,7 @@ class SetupApi(Resource):
                     'step': 'finished',
                     'setup_at': setup_status.setup_at.isoformat()
                 }
-            return {'step': 'not_start'}
+            return {'step': 'not_started'}
         return {'step': 'finished'}
 
     @only_edition_self_hosted
@@ -37,6 +38,9 @@ class SetupApi(Resource):
         tenant_count = TenantService.get_tenant_count()
         if tenant_count > 0:
             raise AlreadySetupError()
+    
+        if not get_init_validate_status():
+            raise NotInitValidateError()
 
         parser = reqparse.RequestParser()
         parser.add_argument('email', type=email,
@@ -71,7 +75,10 @@ def setup_required(view):
     @wraps(view)
     def decorated(*args, **kwargs):
         # check setup
-        if not get_setup_status():
+        if not get_init_validate_status():
+            raise NotInitValidateError()
+        
+        elif not get_setup_status():
             raise NotSetupError()
 
         return view(*args, **kwargs)

api/controllers/console/workspace/account.py

@@ -2,6 +2,7 @@
 from datetime import datetime
 
 import pytz
+from constants.languages import supported_language
 from controllers.console import api
 from controllers.console.setup import setup_required
 from controllers.console.workspace.error import (AccountAlreadyInitedError, CurrentPasswordIncorrectError,
@@ -12,7 +13,6 @@ from flask import current_app, request
 from flask_login import current_user
 from flask_restful import Resource, fields, marshal_with, reqparse
 from libs.helper import TimestampField, timezone
-from constants.languages import supported_language
 from libs.login import login_required
 from models.account import AccountIntegrate, InvitationCode
 from services.account_service import AccountService

api/controllers/console/workspace/members.py

@@ -1,13 +1,12 @@
 # -*- coding:utf-8 -*-
-from flask import current_app
-from flask_login import current_user
-from flask_restful import Resource, abort, fields, marshal_with, reqparse
-
 import services
 from controllers.console import api
 from controllers.console.setup import setup_required
 from controllers.console.wraps import account_initialization_required, cloud_edition_billing_resource_check
 from extensions.ext_database import db
+from flask import current_app
+from flask_login import current_user
+from flask_restful import Resource, abort, fields, marshal_with, reqparse
 from libs.helper import TimestampField
 from libs.login import login_required
 from models.account import Account
@@ -52,10 +51,12 @@ class MemberInviteEmailApi(Resource):
         parser = reqparse.RequestParser()
         parser.add_argument('emails', type=str, required=True, location='json', action='append')
         parser.add_argument('role', type=str, required=True, default='admin', location='json')
+        parser.add_argument('language', type=str, required=False, location='json')
         args = parser.parse_args()
 
         invitee_emails = args['emails']
         invitee_role = args['role']
+        interface_language = args['language']
         if invitee_role not in ['admin', 'normal']:
             return {'code': 'invalid-role', 'message': 'Invalid role'}, 400
 
@@ -64,8 +65,7 @@ class MemberInviteEmailApi(Resource):
         console_web_url = current_app.config.get("CONSOLE_WEB_URL")
         for invitee_email in invitee_emails:
             try:
-                token = RegisterService.invite_new_member(inviter.current_tenant, invitee_email, role=invitee_role,
-                                                          inviter=inviter)
+                token = RegisterService.invite_new_member(inviter.current_tenant, invitee_email, interface_language, role=invitee_role, inviter=inviter)
                 invitation_results.append({
                     'status': 'success',
                     'email': invitee_email,

api/controllers/console/workspace/model_providers.py

@@ -98,7 +98,7 @@ class ModelProviderApi(Resource):
     @login_required
     @account_initialization_required
     def post(self, provider: str):
-        if current_user.current_tenant.current_role not in ['admin', 'owner']:
+        if not current_user.is_admin_or_owner:
             raise Forbidden()
 
         parser = reqparse.RequestParser()
@@ -122,7 +122,7 @@ class ModelProviderApi(Resource):
     @login_required
     @account_initialization_required
     def delete(self, provider: str):
-        if current_user.current_tenant.current_role not in ['admin', 'owner']:
+        if not current_user.is_admin_or_owner:
             raise Forbidden()
 
         model_provider_service = ModelProviderService()
@@ -159,7 +159,7 @@ class PreferredProviderTypeUpdateApi(Resource):
     @login_required
     @account_initialization_required
     def post(self, provider: str):
-        if current_user.current_tenant.current_role not in ['admin', 'owner']:
+        if not current_user.is_admin_or_owner:
             raise Forbidden()
 
         tenant_id = current_user.current_tenant_id
@@ -186,10 +186,11 @@ class ModelProviderPaymentCheckoutUrlApi(Resource):
     def get(self, provider: str):
         if provider != 'anthropic':
             raise ValueError(f'provider name {provider} is invalid')
-
+        BillingService.is_tenant_owner_or_admin(current_user)
         data = BillingService.get_model_provider_payment_link(provider_name=provider,
                                                               tenant_id=current_user.current_tenant_id,
-                                                              account_id=current_user.id)
+                                                              account_id=current_user.id,
+                                                              prefilled_email=current_user.email)
         return data
 
 

api/controllers/console/workspace/tool_providers.py

@@ -1,18 +1,16 @@
+import io
 import json
 
-from libs.login import login_required
-from flask_login import current_user
-from flask_restful import Resource, reqparse
-from flask import send_file
-from werkzeug.exceptions import Forbidden
-
 from controllers.console import api
 from controllers.console.setup import setup_required
 from controllers.console.wraps import account_initialization_required
-
+from flask import send_file
+from flask_login import current_user
+from flask_restful import Resource, reqparse
+from libs.login import login_required
 from services.tools_manage_service import ToolManageService
+from werkzeug.exceptions import Forbidden
 
-import io
 
 class ToolProviderListApi(Resource):
     @setup_required
@@ -43,7 +41,7 @@ class ToolBuiltinProviderDeleteApi(Resource):
     @login_required
     @account_initialization_required
     def post(self, provider):
-        if current_user.current_tenant.current_role not in ['admin', 'owner']:
+        if not current_user.is_admin_or_owner:
             raise Forbidden()
         
         user_id = current_user.id
@@ -60,7 +58,7 @@ class ToolBuiltinProviderUpdateApi(Resource):
     @login_required
     @account_initialization_required
     def post(self, provider):
-        if current_user.current_tenant.current_role not in ['admin', 'owner']:
+        if not current_user.is_admin_or_owner:
             raise Forbidden()
         
         user_id = current_user.id
@@ -90,7 +88,7 @@ class ToolApiProviderAddApi(Resource):
     @login_required
     @account_initialization_required
     def post(self):
-        if current_user.current_tenant.current_role not in ['admin', 'owner']:
+        if not current_user.is_admin_or_owner:
             raise Forbidden()
         
         user_id = current_user.id
@@ -159,7 +157,7 @@ class ToolApiProviderUpdateApi(Resource):
     @login_required
     @account_initialization_required
     def post(self):
-        if current_user.current_tenant.current_role not in ['admin', 'owner']:
+        if not current_user.is_admin_or_owner:
             raise Forbidden()
         
         user_id = current_user.id
@@ -171,8 +169,8 @@ class ToolApiProviderUpdateApi(Resource):
         parser.add_argument('schema', type=str, required=True, nullable=False, location='json')
         parser.add_argument('provider', type=str, required=True, nullable=False, location='json')
         parser.add_argument('original_provider', type=str, required=True, nullable=False, location='json')
-        parser.add_argument('icon', type=str, required=True, nullable=False, location='json')
-        parser.add_argument('privacy_policy', type=str, required=True, nullable=False, location='json')
+        parser.add_argument('icon', type=dict, required=True, nullable=False, location='json')
+        parser.add_argument('privacy_policy', type=str, required=True, nullable=True, location='json')
 
         args = parser.parse_args()
 
@@ -193,7 +191,7 @@ class ToolApiProviderDeleteApi(Resource):
     @login_required
     @account_initialization_required
     def post(self):
-        if current_user.current_tenant.current_role not in ['admin', 'owner']:
+        if not current_user.is_admin_or_owner:
             raise Forbidden()
         
         user_id = current_user.id

api/controllers/console/wraps.py

@@ -1,10 +1,12 @@
 # -*- coding:utf-8 -*-
+import json
 from functools import wraps
 
 from controllers.console.workspace.error import AccountNotInitializedError
-from flask import abort, current_app
+from flask import abort, current_app, request
 from flask_login import current_user
 from services.feature_service import FeatureService
+from services.operation_service import OperationService
 
 
 def account_initialization_required(view):
@@ -73,3 +75,20 @@ def cloud_edition_billing_resource_check(resource: str,
         return decorated
     return interceptor
 
+
+def cloud_utm_record(view):
+    @wraps(view)
+    def decorated(*args, **kwargs):
+        try:
+            features = FeatureService.get_features(current_user.current_tenant_id)
+
+            if features.billing.enabled:
+                utm_info = request.cookies.get('utm_info')
+
+                if utm_info:
+                    utm_info = json.loads(utm_info)
+                    OperationService.record_utm(current_user.current_tenant_id, utm_info)
+        except Exception as e:
+            pass
+        return view(*args, **kwargs)
+    return decorated

api/controllers/files/__init__.py

@@ -6,5 +6,4 @@ bp = Blueprint('files', __name__)
 api = ExternalApi(bp)
 
 
-from . import image_preview
-from . import tool_files
+from . import image_preview, tool_files

api/controllers/files/tool_files.py

@@ -1,10 +1,10 @@
 from controllers.files import api
+from core.tools.tool_file_manager import ToolFileManager
 from flask import Response
 from flask_restful import Resource, reqparse
 from libs.exception import BaseHTTPException
-from werkzeug.exceptions import NotFound, Forbidden
+from werkzeug.exceptions import Forbidden, NotFound
 
-from core.tools.tool_file_manager import ToolFileManager
 
 class ToolFilePreviewApi(Resource):
     def get(self, file_id, extension):

api/controllers/service_api/app/app.py

@@ -1,15 +1,14 @@
 # -*- coding:utf-8 -*-
+import json
+
 from controllers.service_api import api
 from controllers.service_api.wraps import AppApiResource
+from extensions.ext_database import db
 from flask import current_app
 from flask_restful import fields, marshal_with
 from models.model import App, AppModelConfig
 from models.tools import ApiToolProvider
 
-import json
-
-from extensions.ext_database import db
-
 
 class AppParameterApi(AppApiResource):
     """Resource for app variables."""

api/controllers/service_api/app/audio.py

@@ -66,6 +66,7 @@ class TextApi(AppApiResource):
         parser = reqparse.RequestParser()
         parser.add_argument('text', type=str, required=True, nullable=False, location='json')
         parser.add_argument('user', type=str, required=True, nullable=False, location='json')
+        parser.add_argument('streaming', type=bool, required=False, nullable=False, location='json')
         args = parser.parse_args()
 
         try:
@@ -73,7 +74,7 @@ class TextApi(AppApiResource):
                 tenant_id=app_model.tenant_id,
                 text=args['text'],
                 end_user=args['user'],
-                streaming=False
+                streaming=args['streaming']
             )
 
             return response

api/controllers/service_api/app/completion.py

@@ -13,7 +13,7 @@ from core.application_queue_manager import ApplicationQueueManager
 from core.entities.application_entities import InvokeFrom
 from core.errors.error import ModelCurrentlyNotSupportError, ProviderTokenNotInitError, QuotaExceededError
 from core.model_runtime.errors.invoke import InvokeError
-from flask import Response, stream_with_context, request
+from flask import Response, stream_with_context
 from flask_restful import reqparse
 from libs.helper import uuid_value
 from services.completion_service import CompletionService
@@ -75,18 +75,22 @@ class CompletionApi(AppApiResource):
 
 
 class CompletionStopApi(AppApiResource):
-    def post(self, app_model, _, task_id):
+    def post(self, app_model, end_user, task_id):
         if app_model.mode != 'completion':
             raise AppUnavailableError()
 
-        parser = reqparse.RequestParser()
-        parser.add_argument('user', required=True, nullable=False, type=str, location='json')
+        if end_user is None:
+            parser = reqparse.RequestParser()
+            parser.add_argument('user', required=True, nullable=False, type=str, location='json')
+            args = parser.parse_args()
 
-        args = parser.parse_args()
+            user = args.get('user')
+            if user is not None:
+                end_user = create_or_update_end_user_for_user_id(app_model, user)
+            else:
+                raise ValueError("arg user muse be input.")
 
-        end_user_id = args.get('user')
-
-        ApplicationQueueManager.set_stop_flag(task_id, InvokeFrom.SERVICE_API, end_user_id)
+        ApplicationQueueManager.set_stop_flag(task_id, InvokeFrom.SERVICE_API, end_user.id)
 
         return {'result': 'success'}, 200
 
@@ -146,13 +150,22 @@ class ChatApi(AppApiResource):
 
 
 class ChatStopApi(AppApiResource):
-    def post(self, app_model, _, task_id):
+    def post(self, app_model, end_user, task_id):
         if app_model.mode != 'chat':
             raise NotChatAppError()
 
-        end_user_id = request.get_json().get('user')
+        if end_user is None:
+            parser = reqparse.RequestParser()
+            parser.add_argument('user', required=True, nullable=False, type=str, location='json')
+            args = parser.parse_args()
 
-        ApplicationQueueManager.set_stop_flag(task_id, InvokeFrom.SERVICE_API, end_user_id)
+            user = args.get('user')
+            if user is not None:
+                end_user = create_or_update_end_user_for_user_id(app_model, user)
+            else:
+                raise ValueError("arg user muse be input.")
+
+        ApplicationQueueManager.set_stop_flag(task_id, InvokeFrom.SERVICE_API, end_user.id)
 
         return {'result': 'success'}, 200
 

api/controllers/service_api/app/message.py

@@ -44,6 +44,7 @@ class MessageListApi(AppApiResource):
         'position': fields.Integer,
         'thought': fields.String,
         'tool': fields.String,
+        'tool_labels': fields.Raw,
         'tool_input': fields.String,
         'created_at': TimestampField,
         'observation': fields.String,

api/controllers/service_api/dataset/dataset.py

@@ -1,4 +1,3 @@
-from models.dataset import Dataset
 import services.dataset_service
 from controllers.service_api import api
 from controllers.service_api.dataset.error import DatasetNameDuplicateError
@@ -9,6 +8,7 @@ from fields.dataset_fields import dataset_detail_fields
 from flask import request
 from flask_restful import marshal, reqparse
 from libs.login import current_user
+from models.dataset import Dataset
 from services.dataset_service import DatasetService
 
 

api/controllers/service_api/index.py

@@ -1,8 +1,7 @@
+from controllers.service_api import api
 from flask import current_app
 from flask_restful import Resource
 
-from controllers.service_api import api
-
 
 class IndexApi(Resource):
     def get(self):

api/controllers/service_api/wraps.py

@@ -75,8 +75,8 @@ def validate_dataset_token(view=None):
             tenant_account_join = db.session.query(Tenant, TenantAccountJoin) \
                 .filter(Tenant.id == api_token.tenant_id) \
                 .filter(TenantAccountJoin.tenant_id == Tenant.id) \
-                .filter(TenantAccountJoin.role.in_(['owner', 'admin'])) \
-                .one_or_none()
+                .filter(TenantAccountJoin.role.in_(['owner'])) \
+                .one_or_none() # TODO: only owner information is required, so only one is returned.
             if tenant_account_join:
                 tenant, ta = tenant_account_join
                 account = Account.query.filter_by(id=ta.account_id).first()
@@ -86,9 +86,9 @@ def validate_dataset_token(view=None):
                     current_app.login_manager._update_request_context_with_user(account)
                     user_logged_in.send(current_app._get_current_object(), user=_get_user())
                 else:
-                    raise Unauthorized("Tenant owner account is not exist.")
+                    raise Unauthorized("Tenant owner account does not exist.")
             else:
-                raise Unauthorized("Tenant is not exist.")
+                raise Unauthorized("Tenant does not exist.")
             return view(api_token.tenant_id, *args, **kwargs)
         return decorated
 

api/controllers/web/app.py

@@ -1,15 +1,14 @@
 # -*- coding:utf-8 -*-
+import json
+
 from controllers.web import api
 from controllers.web.wraps import WebApiResource
+from extensions.ext_database import db
 from flask import current_app
 from flask_restful import fields, marshal_with
 from models.model import App, AppModelConfig
 from models.tools import ApiToolProvider
 
-from extensions.ext_database import db
-
-import json
-
 
 class AppParameterApi(WebApiResource):
     """Resource for app variables."""

api/controllers/web/audio.py

@@ -31,6 +31,7 @@ class AudioApi(WebApiResource):
             response = AudioService.transcript_asr(
                 tenant_id=app_model.tenant_id,
                 file=file,
+                end_user=end_user
             )
 
             return response

api/core/app_runner/app_runner.py

@@ -2,8 +2,13 @@ import time
 from typing import Generator, List, Optional, Tuple, Union, cast
 
 from core.application_queue_manager import ApplicationQueueManager, PublishFrom
-from core.entities.application_entities import AppOrchestrationConfigEntity, ModelConfigEntity, \
-    PromptTemplateEntity, ExternalDataVariableEntity, ApplicationGenerateEntity, InvokeFrom
+from core.entities.application_entities import (ApplicationGenerateEntity, AppOrchestrationConfigEntity,
+                                                ExternalDataVariableEntity, InvokeFrom, ModelConfigEntity,
+                                                PromptTemplateEntity)
+from core.features.annotation_reply import AnnotationReplyFeature
+from core.features.external_data_fetch import ExternalDataFetchFeature
+from core.features.hosting_moderation import HostingModerationFeature
+from core.features.moderation import ModerationFeature
 from core.file.file_obj import FileObj
 from core.memory.token_buffer_memory import TokenBufferMemory
 from core.model_runtime.entities.llm_entities import LLMResult, LLMResultChunk, LLMResultChunkDelta, LLMUsage
@@ -11,12 +16,9 @@ from core.model_runtime.entities.message_entities import AssistantPromptMessage,
 from core.model_runtime.entities.model_entities import ModelPropertyKey
 from core.model_runtime.errors.invoke import InvokeBadRequestError
 from core.model_runtime.model_providers.__base.large_language_model import LargeLanguageModel
-from core.features.hosting_moderation import HostingModerationFeature
-from core.features.moderation import ModerationFeature
-from core.features.external_data_fetch import ExternalDataFetchFeature
-from core.features.annotation_reply import AnnotationReplyFeature
 from core.prompt.prompt_transform import PromptTransform
-from models.model import App, MessageAnnotation, Message
+from models.model import App, Message, MessageAnnotation
+
 
 class AppRunner:
     def get_pre_calculate_rest_tokens(self, app_record: App,

api/core/app_runner/assistant_app_runner.py

@@ -3,19 +3,19 @@ import logging
 from typing import cast
 
 from core.app_runner.app_runner import AppRunner
+from core.application_queue_manager import ApplicationQueueManager, PublishFrom
+from core.entities.application_entities import AgentEntity, ApplicationGenerateEntity, ModelConfigEntity
 from core.features.assistant_cot_runner import AssistantCotApplicationRunner
 from core.features.assistant_fc_runner import AssistantFunctionCallApplicationRunner
-from core.entities.application_entities import ApplicationGenerateEntity, ModelConfigEntity, \
-    AgentEntity
-from core.application_queue_manager import ApplicationQueueManager, PublishFrom
 from core.memory.token_buffer_memory import TokenBufferMemory
 from core.model_manager import ModelInstance
 from core.model_runtime.entities.llm_entities import LLMUsage
+from core.model_runtime.entities.model_entities import ModelFeature
 from core.model_runtime.model_providers.__base.large_language_model import LargeLanguageModel
 from core.moderation.base import ModerationException
 from core.tools.entities.tool_entities import ToolRuntimeVariablePool
 from extensions.ext_database import db
-from models.model import Conversation, Message, App, MessageChain, MessageAgentThought
+from models.model import App, Conversation, Message, MessageAgentThought, MessageChain
 from models.tools import ToolConversationVariables
 
 logger = logging.getLogger(__name__)
@@ -169,7 +169,7 @@ class AssistantApplicationRunner(AppRunner):
         # load tool variables
         tool_conversation_variables = self._load_tool_variables(conversation_id=conversation.id,
                                                    user_id=application_generate_entity.user_id,
-                                                   tanent_id=application_generate_entity.tenant_id)
+                                                   tenant_id=application_generate_entity.tenant_id)
 
         # convert db variables to tool variables
         tool_variables = self._convert_db_variables_to_tool_variables(tool_conversation_variables)
@@ -194,6 +194,13 @@ class AssistantApplicationRunner(AppRunner):
             memory=memory,
         )
 
+        # change function call strategy based on LLM model
+        llm_model = cast(LargeLanguageModel, model_instance.model_type_instance)
+        model_schema = llm_model.get_model_schema(model_instance.model, model_instance.credentials)
+
+        if set([ModelFeature.MULTI_TOOL_CALL, ModelFeature.TOOL_CALL]).intersection(model_schema.features or []):
+            agent_entity.strategy = AgentEntity.Strategy.FUNCTION_CALLING
+
         # start agent runner
         if agent_entity.strategy == AgentEntity.Strategy.CHAIN_OF_THOUGHT:
             assistant_cot_runner = AssistantCotApplicationRunner(
@@ -209,9 +216,9 @@ class AssistantApplicationRunner(AppRunner):
                 prompt_messages=prompt_message,
                 variables_pool=tool_variables,
                 db_variables=tool_conversation_variables,
+                model_instance=model_instance
             )
             invoke_result = assistant_cot_runner.run(
-                model_instance=model_instance,
                 conversation=conversation,
                 message=message,
                 query=query,
@@ -229,10 +236,10 @@ class AssistantApplicationRunner(AppRunner):
                 memory=memory,
                 prompt_messages=prompt_message,
                 variables_pool=tool_variables,
-                db_variables=tool_conversation_variables
+                db_variables=tool_conversation_variables,
+                model_instance=model_instance
             )
             invoke_result = assistant_fc_runner.run(
-                model_instance=model_instance,
                 conversation=conversation,
                 message=message,
                 query=query,
@@ -246,13 +253,13 @@ class AssistantApplicationRunner(AppRunner):
             agent=True
         )
 
-    def _load_tool_variables(self, conversation_id: str, user_id: str, tanent_id: str) -> ToolConversationVariables:
+    def _load_tool_variables(self, conversation_id: str, user_id: str, tenant_id: str) -> ToolConversationVariables:
         """
         load tool variables from database
         """
         tool_variables: ToolConversationVariables = db.session.query(ToolConversationVariables).filter(
             ToolConversationVariables.conversation_id == conversation_id,
-            ToolConversationVariables.tenant_id == tanent_id
+            ToolConversationVariables.tenant_id == tenant_id
         ).first()
 
         if tool_variables:
@@ -263,7 +270,7 @@ class AssistantApplicationRunner(AppRunner):
             tool_variables = ToolConversationVariables(
                 conversation_id=conversation_id,
                 user_id=user_id,
-                tenant_id=tanent_id,
+                tenant_id=tenant_id,
                 variables_str='[]',
             )
             db.session.add(tool_variables)

api/core/app_runner/basic_app_runner.py

@@ -4,8 +4,7 @@ from typing import Optional
 from core.app_runner.app_runner import AppRunner
 from core.application_queue_manager import ApplicationQueueManager, PublishFrom
 from core.callback_handler.index_tool_callback_handler import DatasetIndexToolCallbackHandler
-from core.entities.application_entities import (ApplicationGenerateEntity, DatasetEntity,
-                                                 InvokeFrom, ModelConfigEntity)
+from core.entities.application_entities import ApplicationGenerateEntity, DatasetEntity, InvokeFrom, ModelConfigEntity
 from core.features.dataset_retrieval import DatasetRetrievalFeature
 from core.memory.token_buffer_memory import TokenBufferMemory
 from core.model_manager import ModelInstance

api/core/app_runner/generate_task_pipeline.py

@@ -6,20 +6,21 @@ from typing import Generator, Optional, Union, cast
 from core.app_runner.moderation_handler import ModerationRule, OutputModerationHandler
 from core.application_queue_manager import ApplicationQueueManager, PublishFrom
 from core.entities.application_entities import ApplicationGenerateEntity, InvokeFrom
-from core.entities.queue_entities import (AnnotationReplyEvent, QueueAgentThoughtEvent, QueueErrorEvent,
-                                          QueueMessageEndEvent, QueueMessageEvent, QueueMessageReplaceEvent,
-                                          QueuePingEvent, QueueRetrieverResourcesEvent, QueueStopEvent,
-                                          QueueMessageFileEvent, QueueAgentMessageEvent)
-from core.errors.error import ProviderTokenNotInitError, QuotaExceededError, ModelCurrentlyNotSupportError
+from core.entities.queue_entities import (AnnotationReplyEvent, QueueAgentMessageEvent, QueueAgentThoughtEvent,
+                                          QueueErrorEvent, QueueMessageEndEvent, QueueMessageEvent,
+                                          QueueMessageFileEvent, QueueMessageReplaceEvent, QueuePingEvent,
+                                          QueueRetrieverResourcesEvent, QueueStopEvent)
+from core.errors.error import ModelCurrentlyNotSupportError, ProviderTokenNotInitError, QuotaExceededError
 from core.model_runtime.entities.llm_entities import LLMResult, LLMResultChunk, LLMResultChunkDelta, LLMUsage
 from core.model_runtime.entities.message_entities import (AssistantPromptMessage, ImagePromptMessageContent,
                                                           PromptMessage, PromptMessageContentType, PromptMessageRole,
                                                           TextPromptMessageContent)
 from core.model_runtime.errors.invoke import InvokeAuthorizationError, InvokeError
 from core.model_runtime.model_providers.__base.large_language_model import LargeLanguageModel
-from core.tools.tool_file_manager import ToolFileManager
 from core.model_runtime.utils.encoders import jsonable_encoder
 from core.prompt.prompt_template import PromptTemplateParser
+from core.tools.tool_file_manager import ToolFileManager
+from core.tools.tool_manager import ToolManager
 from events.message_event import message_was_created
 from extensions.ext_database import db
 from models.model import Conversation, Message, MessageAgentThought, MessageFile
@@ -281,7 +282,7 @@ class GenerateTaskPipeline:
 
                     self._task_state.llm_result.message.content = annotation.content
             elif isinstance(event, QueueAgentThoughtEvent):
-                agent_thought = (
+                agent_thought: MessageAgentThought = (
                     db.session.query(MessageAgentThought)
                     .filter(MessageAgentThought.id == event.agent_thought_id)
                     .first()
@@ -298,6 +299,7 @@ class GenerateTaskPipeline:
                         'thought': agent_thought.thought,
                         'observation': agent_thought.observation,
                         'tool': agent_thought.tool,
+                        'tool_labels': agent_thought.tool_labels,
                         'tool_input': agent_thought.tool_input,
                         'created_at': int(self._message.created_at.timestamp()),
                         'message_files': agent_thought.files

api/core/application_manager.py

@@ -9,11 +9,12 @@ from core.app_runner.basic_app_runner import BasicApplicationRunner
 from core.app_runner.generate_task_pipeline import GenerateTaskPipeline
 from core.application_queue_manager import ApplicationQueueManager, ConversationTaskStoppedException, PublishFrom
 from core.entities.application_entities import (AdvancedChatPromptTemplateEntity,
-                                                AdvancedCompletionPromptTemplateEntity, AgentEntity, AgentToolEntity,
-                                                ApplicationGenerateEntity, AppOrchestrationConfigEntity, DatasetEntity,
+                                                AdvancedCompletionPromptTemplateEntity, AgentEntity, AgentPromptEntity,
+                                                AgentToolEntity, ApplicationGenerateEntity,
+                                                AppOrchestrationConfigEntity, DatasetEntity,
                                                 DatasetRetrieveConfigEntity, ExternalDataVariableEntity,
                                                 FileUploadEntity, InvokeFrom, ModelConfigEntity, PromptTemplateEntity,
-                                                SensitiveWordAvoidanceEntity, AgentPromptEntity)
+                                                SensitiveWordAvoidanceEntity)
 from core.entities.model_entities import ModelStatus
 from core.errors.error import ModelCurrentlyNotSupportError, ProviderTokenNotInitError, QuotaExceededError
 from core.file.file_obj import FileObj

api/core/application_queue_manager.py

@@ -4,10 +4,10 @@ from enum import Enum
 from typing import Any, Generator
 
 from core.entities.application_entities import InvokeFrom
-from core.entities.queue_entities import (AnnotationReplyEvent, AppQueueEvent, QueueAgentThoughtEvent, QueueErrorEvent,
-                                          QueueMessage, QueueMessageEndEvent, QueueMessageEvent,
-                                          QueueMessageReplaceEvent, QueuePingEvent, QueueRetrieverResourcesEvent,
-                                          QueueStopEvent, QueueMessageFileEvent, QueueAgentMessageEvent)
+from core.entities.queue_entities import (AnnotationReplyEvent, AppQueueEvent, QueueAgentMessageEvent,
+                                          QueueAgentThoughtEvent, QueueErrorEvent, QueueMessage, QueueMessageEndEvent,
+                                          QueueMessageEvent, QueueMessageFileEvent, QueueMessageReplaceEvent,
+                                          QueuePingEvent, QueueRetrieverResourcesEvent, QueueStopEvent)
 from core.model_runtime.entities.llm_entities import LLMResult, LLMResultChunk
 from extensions.ext_redis import redis_client
 from models.model import MessageAgentThought, MessageFile

api/core/callback_handler/agent_tool_callback_handler.py

@@ -1,9 +1,10 @@
 import os
 from typing import Any, Dict, Optional, Union
-from pydantic import BaseModel
 
 from langchain.callbacks.base import BaseCallbackHandler
 from langchain.input import print_text
+from pydantic import BaseModel
+
 
 class DifyAgentCallbackHandler(BaseCallbackHandler, BaseModel):
     """Callback Handler that prints to std out."""

api/core/data_loader/loader/unstructured/unstructured_ppt.py

@@ -1,5 +1,6 @@
 import logging
 from typing import List
+
 from langchain.document_loaders.base import BaseLoader
 from langchain.schema import Document
 

api/core/embedding/cached_embedding.py

@@ -8,9 +8,8 @@ from core.model_manager import ModelInstance
 from core.model_runtime.entities.model_entities import ModelPropertyKey
 from core.model_runtime.model_providers.__base.text_embedding_model import TextEmbeddingModel
 from extensions.ext_database import db
-from langchain.embeddings.base import Embeddings
-
 from extensions.ext_redis import redis_client
+from langchain.embeddings.base import Embeddings
 from libs import helper
 from models.dataset import Embedding
 from sqlalchemy.exc import IntegrityError

api/core/entities/application_entities.py

@@ -1,12 +1,11 @@
 from enum import Enum
-from typing import Optional, Any, cast, Literal, Union
-
-from pydantic import BaseModel
+from typing import Any, Literal, Optional, Union, cast
 
 from core.entities.provider_configuration import ProviderModelBundle
 from core.file.file_obj import FileObj
 from core.model_runtime.entities.message_entities import PromptMessageRole
 from core.model_runtime.entities.model_entities import AIModelEntity
+from pydantic import BaseModel
 
 
 class ModelConfigEntity(BaseModel):

api/core/entities/provider_configuration.py

@@ -153,8 +153,16 @@ class ProviderConfiguration(BaseModel):
 
         if provider_record:
             try:
-                original_credentials = json.loads(
-                    provider_record.encrypted_config) if provider_record.encrypted_config else {}
+                # fix origin data
+                if provider_record.encrypted_config:
+                    if not provider_record.encrypted_config.startswith("{"):
+                        original_credentials = {
+                            "openai_api_key": provider_record.encrypted_config
+                        }
+                    else:
+                        original_credentials = json.loads(provider_record.encrypted_config)
+                else:
+                    original_credentials = {}
             except JSONDecodeError:
                 original_credentials = {}
 

api/core/entities/provider_entities.py

@@ -9,6 +9,7 @@ from pydantic import BaseModel
 class QuotaUnit(Enum):
     TIMES = 'times'
     TOKENS = 'tokens'
+    CREDITS = 'credits'
 
 
 class SystemConfigurationStatus(Enum):

api/core/features/agent_runner.py

@@ -1,27 +1,26 @@
 import logging
-from typing import cast, Optional, List
-
-from langchain import WikipediaAPIWrapper
-from langchain.callbacks.base import BaseCallbackHandler
-from langchain.tools import BaseTool, WikipediaQueryRun, Tool
-from pydantic import BaseModel, Field
+from typing import List, Optional, cast
 
 from core.agent.agent.agent_llm_callback import AgentLLMCallback
-from core.agent.agent_executor import PlanningStrategy, AgentConfiguration, AgentExecutor
+from core.agent.agent_executor import AgentConfiguration, AgentExecutor, PlanningStrategy
 from core.application_queue_manager import ApplicationQueueManager
 from core.callback_handler.agent_loop_gather_callback_handler import AgentLoopGatherCallbackHandler
 from core.callback_handler.index_tool_callback_handler import DatasetIndexToolCallbackHandler
 from core.callback_handler.std_out_callback_handler import DifyStdOutCallbackHandler
-from core.entities.application_entities import ModelConfigEntity, InvokeFrom, \
-    AgentEntity, AgentToolEntity, AppOrchestrationConfigEntity
+from core.entities.application_entities import (AgentEntity, AgentToolEntity, AppOrchestrationConfigEntity, InvokeFrom,
+                                                ModelConfigEntity)
 from core.memory.token_buffer_memory import TokenBufferMemory
 from core.model_runtime.entities.model_entities import ModelFeature, ModelType
 from core.model_runtime.model_providers import model_provider_factory
 from core.model_runtime.model_providers.__base.large_language_model import LargeLanguageModel
 from core.tools.tool.dataset_retriever.dataset_retriever_tool import DatasetRetrieverTool
 from extensions.ext_database import db
+from langchain import WikipediaAPIWrapper
+from langchain.callbacks.base import BaseCallbackHandler
+from langchain.tools import BaseTool, Tool, WikipediaQueryRun
 from models.dataset import Dataset
 from models.model import Message
+from pydantic import BaseModel, Field
 
 logger = logging.getLogger(__name__)
 

api/core/features/assistant_base_runner.py

@@ -1,34 +1,32 @@
-import logging
 import json
-
-from typing import Optional, List, Tuple, Union
+import logging
 from datetime import datetime
 from mimetypes import guess_extension
+from typing import List, Optional, Tuple, Union, cast
 
 from core.app_runner.app_runner import AppRunner
-from extensions.ext_database import db
-
-from models.model import MessageAgentThought, Message, MessageFile
-from models.tools import ToolConversationVariables
-
-from core.tools.entities.tool_entities import ToolInvokeMessage, ToolInvokeMessageBinary, \
-    ToolRuntimeVariablePool, ToolParamter
-from core.tools.tool.tool import Tool
-from core.tools.tool_manager import ToolManager
-from core.tools.tool_file_manager import ToolFileManager
-from core.tools.tool.dataset_retriever_tool import DatasetRetrieverTool
-from core.app_runner.app_runner import AppRunner
+from core.application_queue_manager import ApplicationQueueManager
 from core.callback_handler.agent_tool_callback_handler import DifyAgentCallbackHandler
 from core.callback_handler.index_tool_callback_handler import DatasetIndexToolCallbackHandler
-from core.entities.application_entities import ModelConfigEntity, AgentEntity, AgentToolEntity
-from core.application_queue_manager import ApplicationQueueManager
-from core.memory.token_buffer_memory import TokenBufferMemory
-from core.entities.application_entities import ModelConfigEntity, \
-    AgentEntity, AppOrchestrationConfigEntity, ApplicationGenerateEntity, InvokeFrom
-from core.model_runtime.entities.message_entities import PromptMessage, PromptMessageTool
-from core.model_runtime.entities.llm_entities import LLMUsage
-from core.model_runtime.utils.encoders import jsonable_encoder
+from core.entities.application_entities import (AgentEntity, AgentToolEntity, ApplicationGenerateEntity,
+                                                AppOrchestrationConfigEntity, InvokeFrom, ModelConfigEntity)
 from core.file.message_file_parser import FileTransferMethod
+from core.memory.token_buffer_memory import TokenBufferMemory
+from core.model_manager import ModelInstance
+from core.model_runtime.entities.llm_entities import LLMUsage
+from core.model_runtime.entities.message_entities import PromptMessage, PromptMessageTool
+from core.model_runtime.entities.model_entities import ModelFeature
+from core.model_runtime.model_providers.__base.large_language_model import LargeLanguageModel
+from core.model_runtime.utils.encoders import jsonable_encoder
+from core.tools.entities.tool_entities import (ToolInvokeMessage, ToolInvokeMessageBinary, ToolParameter,
+                                               ToolRuntimeVariablePool)
+from core.tools.tool.dataset_retriever_tool import DatasetRetrieverTool
+from core.tools.tool.tool import Tool
+from core.tools.tool_file_manager import ToolFileManager
+from core.tools.tool_manager import ToolManager
+from extensions.ext_database import db
+from models.model import Message, MessageAgentThought, MessageFile
+from models.tools import ToolConversationVariables
 
 logger = logging.getLogger(__name__)
 
@@ -45,6 +43,7 @@ class BaseAssistantApplicationRunner(AppRunner):
                  prompt_messages: Optional[List[PromptMessage]] = None,
                  variables_pool: Optional[ToolRuntimeVariablePool] = None,
                  db_variables: Optional[ToolConversationVariables] = None,
+                 model_instance: ModelInstance = None
                  ) -> None:
         """
         Agent runner
@@ -71,6 +70,7 @@ class BaseAssistantApplicationRunner(AppRunner):
         self.history_prompt_messages = prompt_messages
         self.variables_pool = variables_pool
         self.db_variables_pool = db_variables
+        self.model_instance = model_instance
 
         # init callback
         self.agent_callback = DifyAgentCallbackHandler()
@@ -95,9 +95,17 @@ class BaseAssistantApplicationRunner(AppRunner):
             MessageAgentThought.message_id == self.message.id,
         ).count()
 
-    def _repacket_app_orchestration_config(self, app_orchestration_config: AppOrchestrationConfigEntity) -> AppOrchestrationConfigEntity:
+        # check if model supports stream tool call
+        llm_model = cast(LargeLanguageModel, model_instance.model_type_instance)
+        model_schema = llm_model.get_model_schema(model_instance.model, model_instance.credentials)
+        if model_schema and ModelFeature.STREAM_TOOL_CALL in (model_schema.features or []):
+            self.stream_tool_call = True
+        else:
+            self.stream_tool_call = False
+
+    def _repack_app_orchestration_config(self, app_orchestration_config: AppOrchestrationConfigEntity) -> AppOrchestrationConfigEntity:
         """
-        Repacket app orchestration config
+        Repack app orchestration config
         """
         if app_orchestration_config.prompt_template.simple_prompt_template is None:
             app_orchestration_config.prompt_template.simple_prompt_template = ''
@@ -113,7 +121,7 @@ class BaseAssistantApplicationRunner(AppRunner):
             if response.type == ToolInvokeMessage.MessageType.TEXT:
                 result += response.message
             elif response.type == ToolInvokeMessage.MessageType.LINK:
-                result += f"result link: {response.message}. please dirct user to check it."
+                result += f"result link: {response.message}. please tell user to check it."
             elif response.type == ToolInvokeMessage.MessageType.IMAGE_LINK or \
                  response.type == ToolInvokeMessage.MessageType.IMAGE:
                 result += f"image has been created and sent to user already, you should tell user to check it now."
@@ -128,7 +136,7 @@ class BaseAssistantApplicationRunner(AppRunner):
         """
         tool_entity = ToolManager.get_tool_runtime(
             provider_type=tool.provider_type, provider_name=tool.provider_id, tool_name=tool.tool_name, 
-            tanent_id=self.application_generate_entity.tenant_id,
+            tenant_id=self.application_generate_entity.tenant_id,
             agent_callback=self.agent_callback
         )
         tool_entity.load_variables(self.variables_pool)
@@ -172,20 +180,20 @@ class BaseAssistantApplicationRunner(AppRunner):
         for parameter in parameters:
             parameter_type = 'string'
             enum = []
-            if parameter.type == ToolParamter.ToolParameterType.STRING:
+            if parameter.type == ToolParameter.ToolParameterType.STRING:
                 parameter_type = 'string'
-            elif parameter.type == ToolParamter.ToolParameterType.BOOLEAN:
+            elif parameter.type == ToolParameter.ToolParameterType.BOOLEAN:
                 parameter_type = 'boolean'
-            elif parameter.type == ToolParamter.ToolParameterType.NUMBER:
+            elif parameter.type == ToolParameter.ToolParameterType.NUMBER:
                 parameter_type = 'number'
-            elif parameter.type == ToolParamter.ToolParameterType.SELECT:
+            elif parameter.type == ToolParameter.ToolParameterType.SELECT:
                 for option in parameter.options:
                     enum.append(option.value)
                 parameter_type = 'string'
             else:
                 raise ValueError(f"parameter type {parameter.type} is not supported")
             
-            if parameter.form == ToolParamter.ToolParameterForm.FORM:
+            if parameter.form == ToolParameter.ToolParameterForm.FORM:
                 # get tool parameter from form
                 tool_parameter_config = tool.tool_parameters.get(parameter.name)
                 if not tool_parameter_config:
@@ -194,7 +202,7 @@ class BaseAssistantApplicationRunner(AppRunner):
                     if not tool_parameter_config and parameter.required:
                         raise ValueError(f"tool parameter {parameter.name} not found in tool config")
                     
-                if parameter.type == ToolParamter.ToolParameterType.SELECT:
+                if parameter.type == ToolParameter.ToolParameterType.SELECT:
                     # check if tool_parameter_config in options
                     options = list(map(lambda x: x.value, parameter.options))
                     if tool_parameter_config not in options:
@@ -202,7 +210,7 @@ class BaseAssistantApplicationRunner(AppRunner):
                     
                 # convert tool parameter config to correct type
                 try:
-                    if parameter.type == ToolParamter.ToolParameterType.NUMBER:
+                    if parameter.type == ToolParameter.ToolParameterType.NUMBER:
                         # check if tool parameter is integer
                         if isinstance(tool_parameter_config, int):
                             tool_parameter_config = tool_parameter_config
@@ -213,11 +221,11 @@ class BaseAssistantApplicationRunner(AppRunner):
                                 tool_parameter_config = float(tool_parameter_config)
                             else:
                                 tool_parameter_config = int(tool_parameter_config)
-                    elif parameter.type == ToolParamter.ToolParameterType.BOOLEAN:
+                    elif parameter.type == ToolParameter.ToolParameterType.BOOLEAN:
                         tool_parameter_config = bool(tool_parameter_config)
-                    elif parameter.type not in [ToolParamter.ToolParameterType.SELECT, ToolParamter.ToolParameterType.STRING]:
+                    elif parameter.type not in [ToolParameter.ToolParameterType.SELECT, ToolParameter.ToolParameterType.STRING]:
                         tool_parameter_config = str(tool_parameter_config)
-                    elif parameter.type == ToolParamter.ToolParameterType:
+                    elif parameter.type == ToolParameter.ToolParameterType:
                         tool_parameter_config = str(tool_parameter_config)
                 except Exception as e:
                     raise ValueError(f"tool parameter {parameter.name} value {tool_parameter_config} is not correct type")
@@ -225,7 +233,7 @@ class BaseAssistantApplicationRunner(AppRunner):
                 # save tool parameter to tool entity memory
                 runtime_parameters[parameter.name] = tool_parameter_config
             
-            elif parameter.form == ToolParamter.ToolParameterForm.LLM:
+            elif parameter.form == ToolParameter.ToolParameterForm.LLM:
                 message_tool.parameters['properties'][parameter.name] = {
                     "type": parameter_type,
                     "description": parameter.llm_description or '',
@@ -279,20 +287,20 @@ class BaseAssistantApplicationRunner(AppRunner):
         for parameter in tool_runtime_parameters:
             parameter_type = 'string'
             enum = []
-            if parameter.type == ToolParamter.ToolParameterType.STRING:
+            if parameter.type == ToolParameter.ToolParameterType.STRING:
                 parameter_type = 'string'
-            elif parameter.type == ToolParamter.ToolParameterType.BOOLEAN:
+            elif parameter.type == ToolParameter.ToolParameterType.BOOLEAN:
                 parameter_type = 'boolean'
-            elif parameter.type == ToolParamter.ToolParameterType.NUMBER:
+            elif parameter.type == ToolParameter.ToolParameterType.NUMBER:
                 parameter_type = 'number'
-            elif parameter.type == ToolParamter.ToolParameterType.SELECT:
+            elif parameter.type == ToolParameter.ToolParameterType.SELECT:
                 for option in parameter.options:
                     enum.append(option.value)
                 parameter_type = 'string'
             else:
                 raise ValueError(f"parameter type {parameter.type} is not supported")
         
-            if parameter.form == ToolParamter.ToolParameterForm.LLM:
+            if parameter.form == ToolParameter.ToolParameterForm.LLM:
                 prompt_tool.parameters['properties'][parameter.name] = {
                     "type": parameter_type,
                     "description": parameter.llm_description or '',
@@ -396,6 +404,7 @@ class BaseAssistantApplicationRunner(AppRunner):
             message_chain_id=None,
             thought='',
             tool=tool_name,
+            tool_labels_str='{}',
             tool_input=tool_input,
             message=message,
             message_token=0,
@@ -469,6 +478,21 @@ class BaseAssistantApplicationRunner(AppRunner):
             agent_thought.tokens = llm_usage.total_tokens
             agent_thought.total_price = llm_usage.total_price
 
+        # check if tool labels is not empty
+        labels = agent_thought.tool_labels or {}
+        tools = agent_thought.tool.split(';') if agent_thought.tool else []
+        for tool in tools:
+            if not tool:
+                continue
+            if tool not in labels:
+                tool_label = ToolManager.get_tool_label(tool)
+                if tool_label:
+                    labels[tool] = tool_label.to_dict()
+                else:
+                    labels[tool] = {'en_US': tool, 'zh_Hans': tool}
+
+        agent_thought.tool_labels_str = json.dumps(labels)
+
         db.session.commit()
 
     def get_history_prompt_messages(self) -> List[PromptMessage]:

api/core/features/assistant_cot_runner.py

@@ -1,27 +1,23 @@
 import json
 import logging
 import re
-from typing import Literal, Union, Generator, Dict, List
+from typing import Dict, Generator, List, Literal, Union
 
-from core.entities.application_entities import AgentPromptEntity, AgentScratchpadUnit
 from core.application_queue_manager import PublishFrom
-from core.model_runtime.utils.encoders import jsonable_encoder
-from core.model_runtime.entities.message_entities import PromptMessageTool, PromptMessage, \
-    UserPromptMessage, SystemPromptMessage, AssistantPromptMessage
-from core.model_runtime.entities.llm_entities import LLMResult, LLMUsage, LLMResultChunk, LLMResultChunkDelta
-from core.model_manager import ModelInstance
-
-from core.tools.errors import ToolInvokeError, ToolNotFoundError, \
-    ToolNotSupportedError, ToolProviderNotFoundError, ToolParamterValidationError, \
-          ToolProviderCredentialValidationError
-
+from core.entities.application_entities import AgentPromptEntity, AgentScratchpadUnit
 from core.features.assistant_base_runner import BaseAssistantApplicationRunner
-
+from core.model_manager import ModelInstance
+from core.model_runtime.entities.llm_entities import LLMResult, LLMResultChunk, LLMResultChunkDelta, LLMUsage
+from core.model_runtime.entities.message_entities import (AssistantPromptMessage, PromptMessage, PromptMessageTool,
+                                                          SystemPromptMessage, UserPromptMessage)
+from core.model_runtime.utils.encoders import jsonable_encoder
+from core.tools.errors import (ToolInvokeError, ToolNotFoundError, ToolNotSupportedError, ToolParameterValidationError,
+                               ToolProviderCredentialValidationError, ToolProviderNotFoundError)
 from models.model import Conversation, Message
 
+
 class AssistantCotApplicationRunner(BaseAssistantApplicationRunner):
-    def run(self, model_instance: ModelInstance,
-        conversation: Conversation,
+    def run(self, conversation: Conversation,
         message: Message,
         query: str,
     ) -> Union[Generator, LLMResult]:
@@ -29,7 +25,7 @@ class AssistantCotApplicationRunner(BaseAssistantApplicationRunner):
         Run Cot agent application
         """
         app_orchestration_config = self.app_orchestration_config
-        self._repacket_app_orchestration_config(app_orchestration_config)
+        self._repack_app_orchestration_config(app_orchestration_config)
 
         agent_scratchpad: List[AgentScratchpadUnit] = []
 
@@ -72,7 +68,7 @@ class AssistantCotApplicationRunner(BaseAssistantApplicationRunner):
         }
         final_answer = ''
 
-        def increse_usage(final_llm_usage_dict: Dict[str, LLMUsage], usage: LLMUsage):
+        def increase_usage(final_llm_usage_dict: Dict[str, LLMUsage], usage: LLMUsage):
             if not final_llm_usage_dict['usage']:
                 final_llm_usage_dict['usage'] = usage
             else:
@@ -82,6 +78,8 @@ class AssistantCotApplicationRunner(BaseAssistantApplicationRunner):
                 llm_usage.prompt_price += usage.prompt_price
                 llm_usage.completion_price += usage.completion_price
 
+        model_instance = self.model_instance
+
         while function_call_state and iteration_step <= max_iteration_steps:
             # continue to run until there is not any tool call
             function_call_state = False
@@ -104,7 +102,7 @@ class AssistantCotApplicationRunner(BaseAssistantApplicationRunner):
                 self.queue_manager.publish_agent_thought(agent_thought, PublishFrom.APPLICATION_MANAGER)
 
             # update prompt messages
-            prompt_messages = self._originze_cot_prompt_messages(
+            prompt_messages = self._organize_cot_prompt_messages(
                 mode=app_orchestration_config.model_config.mode,
                 prompt_messages=prompt_messages,
                 tools=prompt_messages_tools,
@@ -137,7 +135,7 @@ class AssistantCotApplicationRunner(BaseAssistantApplicationRunner):
                         
             # get llm usage
             if llm_result.usage:
-                increse_usage(llm_usage, llm_result.usage)
+                increase_usage(llm_usage, llm_result.usage)
             
             # publish agent thought if it's first iteration
             if iteration_step == 1:
@@ -207,7 +205,7 @@ class AssistantCotApplicationRunner(BaseAssistantApplicationRunner):
                         try:
                             tool_response = tool_instance.invoke(
                                 user_id=self.user_id, 
-                                tool_paramters=tool_call_args if isinstance(tool_call_args, dict) else json.loads(tool_call_args)
+                                tool_parameters=tool_call_args if isinstance(tool_call_args, dict) else json.loads(tool_call_args)
                             )
                             # transform tool response to llm friendly response
                             tool_response = self.transform_tool_invoke_messages(tool_response)
@@ -225,15 +223,15 @@ class AssistantCotApplicationRunner(BaseAssistantApplicationRunner):
 
                             message_file_ids = [message_file.id for message_file, _ in message_files]
                         except ToolProviderCredentialValidationError as e:
-                            error_response = f"Plese check your tool provider credentials"
+                            error_response = f"Please check your tool provider credentials"
                         except (
                             ToolNotFoundError, ToolNotSupportedError, ToolProviderNotFoundError
                         ) as e:
                             error_response = f"there is not a tool named {tool_call_name}"
                         except (
-                            ToolParamterValidationError
+                            ToolParameterValidationError
                         ) as e:
-                            error_response = f"tool paramters validation error: {e}, please check your tool paramters"
+                            error_response = f"tool parameters validation error: {e}, please check your tool parameters"
                         except ToolInvokeError as e:
                             error_response = f"tool invoke error: {e}"
                         except Exception as e:
@@ -298,7 +296,7 @@ class AssistantCotApplicationRunner(BaseAssistantApplicationRunner):
             message=AssistantPromptMessage(
                 content=final_answer
             ),
-            usage=llm_usage['usage'],
+            usage=llm_usage['usage'] if llm_usage['usage'] else LLMUsage.empty_usage(),
             system_fingerprint=''
         ), PublishFrom.APPLICATION_MANAGER)
 
@@ -390,7 +388,7 @@ class AssistantCotApplicationRunner(BaseAssistantApplicationRunner):
                     # remove Action: xxx from agent thought
                     agent_thought = re.sub(r'Action:.*', '', agent_thought, flags=re.IGNORECASE)
 
-                    if action_name and action_input:
+                    if action_name and action_input is not None:
                         return AgentScratchpadUnit(
                             agent_response=content,
                             thought=agent_thought,
@@ -468,7 +466,7 @@ class AssistantCotApplicationRunner(BaseAssistantApplicationRunner):
             if not next_iteration.find("{{observation}}") >= 0:
                 raise ValueError("{{observation}} is required in next_iteration")
             
-    def _convert_strachpad_list_to_str(self, agent_scratchpad: List[AgentScratchpadUnit]) -> str:
+    def _convert_scratchpad_list_to_str(self, agent_scratchpad: List[AgentScratchpadUnit]) -> str:
         """
             convert agent scratchpad list to str
         """
@@ -480,7 +478,7 @@ class AssistantCotApplicationRunner(BaseAssistantApplicationRunner):
 
         return result
     
-    def _originze_cot_prompt_messages(self, mode: Literal["completion", "chat"],
+    def _organize_cot_prompt_messages(self, mode: Literal["completion", "chat"],
                                       prompt_messages: List[PromptMessage],
                                       tools: List[PromptMessageTool], 
                                       agent_scratchpad: List[AgentScratchpadUnit],
@@ -489,7 +487,7 @@ class AssistantCotApplicationRunner(BaseAssistantApplicationRunner):
                                       input: str,
         ) -> List[PromptMessage]:
         """
-            originze chain of thought prompt messages, a standard prompt message is like:
+            organize chain of thought prompt messages, a standard prompt message is like:
                 Respond to the human as helpfully and accurately as possible. 
 
                 {{instruction}}
@@ -527,7 +525,7 @@ class AssistantCotApplicationRunner(BaseAssistantApplicationRunner):
                                      .replace("{{tools}}", tools_str) \
                                      .replace("{{tool_names}}", tool_names)
 
-        # originze prompt messages
+        # organize prompt messages
         if mode == "chat":
             # override system message
             overrided = False
@@ -558,7 +556,7 @@ class AssistantCotApplicationRunner(BaseAssistantApplicationRunner):
             return prompt_messages
         elif mode == "completion":
             # parse agent scratchpad
-            agent_scratchpad_str = self._convert_strachpad_list_to_str(agent_scratchpad)
+            agent_scratchpad_str = self._convert_scratchpad_list_to_str(agent_scratchpad)
             # parse prompt messages
             return [UserPromptMessage(
                 content=first_prompt.replace("{{instruction}}", instruction)

api/core/features/assistant_fc_runner.py

@@ -1,27 +1,21 @@
 import json
 import logging
+from typing import Any, Dict, Generator, List, Tuple, Union
 
-from typing import Union, Generator, Dict, Any, Tuple, List
-
-from core.model_runtime.entities.message_entities import PromptMessage, UserPromptMessage,\
-      SystemPromptMessage, AssistantPromptMessage, ToolPromptMessage, PromptMessageTool
-from core.model_runtime.entities.llm_entities import LLMResultChunk, LLMResult, LLMUsage
-from core.model_manager import ModelInstance
 from core.application_queue_manager import PublishFrom
-
-from core.tools.errors import ToolInvokeError, ToolNotFoundError, \
-    ToolNotSupportedError, ToolProviderNotFoundError, ToolParamterValidationError, \
-          ToolProviderCredentialValidationError
-
 from core.features.assistant_base_runner import BaseAssistantApplicationRunner
-
+from core.model_manager import ModelInstance
+from core.model_runtime.entities.llm_entities import LLMResult, LLMResultChunk, LLMResultChunkDelta, LLMUsage
+from core.model_runtime.entities.message_entities import (AssistantPromptMessage, PromptMessage, PromptMessageTool,
+                                                          SystemPromptMessage, ToolPromptMessage, UserPromptMessage)
+from core.tools.errors import (ToolInvokeError, ToolNotFoundError, ToolNotSupportedError, ToolParameterValidationError,
+                               ToolProviderCredentialValidationError, ToolProviderNotFoundError)
 from models.model import Conversation, Message, MessageAgentThought
 
 logger = logging.getLogger(__name__)
 
 class AssistantFunctionCallApplicationRunner(BaseAssistantApplicationRunner):
-    def run(self, model_instance: ModelInstance,
-                conversation: Conversation,
+    def run(self, conversation: Conversation,
                 message: Message,
                 query: str,
     ) -> Generator[LLMResultChunk, None, None]:
@@ -81,6 +75,8 @@ class AssistantFunctionCallApplicationRunner(BaseAssistantApplicationRunner):
                 llm_usage.prompt_price += usage.prompt_price
                 llm_usage.completion_price += usage.completion_price
 
+        model_instance = self.model_instance
+
         while function_call_state and iteration_step <= max_iteration_steps:
             function_call_state = False
 
@@ -96,17 +92,16 @@ class AssistantFunctionCallApplicationRunner(BaseAssistantApplicationRunner):
                 tool_input='',
                 messages_ids=message_file_ids
             )
-            self.queue_manager.publish_agent_thought(agent_thought, PublishFrom.APPLICATION_MANAGER)
 
             # recale llm max tokens
             self.recale_llm_max_tokens(self.model_config, prompt_messages)
             # invoke model
-            chunks: Generator[LLMResultChunk, None, None] = model_instance.invoke_llm(
+            chunks: Union[Generator[LLMResultChunk, None, None], LLMResult] = model_instance.invoke_llm(
                 prompt_messages=prompt_messages,
                 model_parameters=app_orchestration_config.model_config.parameters,
                 tools=prompt_messages_tools,
                 stop=app_orchestration_config.model_config.stop,
-                stream=True,
+                stream=self.stream_tool_call,
                 user=self.user_id,
                 callbacks=[],
             )
@@ -122,11 +117,45 @@ class AssistantFunctionCallApplicationRunner(BaseAssistantApplicationRunner):
 
             current_llm_usage = None
 
-            for chunk in chunks:
+            if self.stream_tool_call:
+                is_first_chunk = True
+                for chunk in chunks:
+                    if is_first_chunk:
+                        self.queue_manager.publish_agent_thought(agent_thought, PublishFrom.APPLICATION_MANAGER)
+                        is_first_chunk = False
+                    # check if there is any tool call
+                    if self.check_tool_calls(chunk):
+                        function_call_state = True
+                        tool_calls.extend(self.extract_tool_calls(chunk))
+                        tool_call_names = ';'.join([tool_call[1] for tool_call in tool_calls])
+                        try:
+                            tool_call_inputs = json.dumps({
+                                tool_call[1]: tool_call[2] for tool_call in tool_calls
+                            }, ensure_ascii=False)
+                        except json.JSONDecodeError as e:
+                            # ensure ascii to avoid encoding error
+                            tool_call_inputs = json.dumps({
+                                tool_call[1]: tool_call[2] for tool_call in tool_calls
+                            })
+
+                    if chunk.delta.message and chunk.delta.message.content:
+                        if isinstance(chunk.delta.message.content, list):
+                            for content in chunk.delta.message.content:
+                                response += content.data
+                        else:
+                            response += chunk.delta.message.content
+
+                    if chunk.delta.usage:
+                        increase_usage(llm_usage, chunk.delta.usage)
+                        current_llm_usage = chunk.delta.usage
+
+                    yield chunk
+            else:
+                result: LLMResult = chunks
                 # check if there is any tool call
-                if self.check_tool_calls(chunk):
+                if self.check_blocking_tool_calls(result):
                     function_call_state = True
-                    tool_calls.extend(self.extract_tool_calls(chunk))
+                    tool_calls.extend(self.extract_blocking_tool_calls(result))
                     tool_call_names = ';'.join([tool_call[1] for tool_call in tool_calls])
                     try:
                         tool_call_inputs = json.dumps({
@@ -138,18 +167,46 @@ class AssistantFunctionCallApplicationRunner(BaseAssistantApplicationRunner):
                             tool_call[1]: tool_call[2] for tool_call in tool_calls
                         })
 
-                if chunk.delta.message and chunk.delta.message.content:
-                    if isinstance(chunk.delta.message.content, list):
-                        for content in chunk.delta.message.content:
+                if result.usage:
+                    increase_usage(llm_usage, result.usage)
+                    current_llm_usage = result.usage
+
+                if result.message and result.message.content:
+                    if isinstance(result.message.content, list):
+                        for content in result.message.content:
                             response += content.data
                     else:
-                        response += chunk.delta.message.content
+                        response += result.message.content
 
-                if chunk.delta.usage:
-                    increase_usage(llm_usage, chunk.delta.usage)
-                    current_llm_usage = chunk.delta.usage
+                if not result.message.content:
+                    result.message.content = ''
 
-                yield chunk
+                self.queue_manager.publish_agent_thought(agent_thought, PublishFrom.APPLICATION_MANAGER)
+                
+                yield LLMResultChunk(
+                    model=model_instance.model,
+                    prompt_messages=result.prompt_messages,
+                    system_fingerprint=result.system_fingerprint,
+                    delta=LLMResultChunkDelta(
+                        index=0,
+                        message=result.message,
+                        usage=result.usage,
+                    )
+                )
+
+            if tool_calls:
+                prompt_messages.append(AssistantPromptMessage(
+                    content='',
+                    name='',
+                    tool_calls=[AssistantPromptMessage.ToolCall(
+                        id=tool_call[0],
+                        type='function',
+                        function=AssistantPromptMessage.ToolCall.ToolCallFunction(
+                            name=tool_call[1],
+                            arguments=json.dumps(tool_call[2], ensure_ascii=False)
+                        )
+                    ) for tool_call in tool_calls]
+                ))
 
             # save thought
             self.save_agent_thought(
@@ -167,6 +224,12 @@ class AssistantFunctionCallApplicationRunner(BaseAssistantApplicationRunner):
             
             final_answer += response + '\n'
 
+            # update prompt messages
+            if response.strip():
+                prompt_messages.append(AssistantPromptMessage(
+                    content=response,
+                ))
+            
             # call tools
             tool_responses = []
             for tool_call_id, tool_call_name, tool_call_args in tool_calls:
@@ -184,7 +247,7 @@ class AssistantFunctionCallApplicationRunner(BaseAssistantApplicationRunner):
                     try:
                         tool_invoke_message = tool_instance.invoke(
                             user_id=self.user_id, 
-                            tool_paramters=tool_call_args, 
+                            tool_parameters=tool_call_args, 
                         )
                         # transform tool invoke message to get LLM friendly message
                         tool_invoke_message = self.transform_tool_invoke_messages(tool_invoke_message)
@@ -203,15 +266,15 @@ class AssistantFunctionCallApplicationRunner(BaseAssistantApplicationRunner):
                             message_file_ids.append(message_file.id)
                             
                     except ToolProviderCredentialValidationError as e:
-                        error_response = f"Plese check your tool provider credentials"
+                        error_response = f"Please check your tool provider credentials"
                     except (
                         ToolNotFoundError, ToolNotSupportedError, ToolProviderNotFoundError
                     ) as e:
                         error_response = f"there is not a tool named {tool_call_name}"
                     except (
-                        ToolParamterValidationError
+                        ToolParameterValidationError
                     ) as e:
-                        error_response = f"tool paramters validation error: {e}, please check your tool paramters"
+                        error_response = f"tool parameters validation error: {e}, please check your tool parameters"
                     except ToolInvokeError as e:
                         error_response = f"tool invoke error: {e}"
                     except Exception as e:
@@ -256,12 +319,6 @@ class AssistantFunctionCallApplicationRunner(BaseAssistantApplicationRunner):
                 )
                 self.queue_manager.publish_agent_thought(agent_thought, PublishFrom.APPLICATION_MANAGER)
 
-            # update prompt messages
-            if response.strip():
-                prompt_messages.append(AssistantPromptMessage(
-                    content=response,
-                ))
-
             # update prompt tool
             for prompt_tool in prompt_messages_tools:
                 self.update_prompt_message_tool(tool_instances[prompt_tool.name], prompt_tool)
@@ -276,7 +333,7 @@ class AssistantFunctionCallApplicationRunner(BaseAssistantApplicationRunner):
             message=AssistantPromptMessage(
                 content=final_answer,
             ),
-            usage=llm_usage['usage'],
+            usage=llm_usage['usage'] if llm_usage['usage'] else LLMUsage.empty_usage(),
             system_fingerprint=''
         ), PublishFrom.APPLICATION_MANAGER)
 
@@ -287,6 +344,14 @@ class AssistantFunctionCallApplicationRunner(BaseAssistantApplicationRunner):
         if llm_result_chunk.delta.message.tool_calls:
             return True
         return False
+    
+    def check_blocking_tool_calls(self, llm_result: LLMResult) -> bool:
+        """
+        Check if there is any blocking tool call in llm result
+        """
+        if llm_result.message.tool_calls:
+            return True
+        return False
 
     def extract_tool_calls(self, llm_result_chunk: LLMResultChunk) -> Union[None, List[Tuple[str, str, Dict[str, Any]]]]:
         """
@@ -304,6 +369,23 @@ class AssistantFunctionCallApplicationRunner(BaseAssistantApplicationRunner):
             ))
 
         return tool_calls
+    
+    def extract_blocking_tool_calls(self, llm_result: LLMResult) -> Union[None, List[Tuple[str, str, Dict[str, Any]]]]:
+        """
+        Extract blocking tool calls from llm result
+
+        Returns:
+            List[Tuple[str, str, Dict[str, Any]]]: [(tool_call_id, tool_call_name, tool_call_args)]
+        """
+        tool_calls = []
+        for prompt_message in llm_result.message.tool_calls:
+            tool_calls.append((
+                prompt_message.id,
+                prompt_message.function.name,
+                json.loads(prompt_message.function.arguments),
+            ))
+
+        return tool_calls
 
     def organize_prompt_messages(self, prompt_template: str,
                                  query: str = None, 

api/core/file/message_file_parser.py

@@ -1,11 +1,11 @@
 from typing import Dict, List, Optional, Union
 
 import requests
-from core.file.file_obj import FileObj, FileTransferMethod, FileType, FileBelongsTo
-from services.file_service import IMAGE_EXTENSIONS
+from core.file.file_obj import FileBelongsTo, FileObj, FileTransferMethod, FileType
 from extensions.ext_database import db
 from models.account import Account
 from models.model import AppModelConfig, EndUser, MessageFile, UploadFile
+from services.file_service import IMAGE_EXTENSIONS
 
 
 class MessageFileParser:

api/core/hosting_configuration.py

@@ -2,7 +2,7 @@ from typing import Optional
 
 from core.entities.provider_entities import QuotaUnit, RestrictModel
 from core.model_runtime.entities.model_entities import ModelType
-from flask import Flask, Config
+from flask import Config, Flask
 from models.provider import ProviderQuotaType
 from pydantic import BaseModel
 
@@ -20,10 +20,6 @@ class TrialHostingQuota(HostingQuota):
 
 class PaidHostingQuota(HostingQuota):
     quota_type: ProviderQuotaType = ProviderQuotaType.PAID
-    stripe_price_id: str = None
-    increase_quota: int = 1
-    min_quantity: int = 20
-    max_quantity: int = 100
 
 
 class FreeHostingQuota(HostingQuota):
@@ -102,7 +98,7 @@ class HostingConfiguration:
         )
 
     def init_openai(self, app_config: Config) -> HostingProvider:
-        quota_unit = QuotaUnit.TIMES
+        quota_unit = QuotaUnit.CREDITS
         quotas = []
 
         if app_config.get("HOSTED_OPENAI_TRIAL_ENABLED"):
@@ -114,6 +110,8 @@ class HostingConfiguration:
                     RestrictModel(model="gpt-3.5-turbo-1106", model_type=ModelType.LLM),
                     RestrictModel(model="gpt-3.5-turbo-instruct", model_type=ModelType.LLM),
                     RestrictModel(model="gpt-3.5-turbo-16k", model_type=ModelType.LLM),
+                    RestrictModel(model="gpt-3.5-turbo-16k-0613", model_type=ModelType.LLM),
+                    RestrictModel(model="gpt-3.5-turbo-0613", model_type=ModelType.LLM),
                     RestrictModel(model="text-davinci-003", model_type=ModelType.LLM),
                     RestrictModel(model="whisper-1", model_type=ModelType.SPEECH2TEXT),
                 ]
@@ -122,10 +120,20 @@ class HostingConfiguration:
 
         if app_config.get("HOSTED_OPENAI_PAID_ENABLED"):
             paid_quota = PaidHostingQuota(
-                stripe_price_id=app_config.get("HOSTED_OPENAI_PAID_STRIPE_PRICE_ID"),
-                increase_quota=int(app_config.get("HOSTED_OPENAI_PAID_INCREASE_QUOTA", "1")),
-                min_quantity=int(app_config.get("HOSTED_OPENAI_PAID_MIN_QUANTITY", "1")),
-                max_quantity=int(app_config.get("HOSTED_OPENAI_PAID_MAX_QUANTITY", "1"))
+                restrict_models=[
+                    RestrictModel(model="gpt-4", model_type=ModelType.LLM),
+                    RestrictModel(model="gpt-4-turbo-preview", model_type=ModelType.LLM),
+                    RestrictModel(model="gpt-4-32k", model_type=ModelType.LLM),
+                    RestrictModel(model="gpt-4-1106-preview", model_type=ModelType.LLM),
+                    RestrictModel(model="gpt-3.5-turbo", model_type=ModelType.LLM),
+                    RestrictModel(model="gpt-3.5-turbo-16k", model_type=ModelType.LLM),
+                    RestrictModel(model="gpt-3.5-turbo-16k-0613", model_type=ModelType.LLM),
+                    RestrictModel(model="gpt-3.5-turbo-1106", model_type=ModelType.LLM),
+                    RestrictModel(model="gpt-4-0125-preview", model_type=ModelType.LLM),
+                    RestrictModel(model="gpt-3.5-turbo-0613", model_type=ModelType.LLM),
+                    RestrictModel(model="gpt-3.5-turbo-instruct", model_type=ModelType.LLM),
+                    RestrictModel(model="text-davinci-003", model_type=ModelType.LLM),
+                ]
             )
             quotas.append(paid_quota)
 
@@ -164,12 +172,7 @@ class HostingConfiguration:
             quotas.append(trial_quota)
 
         if app_config.get("HOSTED_ANTHROPIC_PAID_ENABLED"):
-            paid_quota = PaidHostingQuota(
-                stripe_price_id=app_config.get("HOSTED_ANTHROPIC_PAID_STRIPE_PRICE_ID"),
-                increase_quota=int(app_config.get("HOSTED_ANTHROPIC_PAID_INCREASE_QUOTA", "1000000")),
-                min_quantity=int(app_config.get("HOSTED_ANTHROPIC_PAID_MIN_QUANTITY", "20")),
-                max_quantity=int(app_config.get("HOSTED_ANTHROPIC_PAID_MAX_QUANTITY", "100"))
-            )
+            paid_quota = PaidHostingQuota()
             quotas.append(paid_quota)
 
         if len(quotas) > 0:

api/core/indexing_runner.py

@@ -13,7 +13,7 @@ from core.docstore.dataset_docstore import DatasetDocumentStore
 from core.errors.error import ProviderTokenNotInitError
 from core.generator.llm_generator import LLMGenerator
 from core.index.index import IndexBuilder
-from core.model_manager import ModelManager, ModelInstance
+from core.model_manager import ModelInstance, ModelManager
 from core.model_runtime.entities.model_entities import ModelType, PriceType
 from core.model_runtime.model_providers.__base.large_language_model import LargeLanguageModel
 from core.model_runtime.model_providers.__base.text_embedding_model import TextEmbeddingModel
@@ -562,7 +562,7 @@ class IndexingRunner:
 
             character_splitter = FixedRecursiveCharacterTextSplitter.from_encoder(
                 chunk_size=segmentation["max_tokens"],
-                chunk_overlap=0,
+                chunk_overlap=segmentation.get('chunk_overlap', 0),
                 fixed_separator=separator,
                 separators=["\n\n", "。", ".", " ", ""],
                 embedding_model_instance=embedding_model_instance
@@ -571,7 +571,7 @@ class IndexingRunner:
             # Automatic segmentation
             character_splitter = EnhanceRecursiveCharacterTextSplitter.from_encoder(
                 chunk_size=DatasetProcessRule.AUTOMATIC_RULES['segmentation']['max_tokens'],
-                chunk_overlap=0,
+                chunk_overlap=DatasetProcessRule.AUTOMATIC_RULES['segmentation']['chunk_overlap'],
                 separators=["\n\n", "。", ".", " ", ""],
                 embedding_model_instance=embedding_model_instance
             )
@@ -655,7 +655,9 @@ class IndexingRunner:
                     else:
                         page_content = page_content
                     document_node.page_content = page_content
-                    split_documents.append(document_node)
+
+                    if document_node.page_content:
+                        split_documents.append(document_node)
             all_documents.extend(split_documents)
         # processing qa document
         if document_form == 'qa_model':

api/core/model_manager.py

@@ -12,8 +12,8 @@ from core.model_runtime.model_providers.__base.large_language_model import Large
 from core.model_runtime.model_providers.__base.moderation_model import ModerationModel
 from core.model_runtime.model_providers.__base.rerank_model import RerankModel
 from core.model_runtime.model_providers.__base.speech2text_model import Speech2TextModel
-from core.model_runtime.model_providers.__base.tts_model import TTSModel
 from core.model_runtime.model_providers.__base.text_embedding_model import TextEmbeddingModel
+from core.model_runtime.model_providers.__base.tts_model import TTSModel
 from core.provider_manager import ProviderManager
 
 

api/core/model_runtime/README.md

@@ -13,6 +13,7 @@ This module provides the interface for invoking and authenticating various model
   - `Text Embedding Model` - Text Embedding, pre-computed tokens capability
   - `Rerank Model` - Segment Rerank capability
   - `Speech-to-text Model` - Speech to text capability
+  - `Text-to-speech Model` - Text to speech capability
   - `Moderation` - Moderation capability
 
 - Model provider display

api/core/model_runtime/README_CN.md

@@ -13,6 +13,7 @@
   - `Text Embedidng Model` - 文本 Embedding ，预计算 tokens 能力
   - `Rerank Model` - 分段 Rerank 能力
   - `Speech-to-text Model` - 语音转文本能力
+  - `Text-to-speech Model` - 文本转语音能力
   - `Moderation` - Moderation 能力
 
 - 模型供应商展示

api/core/model_runtime/docs/en_US/interfaces.md

@@ -299,9 +299,7 @@ Inherit the `__base.speech2text_model.Speech2TextModel` base class and implement
 - Invoke Invocation
 
   ```python
-  def _invoke(self, model: str, credentials: dict,
-              file: IO[bytes], user: Optional[str] = None) \
-          -> str:
+  def _invoke(self, model: str, credentials: dict, file: IO[bytes], user: Optional[str] = None) -> str:
       """
       Invoke large language model
   
@@ -331,6 +329,46 @@ Inherit the `__base.speech2text_model.Speech2TextModel` base class and implement
 
     The string after speech-to-text conversion.
 
+### Text2speech
+
+Inherit the `__base.text2speech_model.Text2SpeechModel` base class and implement the following interfaces:
+
+- Invoke Invocation
+
+  ```python
+  def _invoke(elf, model: str, credentials: dict, content_text: str, streaming: bool, user: Optional[str] = None):
+      """
+      Invoke large language model
+  
+      :param model: model name
+      :param credentials: model credentials
+      :param content_text: text content to be translated
+      :param streaming: output is streaming
+      :param user: unique user id
+      :return: translated audio file
+      """	
+  ```
+
+  - Parameters：
+
+    - `model` (string) Model name
+
+    - `credentials` (object) Credential information
+
+      The parameters of credential information are defined by either the `provider_credential_schema` or `model_credential_schema` in the provider's YAML configuration file. Inputs such as `api_key` are included.
+
+    - `content_text` (string) The text content that needs to be converted
+
+    - `streaming` (bool) Whether to stream output
+
+    - `user` (string) [optional] Unique identifier of the user
+
+      This can help the provider monitor and detect abusive behavior.
+
+  - Returns：
+
+    Text converted speech stream。
+
 ### Moderation
 
 Inherit the `__base.moderation_model.ModerationModel` base class and implement the following interfaces:

api/core/model_runtime/docs/en_US/provider_scale_out.md

@@ -94,6 +94,7 @@ The currently supported model types are as follows:
 - `text_embedding` Text Embedding model
 - `rerank` Rerank model
 - `speech2text` Speech to text
+- `tts` Text to speech
 - `moderation` Moderation
 
 Continuing with `Anthropic` as an example, since `Anthropic` only supports LLM, we create a `module` named `llm` in `model_providers.anthropic`.

api/core/model_runtime/docs/en_US/schema.md

@@ -47,6 +47,10 @@
   - `max_chunks` (int) Maximum number of chunks (available for model types `text-embedding`, `moderation`)
   - `file_upload_limit` (int) Maximum file upload limit, in MB (available for model type `speech2text`)
   - `supported_file_extensions` (string) Supported file extension formats, e.g., mp3, mp4 (available for model type `speech2text`)
+  - `default_voice` (string)  default voice, e.g.：alloy,echo,fable,onyx,nova,shimmer（available for model type `tts`）
+  - `word_limit` (int)  Single conversion word limit, paragraphwise by default（available for model type `tts`）
+  - `audio_type` (string)  Support audio file extension format, e.g.：mp3,wav（available for model type `tts`）
+  - `max_workers` (int)  Number of concurrent workers supporting text and audio conversion（available for model type`tts`）
   - `max_characters_per_chunk` (int) Maximum characters per chunk (available for model type `moderation`)
 - `parameter_rules` (array[[ParameterRule](#ParameterRule)]) [optional] Model invocation parameter rules
 - `pricing` ([PriceConfig](#PriceConfig)) [optional] Pricing information
@@ -58,6 +62,7 @@
 - `text-embedding` Text Embedding model
 - `rerank` Rerank model
 - `speech2text` Speech to text
+- `tts` Text to speech
 - `moderation` Moderation
 
 ### ConfigurateMethod

api/core/model_runtime/docs/zh_Hans/customizable_model_scale_out.md

@@ -23,6 +23,7 @@
 - `text_embedding` 文本 Embedding 模型
 - `rerank` Rerank 模型
 - `speech2text` 语音转文字
+- `tts` 文字转语音
 - `moderation` 审查
 
 `Xinference`支持`LLM`和`Text Embedding`和Rerank，那么我们开始编写`xinference.yaml`。

api/core/model_runtime/docs/zh_Hans/interfaces.md

@@ -369,6 +369,46 @@ class XinferenceProvider(Provider):
 
     语音转换后的字符串。
 
+### Text2speech
+
+继承 `__base.text2speech_model.Text2SpeechModel` 基类，实现以下接口：
+
+- Invoke 调用
+
+  ```python
+  def _invoke(elf, model: str, credentials: dict, content_text: str, streaming: bool, user: Optional[str] = None):
+      """
+      Invoke large language model
+  
+      :param model: model name
+      :param credentials: model credentials
+      :param content_text: text content to be translated
+      :param streaming: output is streaming
+      :param user: unique user id
+      :return: translated audio file
+      """	
+  ```
+
+  - 参数：
+
+    - `model` (string) 模型名称
+
+    - `credentials` (object) 凭据信息
+
+      凭据信息的参数由供应商 YAML 配置文件的 `provider_credential_schema` 或 `model_credential_schema` 定义，传入如：`api_key` 等。
+
+    - `content_text` (string) 需要转换的文本内容
+
+    - `streaming` (bool) 是否进行流式输出
+
+    - `user` (string) [optional] 用户的唯一标识符
+
+      可以帮助供应商监控和检测滥用行为。
+
+  - 返回：
+
+    文本转换后的语音流。
+
 ### Moderation
 
 继承 `__base.moderation_model.ModerationModel` 基类，实现以下接口：

api/core/model_runtime/docs/zh_Hans/predefined_model_scale_out.md

@@ -10,6 +10,7 @@
 - `text_embedding` 文本 Embedding 模型
 - `rerank` Rerank 模型
 - `speech2text` 语音转文字
+- `tts` 文字转语音
 - `moderation` 审查
 
 依旧以 `Anthropic` 为例，`Anthropic` 仅支持 LLM，因此在 `model_providers.anthropic` 创建一个 `llm` 为名称的 `module`。

api/core/model_runtime/docs/zh_Hans/schema.md

@@ -48,6 +48,10 @@
   - `max_chunks` (int) 最大分块数量 (模型类型 `text-embedding ` `moderation` 可用)
   - `file_upload_limit` (int) 文件最大上传限制，单位：MB。（模型类型 `speech2text` 可用）
   - `supported_file_extensions` (string)  支持文件扩展格式，如：mp3,mp4（模型类型 `speech2text` 可用）
+  - `default_voice` (string)  缺省音色，可选：alloy,echo,fable,onyx,nova,shimmer（模型类型 `tts` 可用）
+  - `word_limit` (int)  单次转换字数限制，默认按段落分段（模型类型 `tts` 可用）
+  - `audio_type` (string)  支持音频文件扩展格式，如：mp3,wav（模型类型 `tts` 可用）
+  - `max_workers` (int)  支持文字音频转换并发任务数（模型类型 `tts` 可用）
   - `max_characters_per_chunk` (int) 每块最大字符数 (模型类型  `moderation` 可用)
 - `parameter_rules` (array[[ParameterRule](#ParameterRule)]) [optional] 模型调用参数规则
 - `pricing` ([PriceConfig](#PriceConfig)) [optional] 价格信息
@@ -59,6 +63,7 @@
 - `text-embedding` 文本 Embedding 模型
 - `rerank` Rerank 模型
 - `speech2text` 语音转文字
+- `tts` 文字转语音
 - `moderation` 审查
 
 ### ConfigurateMethod

api/core/model_runtime/entities/model_entities.py

@@ -78,6 +78,7 @@ class ModelFeature(Enum):
     MULTI_TOOL_CALL = "multi-tool-call"
     AGENT_THOUGHT = "agent-thought"
     VISION = "vision"
+    STREAM_TOOL_CALL = "stream-tool-call"
 
 
 class DefaultParameterName(Enum):

api/core/model_runtime/model_providers/__base/tts_model.py

@@ -1,7 +1,11 @@
+import hashlib
+import subprocess
+import uuid
 from abc import abstractmethod
 from typing import Optional
 
-from core.model_runtime.entities.model_entities import ModelType
+from core.model_runtime.entities.model_entities import ModelPropertyKey, ModelType
+from core.model_runtime.errors.invoke import InvokeBadRequestError
 from core.model_runtime.model_providers.__base.ai_model import AIModel
 
 
@@ -40,3 +44,96 @@ class TTSModel(AIModel):
         :return: translated audio file
         """
         raise NotImplementedError
+
+    def _get_model_voice(self, model: str, credentials: dict) -> any:
+        """
+        Get voice for given tts model
+
+        :param model: model name
+        :param credentials: model credentials
+        :return: voice
+        """
+        model_schema = self.get_model_schema(model, credentials)
+
+        if model_schema and ModelPropertyKey.DEFAULT_VOICE in model_schema.model_properties:
+            return model_schema.model_properties[ModelPropertyKey.DEFAULT_VOICE]
+
+    def _get_model_audio_type(self, model: str, credentials: dict) -> str:
+        """
+        Get audio type for given tts model
+
+        :param model: model name
+        :param credentials: model credentials
+        :return: voice
+        """
+        model_schema = self.get_model_schema(model, credentials)
+
+        if model_schema and ModelPropertyKey.AUDOI_TYPE in model_schema.model_properties:
+            return model_schema.model_properties[ModelPropertyKey.AUDOI_TYPE]
+
+    def _get_model_word_limit(self, model: str, credentials: dict) -> int:
+        """
+        Get audio type for given tts model
+        :return: audio type
+        """
+        model_schema = self.get_model_schema(model, credentials)
+
+        if model_schema and ModelPropertyKey.WORD_LIMIT in model_schema.model_properties:
+            return model_schema.model_properties[ModelPropertyKey.WORD_LIMIT]
+
+    def _get_model_workers_limit(self, model: str, credentials: dict) -> int:
+        """
+        Get audio max workers for given tts model
+        :return: audio type
+        """
+        model_schema = self.get_model_schema(model, credentials)
+
+        if model_schema and ModelPropertyKey.MAX_WORKERS in model_schema.model_properties:
+            return model_schema.model_properties[ModelPropertyKey.MAX_WORKERS]
+
+    @staticmethod
+    def _split_text_into_sentences(text: str, limit: int, delimiters=None):
+        if delimiters is None:
+            delimiters = set('。！？；\n')
+
+        buf = []
+        word_count = 0
+        for char in text:
+            buf.append(char)
+            if char in delimiters:
+                if word_count >= limit:
+                    yield ''.join(buf)
+                    buf = []
+                    word_count = 0
+                else:
+                    word_count += 1
+            else:
+                word_count += 1
+
+        if buf:
+            yield ''.join(buf)
+
+    @staticmethod
+    def _is_ffmpeg_installed():
+        try:
+            output = subprocess.check_output("ffmpeg -version", shell=True)
+            if "ffmpeg version" in output.decode("utf-8"):
+                return True
+            else:
+                raise InvokeBadRequestError("ffmpeg is not installed, "
+                                            "details: https://docs.dify.ai/getting-started/install-self-hosted"
+                                            "/install-faq#id-14.-what-to-do-if-this-error-occurs-in-text-to-speech")
+        except Exception:
+            raise InvokeBadRequestError("ffmpeg is not installed, "
+                                        "details: https://docs.dify.ai/getting-started/install-self-hosted"
+                                        "/install-faq#id-14.-what-to-do-if-this-error-occurs-in-text-to-speech")
+
+    # Todo: To improve the streaming function
+    @staticmethod
+    def _get_file_name(file_content: str) -> str:
+        hash_object = hashlib.sha256(file_content.encode())
+        hex_digest = hash_object.hexdigest()
+
+        namespace_uuid = uuid.UUID('a5da6ef9-b303-596f-8e88-bf8fa40f4b31')
+        unique_uuid = uuid.uuid5(namespace_uuid, hex_digest)
+        return str(unique_uuid)

api/core/model_runtime/model_providers/azure_openai/_constant.py

@@ -36,6 +36,7 @@ LLM_BASE_MODELS = [
             features=[
                 ModelFeature.AGENT_THOUGHT,
                 ModelFeature.MULTI_TOOL_CALL,
+                ModelFeature.STREAM_TOOL_CALL,
             ],
             fetch_from=FetchFrom.CUSTOMIZABLE_MODEL,
             model_properties={
@@ -80,6 +81,7 @@ LLM_BASE_MODELS = [
             features=[
                 ModelFeature.AGENT_THOUGHT,
                 ModelFeature.MULTI_TOOL_CALL,
+                ModelFeature.STREAM_TOOL_CALL,
             ],
             fetch_from=FetchFrom.CUSTOMIZABLE_MODEL,
             model_properties={
@@ -124,6 +126,7 @@ LLM_BASE_MODELS = [
             features=[
                 ModelFeature.AGENT_THOUGHT,
                 ModelFeature.MULTI_TOOL_CALL,
+                ModelFeature.STREAM_TOOL_CALL,
             ],
             fetch_from=FetchFrom.CUSTOMIZABLE_MODEL,
             model_properties={
@@ -198,6 +201,7 @@ LLM_BASE_MODELS = [
             features=[
                 ModelFeature.AGENT_THOUGHT,
                 ModelFeature.MULTI_TOOL_CALL,
+                ModelFeature.STREAM_TOOL_CALL,
             ],
             fetch_from=FetchFrom.CUSTOMIZABLE_MODEL,
             model_properties={
@@ -272,6 +276,7 @@ LLM_BASE_MODELS = [
             features=[
                 ModelFeature.AGENT_THOUGHT,
                 ModelFeature.MULTI_TOOL_CALL,
+                ModelFeature.STREAM_TOOL_CALL,
             ],
             fetch_from=FetchFrom.CUSTOMIZABLE_MODEL,
             model_properties={

api/core/model_runtime/model_providers/azure_openai/llm/llm.py

@@ -324,6 +324,7 @@ class AzureOpenAILargeLanguageModel(_CommonAzureOpenAI, LargeLanguageModel):
                                               tools: Optional[list[PromptMessageTool]] = None) -> Generator:
         index = 0
         full_assistant_content = ''
+        delta_assistant_message_function_call_storage: ChoiceDeltaFunctionCall = None
         real_model = model
         system_fingerprint = None
         completion = ''
@@ -333,12 +334,32 @@ class AzureOpenAILargeLanguageModel(_CommonAzureOpenAI, LargeLanguageModel):
 
             delta = chunk.choices[0]
 
-            if delta.finish_reason is None and (delta.delta.content is None or delta.delta.content == ''):
+            if delta.finish_reason is None and (delta.delta.content is None or delta.delta.content == '') and \
+                delta.delta.function_call is None:
                 continue
-
+            
             # assistant_message_tool_calls = delta.delta.tool_calls
             assistant_message_function_call = delta.delta.function_call
 
+            # extract tool calls from response
+            if delta_assistant_message_function_call_storage is not None:
+                # handle process of stream function call
+                if assistant_message_function_call:
+                    # message has not ended ever
+                    delta_assistant_message_function_call_storage.arguments += assistant_message_function_call.arguments
+                    continue
+                else:
+                    # message has ended
+                    assistant_message_function_call = delta_assistant_message_function_call_storage
+                    delta_assistant_message_function_call_storage = None
+            else:
+                if assistant_message_function_call:
+                    # start of stream function call
+                    delta_assistant_message_function_call_storage = assistant_message_function_call
+                    if delta_assistant_message_function_call_storage.arguments is None:
+                        delta_assistant_message_function_call_storage.arguments = ''
+                    continue
+
             # extract tool calls from response
             # tool_calls = self._extract_response_tool_calls(assistant_message_tool_calls)
             function_call = self._extract_response_function_call(assistant_message_function_call)
@@ -489,7 +510,7 @@ class AzureOpenAILargeLanguageModel(_CommonAzureOpenAI, LargeLanguageModel):
         else:
             raise ValueError(f"Got unknown type {message}")
 
-        if message.name is not None:
+        if message.name:
             message_dict["name"] = message.name
 
         return message_dict
@@ -586,7 +607,6 @@ class AzureOpenAILargeLanguageModel(_CommonAzureOpenAI, LargeLanguageModel):
         num_tokens = 0
         for tool in tools:
             num_tokens += len(encoding.encode('type'))
-            num_tokens += len(encoding.encode(tool.get("type")))
             num_tokens += len(encoding.encode('function'))
 
             # calculate num tokens for function object

api/core/model_runtime/model_providers/azure_openai/text_embedding/text_embedding.py

@@ -1,7 +1,7 @@
 import base64
 import copy
 import time
-from typing import Optional, Tuple
+from typing import Optional, Tuple, Union
 
 import numpy as np
 import tiktoken
@@ -76,7 +76,7 @@ class AzureOpenAITextEmbeddingModel(_CommonAzureOpenAI, TextEmbeddingModel):
                 embeddings_batch, embedding_used_tokens = self._embedding_invoke(
                     model=model,
                     client=client,
-                    texts=[""],
+                    texts="",
                     extra_model_kwargs=extra_model_kwargs
                 )
 
@@ -147,7 +147,7 @@ class AzureOpenAITextEmbeddingModel(_CommonAzureOpenAI, TextEmbeddingModel):
         return ai_model_entity.entity
 
     @staticmethod
-    def _embedding_invoke(model: str, client: AzureOpenAI, texts: list[str],
+    def _embedding_invoke(model: str, client: AzureOpenAI, texts: Union[list[str], str],
                           extra_model_kwargs: dict) -> Tuple[list[list[float]], int]:
         response = client.embeddings.create(
             input=texts,

api/core/model_runtime/model_providers/bedrock/llm/anthropic.claude-v1.yaml

@@ -8,9 +8,9 @@ model_properties:
 parameter_rules:
   - name: temperature
     use_template: temperature
-  - name: topP
+  - name: top_p
     use_template: top_p
-  - name: topK
+  - name: top_k
     label:
       zh_Hans: 取样数量
       en_US: Top K

api/core/model_runtime/model_providers/bedrock/llm/anthropic.claude-v2.1.yaml

@@ -8,9 +8,9 @@ model_properties:
 parameter_rules:
   - name: temperature
     use_template: temperature
-  - name: topP
+  - name: top_p
     use_template: top_p
-  - name: topK
+  - name: top_k
     label:
       zh_Hans: 取样数量
       en_US: Top K

api/core/model_runtime/model_providers/bedrock/llm/anthropic.claude-v2.yaml

@@ -8,9 +8,9 @@ model_properties:
 parameter_rules:
   - name: temperature
     use_template: temperature
-  - name: topP
+  - name: top_p
     use_template: top_p
-  - name: topK
+  - name: top_k
     label:
       zh_Hans: 取样数量
       en_US: Top K

api/core/model_runtime/model_providers/bedrock/llm/llm.py

@@ -1,15 +1,14 @@
+import json
 import logging
 from typing import Generator, List, Optional, Union
 
 import boto3
-from botocore.exceptions import ClientError, EndpointConnectionError, NoRegionError, ServiceNotInRegionError, UnknownServiceError
 from botocore.config import Config
-import json
-
+from botocore.exceptions import (ClientError, EndpointConnectionError, NoRegionError, ServiceNotInRegionError,
+                                 UnknownServiceError)
 from core.model_runtime.entities.llm_entities import LLMResult, LLMResultChunk, LLMResultChunkDelta
-
-from core.model_runtime.entities.message_entities import (AssistantPromptMessage, PromptMessage,
-                                                          PromptMessageTool, SystemPromptMessage, UserPromptMessage)
+from core.model_runtime.entities.message_entities import (AssistantPromptMessage, PromptMessage, PromptMessageTool,
+                                                          SystemPromptMessage, UserPromptMessage)
 from core.model_runtime.errors.invoke import (InvokeAuthorizationError, InvokeBadRequestError, InvokeConnectionError,
                                               InvokeError, InvokeRateLimitError, InvokeServerUnavailableError)
 from core.model_runtime.errors.validate import CredentialsValidateFailedError
@@ -250,9 +249,12 @@ class BedrockLargeLanguageModel(LargeLanguageModel):
             invoke = runtime_client.invoke_model
 
         try:
+            body_jsonstr=json.dumps(payload)
             response = invoke(
-                body=json.dumps(payload),
                 modelId=model,
+                contentType="application/json",
+                accept= "*/*",
+                body=body_jsonstr
             )
         except ClientError as ex:
             error_code = ex.response['Error']['Code']
@@ -385,7 +387,6 @@ class BedrockLargeLanguageModel(LargeLanguageModel):
             if not chunk:
                 exception_name = next(iter(event))
                 full_ex_msg = f"{exception_name}: {event[exception_name]['message']}"
-
                 raise self._map_client_to_invoke_error(exception_name, full_ex_msg)
 
             payload = json.loads(chunk.get('bytes').decode())
@@ -396,7 +397,7 @@ class BedrockLargeLanguageModel(LargeLanguageModel):
                 finish_reason = payload.get("completion_reason")
  
             elif model_prefix == "anthropic":
-                content_delta = payload
+                content_delta = payload.get("completion")
                 finish_reason = payload.get("stop_reason")
 
             elif model_prefix == "cohere":
@@ -410,12 +411,12 @@ class BedrockLargeLanguageModel(LargeLanguageModel):
             else:
                 raise ValueError(f"Got unknown model prefix {model_prefix} when handling stream response")
 
-            index += 1
-           
+            # transform assistant message to prompt message
             assistant_prompt_message = AssistantPromptMessage(
                 content = content_delta if content_delta else '',
             )
-  
+            index += 1
+           
             if not finish_reason:
                 yield LLMResultChunk(
                     model=model,

api/core/model_runtime/model_providers/chatglm/llm/llm.py

@@ -5,7 +5,8 @@ from typing import Generator, List, Optional, cast
 
 from core.model_runtime.entities.llm_entities import LLMResult, LLMResultChunk, LLMResultChunkDelta
 from core.model_runtime.entities.message_entities import (AssistantPromptMessage, PromptMessage, PromptMessageFunction,
-                                                          PromptMessageTool, SystemPromptMessage, UserPromptMessage)
+                                                          PromptMessageTool, SystemPromptMessage, ToolPromptMessage,
+                                                          UserPromptMessage)
 from core.model_runtime.errors.invoke import (InvokeAuthorizationError, InvokeBadRequestError, InvokeConnectionError,
                                               InvokeError, InvokeRateLimitError, InvokeServerUnavailableError)
 from core.model_runtime.errors.validate import CredentialsValidateFailedError
@@ -194,6 +195,10 @@ class ChatGLMLargeLanguageModel(LargeLanguageModel):
         elif isinstance(message, SystemPromptMessage):
             message = cast(SystemPromptMessage, message)
             message_dict = {"role": "system", "content": message.content}
+        elif isinstance(message, ToolPromptMessage):
+            # check if last message is user message
+            message = cast(ToolPromptMessage, message)
+            message_dict = {"role": "function", "content": message.content}
         else:
             raise ValueError(f"Unknown message type {type(message)}")
         

api/core/model_runtime/model_providers/cohere/llm/llm.py

@@ -1,19 +1,18 @@
 import logging
-from typing import Generator, List, Optional, Union, cast, Tuple
+from typing import Generator, List, Optional, Tuple, Union, cast
 
 import cohere
 from cohere.responses import Chat, Generations
-from cohere.responses.chat import StreamingChat, StreamTextGeneration, StreamEnd
-from cohere.responses.generation import StreamingText, StreamingGenerations
-
+from cohere.responses.chat import StreamEnd, StreamingChat, StreamTextGeneration
+from cohere.responses.generation import StreamingGenerations, StreamingText
 from core.model_runtime.entities.llm_entities import LLMMode, LLMResult, LLMResultChunk, LLMResultChunkDelta
 from core.model_runtime.entities.message_entities import (AssistantPromptMessage, PromptMessage,
-                                                          PromptMessageContentType, SystemPromptMessage,
-                                                          TextPromptMessageContent, UserPromptMessage,
-                                                          PromptMessageTool)
+                                                          PromptMessageContentType, PromptMessageTool,
+                                                          SystemPromptMessage, TextPromptMessageContent,
+                                                          UserPromptMessage)
 from core.model_runtime.entities.model_entities import AIModelEntity, FetchFrom, I18nObject, ModelType
-from core.model_runtime.errors.invoke import InvokeConnectionError, InvokeServerUnavailableError, InvokeError, \
-    InvokeRateLimitError, InvokeAuthorizationError, InvokeBadRequestError
+from core.model_runtime.errors.invoke import (InvokeAuthorizationError, InvokeBadRequestError, InvokeConnectionError,
+                                              InvokeError, InvokeRateLimitError, InvokeServerUnavailableError)
 from core.model_runtime.errors.validate import CredentialsValidateFailedError
 from core.model_runtime.model_providers.__base.large_language_model import LargeLanguageModel
 

api/core/model_runtime/model_providers/cohere/text_embedding/text_embedding.py

@@ -4,11 +4,10 @@ from typing import Optional, Tuple
 import cohere
 import numpy as np
 from cohere.responses import Tokens
-
 from core.model_runtime.entities.model_entities import PriceType
 from core.model_runtime.entities.text_embedding_entities import EmbeddingUsage, TextEmbeddingResult
-from core.model_runtime.errors.invoke import InvokeConnectionError, InvokeServerUnavailableError, InvokeRateLimitError, \
-    InvokeAuthorizationError, InvokeBadRequestError, InvokeError
+from core.model_runtime.errors.invoke import (InvokeAuthorizationError, InvokeBadRequestError, InvokeConnectionError,
+                                              InvokeError, InvokeRateLimitError, InvokeServerUnavailableError)
 from core.model_runtime.errors.validate import CredentialsValidateFailedError
 from core.model_runtime.model_providers.__base.text_embedding_model import TextEmbeddingModel
 
@@ -76,7 +75,7 @@ class CohereTextEmbeddingModel(TextEmbeddingModel):
                 embeddings_batch, embedding_used_tokens = self._embedding_invoke(
                     model=model,
                     credentials=credentials,
-                    texts=[""]
+                    texts=[" "]
                 )
 
                 used_tokens += embedding_used_tokens
@@ -131,6 +130,9 @@ class CohereTextEmbeddingModel(TextEmbeddingModel):
         :param text: text to tokenize
         :return:
         """
+        if not text:
+            return Tokens([], [], {})
+
         # initialize client
         client = cohere.Client(credentials.get('api_key'))
 

api/core/model_runtime/model_providers/minimax/llm/abab5-chat.yaml

@@ -23,7 +23,7 @@ parameter_rules:
   - name: frequency_penalty
     use_template: frequency_penalty
 pricing:
-  input: '0.00'
+  input: '0.015'
   output: '0.015'
   unit: '0.001'
   currency: RMB

api/core/model_runtime/model_providers/minimax/llm/abab5.5-chat.yaml

@@ -4,6 +4,8 @@ label:
 model_type: llm
 features:
   - agent-thought
+  - tool-call
+  - stream-tool-call
 model_properties:
   mode: chat
   context_size: 16384
@@ -36,7 +38,7 @@ parameter_rules:
       en_US: Enable Web Search
       zh_Hans: 开启网页搜索
 pricing:
-  input: '0.00'
+  input: '0.015'
   output: '0.015'
   unit: '0.001'
   currency: RMB

api/core/model_runtime/model_providers/minimax/llm/abab5.5s-chat.yaml

@@ -29,7 +29,7 @@ parameter_rules:
   - name: frequency_penalty
     use_template: frequency_penalty
 pricing:
-  input: '0.00'
+  input: '0.005'
   output: '0.005'
   unit: '0.001'
   currency: RMB

api/core/model_runtime/model_providers/minimax/llm/abab6-chat.yaml

@@ -0,0 +1,37 @@
+model: abab6-chat
+label:
+  en_US: Abab6-Chat
+model_type: llm
+features:
+  - agent-thought
+  - tool-call
+  - stream-tool-call
+model_properties:
+  mode: chat
+  context_size: 32768
+parameter_rules:
+  - name: temperature
+    use_template: temperature
+    min: 0.01
+    max: 1
+    default: 0.1
+  - name: top_p
+    use_template: top_p
+    min: 0.01
+    max: 1
+    default: 0.9
+  - name: max_tokens
+    use_template: max_tokens
+    required: true
+    default: 2048
+    min: 1
+    max: 32768
+  - name: presence_penalty
+    use_template: presence_penalty
+  - name: frequency_penalty
+    use_template: frequency_penalty
+pricing:
+  input: '0.1'
+  output: '0.1'
+  unit: '0.001'
+  currency: RMB

api/core/model_runtime/model_providers/minimax/llm/chat_completion.py

@@ -16,7 +16,7 @@ class MinimaxChatCompletion(object):
     """
     def generate(self, model: str, api_key: str, group_id: str, 
                  prompt_messages: List[MinimaxMessage], model_parameters: dict,
-                 tools: Dict[str, Any], stop: List[str] | None, stream: bool, user: str) \
+                 tools: List[Dict[str, Any]], stop: List[str] | None, stream: bool, user: str) \
         -> Union[MinimaxMessage, Generator[MinimaxMessage, None, None]]:
         """
             generate chat completion
@@ -78,7 +78,7 @@ class MinimaxChatCompletion(object):
 
         try:
             response = post(
-                url=url, data=dumps(body), headers=headers, stream=stream, timeout=10)
+                url=url, data=dumps(body), headers=headers, stream=stream, timeout=(10, 300))
         except Exception as e:
             raise InternalServerError(e)
         
@@ -162,7 +162,6 @@ class MinimaxChatCompletion(object):
                 continue
 
             for choice in choices:
-                print(choice)
                 message = choice['delta']
                 yield MinimaxMessage(
                     content=message,

api/core/model_runtime/model_providers/minimax/llm/chat_completion_pro.py

@@ -17,14 +17,11 @@ class MinimaxChatCompletionPro(object):
     """
     def generate(self, model: str, api_key: str, group_id: str, 
                  prompt_messages: List[MinimaxMessage], model_parameters: dict,
-                 tools: Dict[str, Any], stop: List[str] | None, stream: bool, user: str) \
+                 tools: List[Dict[str, Any]], stop: List[str] | None, stream: bool, user: str) \
         -> Union[MinimaxMessage, Generator[MinimaxMessage, None, None]]:
         """
             generate chat completion
         """
-        if model not in ['abab5.5-chat', 'abab5.5s-chat']:
-            raise BadRequestError(f'Invalid model: {model}')
-        
         if not api_key or not group_id:
             raise InvalidAPIKeyError('Invalid API key or group ID')
         
@@ -85,9 +82,13 @@ class MinimaxChatCompletionPro(object):
             **extra_kwargs
         }
 
+        if tools:
+            body['functions'] = tools
+            body['function_call'] = { 'type': 'auto' }
+
         try:
             response = post(
-                url=url, data=dumps(body), headers=headers, stream=stream, timeout=10)
+                url=url, data=dumps(body), headers=headers, stream=stream, timeout=(10, 300))
         except Exception as e:
             raise InternalServerError(e)
         
@@ -138,6 +139,7 @@ class MinimaxChatCompletionPro(object):
         """
             handle stream chat generate response
         """
+        function_call_storage = None
         for line in response.iter_lines():
             if not line:
                 continue
@@ -151,7 +153,7 @@ class MinimaxChatCompletionPro(object):
                 msg = data['base_resp']['status_msg']
                 self._handle_error(code, msg)
 
-            if data['reply']:
+            if data['reply'] or 'usage' in data and data['usage']:
                 total_tokens = data['usage']['total_tokens']
                 message =  MinimaxMessage(
                     role=MinimaxMessage.Role.ASSISTANT.value,
@@ -163,6 +165,12 @@ class MinimaxChatCompletionPro(object):
                     'total_tokens': total_tokens
                 }
                 message.stop_reason = data['choices'][0]['finish_reason']
+
+                if function_call_storage:
+                    function_call_message = MinimaxMessage(content='', role=MinimaxMessage.Role.ASSISTANT.value)
+                    function_call_message.function_call = function_call_storage
+                    yield function_call_message
+
                 yield message
                 return
 
@@ -171,11 +179,28 @@ class MinimaxChatCompletionPro(object):
                 continue
 
             for choice in choices:
-                message = choice['messages'][0]['text']
-                if not message:
-                    continue
+                message = choice['messages'][0]
+
+                if 'function_call' in message:
+                    if not function_call_storage:
+                        function_call_storage = message['function_call']
+                        if 'arguments' not in function_call_storage or not function_call_storage['arguments']:
+                            function_call_storage['arguments'] = ''
+                            continue
+                    else:
+                        function_call_storage['arguments'] += message['function_call']['arguments']
+                        continue
+                else:
+                    if function_call_storage:
+                        message['function_call'] = function_call_storage
+                        function_call_storage = None
                 
-                yield MinimaxMessage(
-                    content=message,
-                    role=MinimaxMessage.Role.ASSISTANT.value
-                )
+                minimax_message = MinimaxMessage(content='', role=MinimaxMessage.Role.ASSISTANT.value)
+
+                if 'function_call' in message:
+                    minimax_message.function_call = message['function_call']
+
+                if 'text' in message:
+                    minimax_message.content = message['text']
+
+                yield minimax_message

api/core/model_runtime/model_providers/minimax/llm/llm.py

@@ -1,9 +1,8 @@
-from typing import Generator, List, Optional, Union
+from typing import Generator, List
 
-from core.model_runtime.entities.llm_entities import LLMMode, LLMResult, LLMResultChunk, LLMResultChunkDelta, LLMUsage
+from core.model_runtime.entities.llm_entities import LLMResult, LLMResultChunk, LLMResultChunkDelta
 from core.model_runtime.entities.message_entities import (AssistantPromptMessage, PromptMessage, PromptMessageTool,
-                                                          SystemPromptMessage, UserPromptMessage)
-from core.model_runtime.entities.model_entities import AIModelEntity, FetchFrom, ModelType, ParameterRule, ParameterType
+                                                          SystemPromptMessage, ToolPromptMessage, UserPromptMessage)
 from core.model_runtime.errors.invoke import (InvokeAuthorizationError, InvokeBadRequestError, InvokeConnectionError,
                                               InvokeError, InvokeRateLimitError, InvokeServerUnavailableError)
 from core.model_runtime.errors.validate import CredentialsValidateFailedError
@@ -18,6 +17,7 @@ from core.model_runtime.model_providers.minimax.llm.types import MinimaxMessage
 
 class MinimaxLargeLanguageModel(LargeLanguageModel):
     model_apis = {
+        'abab6-chat': MinimaxChatCompletionPro,
         'abab5.5s-chat': MinimaxChatCompletionPro,
         'abab5.5-chat': MinimaxChatCompletionPro,
         'abab5-chat': MinimaxChatCompletion
@@ -55,7 +55,7 @@ class MinimaxLargeLanguageModel(LargeLanguageModel):
                 stream=False,
                 user=''
             )
-        except InvalidAuthenticationError as e:
+        except (InvalidAuthenticationError, InsufficientAccountBalanceError) as e:
             raise CredentialsValidateFailedError(f"Invalid API key: {e}")
 
     def get_num_tokens(self, model: str, credentials: dict, prompt_messages: list[PromptMessage],
@@ -84,6 +84,13 @@ class MinimaxLargeLanguageModel(LargeLanguageModel):
         """
         client: MinimaxChatCompletionPro = self.model_apis[model]()
 
+        if tools:
+            tools = [{
+                "name": tool.name,
+                "description": tool.description,
+                "parameters": tool.parameters
+            } for tool in tools]
+
         response = client.generate(
             model=model,
             api_key=credentials['minimax_api_key'],
@@ -109,7 +116,19 @@ class MinimaxLargeLanguageModel(LargeLanguageModel):
         elif isinstance(prompt_message, UserPromptMessage):
             return MinimaxMessage(role=MinimaxMessage.Role.USER.value, content=prompt_message.content)
         elif isinstance(prompt_message, AssistantPromptMessage):
+            if prompt_message.tool_calls:
+                message = MinimaxMessage(
+                    role=MinimaxMessage.Role.ASSISTANT.value,
+                    content=''
+                )
+                message.function_call={
+                    'name': prompt_message.tool_calls[0].function.name,
+                    'arguments': prompt_message.tool_calls[0].function.arguments
+                }
+                return message
             return MinimaxMessage(role=MinimaxMessage.Role.ASSISTANT.value, content=prompt_message.content)
+        elif isinstance(prompt_message, ToolPromptMessage):
+            return MinimaxMessage(role=MinimaxMessage.Role.FUNCTION.value, content=prompt_message.content)
         else:
             raise NotImplementedError(f'Prompt message type {type(prompt_message)} is not supported')
 
@@ -151,6 +170,28 @@ class MinimaxLargeLanguageModel(LargeLanguageModel):
                         finish_reason=message.stop_reason if message.stop_reason else None,
                     ),
                 )
+            elif message.function_call:
+                if 'name' not in message.function_call or 'arguments' not in message.function_call:
+                    continue
+
+                yield LLMResultChunk(
+                    model=model,
+                    prompt_messages=prompt_messages,
+                    delta=LLMResultChunkDelta(
+                        index=0,
+                        message=AssistantPromptMessage(
+                            content='',
+                            tool_calls=[AssistantPromptMessage.ToolCall(
+                                id='',
+                                type='function',
+                                function=AssistantPromptMessage.ToolCall.ToolCallFunction(
+                                    name=message.function_call['name'],
+                                    arguments=message.function_call['arguments']
+                                )
+                            )]
+                        ),
+                    ),
+                )
             else:
                 yield LLMResultChunk(
                     model=model,