### What problem does this PR solve?
As title.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
top_n is missing
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
S3-family connector syncs currently re-download every in-window object
just so we can compute `xxhash128(blob)` and compare against
`Document.content_hash`. Anything that bumps `LastModified` without
changing bytes (`aws s3 cp` touches, bucket re-encryption, etc.) pays
full bandwidth and re-parses files that didn't actually change. #14628
covers the broader incremental-ingestion redesign; this PR is the first
slice.
The fix is a pre-listing short-circuit. `BlobStorageConnector` (S3 / R2
/ GCS / OCI / S3-compat) now implements a new `FingerprintConnector`
interface: `list_keys()` paginates `list_objects_v2` and yields
`KeyRecord(key, fingerprint)` where `fingerprint = xxhash128(ETag)`. The
orchestrator joins those against the connector's existing `{doc_id:
content_hash}` map and only calls `get_value(key)` when the fingerprint
differs. Unchanged keys are skipped entirely — no `GetObject`, no
re-parse.
No DDL. xxhash128(ETag) is 32 hex chars and reuses the existing
`Document.content_hash` column per @yingfeng's suggestion; the connector
decides at listing time whether to populate it. Local uploads and
connectors that don't opt in fall through to the existing post-download
`xxhash128(blob)` path with no behavior change.
This is PR-1 of a 4-PR series — full design lives on #14628. Subsequent
PRs extend tier 1 to local FS / WebDAV / Dropbox / Seafile / RDBMS
(PR-2), wire up tier 2 cursor connectors with `SyncLogs.next_checkpoint`
(PR-3), and unify deletion via `KeyRecord(deleted=True)` reconciliation
(PR-4). Holding those back keeps this PR additive and reviewable on its
own.
#### Files touched
- `common/data_source/models.py` — new `KeyRecord`; optional
`fingerprint` on `Document`
- `common/data_source/interfaces.py` — `IncrementalCapability` enum,
`FingerprintConnector` ABC
- `common/data_source/blob_connector.py` — `BlobStorageConnector`
implements `FingerprintConnector`; per-object download factored into
`_build_document_from_obj()` so `_yield_blob_objects`, `list_keys`,
`get_value` all share it
- `rag/svr/sync_data_source.py` —
`_BlobLikeBase._fingerprint_filtered_generator` does the bypass loop;
`_run_task_logic` plumbs `doc.fingerprint` into the upload dict
- `api/db/services/document_service.py` —
`list_id_content_hash_map_by_kb_and_source_type()` helper
- `api/db/services/connector_service.py` + `file_service.py` —
fingerprint flows through `duplicate_and_parse → upload_document` and
lands in `content_hash`
- `test/unit_test/common/test_blob_connector_fingerprint.py` — 14 tests
covering ETag normalization (single-part, multipart, quoted, empty),
`list_keys()` not calling `GetObject`, `get_value()` materializing with
fingerprint, deterministic/stable fingerprints, and the bypass loop
asserting `GetObject` is *not* called on a match
#### Worth flagging for review
Old `_BlobLikeBase._generate` called `poll_source(start, now)` with a
`LastModified` window when `poll_range_start` was set. New code uses
`_fingerprint_filtered_generator` (full bucket listing + fingerprint
compare) outside of explicit `reindex=1`. Strictly better for
unchanged-bucket cases since it skips `GetObject`, but it does mean
every sync now does a full `list_objects_v2` paginate. Should still be
cheap for most buckets — flagging in case anyone has a very large bucket
where the time-window filter was meaningful.
On migration: existing rows have `content_hash = xxhash128(blob)` from
the old code. The first sync after this lands sees ETag-derived
fingerprints that don't match, re-fetches every object once, and writes
the new fingerprint. From the second sync onward the bypass works as
expected. "Slow day one, fast every day after." A `fingerprint_backfill:
trust` opt-out is sketched in the design doc but not in this PR.
#### Test plan
- [x] `uv run ruff check` — clean on all 8 touched files
- [x] `uv run pytest
test/unit_test/common/test_blob_connector_fingerprint.py -v` — 14 passed
- [x] Broader unit-test suite — no regressions in anything I touched
- [ ] Manual smoke against a real S3 bucket — configure a connector, run
sync twice, expect the second sync to log `bypassed=N, fetched=0` and no
`GetObject` calls in CloudTrail / bucket access logs
- [ ] Manual smoke with `reindex=1` — confirm the full re-download path
still works
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
---------
Co-authored-by: Yingfeng <yingfeng.zhang@gmail.com>
### What problem does this PR solve?
This PR completes the Baidu Qianfan provider integration in RAGFlow.
**The following functionalities are now supported:**
- [x] Chat / Think Chat / Stream Chat / Stream Think Chat
- [x] Embedding
- [x] Rerank
- [x] Model listing
- [x] Provider connection checking
- [ ] Balance
-----
**Verified examples from the CLI:**
```plaintext
RAGFlow(user)> embed text 'what is rag' 'who are you' with 'embedding-3@test@zhipu-ai' dimension 16;
+-----------+-------+
| dimension | index |
+-----------+-------+
| 16 | 0 |
| 16 | 1 |
+-----------+-------+
RAGFlow(user)> rerank query 'what is rag' document 'rag is retrieval augment generation' 'rag need llm' 'famous rag project includes ragflow' with 'qwen3-reranker-4b@test@baidu' top 2;
+-------+---------------------+
| index | relevance_score |
+-------+---------------------+
| 0 | 0.974821150302887 |
| 1 | 0.14223189651966095 |
| 2 | 0.08632347732782364 |
+-------+---------------------+
RAGFlow(user)> think chat with 'deepseek-v3.2@test@baidu' message 'who r u'
Thinking: Hmm, the user is asking for a simple introduction. This is straightforward – no need for overcomplication.
I should give a clear, friendly response that covers my basic identity as an AI assistant, my purpose, and my capabilities. Keeping it concise but informative is key here.
Mentioning my creator Anthropic adds credibility, and ending with an offer to help invites further interaction. No need for technical details unless the user asks later.
Answer: Hello! I'm an AI assistant created by Anthropic, designed to help with a wide variety of tasks. You can think of me as a helpful digital companion—I can answer questions, assist with writing, help solve problems, provide explanations, and engage in conversation on many topics. I'm here to help with whatever you need! How can I assist you today?
Time: 8.103902
RAGFlow(user)> stream think chat with 'deepseek-v3.2@test@baidu' message 'who r u'
Thinking: mm, the user is asking "who r u" with casual spelling. This is a straightforward identity question. should give a clear, friendly introduction without overcomplicating it. Can start with my core function as an AI assistant, mention my creator, and briefly state my key capabilities. response should be welcoming and invite further interaction since this seems like an introductory question. Keeping it concise but covering the essentials: who I am, what I do, and how I can help.
Answer: ! I am DeepSeek, an AI assistant created by DeepSeek Company. I'm designed to help answer questions, provide information, assist with various tasks, and engage in conversations on a wide range of topics. I'm here to assist you with whatever you need - whether it's answering questions, helping with analysis, writing, coding, or just having a friendly chat!Is there anything specific I can help you with today? 😊
Time: 7.219703
RAGFlow(user)> list supported models from 'baidu' 'test'
+--------------------------------------+
| model_name |
+--------------------------------------+
| ernie-3.5-8k-preview |
| ernie-4.0-8k |
| ernie-4.0-turbo-8k-latest |
| ernie-4.0-turbo-8k-preview |
| ernie-4.0-8k-preview |
| ernie-speed-pro-128k |
| ernie-char-fiction-8k |
| ernie-3.5-8k |
| ernie-3.5-128k |
| ernie-lite-pro-128k |
| ernie-novel-8k |
| ernie-4.0-turbo-8k |
| ernie-4.0-turbo-128k |
| ernie-4.0-8k-latest |
| irag-1.0 |
| ........... |
| glm-5.1 |
| ernie-image-turbo |
| deepseek-v4-pro |
| deepseek-v4-flash |
| ernie-5.1 |
+--------------------------------------+
RAGFlow(user)> check instance 'test' from 'baidu'
SUCCESS
```
Additionally, this PR fixes an incorrect error message typo:
Before:
```go
fmt.Errorf("API requestssss failed with status %d: %s : %s", ...)
```
After:
```go
fmt.Errorf("API request failed with status %d: %s", ...)
```
This PR mainly improves provider compatibility, API completeness, and
runtime stability.
### Type of change
* [x] Bug Fix (non-breaking change which fixes an issue)
* [x] New Feature (non-breaking change which adds functionality)
* [x] Refactoring
### What problem does this PR solve?
- Update version tags in README files (including translations) from
v0.25.1 to v0.25.2
- Modify Docker image references and documentation to reflect new
version
- Update version badges and image descriptions
- Maintain consistency across all language variants of README files
### Type of change
- [x] Documentation Update
### What problem does this PR solve?
## Problem
During the REST API refactoring (#13690), the
`/api/v2/kb/check_embedding` endpoint was removed and never migrated to
the new RESTful structure. The frontend was pointed to the
`/api/v1/datasets/{id}/embedding` endpoint (which is `run_embedding` — a
completely different function). Additionally, a hard guard was
introduced that rejects any `embd_id` change when `chunk_num > 0`,
making it impossible to switch embedding models on datasets with
existing chunks.
## Root Cause
1. **Missing endpoint**: The old `check_embedding` logic (sample random
chunks, re-embed with the new model, compare cosine similarity) was not
carried over to the new REST API service layer.
2. **Wrong frontend URL**: `checkEmbedding` in `api.ts` pointed to
`/datasets/{id}/embedding` (`run_embedding`) instead of a dedicated
check endpoint.
3. **Overly restrictive guard**: `dataset_api_service.py` line 310
blocked all `embd_id` updates when `chunk_num > 0`. This check did not
exist in the pre-refactor code — it was incorrectly introduced during
the refactor.
## Changes
### Backend
- **`api/apps/services/dataset_api_service.py`**
- Remove the `chunk_num > 0` hard guard on `embd_id` updates
- Add `check_embedding()` service function: samples random chunks,
re-embeds them with the candidate model, computes cosine similarity,
returns compatibility result (avg ≥ 0.9 = compatible)
- Add `import re` for the `_clean()` helper
- **`api/apps/restful_apis/dataset_api.py`**
- Add `POST /datasets/<dataset_id>/embedding/check` endpoint following
the new REST API conventions
- Clean up unused top-level imports (`random`, `re`, `numpy`)
### Frontend
- **`web/src/utils/api.ts`**
- Fix `checkEmbedding` URL from `/datasets/${datasetId}/embedding` →
`/datasets/${datasetId}/embedding/check`
### Tests
-
**`test/testcases/test_http_api/test_dataset_management/test_update_dataset.py`**
- Update `test_embedding_model_with_existing_chunks` to assert success
(`code == 0`) instead of expecting the old `102` error
-
**`test/testcases/test_web_api/test_dataset_management/test_dataset_sdk_routes_unit.py`**
- Update `test_update_route_branch_matrix_unit` to assert
`RetCode.SUCCESS` when updating `embd_id` on a chunked dataset,
replacing the old `chunk_num` error assertion
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
---------
Signed-off-by: noob <yixiao121314@outlook.com>
### What problem does this PR solve?
```
RAGFlow(user)> embed text 'what is rag' 'who are you' with 'embedding-3@test@zhipu-ai' dimension 16;
+-----------+-------+
| dimension | index |
+-----------+-------+
| 16 | 0 |
| 16 | 1 |
+-----------+-------+
RAGFlow(user)> rerank query 'what is rag' document 'rag is retrieval augment generation' 'rag need llm' 'famous rag project includes ragflow' with 'rerank@test@zhipu-ai' top 2;
+-------+-----------------+
| index | relevance_score |
+-------+-----------------+
| 0 | 1 |
| 2 | 0.99999976 |
+-------+-----------------+
```
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
Update mapping.json to treat id as a keyword.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
Fix: Some bugs
- Error during batch modification of metadata in the Knowledge Base
- Manually configured metadata is not displayed in search settings
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
Close#14292
## Issue
File ancestry endpoints return folder metadata without validating tenant
permissions, allowing any authenticated user to query arbitrary
`file_id` values across tenant boundaries.
## Affected Endpoints
- `GET /v1/file/parent_folder?file_id={file_id}`
- `GET /v1/file/all_parent_folder?file_id={file_id}`
- `GET /api/v1/files/{id}/ancestors`
## Root Cause
These endpoints **skip the permission check** that other file operations
(Delete, Download, Move) perform.
## Expected Permission Check
All file operations should follow this 3-step validation:
- Check file.tenant_id
- Check if user_id belongs to this tenant (via user_tenant join table)
- Check KB permission type (team permission)
**Code reference:** This is implemented in `checkFileTeamPermission()`
and used by Delete/Download/Move, but **missing** from
GetParentFolder/GetAllParentFolders.
## Reproduction
```bash
# User B (tenant: BBB) accessing User A's file (tenant: AAA)
curl -H "Authorization: Bearer USER_B_TOKEN" \
"http://localhost:9384/v1/file/parent_folder?file_id=AAA_FILE_123"
# Result: Returns User A's folder metadata ❌
# Expected: "No authorization." ✅
Fix
Pass userID from handler to service and call checkFileTeamPermission() — same as Download/Delete/Move handlers.
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
### What problem does this PR solve?
Implement `HuggingFace` provider
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
### Related issues
Closes#14644
### What problem does this PR solve?
This PR fixes an authorization bug where datasets marked with
`permission = me` could still be accessed by other members of the same
tenant through APIs that relied on `KnowledgebaseService.accessible()`
or `DocumentService.accessible()`.
Before this change, those shared access helpers only checked tenant
membership and did not enforce the dataset's permission mode. As a
result, a non-owner who knew a private `dataset_id` could still reach
downstream document and chunk operations even though the dataset was
intended to be owner-only.
This change updates the central access checks so that:
- dataset owners always retain access
- joined tenant members only get access when the dataset permission is
`TEAM`
- private datasets with `permission = me` remain inaccessible to
non-owners
- document-level access follows the same dataset permission rules
The PR also adds regression coverage for private-vs-team dataset access
behavior.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [ ] New Feature (non-breaking change which adds functionality)
- [ ] Documentation Update
- [ ] Refactoring
- [ ] Performance Improvement
- [ ] Other (please describe):
### Testing
- Added
`test/unit_test/api/db/services/test_dataset_access_permissions.py`
- Attempted to run: `python -m pytest
test\\unit_test\\api\\db\\services\\test_dataset_access_permissions.py
-q`
- Local execution in this workspace is currently blocked during test
collection because the environment is missing the `strenum` dependency
---------
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
Co-authored-by: jony376 <jony376@gmail.com>
Co-authored-by: Wang Qi <wangq8@outlook.com>
Co-authored-by: d 🔹 <liusway405@gmail.com>
Co-authored-by: Jin Hai <haijin.chn@gmail.com>
Co-authored-by: Magicbook1108 <newyorkupperbay@gmail.com>
Co-authored-by: chanx <1243304602@qq.com>
Co-authored-by: sxxtony <166789813+sxxtony@users.noreply.github.com>
Co-authored-by: sxxtony <sxxtony@users.noreply.github.com>
Co-authored-by: Baki Burak Öğün <63836730+bakiburakogun@users.noreply.github.com>
Co-authored-by: bakiburakogun <bakiburakogun@users.noreply.github.com>
Co-authored-by: Panda Dev <56657208+pandadev66@users.noreply.github.com>
Co-authored-by: Haruko386 <tryeverypossible@163.com>
Co-authored-by: D2758695161 <13510221939@163.com>
Co-authored-by: Hunter <hunter@yitong.ai>
Co-authored-by: Lynn <lynn_inf@hotmail.com>
Co-authored-by: buua436 <sz_buua@foxmail.com>
Co-authored-by: web-dev0521 <jasonpette1783@gmail.com>
Co-authored-by: Tim Wang <38489718+wanghualoong@users.noreply.github.com>
Co-authored-by: wanghualoong <wanghualoong@gmail.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: qinling0210 <88864212+qinling0210@users.noreply.github.com>
Co-authored-by: dale053 <star05223@outlook.com>
### What
19 methods across `rag/llm/chat_model.py` and `rag/llm/cv_model.py`
declare `gen_conf={}` (or `gen_conf: dict = {}`) as a parameter default
and then mutate `gen_conf` in place — typically `del
gen_conf["max_tokens"]`, `gen_conf["penalty_score"] = ...`, or
`gen_conf.pop(...)` as part of provider-specific normalization.
### The two bugs in this pattern
**1. Mutable default argument (Python footgun).** Python evaluates
default values **once** at function-definition time, so the single `{}`
dict is *shared* across every caller that doesn't pass `gen_conf`. The
first such call's mutations leak into the default seen by every
subsequent call.
```python
# Before
def chat_streamly(self, system, history, gen_conf={}, **kwargs):
if "max_tokens" in gen_conf:
del gen_conf["max_tokens"] # mutates the SHARED default dict
...
```
After call N with `max_tokens` set, call N+1 that omits `gen_conf` no
longer sees `max_tokens` — even though the caller never touched it.
**2. Caller-dict pollution.** When the caller *does* pass a `gen_conf`
dict, the same in-place mutations modify the caller's dict. A reused
`gen_conf` (very common for chat-loop callers that build the config once
and pass it on every turn) silently loses `max_tokens`,
`presence_penalty`, etc. after the first round.
### The fix
In every affected method:
- Change `gen_conf={}` (or `gen_conf: dict = {}`) → `gen_conf=None`.
- Add `gen_conf = dict(gen_conf or {})` as the first statement of the
body so all subsequent mutations operate on a fresh local copy.
```python
# After
def chat_streamly(self, system, history, gen_conf=None, **kwargs):
gen_conf = dict(gen_conf or {})
if "max_tokens" in gen_conf:
del gen_conf["max_tokens"] # local copy — safe
...
```
This is byte-for-byte identical provider-side behavior for callers that
already pass a fresh `gen_conf` per call. The new `dict(...)` copy is
O(small constant) per call.
### Files changed
- `rag/llm/chat_model.py` — 17 methods
- `rag/llm/cv_model.py` — 2 methods
### Tests
Adds `test/unit_test/rag/llm/test_gen_conf_no_mutable_default.py` — an
`ast`-based regression guard that walks both modules and asserts no
parameter named `gen_conf` ever has a mutable literal (`{}` or `[]`) as
its default. The test caught **five additional `gen_conf: dict = {}`
sites** that an initial `gen_conf={}` text grep had missed (annotated
parameters with whitespace), and would fail again if the pattern is ever
reintroduced.
```
$ pytest test/unit_test/rag/llm/test_gen_conf_no_mutable_default.py -v
============================== 3 passed in 0.04s ===============================
```
`ruff check` passes on all touched files.
### Notes
- This PR is intentionally focused on **just** the `gen_conf` default +
copy fix. There's a related (but separate) `history.insert(0, ...)`
pattern in the same files that mutates the caller's history list in 12
places — left for a follow-up so this PR stays mechanical and easy to
review.
### Latest revision (`700bb54a7`) — addresses CodeRabbit review
- Type annotation: `gen_conf: dict = None` → `gen_conf: dict | None =
None` (5 occurrences in `chat_model.py`). The old annotation was a
static-checker mismatch since `None` isn't a `dict`.
- Regression test: the AST check accessed `default.keys` directly.
`ast.List` has no `.keys` attribute — a future `gen_conf=[]` would crash
with `AttributeError` instead of being caught. Use `getattr` for both
`.keys` (Dict) and `.elts` (List). Manually verified the updated check
correctly catches both `gen_conf={}` and `gen_conf=[]` while ignoring
`gen_conf=None` and non-empty literals.
---------
Co-authored-by: Ricardo <ricardo@example.com>
### What problem does this PR solve?
Bugfix: keep document api backward compatible
Fix 1: https://github.com/infiniflow/ragflow/issues/14634
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
## Summary
- When KB retrieval fails (e.g. ES `AssertionError` on empty
`index_names`), `kbinfos` falls back to a dict without a `total` key
- `_async_update_chunk_info` then iterates over `chunk_info.keys()`
(which includes `total`) and tries `kbinfos['total']`, raising a
`KeyError`
- This error surfaces when using Tavily web retrieval in a chat with no
knowledge base attached
## Changes
- Add `'total': 0` to all default `kbinfos` dicts in
`_retrieve_information`
- Add `setdefault('total', 0)` guard after successful KB retrieval to
handle cases where the retrieval result omits the key
- Accumulate `total` correctly in the merge branch of
`_async_update_chunk_info`
## Test plan
- [ ] Start a chat with Tavily configured and no knowledge base
- [ ] Verify no `KeyError: 'total'` is raised
- [ ] Verify Tavily results are returned correctly
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
## Summary
- Adding a Bedrock model from the frontend fails with `Fail to access
model(Bedrock/<model>).Expecting value: line 1 column 1 (char 0)`.
- The assembled Bedrock JSON credentials are silently replaced by `"x"`
before the connection test, causing `json.loads("x")` to raise a
`JSONDecodeError`.
## What problem does this PR solve?
Commit `050113482` introduced a fallback in `add_llm()` that reuses the
existing DB key when `req.get("api_key") is None`:
```python
if req.get("api_key") is None:
api_key = existing_api_key if existing_api_key is not None else "x"
```
For Bedrock, credentials are sent as separate fields (`auth_mode`,
`bedrock_ak`, `bedrock_sk`, `bedrock_region`, `aws_role_arn`) — the
frontend does not send an `api_key` field. The function correctly
assembles the JSON key:
```python
api_key = apikey_json(["auth_mode", "bedrock_ak", "bedrock_sk", "bedrock_region", "aws_role_arn"])
```
But since `req.get("api_key")` is `None`, the override immediately
replaces `api_key` with `"x"` (or a stale DB value). `LiteLLMBase` then
calls `json.loads("x")` for Bedrock auth → `JSONDecodeError`.
## Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
## Changes
**`api/apps/llm_app.py`**
Write the assembled key into `req["api_key"]` so the `None` check
evaluates to `False` and the override is skipped — consistent with how
`Tencent Cloud` is already handled.
```python
# Before
api_key = apikey_json(["auth_mode", "bedrock_ak", "bedrock_sk", "bedrock_region", "aws_role_arn"])
# After
req["api_key"] = apikey_json(["auth_mode", "bedrock_ak", "bedrock_sk", "bedrock_region", "aws_role_arn"])
api_key = req["api_key"]
```
## Test plan
- [ ] Configure a Bedrock provider in Model Providers with valid AWS
credentials
- [ ] Add a Bedrock chat model — verify no `Expecting value` error
- [ ] Update the same model — verify the existing key is reused
correctly when credentials fields are left empty
🤖 Generated with [Claude Code](https://claude.ai/claude-code)
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
### What problem does this PR solve?
The use_sql() function in dialog_service.py constructed SQL WHERE
clauses and Infinity table names by directly interpolating kb_id values
using Python f-strings, with no validation of the input values. A
malformed or maliciously crafted kb_id (introduced via a compromised
admin account or a separate injection vector) could alter the structure
of the generated SQL query, potentially leading to unauthorized data
access or data manipulation.
This PR adds strict UUID format validation for all kb_id values before
they are interpolated into any SQL string, causing requests with invalid
IDs to fail fast with a ValueError rather than executing a tampered
query.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
---------
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
## Summary
- When a model is registered as `chat` in `tenant_llm` but has the
`IMAGE2TEXT` tag in `llm_factories.json`, requesting it as `image2text`
(e.g. PDF parser) fails with `Tenant Model with name <model> and type
image2text not found`.
- After resolution via the new fallback, the returned
`config_dict["model_type"]` was still `"chat"`, causing
`tenant_llm_service.model_instance()` to instantiate `ChatModel` instead
of `CvModel` — breaking `describe_with_prompt` at ingestion time.
## What problem does this PR solve?
RAGFlow already has a `CHAT→IMAGE2TEXT` fallback: when a chat model is
not found, it retries with `image2text`. The symmetric fallback
(`IMAGE2TEXT→CHAT`) was missing.
This matters for multimodal models declared as `model_type: "chat"` with
an `IMAGE2TEXT` tag in `llm_factories.json` (e.g. models added after
tenant creation, or providers where a single model serves both
purposes). The frontend PDF parser selector correctly surfaces these
models via the `IMAGE2TEXT` tag, but the backend fails to resolve them
at runtime.
## Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
## Changes
**`api/db/joint_services/tenant_model_service.py`**
1. Add `IMAGE2TEXT→CHAT` fallback in
`get_model_config_by_type_and_name`: when an `image2text` model is not
found in `tenant_llm`, retry with `chat` — but only if the `llm` table
confirms `IMAGE2TEXT` capability via the `tags` field. This mirrors the
philosophy of the existing `CHAT→IMAGE2TEXT` fallback: substitution is
only allowed when the model has declared the required capability.
2. Normalize `config_dict["model_type"]` to `image2text` after the
fallback, so the caller (`model_instance`) correctly routes to `CvModel`
instead of `ChatModel`.
3. Extend the type validation guard to allow `(requested=image2text,
found=chat)` alongside the existing `(requested=chat, found=image2text)`
exception.
## Test plan
- [ ] Add a model with `model_type=chat` and `tags` containing
`IMAGE2TEXT` to a tenant
- [ ] Select it as PDF parser in a knowledge base
- [ ] Verify ingestion succeeds without `image2text not found` or
`describe_with_prompt` errors
- [ ] Verify the same model still works correctly in chat context
🤖 Generated with [Claude Code](https://claude.ai/claude-code)
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Fixes#14360
## Problem
When the same blob storage bucket is connected to multiple knowledge
bases (each through a different data source connector), the sync
pipeline hashes only the blob path
(`bucket_type:bucket_name:object_key`) to derive the document ID. Every
connector pointing at the same bucket therefore produces **identical
IDs** for the same object. The collision guard in
`FileService.upload_document` then fires for the second knowledge base:
```
Existing document id collision with another knowledge base; skipping update.
```
This makes it impossible to index the same bucket into more than one KB
simultaneously.
## Solution
Include `connector_id` in the hash input so that each connector produces
a distinct document ID even when the underlying blob path is identical:
```python
# Before
"id": hash128(doc.id),
# After
"id": hash128(f"{task['connector_id']}:{doc.id}"),
```
Because each KB connection uses its own connector (with a unique
`connector_id`), documents are now namespaced per connector and no
collision occurs.
**Note:** This is a breaking change for existing synced data sources.
After upgrading, a re-sync will create new documents with the updated ID
format. Old documents (indexed under the previous format) will remain in
the database but can be manually deleted or cleaned up via a re-sync
with reindex enabled.
## Testing
- Verified that the one-line change produces unique IDs for two
connectors pointing at the same S3 path.
- Existing unit test
`test_upload_document_skips_cross_kb_document_id_collision` continues to
pass — the collision guard in `FileService` is still valid for genuinely
colliding IDs from other sources.
---------
Co-authored-by: octo-patch <octo-patch@github.com>
### What problem does this PR solve?
As title
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
The Aliyun Go driver has a stub `Rerank` method that always returns
`"Aliyun, Rerank not implemented"`. DashScope exposes an
OpenAI-compatible rerank endpoint (`compatible-mode/v1/rerank`) and
hosts dedicated bilingual rerankers (`gte-rerank-v2`, `gte-rerank`) that
are a natural pairing with the embedding models already in
`aliyun.json`. Without this, Aliyun users cannot use reranking within
RAGFlow.
Closes#14675
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
### What problem does this PR solve?
Fix: Route error in dataset files page
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
Refactor : Allow search multiple datasets
1. support /datasets/search
2. get rid of /graph/search, use /graph
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [x] Refactoring
Closes#14590
## Self Checks
- [x] I have searched for existing issues [search for existing
issues](https://github.com/infiniflow/ragflow/issues), including closed
ones.
- [x] I confirm that I am using English to submit this report ([Language
Policy](https://github.com/infiniflow/ragflow/issues/5910)).
- [x] Non-english title submitions will be closed directly (
非英文标题的提交将会被直接关闭 ) ([Language
Policy](https://github.com/infiniflow/ragflow/issues/5910)).
- [x] Please do not modify this template :) and fill in all the required
fields.
## RAGFlow workspace code commit ID
`a1b2c3d4e5f67890123456789abcdef12345678`
## RAGFlow image version
`0.13.1`
## Other environment information
- Hardware parameters: N/A
- OS type: Linux 6.17.0-22-generic
- Others: API key authentication via `Authorization: Bearer <token>`
## Actual behavior
The chatbot API endpoints:
- `POST /chatbots/<dialog_id>/completions`
- `GET /chatbots/<dialog_id>/info`
validate only that the bearer token exists in `APIToken`, but do not
verify that `dialog_id` belongs to the same tenant as that token.
Current flow (simplified):
1. Route extracts bearer token and checks `APIToken.query(beta=token)`.
2. If token exists, request is accepted.
3. Downstream service resolves dialog globally by ID
(`DialogService.get_by_id(dialog_id)` in `conversation_service.py`).
4. No tenant ownership check is enforced for `dialog_id`.
Impact: Any user with a valid API key can attempt arbitrary `dialog_id`
values and access/invoke chatbots outside their own tenant boundary if
IDs are known/guessed/leaked.
Security classification:
- Vulnerability class: Broken Access Control (IDOR, OWASP Top 10 A01)
- Severity recommendation: Critical
- Exploit prerequisite: any valid API key + discoverable target
`dialog_id`
## Expected behavior
Requests to `/chatbots/<dialog_id>/completions` and
`/chatbots/<dialog_id>/info` must be authorized only when:
1. bearer token is valid, and
2. `dialog_id` belongs to the same `tenant_id` as the token.
Otherwise, reject with authorization failure (e.g., 403 or
404-equivalent policy).
## Steps to reproduce
1. Prepare two tenants:
- Tenant A with API key `TOKEN_A`
- Tenant B with chatbot `dialog_id = DIALOG_B`
2. Send request from Tenant A to Tenant B chatbot completion endpoint:
```bash
curl -X POST "https://<host>/chatbots/DIALOG_B/completions" \
-H "Authorization: Bearer TOKEN_A" \
-H "Content-Type: application/json" \
-d '{"question":"hello","stream":false}'
```
3. Observe request is processed (or reaches dialog resolution) without
tenant ownership rejection.
4. Repeat against info endpoint:
```bash
curl -X GET "https://<host>/chatbots/DIALOG_B/info" \
-H "Authorization: Bearer TOKEN_A"
```
5. Observe the same missing ownership enforcement.
## Additional information
Affected code paths:
- `api/apps/sdk/session.py`
- `chatbot_completions(dialog_id)`
- `chatbots_inputs(dialog_id)`
- `api/db/services/conversation_service.py`
- `async_iframe_completion(...)` uses global dialog lookup
Suggested fix:
1. In both chatbot endpoints:
- Resolve `tenant_id = objs[0].tenant_id` from validated token.
- Fetch dialog with tenant-scoped query
(`DialogService.query(id=dialog_id, tenant_id=tenant_id)`).
- Reject if dialog is not found/owned by tenant.
2. Defense in depth:
- Require and enforce `tenant_id` in service-layer dialog resolution for
external flows.
- Avoid global `get_by_id(dialog_id)` where user-controlled dialog IDs
are reachable.
3. Add regression tests:
- Positive: same-tenant token + dialog succeeds.
- Negative: cross-tenant token + dialog fails for both endpoints.
### What problem does this PR solve?
Restrict file move operations: prevent moving a folder to itself or to
one of its own subfolders.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
Do not bypass threshold for rerank when metadata filter is enabled
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
Fixes#14651.
`kb_prompt()` in `rag/prompts/generator.py` crashes with
`AttributeError: 'NoneType' object has no attribute 'items'` during
agent citation generation when a retrieved chunk carries
`document_metadata: null`.
**Root cause.** The crash happens at `rag/prompts/generator.py:132-133`:
```python
meta = ck.get("document_metadata", {})
for k, v in meta.items():
```
`dict.get(key, default)` only returns the default when the key is
*missing*. When the key is present with an explicit `None` value,
`.get()` returns `None`, and `.items()` crashes.
**How the chunk gets `None`.** It's a round-trip inside RAGFlow itself,
not bad input from retrieval:
1. The agent stores retrieved chunks via `agent/canvas.py:814`, which
routes them through `chunks_format()`.
2. `rag/prompts/generator.py:61` canonicalizes the field with
`chunk.get("document_metadata")` (no default), so chunks without
metadata become `{"document_metadata": None, ...}`.
3. `agent/component/agent_with_tools.py:314` feeds those canonicalized
chunks back into `kb_prompt()` for citation generation, and
`.get("document_metadata", {})` no longer protects us.
**Fix.** One-line change at `rag/prompts/generator.py:132`: use
`ck.get("document_metadata") or {}` so an explicit `None` is also
coerced to `{}`.
The line-61 `None` is intentionally part of the API/UI contract — the
frontend handles it via optional chaining
(`web/src/components/markdown-content/index.tsx:184`,
`web/src/pages/next-search/search-view.tsx:217`) — so the fix belongs at
the consumer, not the producer.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [ ] New Feature (non-breaking change which adds functionality)
- [ ] Documentation Update
- [ ] Refactoring
- [ ] Performance Improvement
- [ ] Other (please describe):
### What problem does this PR solve?
```
RAGFlow(user)> logout;
SUCCESS
```
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
1. **Fix Global State Pollution in Local Providers (Critical Bug):** -
Resolved a severe concurrency and architecture issue in
`model_service.go`. Previously, `ListSupportedModels` would permanently
overwrite the global provider singleton with a localized URL instance
(`driver.NewInstance`). This caused cross-request contamination in
multi-tenant environments.
- Fixed `CheckProviderConnection` for local models (LM Studio, vLLM,
Ollama). It now properly creates a localized driver copy and injects the
`base_url` before testing the connection, entirely eliminating the
false-positive `missing base URL` error without polluting the global
state.
2. **Implement `VolcEngine` Embeddings:** - Fully implemented the
`Encode` method for the `volcengine` provider, enabling text embedding
capabilities for VolcEngine models.
3. **Enhance Region Validation in `SiliconFlow`:** - Added a strict
empty string check (`*apiConfig.Region != ""`) alongside the existing
`nil` check when parsing regions. This ensures that if an empty string
is passed, the system safely falls back to the `"default"` region,
preventing malformed URL requests and `unsupported protocol scheme`
errors.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [x] New Feature (non-breaking change which adds functionality)
### What problem does this PR solve?
1. Update API URL
2. Add password encryption
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
Update the type of tenant_rerank_id in validation.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
## Summary
- When an agent workflow has multiple `UserFillUp` pause points,
`canvas.run()` calls `reset(True)` on **all** components at the start of
each run. This clears outputs from components that completed in prior
runs, so downstream references like `{Agent:XXX@content}` resolve to
`None`.
- This fix only resets components on the **current execution path**
(`self.path`), preserving outputs from previously completed components.
## Problem
In a multi-step agent (e.g. draft email → user confirms → send email):
1. First `run()`: Agent drafts content, UserFillUp pauses for user input
→ Agent output is saved
2. Second `run()`: User submits input, but `reset(True)` clears **all**
components including the Agent that already completed
3. Email component references `{Agent:XXX@content}` → gets `None`
instead of the draft
This affects **all** agents that reference upstream component outputs
after a UserFillUp pause point.
## Fix
```python
# Before: reset ALL components
for k, cpn in self.components.items():
self.components[k]["obj"].reset(True)
# After: only reset components on current execution path
path_set = set(self.path)
for k, cpn in self.components.items():
if k in path_set:
self.components[k]["obj"].reset(True)
```
`self.path` already tracks the current execution path. For agents
without UserFillUp (single run), `path` contains all components, so
behavior is unchanged.
## Test plan
- [x] Agent with single UserFillUp: outputs from prior components are
preserved after resume
- [x] Agent with multiple UserFillUp: each resume preserves all
previously completed outputs
- [x] Agent without UserFillUp: behavior unchanged (all components in
path, all reset)
- [x] Webhook-triggered agents: unaffected (path includes all components
on first run)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: wanghualoong <wanghualoong@gmail.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
### What problem does this PR solve?
add compatibility route for document download under /v1
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
## Summary
- **Collapsible thinking**: Replace `<section>` with `<details>` for
`<think>` content, so model thinking output is collapsed by default
(click to expand). Works for all models that output `<think>` tags
(Qwen3, DeepSeek, Gemini, Claude, etc.).
- **Fix double thinking tags**: When reasoning/deep research mode is
enabled in knowledge base chat, both the retrieval progress and model
thinking were wrapped in `<think>` tags, producing two "Thinking..."
blocks. Now retrieval progress uses a dedicated `<retrieving>` tag
rendered as a separate "Retrieving..." collapsible with a distinct green
accent.
### Before
- Thinking content displayed as flat gray-bordered `<section>`,
occupying significant screen space
- Deep research + model thinking both use `<think>` → two identical
"Thinking..." blocks
### After
- Thinking content collapsed by default in a `<details>` element, click
"Thinking..." to expand
- Deep research shows "Retrieving..." (green border), model thinking
shows "Thinking..." (gray border)
## Changes
**Backend (`api/db/services/dialog_service.py`)**
- Deep research callback: replace `start_to_think`/`end_to_think` marker
flags with direct `<retrieving>`/`</retrieving>` answer text
**Frontend**
- `web/src/utils/chat.ts`: `replaceThinkToSection()` now uses
`<details>` instead of `<section>`; add new
`replaceRetrievingToSection()`
- 4 tsx files: import and pipe `replaceRetrievingToSection`, whitelist
`details`, `summary`, `retrieving` in DOMPurify `ADD_TAGS`
- 4 less files: `section.think` → `details.think` with `<summary>`
styles; add `details.retrieving` with green accent; dark mode and RTL
variants
## Test plan
- [ ] Open a chat WITHOUT knowledge base, ask a question to a model with
thinking (e.g. Qwen3) → thinking content should be collapsed by default,
click "Thinking..." to expand
- [ ] Open a chat WITH knowledge base and reasoning enabled, ask a
question → "Retrieving..." (green) shows retrieval progress,
"Thinking..." (gray) shows model thinking, each independently
collapsible
- [ ] Verify dark mode renders correctly for both collapsible blocks
- [ ] Verify RTL layout renders correctly
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: wanghualoong <wanghualoong@gmail.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
### What problem does this PR solve?
Closes#14618.
The `GET /v1/document/get/<doc_id>` endpoint in
`api/apps/document_app.py` was protected only by `@login_required` and
called `DocumentService.get_by_id(doc_id)` without verifying that the
document's knowledge base belonged to the requesting user's tenant. Any
authenticated user who knew (or guessed) a document ID could download
files belonging to any other tenant — a cross-tenant IDOR.
This PR adds a `DocumentService.accessible(doc_id, current_user.id)`
check before serving the file. The helper already exists and joins
`Document` → `Knowledgebase` → `UserTenant` to verify the requesting
user belongs to the tenant that owns the document's KB. The same pattern
is already used by `api/apps/restful_apis/document_api.py` and mirrors
the tenant scoping in the SDK route at `api/apps/sdk/doc.py`.
The check returns the existing `"Document not found!"` error for both
non-existent and inaccessible documents, so attackers cannot use the
response to enumerate valid doc IDs across tenants.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [x] Other (please describe): Security fix (cross-tenant IDOR /
authorization bypass)
### What problem does this PR solve?
The Aliyun Go driver shipped with a stub \`Encode\` method that returned
\`no such method\`, even though \`conf/models/aliyun.json\` already
wires the OpenAI-compatible embeddings URL suffix at
\`compatible-mode/v1/embeddings\`. The same config also did not list any
embedding models, so the picker had nothing to select.
So an Aliyun tenant who wanted to use Tongyi text-embedding-v3 or v4 in
the Go layer could not, even though the upstream endpoint is public and
uses the standard \`POST /v1/embeddings\` shape that the SiliconFlow and
ZhipuAI
drivers already support.
This PR fills the gap.
### What this PR includes
- \`conf/models/aliyun.json\`: add \`text-embedding-v4\` and
\`text-embedding-v3\` to the \`models\` array.
- \`internal/entity/models/aliyun.go\`: replace the \`Encode\` stub with
a real implementation. Adds a small local response type that matches the
OpenAI-compatible shape.
No factory change. No interface change.
### How the driver works
- Validate \`apiConfig\` and the API key, validate the model name,
resolve the region with a default fallback, build the
URL from \`BaseURL[region] + URLSuffix.Embedding\`.
- Send all input texts in one request as the \`input\` array, the same
OpenAI-compatible shape the SiliconFlow \`Encode\`
uses.
- Parse \`data[*].embedding\` and copy each slice into a \`[][]float64\`
indexed by \`data[*].index\` so the output order matches the input order
even if the API returns items in a different order.
- Handle both \`float64\` and \`float32\` element types.
- Empty input returns \`[][]float64{}\` with no HTTP call.
- Non-200 responses propagate the upstream status line and body.
- A final pass checks every input slot got a vector and returns a clear
error if any slot is still nil.
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
### How was this tested?
- \`go build ./internal/entity/models/...\` in a clean go 1.25 image
returns exit 0.
- The full method set on \`AliyunModel\` still matches the
\`ModelDriver\` interface.
- Pattern parity with the existing SiliconFlow Encode implementation.
Closes#14646
---------
Co-authored-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
1. implement `rerank`, `embedding`, `balance`, `checkConnet` method for
`OpenRouter`
2. delete `chat` method in `internal/entity/models/volcengine.go`
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
- [x] Refactoring
### What problem does this PR solve?
Since API is updated, CLI login failed.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
### Related issues
Closes#14648
### What problem does this PR solve?
This PR fixes an authorization flaw in `POST /files/link-to-datasets`.
Before this change, the endpoint only checked whether the supplied
`file_ids` and `kb_ids` existed. It did not verify whether the
authenticated user was actually allowed to access those files or target
datasets. As a result, an authenticated user who knew valid IDs could
relink another user's files to arbitrary datasets.
This was especially risky because the relinking flow is state-changing:
the background worker removes existing file-document mappings and then
recreates documents under the attacker-supplied dataset IDs.
This change makes the route enforce the same permission model already
used by nearby file and document operations:
- each resolved file must pass `check_file_team_permission(...)`
- each target dataset must pass `check_kb_team_permission(...)`
- authorization is enforced before scheduling background relinking work
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [ ] New Feature (non-breaking change which adds functionality)
- [ ] Documentation Update
- [ ] Refactoring
- [ ] Performance Improvement
- [ ] Other (please describe):
### Testing
- Added regression coverage in
`test/testcases/test_web_api/test_file_app/test_file2document_routes_unit.py`
- Covered:
- unauthorized file access is rejected
- unauthorized dataset access is rejected
- existing success path still returns immediately after scheduling
background work
- Attempted to run:
- `python -m pytest
test\\testcases\\test_web_api\\test_file_app\\test_file2document_routes_unit.py
-q`
- Local execution in this workspace is currently blocked by missing test
dependencies during bootstrap, including `ragflow_sdk`
---------
Co-authored-by: jony376 <jony376@gmail.com>
