mirror of
https://github.com/infiniflow/ragflow.git
synced 2026-05-20 16:26:42 +08:00
4dcc42e0e1
### What problem does this PR solve?
## Summary
Refactor the dataset API layer into a clean service/REST separation
pattern, add a unified `/index` API for graph/raptor/mindmap operations,
and introduce several new dataset management endpoints with full test
coverage.
## Changes
### Service Layer (`dataset_api_service.py`)
- Added `trace_index(dataset_id, tenant_id, index_type)` — unified trace
function for all index types
- Added `run_index`, `delete_index` service functions
- Added `get_dataset`, `get_ingestion_summary`, `list_ingestion_logs`,
`get_ingestion_log`
- Added `run_embedding`, `list_tags`, `aggregate_tags`, `delete_tags`,
`rename_tag`
- Added `get_flattened_metadata`, `get_auto_metadata`,
`update_auto_metadata`
### REST API Layer (`dataset_api.py`)
**New unified routes:**
| Method | Route | Description |
|--------|-------|-------------|
| POST | `/datasets/<id>/index?type=graph\|raptor\|mindmap` | Run index
task |
| GET | `/datasets/<id>/index?type=graph\|raptor\|mindmap` | Trace index
task |
| DELETE | `/datasets/<id>/<index_type>` | Delete index |
| GET | `/datasets/<id>` | Get dataset details |
| GET | `/datasets/<id>/ingestions/summary` | Ingestion summary |
| GET | `/datasets/<id>/ingestions` | List ingestion logs |
| GET | `/datasets/<id>/ingestions/<log_id>` | Get single ingestion log
|
| POST | `/datasets/<id>/embedding` | Run embedding |
| GET | `/datasets/<id>/tags` | List tags |
| GET | `/datasets/tags/aggregation` | Aggregate tags across datasets |
| DELETE | `/datasets/<id>/tags` | Delete tags |
| PUT | `/datasets/<id>/tags` | Rename tag |
| GET | `/datasets/metadata/flattened` | Get flattened metadata |
| GET/PUT | `/datasets/<id>/metadata/config` | New metadata config path
|
**Removed routes (replaced by unified `/index`):**
- `POST /datasets/<id>/mindmap`
- `GET /datasets/<id>/mindmap`
**Preserved legacy routes (backward compatibility):**
- `/run_graphrag`, `/trace_graphrag`, `/run_raptor`, `/trace_raptor`
- `/auto_metadata` GET/PUT
### Test Suite
- Updated `common.py` helpers: added `trace_index`, removed
`run_mindmap`/`trace_mindmap`
- Added 7 new test files with 39 test cases total:
| Test File | Cases |
|-----------|-------|
| `test_get_dataset.py` | 4 |
| `test_ingestion_summary.py` | 2 |
| `test_ingestion_logs.py` | 5 |
| `test_index_api.py` | 14 |
| `test_embedding.py` | 2 |
| `test_tags.py` | 8 |
| `test_flattened_metadata.py` | 4 |
- Deleted `test_mindmap_tasks.py` (covered by unified index tests)
## Design Decisions
1. **Unified `/index?type=...`** — single endpoint replaces 3 separate
route pairs for graph/raptor/mindmap
2. **Backward compatibility** — old routes (`/run_graphrag`,
`/run_raptor`, `/auto_metadata`) preserved alongside new paths
3. **`_VALID_INDEX_TYPES = {"graph", "raptor", "mindmap"}`** — input
validation via constant set
4. **`_INDEX_TYPE_TO_TASK_ID_FIELD`** — maps index type to KB model task
ID field for clean dispatch
## Files Changed
- `api/apps/restful_apis/dataset_api.py`
- `api/apps/services/dataset_api_service.py`
- `sdk/python/ragflow_sdk/modules/dataset.py`
- `test/testcases/test_http_api/common.py`
- `test/testcases/test_http_api/test_dataset_management/` (7 new files)
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
- [x] Refactoring
---------
Signed-off-by: noob <yixiao121314@outlook.com>
75 lines
2.5 KiB
Python
75 lines
2.5 KiB
Python
#
|
|
# Copyright 2024 The InfiniFlow Authors. All Rights Reserved.
|
|
#
|
|
# Licensed under the Apache License, Version 2.0 (the "License");
|
|
# you may not use this file except in compliance with the License.
|
|
# You may obtain a copy of the License at
|
|
#
|
|
# http://www.apache.org/licenses/LICENSE-2.0
|
|
#
|
|
# Unless required by applicable law or agreed to in writing, software
|
|
# distributed under the License is distributed on an "AS IS" BASIS,
|
|
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
# See the License for the specific language governing permissions and
|
|
# limitations under the License.
|
|
|
|
from common import create_dataset, list_dataset, rm_dataset, upload_file
|
|
from common import list_document, get_docs_info, parse_docs
|
|
from time import sleep
|
|
from timeit import default_timer as timer
|
|
|
|
|
|
def test_parse_txt_document(get_auth):
|
|
# create dataset
|
|
res = create_dataset(get_auth, {"name": "test_parse_txt_document"})
|
|
assert res.get("code") == 0, f"{res.get('message')}"
|
|
|
|
# list dataset
|
|
page_number = 1
|
|
dataset_list = []
|
|
dataset_id = None
|
|
while True:
|
|
res = list_dataset(get_auth, {"page": page_number, "page_size": 150})
|
|
data = res.get("data")
|
|
if isinstance(data, dict):
|
|
data = data.get("kbs", [])
|
|
for item in data:
|
|
dataset_id = item.get("id")
|
|
dataset_list.append(dataset_id)
|
|
if len(dataset_list) < page_number * 150:
|
|
break
|
|
page_number += 1
|
|
|
|
filename = 'ragflow_test.txt'
|
|
res = upload_file(get_auth, dataset_id, f"../test_sdk_api/test_data/{filename}")
|
|
assert res.get("code") == 0, f"{res.get('message')}"
|
|
|
|
res = list_document(get_auth, dataset_id)
|
|
|
|
doc_id_list = []
|
|
for doc in res['data']['docs']:
|
|
doc_id_list.append(doc['id'])
|
|
|
|
res = get_docs_info(get_auth, dataset_id, doc_ids=doc_id_list)
|
|
print(doc_id_list)
|
|
doc_count = len(doc_id_list)
|
|
res = parse_docs(get_auth, doc_id_list)
|
|
|
|
start_ts = timer()
|
|
while True:
|
|
res = get_docs_info(get_auth, dataset_id, doc_ids=doc_id_list)
|
|
finished_count = 0
|
|
for doc_info in res['data']:
|
|
if doc_info['progress'] == 1:
|
|
finished_count += 1
|
|
if finished_count == doc_count:
|
|
break
|
|
sleep(1)
|
|
print('time cost {:.1f}s'.format(timer() - start_ts))
|
|
|
|
# delete dataset
|
|
if dataset_list:
|
|
res = rm_dataset(get_auth, dataset_list)
|
|
assert res.get("code") == 0, f"{res.get('message')}"
|
|
print(f"{len(dataset_list)} datasets are deleted")
|