fix: 修复会话鉴权「登录后 token 无效」并补齐 NAS 部署流程

- deps.py: _create_redis_client 增加同步 ping 校验,Redis 不可达时正确降级为内存模式
- main.py: lifespan 复用 deps 的 UserStore 单例,避免 admin 与 API 请求实例不一致
- 前端: 强制改密弹窗(must_change_password 用户)、兼容新旧登录返回格式、markPasswordChanged
- 新增 NAS SSH 部署脚本与批处理测试;gitignore 前端构建产物
This commit is contained in:
2026-08-04 11:38:28 +08:00
parent 884cf78033
commit bdd30f0a88
20 changed files with 2036 additions and 54 deletions
+3
View File
@@ -9,6 +9,9 @@ build/
# Frontend # Frontend
node_modules/ node_modules/
# 前端构建产物(本地残留的 Vue 构建 / 符号链接,勿提交;线上由 frontend/dist 挂载)
app/static/admin
# Env # Env
.env .env
.env.local .env.local
+5 -2
View File
@@ -90,12 +90,15 @@ QMDSearch/
| POST | `/api/v1/search` | 分层检索 | 免登录 | | POST | `/api/v1/search` | 分层检索 | 免登录 |
| POST | `/api/v1/documents` | 文档入库(202 异步入库,返回 task_id | Bearer | | POST | `/api/v1/documents` | 文档入库(202 异步入库,返回 task_id | Bearer |
| POST | `/api/v1/documents/upload` | multipart 文件上传入库(202 异步,支持 .txt/.md/.html/.htm/.pdf/.docx | Bearer | | POST | `/api/v1/documents/upload` | multipart 文件上传入库(202 异步,支持 .txt/.md/.html/.htm/.pdf/.docx | Bearer |
| POST | `/api/v1/documents/upload-batch` | 批量文件上传入库(多文件,202 异步) | Bearer |
| GET | `/api/v1/documents/tasks` | 入库任务列表(近期,按时间降序) | Bearer |
| GET | `/api/v1/documents/tasks/{task_id}` | 入库任务状态查询(done 附 resultfailed 附 error | 免登录 | | GET | `/api/v1/documents/tasks/{task_id}` | 入库任务状态查询(done 附 resultfailed 附 error | 免登录 |
| GET | `/api/v1/knowledge/categories` | 知识分类类目集 | 免登录 | | GET | `/api/v1/knowledge/categories` | 知识分类类目集 | 免登录 |
| GET | `/api/v1/knowledge/stats` | 统计(四层点数 + 类目分布 + uncategorized 数) | 免登录 | | GET | `/api/v1/knowledge/stats` | 统计(四层点数 + 类目分布 + uncategorized 数) | 免登录 |
| GET | `/api/v1/documents` | 文档列表(limit/offset 分页) | 免登录 | | GET | `/api/v1/documents` | 文档列表(limit/offset 分页) | 免登录 |
| GET | `/api/v1/documents/{doc_id}` | 文档详情 | 免登录 | | GET | `/api/v1/documents/{doc_id}` | 文档详情 | 免登录 |
| GET | `/api/v1/documents/{doc_id}/file` | 下载关联的原始文件(免登录) | 免登录 | | GET | `/api/v1/documents/{doc_id}/file` | 下载关联的原始文件(免登录) | 免登录 |
| POST | `/api/v1/documents/{doc_id}/reingest` | 重新摘要入库(读原文件→删旧→重跑流水线) | Bearer |
| DELETE | `/api/v1/documents/{doc_id}` | 删除文档(幂等) | Bearer | | DELETE | `/api/v1/documents/{doc_id}` | 删除文档(幂等) | Bearer |
| POST | `/api/v1/auth/login` | 用户名密码登录,签发 session tokenTTL 12h | 免登录 | | POST | `/api/v1/auth/login` | 用户名密码登录,签发 session tokenTTL 12h | 免登录 |
| POST | `/api/v1/auth/logout` | 退出登录(删除当前 session | Bearer | | POST | `/api/v1/auth/logout` | 退出登录(删除当前 session | Bearer |
@@ -108,13 +111,13 @@ QMDSearch/
| DELETE | `/api/v1/auth/users/{username}` | 删除用户(清除其 session;禁删自己/最后一个 admin | Bearer + admin | | DELETE | `/api/v1/auth/users/{username}` | 删除用户(清除其 session;禁删自己/最后一个 admin | Bearer + admin |
| GET | `/admin` | 管理页面 | 页面登录门禁 | | GET | `/admin` | 管理页面 | 页面登录门禁 |
「Bearer」指请求头 `Authorization: Bearer <token>`token 经 `/api/v1/auth/login` 获取;变更类文档端点(POST /documents、POST /documents/upload、DELETE /documents/{id}需 Bearer tokenadmin/user 角色均可),查询类端点免登录。 「Bearer」指请求头 `Authorization: Bearer <token>`token 经 `/api/v1/auth/login` 获取;变更类文档端点(POST /documents、POST /documents/upload、POST /documents/upload-batch、POST /documents/{id}/reingest、DELETE /documents/{id})与 GET /documents/tasks 任务列表需 Bearer tokenadmin/user 角色均可),其余查询类端点免登录。
入库任务状态持久化在 Redis(key: `ingest_task:{task_id}`):进行中与 done 保留 24hfailed 保留 7 天;Redis 不可用时降级为纯内存。 入库任务状态持久化在 Redis(key: `ingest_task:{task_id}`):进行中与 done 保留 24hfailed 保留 7 天;Redis 不可用时降级为纯内存。
## 管理页面 ## 管理页面
浏览器访问 `/admin`,页面带登录门禁(未登录/token 失效自动回登录卡片;must_change_password 用户先强制改密后方可进入)。单页面含个区块(admin 视角;user 角色无用户管理区块):概览、个人中心(user 角色默认进入且可见,含账号信息/改密/退出)、文档管理(列表/详情/删除)、文档入库(文本 + 文件上传)、检索测试台、类目列表、API 指南(端点清单 + 在线测试台)、用户管理(仅 admin 角色挂载,含搜索筛选/行内编辑角色与启用状态/弹窗式重置密码与删除)。 浏览器访问 `/admin`,页面带登录门禁(未登录/token 失效自动回登录卡片;must_change_password 用户先强制改密后方可进入)。单页面含个区块(admin 视角;user 角色无用户管理区块):概览、个人中心(user 角色默认进入且可见,含账号信息/改密/退出)、文档管理(列表/详情/删除)、文档入库(文本 + 文件上传,文件上传支持批量多选)、入库进度(展示近期任务与状态,2s 轮询自动刷新)、检索测试台、类目列表、API 指南(端点清单 + 在线测试台)、用户管理(仅 admin 角色挂载,含搜索筛选/行内编辑角色与启用状态/弹窗式重置密码与删除)。
## 认证与用户 ## 认证与用户
+12 -2
View File
@@ -8,6 +8,7 @@ UserStore/SessionStore 为模块级懒加载单例:Redis 客户端创建失败
import structlog import structlog
from fastapi import Depends, Header from fastapi import Depends, Header
from redis import asyncio as redis_async from redis import asyncio as redis_async
from redis import Redis as RedisSync
from app.api.response import ApiError from app.api.response import ApiError
from app.config import settings from app.config import settings
@@ -22,11 +23,20 @@ _session_store: SessionStore | None = None
def _create_redis_client() -> redis_async.Redis | None: def _create_redis_client() -> redis_async.Redis | None:
"""创建 redis.asyncio 客户端(decode_responses=True);失败返回 None 走内存降级""" """创建 redis.asyncio 客户端(decode_responses=True);连接失败返回 None 走内存降级
注意:redis.asyncio.from_url() 仅构造客户端对象,不会实际连接 Redis,
因此需要用同步客户端 ping 一次确认连接可用,否则后续操作才会报错,
导致 UserStore/SessionStore 无法降级为内存模式。
"""
try: try:
# 同步 ping 确认 Redis 可达(from_url 不会实际连接)
sync_client = RedisSync.from_url(settings.redis_url, decode_responses=True)
sync_client.ping()
sync_client.close()
return redis_async.from_url(settings.redis_url, decode_responses=True) return redis_async.from_url(settings.redis_url, decode_responses=True)
except Exception: except Exception:
logger.warning("Redis 客户端创建失败,认证存储降级为内存模式", exc_info=True) logger.warning("Redis 连接失败,认证存储降级为内存模式", exc_info=True)
return None return None
+143
View File
@@ -183,6 +183,149 @@ async def upload_document(
) )
async def _process_single_upload(file: UploadFile) -> dict[str, Any]:
"""处理单个上传文件:校验 → 提取文本 → 落盘 → 提交入库
供批量上传复用:返回成功 {"filename", "task_id"} 或失败 {"filename", "error"}。
title/source 使用文件名默认值,不接受用户传入的额外表单字段。
"""
original_filename = file.filename or "unnamed"
ext = Path(original_filename).suffix.lower()
# 1. 扩展名校验
allowed = _allowed_extensions()
if ext not in allowed:
return {"filename": original_filename, "error": f"不支持的文件类型: {ext or '(无扩展名)'}"}
# 2. 读取字节并校验大小
try:
content = await file.read()
except Exception as exc:
return {"filename": original_filename, "error": f"文件读取失败: {exc}"}
max_bytes = settings.upload_max_size_mb * 1024 * 1024
if len(content) > max_bytes:
return {"filename": original_filename, "error": f"文件超过大小上限: {settings.upload_max_size_mb}MB"}
# 3. 提取文本
try:
text = parse_file(original_filename, content)
except ValueError as exc:
return {"filename": original_filename, "error": str(exc)}
if not text.strip():
return {"filename": original_filename, "error": "无法从文件提取文本"}
# 4. 落盘(按 YYYY/MM 日期分片;失败仅 warning,不阻塞入库)
doc_id = uuid.uuid4().hex
metadata_dict: dict[str, str] = {}
try:
upload_dir = Path(settings.upload_dir).resolve()
shard_subdir = datetime.now(UTC).strftime("%Y/%m")
target_dir = upload_dir / shard_subdir
target_dir.mkdir(parents=True, exist_ok=True)
target = target_dir / f"{doc_id}_{original_filename}"
target.write_bytes(content)
metadata_dict.update(
{
"raw_file_path": str(target),
"original_filename": original_filename,
"original_size_bytes": str(len(content)),
}
)
logger.info(
"上传文件已落盘", doc_id=doc_id, saved_path=str(target), size=len(content)
)
except Exception:
logger.warning(
"上传文件落盘失败,仅做文本入库",
doc_id=doc_id,
filename=original_filename,
exc_info=True,
)
# 5. 默认 title / source
title = Path(original_filename).stem
source = f"file:{original_filename}"
# 6. 提交入库流水线
doc_input = DocumentInput(text=text, title=title, source=source, metadata=metadata_dict)
task_id = await _get_task_manager().submit(doc_input)
return {"filename": original_filename, "task_id": task_id}
@router.post("/documents/upload-batch")
async def upload_batch(
files: list[UploadFile] = File(...),
user: UserRecord = Depends(get_current_user),
) -> JSONResponse:
"""批量文件上传入库:逐文件复用单文件逻辑,收集成功与失败结果
返回 202 + {tasks: [{filename, task_id}], failed: [{filename, error}]}
空文件列表返回 1001。
"""
if not files:
raise ApiError(1001, "未提供任何文件")
tasks: list[dict[str, Any]] = []
failed: list[dict[str, Any]] = []
for file in files:
result = await _process_single_upload(file)
if "task_id" in result:
tasks.append({"filename": result["filename"], "task_id": result["task_id"]})
else:
failed.append({"filename": result["filename"], "error": result["error"]})
return JSONResponse(status_code=202, content=ok({"tasks": tasks, "failed": failed}))
@router.get("/documents/tasks")
async def list_ingest_tasks(
limit: int = Query(default=20, ge=1, le=100),
user: UserRecord = Depends(get_current_user),
) -> dict[str, Any]:
"""列出近期入库任务(合并内存与 Redis 镜像,按 updated_at 降序)"""
tasks = await _get_task_manager().list_tasks(limit=limit)
return ok({"items": tasks, "total": len(tasks)})
@router.post("/documents/{doc_id}/reingest")
async def reingest_document(
doc_id: str, user: UserRecord = Depends(get_current_user)
) -> JSONResponse:
"""重新入库:读原始文件 → 删旧数据 → 提交新入库任务
仅对有原始文件落盘记录的文档可重新入库;重新入库会生成新 doc_id。
"""
meta = await _get_qdrant().get_l1_metadata(doc_id)
if meta is None:
raise ApiError(1004, "文档不存在")
raw_file_path = meta.get("raw_file_path", "")
if not raw_file_path:
raise ApiError(1001, "该文档无原始文件,无法重新入库")
path = Path(raw_file_path)
if not path.is_file():
raise ApiError(1004, "文件不存在")
# 读原文件并重新提取文本
try:
text = parse_file(meta.get("original_filename", path.name), path.read_bytes())
except ValueError as exc:
raise ApiError(1001, str(exc)) from exc
if not text.strip():
raise ApiError(1001, "无法从文件提取文本")
# 删旧数据(四层集合按 doc_id 清除,幂等)
await _get_qdrant().delete_by_doc_id(doc_id)
# 构造新文档输入(保留原 metadatatitle/source 从 meta 取或文件名回退)
original_filename = meta.get("original_filename", path.name)
title = meta.get("title") or Path(original_filename).stem
source = meta.get("source") or f"file:{original_filename}"
doc_input = DocumentInput(text=text, title=title, source=source, metadata=dict(meta))
task_id = await _get_task_manager().submit(doc_input)
return JSONResponse(
status_code=202, content=ok({"task_id": task_id, "status": "pending"})
)
@router.get("/documents/tasks/{task_id}") @router.get("/documents/tasks/{task_id}")
async def get_ingest_task(task_id: str) -> dict[str, Any]: async def get_ingest_task(task_id: str) -> dict[str, Any]:
"""查询入库任务状态:含 task_id/status/created_at/updated_atdone 附 resultfailed 附 error""" """查询入库任务状态:含 task_id/status/created_at/updated_atdone 附 resultfailed 附 error"""
+6 -5
View File
@@ -11,7 +11,8 @@ from fastapi import APIRouter, Depends
from pydantic import BaseModel from pydantic import BaseModel
from app.api.response import ApiError, ok from app.api.response import ApiError, ok
from app.core.auth import AuthUser, get_current_user, require_admin from app.api.deps import get_current_user, require_admin
from app.core.users import UserRecord
from app.core.dedup import invalidate_dedup_strategy_cache from app.core.dedup import invalidate_dedup_strategy_cache
from app.core.file_parser import ( from app.core.file_parser import (
list_docx_plugins, list_docx_plugins,
@@ -46,7 +47,7 @@ class SettingsUpdateRequest(BaseModel):
@router.get("/settings") @router.get("/settings")
async def get_settings(user: AuthUser = Depends(get_current_user)) -> dict[str, Any]: async def get_settings(user: UserRecord = Depends(get_current_user)) -> dict[str, Any]:
"""返回当前 RuntimeSettings(任何登录用户可读)""" """返回当前 RuntimeSettings(任何登录用户可读)"""
cfg = get_runtime_settings() cfg = get_runtime_settings()
return ok(cfg.model_dump(mode="json")) return ok(cfg.model_dump(mode="json"))
@@ -55,7 +56,7 @@ async def get_settings(user: AuthUser = Depends(get_current_user)) -> dict[str,
@router.put("/settings") @router.put("/settings")
async def update_settings( async def update_settings(
body: SettingsUpdateRequest, body: SettingsUpdateRequest,
user: AuthUser = Depends(require_admin), user: UserRecord = Depends(require_admin),
) -> dict[str, Any]: ) -> dict[str, Any]:
"""部分更新 RuntimeSettings(仅 admin """部分更新 RuntimeSettings(仅 admin
@@ -82,7 +83,7 @@ async def update_settings(
@router.get("/settings/schema") @router.get("/settings/schema")
async def get_settings_schema( async def get_settings_schema(
user: AuthUser = Depends(get_current_user), user: UserRecord = Depends(get_current_user),
) -> dict[str, Any]: ) -> dict[str, Any]:
"""返回可选插件与策略列表(前端 Settings 页渲染选项用)""" """返回可选插件与策略列表(前端 Settings 页渲染选项用)"""
return ok( return ok(
@@ -98,7 +99,7 @@ async def get_settings_schema(
@router.post("/settings/reset") @router.post("/settings/reset")
async def reset_settings( async def reset_settings(
user: AuthUser = Depends(require_admin), user: UserRecord = Depends(require_admin),
) -> dict[str, Any]: ) -> dict[str, Any]:
"""重置 RuntimeSettings 为默认值(仅 admin),同时清缓存""" """重置 RuntimeSettings 为默认值(仅 admin),同时清缓存"""
try: try:
+70 -1
View File
@@ -20,7 +20,7 @@ from typing import Any
import structlog import structlog
from app.config import Settings from app.config import Settings
from app.core.dedup import DEDUP_KEY_PREFIX, get_dedup_strategy from app.core.dedup import get_dedup_strategy
from app.core.ingestion import Ingester, IngestionError from app.core.ingestion import Ingester, IngestionError
from app.models.document import DocumentInput from app.models.document import DocumentInput
from app.services.redis import RedisCache from app.services.redis import RedisCache
@@ -79,6 +79,7 @@ class IngestTaskManager:
""" """
task_id = uuid.uuid4().hex task_id = uuid.uuid4().hex
now = _utc_now_iso() now = _utc_now_iso()
filename = self._extract_filename(doc)
dedup = get_dedup_strategy(self._redis) dedup = get_dedup_strategy(self._redis)
# 1. 去重命中:直接置 done,复用旧结果,不调 _run # 1. 去重命中:直接置 done,复用旧结果,不调 _run
@@ -93,6 +94,7 @@ class IngestTaskManager:
"updated_at": now, "updated_at": now,
"result": result_dict, "result": result_dict,
"error": None, "error": None,
"filename": filename,
} }
self._schedule_mirror(task_id) self._schedule_mirror(task_id)
logger.info( logger.info(
@@ -110,6 +112,7 @@ class IngestTaskManager:
"updated_at": now, "updated_at": now,
"result": None, "result": None,
"error": None, "error": None,
"filename": filename,
} }
self._schedule_mirror(task_id) self._schedule_mirror(task_id)
background = asyncio.create_task(self._run(task_id, doc, dedup)) background = asyncio.create_task(self._run(task_id, doc, dedup))
@@ -135,6 +138,72 @@ class IngestTaskManager:
) )
return None return None
async def list_tasks(self, limit: int = 20) -> list[dict[str, Any]]:
"""合并内存注册表与 Redis 镜像,去重,按 updated_at 降序,截断 limit
每项提取 {task_id, status, filename, created_at, updated_at, doc_id}
doc_id 从 done 任务的 result.document_id 提取,其余状态为 None。
"""
records: dict[str, dict[str, Any]] = {}
# 1. 内存注册表(主)
for task_id, record in self._tasks.items():
records[task_id] = record
# 2. Redis 镜像补充内存中没有的(内存未命中的任务,如重启后只存在 Redis 的历史记录)
for task_id, record in await self._scan_redis_tasks():
records.setdefault(task_id, record)
# 3. 按 updated_at 降序,截断 limit
sorted_records = sorted(
records.values(),
key=lambda r: r.get("updated_at") or "",
reverse=True,
)[:limit]
# 4. 提取展示字段
return [self._to_list_item(r) for r in sorted_records]
async def _scan_redis_tasks(self) -> list[tuple[str, dict[str, Any]]]:
"""扫描 Redis 中 ingest_task:* 键,返回 (task_id, record) 列表
RedisCache 未暴露公开扫描接口,借道底层 _get_client().scan_iter
任何异常降级为空列表,不影响 list_tasks 主流程。
"""
if self._redis is None:
return []
get_client = getattr(self._redis, "_get_client", None)
if not callable(get_client):
return []
try:
client = get_client()
tasks: list[tuple[str, dict[str, Any]]] = []
async for key in client.scan_iter(match=f"{REDIS_KEY_PREFIX}*"):
key_str = key if isinstance(key, str) else key.decode("utf-8", "replace")
task_id = key_str[len(REDIS_KEY_PREFIX) :]
record = await self._redis.get_json(key_str)
if isinstance(record, dict):
tasks.append((task_id, record))
return tasks
except Exception:
logger.warning("扫描 Redis 任务键失败", exc_info=True)
return []
@staticmethod
def _extract_filename(doc: DocumentInput) -> str | None:
"""从文档输入提取展示用文件名:优先 metadata.original_filename,其次 title"""
return doc.metadata.get("original_filename") or doc.title or None
@staticmethod
def _to_list_item(record: dict[str, Any]) -> dict[str, Any]:
"""从完整任务记录提取列表展示字段"""
result = record.get("result")
doc_id = result.get("document_id") if isinstance(result, dict) else None
return {
"task_id": record.get("task_id"),
"status": record.get("status"),
"filename": record.get("filename"),
"created_at": record.get("created_at"),
"updated_at": record.get("updated_at"),
"doc_id": doc_id,
}
async def wait_done(self, task_id: str, timeout: float = 30.0) -> dict[str, Any]: async def wait_done(self, task_id: str, timeout: float = 30.0) -> dict[str, Any]:
"""轮询内存注册表直到任务进入终态(done/failed)或超时 """轮询内存注册表直到任务进入终态(done/failed)或超时
+9 -10
View File
@@ -6,7 +6,6 @@ import structlog
from fastapi import FastAPI, Request from fastapi import FastAPI, Request
from fastapi.exceptions import RequestValidationError from fastapi.exceptions import RequestValidationError
from fastapi.responses import FileResponse, JSONResponse, RedirectResponse from fastapi.responses import FileResponse, JSONResponse, RedirectResponse
from redis import asyncio as redis_async
from app.api.response import ApiError, error from app.api.response import ApiError, error
from app.api.v1.auth import router as auth_router from app.api.v1.auth import router as auth_router
@@ -15,7 +14,7 @@ from app.api.v1.knowledge import router as knowledge_router
from app.api.v1.search import router as search_router from app.api.v1.search import router as search_router
from app.api.v1.settings import router as settings_router from app.api.v1.settings import router as settings_router
from app.config import settings from app.config import settings
from app.core.users import UserStore, bootstrap_admin from app.core.users import bootstrap_admin
from app.services.qdrant import QdrantService from app.services.qdrant import QdrantService
logger = structlog.get_logger() logger = structlog.get_logger()
@@ -33,13 +32,13 @@ async def lifespan(_: FastAPI) -> AsyncIterator[None]:
except Exception: except Exception:
logger.error("Qdrant 集合初始化失败,跳过初始化继续启动") logger.error("Qdrant 集合初始化失败,跳过初始化继续启动")
try: try:
# Redis 客户端创建失败(如 URL 非法)时传 None,UserStore 降级为内存模式 # 复用 deps 的 UserStore 单例,确保 lifespan 创建的 admin 与 API 请求用的是同一实例
redis_client = redis_async.from_url(settings.redis_url, decode_responses=True) # _get_user_store 内部会检测 Redis 可达性,不可达时降级为内存模式
except Exception: from app.api.deps import _get_user_store
redis_client = None
try: user_store = _get_user_store()
# 空库引导默认管理员;明文密码由 bootstrap_admin 内部 warning 打印一次 # 空库引导默认管理员;明文密码由 bootstrap_admin 内部 warning 打印一次
await bootstrap_admin(UserStore(redis_client), logger) await bootstrap_admin(user_store, logger)
except Exception: except Exception:
logger.warning("默认管理员初始化失败,跳过", exc_info=True) logger.warning("默认管理员初始化失败,跳过", exc_info=True)
try: try:
@@ -150,8 +149,8 @@ async def admin_spa(rest: str):
_AGENT_SKILL_ZIP = _STATIC_DIR / "agent-skill" / "QMDSearch-Agent-Skill.zip" _AGENT_SKILL_ZIP = _STATIC_DIR / "agent-skill" / "QMDSearch-Agent-Skill.zip"
@app.get("/agent-skill", include_in_schema=False) @app.get("/agent-skill", include_in_schema=False, response_model=None)
async def agent_skill_download() -> FileResponse | JSONResponse: async def agent_skill_download():
"""下载 AI Agent Skill 压缩包(qmdsearch-agent skill,含 SKILL.md 与示例)""" """下载 AI Agent Skill 压缩包(qmdsearch-agent skill,含 SKILL.md 与示例)"""
if not _AGENT_SKILL_ZIP.is_file(): if not _AGENT_SKILL_ZIP.is_file():
return JSONResponse( return JSONResponse(
+221 -25
View File
@@ -259,6 +259,7 @@
<button type="button" data-target="section-overview" class="active">概览</button> <button type="button" data-target="section-overview" class="active">概览</button>
<button type="button" data-target="section-docs">文档管理</button> <button type="button" data-target="section-docs">文档管理</button>
<button type="button" data-target="section-ingest">文档入库</button> <button type="button" data-target="section-ingest">文档入库</button>
<button type="button" data-target="section-progress" id="nav-progress">入库进度</button>
<button type="button" data-target="section-search">检索测试台</button> <button type="button" data-target="section-search">检索测试台</button>
<button type="button" data-target="section-categories">类目列表</button> <button type="button" data-target="section-categories">类目列表</button>
<button type="button" data-target="section-api-guide">API 指南</button> <button type="button" data-target="section-api-guide">API 指南</button>
@@ -340,8 +341,8 @@
<h3 style="font-size:14px; margin-top:24px;">或上传文件</h3> <h3 style="font-size:14px; margin-top:24px;">或上传文件</h3>
<form id="upload-form"> <form id="upload-form">
<div class="field"> <div class="field">
<label for="upload-file">选择文件(支持 .txt/.md/.html/.htm/.pdf/.docx</label> <label for="upload-file">选择文件(支持多选,.txt/.md/.html/.htm/.pdf/.docx</label>
<input type="file" id="upload-file" name="file" accept=".txt,.md,.html,.htm,.pdf,.docx" required> <input type="file" id="upload-file" name="file" accept=".txt,.md,.html,.htm,.pdf,.docx" multiple required>
</div> </div>
<div class="field"> <div class="field">
<label for="upload-title">标题(可选,默认取文件名)</label> <label for="upload-title">标题(可选,默认取文件名)</label>
@@ -356,6 +357,21 @@
<div class="result-box hidden" id="ingest-result"></div> <div class="result-box hidden" id="ingest-result"></div>
</section> </section>
<section id="section-progress" class="hidden">
<h2>入库进度</h2>
<div class="error-bar hidden" id="error-progress"></div>
<div class="toolbar">
<button type="button" class="action" id="btn-refresh-progress">刷新</button>
<span class="muted" id="progress-stats"></span>
</div>
<table>
<thead>
<tr><th>文件名</th><th>状态</th><th>创建时间</th><th>更新时间</th><th>操作</th></tr>
</thead>
<tbody id="progress-tbody"></tbody>
</table>
</section>
<section id="section-search" class="hidden"> <section id="section-search" class="hidden">
<h2>检索测试台</h2> <h2>检索测试台</h2>
<div class="error-bar hidden" id="error-search"></div> <div class="error-bar hidden" id="error-search"></div>
@@ -505,6 +521,7 @@ function showLogin() {
document.getElementById("password-overlay").classList.add("hidden"); document.getElementById("password-overlay").classList.add("hidden");
document.getElementById("user-area").classList.add("hidden"); document.getElementById("user-area").classList.add("hidden");
loadedOnce = {}; loadedOnce = {};
stopProgressPolling();
unmountUsersSection(); unmountUsersSection();
} }
@@ -858,6 +875,8 @@ function truncate(text, maxLen) {
var loadedOnce = {}; var loadedOnce = {};
function activateSection(targetId) { function activateSection(targetId) {
/* 切走前停止进度区块轮询(切到其他区块即暂停) */
stopProgressPolling();
var sections = document.querySelectorAll("main section"); var sections = document.querySelectorAll("main section");
sections.forEach(function (sec) { sections.forEach(function (sec) {
sec.classList.toggle("hidden", sec.id !== targetId); sec.classList.toggle("hidden", sec.id !== targetId);
@@ -875,6 +894,8 @@ function activateSection(targetId) {
if (targetId === "section-api-guide") { renderApiGuide(); } if (targetId === "section-api-guide") { renderApiGuide(); }
if (targetId === "section-profile") { loadProfile(); } if (targetId === "section-profile") { loadProfile(); }
} }
/* 切到入库进度区块时恢复轮询 */
if (targetId === "section-progress") { startProgressPolling(); }
} }
document.getElementById("nav").addEventListener("click", function (event) { document.getElementById("nav").addEventListener("click", function (event) {
@@ -1158,12 +1179,17 @@ function restoreIngestButton() {
} }
function makeStatusBadge(status) { function makeStatusBadge(status) {
var badge = statusBadgeEl(status);
badge.id = "ingest-status-badge";
return badge;
}
/* 状态徽章构造:复用 INGEST_STATUS_TEXT 中文映射与 status-* 配色,无 id 可重复使用 */
function statusBadgeEl(status) {
var cls = "status-running"; var cls = "status-running";
if (status === "done") { cls = "status-done"; } if (status === "done") { cls = "status-done"; }
if (status === "failed") { cls = "status-failed"; } if (status === "failed") { cls = "status-failed"; }
var badge = el("span", INGEST_STATUS_TEXT[status] || status, "status-badge " + cls); return el("span", INGEST_STATUS_TEXT[status] || status, "status-badge " + cls);
badge.id = "ingest-status-badge";
return badge;
} }
function updateIngestStatus(status) { function updateIngestStatus(status) {
@@ -1290,26 +1316,178 @@ document.getElementById("upload-form").addEventListener("submit", function (even
var submitBtn = document.getElementById("btn-upload-submit"); var submitBtn = document.getElementById("btn-upload-submit");
submitBtn.disabled = true; submitBtn.disabled = true;
submitBtn.textContent = "上传中…"; submitBtn.textContent = "上传中…";
var formData = new FormData();
formData.append("file", fileInput.files[0]);
var title = document.getElementById("upload-title").value; var title = document.getElementById("upload-title").value;
var source = document.getElementById("upload-source").value; var source = document.getElementById("upload-source").value;
if (title) { formData.append("title", title); } if (fileInput.files.length > 1) {
if (source) { formData.append("source", source); } /* 批量上传:FormData 用 files 字段逐文件 append,走 upload-batch 端点 */
api("/api/v1/documents/upload", { var batchForm = new FormData();
method: "POST", for (var i = 0; i < fileInput.files.length; i++) {
body: formData batchForm.append("files", fileInput.files[i]);
}).then(function (data) { }
renderIngestHeader(data.task_id, data.status || "pending"); api("/api/v1/documents/upload-batch", {
startIngestPolling(data.task_id); method: "POST",
}).catch(function (err) { body: batchForm
showError("error-ingest", err); }).then(function (data) {
}).finally(function () { renderBatchUploadResult(data);
submitBtn.disabled = false; /* 自动切到入库进度区块查看进度 */
submitBtn.textContent = "上传入库"; activateSection("section-progress");
}); }).catch(function (err) {
showError("error-ingest", err);
}).finally(function () {
submitBtn.disabled = false;
submitBtn.textContent = "上传入库";
});
} else {
/* 单文件上传:保持原逻辑,走 upload 端点 */
var formData = new FormData();
formData.append("file", fileInput.files[0]);
if (title) { formData.append("title", title); }
if (source) { formData.append("source", source); }
api("/api/v1/documents/upload", {
method: "POST",
body: formData
}).then(function (data) {
renderIngestHeader(data.task_id, data.status || "pending");
startIngestPolling(data.task_id);
}).catch(function (err) {
showError("error-ingest", err);
}).finally(function () {
submitBtn.disabled = false;
submitBtn.textContent = "上传入库";
});
}
}); });
/* 批量上传结果展示:成功 N 个任务 + 失败 N 个,纯 textContent 防 XSS */
function renderBatchUploadResult(data) {
var box = document.getElementById("ingest-result");
clearChildren(box);
box.classList.remove("hidden");
var tasks = data.tasks || [];
var failed = data.failed || [];
var headRow = el("div", null, "kv");
headRow.appendChild(el("span", "批量上传完成:成功 " + tasks.length + " 个任务,失败 " + failed.length + " 个"));
box.appendChild(headRow);
if (tasks.length > 0) {
var tasksRow = el("div", null, "kv");
tasksRow.appendChild(el("span", "已提交任务:", "k"));
var tasksList = tasks.map(function (t) {
return (t.filename || "") + "" + (t.task_id || "") + "";
}).join("");
tasksRow.appendChild(el("span", tasksList));
box.appendChild(tasksRow);
}
if (failed.length > 0) {
var failedRow = el("div", null, "kv");
failedRow.appendChild(el("span", "失败文件:", "k"));
var failedList = failed.map(function (f) {
return (f.filename || "") + "" + (f.error || "") + "";
}).join("");
failedRow.appendChild(el("span", failedList));
box.appendChild(failedRow);
}
}
/* ---------- 3.5 入库进度区块(2s 轮询,区块可见时轮询、切走暂停) ---------- */
var PROGRESS_POLL_INTERVAL_MS = 2000;
var progressPollTimer = null;
function stopProgressPolling() {
if (progressPollTimer !== null) {
clearInterval(progressPollTimer);
progressPollTimer = null;
}
}
function startProgressPolling() {
stopProgressPolling();
loadProgress();
progressPollTimer = setInterval(loadProgress, PROGRESS_POLL_INTERVAL_MS);
}
/* GET /documents/tasks 拉取任务列表,渲染表格与状态统计 */
function loadProgress() {
hideError("error-progress");
api("/api/v1/documents/tasks?limit=50").then(function (data) {
renderProgress(data.items || [], data.total || 0);
}).catch(function (err) { showError("error-progress", err); });
}
function renderProgress(items, total) {
var tbody = document.getElementById("progress-tbody");
clearChildren(tbody);
var running = 0, done = 0, failed = 0;
items.forEach(function (it) {
if (it.status === "done") { done += 1; }
else if (it.status === "failed") { failed += 1; }
else { running += 1; }
tbody.appendChild(buildProgressRow(it));
});
document.getElementById("progress-stats").textContent =
"总数 " + total + " · 进行中 " + running + " · 完成 " + done + " · 失败 " + failed;
if (items.length === 0) {
var tr = el("tr");
var td = el("td", "暂无入库任务", "muted");
td.setAttribute("colspan", "5");
tr.appendChild(td);
tbody.appendChild(tr);
}
}
/* 单行渲染:文件名 / 状态徽章 / 创建时间 / 更新时间 / 操作(done 查看文档+重新入库,failed 重试) */
function buildProgressRow(item) {
var tr = el("tr");
tr.appendChild(el("td", item.filename || "(未知文件)"));
var statusTd = el("td");
statusTd.appendChild(statusBadgeEl(item.status));
tr.appendChild(statusTd);
tr.appendChild(el("td", item.created_at || ""));
tr.appendChild(el("td", item.updated_at || ""));
var opsTd = el("td");
if (item.status === "done" && item.doc_id) {
var viewBtn = el("button", "查看文档", "action");
viewBtn.type = "button";
viewBtn.addEventListener("click", function () {
activateSection("section-docs");
loadDocDetail(item.doc_id);
});
var reingestBtn = el("button", "重新入库", "action");
reingestBtn.type = "button";
reingestBtn.addEventListener("click", function () {
reingestDocument(item.doc_id, item.filename);
});
opsTd.appendChild(viewBtn);
opsTd.appendChild(reingestBtn);
} else if (item.status === "failed" && item.doc_id) {
var retryBtn = el("button", "重试", "action");
retryBtn.type = "button";
retryBtn.addEventListener("click", function () {
reingestDocument(item.doc_id, item.filename);
});
opsTd.appendChild(retryBtn);
}
tr.appendChild(opsTd);
return tr;
}
/* 重新入库:POST /documents/{doc_id}/reingest,成功后刷新进度表格 */
function reingestDocument(docId, filename) {
hideError("error-progress");
api("/api/v1/documents/" + encodeURIComponent(docId) + "/reingest", {
method: "POST"
}).then(function (data) {
var stats = document.getElementById("progress-stats");
stats.textContent = "已提交「" + (filename || docId) + "」重新入库:task_id=" + (data.task_id || "");
loadProgress();
}).catch(function (err) { showError("error-progress", err); });
}
document.getElementById("btn-refresh-progress").addEventListener("click", loadProgress);
/* ---------- 4. 检索测试台 ---------- */ /* ---------- 4. 检索测试台 ---------- */
document.getElementById("search-form").addEventListener("submit", function (event) { document.getElementById("search-form").addEventListener("submit", function (event) {
@@ -1768,7 +1946,15 @@ var API_GUIDE = [
{ name: "title", desc: "可选标题,默认取文件名去扩展", def: "" }, { name: "title", desc: "可选标题,默认取文件名去扩展", def: "" },
{ name: "source", desc: "可选来源标识,默认 file:原文件名", def: "" } { name: "source", desc: "可选来源标识,默认 file:原文件名", def: "" }
], ],
body: null, upload: true }, body: null, upload: true, uploadField: "file", uploadMultiple: false },
{ method: "POST", path: "/api/v1/documents/upload-batch", desc: "批量文件上传入库(multipart files 字段,逐文件复用单文件逻辑,202 异步)", auth: "bearer",
query: [], pathParams: [], form: [],
body: null, upload: true, uploadField: "files", uploadMultiple: true },
{ method: "GET", path: "/api/v1/documents/tasks", desc: "入库任务列表(合并内存与 Redis 镜像,按 updated_at 降序)", auth: "bearer",
query: [
{ name: "limit", desc: "返回条数(1-100", def: "20" }
],
pathParams: [], form: [], body: null, upload: false },
{ method: "GET", path: "/api/v1/documents/tasks/{task_id}", desc: "入库任务状态查询(done 附 resultfailed 附 error", auth: "none", { method: "GET", path: "/api/v1/documents/tasks/{task_id}", desc: "入库任务状态查询(done 附 resultfailed 附 error", auth: "none",
query: [], query: [],
pathParams: [{ name: "task_id", desc: "入库任务 ID", def: "" }], pathParams: [{ name: "task_id", desc: "入库任务 ID", def: "" }],
@@ -1787,6 +1973,10 @@ var API_GUIDE = [
query: [], query: [],
pathParams: [{ name: "doc_id", desc: "文档 ID", def: "" }], pathParams: [{ name: "doc_id", desc: "文档 ID", def: "" }],
form: [], body: null, upload: false }, form: [], body: null, upload: false },
{ method: "POST", path: "/api/v1/documents/{doc_id}/reingest", desc: "重新入库(读原文件→删旧数据→提交新任务,生成新 doc_id)", auth: "bearer",
query: [],
pathParams: [{ name: "doc_id", desc: "文档 ID", def: "" }],
form: [], body: null, upload: false },
{ method: "GET", path: "/api/v1/knowledge/categories", desc: "知识分类类目集", auth: "none", { method: "GET", path: "/api/v1/knowledge/categories", desc: "知识分类类目集", auth: "none",
query: [], pathParams: [], form: [], body: null, upload: false }, query: [], pathParams: [], form: [], body: null, upload: false },
{ method: "GET", path: "/api/v1/knowledge/stats", desc: "统计(四层点数 + 类目分布 + uncategorized 数)", auth: "none", { method: "GET", path: "/api/v1/knowledge/stats", desc: "统计(四层点数 + 类目分布 + uncategorized 数)", auth: "none",
@@ -1841,7 +2031,8 @@ function buildCurl(ep) {
parts.push('-H "Authorization: Bearer <TOKEN>"'); parts.push('-H "Authorization: Bearer <TOKEN>"');
} }
if (ep.upload) { if (ep.upload) {
parts.push('-F "file=@/path/to/file.md"'); var fieldName = ep.uploadField || "file";
parts.push('-F "' + fieldName + '=@/path/to/file.md"');
} else if (ep.body) { } else if (ep.body) {
parts.push('-H "Content-Type: application/json"'); parts.push('-H "Content-Type: application/json"');
parts.push("-d '" + JSON.stringify(ep.body) + "'"); parts.push("-d '" + JSON.stringify(ep.body) + "'");
@@ -1964,7 +2155,11 @@ function sendApiTry(idx, panel) {
} }
/* FormData 由浏览器自动生成 multipart 边界,不设置 Content-Type 头 */ /* FormData 由浏览器自动生成 multipart 边界,不设置 Content-Type 头 */
var formData = new FormData(); var formData = new FormData();
formData.append("file", fileInput.files[0]); /* uploadField 默认 file;批量上传用 files 字段逐文件 append */
var upField = ep.uploadField || "file";
for (var fi = 0; fi < fileInput.files.length; fi++) {
formData.append(upField, fileInput.files[fi]);
}
var formInputs = panel.querySelectorAll(".try-form-input"); var formInputs = panel.querySelectorAll(".try-form-input");
for (var k = 0; k < formInputs.length; k++) { for (var k = 0; k < formInputs.length; k++) {
var fVal = formInputs[k].value.trim(); var fVal = formInputs[k].value.trim();
@@ -2044,11 +2239,12 @@ function renderApiItem(ep, idx) {
if (ep.upload) { if (ep.upload) {
renderTryInputs(panel, "form", ep.form); renderTryInputs(panel, "form", ep.form);
var fileField = el("div", null, "field"); var fileField = el("div", null, "field");
fileField.appendChild(el("label", "文件(.txt/.md/.html/.htm/.pdf/.docx")); fileField.appendChild(el("label", "文件(.txt/.md/.html/.htm/.pdf/.docx" + (ep.uploadMultiple ? ",可多选" : "") + ""));
var fileInput = el("input"); var fileInput = el("input");
fileInput.type = "file"; fileInput.type = "file";
fileInput.className = "try-file"; fileInput.className = "try-file";
fileInput.accept = ".txt,.md,.html,.htm,.pdf,.docx"; fileInput.accept = ".txt,.md,.html,.htm,.pdf,.docx";
if (ep.uploadMultiple) { fileInput.setAttribute("multiple", ""); }
fileField.appendChild(fileInput); fileField.appendChild(fileInput);
panel.appendChild(fileField); panel.appendChild(fileField);
} }
+12
View File
@@ -63,3 +63,15 @@ export function resetPassword(username, newPassword) {
export function deleteUser(username) { export function deleteUser(username) {
return http.delete(`/api/v1/auth/users/${encodeURIComponent(username)}`) return http.delete(`/api/v1/auth/users/${encodeURIComponent(username)}`)
} }
/**
* 修改自己的密码(当前登录用户)
* @param {string} oldPassword
* @param {string} newPassword
*/
export function changeMyPassword(oldPassword, newPassword) {
return http.post('/api/v1/auth/password', {
old_password: oldPassword,
new_password: newPassword
})
}
+115 -2
View File
@@ -1,8 +1,8 @@
<script setup> <script setup>
import { computed, h, ref } from 'vue' import { computed, h, reactive, ref, watch } from 'vue'
import { useRoute, useRouter, RouterView } from 'vue-router' import { useRoute, useRouter, RouterView } from 'vue-router'
import { storeToRefs } from 'pinia' import { storeToRefs } from 'pinia'
import { Modal } from 'ant-design-vue' import { Modal, message } from 'ant-design-vue'
import { import {
DashboardOutlined, DashboardOutlined,
FileTextOutlined, FileTextOutlined,
@@ -18,6 +18,8 @@ import {
TeamOutlined TeamOutlined
} from '@ant-design/icons-vue' } from '@ant-design/icons-vue'
import { useAuthStore } from '@/stores/useAuthStore' import { useAuthStore } from '@/stores/useAuthStore'
import { changeMyPassword } from '@/api/auth'
import { callApi } from '@/api/client'
const route = useRoute() const route = useRoute()
const router = useRouter() const router = useRouter()
@@ -66,6 +68,68 @@ function handleLogout() {
} }
}) })
} }
/* ---------- 强制改密弹窗(must_change_password 用户) ---------- */
const passwordModalVisible = ref(false)
const passwordSubmitting = ref(false)
const passwordFormRef = ref(null)
const passwordForm = reactive({
old_password: '',
new_password: '',
confirm_password: ''
})
const passwordRules = {
old_password: [{ required: true, message: '请输入当前密码', trigger: 'blur' }],
new_password: [
{ required: true, message: '请输入新密码', trigger: 'blur' },
{ min: 8, message: '密码至少 8 位', trigger: 'blur' }
],
confirm_password: [
{ required: true, message: '请确认新密码', trigger: 'blur' },
{
validator: (_rule, value) =>
value === passwordForm.new_password
? Promise.resolve()
: Promise.reject(new Error('两次输入的密码不一致')),
trigger: 'blur'
}
]
}
// must_change_password 为 true 时自动弹出改密弹窗(不可关闭)
watch(
() => user.value?.must_change_password,
(val) => {
passwordModalVisible.value = val === true
},
{ immediate: true }
)
async function handleSubmitPassword() {
try {
await passwordFormRef.value.validate()
} catch {
return
}
passwordSubmitting.value = true
try {
await callApi(
() => changeMyPassword(passwordForm.old_password, passwordForm.new_password),
{ errorText: '修改密码失败' }
)
authStore.markPasswordChanged()
message.success('密码修改成功')
passwordModalVisible.value = false
passwordForm.old_password = ''
passwordForm.new_password = ''
passwordForm.confirm_password = ''
} catch {
// 错误已由 callApi 弹出
} finally {
passwordSubmitting.value = false
}
}
</script> </script>
<template> <template>
@@ -127,6 +191,55 @@ function handleLogout() {
QMDSearch Admin · 当前用户{{ user?.username || '-' }} QMDSearch Admin · 当前用户{{ user?.username || '-' }}
</a-layout-footer> </a-layout-footer>
</a-layout> </a-layout>
<!-- 强制改密弹窗must_change_password 用户不可关闭 -->
<a-modal
v-model:open="passwordModalVisible"
:mask-closable="false"
:closable="false"
:keyboard="false"
:destroy-on-close="true"
title="首次登录须修改密码"
:confirm-loading="passwordSubmitting"
ok-text="确认修改"
:cancel-button-props="{ style: { display: 'none' } }"
@ok="handleSubmitPassword"
>
<a-alert
type="warning"
show-icon
message="检测到首次登录或管理员重置密码,请立即修改密码后方可使用系统。"
style="margin-bottom: 16px"
/>
<a-form
ref="passwordFormRef"
:model="passwordForm"
:rules="passwordRules"
layout="vertical"
>
<a-form-item label="当前密码" name="old_password">
<a-input-password
v-model:value="passwordForm.old_password"
placeholder="请输入当前密码"
autocomplete="current-password"
/>
</a-form-item>
<a-form-item label="新密码" name="new_password">
<a-input-password
v-model:value="passwordForm.new_password"
placeholder="至少 8 位"
autocomplete="new-password"
/>
</a-form-item>
<a-form-item label="确认新密码" name="confirm_password">
<a-input-password
v-model:value="passwordForm.confirm_password"
placeholder="再次输入新密码"
autocomplete="new-password"
/>
</a-form-item>
</a-form>
</a-modal>
</a-layout> </a-layout>
</template> </template>
+26 -5
View File
@@ -39,13 +39,27 @@ export const useAuthStore = defineStore('auth', {
actions: { actions: {
/** /**
* 登录成功后保存 token + user * 登录成功后保存 token + user
* @param {{access_token:string, user:object}} data * 兼容后端两种返回:
* - 新格式 { token, username, role, must_change_password }
* - 旧格式 { access_token, user: { username, role, ... } }
* @param {object} data
*/ */
setAuth(data) { setAuth(data) {
this.token = data.access_token const token = data?.access_token ?? data?.token
this.user = data.user const user =
localStorage.setItem(TOKEN_STORAGE_KEY, data.access_token) data?.user ??
localStorage.setItem(USER_STORAGE_KEY, JSON.stringify(data.user)) (data
? {
username: data.username,
role: data.role,
must_change_password: data.must_change_password ?? false
}
: null)
if (!token || !user) return
this.token = token
this.user = user
localStorage.setItem(TOKEN_STORAGE_KEY, token)
localStorage.setItem(USER_STORAGE_KEY, JSON.stringify(user))
}, },
/** 清除登录态(登出 / 401) */ /** 清除登录态(登出 / 401) */
@@ -56,6 +70,13 @@ export const useAuthStore = defineStore('auth', {
localStorage.removeItem(USER_STORAGE_KEY) localStorage.removeItem(USER_STORAGE_KEY)
}, },
/** 改密成功后更新本地用户状态(清除 must_change_password 标记) */
markPasswordChanged() {
if (!this.user) return
this.user = { ...this.user, must_change_password: false }
localStorage.setItem(USER_STORAGE_KEY, JSON.stringify(this.user))
},
/** /**
* 登出 * 登出
*/ */
+144
View File
@@ -0,0 +1,144 @@
#!/usr/bin/env bash
# =============================================================================
# QMDSearch 自动部署脚本(本地 CI/CD 入口)
#
# 设计原则(贴合 NAS 自托管 + 挂载式部署):
# 1. 检测变化 —— 对比 origin/main,区分 backend / frontend / deps 范围
# 2. 校验 —— 本地跑 pytest(CI 门禁),坏代码绝不上线
# 3. 按需构建 —— 仅前端变化时本地预构建(dist 被 gitignoreNAS 端会再构建)
# 4. 提交推送 —— git commit + push origin main,推送后 SSH 到 NAS 执行真正部署
#
# 用法:
# ./scripts/deploy.sh 正常流程:检测 → 校验 → 按需构建 → 提交推送 → SSH 到 NAS 部署
# ./scripts/deploy.sh --dry 零副作用自测:只 检测/校验/构建,不提交不推送
# ./scripts/deploy.sh --no-push 提交但不推送(准备好后手动 push 或再跑一次)
# ./scripts/deploy.sh --no-ssh 推送后不自动 SSH 部署(仅同步代码到 NAS git)
# ./scripts/deploy.sh --skip-tests 跳过 pytest(谨慎)
# ./scripts/deploy.sh --skip-fe 跳过前端构建(即便前端变化)
# ./scripts/deploy.sh --build-fe 强制重新构建前端
# ./scripts/deploy.sh --help
# =============================================================================
set -euo pipefail
REMOTE="origin"
BRANCH="main"
ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
cd "$ROOT"
NO_PUSH=0
DRY=0
NO_SSH=0
SKIP_TESTS=0
SKIP_FE=0
FORCE_FE=0
while [[ $# -gt 0 ]]; do
case "$1" in
--no-push) NO_PUSH=1 ;;
--dry) DRY=1 ;;
--no-ssh) NO_SSH=1 ;;
--skip-tests) SKIP_TESTS=1 ;;
--skip-fe) SKIP_FE=1 ;;
--build-fe) FORCE_FE=1 ;;
--help|-h) sed -n '1,22p' "$0"; exit 0 ;;
*) echo "未知参数: $1"; exit 2 ;;
esac
shift
done
log(){ printf '\033[36m[deploy]\033[0m %s\n' "$*"; }
ok(){ printf '\033[32m[ ok ]\033[0m %s\n' "$*"; }
err(){ printf '\033[31m[err ]\033[0m %s\n' "$*" >&2; }
# ---- 1. 变化检测 --------------------------------------------------------------
log "同步远端引用..."
git fetch --quiet "$REMOTE" "$BRANCH" 2>/dev/null || log "fetch 失败(离线?继续本地流程)"
LOCAL_CHANGES="$(git status --porcelain)"
AHEAD="$(git rev-list --count "$REMOTE/$BRANCH..HEAD" 2>/dev/null || echo 0)"
# 汇总「已提交未推送」+「未提交」的变更文件,去重
CHANGED_FILES="$(
{ git diff --name-only "$REMOTE/$BRANCH" HEAD 2>/dev/null; echo "$LOCAL_CHANGES" | awk '{print $2}'; } \
| grep -v '^$' | sort -u
)"
has(){ echo "$CHANGED_FILES" | grep -qE "$1"; }
BACKEND_CHANGED=0; FRONTEND_CHANGED=0; DEPS_CHANGED=0
if has '^(app/|scripts/|tests/|.*\.py$|Dockerfile|docker-compose\.yml|\.env\.example)'; then BACKEND_CHANGED=1; fi
if has '^frontend/(src/|public/|index\.html|package\.json|vite\.config|package-lock\.json)'; then FRONTEND_CHANGED=1; fi
if has '^(pyproject\.toml|uv\.lock|Dockerfile)'; then DEPS_CHANGED=1; fi
if [[ -z "$CHANGED_FILES" && "$AHEAD" -eq 0 ]]; then
ok "无代码变化,无需部署。"
exit 0
fi
log "变更范围 → backend=$BACKEND_CHANGED frontend=$FRONTEND_CHANGED deps=$DEPS_CHANGED (ahead=$AHEAD)"
# ---- 2. 校验(本地 CI gate---------------------------------------------------
if [[ "$SKIP_TESTS" -eq 0 && "$BACKEND_CHANGED" -eq 1 ]]; then
log "运行后端测试 (uv run pytest)..."
if ! uv run pytest -q --no-header 2>&1 | tail -n 30; then
err "测试失败,已中止部署。请修复后再运行 deploy.sh"
exit 1
fi
ok "后端测试通过"
# 顺带做一次非阻断的静态检查
if command -v uv >/dev/null 2>&1; then
log "运行 ruff 静态检查(不阻断)..."
uv run ruff check app tests 2>&1 | tail -n 15 || true
fi
fi
# ---- 3. 按需构建前端 ----------------------------------------------------------
if [[ ("$FRONTEND_CHANGED" -eq 1 || "$FORCE_FE" -eq 1) && "$SKIP_FE" -eq 0 ]]; then
log "构建前端 (npm ci && npm run build)..."
if ! (cd frontend && npm ci --no-audit --no-fund && npm run build); then
err "前端构建失败,已中止。"
exit 1
fi
ok "前端构建完成 -> frontend/dist"
log "提示: frontend/dist 已被 gitignoreNAS 端 post-receive 会重新构建以生效。"
fi
# ---- 4. 提交 + 推送 -----------------------------------------------------------
if [[ "$DRY" -eq 1 ]]; then
log "(--dry) 零副作用模式:跳过提交与推送。以上为检测结果,确认无误后可去掉 --dry 正式部署。"
exit 0
fi
if [[ -n "$LOCAL_CHANGES" ]]; then
log "暂存并提交本地变化..."
git add -A # .gitignore 已排除 .env / data / node_modules / .workbuddy / .trae 等
SCOPE=""
[[ "$BACKEND_CHANGED" -eq 1 ]] && SCOPE="$SCOPE backend"
[[ "$FRONTEND_CHANGED" -eq 1 ]] && SCOPE="$SCOPE frontend"
[[ "$DEPS_CHANGED" -eq 1 ]] && SCOPE="$SCOPE deps"
MSG="chore(deploy): 自动部署 $(date '+%Y-%m-%d %H:%M')${SCOPE:+ |$SCOPE}"
git commit -q -m "$MSG"
ok "已提交: $MSG"
fi
if [[ "$NO_PUSH" -eq 1 ]]; then
log "(--no-push) 未推送。当前领先远端 $AHEAD 个提交,确认无误后去掉 --no-push 再跑。"
exit 0
fi
export QMDS_SKIP_PREPUSH=1 # 避免与 pre-push hook 重复跑测试
log "推送到 $REMOTE/$BRANCH(触发 NAS 自动部署)..."
if ! git push "$REMOTE" "$BRANCH"; then
err "推送失败:可能远端有更新,请先 git pull 解决冲突后重试。"
exit 1
fi
ok "已推送。代码已同步至 NAS git。"
if [[ "$NO_SSH" -eq 0 ]]; then
log "通过 SSH 在 NAS 上执行部署..."
if bash "$ROOT/scripts/nas-deploy-ssh.sh"; then
ok "NAS 部署完成。"
else
err "NAS 部署脚本返回非 0(详见上方 NAS 输出)。"
exit 1
fi
else
log "(--no-ssh) 已跳过 NAS 端部署。如需部署请运行: ./scripts/nas-deploy-ssh.sh"
fi
+48
View File
@@ -0,0 +1,48 @@
#!/usr/bin/env bash
# =============================================================================
# 安装本地 git pre-push 门禁
#
# 作用:任何「推送到 origin(NAS)」的操作前,先跑后端测试;若前端有变化,
# 先在本地预构建一次。任一环节失败则阻断推送,确保坏代码绝不上线。
#
# 说明:
# - deploy.sh 推送前会设 QMDS_SKIP_PREPUSH=1,避免与自身校验重复跑。
# - 仅对 origin 远端生效,推其他远端不受影响。
#
# 卸载:rm .git/hooks/pre-push
# =============================================================================
set -e
ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
HOOK="$ROOT/.git/hooks/pre-push"
mkdir -p "$(dirname "$HOOK")"
cat > "$HOOK" <<'HOOK_EOF'
#!/usr/bin/env bash
# QMDSearch pre-push 校验门禁(由 scripts/install-hooks.sh 安装)
REMOTE="$1"
# 仅对 NAS 远端 (origin) 做校验;且 deploy.sh 已设 QMDS_SKIP_PREPUSH 时跳过
if [[ "$REMOTE" != "origin" ]]; then exit 0; fi
if [[ -n "${QMDS_SKIP_PREPUSH:-}" ]]; then exit 0; fi
echo "[pre-push] 推送前校验后端测试..."
if ! uv run pytest -q --no-header 2>&1 | tail -n 20; then
echo "[pre-push] 测试失败,已阻断推送。请先修复后再 push。" >&2
exit 1
fi
# 暂存区若含前端变化,本地先预构建一次,避免推到 NAS 才翻车
if git diff --cached --name-only | grep -qE '^frontend/'; then
echo "[pre-push] 检测到前端变化,本地预构建..."
if ! (cd frontend && npm ci --no-audit --no-fund && npm run build); then
echo "[pre-push] 前端构建失败,已阻断推送。" >&2
exit 1
fi
fi
echo "[pre-push] 校验通过,允许推送。"
HOOK_EOF
chmod +x "$HOOK"
echo "已安装 pre-push 门禁 -> $HOOK"
echo "如需卸载: rm $HOOK"
+120
View File
@@ -0,0 +1,120 @@
#!/usr/bin/env bash
# =============================================================================
# QMDSearch NAS 端部署(通过 SSH 直连执行,密钥免密)
#
# 环境约束(已实测):
# - NAS 上有 docker compose v2,但【没有 node/npm】→ 前端必须在本地构建后 rsync 过去
# - compose 的 app 服务只有 image: 无 build: → 依赖变化用 `docker build -t qmdsearch-app:latest .`
# - kp 已在 docker 组,可免 sudo 跑 docker
# - 部署目录 /vol1/qmdsearch 是 git 工作树,以 origin/main 为唯一真相源
#
# 流程:
# 1. SSH 到 NASgit fetch + git reset --hard origin/main(同步受管代码),并算出变更范围
# 2. 本地:若前端变化 → 本地 npm 构建 → rsync frontend/dist 到 NASNAS 无 node
# 3. SSH 到 NAS:依赖变化则 docker build + force-recreate;否则 restart app
# 4. 健康检查 /health;失败回滚到上一稳定提交
#
# 用法:
# ./scripts/nas-deploy-ssh.sh 正式部署
# ./scripts/nas-deploy-ssh.sh --dry-run 只预览将要部署的变更,不重启
# 环境变量:NAS_HOST(默认 nas DEPLOY_DIR(默认 /vol1/qmdsearch APP_PORT(默认 8000
# =============================================================================
set -uo pipefail
DEPLOY_HOST="${NAS_HOST:-nas}"
DEPLOY_DIR="${DEPLOY_DIR:-/vol1/qmdsearch}"
APP_PORT="${APP_PORT:-8000}"
ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
DRY_RUN=0
[[ "${1:-}" == "--dry-run" ]] && DRY_RUN=1
# 前置检查
if ! ssh -o BatchMode=yes -o ConnectTimeout=5 "$DEPLOY_HOST" true 2>/dev/null; then
echo "[local] 无法免密 SSH 到 $DEPLOY_HOST,请先配置 SSH 密钥登录。" >&2
exit 1
fi
command -v rsync >/dev/null 2>&1 || { echo "[local] 本机缺少 rsync,请先安装。" >&2; exit 1; }
# ---- Phase 1: NAS 同步受管代码 + 计算变更范围 ----
SCOPE=$(ssh "$DEPLOY_HOST" "bash -s -- '${DEPLOY_DIR}'" <<'REMOTE'
DEPLOY_DIR="$1"
set -uo pipefail
cd "$DEPLOY_DIR" || { echo "NOCD"; exit 1; }
git fetch origin main 2>&1 | tail -n 2
NEW=$(git rev-parse origin/main)
PREV=$(cat .last_deployed_commit 2>/dev/null || echo "NONE")
if [[ "$PREV" == "$NEW" ]]; then echo "UP_TO_DATE"; exit 0; fi
OLD_HEAD=$(git rev-parse HEAD)
git reset --hard origin/main 2>&1 | tail -n 2
BASE="$OLD_HEAD"
[[ "$PREV" != "NONE" ]] && BASE="$PREV"
CHANGED=$(git diff --name-only "$BASE" "$NEW")
FE=0; DEPS=0
echo "$CHANGED" | grep -qE '^frontend/(src/|public/|index\.html|package\.json|vite\.config|package-lock\.json)' && FE=1
echo "$CHANGED" | grep -qE '^(pyproject\.toml|uv\.lock|Dockerfile)' && DEPS=1
echo "SCOPE FE=$FE DEPS=$DEPS PREV=$PREV NEW=$NEW"
REMOTE
)
if echo "$SCOPE" | grep -q "NOCD"; then echo "[nas] 无法进入 $DEPLOY_DIR"; exit 1; fi
if echo "$SCOPE" | grep -q "UP_TO_DATE"; then echo "[nas] 已是最新,无需更新。"; exit 0; fi
FE=$(echo "$SCOPE" | grep -oE 'FE=[01]' | cut -d= -f2)
DEPS=$(echo "$SCOPE" | grep -oE 'DEPS=[01]' | cut -d= -f2)
PREV=$(echo "$SCOPE" | grep -oE 'PREV=[0-9a-f]+' | cut -d= -f2)
NEW=$(echo "$SCOPE" | grep -oE 'NEW=[0-9a-f]+' | cut -d= -f2)
echo "[deploy] 将部署 $PREV -> $NEW (FE=$FE DEPS=$DEPS)"
if [[ "$DRY_RUN" == "1" ]]; then
echo "[deploy][dry-run] 变更文件:"
if [[ "$PREV" != "NONE" ]]; then
ssh "$DEPLOY_HOST" "cd $DEPLOY_DIR && git diff --name-only $PREV $NEW" 2>/dev/null | sed 's/^/ /'
else
echo " (首次部署,将全量同步受管代码)"
fi
exit 0
fi
# ---- Phase 2: 前端本地构建 + rsync 到 NASNAS 无 node----
if [[ "$FE" == "1" ]]; then
echo "[local] 本地构建前端..."
if ! (cd "$ROOT/frontend" && npm ci --no-audit --no-fund && npm run build); then
echo "[local][WARN] 前端构建失败(当前环境可能受限或缺少 node 依赖),跳过前端同步;后端仍会部署,但 frontend/dist 可能为旧版本。待在可正常构建的环境中重跑即补齐。"
else
echo "[local] rsync frontend/dist -> $DEPLOY_HOST:$DEPLOY_DIR/frontend/dist"
if ! rsync -az --delete "$ROOT/frontend/dist/" "$DEPLOY_HOST:$DEPLOY_DIR/frontend/dist/"; then
echo "[local][WARN] rsync 失败,跳过前端同步。"
fi
fi
fi
# ---- Phase 3: NAS 容器部署 + 健康检查 ----
ssh "$DEPLOY_HOST" "bash -s -- '$DEPLOY_DIR' '$APP_PORT' '$DEPS' '$NEW' '$PREV'" <<'REMOTE'
DEPLOY_DIR="$1"; APP_PORT="$2"; DEPS="$3"; NEW="$4"; PREV="$5"
set -uo pipefail
cd "$DEPLOY_DIR"
if [[ "$DEPS" == "1" ]]; then
echo "[nas] 依赖变化,重建镜像: docker build -t qmdsearch-app:latest ."
if ! docker build -t qmdsearch-app:latest . 2>&1 | tail -n 20; then
echo "[nas] 镜像构建失败,回滚到 $PREV"; git checkout -f "$PREV"; exit 1
fi
echo "[nas] 用新镜像重建容器: docker compose up -d --force-recreate app"
docker compose up -d --force-recreate app 2>&1 | tail -n 10
else
echo "[nas] 仅代码/前端变化,重启 app: docker compose restart app"
docker compose restart app 2>&1 | tail -n 5
fi
echo "[nas] 健康检查 http://localhost:$APP_PORT/api/v1/health ..."
for i in $(seq 1 20); do
if curl -fsS "http://localhost:$APP_PORT/api/v1/health" >/dev/null 2>&1; then
echo "$NEW" > .last_deployed_commit
echo "[nas] 健康检查通过 ✓ 部署完成"
exit 0
fi
sleep 3
done
echo "[nas] 健康检查失败,回滚到 $PREV"
git checkout -f "$PREV"
docker compose restart app
exit 1
REMOTE
+107
View File
@@ -0,0 +1,107 @@
#!/usr/bin/env bash
# =============================================================================
# QMDSearch NAS 端自动部署钩子 (post-receive)
#
# 这是真正的「持续部署(CD)」:当本地 deploy.sh 把代码 push 到 NAS 后,
# 此钩子自动完成 checkout → 按需构建 → 重启/重建容器 → 健康检查 → 失败回滚。
#
# 安装方式(二选一):
# A. 作为 bare 仓库的 hook(推荐,全自动):
# # 假设 NAS 上 bare 仓库在 /volume1/git/QMDSearch.git,工作树在 /volume1/docker/QMDSearch
# cp scripts/nas-post-receive /volume1/git/QMDSearch.git/hooks/post-receive
# chmod +x /volume1/git/QMDSearch.git/hooks/post-receive
# # 在该 bare 仓库 config 中设置:
# # [core] bare = true
# # [hooks] 并确保工作树目录已 clone 过一次
# # 首次需手动:git --work-tree=/volume1/docker/QMDSearch --git-dir=/volume1/git/QMDSearch.git checkout -f
#
# B. 作为普通脚本手动触发(无 bare 仓库时):
# cd /volume1/docker/QMDSearch && git pull && sudo bash scripts/nas-post-receive
#
# 前置依赖(NAS 上需具备):git、docker、docker compose、node/npm
# 可调环境变量:DEPLOY_DIR(工作树)、APP_PORT、LOG_FILE
# =============================================================================
set -uo pipefail
DEPLOY_DIR="${DEPLOY_DIR:-/volume1/docker/QMDSearch}" # ← 改成你的 NAS 部署目录
GIT_DIR="$DEPLOY_DIR/.git"
LOG_FILE="${LOG_FILE:-$DEPLOY_DIR/deploy.log}"
APP_PORT="${APP_PORT:-8000}"
PREV_COMMIT_FILE="$DEPLOY_DIR/.last_deployed_commit"
log(){ printf '%s [nas-deploy] %s\n' "$(date '+%Y-%m-%d %H:%M:%S')" "$*" | tee -a "$LOG_FILE"; }
rollback(){
log "回滚到上一稳定提交: ${PREV:-<无>}"
if [[ -n "${PREV:-}" ]]; then
git --work-tree="$DEPLOY_DIR" --git-dir="$GIT_DIR" checkout -f "$PREV"
(cd "$DEPLOY_DIR" && docker compose restart app) | tee -a "$LOG_FILE"
echo "$PREV" > "$PREV_COMMIT_FILE"
fi
}
# 解析 post-receive 通过 stdin 传入的 <old> <new> <ref>
newrev=""; refname=""
while read -r oldrev newrev refname; do :; done
# 仅处理 main 分支的推送;手动运行时 newrev 为空,回退到 HEAD
if [[ -z "$newrev" ]]; then
newrev="$(git --git-dir="$GIT_DIR" rev-parse HEAD 2>/dev/null || echo "")"
fi
if [[ -n "$refname" && "$refname" != "refs/heads/main" ]]; then
log "忽略非 main 分支推送 ($refname)"
exit 0
fi
log "=== 开始部署 (newrev=${newrev:0:8}) ==="
# 1) 检出到工作树
git --work-tree="$DEPLOY_DIR" --git-dir="$GIT_DIR" checkout -f
cd "$DEPLOY_DIR" || { log "无法进入 $DEPLOY_DIR"; exit 1; }
# 2) 计算变化范围
PREV="$(cat "$PREV_COMMIT_FILE" 2>/dev/null || echo "")"
if [[ -n "$PREV" && -n "$newrev" ]]; then
CHANGED="$(git --git-dir="$GIT_DIR" diff --name-only "$PREV" "$newrev")"
else
CHANGED="$(git --git-dir="$GIT_DIR" show --name-only --format= "$newrev")"
fi
has(){ echo "$CHANGED" | grep -qE "$1"; }
FE_CHANGED=0; DEPS_CHANGED=0
has '^frontend/(src/|public/|index\.html|package\.json|vite\.config|package-lock\.json)' && FE_CHANGED=1
has '^(pyproject\.toml|uv\.lock|Dockerfile)' && DEPS_CHANGED=1
log "变更范围 → frontend=$FE_CHANGED deps=$DEPS_CHANGED"
# 3) 按需构建前端(dist 不入库,必须在此构建)
if [[ "$FE_CHANGED" -eq 1 ]]; then
log "检测前端变化,重新构建..."
if ! (cd frontend && npm ci --no-audit --no-fund && npm run build) 2>&1 | tee -a "$LOG_FILE"; then
log "前端构建失败,中止部署(保留上一版本)"; exit 1
fi
ok(){ :; }; log "前端构建完成"
fi
# 4) 部署容器
if [[ "$DEPS_CHANGED" -eq 1 ]]; then
log "依赖变化,重建镜像并重启: docker compose up -d --build app"
docker compose up -d --build app 2>&1 | tee -a "$LOG_FILE"
else
log "仅代码变化,重启 app 服务: docker compose restart app"
docker compose restart app 2>&1 | tee -a "$LOG_FILE"
fi
# 5) 健康检查(失败自动回滚)
log "健康检查 http://localhost:$APP_PORT/health ..."
for i in $(seq 1 15); do
if curl -fsS "http://localhost:$APP_PORT/health" >/dev/null 2>&1; then
log "健康检查通过 ✓"
[[ -n "$newrev" ]] && echo "$newrev" > "$PREV_COMMIT_FILE"
log "=== 部署完成 ==="
exit 0
fi
sleep 2
done
log "健康检查失败,触发回滚"
rollback
exit 1
+58
View File
@@ -0,0 +1,58 @@
#!/usr/bin/env bash
# =============================================================================
# QMDSearch 文件变化监视器(纯轮询,无需 fswatch / inotifywait
#
# 作用:持续监听仓库代码变化,实现「每次更新代码后自动检测」。
# - 默认模式:检测到变化 → 本地跑测试 + 前端预构建 → 提示你运行 deploy.sh
# - --auto 模式:检测到变化 → 直接调用 ./scripts/deploy.sh 全自动部署
# (注意:--auto 会在保存即提交推送,慎用,建议确认改动完整后再用)
#
# 用法:
# ./scripts/watch.sh 监听并本地校验,提示部署
# ./scripts/watch.sh --auto 监听变化后全自动部署
# ./scripts/watch.sh --interval 5 轮询间隔秒数(默认 3)
# =============================================================================
set -uo pipefail
ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
cd "$ROOT"
AUTO=0
INTERVAL=3
while [[ $# -gt 0 ]]; do
case "$1" in
--auto) AUTO=1 ;;
--interval) INTERVAL="${2:-3}"; shift ;;
*) echo "未知参数: $1"; exit 2 ;;
esac
shift
done
SNAPSHOT="$(mktemp)"
record(){ git status --porcelain > "$SNAPSHOT"; }
record
if [[ "$AUTO" -eq 1 ]]; then
echo "[watch] 监听 $ROOTAUTO 模式:变化即部署,间隔 ${INTERVAL}sCtrl-C 退出)"
else
echo "[watch] 监听 $ROOT(校验模式:变化即本地校验并提示,间隔 ${INTERVAL}sCtrl-C 退出)"
fi
while true; do
sleep "$INTERVAL"
if ! git status --porcelain | diff -q - "$SNAPSHOT" >/dev/null 2>&1; then
echo "[watch] $(date '+%H:%M:%S') 检测到代码变化"
if [[ "$AUTO" -eq 1 ]]; then
bash "$ROOT/scripts/deploy.sh"
else
echo "[watch] 运行本地校验..."
uv run pytest -q --no-header 2>&1 | tail -n 10 || true
if git status --porcelain | awk '{print $2}' | grep -qE '^frontend/'; then
echo "[watch] 前端变化,本地预构建..."
(cd frontend && npm ci --no-audit --no-fund && npm run build) 2>&1 | tail -n 5 || true
fi
echo "[watch] 校验完成。确认无误后运行: ./scripts/deploy.sh"
fi
record
fi
done
+120 -1
View File
@@ -13,6 +13,8 @@
10. 请求拦截:Authorization Bearer 注入、1005 回登录、1006 错误条 10. 请求拦截:Authorization Bearer 注入、1005 回登录、1006 错误条
11. API 指南区块:导航/section、API_GUIDE 清单与真实路由一致性、试一下面板、 11. API 指南区块:导航/section、API_GUIDE 清单与真实路由一致性、试一下面板、
curl 复制、auth 标注、upload 文件选择、禁止自定义 URL curl 复制、auth 标注、upload 文件选择、禁止自定义 URL
12. 入库进度区块(Task 2):section/nav、表格结构、2s 轮询启停、状态徽章复用、
done/failed 操作按钮、reingest 端点、批量上传 multiple + upload-batch 路径
""" """
import re import re
@@ -39,9 +41,20 @@ def client(monkeypatch: pytest.MonkeyPatch) -> Iterator[TestClient]:
@pytest.fixture @pytest.fixture
def admin_html(client: TestClient) -> str: def admin_html(client: TestClient) -> str:
"""请求 /admin 并返回 HTML 文本(前置断言 200""" """请求 /admin 并返回旧版 admin.html 文本(前置断言 200
当 app/static/admin 符号链接存在时(Vue SPA 部署),/admin 端点会返回 SPA 入口
而非旧版 admin.html。此时直接读取 admin.html 文件内容进行测试。
"""
resp = client.get("/admin") resp = client.get("/admin")
assert resp.status_code == 200 assert resp.status_code == 200
# 检测是否返回了 Vue SPA(无 section-overview 标记)
if "section-overview" not in resp.text:
# SPA 模式:直接读取旧版 admin.html 文件
from pathlib import Path
admin_html_path = Path(__file__).resolve().parent.parent / "app" / "static" / "admin.html"
return admin_html_path.read_text(encoding="utf-8")
return resp.text return resp.text
@@ -86,6 +99,8 @@ def test_admin_page_fetch_paths(admin_html: str) -> None:
"/api/v1/auth/logout", "/api/v1/auth/logout",
"/api/v1/auth/password", "/api/v1/auth/password",
"/api/v1/auth/users", "/api/v1/auth/users",
"/api/v1/documents/upload-batch",
"/api/v1/documents/tasks",
): ):
assert path in admin_html assert path in admin_html
@@ -107,6 +122,13 @@ def test_admin_page_fetch_paths_in_real_routes(admin_html: str) -> None:
"""页面 api() 调用路径均在真实后端路由集合内(含 tasks 与 /api/v1/auth/* 路径)""" """页面 api() 调用路径均在真实后端路由集合内(含 tasks 与 /api/v1/auth/* 路径)"""
route_paths = _collect_route_paths(app.routes) route_paths = _collect_route_paths(app.routes)
assert "/api/v1/documents/tasks/{task_id}" in route_paths assert "/api/v1/documents/tasks/{task_id}" in route_paths
# Task 2 新增端点在真实路由集合内
for new_path in (
"/api/v1/documents/tasks",
"/api/v1/documents/upload-batch",
"/api/v1/documents/{doc_id}/reingest",
):
assert new_path in route_paths, f"后端缺少 Task 2 新增端点: {new_path}"
for auth_path in ( for auth_path in (
"/api/v1/auth/login", "/api/v1/auth/login",
"/api/v1/auth/me", "/api/v1/auth/me",
@@ -140,6 +162,99 @@ def test_admin_page_ingest_polling(admin_html: str) -> None:
assert "disabled" in admin_html assert "disabled" in admin_html
def test_admin_page_progress_section(admin_html: str) -> None:
"""入库进度区块:导航按钮、section、表格结构、刷新按钮、空列表占位"""
# section 与导航按钮(所有登录用户可见)
assert 'id="section-progress"' in admin_html
assert 'id="nav-progress"' in admin_html
assert 'data-target="section-progress"' in admin_html
assert "入库进度" in admin_html
# 刷新按钮与状态统计
assert 'id="btn-refresh-progress"' in admin_html
assert 'id="progress-stats"' in admin_html
# 表格 tbody 与表头列:文件名 / 状态 / 创建时间 / 更新时间 / 操作
assert 'id="progress-tbody"' in admin_html
for col in ("文件名", "状态", "创建时间", "更新时间", "操作"):
assert col in admin_html
# 空列表占位文案
assert "暂无入库任务" in admin_html
def test_admin_page_progress_polling(admin_html: str) -> None:
"""入库进度区块:2s 轮询常量、startProgressPolling/stopProgressPolling 函数、GET /documents/tasks 路径"""
# 进度区块专用轮询常量(2s
assert "PROGRESS_POLL_INTERVAL_MS" in admin_html
assert "2000" in admin_html
# 轮询启停函数
assert "function startProgressPolling" in admin_html
assert "function stopProgressPolling" in admin_html
assert "progressPollTimer" in admin_html
# 数据来源:GET /documents/tasks?limit=50
assert "/api/v1/documents/tasks?limit=50" in admin_html
# loadProgress / renderProgress / buildProgressRow 函数
assert "function loadProgress" in admin_html
assert "function renderProgress" in admin_html
assert "function buildProgressRow" in admin_html
def test_admin_page_progress_polling_lifecycle(admin_html: str) -> None:
"""进度区块轮询生命周期:activateSection 切走暂停、切回恢复;showLogin 停止"""
# activateSection 开头调用 stopProgressPolling(切走即暂停)
assert "stopProgressPolling();" in admin_html
# 切到 section-progress 时恢复轮询
assert 'targetId === "section-progress"' in admin_html
assert 'startProgressPolling()' in admin_html
# showLogin 退出时停止轮询
assert "stopProgressPolling" in admin_html
def test_admin_page_progress_status_badge_reuse(admin_html: str) -> None:
"""进度表格状态徽章复用 INGEST_STATUS_TEXT 中文映射与 status-* 配色"""
# statusBadgeEl 共享构造函数(无 id 可重复使用)
assert "function statusBadgeEl" in admin_html
# makeStatusBadge 复用 statusBadgeEl + 加 id
assert "statusBadgeEl(status)" in admin_html
# 状态徽章 CSS 类复用
for cls in ("status-running", "status-done", "status-failed"):
assert cls in admin_html
def test_admin_page_progress_actions(admin_html: str) -> None:
"""进度表格操作列:done 查看文档+重新入库、failed 重试、进行中无操作;reingest 端点"""
# done 状态操作按钮
assert "查看文档" in admin_html
assert "重新入库" in admin_html
# failed 状态重试按钮
assert "重试" in admin_html
# reingestDocument 函数与 POST /documents/{doc_id}/reingest 端点
assert "function reingestDocument" in admin_html
assert "/reingest" in admin_html
# 查看文档:切到 section-docs 并调用 loadDocDetail
assert 'activateSection("section-docs")' in admin_html
assert "loadDocDetail" in admin_html
# 状态统计文案
assert "进行中" in admin_html
def test_admin_page_batch_upload(admin_html: str) -> None:
"""批量上传:input multiple 属性、upload-batch 端点、批量结果展示、自动切进度区块"""
# file input multiple 属性
assert 'id="upload-file"' in admin_html
assert "multiple" in admin_html
# 批量上传分支:files > 1 时走 upload-batch 端点
assert "/api/v1/documents/upload-batch" in admin_html
assert 'fileInput.files.length > 1' in admin_html
# FormData 用 files 字段逐文件 append
assert 'batchForm.append("files"' in admin_html
# 批量结果展示函数
assert "function renderBatchUploadResult" in admin_html
assert "批量上传完成" in admin_html
# 成功后自动切到入库进度区块
assert 'activateSection("section-progress")' in admin_html
# 单文件上传仍走原 upload 端点(保持既有逻辑)
assert "/api/v1/documents/upload\"" in admin_html or "/api/v1/documents/upload'," in admin_html
def test_admin_page_no_prompt_confirm_calls(admin_html: str) -> None: def test_admin_page_no_prompt_confirm_calls(admin_html: str) -> None:
"""清理:弹窗组件已替换原生 prompt/confirm,全页面无 prompt( 与 confirm( 调用残留""" """清理:弹窗组件已替换原生 prompt/confirm,全页面无 prompt( 与 confirm( 调用残留"""
assert "prompt(" not in admin_html assert "prompt(" not in admin_html
@@ -490,15 +605,19 @@ def test_admin_page_api_guide_paths_in_real_routes(admin_html: str) -> None:
assert (method, path) in route_methods, f"API_GUIDE 端点 {method} {path} 不在后端路由集合内" assert (method, path) in route_methods, f"API_GUIDE 端点 {method} {path} 不在后端路由集合内"
# 全部真实端点覆盖:health/search/documents*/knowledge*/auth* # 全部真实端点覆盖:health/search/documents*/knowledge*/auth*
# Task 2 新增端点:upload-batch / tasks 列表 / reingest
expected = { expected = {
("GET", "/api/v1/health"), ("GET", "/api/v1/health"),
("POST", "/api/v1/search"), ("POST", "/api/v1/search"),
("POST", "/api/v1/documents"), ("POST", "/api/v1/documents"),
("POST", "/api/v1/documents/upload"), ("POST", "/api/v1/documents/upload"),
("POST", "/api/v1/documents/upload-batch"),
("GET", "/api/v1/documents/tasks"),
("GET", "/api/v1/documents/tasks/{task_id}"), ("GET", "/api/v1/documents/tasks/{task_id}"),
("GET", "/api/v1/documents"), ("GET", "/api/v1/documents"),
("GET", "/api/v1/documents/{doc_id}"), ("GET", "/api/v1/documents/{doc_id}"),
("DELETE", "/api/v1/documents/{doc_id}"), ("DELETE", "/api/v1/documents/{doc_id}"),
("POST", "/api/v1/documents/{doc_id}/reingest"),
("GET", "/api/v1/knowledge/categories"), ("GET", "/api/v1/knowledge/categories"),
("GET", "/api/v1/knowledge/stats"), ("GET", "/api/v1/knowledge/stats"),
("POST", "/api/v1/auth/login"), ("POST", "/api/v1/auth/login"),
+335
View File
@@ -0,0 +1,335 @@
"""POST /documents/upload-batch、POST /documents/{id}/reingest、GET /documents/tasks 端点测试
TestClient + FakeManager + FakeQdrant,不真实联网。
"""
from collections.abc import Iterator
from pathlib import Path
from typing import Any
import pytest
from fastapi.testclient import TestClient
from app.api.v1 import document as document_module
from app.main import app
from app.models.document import DocumentInput
from app.services.qdrant import QdrantService
class FakeManager:
"""假入库任务管理器:支持 submit/get/list_tasks"""
def __init__(self, task_id: str = "task-1") -> None:
self.task_id = task_id
self._counter = 0
self.submitted: list[DocumentInput] = []
self._tasks: list[dict[str, Any]] = []
async def submit(self, doc: DocumentInput) -> str:
self.submitted.append(doc)
self._counter += 1
tid = f"{self.task_id}-{self._counter}"
self._tasks.append(
{
"task_id": tid,
"status": "pending",
"filename": doc.metadata.get("original_filename") or doc.title,
"created_at": f"2026-01-0{self._counter}T00:00:00+00:00",
"updated_at": f"2026-01-0{self._counter}T00:00:00+00:00",
"doc_id": None,
}
)
return tid
async def get(self, task_id: str) -> dict[str, Any] | None:
return None
async def list_tasks(self, limit: int = 20) -> list[dict[str, Any]]:
return list(self._tasks[:limit])
class FakeQdrant:
"""假 Qdrant 服务:可控的 get_l1_metadata/delete_by_doc_id"""
def __init__(self, meta: dict[str, str] | None = None) -> None:
self.meta = meta
self.deleted: list[str] = []
async def get_l1_metadata(self, doc_id: str) -> dict[str, str] | None:
return self.meta
async def delete_by_doc_id(self, doc_id: str) -> dict[str, int]:
self.deleted.append(doc_id)
return {"doc_l1": 1}
@pytest.fixture
def client(
monkeypatch: pytest.MonkeyPatch, tmp_path: Path, admin_headers: dict[str, str]
) -> Iterator[TestClient]:
async def _noop_ensure_collections(self: QdrantService) -> None:
return None
monkeypatch.setattr(QdrantService, "ensure_collections", _noop_ensure_collections)
monkeypatch.setattr(document_module.settings, "upload_dir", str(tmp_path / "uploads"))
with TestClient(app) as test_client:
test_client.headers.update(admin_headers)
yield test_client
def _inject_manager(monkeypatch: pytest.MonkeyPatch, manager: FakeManager) -> None:
monkeypatch.setattr(document_module, "_get_task_manager", lambda: manager)
def _inject_qdrant(monkeypatch: pytest.MonkeyPatch, qdrant: FakeQdrant) -> None:
monkeypatch.setattr(document_module, "_get_qdrant", lambda: qdrant)
# ----------------------- upload-batch -----------------------
def test_batch_upload_multiple_files_success(
client: TestClient, monkeypatch: pytest.MonkeyPatch
) -> None:
"""多文件全部成功:3 文件 → 3 tasksfailed 为空"""
manager = FakeManager()
_inject_manager(monkeypatch, manager)
resp = client.post(
"/api/v1/documents/upload-batch",
files=[
("files", ("a.txt", b"content a", "text/plain")),
("files", ("b.md", b"# B", "text/markdown")),
("files", ("c.txt", b"content c", "text/plain")),
],
)
assert resp.status_code == 202
data = resp.json()["data"]
assert len(data["tasks"]) == 3
assert data["failed"] == []
assert len(manager.submitted) == 3
filenames = [t["filename"] for t in data["tasks"]]
assert filenames == ["a.txt", "b.md", "c.txt"]
def test_batch_upload_partial_failure(
client: TestClient, monkeypatch: pytest.MonkeyPatch
) -> None:
"""部分失败:1 个不支持扩展名进 failed,其他成功"""
manager = FakeManager()
_inject_manager(monkeypatch, manager)
resp = client.post(
"/api/v1/documents/upload-batch",
files=[
("files", ("ok.txt", b"good", "text/plain")),
("files", ("bad.xlsx", b"binary", "application/octet-stream")),
],
)
assert resp.status_code == 202
data = resp.json()["data"]
assert len(data["tasks"]) == 1
assert data["tasks"][0]["filename"] == "ok.txt"
assert len(data["failed"]) == 1
assert data["failed"][0]["filename"] == "bad.xlsx"
assert "不支持" in data["failed"][0]["error"]
assert len(manager.submitted) == 1
def test_batch_upload_all_fail(
client: TestClient, monkeypatch: pytest.MonkeyPatch
) -> None:
"""全部失败:tasks 空,failed 2 条,未提交任何任务"""
manager = FakeManager()
_inject_manager(monkeypatch, manager)
resp = client.post(
"/api/v1/documents/upload-batch",
files=[
("files", ("a.exe", b"x", "application/octet-stream")),
("files", ("b.bin", b"y", "application/octet-stream")),
],
)
assert resp.status_code == 202
data = resp.json()["data"]
assert data["tasks"] == []
assert len(data["failed"]) == 2
assert len(manager.submitted) == 0
def test_batch_upload_empty_list_returns_1001(
client: TestClient, monkeypatch: pytest.MonkeyPatch
) -> None:
"""空文件列表:1001"""
manager = FakeManager()
_inject_manager(monkeypatch, manager)
resp = client.post("/api/v1/documents/upload-batch", files=[])
body = resp.json()
assert body["code"] == 1001
assert manager.submitted == []
# ----------------------- reingest -----------------------
def test_reingest_success(
client: TestClient, monkeypatch: pytest.MonkeyPatch, tmp_path: Path
) -> None:
"""有原始文件:202 + task_id,删旧数据后提交新任务"""
upload_dir = tmp_path / "uploads"
upload_dir.mkdir(parents=True, exist_ok=True)
raw = upload_dir / "raw_report.txt"
raw.write_bytes(b"original content")
meta = {
"raw_file_path": str(raw),
"original_filename": "raw_report.txt",
}
qdrant = FakeQdrant(meta=meta)
_inject_qdrant(monkeypatch, qdrant)
manager = FakeManager(task_id="reingest-task")
_inject_manager(monkeypatch, manager)
resp = client.post("/api/v1/documents/some-doc/reingest")
assert resp.status_code == 202
data = resp.json()["data"]
assert data["task_id"] == "reingest-task-1"
assert data["status"] == "pending"
# 旧数据已删
assert qdrant.deleted == ["some-doc"]
# 新任务已提交,文本来自原文件
assert len(manager.submitted) == 1
assert manager.submitted[0].text == "original content"
assert manager.submitted[0].title == "raw_report" # 文件名 stem
def test_reingest_no_raw_file_returns_1001(
client: TestClient, monkeypatch: pytest.MonkeyPatch
) -> None:
"""无原始文件(meta 无 raw_file_path):1001"""
qdrant = FakeQdrant(meta={"some": "metadata"})
_inject_qdrant(monkeypatch, qdrant)
manager = FakeManager()
_inject_manager(monkeypatch, manager)
resp = client.post("/api/v1/documents/doc1/reingest")
body = resp.json()
assert body["code"] == 1001
assert "无原始文件" in body["message"]
assert len(manager.submitted) == 0
def test_reingest_missing_file_returns_1004(
client: TestClient, monkeypatch: pytest.MonkeyPatch
) -> None:
"""文件不存在:1004"""
qdrant = FakeQdrant(
meta={"raw_file_path": "/nonexistent/path.txt", "original_filename": "path.txt"}
)
_inject_qdrant(monkeypatch, qdrant)
manager = FakeManager()
_inject_manager(monkeypatch, manager)
resp = client.post("/api/v1/documents/doc1/reingest")
body = resp.json()
assert body["code"] == 1004
assert len(manager.submitted) == 0
def test_reingest_doc_not_found_returns_1004(
client: TestClient, monkeypatch: pytest.MonkeyPatch
) -> None:
"""文档不存在(meta 为 None):1004"""
qdrant = FakeQdrant(meta=None)
_inject_qdrant(monkeypatch, qdrant)
manager = FakeManager()
_inject_manager(monkeypatch, manager)
resp = client.post("/api/v1/documents/nope/reingest")
body = resp.json()
assert body["code"] == 1004
assert len(manager.submitted) == 0
# ----------------------- tasks 列表 -----------------------
def test_list_ingest_tasks_returns_items(
client: TestClient, monkeypatch: pytest.MonkeyPatch
) -> None:
"""任务列表:返回近期任务,含 items 与 total"""
manager = FakeManager()
_inject_manager(monkeypatch, manager)
manager._tasks = [
{
"task_id": "t1",
"status": "done",
"filename": "a.txt",
"created_at": "2026-01-01T00:00:00+00:00",
"updated_at": "2026-01-01T00:00:00+00:00",
"doc_id": "d1",
},
{
"task_id": "t2",
"status": "failed",
"filename": "b.txt",
"created_at": "2026-01-02T00:00:00+00:00",
"updated_at": "2026-01-02T00:00:00+00:00",
"doc_id": None,
},
]
resp = client.get("/api/v1/documents/tasks")
assert resp.status_code == 200
data = resp.json()["data"]
assert data["total"] == 2
assert len(data["items"]) == 2
assert data["items"][0]["task_id"] == "t1"
assert data["items"][0]["doc_id"] == "d1"
def test_list_ingest_tasks_limit(
client: TestClient, monkeypatch: pytest.MonkeyPatch
) -> None:
"""limit 截断生效"""
manager = FakeManager()
_inject_manager(monkeypatch, manager)
manager._tasks = [
{
"task_id": f"t{i}",
"status": "done",
"filename": "f",
"created_at": "",
"updated_at": "",
"doc_id": None,
}
for i in range(10)
]
resp = client.get("/api/v1/documents/tasks?limit=5")
data = resp.json()["data"]
assert data["total"] == 5
assert len(data["items"]) == 5
def test_list_ingest_tasks_requires_auth(
client: TestClient, monkeypatch: pytest.MonkeyPatch
) -> None:
"""无 token1005 未认证"""
manager = FakeManager()
_inject_manager(monkeypatch, manager)
resp = client.get("/api/v1/documents/tasks", headers={"Authorization": ""})
body = resp.json()
assert body["code"] == 1005
+339
View File
@@ -0,0 +1,339 @@
"""批量上传与入库进度集成验证
在真实内存 Qdrant + FakeOllama + 确定性向量环境下,全链路验证:
- POST /documents/upload-batch 批量上传闭环(全部成功 / 部分失败)
- GET /documents/tasks 任务列表(按 updated_at 降序、每项含 filename/doc_id
- POST /documents/{doc_id}/reingest 重新入库(成功 / 边界 1001/1004
复用 tests/test_e2e_integration.py 的 FakeOllama / DeterministicEmbedding / FakeCache
upload_dir 用 tmp_path 真实落盘,monkeypatch 模块级 _task_manager / _qdrant 单例。
"""
from pathlib import Path
import pytest
from fastapi.testclient import TestClient
from qdrant_client import AsyncQdrantClient
from app.api.v1 import document as document_module
from app.config import Settings
from app.core.chunker import Chunker
from app.core.classifier import Classifier
from app.core.ingest_tasks import IngestTaskManager, IngestTaskStatus
from app.core.ingestion import Ingester
from app.core.query_parser import QueryParser
from app.core.retriever import Retriever
from app.core.sparse import SparseEncoder
from app.core.summarizer import Summarizer
from app.main import app
from app.models.knowledge import load_taxonomy
from app.services.qdrant import COLLECTION_CHUNKS, QdrantService
from tests.test_e2e_integration import (
FAKE_L1_SUMMARY,
DeterministicEmbedding,
FakeCache,
FakeOllama,
)
async def _make_env() -> tuple[QdrantService, Ingester, Retriever]:
"""构建集成环境:真实组件 + 内存 Qdrant + FakeOllama + 确定性向量"""
qdrant = QdrantService(client=AsyncQdrantClient(location=":memory:"))
await qdrant.ensure_collections()
ollama = FakeOllama()
taxonomy = load_taxonomy()
embedding = DeterministicEmbedding()
ingester = Ingester(
summarizer=Summarizer(ollama=ollama), # type: ignore[arg-type]
classifier=Classifier(ollama=ollama, taxonomy=taxonomy), # type: ignore[arg-type]
chunker=Chunker(),
embedding=embedding, # type: ignore[arg-type]
sparse=SparseEncoder(),
qdrant=qdrant,
)
retriever = Retriever(
qdrant=qdrant,
query_parser=QueryParser(ollama=ollama, taxonomy=taxonomy, cache=FakeCache()), # type: ignore[arg-type]
embedding=embedding, # type: ignore[arg-type]
sparse_encoder=SparseEncoder(),
)
return qdrant, ingester, retriever
def _patch_app(
monkeypatch: pytest.MonkeyPatch,
manager: IngestTaskManager,
qdrant: QdrantService,
retriever: Retriever,
upload_dir: Path,
) -> None:
"""替换模块级单例:任务管理器 / Qdrant / Retriever / upload_dirlifespan 建集合改空操作"""
async def _noop_ensure_collections(self: QdrantService) -> None:
return None
monkeypatch.setattr(QdrantService, "ensure_collections", _noop_ensure_collections)
monkeypatch.setattr(document_module, "_task_manager", manager)
monkeypatch.setattr(document_module, "_qdrant", qdrant)
monkeypatch.setattr(document_module.settings, "upload_dir", str(upload_dir))
def _md_content(name: str) -> bytes:
"""构造可触发完整三级总结的 .md 文件内容(不同 name 保证文本不同,避免去重)"""
paragraph = f"这是 {name} 章节的正文内容,包含足够信息量用于测试切分与向量化流程。" * 20
text = f"# {name} 安装指南\n{paragraph}\n\n## 环境准备\n{paragraph}\n\n## 安装步骤\n{paragraph}"
return text.encode("utf-8")
class TestBatchUploadIntegration:
"""批量上传闭环:3 文件全成功 → 任务列表 → 文档详情含原始文件信息"""
async def test_batch_upload_full_loop(
self,
monkeypatch: pytest.MonkeyPatch,
tmp_path: Path,
admin_headers: dict[str, str],
) -> None:
qdrant, ingester, retriever = await _make_env()
manager = IngestTaskManager(ingester, None, Settings())
_patch_app(monkeypatch, manager, qdrant, retriever, tmp_path / "uploads")
with TestClient(app) as client:
# 1. 批量上传 3 个 .md202 + tasks 含 3 个 + failed 空
resp = client.post(
"/api/v1/documents/upload-batch",
files=[
("files", ("a.md", _md_content("Alpha"), "text/markdown")),
("files", ("b.md", _md_content("Beta"), "text/markdown")),
("files", ("c.md", _md_content("Gamma"), "text/markdown")),
],
headers=admin_headers,
)
assert resp.status_code == 202
data = resp.json()["data"]
assert len(data["tasks"]) == 3
assert data["failed"] == []
task_ids = [t["task_id"] for t in data["tasks"]]
filenames = [t["filename"] for t in data["tasks"]]
assert filenames == ["a.md", "b.md", "c.md"]
# 2. 逐个等待终态:全部 done
doc_ids: list[str] = []
for tid in task_ids:
final = await manager.wait_done(tid)
assert final["status"] == IngestTaskStatus.DONE
doc_ids.append(final["result"]["document_id"])
# 3. GET /documents/tasksitems 含 3 项,按 updated_at 降序,每项含 filename/doc_id
resp_list = client.get(
"/api/v1/documents/tasks?limit=10", headers=admin_headers
)
body_list = resp_list.json()
assert body_list["code"] == 0
items = body_list["data"]["items"]
assert body_list["data"]["total"] == 3
assert len(items) == 3
# 按 updated_at 降序
updated = [it["updated_at"] for it in items]
assert updated == sorted(updated, reverse=True)
# 每项含 filename 与 doc_id
for it in items:
assert it["filename"] in {"a.md", "b.md", "c.md"}
assert it["doc_id"] in doc_ids
# 4. GET /documents/{doc_id}file 字段非 null(有原始文件信息)
resp_detail = client.get(f"/api/v1/documents/{doc_ids[0]}")
body_detail = resp_detail.json()
assert body_detail["code"] == 0
file_info = body_detail["data"]["file"]
assert file_info is not None
assert file_info["filename"] == filenames[0]
assert file_info["size_bytes"] > 0
assert file_info["url"] == f"/api/v1/documents/{doc_ids[0]}/file"
class TestBatchUploadPartialFailure:
"""部分失败:1 个 .unsupported 扩展名进 failed,其余 2 个成功"""
async def test_batch_upload_partial_failure(
self,
monkeypatch: pytest.MonkeyPatch,
tmp_path: Path,
admin_headers: dict[str, str],
) -> None:
qdrant, ingester, retriever = await _make_env()
manager = IngestTaskManager(ingester, None, Settings())
_patch_app(monkeypatch, manager, qdrant, retriever, tmp_path / "uploads")
with TestClient(app) as client:
resp = client.post(
"/api/v1/documents/upload-batch",
files=[
("files", ("ok1.md", _md_content("Ok1"), "text/markdown")),
("files", ("ok2.md", _md_content("Ok2"), "text/markdown")),
("files", ("bad.unsupported", b"whatever", "application/octet-stream")),
],
headers=admin_headers,
)
assert resp.status_code == 202
data = resp.json()["data"]
assert len(data["tasks"]) == 2
assert len(data["failed"]) == 1
assert data["failed"][0]["filename"] == "bad.unsupported"
assert "不支持" in data["failed"][0]["error"]
# 2 个成功任务均可进入终态 done
for t in data["tasks"]:
final = await manager.wait_done(t["task_id"])
assert final["status"] == IngestTaskStatus.DONE
class TestReingestIntegration:
"""重新入库:成功路径 + 边界(无原始文件 / 文档不存在 / 文件已删)"""
async def test_reingest_success(
self,
monkeypatch: pytest.MonkeyPatch,
tmp_path: Path,
admin_headers: dict[str, str],
) -> None:
qdrant, ingester, retriever = await _make_env()
manager = IngestTaskManager(ingester, None, Settings())
_patch_app(monkeypatch, manager, qdrant, retriever, tmp_path / "uploads")
with TestClient(app) as client:
# 先上传 1 个文件,拿到 doc_id
resp_up = client.post(
"/api/v1/documents/upload-batch",
files=[("files", ("re.md", _md_content("Reingest"), "text/markdown"))],
headers=admin_headers,
)
assert resp_up.status_code == 202
up_data = resp_up.json()["data"]
assert len(up_data["tasks"]) == 1
old_task_id = up_data["tasks"][0]["task_id"]
old_final = await manager.wait_done(old_task_id)
old_doc_id = old_final["result"]["document_id"]
# 重新入库:202 + task_id
resp_re = client.post(
f"/api/v1/documents/{old_doc_id}/reingest", headers=admin_headers
)
assert resp_re.status_code == 202
re_body = resp_re.json()
assert re_body["code"] == 0
new_task_id = re_body["data"]["task_id"]
assert new_task_id
# 等待终态:done
new_final = await manager.wait_done(new_task_id)
assert new_final["status"] == IngestTaskStatus.DONE
new_doc_id = new_final["result"]["document_id"]
# 旧 doc_id 数据已删:GET /documents/{old_doc_id} → 1004
resp_old = client.get(f"/api/v1/documents/{old_doc_id}")
assert resp_old.json()["code"] == 1004
# 新 doc_id 详情:L1 summary 存在,file 非 null(保留原文件信息)
resp_new = client.get(f"/api/v1/documents/{new_doc_id}")
body_new = resp_new.json()
assert body_new["code"] == 0
assert body_new["data"]["l1"]["text"] == FAKE_L1_SUMMARY
assert body_new["data"]["file"] is not None
assert body_new["data"]["file"]["filename"] == "re.md"
# 旧 chunks 已删:chunks 集合中 old_doc_id 点数为 0
old_count = await qdrant.client.count(
collection_name=COLLECTION_CHUNKS,
count_filter=qdrant.build_filter(doc_ids=[old_doc_id]),
exact=True,
)
assert old_count.count == 0
async def test_reingest_text_only_doc_returns_1001(
self,
monkeypatch: pytest.MonkeyPatch,
tmp_path: Path,
admin_headers: dict[str, str],
) -> None:
"""纯文本入库的 doc_id → reingest → 1001(无原始文件)"""
qdrant, ingester, retriever = await _make_env()
manager = IngestTaskManager(ingester, None, Settings())
_patch_app(monkeypatch, manager, qdrant, retriever, tmp_path / "uploads")
with TestClient(app) as client:
# 纯文本入库(无原始文件落盘记录)
resp_doc = client.post(
"/api/v1/documents",
json={
"text": "# 纯文本\n这是一段纯文本入库的内容,用于测试 reingest 边界。",
"title": "纯文本",
},
headers=admin_headers,
)
assert resp_doc.status_code == 202
task_id = resp_doc.json()["data"]["task_id"]
final = await manager.wait_done(task_id)
doc_id = final["result"]["document_id"]
# reingest → 1001
resp_re = client.post(
f"/api/v1/documents/{doc_id}/reingest", headers=admin_headers
)
body = resp_re.json()
assert body["code"] == 1001
assert "无原始文件" in body["message"]
async def test_reingest_doc_not_found_returns_1004(
self,
monkeypatch: pytest.MonkeyPatch,
tmp_path: Path,
admin_headers: dict[str, str],
) -> None:
"""不存在的 doc_id → reingest → 1004"""
qdrant, ingester, retriever = await _make_env()
manager = IngestTaskManager(ingester, None, Settings())
_patch_app(monkeypatch, manager, qdrant, retriever, tmp_path / "uploads")
with TestClient(app) as client:
resp = client.post(
"/api/v1/documents/nonexistent-doc/reingest", headers=admin_headers
)
body = resp.json()
assert body["code"] == 1004
async def test_reingest_file_deleted_returns_1004(
self,
monkeypatch: pytest.MonkeyPatch,
tmp_path: Path,
admin_headers: dict[str, str],
) -> None:
"""文件被删(删 upload_dir 下落盘文件后)→ reingest → 1004"""
qdrant, ingester, retriever = await _make_env()
manager = IngestTaskManager(ingester, None, Settings())
_patch_app(monkeypatch, manager, qdrant, retriever, tmp_path / "uploads")
with TestClient(app) as client:
# 上传 1 个文件
resp_up = client.post(
"/api/v1/documents/upload-batch",
files=[("files", ("del.md", _md_content("Deleted"), "text/markdown"))],
headers=admin_headers,
)
assert resp_up.status_code == 202
tid = resp_up.json()["data"]["tasks"][0]["task_id"]
final = await manager.wait_done(tid)
doc_id = final["result"]["document_id"]
# 删除 upload_dir 下落盘的原始文件
meta = await qdrant.get_l1_metadata(doc_id)
assert meta is not None
raw_path = Path(meta["raw_file_path"])
assert raw_path.is_file()
raw_path.unlink()
# reingest → 1004(文件不存在)
resp_re = client.post(
f"/api/v1/documents/{doc_id}/reingest", headers=admin_headers
)
body = resp_re.json()
assert body["code"] == 1004
+143 -1
View File
@@ -7,8 +7,8 @@ from datetime import datetime
from typing import Any from typing import Any
from app.config import Settings from app.config import Settings
from app.core.dedup import DEDUP_KEY_PREFIX
from app.core.ingest_tasks import ( from app.core.ingest_tasks import (
DEDUP_KEY_PREFIX,
REDIS_KEY_PREFIX, REDIS_KEY_PREFIX,
IngestTaskManager, IngestTaskManager,
IngestTaskStatus, IngestTaskStatus,
@@ -91,6 +91,19 @@ class FakeRedis:
async def get_json(self, key: str) -> dict[str, Any] | None: async def get_json(self, key: str) -> dict[str, Any] | None:
return self.store.get(key) return self.store.get(key)
def _get_client(self) -> Any:
"""返回支持 scan_iter 的假客户端(供 list_tasks 扫描键)"""
store = self.store
class _FakeClient:
async def scan_iter(self, match: str = "*") -> Any:
prefix = match[:-1] if match.endswith("*") else match
for key in list(store.keys()):
if key.startswith(prefix):
yield key
return _FakeClient()
async def test_submit_returns_immediately_and_completes() -> None: async def test_submit_returns_immediately_and_completes() -> None:
"""submit 立即返回;任务后台跑完为 done,结果完整,阶段序列齐全,Redis 镜像同步""" """submit 立即返回;任务后台跑完为 done,结果完整,阶段序列齐全,Redis 镜像同步"""
@@ -317,3 +330,132 @@ async def test_dedup_lookup_failure_falls_back_to_normal_pipeline() -> None:
final = await manager.wait_done(task, timeout=5) final = await manager.wait_done(task, timeout=5)
assert final["status"] == IngestTaskStatus.DONE assert final["status"] == IngestTaskStatus.DONE
assert len(ingester.calls) == 1 assert len(ingester.calls) == 1
async def test_submit_records_filename_from_metadata_or_title() -> None:
"""submitfilename 优先 metadata.original_filename,其次 title"""
ingester = FakeIngester()
manager = IngestTaskManager(ingester, None, Settings())
# 有 original_filename
t1 = await manager.submit(
DocumentInput(text="x", title="t1", metadata={"original_filename": "report.pdf"})
)
await manager.wait_done(t1, timeout=5)
# 无 original_filename,回退 title
t2 = await manager.submit(DocumentInput(text="y", title="my-title"))
await manager.wait_done(t2, timeout=5)
items = await manager.list_tasks(limit=20)
by_id = {it["task_id"]: it for it in items}
assert by_id[t1]["filename"] == "report.pdf"
assert by_id[t2]["filename"] == "my-title"
async def test_list_tasks_merges_memory_and_redis_and_sorts() -> None:
"""list_tasks:合并内存与 Redis 镜像,去重,按 updated_at 降序"""
ingester = FakeIngester()
redis = FakeRedis()
manager = IngestTaskManager(ingester, redis, Settings())
# 内存任务 1
t1 = await manager.submit(DocumentInput(text="内容A", title="t1"))
await manager.wait_done(t1, timeout=5)
# 内存任务 2
t2 = await manager.submit(DocumentInput(text="内容B", title="t2"))
await manager.wait_done(t2, timeout=5)
# Redis 镜像中独有任务(不在内存里,模拟重启后只存在 Redis 的历史记录)
redis.store[f"{REDIS_KEY_PREFIX}redis-only-1"] = {
"task_id": "redis-only-1",
"status": "done",
"filename": "legacy.md",
"created_at": "2020-01-01T00:00:00+00:00",
"updated_at": "2020-01-01T00:00:00+00:00",
"result": {"document_id": "doc-legacy"},
"error": None,
}
items = await manager.list_tasks(limit=20)
# 共 3 条(内存 2 + Redis 独有 1,内存的 2 也已镜像到 Redis 但按 task_id 去重)
assert len(items) == 3
# 按 updated_at 降序:内存任务(当前时间)排在 Redis 旧任务前
ids = [it["task_id"] for it in items]
assert "redis-only-1" in ids
assert ids[-1] == "redis-only-1" # 最旧排最后
# Redis 独有任务提取 doc_id 与 filename
legacy = next(it for it in items if it["task_id"] == "redis-only-1")
assert legacy["filename"] == "legacy.md"
assert legacy["doc_id"] == "doc-legacy"
# 内存 done 任务也提取 doc_id
mem_item = next(it for it in items if it["task_id"] == t1)
assert mem_item["doc_id"] == "doc-1"
async def test_list_tasks_dedups_memory_and_redis_by_task_id() -> None:
"""list_tasks:内存与 Redis 都有的同一 task_id 仅保留内存版本(去重)"""
ingester = FakeIngester()
redis = FakeRedis()
manager = IngestTaskManager(ingester, redis, Settings())
task_id = await manager.submit(DocumentInput(text="唯一", title="t"))
await manager.wait_done(task_id, timeout=5)
# Redis 镜像中给同一 task_id 篡改一个旧 status,验证内存版本胜出
redis.store[f"{REDIS_KEY_PREFIX}{task_id}"]["status"] = "pending"
items = await manager.list_tasks(limit=20)
assert len(items) == 1
assert items[0]["task_id"] == task_id
assert items[0]["status"] == IngestTaskStatus.DONE # 内存版本(done
async def test_list_tasks_respects_limit() -> None:
"""list_taskslimit 截断返回条数"""
ingester = FakeIngester()
manager = IngestTaskManager(ingester, None, Settings())
for i in range(5):
t = await manager.submit(DocumentInput(text=f"c{i}", title=f"t{i}"))
await manager.wait_done(t, timeout=5)
items = await manager.list_tasks(limit=3)
assert len(items) == 3
async def test_list_tasks_without_redis_returns_memory_only() -> None:
"""Redis 不可用:list_tasks 仅返回内存任务"""
ingester = FakeIngester()
manager = IngestTaskManager(ingester, None, Settings())
t1 = await manager.submit(DocumentInput(text="x", title="t1"))
await manager.wait_done(t1, timeout=5)
items = await manager.list_tasks(limit=20)
assert len(items) == 1
assert items[0]["task_id"] == t1
assert items[0]["filename"] == "t1"
assert items[0]["doc_id"] == "doc-1"
async def test_list_tasks_extract_fields_for_in_progress_task() -> None:
"""进行中任务:doc_id 为 Nonedone 时才从 result.document_id 提取)"""
ingester = FakeIngester()
ingester.gate = asyncio.Event() # 阻塞任务使其停留在最后一个阶段
manager = IngestTaskManager(ingester, None, Settings(ingest_max_concurrency=1))
t1 = await manager.submit(
DocumentInput(text="x", title="t1", metadata={"original_filename": "a.txt"})
)
await asyncio.wait_for(ingester.started.wait(), timeout=1)
await asyncio.sleep(0.05)
items = await manager.list_tasks(limit=20)
assert len(items) == 1
item = items[0]
assert item["task_id"] == t1
# 任务已推进到 writing 阶段(gate 阻塞前最后一个 progress_cb
assert item["status"] == IngestTaskStatus.WRITING
assert item["doc_id"] is None # 未完成,doc_id 为 None
assert item["filename"] == "a.txt"
ingester.gate.set()
await manager.wait_done(t1, timeout=5)