Compare commits
3 Commits
5d3c895542
...
v0.2.3
| Author | SHA1 | Date | |
|---|---|---|---|
| 144431b300 | |||
| c0cb292350 | |||
| da8496adfd |
32
README.md
32
README.md
@@ -1,6 +1,6 @@
|
|||||||
# MeshSafe · 文件安全检测服务
|
# MeshSafe · 文件安全检测服务
|
||||||
|
|
||||||

|

|
||||||

|

|
||||||

|

|
||||||
|
|
||||||
@@ -200,6 +200,36 @@ MeshSafe 会**分析宏的实际行为**:
|
|||||||
|
|
||||||
## 版本历史
|
## 版本历史
|
||||||
|
|
||||||
|
### v0.2.3 (2026-10-29)
|
||||||
|
|
||||||
|
**Bug 修复:**
|
||||||
|
|
||||||
|
- **Policy 加载失败(每次启动都回退 YAML)** — `policy.py _load_db_or_yaml()` 用 `scalar_one_or_none()` 加载最新策略版本,但 `policy_versions` 表有多条历史版本 → SQLAlchemy 抛 `Multiple rows` → 永远回退 YAML 配置文件,导致不同重启间规则数波动。修复为 `db.query(PolicyVersion).order_by(created_at.desc()).first()`,直接取最新一条 ORM 对象
|
||||||
|
- **控制台任务历史被内部测试记录淹没** — `scan_tasks` 表累积 5,800+ 条内部对账/诊断脚本产生的测试数据(channel 包括 `test_center` / `v` / `verify` / `diag` / `debug` / `final` 等 19 个),dashboard 和 tasks_page 无过滤全部展示。修复:① DB 清理所有内部 channel 记录(保留 3,488 条生产数据);② console.py 加 `INTERNAL_TEST_CHANNELS` 黑名单常量,dashboard 6 个查询 + tasks_page 2 个查询全部 `.filter(ScanTask.channel.notin_(...))`
|
||||||
|
- **测试靶场显示 "未知" 层和 "—" 编号** — 控制台用 `iterdir()` 遍历目录收集样本,但目录里混入 4 个 YARA 调试时的临时文件(`L4-005_hotreload.txt` 等),index.json 没登记 → category 取不到显示 "未知"。修复:① 清理 4 个临时文件;② `test_center_page` 改为只从 index.json 收集样本(目录遍历替换为 index.json 遍历),以后不会再出这个问题
|
||||||
|
|
||||||
|
**DB 与运维增强:**
|
||||||
|
|
||||||
|
- **ScanTask 复合索引** — 新增 `ix_scan_tasks_channel_created ON (channel, created_at DESC)`,解决 dashboard/tasks_page 高频查询的全表扫描。models.py `ScanTask.__table_args__` 也同步更新,新建实例会自动带上
|
||||||
|
- **归档表 + 归档脚本** — 建 `scan_tasks_archive` 表(含 4 个索引);新增 `scripts/archive_old_tasks.py` 支持 4 项清理策略:
|
||||||
|
- scan_tasks **90 天前**:清 detail JSONB(每条 1-5 KB,省空间)
|
||||||
|
- scan_tasks **365 天前**:移到 scan_tasks_archive 归档表(CASCADE 删 ScanFinding)
|
||||||
|
- idempotency_records:清 **7 天前**(幂等缓存过期可以重扫)
|
||||||
|
- policy_versions:只留**最新 5 个**(历史版本无保留价值)
|
||||||
|
- 支持 `--dry-run`(默认,只报告不动数据)+ `--apply`(真正执行),分批 1,000 条处理避免长事务锁表
|
||||||
|
- **自动定时任务 Cron 服务** — docker-compose.yml 新增 `cron` 服务(共享 meshsafe:latest 镜像,零额外依赖,`restart: unless-stopped`),`scripts/cron_entrypoint.py` 纯 Python 调度器:
|
||||||
|
- **每天 03:00** 跑 archive_old_tasks.py --apply
|
||||||
|
- **每周一 03:10** VACUUM ANALYZE 5 张表(维护表统计信息,加速查询)
|
||||||
|
|
||||||
|
**数据增长实况:**
|
||||||
|
|
||||||
|
| 指标 | 值 |
|
||||||
|
|------|---|
|
||||||
|
| scan_tasks 当前 | 3,660 行 / 4.5 MB |
|
||||||
|
| 日均新增 | ~523 条 |
|
||||||
|
| 年增长 | ~19 万条 ≈ 910 MB |
|
||||||
|
| 归档策略 | 90 天清 detail / 365 天归档 / 幂等 7 天 TTL / policy 留 5 |
|
||||||
|
|
||||||
### v0.2.2 (2026-10-29)
|
### v0.2.2 (2026-10-29)
|
||||||
|
|
||||||
**引擎能力补全 — 测试靶场 168/168 100% 通过:**
|
**引擎能力补全 — 测试靶场 168/168 100% 通过:**
|
||||||
|
|||||||
@@ -960,15 +960,13 @@ async def test_page(request: Request, current=Depends(require_login)):
|
|||||||
index = json.loads(index_path.read_text(encoding="utf-8"))
|
index = json.loads(index_path.read_text(encoding="utf-8"))
|
||||||
meta_map = {s["filename"]: s for s in index.get("samples", [])}
|
meta_map = {s["filename"]: s for s in index.get("samples", [])}
|
||||||
|
|
||||||
# 收集所有样本文件
|
# ★ 只展示 index.json 里登记的样本(不遍历目录,避免临时/调试文件混入)
|
||||||
for fpath in sorted(samples_dir.iterdir()):
|
# 每个样本必须: index.json 有登记 + 磁盘上确实存在
|
||||||
if not fpath.is_file():
|
for s in index.get("samples", []):
|
||||||
continue
|
fname = s["filename"]
|
||||||
if fpath.name == "index.json" or fpath.name.startswith("_"):
|
fpath = samples_dir / fname
|
||||||
continue
|
if fpath.is_file():
|
||||||
if fpath.suffix == ".json" and fpath.name.startswith("_scan_results"):
|
samples_list.append(fpath)
|
||||||
continue
|
|
||||||
samples_list.append(fpath)
|
|
||||||
|
|
||||||
results: list[dict] = []
|
results: list[dict] = []
|
||||||
pass_cnt = fail_cnt = 0
|
pass_cnt = fail_cnt = 0
|
||||||
|
|||||||
@@ -133,6 +133,29 @@ services:
|
|||||||
start_period: 15s
|
start_period: 15s
|
||||||
retries: 3
|
retries: 3
|
||||||
|
|
||||||
|
# ---------- Cron (定时任务: DB 归档 + VACUUM) ----------
|
||||||
|
# 纯 Python 实现, 零额外依赖, 和 api 共享 meshsafe:latest 镜像
|
||||||
|
# 任务:
|
||||||
|
# 每天 03:00 archive_old_tasks.py --apply (清 90 天 detail / 365 天归档 / 幂等 TTL)
|
||||||
|
# 每周一 03:10 VACUUM ANALYZE (维护表统计信息)
|
||||||
|
cron:
|
||||||
|
image: meshsafe:latest
|
||||||
|
container_name: meshsafe-cron
|
||||||
|
restart: unless-stopped
|
||||||
|
networks: [meshsafe_net]
|
||||||
|
depends_on:
|
||||||
|
postgres:
|
||||||
|
condition: service_healthy
|
||||||
|
environment:
|
||||||
|
- TZ=Asia/Shanghai
|
||||||
|
- MESHSAFE_ENV=prod
|
||||||
|
- MESHSAFE_DB_URL=postgresql+psycopg2://meshsafe:${MESHSAFE_DB_PASSWORD:-meshsafe}@postgres:5432/meshsafe
|
||||||
|
- MESHSAFE_MASTER_KEY=${MESHSAFE_MASTER_KEY:?MASTER_KEY_REQUIRED}
|
||||||
|
volumes:
|
||||||
|
- ./data/logs:/var/log/meshsafe # 归档日志共享目录
|
||||||
|
- ./scripts:/app/scripts:ro # archive_old_tasks.py + cron_entrypoint.py
|
||||||
|
entrypoint: ["python3", "/app/scripts/cron_entrypoint.py"]
|
||||||
|
|
||||||
# ---------- Nginx(反向代理,内网 HTTP 明文)----------
|
# ---------- Nginx(反向代理,内网 HTTP 明文)----------
|
||||||
# 外部端口 44380 → 内部 80 → api:8443
|
# 外部端口 44380 → 内部 80 → api:8443
|
||||||
# 无 TLS,外层防护靠光闸
|
# 无 TLS,外层防护靠光闸
|
||||||
|
|||||||
121
scripts/cron_entrypoint.py
Normal file
121
scripts/cron_entrypoint.py
Normal file
@@ -0,0 +1,121 @@
|
|||||||
|
"""MeshSafe 定时任务调度器 — 零依赖,Docker 容器前台常驻。
|
||||||
|
|
||||||
|
和 api 共享同一个镜像,容器 entrypoint 换成本脚本即可。
|
||||||
|
用法: docker run meshsafe:latest python3 scripts/cron_entrypoint.py
|
||||||
|
|
||||||
|
支持的任务:
|
||||||
|
每天 03:00 归档 + 清理 DB (archive_old_tasks.py --apply)
|
||||||
|
每周一 03:10 VACUUM ANALYZE (维护表统计信息)
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import logging
|
||||||
|
import os
|
||||||
|
import subprocess
|
||||||
|
import sys
|
||||||
|
import time
|
||||||
|
from datetime import datetime, timedelta
|
||||||
|
|
||||||
|
sys.path.insert(0, "/app")
|
||||||
|
|
||||||
|
logging.basicConfig(
|
||||||
|
level=logging.INFO,
|
||||||
|
format="%(asctime)s [%(levelname)s] %(message)s",
|
||||||
|
)
|
||||||
|
log = logging.getLogger("scheduler")
|
||||||
|
|
||||||
|
|
||||||
|
def run(cmd: list[str], label: str) -> None:
|
||||||
|
"""跑一个命令,捕获输出并记日志。"""
|
||||||
|
log.info(f"[{label}] 开始执行: {' '.join(cmd)}")
|
||||||
|
try:
|
||||||
|
proc = subprocess.run(cmd, capture_output=True, text=True, timeout=1800)
|
||||||
|
if proc.stdout:
|
||||||
|
for line in proc.stdout.strip().splitlines():
|
||||||
|
log.info(f"[{label}] {line}")
|
||||||
|
if proc.stderr:
|
||||||
|
for line in proc.stderr.strip().splitlines():
|
||||||
|
log.warning(f"[{label}] {line}")
|
||||||
|
if proc.returncode == 0:
|
||||||
|
log.info(f"[{label}] ✅ 完成")
|
||||||
|
else:
|
||||||
|
log.error(f"[{label}] ❌ 返回码 {proc.returncode}")
|
||||||
|
except subprocess.TimeoutExpired:
|
||||||
|
log.error(f"[{label}] ❌ 超时 (30min)")
|
||||||
|
except Exception as e:
|
||||||
|
log.error(f"[{label}] ❌ 异常: {e}")
|
||||||
|
|
||||||
|
|
||||||
|
def run_vacuum() -> None:
|
||||||
|
"""VACUUM ANALYZE — 维护表统计信息。"""
|
||||||
|
try:
|
||||||
|
from app.db import engine
|
||||||
|
from sqlalchemy import text
|
||||||
|
|
||||||
|
tables = [
|
||||||
|
"scan_tasks", "scan_tasks_archive",
|
||||||
|
"scan_findings", "idempotency_records",
|
||||||
|
"policy_versions", "audit_logs",
|
||||||
|
]
|
||||||
|
with engine.begin() as conn:
|
||||||
|
for tbl in tables:
|
||||||
|
log.info(f"[VACUUM] {tbl} ...")
|
||||||
|
conn.execute(text(f"VACUUM ANALYZE {tbl}"))
|
||||||
|
log.info("[VACUUM] ✅ 全部完成")
|
||||||
|
except Exception as e:
|
||||||
|
log.error(f"[VACUUM] ❌ 异常: {e}")
|
||||||
|
|
||||||
|
|
||||||
|
def seconds_until(target_hour: int, target_minute: int, weekday: int | None = None) -> float:
|
||||||
|
"""计算到下次目标时间的秒数。
|
||||||
|
|
||||||
|
weekday: 0=周一...6=周日, None=每天
|
||||||
|
"""
|
||||||
|
now = datetime.now()
|
||||||
|
candidate = now.replace(hour=target_hour, minute=target_minute, second=0, microsecond=0)
|
||||||
|
|
||||||
|
if weekday is not None:
|
||||||
|
# 找到本周目标日
|
||||||
|
days_ahead = weekday - now.weekday()
|
||||||
|
if days_ahead < 0 or (days_ahead == 0 and now >= candidate):
|
||||||
|
days_ahead += 7
|
||||||
|
candidate += timedelta(days=days_ahead)
|
||||||
|
else:
|
||||||
|
if candidate <= now:
|
||||||
|
candidate += timedelta(days=1)
|
||||||
|
|
||||||
|
return (candidate - now).total_seconds()
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> None:
|
||||||
|
log.info("=" * 60)
|
||||||
|
log.info("MeshSafe 定时任务调度器启动")
|
||||||
|
log.info(f"PID={os.getpid()} TZ=Asia/Shanghai")
|
||||||
|
log.info("任务:")
|
||||||
|
log.info(" - 每天 03:00 archive_old_tasks.py --apply")
|
||||||
|
log.info(" - 每周一 03:10 VACUUM ANALYZE")
|
||||||
|
log.info("=" * 60)
|
||||||
|
|
||||||
|
while True:
|
||||||
|
# 计算两个任务下次执行时间,取更早的那个
|
||||||
|
secs_archive = seconds_until(3, 0, weekday=None) # 每天 03:00
|
||||||
|
secs_vacuum = seconds_until(3, 10, weekday=0) # 周一 03:10
|
||||||
|
|
||||||
|
if secs_archive <= secs_vacuum:
|
||||||
|
log.info(f"下一个任务: 归档 → {secs_archive/3600:.1f}h 后")
|
||||||
|
time.sleep(secs_archive)
|
||||||
|
run(
|
||||||
|
[sys.executable, "/app/scripts/archive_old_tasks.py", "--apply"],
|
||||||
|
"ARCHIVE",
|
||||||
|
)
|
||||||
|
# 跑完后清下一次时间 (避免在 03:00 这一刻两个任务重叠)
|
||||||
|
continue
|
||||||
|
else:
|
||||||
|
log.info(f"下一个任务: VACUUM → {secs_vacuum/3600:.1f}h 后")
|
||||||
|
time.sleep(secs_vacuum)
|
||||||
|
run_vacuum()
|
||||||
|
continue
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
@@ -1 +0,0 @@
|
|||||||
meshsafe-hot-reload-test-string present here
|
|
||||||
@@ -1 +0,0 @@
|
|||||||
trojan-metadata-test payload here
|
|
||||||
@@ -1 +0,0 @@
|
|||||||
powershell -enc ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwx+/==
|
|
||||||
@@ -1 +0,0 @@
|
|||||||
size_condition_test_payload small file
|
|
||||||
Reference in New Issue
Block a user