13 Commits

Author SHA1 Message Date
21aa657ff4 chore: .gitignore 忽略 MeshSafe.zip 打包快照
10-30 的项目打包件,内含 .env 副本;.gitignore 的 .env 规则挡不住打进 zip 的副本,
整包一旦误提交即泄密。zip 本身未入过库,无需 rm --cached。
2026-11-02 20:28:57 +08:00
c581bfc4bb docs: README v0.2.4 四轮误报标定记录 + YARA 范围文档移入 docs/
第四轮记录:逐规则独立贡献重放(60 条命中/25 条有独立贡献/前 10 条占 92.6%)、
批量按族退役零 verdict 变化、复核地板重构与复测表;另记 meshsafe-cron 自查结论与
ClamAV 签名库新鲜度机制(库龄不影响处置结果是剩余缺口)。遗留项明确 35 条零贡献规则
暂不退役的原因。YARA 检测范围文档从仓库根目录挪进 docs/。
2026-11-02 20:27:39 +08:00
f1a176aae0 fix: 裁决口径重构 — rule_overrides 改处置地板,YARA 权重分档与阈值解耦
三件事,都围绕"分数只反映证据,不放行由地板保证":

- rule_overrides 的 manual_review 旧实现是 score=max(score,65) 并跳过该规则自身权重:
  注进去的 65 不是证据,再叠任意 ≥5 分噪声就越过 block_score=70 变拒绝,同时显示分数
  与真实证据脱钩。改为步骤 9「复核地板」:权重照常累加,裁决后处置低于复核才抬到
  manual_review,已判 reject 不降回。3661 条历史扫描重放 0 收紧 0 放行;把 review_score
  下界抬到 36/40/50 或把 DEE 权重降到 10,无地板时干净 exe 静默放行,有地板恒为人工复核。
- DangerousExecutableExtension 进 rule_overrides(manual_review),落实光闸口径「EXE/DLL/
  脚本不自动放行」,不再依赖 weight=30 恰好等于 review_score 下界这种数值巧合。
- YARA 权重分档 85/65/45/25 → 85/60/30/10,只有 critical 一档能独立越过 70;
  档位与 rules/*.yar 的 meta.risk_level 一一对应,控制台注释与 policy.html 同步。
- 词表:RemoteDownloadUnix/PowerShell 移出零容忍、改判"工具与 URL/IP 同框",
  新增 RemoteDownloadExec(40)/PayloadDropToDisk(45) 带内网源站白名单。裸 cmdlet 名会把
  nvidiaInspector.exe 的 add_DownloadFileCompleted 白送 30 分,把 45 分复核顶成 75 分拒绝。
- 风险等级文案:裁决已是 malicious/error 时仍输出"打开风险低"是自相矛盾的对外文案,
  强制收敛到 critical。
- 靶场 L5-005 expected_verdict reject→review:旧 85 分靠载体45+时间戳20+未签名10+架构10
  堆叠,属权重堆叠误报;真实注入器由 C1_PE_ProcessInjection(85) 拦截。
2026-11-02 20:27:30 +08:00
b6a004bdb7 fix: 六引擎误报根因修复 — 解包递归、YARA 恒真规则、签名分级、real_type 溢出
检测范围与证据链修复,不改裁决口径(口径改动在下一条提交)。

- extractor:zip/tar 成员在超限分支外漏加回扫描队列,解包后内容永远不参与检测;
  zip 目录条目被当成内嵌对象;tar 超限成员先落盘不删。递归真正生效后深扫耗时可测。
- clamav:_HIGH/_MEDIUM/_UNWANTED 关键词是小写,而签名名是 "Win.Trojan..." 大小写混排,
  分级判定恒不命中全走默认档;改统一大写匹配。超过 clamd MaxFileSize 的裸奔区间
  (1G~1.5G) 新增 ClamAVSkippedSizeLimit(30) — 按"不可扫描不放行"进复核。
- yara:pe.yar C1 系列改用导入表判定,不再靠裸子串命中字符串表;
  linux.yar C7 多条规则条件恒真(裸 nc/fetch 之类)已收紧;C8/C3 相应收窄。
- structure:DangerousExecutableExtension 是载体信号而非恶意行为,critical 权重 45→35,
  不再作为把中低置信度信号一起顶过 block_score 的底座。
- scanner/documents:real_type 拼进 libmagic 对 MSI/OLE 的元数据串实测 334+ 字符,
  超列宽导致整条 scan_tasks INSERT 连 findings 一起丢(被拒文件不留痕)→
  列宽 256→512,migrate.sql 补 ALTER。
2026-11-02 20:26:56 +08:00
64b682319e fix: meshsafe-cron 定时任务自引入以来从未成功执行 — 三处缺陷
复现而非等日志:容器内直接跑 archive_old_tasks.py 即崩。

1. app/config.py 的 admin_password 是必填字段,cron 环境没有它 →
   get_settings() 在 import 阶段抛 ValidationError,归档与 VACUUM 全崩。
   证据:policy_versions 留 10 行(策略写的留 5 删 5)、scan_tasks_archive 0 行,
   而容器 Up 且 RestartCount=0,没有任何存活信号暴露。给占位 env,不把真实口令复制进 cron。
2. VACUUM 不能在事务块里跑(ActiveSqlTransaction),原代码用了 engine.begin(),
   改用 AUTOCOMMIT 连接;实测 6 张表 0.25s 完成。
3. ValidationError 把整个 settings dict 打进 stderr,被 run() 原样写进 docker 日志,
   含 DB 连接串与 master key 尾部 → 加 _redact() 落盘前抹掉,验证输出中不可见。

cron 不监听 8443,镜像层 HEALTHCHECK 恒为 unhealthy,调度器真挂掉时该信号失效,故关闭。
2026-11-02 20:26:08 +08:00
1079ca16de fix: 空文件 verdict error→pass
console.py upload_multiple_files L1278-1291:
  空文件被硬编码成 verdict='error' ok=False
  根因: 把'没内容'当成'检测异常'
  修复: 空文件是合法文件, 改成 verdict='pass' ok=True reason='空文件,无风险'
2026-10-29 17:33:19 +08:00
bc486df250 fix: L0-004/L1-008 expected_verdict pass→suspicious
L0-004_pdf_normal.pdf (14字节):
  只有 %PDF-1.4 头和 %%EOF 尾, 缺 xref/trailer/catalog
  引擎报 MalformedStructure/PdfParseFailed 是对的
  expected_rules 本来就写了这两个, expected_verdict 写成 pass 矛盾
  → pass → suspicious

L1-008_L0-005_pe_clean.exe (926字节):
  e_lfanew=0x80 但 PE\\0\\0 实际在 0x7e (差2字节)
  pefile 报 Invalid NT Headers signature, PeParseFailed 是对的
  expected_verdict 写成 pass 不对
  → pass → suspicious

两个样本都是'测试畸形检测'用的负样本, 不是'干净文件对照'
引擎行为正确, 是 expected_verdict 本身写错了
2026-10-29 17:31:53 +08:00
9834930299 fix: PDF/Office/压缩包关键词误报 — 两层修复
【regex 层收紧】policy.yaml keyword_rules

1) ReverseShellNetcat: \\bnc\\b → \\bnc\\s+-[elpvs]\\w*
   nc 后必须跟参数 (-e/-l/-p/-v/-s), 单独 nc 在二进制里必然误报
   ncat 同理: \\bncat\\b → \\bncat\\s+-\\w+
   保留 \\bnetcat\\b (8 字母, 碰撞概率极低)

2) CredentialStealer: 全部关键词加 \\b 词边界
   wce 单独加后续约束: wce(\\.exe\\b|\\s|\\/)
   要求 wce 后跟 .exe / 空格 / / (二进制随机字节里 WCE 不再触发)

【引擎层架构修复】keywords.py

新增 BINARY_SKIP_EXTS (48 个扩展名):
  - PDF / Office (.pdf/.docx/.xlsx/.pptx/...)
  - 压缩包 (.rar/.zip/.7z/.tar/.gz/...)
  - 图片 (.png/.jpg/.gif/.svg/...)
  - 视频 (.mp4/.avi/.mkv/...)
  - 音频 (.mp3/.wav/.flac/...)

KeywordEngine.scan() 开头检查:
  如果 declared_ext / path ext 在 BINARY_SKIP_EXTS → 直接返回空

根因:
  PDF/DOCX/ZIP 等文件是 DEFLATE 压缩流 (伪随机字节)
  短关键词 (nc=2w/wce=3w) 在压缩流里必然出现
  误报导致: 云启智能体平台生成效果.pdf 被拦截
    ReverseShellNetcat: 'g%ew;V6Z?)Nc{>72z}*&96'
    CredentialStealer: '2>>~:pa}w*WCE|2MVB85t/$'

验证:
  ✅ 新 regex: 真实 nc -e / wce.exe 命中, PDF 压缩流字节不命中
  ✅ BINARY_SKIP_EXTS: .pdf/.docx/.zip 跳过, .sh/.ps1/.txt 扫描
  ✅ scan() 方法已含跳过逻辑
2026-10-29 17:28:23 +08:00
144431b300 docs: README 版本号 v0.2.2→v0.2.3 + 版本历史
v0.2.3 覆盖 (c0cb292~5d3c895):
  - Policy 加载 bug (scalar_one_or_none→first) → 规则数波动根治
  - 控制台隐藏 19 个内部测试 channel + DB 清 4624 条
  - 测试靶场 index.json 来源加固 + 清 4 个临时调试文件
  - ScanTask 复合索引 ix_scan_tasks_channel_created
  - scan_tasks_archive 归档表 + archive_old_tasks.py 脚本
  - docker-compose cron 服务 (每天 03:00 归档 + 每周一 VACUUM)
2026-10-29 17:15:23 +08:00
c0cb292350 fix: 测试靶场只展示 index.json 登记的样本 + 清 4 个临时调试文件
问题:
  控制台测试靶场页面样本数: 目录 172 个 vs index.json 168 条
  差了 4 个是 YARA 调试时创建的临时文件 (L4-005_hotreload.txt 等)
  这些临时文件没进 index.json → meta_map 查不到 → category='未知' case_id='—'

修复:
  1) console.py test_center_page:
     samples_list 从遍历目录 (iterdir) 改成只从 index.json 来
     每个样本必须 index.json 有登记 + 磁盘上确实存在
     以后目录里即使有临时文件也不会混入靶场列表
  2) 清理 4 个重复临时调试文件:
     L4-005_hotreload.txt      → index.json 正式: L4-005_005_sample.txt
     L4-007_meta_rule.txt       → index.json 正式: L4-007_meta_rule_match.txt
     L4-010_regex_pattern.txt   → index.json 正式: L4-010_010_sample.txt
     L4-012_size_condition.txt  → index.json 正式: L4-012_size_condition.bin

控制台页面现在: 168 条, 全部有 case_id 和 category, 无 '未知' 和 '—'
2026-10-29 17:11:51 +08:00
da8496adfd infra: 自动定时任务 — cron 服务
新增:
  - scripts/cron_entrypoint.py: 纯 Python 调度器, 零依赖
    任务1: 每天 03:00 跑 archive_old_tasks.py --apply
    任务2: 每周一 03:10 VACUUM ANALYZE (维护表统计信息)
  - docker-compose.yml: 加 cron 服务 (共享 meshsafe:latest 镜像)

归档策略 (archive_old_tasks.py):
  - scan_tasks  90天前: 清 detail JSONB (省空间)
  - scan_tasks  365天前: 移到 scan_tasks_archive
  - idempotency_records:  清 7 天前 (幂等 TTL)
  - policy_versions:      只留最新 5 个

启动: docker compose up -d cron
日志: docker compose logs -f cron
docker compose up 会自动拉起 (restart: unless-stopped)
2026-10-29 17:05:41 +08:00
5d3c895542 fix: policy 加载 bug + DB 索引 + 归档脚本
【Bug 修复】policy.py _load_db_or_yaml
  根因: select(PolicyVersion).order_by(desc).scalar_one_or_none()
        表有多条历史版本 → Multiple rows 异常 → 每次启动都回退 YAML
  修复: 改成 db.query(PolicyVersion).order_by(desc).first()
        用 ORM Query API 取最新一条, 返回真正的 PolicyVersion 对象
  验证: 容器启动后日志中不再出现 'DB 加载策略失败(回退 YAML)'

【索引】models.py ScanTask 加复合索引
  ix_scan_tasks_channel_created ON (channel, created_at DESC)
  解决 dashboard/tasks_page 高频查询的全表扫描问题
  (已在运行中 DB 用 CREATE INDEX CONCURRENTLY 补上)

【归档】scripts/archive_old_tasks.py
  - scan_tasks 90天前 清 detail JSONB (省空间)
  - scan_tasks 365天前 移到 scan_tasks_archive 归档表
  - idempotency_records 7天 TTL 清过期幂等缓存
  - policy_versions 只留最新 5 个
  - 支持 --dry-run (默认) + --apply (真正执行)
  - 分批 1000 条处理, 避免长事务锁表
  - 归档表 scan_tasks_archive 已建 (含 4 个索引)

数据增长: 日均 523 条 → 年 ~19万条 ≈ 910 MB
当前 3660 条 ≈ 4.5 MB, 增长可控, 脚本可每周/每月 cron 跑一次
2026-10-29 17:02:37 +08:00
1222238bf3 fix: 控制台隐藏内部测试 channel + DB 清理
问题:
  任务历史 / Dashboard 全是对账脚本/诊断脚本灌进 DB 的内部测试记录
  (test_center/v/verify/diag/debug/final/... 共 4624 条)
  用户误以为测试靶场页面在自动触发批量扫描

根因:
  scanner.scan_file() 内部无条件 persist_to_db()
  控制台页面查询完全没有 channel 过滤 — 直接 SELECT * FROM scan_tasks

修复:
  1) 代码层: INTERNAL_TEST_CHANNELS frozenset (19 个内部 channel 黑名单)
     - dashboard 6 个查询全加 .filter(ScanTask.channel.notin_(...))
     - tasks_page 2 个查询全加 .filter(...)
  2) 数据层: 删除内部测试 channel 记录
     - ScanTask 4624 条 (test_center 1929 / v 688 / verify 501 / ...)
     - ScanFinding 9101 条 (外键先删)
     - 保留生产: console_upload 3465 + duandao 16 + default 7

控制台现在只展示真实生产数据, 后续新增内部脚本 channel 只需在
INTERNAL_TEST_CHANNELS 补充一行即可自动隐藏
2026-10-29 16:46:52 +08:00
29 changed files with 1480 additions and 419 deletions

3
.gitignore vendored
View File

@@ -45,6 +45,9 @@ Thumbs.db
benchmark_report*.json benchmark_report*.json
out.json out.json
# 打包快照(含 .env 副本,不入库)
/MeshSafe.zip
# 截图 # 截图
/.screenshots/ /.screenshots/

141
README.md
View File

@@ -1,6 +1,6 @@
# MeshSafe · 文件安全检测服务 # MeshSafe · 文件安全检测服务
![Version](https://img.shields.io/badge/version-v0.2.2-blue) ![Version](https://img.shields.io/badge/version-v0.2.4-blue)
![Python](https://img.shields.io/badge/python-3.12%2B-green) ![Python](https://img.shields.io/badge/python-3.12%2B-green)
![License](https://img.shields.io/badge/license-proprietary-orange) ![License](https://img.shields.io/badge/license-proprietary-orange)
@@ -171,7 +171,7 @@ MeshSafe 会**分析宏的实际行为**:
| 可执行程序结构分析 | 异常特征、签名目录、导入表 | | 可执行程序结构分析 | 异常特征、签名目录、导入表 |
| 外部链接与二维码提取 | 文档和图片 | | 外部链接与二维码提取 | 文档和图片 |
| 恶意链接库查询 | 分级处置 | | 恶意链接库查询 | 分级处置 |
| 零容忍规则集(15 条) | 打开即执行载体全覆盖 | | 零容忍规则集(25 条) | 打开即执行载体全覆盖 + 高危行为词表 |
| 宏行为例外机制 | 合法公文模板受控放行 | | 宏行为例外机制 | 合法公文模板受控放行 |
| 引擎故障安全降级 | 任何检测引擎不可用时整体拒绝(安全优先) | | 引擎故障安全降级 | 任何检测引擎不可用时整体拒绝(安全优先) |
| 同步 + 异步双模式 | 小文件同步返回,大文件异步轮询 | | 同步 + 异步双模式 | 小文件同步返回,大文件异步轮询 |
@@ -200,6 +200,143 @@ MeshSafe 会**分析宏的实际行为**:
## 版本历史 ## 版本历史
### v0.2.4 (2026-11-02)
**误报清零专项(现场反馈的三类误判):**
真实 PPT(内嵌图表对象)、电脑上正常无病毒的 EXE/DLL、写有安装命令的 skill.md 文档,在旧策略下分别被判「中危可疑」「严重拒绝」「零容忍拒绝」。根因不是某一个规则,而是**弱信号可以靠权重堆叠越过 block_score=70**,以及**关键词/YARA 用裸子串匹配**。本轮把"载体信号"和"行为信号"彻底分档,行为如下:
- **文档内嵌对象分级** — `documents.py _scan_ooxml()` 不再把 zip 目录条目当成内嵌对象;对 `ppt/embeddings/*` 逐条做**魔数嗅探**:MZ/ELF/Mach-O 或 `.exe/.dll/.js/...` → `EmbeddedExecutable`(严重 75,零容忍);Office 部件(`.bin` 里装的是 OLE/子文档)→ `EmbeddedObject`(低危 5);其余 → 中危 20。正常 PPT 的图表/嵌入对象不再产生 20 分可疑
- **YARA 判定改用导入表邻接窗口** — `pe.yar` 全面重写:API 名必须在**同一张导入名表内相邻 512B**才算命中(`$b in (@a[i]-512..@a[i]+512)`),导出表、i18n 资源、语言包里的字符串堆不再触发;RAT 规则要求真实 C2 痕迹(IP:端口 / .exe URL / 网关面板 php),打包器 UPX 降为 low
- **Linux/脚本规则去恒真** — `linux.yar` 修掉 `$a and $b` 这类"两个独立字符串同时存在即命中"的恒真写法;`ettercap/responder/nmap` 加词边界并需与攻击参数成组;`LD_PRELOAD` 只在指向可写临时目录或出现 `ld.so.preload` 时命中;`C7_Misc_Ruby_Lua_ReverseShell` 需要 `exec` 与 `TCPSocket/loadstring` 同窗口
- **PE 信号分档(不再单独定罪)** — `binary.py`:导入 API 分三档,单个注入级 API 只记 5 分(低危),≥2 个才记 20 分;`WinExec/ShellExecuteEx/CreateService/InternetOpenUrl` 从"高危"降到能力档;`IsDebuggerPresent/GetTickCount/FindWindow` 等反调试探测记 0 分;调试信息、异常时间戳 10 → 0;未签名 20 → 10
- **YARA 严重度取 `meta.risk_level`** — 旧版只读 tag 和规则名,规则文件里的 `risk_level` 被完全忽略;现在 severity/weight 一律先取 meta,再按 `policy.yaml yara_rule_weights` 覆盖,档位统一为 **critical=85 / high=60 / medium=30 / low=10**,44 条规则全部按 meta 重新归位
- **可执行扩展名降为"载体分"** — `DangerousExecutableExtension` 45 → 35:单独命中仍进人工复核(≥30),但不再充当把后续中低置信度信号顶过 70 分的底座
- **关键词引擎不再无条件删换行** — v0.2.2 的"整篇文本去换行"是 `skill.md` 误报的直接原因(上一行行尾和下一行行首黏成一句话)。改为默认按行匹配,仅显式 `cross_line: true` 的规则额外跑一遍**换行压成空格**的文本(保留分隔符,兼容续行绕过,不再产生黏连)
- **下载类词表重写 + 移出零容忍** — 新增 `RemoteDownloadExec`(下载**即执行**:`curl|sh`、`wget -O … && chmod`、`iwr|iex`、`certutil -urlcache … .exe`、`mkfifo+nc`,40 分)与 `PayloadDropToDisk`(下载落地成 `.exe/.ps1/.so/…`,45 分);`RemoteDownloadUnix` / `RemoteDownloadPowerShell` 从"零容忍直接拒"降为 12 分信息项。裸 `curl https://…` 出现在 README/运维文档里不再拦截,内网源站另有白名单放行
- **反弹 Shell 词表只认执行原语** — `ReverseShellNetcat` 要求 `nc|ncat|netcat` 带 `-e/-c` **且跟 shell 路径**(`/bin/`、`*sh`、`cmd.exe`),或 `/dev/tcp/`、`bash -i >&`;裸 `nc`、`nc -l`、`nc -z`(速查文档、二进制字符串表里必然出现)不再命中;英文公文里的 "NC -E" 之类缩写组合也不再误触
- **`rules/ransom.yar` / `webshell.yar` 收尾** — 卷影删除不再接受裸 `vss` / `ShadowCopy` 子串与加密 API 同框(正常备份组件的必然组合),要求 `vssadmin … delete … shadows`;混淆 PHP 马要求存在 `<?php` 或超全局变量上下文
**验证结果(离线,ClamAV/DB 不在本机):**
| 指标 | 修复前 | 修复后 |
|---|---|---|
| 32 个干净样本(系统 DLL/EXE + Git/node 工具 + 真实 PPTX)裁决 | 18 malicious/reject、14 suspicious、0 pass | **0 reject**、31 manual_review(仅因"是可执行文件")、1 pass(PPTX 全程 0 命中) |
| 2162 个真实 Windows 二进制 YARA 误报 | **407 条命中 / 14 条规则**(`C1_PE_Downloader_Loader` 单独 290 条,ActionCenter.dll 一类正常组件全中) | **4 条**,全部为 `C1_PE_ProcessInjection` 命中调试/注入基础设施组件(dbgeng.dll、ttdinject.exe、PayloadRestrictions.dll、VsGraphics*),属真实能力特征 |
| 靶场 91 个恶意样本 YARA 覆盖 | 全部命中 | **全部命中,无一条漏检** |
| 168 条样本逐条对账(同一离线管线,HEAD vs 本轮) | 124 Fail | 124 Fail,其中 2 条由 Fail 转 Pass,仅 `L5-005_injection_imports.exe` 从 reject 变为 manual_review(928B 占位假 PE,无导入表,旧结论完全靠权重堆叠),expected_verdict 已同步为 review |
**处置口径(本轮已确认,不再作为开放项):**
- **干净 EXE/DLL 落人工复核。** 可执行文件本身是载体,单独命中 `DangerousExecutableExtension`(35 分)落在 `review_score` 区间是设计目标,不改自动放行;"严重/拒绝"只留给真实恶意特征。
- **调试/注入基础设施组件的命中保留,不做白名单。** dbgeng.dll、ttdinject.exe、PayloadRestrictions.dll、VsGraphics* 这 4 个 Windows 组件确实具备进程注入能力,`C1_PE_ProcessInjection` 命中它们属真阳性,**维持现状**(critical=85,单条即判恶意拒绝)。**不使用 `yara_hash_whitelist` 豁免,也不在规则里判断 Microsoft 版本信息**——版本信息可伪造,任何基于它的放行都会开出一条绕过路径;这类文件本就不该走通用摆渡通道传输,如业务确有需要,由管理员单独确认后处置。
**容器内实测(v0.2.4 部署后,ClamAV + PostgreSQL 全部可达):**
| 指标 | 结果 |
|---|---|
| 6 个引擎可用性 | `engine_errors` **0 条**(168 个样本全程无引擎降级) |
| 三个原始误报复测 | `北京简介 (1).pptx` → clean/pass、0 命中(现已真正展开 61 个 OOXML 部件仍 0 命中);`SKILL.md` → clean/pass、0 命中;干净 PE(dxdiag.exe 50 分、shell32.dll 65 分、cmd.exe 45 分)→ manual_review,均低于 `block_score=70` |
| 靶场 168 条(控制台对账口径) | **168/168 通过**(初测 4 条未过,见下) |
**第二轮修补的 4 条检出缺口(容器内才暴露):**
- **zip/OOXML 成员从未参与扫描(HEAD 既有缺陷)** — `extractor.py` 的成员写盘循环用 `while True: … if not chunk: break` 正常结束,`else:` 分支因此永不执行,解出来的成员一个都没进扫描队列:压缩包和 docx/xlsx/pptx 正文里的高危串全部放行。改为显式 `overflow` 标志后 `scanned_objects` 才真正大于 1;tar 分支同修(原先超限成员既不丢弃也不报,还把截断文件塞进队列)。这条修复让 `L3-015`(docx 正文的 `Invoke-WebRequest`)、`L3-023`(zip 内的 `mimikatz`)从 0 命中变检出。
- **不可解析即放行** — 畸形 OLE 触发 olefile 内部异常后被 `documents.py` 的宽 `except` 兜成 `OleToolsUnavailable`,权重 0 → clean/pass。改为 medium/30,宏解析不了的文档一律进人工复核。
- **换行拆分绕过** — `L3-009` 把 `Invoke-WebRequest` 拆成两行,`-OutFile evil.exe` 因跨行漏检。`RemoteDownloadExec`/`PayloadDropToDisk` 恢复 `cross_line`,并让 cmdlet 词容忍 `invoke-web_request`/换行拆分形态;仍要求"下载工具 + 执行或落地动作"同框,因此 `SKILL.md`、README 类文档保持 0 命中。
- **裸下载命令权重回正** — 为压误报把 `RemoteDownloadPowerShell`/`RemoteDownloadUnix` 降到 12 分是矫枉过正:靶场里没有任何"期望放行"的样本含下载命令,误报源头是 `curl|wget -o 任意值` 这种松散写法而非权重。现改回 30/medium 并删掉裸 `-o` 分支,`L3-018`/`L3-024`/`L3-025` 随之回到 review。
**第三轮:生产日志暴露的 4 个问题(现场上传 `go1.25.0.windows-amd64.msi` / `nvidiaInspector.exe`):**
- **被拒绝的文件在库里查不到记录(留痕丢失)** — `scan_tasks.real_type` 是 `VARCHAR(256)`,而这个 MSI 的 libmagic 描述串有 **632 字符**(CFB 会把 Title/Subject/Creating Application/Number of Pages 全部拼进描述)。Postgres 对超长值不是截断存储而是**整条 INSERT 报错回滚**,因此这次扫描的记录连同 3 条 `scan_findings` 一起丢失——一个 `malicious/reject` 的文件没有任何痕迹。修复三处:模型列宽 256 → 512(`app/models.py`)、线上 `ALTER TABLE scan_tasks ALTER COLUMN real_type TYPE VARCHAR(512)`(已入 `scripts/migrate.sql` 并在 meshsafe-postgres 执行)、落库前按列宽截断 `filename/declared_ext/real_type/extracted_path`(`scanner.py` 新增 `_clip()`)。外部可控长度(libmagic 串、归档成员路径、用户文件名)从此不可能再撑爆 INSERT:实测 600 字符文件名 + 900 字符 reason + 1600 字符成员路径 → 记录与 findings 全部入库(512/512/1024)。
- **`verdict=malicious` 却给出 `open_risk.level=low`** — 同一条日志里的自相矛盾。`compute_open_risk()` 只看命中因子,不看裁决。补上裁决下限:`malicious/error/reject` 一律至少 critical,并把 `.msi/.jar/.ps1/.lnk/.hta/…` 这类真实载体扩展名纳入 `contains_executable`。
- **`C8_Ransomware_Note_Filename` 把 Go 官方安装包判成勒索** — 单文件实测命中量:`$wallet` **76,526 次**(`1[A-Za-z0-9]{25,34}` 无边界,任何 GUID/哈希子串都算比特币地址)、`$bitcoin` **44 次**(压缩流里的 `BTC`/`Xmr` 随机字节)、`$n4` 命中安装包内字面量 `README.txt`。三者拼出 70 分 → 官方 MSI 被拒。仓库自己的 `tests/samples/index.json` 同样中招(`ransom` 是 `Ransomware` 的子串)。重写为:文件名只保留家族专有名 + 扩展名(去掉裸 `README.txt`/`_readme.txt`,`\b_readme` 不再命中 `normal_readme.txt`);赎金语言必须是完整词组(`bitcoin address`、`send the money`、`\bransom\b note`);地址按 base58/bech32/ Monero 真实字符集与长度并用 `\b` 锚定整个 token;条件要求**文件名 + (加密 API + 赎金语言)或(赎金语言 + 钱包地址)**。修复后 MSI 该规则 0 命中,`index.json`/`_scan_results.json` 0 命中,真勒索样本 `ransom_dropper.exe.txt` 仍命中(100 分拒绝)。
- **`PeParseFailed` 打在非 PE 容器上** — `BINARY_EXTS` 里的 `.msi/.msp/.jar/.apk/.war/.so/.elf` 本来就不是 PE,`pefile` 解析必然抛错,于是每个合法安装包/共享库白送 10 分。新增 `_is_native_container()`:内容魔数与声明格式一致(MSI=CFB、JAR/APK=ZIP、SO/ELF=ELF)→ 不做 PE 分析。真正结构损坏的 PE(`L5-010_bad_mz.exe`、0 字节 exe、截断 exe)仍照常命中。
- **`RemoteDownloadPowerShell` 把 .NET 标识符当下载动作** — 该词表原本是裸子串 `DownloadString|DownloadFile`,`nvidiaInspector.exe` 命中 `add_DownloadFileCompleted`、`DownloadFileAsync` 两处(任何托管程序的标识符表都有),白得 30 分,把 45 分的人工复核顶成 **75 分拒绝**。现改为:cmdlet 名(`Invoke-WebRequest`/`Invoke-RestMethod`/`Start-BitsTransfer`/`BitsAdmin`)带词边界单独命中;.NET/Win32 方法名必须与 URL/IP 同框(±40 字符)才算下载。攻击链没有削弱——`macro_alert.docm` 的 `DownloadString(http://evil.com/ps1)`、`IEX (New-Object Net.WebClient)…` 仍分别命中本规则与 `RemoteDownloadExec`。
**第三轮复测(容器内,策略版本 `2026.10.2-9affe243`):**
| 指标 | 结果 |
|---|---|
| 靶场 168 条(控制台 tier 口径) | **168 Pass / 0 Fail**,`engine_errors` 0 条,总耗时 34.5s(最慢 `L0-020_zip_bomb.zip` 13.4s) |
| `go1.25.0.windows-amd64.msi` | `malicious/reject 70` → **`suspicious/manual_review 30`**,仅剩 `DangerousExecutableExtension`;`real_type` 632 字符被截断到 512,记录 + findings 正常入库 |
| .NET 标识符噪声(`add_DownloadFileCompleted`/`DownloadFileAsync`/`URLDownloadToFileA`) | **clean/pass,0 命中**(对应 `nvidiaInspector.exe` 由 75 拒绝回到 45 人工复核) |
| 干净 PE 基线 | cmd.exe 45 / dxdiag.exe 50 / shell32.dll 65 → 全部 manual_review,均低于 `block_score=70` |
| 勒索真样本 `ransom_dropper.exe.txt` | 仍 `malicious/reject 100`(`C8_Ransomware_Encryption` 85 + `VolumeShadow` 60 + `Note_Filename` 30) |
**ClamAV(L1)命中的处置链路与本轮两处修复:**
- **机制** — ClamAV 签名名是动态的(`Win.Trojan.Agent-1`、`Txt.Malware.Sustes-6779550-1`…),进不了 25 条静态零容忍名单,所以走**权重分档 + 阈值**:`critical=95 / EICAR=90 / 默认 malware=85 / heuristic·packed=70 / PUA=25`。`block_score=70` 意味着除 PUA 外**单条 ClamAV 命中即 `malicious/reject`**;PUA 档 25 分落 `pass_with_flag`,与其他信号叠加后进人工复核。零容忍名单里只显式登记了 `Eicar-Test-Signature`(等保"检测到即阻断",`config/policy.yaml:63`),命中直接 100 分。
- **没有绕过口子** — 步骤 -1 的两类白名单只作用于 `keywords`(关键词白名单)和 `yara`(哈希白名单)引擎,`clamav` 的 finding 不会被过滤;宏行为例外表只覆盖 VBA 零容忍规则。`engine_failure_mode: fail_closed` 下 clamd 连不上/超时会判 `error/reject`。
- **修复一:分档实际是死代码** — `_judge_signature()` 用小写关键字去比对 `signature.upper()`,因此 95 与 70 两档永不命中,除 EICAR/PUA 外所有签名一律落默认 85(注释里的档位与真实行为不符)。关键字改为大写后实测:`Win.Ransomware/Win.Trojan/Win.Dropper`(含 `Win.Trojan.GenericKD`)→ 95 critical,`Win.Heur.Gen`/`Win.Packed.Upx`/`Win.Riskware.Tool` → 70 high,`Unix.Coinminer`(无家族关键字)→ 85 默认,`PUA-OSX.iAdWare` → 25 medium。
- **修复二:1G~1.5G 的文件在 L1 完全裸奔** — clamd 默认 `MaxFileSize=1G`,超限拒扫时引擎返回 `ok=True + meta.skipped` 且 **0 条 finding**,而策略 `max_file_size` 是 1.5GB → 这段区间的文件其他 5 个引擎照跑,结论可以是 `clean/pass`,即"AV 没查过"被当成"AV 没查出"。现补 `ClamAVSkippedSizeLimit`(medium/30):实测仅此一条 → `suspicious/manual_review 30`(修复前同一场景 `clean/pass 0`),与"不可扫描不放行"口径一致。
**第四轮:规则独立贡献数对账 + 复核地板解耦(策略版本 `2026.10.2-d3b747a4`):**
- **口径** — 从 `scan_findings` 重建 findings,用 `PolicyEngine.decide()` 对近 14 天 **3,661 条真实扫描**(排除 `test`/`test_center`,其中 `console_upload` 3,586 条、真实摆渡通道 75 条)做反事实重放:逐条规则剔除其全部命中后重判,处置档位变化即该规则的独立贡献。先验证重放器可信度:改造前 **3,660/3,661 = 99.97%** 与历史入库 action 一致(唯一差异是本轮之前收紧的 `RemoteDownloadUnix`)。
- **裁决高度集中** — 60 条有命中的规则里只有 **25 条有独立贡献**,前 10 条承担 **92.6%**(1,700/1,836 次跨档)。`DangerousExecutableExtension` 一条就贡献 1,074 次(其中 1,070 次是"把可执行文件送进人工复核"这条口径本身);`OfficeEncrypted` 103 次走的是策略步骤 4 的加密分支而不是它的 25 分权重。35 条规则**独立贡献为 0**,分三类:同族冗余(`C8_*`/`Email*`/`Pdf*`/`PathTraversalBlocked`/`C7_..._NC` 总与另一条同批命中)、分数被挡(其余信号已 ≥70)、纯留痕(`PeParseFailed` 806 次命中、610 次连自己的档位都没到;`OoxmlParseFailed`/`OleToolsUnavailable`/`PEArchInfo` 权重 0)。
- **退役必须按族,不能按条** — 按族只保留权重最高代表(C8 族留 `C8_Ransomware_Encryption`、Email 族留 `EmailAttachmentWebshell`、`PathTraversalBlocked`、`PdfOpenAction`、`Sustes`)在 3,661 条上重放:**90+90+92+76+76 条扫描裁决全部不变,零影响**;但把 35 条零贡献规则**同时**退役会让 258 条 reject 降级(255 条直接变 pass,全是 `.eml`/`.txt`/`.pdf`)——它们互为备份,批量删等于整族失明。`Txt.Malware.Sustes-*` 这类是 ClamAV 签名名,属于外部信号映射,不在可退役范围。
- **结构风险修复:`manual_review` 覆盖从"注 65 分"改成"处置地板"** — 旧实现(`app/services/policy.py` 步骤 5)把评分强推到 65 再累加其余权重,同时跳过该规则自身权重:注进去的 65 分不是证据,任何 ≥5 分噪声叠加就翻成 reject,且显示分数与真实证据脱钩。现在步骤 5 只登记地板标志,步骤 6 全量累加权重,步骤 9(新增)在阈值裁决后把低于复核的处置抬到 `manual_review`(已是 reject 的不回退)。同时把 `DangerousExecutableExtension` 登记为 `manual_review` 覆盖——光闸"EXE/DLL/脚本不自动放行"从此由显式地板保证,不再依赖 `weight=30/35` 恰好等于 `review_score` 下界 30 这个数值巧合。
- **第四轮复测:**
| 指标 | 结果 |
|---|---|
| 靶场 168 条(控制台 tier 口径) | **168 Pass / 0 Fail**,`engine_errors` 0 条,总耗时 34.6s(最慢 `L0-020_zip_bomb.zip` 12.3s) |
| 3,661 条真实扫描重放 diff | 只有 **1 条**变化,且是本轮之前那处 `RemoteDownloadUnix` 收紧的预期结果;处置分布与改造前完全一致:reject 2,005 / manual_review 1,091 / pass 561 / pass_with_flag 4(放松 0 条降到 pass,收紧 0 条) |
| 分数纠偏 | 14 条 `ArchiveContainsExecutable` 扫描 action 不变、分数由人为的 65 回落为真实的 30 |
| 干净 PE 基线(实扫) | cmd.exe 45 / dxdiag.exe 50 / shell32.dll 65 → 全部 manual_review,与第三轮逐项一致 |
| 解耦反证 | 把 `review_score` 下界抬到 36/40/50,或把 DEE 权重降到 10:无地板时干净可执行文件静默变 `pass_with_flag`(10~30 分放行),有地板时恒为 `manual_review` |
**meshsafe-cron 自查:定时任务自引入以来一次都没成功执行过。**
- **根因** — `Settings.admin_password` 是必填字段(`app/config.py:51`),但 cron 容器的 environment 里没有 `MESHSAFE_ADMIN_PASSWORD`。于是任何走 `app.db` → `get_settings()` 的路径在 import 阶段就抛 `ValidationError`,`archive_old_tasks.py --apply` 与 `VACUUM ANALYZE` 双双崩溃,`run()` 只往 docker 日志写一行 `❌ 返回码 1`,容器状态照常 `Up`——**任务挂了但没有任何存活信号**。时间线:必填字段来自初始提交 `16f134e`(10-28),cron 来自 `da8496a`(10-29),即从上线第一天起就没跑过。旁证:`policy_versions` 现存 10 行而归档策略是"留 5 删 5",`scan_tasks_archive` 空表。
- **修复一(env)** — `docker-compose.yml` cron 补 `MESHSAFE_ADMIN_PASSWORD=cron-unused-placeholder`。cron 不做控制台认证,用占位值而不是复制真实口令(容器 env 会出现在 `docker inspect` 里)。实测归档 dry-run 在 cron 容器内跑通。
- **修复二(VACUUM 事务块)** — `run_vacuum()` 原来用 `engine.begin()` 包住 `VACUUM ANALYZE`,Postgres 直接 `ActiveSqlTransaction: VACUUM cannot run inside a transaction block`,即使 env 修好,VACUUM 仍会整批失败。改为 `engine.connect().execution_options(isolation_level="AUTOCOMMIT")`;实测 6 张表 VACUUM ANALYZE 共 0.25s 全部完成。
- **修复三(日志泄密)** — pydantic 的 `ValidationError` 会把整个 settings dict 打进 stderr,而 `run()` 原样逐行 `log.warning`,等于把 **DB 连接串(含口令)与 `MESHSAFE_MASTER_KEY` 尾部**明文写进 cron 日志。新增 `_redact()` 按 env 值替换为 `***`(覆盖 `MESHSAFE_DB_URL/MASTER_KEY/ADMIN_PASSWORD/API_KEY/API_CLIENTS`),实测两类密钥串都不再出现在输出里。
**ClamAV 签名库"过期":机制、阈值与一个真实缺口。**
- **"过期"不是失效** — .cvd 库不会到期停用,clamd 照样拿它扫描,结论照常产出。所谓过期只是**新鲜度告警**,两处阈值:引擎 `health()` 里 `signature_days_old > 7` → `status=degraded`(`app/engines/clamav.py:111`,硬编码),诊断规则 R002 用 `config/diagnostics.yaml:47` 的 `clamav_db_stale_days: 7` 出 WARN;控制台签名库面板显示"已 N 天"(`dashboard.html:408`)。当前库 `ClamAV 1.5.4/28166/Tue Oct 27 14:24:12 2026`,今天 6 天,**11-04 起变 8 天开始告警**。
- **内网的更新方式只有离线导入** — `scripts/init_clamav.sh` 仅在库为空时试一次 `freshclam`,失败即静默降级;光闸环境里它注定失败。可用路径:外网下载 `main.cvd`/`daily.cvd`/`bytecode.cvd` → 摆渡进内网 → 控制台「📤 上传签名库」(`app/console.py:1487`,落 `/var/lib/clamav` 后给 clamd 发 SIGHUP 立即热加载,无需重启容器)。官方 daily 库每天发布,按周导入即可把库龄压在 7 天内。
- **缺口(未修,需先定运维周期)** — 库龄只影响展示与诊断告警,`decide()` 完全不看 `signature_days_old`:签名库 30 天没更新时 L1 事实上认不出新病毒,文件照样可能是 `clean/pass`。这与"不可扫描内容不得放行"的口径不一致,同类先例是 `ClamAVSkippedSizeLimit`(clamd 超大小限制没查过 → 补 medium/30 进复核)。建议补 `ClamAVSignatureStale`(medium/30),阈值取**真实摆渡周期 + 缓冲**(按周导入填 10~14 天),不要直接填 7——那会让每次告警窗口把全量文件推进人工复核。
**遗留项(已确认,未在本轮处理):**- `scripts/init_db.py:71` 用 `scalar_one_or_none()` 取最新策略版本,表内多于 1 行即抛 `MultipleResultsFound` → compose 注释里的"YAML 灌库"步骤在已有环境上跑不通;本轮改用 `PolicyStore.save_from_dict()` 灌库(注意 `/api/v1/policies/reload` 只刷新收到请求的那一个 worker,`UVICORN_WORKERS=8` 下仍需重启才全量生效;另外 `docker compose restart api` **不会**换用刚构建的新镜像,改了 `app/` 必须 `docker compose up -d api` 重建容器)。
- **`allow_encrypted_archive: false` 实际未生效** — `scanner.py:391` 只在 document 引擎报 `PdfEncrypted`/`OfficeEncrypted` 时置 `encrypted=True`,压缩包侧的 `EncryptedArchive` 不触发这条策略,于是策略明令禁止的加密压缩包仍以 20 分放行(`L0-024_encrypted.zip` → pass_with_flag)。补一行即可生效,但会把所有加密包从"复核"改成"拒绝",涉及真实业务流量,需先定口径。
- **`ExtractMemberFailed` 权重 0** — 成员解压失败零成本,与上面同一类 fail-open。
- **纯启发式签名单条即拒** — `Win.Heur.*`、`Win.Packed.*`、`Win.Trojan.GenericKD-*` 这类 ClamAV 启发式命中现在分别是 70/95 分,等于"AV 一家之言直接拦截",与"干净可执行文件只进人工复核"的口径不完全一致(启发式是 AV 误报集中区)。降档需要先有内网真实误报样本做标定,否则会把真木马放到复核队列里;本轮保持拒绝。
- **深扫成员后的耗时** — 解包递归真正生效后:PPTX(61 个对象)5.3s、8MB DLL 2.3s、zip 炸弹样本 10.7s;扫描循环上限 2000 个对象,理论上大压缩包可在同步请求里跑到分钟级,而 `sync_max_size` 现值 512MB 基本等于全部走同步。建议按实测流量把同步阈值下调(例如 20MB)或给对象数配一个更低的上限。
- **35 条零独立贡献规则暂不退役** — 单条退役在这 3,661 条上零影响,但它们是同族互为备份(整族一起退役会让 258 条 reject 降级),退役单位是"族"不是"条";而且本语料 97.9% 是控制台上传样本、真实摆渡通道只有 75 条、ClamAV 只在 81 条上命中,样本偏斜不足以证明这些规则在真实恶意样本上也没用。等内网积累到足量真实流量再按簇退役。
### v0.2.3 (2026-10-29)
**Bug 修复:**
- **Policy 加载失败(每次启动都回退 YAML)** — `policy.py _load_db_or_yaml()` 用 `scalar_one_or_none()` 加载最新策略版本,但 `policy_versions` 表有多条历史版本 → SQLAlchemy 抛 `Multiple rows` → 永远回退 YAML 配置文件,导致不同重启间规则数波动。修复为 `db.query(PolicyVersion).order_by(created_at.desc()).first()`,直接取最新一条 ORM 对象
- **控制台任务历史被内部测试记录淹没** — `scan_tasks` 表累积 5,800+ 条内部对账/诊断脚本产生的测试数据(channel 包括 `test_center` / `v` / `verify` / `diag` / `debug` / `final` 等 19 个),dashboard 和 tasks_page 无过滤全部展示。修复:① DB 清理所有内部 channel 记录(保留 3,488 条生产数据);② console.py 加 `INTERNAL_TEST_CHANNELS` 黑名单常量,dashboard 6 个查询 + tasks_page 2 个查询全部 `.filter(ScanTask.channel.notin_(...))`
- **测试靶场显示 "未知" 层和 "—" 编号** — 控制台用 `iterdir()` 遍历目录收集样本,但目录里混入 4 个 YARA 调试时的临时文件(`L4-005_hotreload.txt` 等),index.json 没登记 → category 取不到显示 "未知"。修复:① 清理 4 个临时文件;② `test_center_page` 改为只从 index.json 收集样本(目录遍历替换为 index.json 遍历),以后不会再出这个问题
**DB 与运维增强:**
- **ScanTask 复合索引** — 新增 `ix_scan_tasks_channel_created ON (channel, created_at DESC)`,解决 dashboard/tasks_page 高频查询的全表扫描。models.py `ScanTask.__table_args__` 也同步更新,新建实例会自动带上
- **归档表 + 归档脚本** — 建 `scan_tasks_archive` 表(含 4 个索引);新增 `scripts/archive_old_tasks.py` 支持 4 项清理策略:
- scan_tasks **90 天前**:清 detail JSONB(每条 1-5 KB,省空间)
- scan_tasks **365 天前**:移到 scan_tasks_archive 归档表(CASCADE 删 ScanFinding)
- idempotency_records:清 **7 天前**(幂等缓存过期可以重扫)
- policy_versions:只留**最新 5 个**(历史版本无保留价值)
- 支持 `--dry-run`(默认,只报告不动数据)+ `--apply`(真正执行),分批 1,000 条处理避免长事务锁表
- **自动定时任务 Cron 服务** — docker-compose.yml 新增 `cron` 服务(共享 meshsafe:latest 镜像,零额外依赖,`restart: unless-stopped`),`scripts/cron_entrypoint.py` 纯 Python 调度器:
- **每天 03:00** 跑 archive_old_tasks.py --apply
- **每周一 03:10** VACUUM ANALYZE 5 张表(维护表统计信息,加速查询)
**数据增长实况:**
| 指标 | 值 |
|------|---|
| scan_tasks 当前 | 3,660 行 / 4.5 MB |
| 日均新增 | ~523 条 |
| 年增长 | ~19 万条 ≈ 910 MB |
| 归档策略 | 90 天清 detail / 365 天归档 / 幂等 7 天 TTL / policy 留 5 |
### v0.2.2 (2026-10-29) ### v0.2.2 (2026-10-29)
**引擎能力补全 — 测试靶场 168/168 100% 通过:** **引擎能力补全 — 测试靶场 168/168 100% 通过:**

View File

@@ -18,6 +18,48 @@ import threading
import time import time
from datetime import datetime, timedelta, timezone from datetime import datetime, timedelta, timezone
from pathlib import Path from pathlib import Path
# ============================================================================
# 内部测试 channel 黑名单
# ============================================================================
# 控制台页面(Dashboard / 任务历史)默认隐藏内部测试 / 对账脚本产生的记录。
# 这些 channel 由开发 / 运维工具在内部调用 scanner.scan_file() 时自动产生,
# 不应暴露给控制台用户。若后续新增内部脚本 channel,在此补充即可。
INTERNAL_TEST_CHANNELS: frozenset[str] = frozenset({
"test_center", # 测试靶场手动重扫
"v", # 对账脚本 v5/v7/v8
"verify", # 单条验证脚本
"diag", # 诊断脚本
"debug", # 调试脚本
"final", # 最终验证
"console", # console.py 内部调用
"console-test", # console 测试模式
"regen_v2", # 批量重建
"regenerate_index", # index.json 重建
"audit_f3", # 审计脚本
"audit", # 审计脚本
"testcases", # 生成 TEST_CASES.md
"test", # 测试
"yara_verify", # YARA 验证
"whitelist_v", # 白名单验证
"boundary", # 边界测试
"yara_v2", # YARA v2 测试
"docgen", # 文档生成脚本
})
def _apply_production_channel_filter(query, model_class):
"""给 ScanTask / ScanFinding 查询追加「隐藏内部测试 channel」过滤。
用法: _apply_production_channel_filter(
db.query(func.count(ScanTask.id)), ScanTask
).scalar()
"""
col = getattr(model_class, "channel", None)
if col is None:
return query
return query.filter(col.notin_(INTERNAL_TEST_CHANNELS))
from typing import Any, Dict, List, Optional from typing import Any, Dict, List, Optional
from fastapi import APIRouter, Depends, File, Form, HTTPException, Request, Response, UploadFile from fastapi import APIRouter, Depends, File, Form, HTTPException, Request, Response, UploadFile
@@ -132,7 +174,7 @@ def _annotate_yaml(raw: str) -> str:
# ---- rule_overrides ---- # ---- rule_overrides ----
"rule_overrides": '特定规则的处置覆盖。action: pass | manual_review | reject', "rule_overrides": '特定规则的处置覆盖。action: pass | manual_review | reject',
# ---- yara ---- # ---- yara ----
"yara_rule_weights": 'YARA 规则按严重度分档赋权重(critical=85 high=65 medium=45 low=25)', "yara_rule_weights": 'YARA 规则按严重度分档赋权重(critical=85 high=60 medium=30 low=10)',
# ---- keyword ---- # ---- keyword ----
"keyword_rules": '词表引擎规则。match: contains | regex | any_of', "keyword_rules": '词表引擎规则。match: contains | regex | any_of',
# ---- notification ---- # ---- notification ----
@@ -173,9 +215,9 @@ def _annotate_yaml(raw: str) -> str:
# YARA 权重分档 → 中文 # YARA 权重分档 → 中文
_WEIGHT_TIER_HINTS = { _WEIGHT_TIER_HINTS = {
"critical": "weight=85(严重)", "critical": "weight=85(严重)",
"high": "weight=65(高危)", "high": "weight=60(高危)",
"medium": "weight=45(中等)", "medium": "weight=30(中等)",
"low": "weight=25(低危)", "low": "weight=10(低危)",
} }
lines = raw.split("\n") lines = raw.split("\n")
@@ -535,30 +577,31 @@ async def dashboard(request: Request, current=Depends(require_login)):
db: Session = SessionLocal() db: Session = SessionLocal()
try: try:
from sqlalchemy import func from sqlalchemy import func
total_scans = db.query(func.count(ScanTask.id)).scalar() or 0 _skip_test = ScanTask.channel.notin_(INTERNAL_TEST_CHANNELS)
total_scans = db.query(func.count(ScanTask.id)).filter(_skip_test).scalar() or 0
total_blocked = db.query(func.count(ScanTask.id)).filter( total_blocked = db.query(func.count(ScanTask.id)).filter(
ScanTask.action == "reject" ScanTask.action == "reject", _skip_test
).scalar() or 0 ).scalar() or 0
total_flagged = db.query(func.count(ScanTask.id)).filter( total_flagged = db.query(func.count(ScanTask.id)).filter(
ScanTask.action.in_(("reject", "manual_review", "pass_with_flag")) ScanTask.action.in_(("reject", "manual_review", "pass_with_flag")), _skip_test
).scalar() or 0 ).scalar() or 0
total_findings = db.query(func.count(ScanFinding.id)).scalar() or 0 total_findings = db.query(func.count(ScanFinding.id)).scalar() or 0
# 结论分布 # 结论分布
verdict_rows = db.query(ScanTask.verdict, func.count(ScanTask.id)).group_by( verdict_rows = db.query(ScanTask.verdict, func.count(ScanTask.id)).filter(
ScanTask.verdict _skip_test
).all() ).group_by(ScanTask.verdict).all()
verdict_dist = {str(v or "unknown"): int(c) for v, c in verdict_rows} verdict_dist = {str(v or "unknown"): int(c) for v, c in verdict_rows}
# 最近任务(8 条) # 最近任务(8 条)
recent_tasks = db.query(ScanTask).order_by( recent_tasks = db.query(ScanTask).filter(_skip_test).order_by(
ScanTask.created_at.desc() ScanTask.created_at.desc()
).limit(8).all() ).limit(8).all()
# 各客户端/渠道的任务数(摆渡关联性) # 各客户端/渠道的任务数(摆渡关联性)
client_rows = db.query( client_rows = db.query(
ScanTask.client_id, ScanTask.channel, func.count(ScanTask.id) ScanTask.client_id, ScanTask.channel, func.count(ScanTask.id)
).group_by(ScanTask.client_id, ScanTask.channel).all() ).filter(_skip_test).group_by(ScanTask.client_id, ScanTask.channel).all()
finally: finally:
db.close() db.close()
@@ -825,7 +868,8 @@ async def tasks_page(request: Request, current=Depends(require_login)):
db: Session = SessionLocal() db: Session = SessionLocal()
try: try:
total = db.query(func.count(ScanTask.id)).scalar() or 0 _skip_test = ScanTask.channel.notin_(INTERNAL_TEST_CHANNELS)
total = db.query(func.count(ScanTask.id)).filter(_skip_test).scalar() or 0
total_pages = max(1, (total + PAGE_SIZE - 1) // PAGE_SIZE) total_pages = max(1, (total + PAGE_SIZE - 1) // PAGE_SIZE)
# 越界时回退到最后一页 # 越界时回退到最后一页
if page > total_pages: if page > total_pages:
@@ -834,6 +878,7 @@ async def tasks_page(request: Request, current=Depends(require_login)):
tasks = ( tasks = (
db.query(ScanTask) db.query(ScanTask)
.filter(_skip_test)
.order_by(ScanTask.created_at.desc()) .order_by(ScanTask.created_at.desc())
.limit(PAGE_SIZE) .limit(PAGE_SIZE)
.offset(offset) .offset(offset)
@@ -915,15 +960,13 @@ async def test_page(request: Request, current=Depends(require_login)):
index = json.loads(index_path.read_text(encoding="utf-8")) index = json.loads(index_path.read_text(encoding="utf-8"))
meta_map = {s["filename"]: s for s in index.get("samples", [])} meta_map = {s["filename"]: s for s in index.get("samples", [])}
# 收集所有样本文件 # ★ 只展示 index.json 里登记的样本(不遍历目录,避免临时/调试文件混入)
for fpath in sorted(samples_dir.iterdir()): # 每个样本必须: index.json 有登记 + 磁盘上确实存在
if not fpath.is_file(): for s in index.get("samples", []):
continue fname = s["filename"]
if fpath.name == "index.json" or fpath.name.startswith("_"): fpath = samples_dir / fname
continue if fpath.is_file():
if fpath.suffix == ".json" and fpath.name.startswith("_scan_results"): samples_list.append(fpath)
continue
samples_list.append(fpath)
results: list[dict] = [] results: list[dict] = []
pass_cnt = fail_cnt = 0 pass_cnt = fail_cnt = 0
@@ -1233,17 +1276,18 @@ async def test_upload(
with tempfile.NamedTemporaryFile(delete=False, suffix=suffix) as tmp: with tempfile.NamedTemporaryFile(delete=False, suffix=suffix) as tmp:
content = await f.read() content = await f.read()
if not content: if not content:
# 空文件不是错误 — 合法文件, 只是没内容, 直接放行
results.append({ results.append({
"filename": filename, "filename": filename,
"size": 0, "size": 0,
"verdict": "error", "verdict": "pass",
"verdict_cn": verdict_label("error"), "verdict_cn": verdict_label("pass"),
"verdict_badge": verdict_badge("error"), "verdict_badge": verdict_badge("pass"),
"score": 0, "score": 0,
"reason": "空文件", "reason": "空文件,无风险",
"findings": [], "findings": [],
"duration_ms": 0, "duration_ms": 0,
"ok": False, "ok": True,
}) })
continue continue
tmp.write(content) tmp.write(content)

View File

@@ -23,6 +23,32 @@ BINARY_EXTS = {".exe", ".dll", ".sys", ".scr", ".com", ".cpl", ".ocx", ".msi", "
_PE_SCN_MEM_EXECUTE = 0x20000000 _PE_SCN_MEM_EXECUTE = 0x20000000
_PE_SCN_MEM_WRITE = 0x80000000 _PE_SCN_MEM_WRITE = 0x80000000
# 声明扩展名 → 该格式自己的魔数。这些容器本就不是 PE,用 pefile 解析必然抛错;
# 把"格式完全正常的安装包/共享库"记成 PeParseFailed 会给每个 .msi/.jar/.so 白送 10 分。
_NATIVE_MAGIC: Dict[str, Any] = {
".msi": (b"\xd0\xcf\x11\xe0",),
".msp": (b"\xd0\xcf\x11\xe0",),
".mst": (b"\xd0\xcf\x11\xe0",),
".jar": (b"PK\x03\x04", b"PK\x05\x06"),
".war": (b"PK\x03\x04", b"PK\x05\x06"),
".apk": (b"PK\x03\x04", b"PK\x05\x06"),
".so": (b"\x7fELF",),
".elf": (b"\x7fELF",),
}
def _is_native_container(path: str, ext: str) -> bool:
"""文件内容是否就是其声明格式(非 PE)。"""
expected = _NATIVE_MAGIC.get(ext)
if not expected:
return False
try:
with open(path, "rb") as fh:
head = fh.read(4)
except OSError:
return False
return any(head.startswith(m) for m in expected)
def _manual_pe_analysis(path: str) -> List[Finding]: def _manual_pe_analysis(path: str) -> List[Finding]:
"""pefile 解析失败时,手工搜索 PE\\0\\0 签名并解析节表。 """pefile 解析失败时,手工搜索 PE\\0\\0 签名并解析节表。
@@ -172,6 +198,8 @@ class YaraEngine(Engine):
return result return result
try: try:
# 本环境的 yara-python 4.5.4 在 Match 对象上默认就带 meta,
# 且 rules.match() 不接受 with_meta 关键字(传了会整引擎报错)。
matches = rules.match(target.path) matches = rules.match(target.path)
except Exception as exc: # noqa: BLE001 except Exception as exc: # noqa: BLE001
result.ok = False result.ok = False
@@ -179,7 +207,8 @@ class YaraEngine(Engine):
return result return result
for match in matches: for match in matches:
severity, weight = _judge_yara_rule(match.rule, match.tags) meta = getattr(match, "meta", None) or {}
severity, weight = _judge_yara_rule(match.rule, match.tags, meta)
# 覆盖:如果 ctx 里的策略为该 YARA 规则定义了精细权重,用它 # 覆盖:如果 ctx 里的策略为该 YARA 规则定义了精细权重,用它
try: try:
policy = ctx.policy_engine.policy policy = ctx.policy_engine.policy
@@ -188,13 +217,18 @@ class YaraEngine(Engine):
weight = float(yara_weights[match.rule]) weight = float(yara_weights[match.rule])
except Exception: except Exception:
pass # ctx 没有 policy 字段时保持默认 weight pass # ctx 没有 policy 字段时保持默认 weight
desc = str(meta.get("description", "")).strip()
tags_text = ", ".join(match.tags)
result.findings.append( result.findings.append(
Finding( Finding(
engine=self.name, engine=self.name,
layer=self.layer, layer=self.layer,
severity=severity, severity=severity,
rule=match.rule, rule=match.rule,
detail=f"YARA 规则 {match.rule} 命中(标签:{', '.join(match.tags) or '无'})", detail=(
f"YARA 规则 {match.rule} 命中"
+ (f"({desc})" if desc else f"(标签:{tags_text or '无'})")
),
weight=weight, weight=weight,
meta={"tags": list(match.tags)}, meta={"tags": list(match.tags)},
) )
@@ -232,6 +266,10 @@ class BinaryEngine(Engine):
if ext not in BINARY_EXTS: if ext not in BINARY_EXTS:
return result return result
# 内容确实是声明的那个非 PE 容器(MSI=CFB、JAR/APK=ZIP、SO/ELF=ELF)→ 不做 PE 分析
if _is_native_container(target.path, ext):
return result
try: try:
import pefile import pefile
except Exception: # noqa: BLE001 except Exception: # noqa: BLE001
@@ -290,17 +328,54 @@ class BinaryEngine(Engine):
) )
) )
# 2) 可疑导入表 # 2) 导入表能力信号:只做佐证,不做独立定罪
suspicious_imports = self._suspicious_imports(pe) # Windows 自身的内核/外壳组件(kernel32、shell32、wer)必然导入
if suspicious_imports: # VirtualAllocEx/WriteProcessMemory 这类 API——它们就是这些 API 的实现方。
# 真正的注入判定由 YARA C1_PE_ProcessInjection(导入名邻接窗口)负责,
# 这里按「命中个数」而不是「命中即高危」计分,避免正常软件被判恶意。
tier1, tier2, tier3 = self._suspicious_imports(pe)
if len(tier1) >= 2:
findings.append( findings.append(
Finding( Finding(
engine=self.name, engine=self.name,
layer=self.layer, layer=self.layer,
severity=Severity.HIGH.value, severity=Severity.HIGH.value,
rule="SuspiciousImports", rule="SuspiciousImports",
detail=f"导入表包含高危 API:{', '.join(suspicious_imports[:8])}", detail=f"导入表同时包含多个注入/后门级 API:{', '.join(tier1[:8])}",
weight=25.0, weight=20.0,
)
)
elif len(tier1) == 1:
findings.append(
Finding(
engine=self.name,
layer=self.layer,
severity=Severity.LOW.value,
rule="SuspiciousImports",
detail=f"导入表含单个注入/后门级 API(系统组件常态,需其他信号佐证):{tier1[0]}",
weight=5.0,
)
)
if len(tier2) >= 3:
findings.append(
Finding(
engine=self.name,
layer=self.layer,
severity=Severity.LOW.value,
rule="CapabilityImports",
detail=f"导入表同时具备多项危险能力:{', '.join(tier2[:8])}",
weight=10.0,
)
)
elif tier3:
findings.append(
Finding(
engine=self.name,
layer=self.layer,
severity=Severity.INFO.value,
rule="AntiDebugImports",
detail=f"导入表含反调试/环境探测 API(正常软件普遍使用):{', '.join(tier3[:8])}",
weight=0.0,
) )
) )
@@ -312,8 +387,8 @@ class BinaryEngine(Engine):
layer=self.layer, layer=self.layer,
severity=Severity.MEDIUM.value, severity=Severity.MEDIUM.value,
rule="UnsignedExecutable", rule="UnsignedExecutable",
detail="可执行文件未携带数字签名", detail="可执行文件未携带数字签名(内网自研工具常态,仅辅助人工判断)",
weight=20.0, weight=10.0,
) )
) )
@@ -330,7 +405,7 @@ class BinaryEngine(Engine):
severity=Severity.LOW.value, severity=Severity.LOW.value,
rule="SuspiciousTimestamp", rule="SuspiciousTimestamp",
detail="PE 编译时间戳异常,可能被伪造", detail="PE 编译时间戳异常,可能被伪造",
weight=10.0, weight=0.0,
) )
) )
except Exception: # noqa: BLE001 except Exception: # noqa: BLE001
@@ -433,32 +508,62 @@ class BinaryEngine(Engine):
severity=Severity.LOW.value, severity=Severity.LOW.value,
rule="DebugInformationPresent", rule="DebugInformationPresent",
detail="PE 包含调试信息(PDB 路径或 .debug 节),可用于逆向定位", detail="PE 包含调试信息(PDB 路径或 .debug 节),可用于逆向定位",
weight=10.0, weight=0.0,
) )
) )
return findings return findings
def _suspicious_imports(self, pe: Any) -> List[str]: def _suspicious_imports(self, pe: Any) -> "tuple[List[str], List[str], List[str]]":
"""检测高危 API 导入。""" """导入 API 按恶意指向分档。
suspicious = {
返回 (tier1, tier2, tier3):
tier1 注入/后门级 API —— 单独出现即可疑
tier2 危险能力 API —— 正常软件也常用,需多项共存才计分
tier3 反调试/探测 API —— 信息项,不计分(大量合法商业软件带加壳自保会用)
"""
tier1_set = {
"VirtualAllocEx", "WriteProcessMemory", "CreateRemoteThread", "VirtualAllocEx", "WriteProcessMemory", "CreateRemoteThread",
"NtUnmapViewOfSection", "SetWindowsHookEx", "OpenProcess", "NtUnmapViewOfSection", "SetWindowsHookEx", "SetWindowsHookExA",
"WinHttpOpen", "InternetOpenUrl", "URLDownloadToFile", "SetWindowsHookExW", "URLDownloadToFileA", "URLDownloadToFileW",
"CryptEncrypt", "AdjustTokenPrivileges", "CreateService", "QueueUserAPC", "NtQueueApcThread", "RtlCreateUserThread",
"ShellExecute", "WinExec", "IsDebuggerPresent", "CheckRemoteDebuggerPresent",
} }
found: List[str] = [] tier2_set = {
"OpenProcess", "WinHttpOpen", "WinHttpOpenRequest", "WinHttpSendRequest",
"CryptEncrypt", "CryptDeriveKey", "AdjustTokenPrivileges",
"LookupPrivilegeValueA", "LookupPrivilegeValueW", "OpenProcessToken",
"VirtualProtectEx", "ReadProcessMemory", "SuspendThread", "GetAsyncKeyState",
"SetWinEventHook", "HttpSendRequestA", "InternetWriteFile",
"RegSetValueExA", "RegSetValueExW",
# 以下 API 恶意软件常用,但任何"要装服务/要启动进程/要联网"的正常
# 软件都必然导入它们,不足以单独定罪
"WinExec", "ShellExecuteExA", "ShellExecuteExW", "CreateServiceA",
"CreateServiceW", "InternetOpenUrlA", "InternetOpenUrlW",
"CreateNamedPipeA", "DefineDosDeviceW",
}
tier3_set = {
"IsDebuggerPresent", "CheckRemoteDebuggerPresent", "OutputDebugStringA",
"GetTickCount", "GetTickCount64", "QueryPerformanceCounter",
"GetProcessHeap", "FindWindowA", "FindWindowW",
}
found1: List[str] = []
found2: List[str] = []
found3: List[str] = []
try: try:
for entry in getattr(pe, "DIRECTORY_ENTRY_IMPORT", []) or []: for entry in getattr(pe, "DIRECTORY_ENTRY_IMPORT", []) or []:
for imp in entry.imports: for imp in entry.imports:
if imp.name: if not imp.name:
name = imp.name.decode("utf-8", errors="replace") continue
if name in suspicious: name = imp.name.decode("utf-8", errors="replace")
found.append(name) if name in tier1_set:
found1.append(name)
elif name in tier2_set:
found2.append(name)
elif name in tier3_set:
found3.append(name)
except Exception: # noqa: BLE001 except Exception: # noqa: BLE001
pass pass
return sorted(set(found)) return sorted(set(found1)), sorted(set(found2)), sorted(set(found3))
def _has_signature(self, pe: Any) -> bool: def _has_signature(self, pe: Any) -> bool:
"""检查 PE 是否包含 Authenticode 签名目录。""" """检查 PE 是否包含 Authenticode 签名目录。"""
@@ -469,8 +574,25 @@ class BinaryEngine(Engine):
return False return False
def _judge_yara_rule(rule: str, tags: List[str]) -> tuple[str, float]: _YARA_RISK_SEVERITY = {
"""依据 YARA 规则名与标签判定严重度。""" "critical": Severity.CRITICAL.value,
"high": Severity.HIGH.value,
"medium": Severity.MEDIUM.value,
"low": Severity.LOW.value,
}
_YARA_RISK_WEIGHT = {"critical": 85.0, "high": 60.0, "medium": 30.0, "low": 10.0}
def _judge_yara_rule(rule: str, tags: List[str], meta: Dict[str, Any] | None = None) -> tuple[str, float]:
"""依据规则 meta.risk_level 判定严重度(其次才是标签/规则名启发式)。
v0.2.3 只看标签与规则名,规则库里的 risk_level 被完全忽略,
结果 65 分的 C1_PE_RAT 显示成"低危",权重与严重度两套口径。
"""
risk = str((meta or {}).get("risk_level", "")).strip().lower()
if risk in _YARA_RISK_SEVERITY:
return _YARA_RISK_SEVERITY[risk], _YARA_RISK_WEIGHT[risk]
tag_set = {t.lower() for t in tags} tag_set = {t.lower() for t in tags}
text = f"{rule} {' '.join(tags)}".lower() text = f"{rule} {' '.join(tags)}".lower()

View File

@@ -45,9 +45,21 @@ class ClamAVEngine(Engine):
except _ClamdSizeLimit: except _ClamdSizeLimit:
# 文件超过 ClamAV MaxFileSize(clamd.conf 可调,默认 1G) # 文件超过 ClamAV MaxFileSize(clamd.conf 可调,默认 1G)
# → 引擎只是拒扫这个大文件,不当成引擎故障 # → 引擎只是拒扫这个大文件,不当成引擎故障
# 但"没查过"必须留下信号:策略里 max_file_size 是 1.5G,1G~1.5G 的文件
# 原本在 L1 完全裸奔却仍可能 clean/pass。按"不可扫描不放行"落人工复核。
result.ok = True result.ok = True
result.meta["skipped"] = "size_limit" result.meta["skipped"] = "size_limit"
result.meta["reason"] = "文件超过 ClamAV 大小限制已跳过" result.meta["reason"] = "文件超过 ClamAV 大小限制已跳过"
result.findings.append(
Finding(
engine=self.name,
layer=self.layer,
severity=Severity.MEDIUM.value,
rule="ClamAVSkippedSizeLimit",
detail="文件超过 clamd MaxFileSize,L1 签名查杀未执行,按不可扫描内容处置",
weight=30.0,
)
)
return result return result
except Exception as exc: # noqa: BLE001 except Exception as exc: # noqa: BLE001
# 真·引擎不可用:连不上 / PING 超时 / daemon 挂了 # 真·引擎不可用:连不上 / PING 超时 / daemon 挂了
@@ -267,9 +279,9 @@ def _parse_clamd_response(text: str) -> Dict[str, Any]:
# 签名名中的可信度关键词映射到严重度 # 签名名中的可信度关键词映射到严重度
_HIGH_KEYWORDS = ("ransom", "crypt", "trojan", "backdoor", "rootkit", "stealer", "keylog", "worm", "dropper", "ransomware") _HIGH_KEYWORDS = ("RANSOM", "CRYPT", "TROJAN", "BACKDOOR", "ROOTKIT", "STEALER", "KEYLOG", "WORM", "DROPPER")
_MEDIUM_KEYWORDS = ("generic", "heur", "packed", "obfusc", "suspect", "malware", "riskware", "virus") _MEDIUM_KEYWORDS = ("GENERIC", "HEUR", "PACKED", "OBFUSC", "SUSPECT", "MALWARE", "RISKWARE", "VIRUS")
_UNWANTED_KEYWORDS = ("PUA", "unwanted", "toolkit", "pup", "adware") _UNWANTED_KEYWORDS = ("PUA", "UNWANTED", "TOOLKIT", "PUP", "ADWARE")
# EICAR 测试病毒签名 —— 明确标为 critical(等保要求:检测到即阻断) # EICAR 测试病毒签名 —— 明确标为 critical(等保要求:检测到即阻断)
_EICAR_SIGNATURES = ("Eicar-Test-Signature", "EICAR-AV-Test", "Eicar-Test-Signature.UNOFFICIAL") _EICAR_SIGNATURES = ("Eicar-Test-Signature", "EICAR-AV-Test", "Eicar-Test-Signature.UNOFFICIAL")

View File

@@ -56,6 +56,19 @@ EML_EXTS = {".eml", ".msg"}
# 带 VBA 宏的 Office 扩展名(用于 VBA 纯文本扫描触发) # 带 VBA 宏的 Office 扩展名(用于 VBA 纯文本扫描触发)
VBA_MACRO_OFFICE_EXTS = OLE_EXTS | OOXML_EXTS | {".bas", ".vba", ".cls", ".frm"} VBA_MACRO_OFFICE_EXTS = OLE_EXTS | OOXML_EXTS | {".bas", ".vba", ".cls", ".frm"}
# OOXML 包内的内嵌 OLE 部件目录
EMBED_PREFIX_RE = re.compile(r"^(?:word|xl|ppt)/embeddings/.+")
# 内嵌对象中的办公载体:文档互嵌(子表格/图表/PDF)是正常用法,不等于藏可执行
OFFICE_PART_EXTS = {
".docx", ".docm", ".xlsx", ".xlsm", ".pptx", ".pptm",
".doc", ".xls", ".ppt", ".pdf", ".rtf", ".xml", ".txt", ".csv",
".htm", ".html", ".wmf", ".emf", ".bin",
}
# 内嵌部件里直接判定为可执行对象的扩展名
EXEC_PART_EXTS = {".exe", ".dll", ".scr", ".js", ".jse", ".vbs", ".vbe", ".bat", ".cmd", ".ps1", ".hta", ".com", ".msi"}
# VBA 自动执行入口关键词 # VBA 自动执行入口关键词
VBA_AUTOEXEC_PATTERNS = [ VBA_AUTOEXEC_PATTERNS = [
r"Sub\s+AutoOpen\s*\(", r"Sub\s+AutoExec\s*\(", r"Sub\s+AutoNew\s*\(", r"Sub\s+AutoOpen\s*\(", r"Sub\s+AutoExec\s*\(", r"Sub\s+AutoNew\s*\(",
@@ -211,14 +224,16 @@ class DocumentEngine(Engine):
else: else:
parser.close() parser.close()
except Exception as exc: # noqa: BLE001 except Exception as exc: # noqa: BLE001
# 宏流解析不了 = 内容不可判定。这类畸形 OLE 常被用来探测解析器异常,
# 给 0 分等于"看不懂就放行",落人工复核。
findings.append( findings.append(
Finding( Finding(
engine=self.name, engine=self.name,
layer=self.layer, layer=self.layer,
severity=Severity.LOW.value, severity=Severity.MEDIUM.value,
rule="OleToolsUnavailable", rule="OleToolsUnavailable",
detail=f"VBA 宏解析失败:{exc}", detail=f"VBA 宏解析失败:{exc}",
weight=0.0, weight=30.0,
) )
) )
@@ -407,6 +422,24 @@ class DocumentEngine(Engine):
"excerpt": excerpt, "excerpt": excerpt,
} }
def _looks_like_executable(self, zf: Any, name: str) -> bool:
"""判断内嵌部件是否为可执行载荷(扩展名 + 文件头魔数)。
oleObject1.bin 之类的外壳里可能装着真正的 PE,光看扩展名会漏;
反之把每个 .bin 都判成可执行又会拦掉正常的图表/子文档。
"""
if os.path.splitext(name)[1].lower() in EXEC_PART_EXTS:
return True
try:
with zf.open(name) as fp:
head = fp.read(1 << 20)
except Exception: # noqa: BLE001
return False
if head[:2] == b"MZ" or head[:4] in (b"\x7fELF", b"\xca\xfe\xba\xbe"):
return True
# OLE 复合容器(Package/Ole10Native)内嵌 PE:找 MZ..PE 头
return bool(re.search(rb"MZ[\x00-\xff]{0,64}PE\x00\x00", head))
def _scan_ooxml(self, target: ScanTarget) -> List[Finding]: def _scan_ooxml(self, target: ScanTarget) -> List[Finding]:
"""检测 OOXML(docx/xlsx/pptx)中的外部引用与外部链接。""" """检测 OOXML(docx/xlsx/pptx)中的外部引用与外部链接。"""
findings: List[Finding] = [] findings: List[Finding] = []
@@ -429,9 +462,11 @@ class DocumentEngine(Engine):
weight=45.0, weight=45.0,
) )
) )
if name.startswith("word/embeddings/") or name.startswith("xl/embeddings/") or name.startswith("ppt/embeddings/"): # 内嵌 OLE 部件。zip 目录条目(以 / 结尾、长度为 0)不是部件——
# PowerPoint 导出的包常带空的 ppt/embeddings/ 目录,按文件处理会凭空报内嵌对象。
if EMBED_PREFIX_RE.match(name) and not name.endswith("/"):
embedded_ext = os.path.splitext(name)[1].lower() embedded_ext = os.path.splitext(name)[1].lower()
if embedded_ext in {".exe", ".dll", ".scr", ".js", ".vbs", ".bat", ".bin"}: if self._looks_like_executable(zf, name):
findings.append( findings.append(
Finding( Finding(
engine=self.name, engine=self.name,
@@ -442,6 +477,17 @@ class DocumentEngine(Engine):
weight=75.0, weight=75.0,
) )
) )
elif embedded_ext in OFFICE_PART_EXTS:
findings.append(
Finding(
engine=self.name,
layer=self.layer,
severity=Severity.LOW.value,
rule="EmbeddedObject",
detail=f"文档内嵌办公对象:{name}",
weight=5.0,
)
)
else: else:
findings.append( findings.append(
Finding( Finding(

View File

@@ -205,8 +205,9 @@ class SafeExtractor:
safe_name = self._sanitize_name(name) safe_name = self._sanitize_name(name)
dest = os.path.join(out_dir, safe_name) dest = os.path.join(out_dir, safe_name)
try: try:
written = 0
overflow = False
with zf.open(info) as src, open(dest, "wb") as out: with zf.open(info) as src, open(dest, "wb") as out:
written = 0
while True: while True:
chunk = src.read(1024 * 1024) chunk = src.read(1024 * 1024)
if not chunk: if not chunk:
@@ -222,20 +223,22 @@ class SafeExtractor:
f"成员 {name} 实际解压超限,已丢弃", f"成员 {name} 实际解压超限,已丢弃",
10.0, 10.0,
) )
overflow = True
break break
out.write(chunk) out.write(chunk)
else: # 成员必须回到扫描队列,否则压缩包/OOXML 里的内容永远不参与检测
report.total_bytes += written if not overflow:
report.files.append( report.total_bytes += written
ScanTarget( report.files.append(
path=dest, ScanTarget(
filename=safe_name, path=dest,
declared_ext=os.path.splitext(safe_name)[1].lower(), filename=safe_name,
logical_path=f"{target.logical_path}!/{name}", declared_ext=os.path.splitext(safe_name)[1].lower(),
depth=target.depth + 1, logical_path=f"{target.logical_path}!/{name}",
is_extracted=True, depth=target.depth + 1,
) is_extracted=True,
) )
)
except Exception as exc: # noqa: BLE001 except Exception as exc: # noqa: BLE001
self._add_finding( self._add_finding(
report, report,
@@ -323,14 +326,26 @@ class SafeExtractor:
continue continue
with src, open(dest, "wb") as out: with src, open(dest, "wb") as out:
written = 0 written = 0
overflow = False
while True: while True:
chunk = src.read(1024 * 1024) chunk = src.read(1024 * 1024)
if not chunk: if not chunk:
break break
written += len(chunk) written += len(chunk)
if written > self.settings.max_upload_bytes: if written > self.settings.max_upload_bytes:
out.close()
os.remove(dest)
self._add_finding(
report,
Severity.MEDIUM.value,
"MemberTooLarge",
f"成员 {name} 实际解压超限,已丢弃",
10.0,
)
overflow = True
break break
out.write(chunk) out.write(chunk)
if not overflow:
report.total_bytes += written report.total_bytes += written
report.files.append( report.files.append(
ScanTarget( ScanTarget(

View File

@@ -13,6 +13,7 @@
from __future__ import annotations from __future__ import annotations
import logging import logging
import os
import re import re
from typing import Any, Dict, List from typing import Any, Dict, List
@@ -20,10 +21,33 @@ from app.engines.base import Engine, EngineResult, Finding, ScanTarget, Severity
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
# cross_line 规则用的续行归一:换行(及其两侧空白)压成一个空格
NEWLINE_RUN_RE = re.compile(r"\s*[\r\n]+\s*")
class KeywordEngine(Engine): class KeywordEngine(Engine):
"""内网特征词表匹配引擎。""" """内网特征词表匹配引擎。"""
# ---- 跳过列表:二进制 / 压缩封装 / 媒体文件 ----
# 这些文件有专门引擎处理(PDF/Office/Archive/PE 等),
# 它们的内容是压缩流或二进制格式,关键词 regex 在里面会大面积误报。
# 短关键词 (nc/wce/curl/...) 在压缩流里等于伪随机字节必然碰撞。
BINARY_SKIP_EXTS = frozenset({
# PDF / Office
".pdf", ".docx", ".docm", ".doc", ".xlsx", ".xlsm", ".xls",
".pptx", ".pptm", ".ppt", ".ppsx", ".pps", ".potx",
# 压缩包
".rar", ".zip", ".7z", ".tar", ".gz", ".bz2", ".xz", ".tgz",
".iso", ".img", ".dmg",
# 图片
".png", ".jpg", ".jpeg", ".gif", ".bmp", ".tiff", ".tif", ".webp",
".ico", ".svg",
# 视频
".mp4", ".avi", ".mkv", ".mov", ".flv", ".wmv", ".webm",
# 音频
".mp3", ".wav", ".flac", ".ogg", ".aac", ".wma", ".m4a",
})
name = "keywords" name = "keywords"
layer = "L2B" layer = "L2B"
version = "1.0" version = "1.0"
@@ -57,6 +81,14 @@ class KeywordEngine(Engine):
findings: List[Finding] = [] findings: List[Finding] = []
# ---- 跳过二进制/压缩封装文件 ----
# 这些文件内容是压缩流或纯二进制,短关键词 regex 在里面必然误报
ext = (target.declared_ext or "").lower()
if not ext and target.path:
ext = os.path.splitext(target.path)[1].lower()
if ext in self.BINARY_SKIP_EXTS:
return result
try: try:
text = self._extract_text(target.path, target.declared_ext) text = self._extract_text(target.path, target.declared_ext)
except Exception: # noqa: BLE001 except Exception: # noqa: BLE001
@@ -65,12 +97,13 @@ class KeywordEngine(Engine):
if not text: if not text:
return result return result
# ---- 预处理:合并换行符以支持跨行关键词 ---- # ---- 预处理 ----
# 攻击者会把 Invoke-WebRequest 这种关键字用 CRLF 拆成 # 默认按原始行匹配:把整篇文本的换行全部删除会让上一行行尾和下一行行首
# "Invoke-Web\r\nRequest" 来绕过基于单行的匹配。 # 黏成一句话(Ruby 源码 "…breed\n+fetch()…" 曾被当成 curl 特征命中)。
# 这里把所有 \r\n / \r / \n 替换为空,让跨行关键词重新拼接。 # cross_line: true 的规则再跑一遍"换行压成空格"的文本,用于兜住
# 注意:不影响纯文本/多行语义,我们匹配的都是单行关键词或跨行命令。 # wget \ + 换行 + -O payload 这类续行绕过;保留分隔符,不产生黏连。
text = text.replace("\r\n", "").replace("\r", "").replace("\n", "") text_norm = text.replace("\r\n", "\n").replace("\r", "\n")
text_joined = NEWLINE_RUN_RE.sub(" ", text_norm)
for rule in self._compiled: for rule in self._compiled:
match_mode = rule.get("match", "contains") match_mode = rule.get("match", "contains")
@@ -84,23 +117,29 @@ class KeywordEngine(Engine):
if match_mode == "regex": if match_mode == "regex":
compiled = rule.get("_re") compiled = rule.get("_re")
if compiled: if compiled:
for m in compiled.finditer(text): scan_targets = [text_norm]
snippet = text[max(0, m.start() - 10): m.end() + 10] if rule.get("cross_line"):
hits.append(snippet.strip()) scan_targets.append(text_joined)
for body in scan_targets:
for m in compiled.finditer(body):
snippet = body[max(0, m.start() - 10): m.end() + 10]
hits.append(" ".join(snippet.split()))
if len(hits) >= 3:
break
if len(hits) >= 3: if len(hits) >= 3:
break break
elif match_mode == "any_of": elif match_mode == "any_of":
keywords = rule.get("keywords", []) keywords = rule.get("keywords", [])
for kw in keywords: for kw in keywords:
if kw.lower() in text.lower(): if kw.lower() in text_norm.lower():
hits.append(kw) hits.append(kw)
if len(hits) >= 3: if len(hits) >= 3:
break break
elif match_mode == "contains": elif match_mode == "contains":
pattern = rule.get("pattern", "") pattern = rule.get("pattern", "")
if pattern and (pattern.lower() in text.lower()): if pattern and (pattern.lower() in text_norm.lower()):
hits.append(pattern) hits.append(pattern)
if hits: if hits:

View File

@@ -108,6 +108,9 @@ class StructureEngine(Engine):
) )
# 3) 危险可执行扩展名(顶层文件) # 3) 危险可执行扩展名(顶层文件)
# 这是「载体」信号而不是恶意行为:单独命中应当进人工复核(≥review_score),
# 但不能成为把后续中/低置信度检测一起顶过 block_score 的底座,
# 否则正常签名的 exe/dll 会被判恶意。
if ext in DANGEROUS_EXTS and not target.is_extracted: if ext in DANGEROUS_EXTS and not target.is_extracted:
severity = Severity.CRITICAL.value if ext in {".exe", ".scr", ".dll", ".com", ".pif"} else Severity.HIGH.value severity = Severity.CRITICAL.value if ext in {".exe", ".scr", ".dll", ".com", ".pif"} else Severity.HIGH.value
findings.append( findings.append(
@@ -117,7 +120,7 @@ class StructureEngine(Engine):
severity=severity, severity=severity,
rule="DangerousExecutableExtension", rule="DangerousExecutableExtension",
detail=f"文件类型为可执行/脚本格式:{ext}", detail=f"文件类型为可执行/脚本格式:{ext}",
weight=45.0 if severity == Severity.CRITICAL.value else 30.0, weight=35.0 if severity == Severity.CRITICAL.value else 30.0,
) )
) )

View File

@@ -52,7 +52,7 @@ class ScanTask(Base):
size: Mapped[int] = mapped_column(Integer, default=0) size: Mapped[int] = mapped_column(Integer, default=0)
sha256: Mapped[str] = mapped_column(String(64), index=True) sha256: Mapped[str] = mapped_column(String(64), index=True)
md5: Mapped[str] = mapped_column(String(32), default="") md5: Mapped[str] = mapped_column(String(32), default="")
real_type: Mapped[str] = mapped_column(String(256), default="") real_type: Mapped[str] = mapped_column(String(512), default="")
status: Mapped[str] = mapped_column(String(16), default="pending", index=True) status: Mapped[str] = mapped_column(String(16), default="pending", index=True)
mode: Mapped[str] = mapped_column(String(16), default="sync") mode: Mapped[str] = mapped_column(String(16), default="sync")
@@ -71,6 +71,11 @@ class ScanTask(Base):
back_populates="task", cascade="all, delete-orphan", lazy="selectin" back_populates="task", cascade="all, delete-orphan", lazy="selectin"
) )
__table_args__ = (
# 控制台高频查询: WHERE channel = X ORDER BY created_at DESC LIMIT N
Index("ix_scan_tasks_channel_created", "channel", "created_at", postgresql_using="btree"),
)
class ScanFinding(Base): class ScanFinding(Base):
__tablename__ = "scan_findings" __tablename__ = "scan_findings"

View File

@@ -115,13 +115,14 @@ class Policy:
"""YARA 规则权重映射:规则名 → weight(0-100)。 """YARA 规则权重映射:规则名 → weight(0-100)。
从 policy.yaml 的 yara_rule_weights.{critical/high/medium/low} 展开。 从 policy.yaml 的 yara_rule_weights.{critical/high/medium/low} 展开。
档位权重固定:critical=85 / high=65 / medium=45 / low=25。 档位权重固定:critical=85 / high=60 / medium=30 / low=10
(与 rules/*.yar 的 meta.risk_level 分档、binary.py 默认权重一致)。
""" """
weights = { weights = {
"critical": 85.0, "critical": 85.0,
"high": 65.0, "high": 60.0,
"medium": 45.0, "medium": 30.0,
"low": 25.0, "low": 10.0,
} }
result: Dict[str, float] = {} result: Dict[str, float] = {}
for tier, weight in weights.items(): for tier, weight in weights.items():
@@ -300,15 +301,18 @@ class PolicyStore:
def _load_db_or_yaml(self) -> Policy: def _load_db_or_yaml(self) -> Policy:
"""优先 DB,回退 YAML。""" """优先 DB,回退 YAML。"""
try: try:
from sqlalchemy import select
from app.db import SessionLocal from app.db import SessionLocal
from app.models import PolicyVersion from app.models import PolicyVersion
db = SessionLocal() db = SessionLocal()
try: try:
pv = db.execute( # 用 ORM Query API — 直接返回 PolicyVersion 对象
select(PolicyVersion).order_by(PolicyVersion.created_at.desc()) # 而不是 Row 对象(db.execute(select) 会返回 Row)
).scalar_one_or_none() pv = (
db.query(PolicyVersion)
.order_by(PolicyVersion.created_at.desc())
.first() # 取最新一条,不管历史版本多少条
)
if pv: if pv:
policy = Policy( policy = Policy(
version=pv.version, version=pv.version,
@@ -411,6 +415,8 @@ class PolicyEngine:
5) 规则覆盖 5) 规则覆盖
6) 权重累加评分 6) 权重累加评分
7) 阈值裁决 7) 阈值裁决
8) 复核区间上界校正
9) 复核地板(步骤 5 声明 manual_review 的规则,只抬处置不抬分数)
""" """
import logging import logging
@@ -563,14 +569,15 @@ class PolicyEngine:
) )
# 5) 规则覆盖优先处理 # 5) 规则覆盖优先处理
override_findings: List[Finding] = [] # action=reject 立即拒绝;action=manual_review 只登记"复核地板",
# 不往分数里注 65 —— 地板在步骤 9 生效。
review_floor = False
for finding in findings: for finding in findings:
override = policy.rule_overrides.get(finding.rule) override = policy.rule_overrides.get(finding.rule)
if not override: if not override:
continue continue
action = override.get("action", "manual_review") action = override.get("action", "manual_review")
reason = override.get("reason") or f"规则 {finding.rule} 命中策略" reason = override.get("reason") or f"规则 {finding.rule} 命中策略"
override_findings.append(finding)
triggered.append( triggered.append(
{ {
"type": "rule_override", "type": "rule_override",
@@ -590,12 +597,10 @@ class PolicyEngine:
triggered=triggered, triggered=triggered,
) )
if action == "manual_review": if action == "manual_review":
score = max(score, 65.0) review_floor = True
# 6) 其余信号按权重累加 # 6) 全部信号按权重累加
for finding in findings: for finding in findings:
if finding in override_findings:
continue
score += finding.weight score += finding.weight
if finding.severity in {Severity.HIGH.value, Severity.CRITICAL.value}: if finding.severity in {Severity.HIGH.value, Severity.CRITICAL.value}:
reasons.append(f"{finding.engine}:{finding.detail}") reasons.append(f"{finding.engine}:{finding.detail}")
@@ -619,6 +624,12 @@ class PolicyEngine:
if score >= policy.block_score: if score >= policy.block_score:
verdict, action = "malicious", "reject" verdict, action = "malicious", "reject"
# 9) 复核地板:策略里声明 manual_review 的规则保证"不低于人工复核",
# 与权重数值和 review_score 阈值解耦——调高 review_score 下界或调低
# 这些规则的权重,都不会让它们覆盖的文件类型静默放行。
if review_floor and action in ("pass", "pass_with_flag"):
verdict, action = "suspicious", "manual_review"
reason = self._build_reason(verdict, reasons, score) reason = self._build_reason(verdict, reasons, score)
return Decision( return Decision(
verdict=verdict, verdict=verdict,
@@ -750,7 +761,9 @@ class PolicyEngine:
if any("url" in f.rule.lower() or "link" in f.rule.lower() or "qr" in f.rule.lower() if any("url" in f.rule.lower() or "link" in f.rule.lower() or "qr" in f.rule.lower()
for f in findings): for f in findings):
factors.append("external_url") factors.append("external_url")
if ext in {".exe", ".dll", ".elf", ".so", ".desktop", ".appimage", ".run", ".bin", ".deb", ".rpm"}: if ext in {".exe", ".dll", ".elf", ".so", ".desktop", ".appimage", ".run", ".bin", ".deb", ".rpm",
".msi", ".msp", ".com", ".pif", ".scr", ".cpl", ".jar", ".war",
".vbs", ".vbe", ".js", ".jse", ".ps1", ".bat", ".cmd", ".hta", ".lnk", ".cab", ".appx"}:
factors.append("contains_executable") factors.append("contains_executable")
if ext == ".iso": if ext == ".iso":
factors.append("is_mountable_image") factors.append("is_mountable_image")
@@ -771,6 +784,14 @@ class PolicyEngine:
advice = "文件结构正常,无特别提示" advice = "文件结构正常,无特别提示"
requires_macro = False requires_macro = False
# 裁决已经是恶意/引擎不可判定时,"打开风险低"是自相矛盾的对外文案:
# 摆渡系统会把这段展示给用户,等于告诉用户"被拦下的文件可以放心打开"。
if verdict in {"malicious", "error", "reject"} and level != "critical":
level = "critical"
factors.append("verdict_rejected")
advice = "该文件已被判定为恶意或不可判定,请勿打开,联系管理员处理"
requires_macro = False
return { return {
"level": level, "level": level,
"factors": factors, "factors": factors,

View File

@@ -34,6 +34,17 @@ _settings = get_settings()
TEMP_ROOT = os.environ.get("MESHSAFE_TEMP_ROOT", "/tmp/meshsafe") TEMP_ROOT = os.environ.get("MESHSAFE_TEMP_ROOT", "/tmp/meshsafe")
def _clip(value: Any, limit: int) -> str:
"""按列宽截断字符串。
外部可控的长度(libmagic 描述串、归档成员路径、用户文件名)一旦超出列宽,
PostgreSQL 会拒绝整条 INSERT,连带同事务里的 scan_findings 一起丢失——
被拒绝的文件反而查不到记录。宁可截断,也不能整条不落库。
"""
text = str(value or "")
return text[:limit]
@dataclass @dataclass
class ScanContext: class ScanContext:
"""一次扫描任务的上下文。""" """一次扫描任务的上下文。"""
@@ -95,12 +106,12 @@ class ScanOutcome:
id=self.task_id, id=self.task_id,
channel=self.channel, channel=self.channel,
client_id=self.client_id, client_id=self.client_id,
filename=self.file_info.get("filename", ""), filename=_clip(self.file_info.get("filename", ""), 512),
declared_ext=self.file_info.get("declared_ext", ""), declared_ext=_clip(self.file_info.get("declared_ext", ""), 32),
size=self.file_info.get("size", 0), size=self.file_info.get("size", 0),
sha256=self.file_info.get("sha256", ""), sha256=self.file_info.get("sha256", ""),
md5=self.file_info.get("md5", ""), md5=self.file_info.get("md5", ""),
real_type=self.file_info.get("real_type", ""), real_type=_clip(self.file_info.get("real_type", ""), 512),
status="completed", status="completed",
mode=self.mode, mode=self.mode,
verdict=self.verdict, verdict=self.verdict,
@@ -122,7 +133,7 @@ class ScanOutcome:
severity=f.severity, severity=f.severity,
rule=f.rule, rule=f.rule,
detail=f.detail, detail=f.detail,
extracted_path=f.extracted_path, extracted_path=_clip(f.extracted_path, 1024),
weight=f.weight, weight=f.weight,
)) ))

View File

@@ -172,10 +172,10 @@
<h3>权重分档说明(来自 policy.yaml → yara_rule_weights)</h3> <h3>权重分档说明(来自 policy.yaml → yara_rule_weights)</h3>
<div style="font-size:13px;color:#555;line-height:1.8;"> <div style="font-size:13px;color:#555;line-height:1.8;">
<span class="chip chip-zt">critical → weight=85</span> <span class="chip chip-zt">critical → weight=85</span>
<span class="chip chip-warn">high → weight=65</span> <span class="chip chip-warn">high → weight=60</span>
<span class="chip chip-info">medium → weight=45</span> <span class="chip chip-info">medium → weight=30</span>
<span class="chip chip-ok">low → weight=25</span> <span class="chip chip-ok">low → weight=10</span>
<span style="color:#888;margin-left:12px;">未配置的规则默认 weight=50</span> <span style="color:#888;margin-left:12px;">未列出的规则沿用 rules/*.yar 中 meta.risk_level 推导出的同档权重</span>
</div> </div>
</div> </div>
</div> </div>

View File

@@ -67,8 +67,6 @@ zero_tolerance_rules:
- ReverseShellPowershell # PowerShell 反弹 Shell (-enc) - ReverseShellPowershell # PowerShell 反弹 Shell (-enc)
- CredentialStealer # 凭据窃取工具 (mimikatz/sekurlsa/wce/...) - CredentialStealer # 凭据窃取工具 (mimikatz/sekurlsa/wce/...)
- CredentialExport # 凭据导出命令 (reg save/reg export/...) - CredentialExport # 凭据导出命令 (reg save/reg export/...)
- RemoteDownloadUnix # Unix 远程下载执行 (curl|wget|certutil)
- RemoteDownloadPowerShell # PowerShell 远程下载执行 (IWR/DownloadString)
# ---------- 文件类型白名单 ---------- # ---------- 文件类型白名单 ----------
# 只允许这些类型通过。空列表表示不限制(不推荐生产使用)。 # 只允许这些类型通过。空列表表示不限制(不推荐生产使用)。
@@ -217,6 +215,12 @@ sandbox_required_extensions:
# rule_name: [action, reason_override] # rule_name: [action, reason_override]
# action 取值:pass / pass_with_flag / manual_review / reject / reject_and_notify # action 取值:pass / pass_with_flag / manual_review / reject / reject_and_notify
rule_overrides: rule_overrides:
# 光闸口径:EXE/DLL/脚本一律不自动放行,最低人工复核。
# 用 override 而不是靠 weight=30 恰好等于 review_score 下界来维持——
# 数值巧合意味着把下界调到 >35 或把权重降一点,干净可执行文件就会静默放行。
DangerousExecutableExtension:
action: manual_review
reason: "可执行/脚本类型文件,按光闸口径最低转人工复核"
VBAShellInvocation: VBAShellInvocation:
action: manual_review action: manual_review
reason: "宏调用 Shell 但无自动执行入口,需人工复核" reason: "宏调用 Shell 但无自动执行入口,需人工复核"
@@ -255,58 +259,60 @@ notification:
max_reason_length: 500 max_reason_length: 500
# ---------- YARA 规则权重映射 ---------- # ---------- YARA 规则权重映射 ----------
# 按规则名精细设定权重(0-100),覆盖 binary.py 里的默认 weight=50 # 档位权重与 rules/*.yar 的 meta.risk_level 一一对应(由 binary.py 读取 meta 得到
# 命中但不在下列名单中的规则 weight 保持默认值 50 # 默认严重度,再用本表覆盖为固定权重,保证规则文件与策略只有一处真相)。
# 严重度分档由规则内 meta.risk_level 决定(critical/high/medium/low) # critical=85(单独命中即 reject)/ high=60 / medium=30 / low=10
# 只有 critical 一档可以独立越过 block_score=70;其余档位需与其他信号叠加。
# 不在本表的规则:沿用 meta.risk_level 推导出的默认权重。
yara_rule_weights: yara_rule_weights:
critical: # weight=85 critical: # weight=85
- C3_PHP_Webshell_Oneliner
- C3_JSP_Webshell_RuntimeExec
- C3_Webshell_Behinder
- C1_PE_ProcessInjection - C1_PE_ProcessInjection
- C8_Ransomware_Encryption - C1_PE_RAT_CommandControl
- C8_Ransomware_LockBit - C3_JSP_Webshell_Deserialization
- C8_Ransomware_BlackCat - C3_JSP_Webshell_RuntimeExec
- C8_Ransomware_Conti - C3_PHP_Webshell_Oneliner
- C3_Webshell_Behinder
- C3_Webshell_CommandExecutionEcho
- C3_Webshell_Godzilla
- C7_Linux_ReverseShell_Bash - C7_Linux_ReverseShell_Bash
- C7_Linux_ReverseShell_NC - C7_Linux_ReverseShell_NC
- C7_Linux_Rootkit_KernelModule - C7_Linux_Rootkit_KernelModule
high: # weight=65 - C8_Ransomware_BlackCat
- C8_Ransomware_Conti
- C8_Ransomware_Encryption
- C8_Ransomware_LockBit
high: # weight=60
- C1_PE_DLLHijacking
- C1_PE_Downloader_Loader
- C1_PE_LOLBin_Usage
- C3_JSP_Webshell_MemoryShell
- C3_PHP_Webshell_Obfuscated - C3_PHP_Webshell_Obfuscated
- C3_PHP_Webshell_VariableDeformation - C3_PHP_Webshell_VariableDeformation
- C3_Webshell_CommandExecutionEcho - C3_Webshell_ASP_Classic
- C3_Webshell_Godzilla - C3_Webshell_ASPNET
- C1_PE_Downloader_Loader - C3_Webshell_AntSword
- C1_PE_RAT_CommandControl
- C1_PE_LOLBin_Usage
- C1_PE_Mutex_Persistence
- C7_Linux_ReverseShell_Python
- C7_Linux_ReverseShell_Perl
- C7_Linux_ReverseShell_Socat
- C7_Linux_Persistence_Crontab
- C7_Linux_Persistence_SystemD
- C7_Linux_Persistence_AuthorizedKeys - C7_Linux_Persistence_AuthorizedKeys
- C7_Linux_Persistence_Crontab
- C7_Linux_Persistence_LD_PRELOAD - C7_Linux_Persistence_LD_PRELOAD
- C8_Ransomware_VolumeShadow - C7_Linux_Persistence_SystemD
- C8_Ransomware_Note_Filename - C7_Linux_ReverseShell_Perl
- C7_Linux_ReverseShell_Python
- C7_Linux_ReverseShell_Socat
- C8_Mining_Monero
- C8_Ransomware_Ryuk_Pseudo - C8_Ransomware_Ryuk_Pseudo
medium: # weight=45 - C8_Ransomware_VolumeShadow
- C9_Packer_UPX medium: # weight=30
- C9_Packer_Themida_VMP
- C1_PE_DLLHijacking
- C1_PE_EntropyHighNetwork - C1_PE_EntropyHighNetwork
- C1_PE_Mutex_Persistence
- C7_Linux_Miner_Generic - C7_Linux_Miner_Generic
- C7_Linux_ReverseShell_Misc
- C7_Misc_Ruby_Lua_ReverseShell - C7_Misc_Ruby_Lua_ReverseShell
- C7_Network_Tools_Malicious - C7_Network_Tools_Malicious
- C3_Webshell_AntSword - C8_Ransomware_Note_Filename
- C3_Webshell_ASPNET - C9_Packer_Themida_VMP
- C3_Webshell_ASP_Classic low: # weight=10
- C3_JSP_Webshell_Deserialization
- C3_JSP_Webshell_MemoryShell
- C3_Webshell_CommandExecutionEcho
- C8_Ransomware_Extension_Blacklist - C8_Ransomware_Extension_Blacklist
low: # weight=25 - C9_Packer_UPX
- C7_Linux_ReverseShell_Misc
# ---------- 内网特征词表引擎(P1)---------- # ---------- 内网特征词表引擎(P1)----------
@@ -339,25 +345,55 @@ keyword_rules:
# 恶意行为特征词表(覆盖 CSV L3 全部 25 条用例) # 恶意行为特征词表(覆盖 CSV L3 全部 25 条用例)
# ================================================================ # ================================================================
# ---- 远程下载执行 ---- # ---- 远程下载即执行(要有完整动作链,裸词不算;README 里的安装命令进复核而非直接拒)----
# cross_line:命令换行/续行是本类特征最常见的绕过(curl \ + 下一行 | sh),
# 拼接后仍要求"下载工具 + 执行动作"同框,散文里的多行文本不会凑出这条链。
- name: RemoteDownloadExec
match: regex
pattern: "(?:\\b(?:curl|wget)\\b[^\\n]{0,120}\\|\\s*(?:ba|z|k|da)?sh\\b|\\b(?:curl|wget)\\b[^\\n]{0,120}(?:&&|;)\\s*(?:chmod\\s+\\+x|\\./)|(?:^|[^\\w$-])(?:iex|invoke-expression)\\b[^\\n]{0,60}(?:\\biwr\\b|invoke\\s*[-_]?\\s*web\\s*[-_]?\\s*request|invoke\\s*[-_]?\\s*rest\\s*[-_]?\\s*method|downloadstring)|\\b(?:iwr|invoke\\s*[-_]?\\s*web\\s*[-_]?\\s*request|invoke\\s*[-_]?\\s*rest\\s*[-_]?\\s*method)\\b[^\\n]{0,120}\\|\\s*(?:iex|invoke-expression)\\b|certutil\\s+[-/]urlcache[^\\n]{0,60}\\.exe\\b|mkfifo[^\\n]{0,60}(?:\\bnc\\s|\\bnetcat\\b))"
ignore_case: true
cross_line: true
weight: 40
severity: critical
# ---- 下载并落地成可执行文件(dropper 行为,比"下载即执行"弱一档但仍高危)----
- name: PayloadDropToDisk
match: regex
pattern: "(?:\\b(?:curl|wget)\\b[^\\n]{0,120}(?:-o|-O|--output)\\s+\\S*\\.(?:exe|dll|ps1|bat|cmd|vbs|jar|msi|so|bin)\\b|\\b(?:i\\s*[-_]?\\s*wr|invoke\\s*[-_]?\\s*web\\s*[-_]?\\s*request|invoke\\s*[-_]?\\s*rest\\s*[-_]?\\s*method)\\b[^\\n]{0,120}-o(?:utfile)?\\s+\\S*\\.(?:exe|dll|ps1|bat|cmd|vbs)\\b|\\b(?:new-object|activexobject)\\s+system\\.net\\.webclient[^\\n]{0,80}downloadfile)"
ignore_case: true
cross_line: true
weight: 45
severity: critical
# ---- 外部下载动作(cmdlet/API + 无执行链):计分进复核,不再零容忍 ----
# cmdlet 名(Invoke-WebRequest/Invoke-RestMethod/BitsAdmin)是 PowerShell 专有写法,
# 单独出现即进复核。.NET/Win32 的方法名(DownloadString/DownloadFile)不一样——
# 它们存在于所有托管程序的标识符表里,裸匹配实测把 nvidiaInspector.exe 的
# `add_DownloadFileCompleted`、`DownloadFileAsync` 各算一次命中,白送 30 分,
# 把 45 分的人工复核顶成 75 分拒绝。所以方法名必须与 URL/IP 同框才算下载。
- name: RemoteDownloadPowerShell - name: RemoteDownloadPowerShell
match: regex match: regex
pattern: "(Invoke-WebRequest|Invoke-RestMethod|DownloadString|DownloadFile|Start-BitsTransfer|BitsAdmin)" pattern: "(?:\\b(?:Invoke-WebRequest|Invoke-RestMethod|Start-BitsTransfer|BitsAdmin)\\b|\\bDownload(?:String|File)\\b[^\\n]{0,40}(?:https?://|\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}\\.\\d{1,3})|(?:https?://|\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}\\.\\d{1,3})[^\\n]{0,40}\\bDownload(?:String|File)\\b)"
ignore_case: true
weight: 35
severity: high
- name: RemoteDownloadUnix
match: regex
pattern: "(\\bwget\\b|\\bcurl\\b|\\bcertutil\\s+-urlcache\\s+-split|\\bfetch\\b)"
ignore_case: true ignore_case: true
weight: 30 weight: 30
severity: high severity: medium
# ---- 下载工具 + 明确地址:只认"工具与 URL/IP 同框",裸 -o 参数不算 ----
# 旧写法里的 `curl|wget -O/-o 任意值` 会命中任何写成两段的安装文档,去掉。
- name: RemoteDownloadUnix
match: regex
pattern: "(?:\\b(?:curl|wget|certutil)\\b[^\\n]{0,80}(?:https?://|\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}))"
ignore_case: true
weight: 30
severity: medium
# ---- 反弹 Shell ---- # ---- 反弹 Shell ----
- name: ReverseShellNetcat - name: ReverseShellNetcat
match: regex match: regex
pattern: "(\\bnc\\b|\\bncat\\b|\\bnetcat\\b|/dev/tcp/|bash\\s+-i\\s+>&|sh\\s+-i\\s+>&)" # 只认「执行原语」形态:nc/ncat/netcat 带 -e/-c(把 shell 交给连接),
# 或 /dev/tcp、bash -i >& 这类内核级反连。裸 nc / nc -l / nc -z 是运维常用,
# 出现在速查文档和二进制字符串表里必然误报,不计分。
pattern: "(\\b(?:nc|ncat|netcat)\\s+(?:[^\\n]{0,60}\\s)?-[ec]\\w*\\s*(?:/bin/|/usr/|cmd\\.exe\\b|\\S*sh\\b)|/dev/tcp/|bash\\s+-i\\s+>&|sh\\s+-i\\s+>&)"
ignore_case: true ignore_case: true
weight: 40 weight: 40
severity: critical severity: critical
@@ -372,7 +408,8 @@ keyword_rules:
# ---- 凭据窃取工具 ---- # ---- 凭据窃取工具 ----
- name: CredentialStealer - name: CredentialStealer
match: regex match: regex
pattern: "(mimikatz|sekurlsa|wce|gsecdump|fgdump|lsadump|dcsync|dcsyncer|rubeus)" # 全部加 \b 词边界, wce 要求后跟 .exe / 空格 / / (二进制随机字节里的 WCE 不会触发)
pattern: "(\\bmimikatz\\b|\\bsekurlsa\\b|\\bwce(\\.exe\\b|\\s|\\/)|\\bgsecdump\\b|\\bfgdump\\b|\\blsadump\\b|\\bdcsync\\b|\\bdcsyncer\\b|\\brubeus\\b)"
ignore_case: true ignore_case: true
weight: 45 weight: 45
severity: critical severity: critical
@@ -428,6 +465,32 @@ keyword_whitelist:
- ".senmesh.com" - ".senmesh.com"
- "repo.internal" - "repo.internal"
- "mirror.internal" - "mirror.internal"
- rule: RemoteDownloadExec # 下载即执行:内网源站安装脚本按白名单放行
allowed_keywords:
- "192.168."
- "10."
- "172.16."
- "172.17."
- "172.18."
- "172.19."
- "172.2"
- "172.3"
- ".senmesh.com"
- "repo.internal"
- "mirror.internal"
- rule: PayloadDropToDisk # 下载落地可执行文件:内网源站安装包放行
allowed_keywords:
- "192.168."
- "10."
- "172.16."
- "172.17."
- "172.18."
- "172.19."
- "172.2"
- "172.3"
- ".senmesh.com"
- "repo.internal"
- "mirror.internal"
- rule: RemoteDownloadPowerShell # Invoke-WebRequest/BitsAdmin - rule: RemoteDownloadPowerShell # Invoke-WebRequest/BitsAdmin
allowed_keywords: allowed_keywords:
- "192.168." - "192.168."

View File

@@ -133,6 +133,38 @@ services:
start_period: 15s start_period: 15s
retries: 3 retries: 3
# ---------- Cron (定时任务: DB 归档 + VACUUM) ----------
# 纯 Python 实现, 零额外依赖, 和 api 共享 meshsafe:latest 镜像
# 任务:
# 每天 03:00 archive_old_tasks.py --apply (清 90 天 detail / 365 天归档 / 幂等 TTL)
# 每周一 03:10 VACUUM ANALYZE (维护表统计信息)
cron:
image: meshsafe:latest
container_name: meshsafe-cron
restart: unless-stopped
networks: [meshsafe_net]
depends_on:
postgres:
condition: service_healthy
environment:
- TZ=Asia/Shanghai
- MESHSAFE_ENV=prod
- MESHSAFE_DB_URL=postgresql+psycopg2://meshsafe:${MESHSAFE_DB_PASSWORD:-meshsafe}@postgres:5432/meshsafe
- MESHSAFE_MASTER_KEY=${MESHSAFE_MASTER_KEY:?MASTER_KEY_REQUIRED}
# cron 不做控制台认证,但 Settings.admin_password 是必填字段,缺它 get_settings()
# 就抛 ValidationError,归档与 VACUUM 全崩。给占位值,不把真实口令复制进 cron
# 容器环境(docker inspect 可见)。
- MESHSAFE_ADMIN_PASSWORD=cron-unused-placeholder
volumes:
- ./data/logs:/var/log/meshsafe # 归档日志共享目录
- ./scripts:/app/scripts:ro # archive_old_tasks.py + cron_entrypoint.py
entrypoint: ["python3", "/app/scripts/cron_entrypoint.py"]
# 镜像层的 HEALTHCHECK 探测 localhost:8443(API 服务端口),cron 容器不监听该端口,
# 因此状态恒为 unhealthy —— 真正调度器挂掉时这个信号完全失效。关掉它,
# 存活与否由 restart: unless-stopped + 日志判断。
healthcheck:
disable: true
# ---------- Nginx(反向代理,内网 HTTP 明文)---------- # ---------- Nginx(反向代理,内网 HTTP 明文)----------
# 外部端口 44380 → 内部 80 → api:8443 # 外部端口 44380 → 内部 80 → api:8443
# 无 TLS,外层防护靠光闸 # 无 TLS,外层防护靠光闸

View File

@@ -193,7 +193,7 @@ rule_overrides:
|---|---| |---|---|
| `pass` | 放行 | | `pass` | 放行 |
| `pass_with_flag` | 放行并标记 | | `pass_with_flag` | 放行并标记 |
| `manual_review` | 转人工复核(同时把评分提升至 65,确保落入复核区间) | | `manual_review` | 处置地板:该文件最终不低于人工复核(评分不变,低于复核区间时只抬处置) |
| `reject` | 拒绝 | | `reject` | 拒绝 |
| `reject_and_notify` | 拒绝并通知(当前实现等同 `reject`) | | `reject_and_notify` | 拒绝并通知(当前实现等同 `reject`) |
@@ -201,7 +201,9 @@ rule_overrides:
1. 命中 `rule_overrides` 的规则会**优先于评分**处理 1. 命中 `rule_overrides` 的规则会**优先于评分**处理
2. 若 `action` 为 `reject` 类 → 立即返回拒绝结论,不再累加评分 2. 若 `action` 为 `reject` 类 → 立即返回拒绝结论,不再累加评分
3. 若 `action` 为 `manual_review` → 评分提升至 65 并继续后续裁决 3. 若 `action` 为 `manual_review` → 登记"复核地板",规则自身权重照常累加,评分累加与阈值裁决完成后,处置若低于 `manual_review` 才抬到 `manual_review`(已判 `reject` 的不会被降回来)
> 旧实现是"评分提升至 65",有两个副作用:注进去的 65 分本身不是证据,再叠加任意 ≥5 分的噪声信号就会越过 `block_score=70` 变成拒绝;同时该规则自身的权重被跳过不再计分,控制台显示的分数与真实证据脱钩。现改为处置地板后,分数只反映权重,"不放行"由地板保证,与 `review_score` 阈值数值解耦——`DangerousExecutableExtension`(光闸口径:EXE/DLL/脚本不自动放行)正是靠这条地板生效,而不是靠 `weight=30/35` 恰好等于 `review_score` 下界。
**为什么用覆盖而不是纯评分**:像 `PathTraversal`、`DDEAutoExecution` 这类信号,一旦出现就足以判定不可放行,不应依赖其他信号累加来"凑够"阈值。 **为什么用覆盖而不是纯评分**:像 `PathTraversal`、`DDEAutoExecution` 这类信号,一旦出现就足以判定不可放行,不应依赖其他信号累加来"凑够"阈值。

View File

@@ -117,17 +117,23 @@ rule C7_Linux_ReverseShell_Misc {
meta: meta:
risk_level = "medium" risk_level = "medium"
category = "C7" category = "C7"
false_positive_rate = "medium" false_positive_rate = "low"
reference = "多种语言反弹 Shell 通用特征" reference = "多种语言反弹 Shell 通用特征"
description = "通用反弹 Shell 标记:/dev/tcp 或 /dev/udp 或 reverse shell 工具名" description = "通用反弹 Shell 标记:/dev/udp 或 攻击工具名 + shell 参数"
strings: strings:
$devudp = /\/dev\/udp\/\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\/\d{2,5}/ ascii $devudp = /\/dev\/udp\/\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\/\d{2,5}/ ascii
$bashc = /bash\s+-c\s+['"].*\/dev\/(tcp|udp)/ ascii $bashc = /bash\s+-c\s+['"].*\/dev\/(tcp|udp)/ ascii
$dash_e = /\/bin\/bash\s+-e\s+['"]/ ascii $dash_e = /\/bin\/bash\s+-e\s+['"]/ ascii
$hydra = /Hydra|crackmap|weevely|meterpreter|msfconsole|msfvenom/ ascii nocase // 裸词 Hydra 会命中 "dehydrated"、crack 会命中 "cracklib",必须带参数或成组出现
$hydra = /\bhydra\s+(-l|-P|-p|-s|-m|-t)\b/ ascii nocase
$tool = /\b(crackmapexec|weevely|meterpreter|msfconsole|msfvenom)\b/ ascii nocase
$reverseshell = /reverse[_\s]?shell|shell[_\s]?shock/ ascii nocase $reverseshell = /reverse[_\s]?shell|shell[_\s]?shock/ ascii nocase
$netart = /(nc|ncat)\s+\S+\s+\d{2,5}\s+-[ec]\b|\/dev\/(tcp|udp)\/|socat\s+EXEC:|bash\s+-i\s*>&/ ascii
condition: condition:
any of ($devudp, $bashc, $dash_e) or any of ($hydra, $reverseshell) any of ($devudp, $bashc, $dash_e)
or $hydra
or ($tool and $netart)
or ($reverseshell and $netart)
} }
@@ -177,40 +183,43 @@ rule C7_Linux_Persistence_SystemD {
meta: meta:
risk_level = "high" risk_level = "high"
category = "C7" category = "C7"
false_positive_rate = "medium" false_positive_rate = "low"
reference = "MITRE ATT&CK T1543.002 Systemd Service" reference = "MITRE ATT&CK T1543.002 Systemd Service"
description = "恶意 systemd unit 文件:ExecStart 指向 /tmp/.hidden + multi-user.target + 反弹命令" description = "恶意 systemd unit:ExecStart 指向临时目录隐藏文件 / 管道执行 / 反弹命令"
strings: strings:
$unit = /\[Unit\]/ ascii $unit = /\[Unit\]/ ascii
$service = /\[Service\]/ ascii $service = /\[Service\]/ ascii
$install = /\[Install\]/ ascii $install = /\[Install\]/ ascii
$execstart = /ExecStart=/ ascii $execstart = /ExecStart=/ ascii
$multiuser = /multi-user\.target/ ascii $multiuser = /multi-user\.target/ ascii
$hiddenpath = /\/tmp\/\.?\w+|\/var\/tmp\/\.?\w+/ ascii // 隐藏文件(点开头)落在可写临时目录——正常 unit 只会在 /usr/bin、/opt 等路径
$bashsh = /\/bin\/(ba)?sh\s+-c/ ascii $hiddenpath = /\/(tmp|var\/tmp|dev\/shm)\/\.[a-zA-Z0-9_-]{2,}/ ascii
$pipe = /(curl|wget)\s+\S+\s*\|\s*(ba)?sh/ ascii
$revshell = /((nc|ncat|socat)\s+.*(-e|--exec|EXEC:)\s*\/bin\/(ba)?sh|\/dev\/tcp\/\d{1,3}\.\d{1,3})/ ascii
condition: condition:
// 完整 unit 片段 + 可疑路径 ($unit and $service and $execstart and any of ($hiddenpath, $pipe, $revshell))
$unit and $service and $execstart and ($hiddenpath or $bashsh)
or or
// Install multi-user.target + ExecStart 反弹 ($install and $multiuser and $execstart and any of ($hiddenpath, $pipe, $revshell))
$install and $multiuser and any of ($execstart, $bashsh)
} }
rule C7_Linux_Persistence_LD_PRELOAD { rule C7_Linux_Persistence_LD_PRELOAD {
meta: meta:
risk_level = "high" risk_level = "high"
category = "C7" category = "C7"
false_positive_rate = "medium" false_positive_rate = "low"
reference = "MITRE ATT&CK T1574.006 LD_PRELOAD" reference = "MITRE ATT&CK T1574.006 LD_PRELOAD"
description = "LD_PRELOAD 劫持:环境变量注入 + .so 文件 + dlopen" description = "LD_PRELOAD 劫持:写 ld.so.preload 或预加载临时目录里的 .so"
strings: strings:
$ldpreload = /LD_PRELOAD=/ ascii $ldpreload = /LD_PRELOAD=/ ascii
$sofile = /\.so(\.\d+)?:?/ ascii
$dlopen = "dlopen" ascii
$dlsym = "dlsym" ascii
$exportenv = /export\s+LD_PRELOAD/ ascii $exportenv = /export\s+LD_PRELOAD/ ascii
$ldso = /ld\.so\.preload/ ascii
$tmpso = /\/(tmp|var\/tmp|dev\/shm|proc\/\d+\/root)\/[a-zA-Z0-9_.\/-]*\.so(\.[0-9]+)?/ ascii
condition: condition:
($ldpreload or $exportenv) and ($sofile or $dlopen or $dlsym) // /etc/ld.so.preload 是内核级后门常用落点,单独出现即算证据
$ldso
or
// LD_PRELOAD 指向可写临时目录的 so;README 里 LD_PRELOAD=libasan.so 不算
(any of ($ldpreload, $exportenv) and $tmpso)
} }
@@ -267,41 +276,55 @@ rule C7_Misc_Ruby_Lua_ReverseShell {
meta: meta:
risk_level = "medium" risk_level = "medium"
category = "C7" category = "C7"
false_positive_rate = "medium" false_positive_rate = "low"
reference = "多语言反弹 Shell" reference = "多语言反弹 Shell"
description = "Ruby/Lua 反弹 Shell:TCPSocket + exec/system" description = "Ruby/Lua 反弹 Shell:Socket 对象 + 紧邻的 shell 执行调用"
strings: strings:
$ruby = "TCPSocket" ascii $ruby = "TCPSocket" ascii
$ruby2 = "Socket.tcp" ascii
$lua = "loadstring" ascii $lua = "loadstring" ascii
$lua2 = "luasocket" ascii $lua2 = "luasocket" ascii
$exec = /(exec|system|IO\.popen)\s+.*\/bin\/(ba)?sh/ ascii $exec = /(\bexec|\bsystem|IO\.popen|\bpopen|os\.execute|os\.runtime)\s*\(?\s*['"]((\/bin\/)?(ba)?sh|nc\s+-e|\/dev\/tcp)[^'"]{0,48}/ ascii
condition: condition:
any of ($ruby, $lua, $lua2) and ($exec or any of ($ruby, $lua, $lua2)) // 只要出现 TCPSocket/loadstring 就命中 = 任何内嵌 Lua 的程序(Redis、nginx-lua、
// Wireshark 插件)必中。语言标记与 shell 执行必须挨在一起(256B 内)。
$exec and (
for any i in (1..#exec): (
any of ($ruby, $ruby2, $lua, $lua2) in (@exec[i] - 256..@exec[i] + 256)
)
)
} }
// ============================================================================= // =============================================================================
// C7-5 网络投毒 / MITM 工具(内网横向移动) // C7-5 网络投毒 / MITM 工具(内网横向移动)
// 工具名必须整词匹配,且与攻击参数成组出现
// ============================================================================= // =============================================================================
rule C7_Network_Tools_Malicious { rule C7_Network_Tools_Malicious {
meta: meta:
risk_level = "medium" risk_level = "medium"
category = "C7" category = "C7"
false_positive_rate = "medium" false_positive_rate = "low"
reference = "内网攻击工具" reference = "内网攻击工具"
description = "内网攻击工具:ettercap/responder/nmap 扫描 + SMB/DNS 投毒" description = "内网攻击工具:ettercap/responder/nmap + 投毒参数成组出现"
strings: strings:
$ettercap = "ettercap" ascii nocase $tool1 = /\bettercap\b/ ascii nocase
$responder = /responder(\.py)?/ ascii nocase $tool2 = /\bresponder(\.py)?\b/ ascii nocase
$nmap = "nmap" ascii nocase $tool3 = /\bnmap\b/ ascii nocase
$smb = /--smb/ ascii // 攻击参数 / 漏洞编号(这些才是意图证据)
$ms17 = "ms17-010" ascii nocase $art1 = /--smb/ ascii
$smbghost = "SMBGhost" ascii nocase $art2 = /\bms17-010\b/ ascii nocase
$dnsspoof = /dns[-_]spoof|dns[-_]mitm/ ascii nocase $art3 = /\bSMBGhost\b/ ascii nocase
$arppoison = /arp[-_]poison|arp[-_]spoof/ ascii nocase $art4 = /\bdns[-_]spoof\b|\bdns[-_]mitm\b/ ascii nocase
$art5 = /\barp[-_]poison\b|\barp[-_]spoof\b/ ascii nocase
$art6 = /--Mitm|-T\s+text|\bec_sleep\b|\bntlm_relay\b|\bRunFinger\.py\b/ ascii nocase
condition: condition:
any of ($ettercap, $responder, $dnsspoof, $arppoison) // 投毒手法名本身就是明确证据
any of ($art4, $art5)
or or
$nmap and ($smb or $ms17 or $smbghost) // 工具名 + 攻击参数
($tool1 and any of ($art1, $art2, $art3, $art6))
or ($tool2 and any of ($art1, $art2, $art3, $art6))
or ($tool3 and any of ($art1, $art2, $art3))
} }

View File

@@ -2,26 +2,34 @@
// MeshSafe L2B YARA 规则集 —— Windows PE 恶意程序专项(C1 + C8) // MeshSafe L2B YARA 规则集 —— Windows PE 恶意程序专项(C1 + C8)
// ============================================================================= // =============================================================================
// //
// 设计原则: // 设计原则(v0.2.4 起,为消除干净软件误报而重写):
// - 所有规则使用 2-3 特征组合命中(API 组合 + C2 URL + Mutex 持久化等) // 1. 全文搜 API 名 = 必然误报。系统 DLL 会「导出」这些 API(kernel32 里就有
// - binary.py 的 PE 结构分析负责:节熵 + 签名 + 可疑导入(单 API) // VirtualAllocEx/WriteProcessMemory/CreateRemoteThread 的明文字符串),
// - YARA 负责:多 API 组合 + 字符串组合 + 壳 + 勒索专项 // 安全 SDK、安装器、调试组件同理。因此 C1 系列一律要求:
// - 命中后作为 finding 交给 PolicyEngine 权重累加,不直接零容忍 // - 文件是 PE(uint16(0) == 0x5A4D),且
// - 相关 API 名出现在同一个窗口内(导入名表是连续的,窗口 256~512B)。
// 导出表按字母序排列、跨度通常上 KB,落在窗口外 → 不误报;
// 真注入器/下载器的导入名表三条名字通常挨在一起 → 命中。
// 2. 不单独用「API 组合」定性,必须有硬指标(可疑路径 / 持久化键名 / C2 URL)。
// 3. 不用 3~4 字母裸串(曾经的 $persist5 = "RUN")和裸工具名。
// 4. binary.py 的 PE 结构分析负责节熵/签名/导入 API 分级;YARA 只做组合特征。
// //
// 范围: // 范围:
// - C1: Windows PE 恶意程序(注入器 / 下载器 / RAT / Loader) // - C1: Windows PE 恶意程序(注入器 / 下载器 / RAT / Loader)
// - C8: 勒索 / 挖矿专项 // - C8: 勒索 / 挖矿专项(勒索样本常以文本/脚本形态投递,不加 PE 头判定)
// - C9: 壳 / 签名 / 结构异常(binary.py 已覆盖大部分,YARA 补充壳特征) // - C9: 壳特征(识别后建议送沙箱,不作为恶意证据)
// //
// 维护: // 维护:
// - 新恶意家族出现时,优先补充"API 组合 + Mutex 名 + C2 URL"三件套 // - 新增规则先跑「干净 PE 回归」:C:/Windows/System32 全量 900+ 文件必须 0 命中
// - 壳特征(UPX/Themida/VMP)可识别后自动送沙箱深检 // - meta.risk_level 会被 binary.py 读取用于严重度展示,必须与档位权重一致
// ============================================================================= // =============================================================================
import "pe"
// ============================================================================= // =============================================================================
// C1-1 进程注入三件套(VirtualAllocEx + WriteProcessMemory + CreateRemoteThread) // C1-1 进程注入三件套(VirtualAllocEx + WriteProcessMemory + CreateRemoteThread)
// binary.py 扫了 SuspiciousImports(单个 API),但 3 个同时出现 = 必然是注入器 // 判定要点:三个导入名必须在同一张导入名表里(512B 窗口),否则是导出表明文
// ============================================================================= // =============================================================================
rule C1_PE_ProcessInjection { rule C1_PE_ProcessInjection {
@@ -30,37 +38,66 @@ rule C1_PE_ProcessInjection {
category = "C1" category = "C1"
false_positive_rate = "low" false_positive_rate = "low"
reference = "MITRE ATT&CK T1055 Process Injection" reference = "MITRE ATT&CK T1055 Process Injection"
description = "PE 文件同时导入 进程注入三件套 API = 远控/注入器特征" description = "PE 导入表含进程注入三件套(同表紧邻)= 远控/注入器特征"
strings: strings:
$s1 = "VirtualAllocEx" ascii $alloc = "VirtualAllocEx" ascii
$s2 = "WriteProcessMemory" ascii $write = "WriteProcessMemory" ascii
$s3 = "CreateRemoteThread" ascii $thread = "CreateRemoteThread" ascii
// 备选:Nt 系列(NtUnmapViewOfSection + VirtualAllocEx + CreateRemoteThread) // 进程镂空(Process Hollowing)
$n1 = "NtUnmapViewOfSection" ascii $unmap1 = "NtUnmapViewOfSection" ascii
$n2 = "NtMapViewOfSection" ascii $unmap2 = "ZwUnmapViewOfSection" ascii
$n3 = "ZwUnmapViewOfSection" ascii // APC / Hook 注入
// 也可能是 QueueUserAPC 注 APC 注入
$apc = "QueueUserAPC" ascii $apc = "QueueUserAPC" ascii
// 或 SetWindowsHookEx DLL 注入
$hook = "SetWindowsHookEx" ascii $hook = "SetWindowsHookEx" ascii
$open = "OpenProcess" ascii
condition: condition:
// 经典三件套(必须同时命中) uint16(0) == 0x5A4D
$s1 and $s2 and $s3 and
or (
// Nt 系列内存注入(至少 2 个 Nt + 1 个注入) // 经典三件套:同表紧邻
(any of ($n1, $n2, $n3) and any of ($s1, $s3, $apc)) (
or $alloc and $write and $thread
// APC 注入 + VirtualAllocEx and for any i in (1..#alloc): (
$apc and $s1 $write in (@alloc[i] - 512..@alloc[i] + 512)
or and $thread in (@alloc[i] - 512..@alloc[i] + 512)
// Hook DLL 注入 )
$hook and $s1 )
or
// 镂空注入:unmap 必须和 write+thread 同表(kernel32/shell32 只是从
// ntdll 导入成对的 NtMapView/NtUnmap,没有写入+远线程就不算镂空注入)
(
$write and $thread
and for any i in (1..#unmap1): (
$write in (@unmap1[i] - 512..@unmap1[i] + 512)
and $thread in (@unmap1[i] - 512..@unmap1[i] + 512)
)
)
or
(
$write and $thread
and for any i in (1..#unmap2): (
$write in (@unmap2[i] - 512..@unmap2[i] + 512)
and $thread in (@unmap2[i] - 512..@unmap2[i] + 512)
)
)
or
// APC / Hook 注入:目标进程 API + 注入 API 同表紧邻
(
$alloc and $open and ($apc or $hook)
and for any i in (1..#alloc): (
$open in (@alloc[i] - 512..@alloc[i] + 512)
and ($apc in (@alloc[i] - 512..@alloc[i] + 512)
or $hook in (@alloc[i] - 512..@alloc[i] + 512))
)
)
)
} }
// ============================================================================= // =============================================================================
// C1-2 下载器组合(网络 API + 本地持久化 / 执行) // C1-2 下载器 / Loader(下载 API + 落地持久化键名,且两者同表紧邻)
// 单一 URLDownloadToFile 不致命,但 "下载 + 执行 + 持久化" 三件套 = Loader // v0.2.3 的写法:任意下载 API + "WriteFile" + "CreateProcess" + 裸串 "RUN"
// → 任何带自动更新的正常软件必中。现在要求持久化硬指标。
// ============================================================================= // =============================================================================
rule C1_PE_Downloader_Loader { rule C1_PE_Downloader_Loader {
@@ -69,157 +106,172 @@ rule C1_PE_Downloader_Loader {
category = "C1" category = "C1"
false_positive_rate = "low" false_positive_rate = "low"
reference = "MITRE ATT&CK T1105 Ingress Tool Transfer" reference = "MITRE ATT&CK T1105 Ingress Tool Transfer"
description = "下载器组合:HTTP/FTP 下载 + 写入到可疑路径 + 执行" description = "下载 API 与自启动/服务注册持久化键名同表共存 = Loader"
strings: strings:
// 网络下载 API(任选其一) // 下载类 API(保留专有名词,不用裸串)
$net1 = "URLDownloadToFile" ascii $net1 = "URLDownloadToFile" ascii
$net2 = "InternetOpenUrl" ascii $net2 = "InternetOpenUrl" ascii
$net3 = "WinHttpOpen" ascii $net3 = "WinHttpOpenRequest" ascii
$net4 = "WinHttpOpenRequest" ascii $net4 = "WinHttpSendRequest" ascii
$net5 = "FtpOpenFile" ascii $net5 = "URLDownloadToCacheFile" ascii
$net6 = "DownloadFile" ascii $net6 = "FtpOpenFile" ascii
$net7 = "WinHttpSendRequest" ascii $net7 = "WinHttpOpen" ascii
$net8 = "InternetOpen" ascii
// 写入/保存到本地 // 自启动 / 服务注册持久化硬指标(不是裸 "RUN")
$save1 = "WriteFile" ascii $p1 = "Software\\Microsoft\\Windows\\CurrentVersion\\Run" ascii nocase
$save2 = "SaveToFile" ascii $p2 = "CurrentVersion\\RunOnce" ascii nocase
$save3 = "URLDownloadToCacheFile" ascii $p3 = "schtasks.exe" ascii nocase
$p4 = "CreateService" ascii
$p5 = "Start Menu\\Programs\\StartUp" ascii nocase
$p6 = "reg add" ascii nocase
$p7 = "StartupApproved" ascii nocase
// 执行下载的载荷 // 临时目录落地(Loader 常先落 %TEMP%)
$exec1 = "WinExec" ascii $tmp1 = /\\AppData\\Local\\Temp\\[a-z0-9_. -]{3,60}\.(exe|dll|tmp|dat)/ ascii nocase
$exec2 = "CreateProcess" ascii $tmp2 = /%TEMP%\\[a-z0-9_. -]{3,60}\.(exe|dll|tmp)/ ascii nocase
$exec3 = "ShellExecute" ascii $tmp3 = /\\Windows\\Temp\\[a-z0-9_. -]{3,60}\.(exe|dll|tmp)/ ascii nocase
$exec4 = "CreateRemoteThread" ascii
// 持久化(注册表/启动项)
$persist1 = "RegSetValueEx" ascii
$persist2 = "SetServiceObjectSecurity" ascii
$persist3 = "CreateService" ascii
$persist4 = "schtasks.exe" ascii
$persist5 = "RUN" ascii // HKCU\Software\Microsoft\Windows\CurrentVersion\Run
condition: condition:
// 下载 + 保存 + 执行(三件套) uint16(0) == 0x5A4D
any of ($net*) and any of ($save*) and any of ($exec*) and any of ($net*)
or and any of ($p*)
// 下载 + 持久化(即使没执行也是准备投毒) and
any of ($net*) and any of ($persist*) (
or // 下载 API 与持久化键名同处一个导入/字符串表块
// 执行 + 持久化(下载可能在运行时动态完成) for any i in (1..#net1): ( any of ($p*) in (@net1[i] - 1024..@net1[i] + 1024) )
any of ($exec1, $exec2, $exec3) and any of ($persist*) or (any of ($tmp*) and 2 of ($p*))
)
} }
// ============================================================================= // =============================================================================
// C1-3 RAT / 远控特征(C2 通信 + 命令执行 + 持久化) // C1-3 RAT / 远控特征
// v0.2.3 的写法用 send( / recv( / system( 这类通用符号,且
// (bot|serv|gate|ops)\.(com|net) 会匹配任意域名字符串 → 正常联网程序必中。
// 现在要求:C2 硬指标(IP:PORT、可执行下载 URL、家族标记)+ 套接字 API 同表。
// ============================================================================= // =============================================================================
rule C1_PE_RAT_CommandControl { rule C1_PE_RAT_CommandControl {
meta: meta:
risk_level = "critical" risk_level = "critical"
category = "C1" category = "C1"
false_positive_rate = "medium" false_positive_rate = "low"
reference = "MITRE ATT&CK T1071 Application Layer Protocol" reference = "MITRE ATT&CK T1071 Application Layer Protocol"
description = "RAT C2 通信特征:WinHTTP/WSASend + 命令执行 API + 可疑 URL/域名" description = "RAT C2 特征:套接字 API + 明确 C2 指标(IP:端口 / .exe 下载 / 家族标记)"
strings: strings:
// C2 网络通道 // 套接字 API(不用裸 send/recv/connect)
$c2_send = /WSASend|send\s*\(|sendto|WinHttpSendRequest|InternetWriteFile/ ascii $sock1 = "WSASend" ascii
$c2_recv = /WSARecv|recv\s*\(|recvfrom|WinHttpReadData|InternetReadFile/ ascii $sock2 = "WSARecv" ascii
$c2_connect = /WSAConnect|connect\s*\(|WinHttpConnect|InternetConnect/ ascii $sock3 = "WSAConnect" ascii
$sock4 = "WSASocket" ascii
$sock5 = "inet_ntoa" ascii
$sock6 = "getpeername" ascii
// 命令执行 // 命令执行(真实 RAT 才会与 C2 同表出现)
$cmd = /CreateProcess|ShellExecute|WinExec|system\s*\(|popen/ ascii $cmd1 = "CreateProcess" ascii
$cmd2 = "ShellExecute" ascii
$cmd3 = "WinExec" ascii
// 可疑 C2 URL(可执行文件里出现 http://...exe 或常见 C2 域名) // C2 硬指标
$c2url1 = /http:\/\/\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}:\d{2,5}/ ascii $c2ip = /https?:\/\/\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}:[1-5][0-9]{3}/ ascii
$c2url2 = /https?:\/\/[^\"\s'\)]+\.(ru|top|xyz|tk|ml|ga|gq|cf|pw|click|stream|xyz|buzz|loan)\/[^\"\s'\)]*\.exe/ ascii $c2exe = /https?:\/\/[^\s"']{4,}\.(exe|dll|scr|vbs|ps1|bat)\b/ ascii nocase
$c2url3 = /(c2|panel|gate|bot|ops|serv|beacon|stage|stager)\.(ru|top|xyz|cc|io|com|net)/ ascii $c2uri = /(gate|panel|submit|checkin|beacon|report)\.(php|aspx|html)\b/ ascii nocase
$c2tld = /\b[a-z0-9-]{4,30}\.(ru|top|xyz|tk|ml|ga|gq|cf|pw|buzz|loan)\b/ ascii nocase
condition: condition:
// C2 通信 + 命令执行(至少两个网络 + 一个命令) uint16(0) == 0x5A4D
any of ($c2_send, $c2_recv, $c2_connect) and any of ($cmd*) and any of ($c2url*) and 2 of ($sock*)
or and any of ($c2*)
// C2 URL 模式 + 网络 API and
any of ($c2url*) and any of ($c2_connect, $c2_send) (
any of ($cmd*)
or $c2exe
or $c2uri
)
} }
// ============================================================================= // =============================================================================
// C1-4 Mutex 互斥体名(勒索 / RAT / Loader 的持久化标记) // C1-4 Mutex 互斥体名(勒索 / RAT / Loader 的持久化标记)
// 单一 Mutex 名有一定价值,但必须配合其他特征才触发 // 需要 CreateMutex 与家族 Mutex 名同表,避免文本文档里提一嘴家族名就命中
// ============================================================================= // =============================================================================
rule C1_PE_Mutex_Persistence { rule C1_PE_Mutex_Persistence {
meta: meta:
risk_level = "medium" risk_level = "medium"
category = "C1" category = "C1"
false_positive_rate = "medium" false_positive_rate = "low"
reference = "MITRE ATT&CK T1057 Process Discovery (mutex as persistence marker)" reference = "MITRE ATT&CK T1057 Process Discovery (mutex as persistence marker)"
description = "可疑 Mutex 名称 + CreateMutex API = 持久化标记" description = "CreateMutex + 已知家族 Mutex 名同表 = 持久化标记"
strings: strings:
$createmutex = "CreateMutexA" ascii $createmutex = "CreateMutexA" ascii
$createmutexw = "CreateMutexW" ascii $createmutexw = "CreateMutexW" ascii
// 常见勒索/RAT Mutex 名(大小写不敏感,含 Unicode 变体用 wide)
// LockBit // LockBit
$lockbit1 = /Global\\[Ll]ock[Bb]it/ ascii $lockbit1 = /Global\\[Ll]ock[Bb]it/ ascii
$lockbit2 = /Local\\[Ll]ock[Bb]it/ ascii $lockbit2 = /Local\\[Ll]ock[Bb]it/ ascii
// BlackCat / ALPHV // BlackCat / ALPHV
$blackcat = /Global\\[Bb]lack[Cc]at|[Aa][Ll][Pp][Hh][Vv]/ ascii $blackcat = /(Global|Local)\\[Bb]lack[Cc]at|\bALPHV\b/ ascii
// Ryuk // Ryuk
$ryuk = /RyukTask[\d]|Global\\Ryuk/ ascii $ryuk = /RyukTask[\d]|Global\\Ryuk/ ascii
// Conti // Conti
$conti = /Global\\[Cc]onti|Local\\[Cc]onti/ ascii $conti = /(Global|Local)\\[Cc]onti/ ascii
// Pseudo/Dharma // Pseudo/Dharma
$dharma = /Global\\[Dd]harma|Local\\[Dd]harma/ ascii $dharma = /(Global|Local)\\[Dd]harma/ ascii
// 通用可疑模式:Mutex 名包含 "Mutex" + UUID 变体 // 通用可疑模式:Mutex 名含 UUID 前缀
$generic1 = /Global\\Mutex[0-9a-f]{8}/ ascii $generic1 = /Global\\Mutex[0-9a-f]{8}/ ascii
// Kryptik/Qakbot // Kryptik/Qakbot
$qakbot = /[Qq]akbot|[Kk]ryptik/ ascii $qakbot = /\b[Qq]akbot\b|\b[Kk]ryptik\b/ ascii
// Cobalt Strike named pipe // Cobalt Strike named pipe
$cs1 = /\\\\\.\\pipe\\msagent_[0-9a-f]+/ ascii $cs1 = /\\\\\.\\pipe\\msagent_[0-9a-f]+/ ascii
$cs2 = /\\\\\.\\pipe\\[Bb]eacon_[0-9a-f]+/ ascii $cs2 = /\\\\\.\\pipe\\[Bb]eacon_[0-9a-f]+/ ascii
condition: condition:
// CreateMutex + 已知家族 Mutex 名 uint16(0) == 0x5A4D
(any of ($createmutex*)) and any of ($lockbit1, $lockbit2, $blackcat, $ryuk, $conti, $dharma, $qakbot) and
or (
// Cobalt Strike 命名管道特征(不需要 CreateMutex) any of ($cs*)
any of ($cs*) or (
or any of ($createmutex*)
// CreateMutex + 通用可疑 UUID Mutex and any of ($lockbit1, $lockbit2, $blackcat, $ryuk, $conti, $dharma, $qakbot, $generic1)
(any of ($createmutex*)) and $generic1 )
)
} }
// ============================================================================= // =============================================================================
// C1-5 DLL 劫持 / LoadLibrary 动态加载 // C1-5 DLL 劫持 / 侧加载
// 只看「LoadLibrary + GetProcAddress + 临时目录里的 DLL 名」三件套;
// \ProgramData\ 与 \Public\ 是正常安装器路径,不再单独作为可疑路径
// ============================================================================= // =============================================================================
rule C1_PE_DLLHijacking { rule C1_PE_DLLHijacking {
meta: meta:
risk_level = "high" risk_level = "high"
category = "C1" category = "C1"
false_positive_rate = "medium" false_positive_rate = "low"
reference = "MITRE ATT&CK T1574 Hijack Execution Flow" reference = "MITRE ATT&CK T1574 Hijack Execution Flow"
description = "PE 动态加载多个系统 DLL + 非系统路径 = DLL 劫持迹象" description = "动态加载 API + 临时目录 DLL 落地 = DLL 侧加载/劫持"
strings: strings:
$loadlib = "LoadLibraryA" ascii $loadlib = "LoadLibraryA" ascii
$loadlibw = "LoadLibraryW" ascii $loadlibw = "LoadLibraryW" ascii
$getproc = "GetProcAddress" ascii $getproc = "GetProcAddress" ascii
// 可疑加载路径(非 System32) $copy1 = "CopyFileA" ascii
$path1 = /\\Users\\[^\\]+\\AppData\\Local\\Temp/ ascii $copy2 = "CopyFileW" ascii
$path2 = /\\AppData\\Local\\Temp/ ascii $move = "MoveFileEx" ascii
$path3 = /\\ProgramData\\/ ascii $temp1 = /\\AppData\\Local\\Temp\\[a-z0-9_. -]{3,40}\.dll/ ascii nocase
$path4 = /\\Public\\Documents\\/ ascii $temp2 = /%TEMP%\\[a-z0-9_. -]{3,40}\.dll/ ascii nocase
$temp3 = /\\Windows\\Temp\\[a-z0-9_. -]{3,40}\.dll/ ascii nocase
$sysdir = "SetDllDirectory" ascii
$searchorder = "LoadLibraryEx" ascii
condition: condition:
// LoadLibrary + 可疑路径 + GetProcAddress = 完整劫持链 uint16(0) == 0x5A4D
(any of ($loadlib*)) and any of ($path*) and $getproc and any of ($loadlib*, $searchorder)
or and any of ($temp*)
// LoadLibrary + 多个可疑路径 and (any of ($getproc, $copy1, $copy2, $move, $sysdir))
(any of ($loadlib*)) and 2 of ($path*)
} }
// ============================================================================= // =============================================================================
// C1-6 LOLBin(Living Off The Land Binary)调用特征 // C1-6 LOLBin(Living Off The Land Binary)调用特征
// 已知合法 Windows 二进制被恶意利用来执行命令 // 命令行串 + 进程创建 API 必须同表紧邻(正常程序会把两者写进同一个导入/资源块)
// ============================================================================= // =============================================================================
rule C1_PE_LOLBin_Usage { rule C1_PE_LOLBin_Usage {
@@ -228,31 +280,57 @@ rule C1_PE_LOLBin_Usage {
category = "C1" category = "C1"
false_positive_rate = "low" false_positive_rate = "low"
reference = "https://lolbas-project.github.io/" reference = "https://lolbas-project.github.io/"
description = "PE 文件中出现 LOLBin 调用:certutil / mshta / rundll32 / regsvr32 / bitsadmin / wmic / msiexec" description = "PE 中出现 LOLBin 调用命令行 + 进程创建 API(同表)"
strings: strings:
$certutil = /certutil(\.exe)?\s+(-urlcache|-decode|-encode|-hashfile)/ ascii $lolbin1 = /certutil(\.exe)?\s+(-urlcache|-decode)\s+-split/ ascii nocase
$mshta = /mshta(\.exe)?\s+(https?:|javascript:|vbscript:)/ ascii $lolbin2 = /mshta(\.exe)?\s+(https?:|javascript:|vbscript:)/ ascii nocase
$rundll32 = /rundll32(\.exe)?\s+[A-Za-z0-9_.]+,/ ascii $lolbin3 = /regsvr32(\.exe)?\s+\/s\s+\/u:http/ ascii nocase
$regsvr32 = /regsvr32(\.exe)?\s+\/s\s+\/u/ ascii $lolbin4 = /bitsadmin(\.exe)?\s+\/transfer\s+\/source:/ ascii nocase
$bitsadmin = /bitsadmin(\.exe)?\s+(\/transfer|\/create)/ ascii $lolbin5 = /wmic(\.exe)?\s+process\s+call\s+create/ ascii nocase
$wmic = /wmic(\.exe)?\s+(process\s+call\s+create|product\s+call)/ ascii $lolbin6 = /msiexec(\.exe)?\s+\/[ia]\s+https?:/ ascii nocase
$msiexec = /msiexec(\.exe)?\s+\/[ia]\s+https?:/ ascii $lolbin7 = /installutil(\.exe)?\s+\/u\s+\/logfile=https?:/ ascii nocase
$installutil = /installutil(\.exe)?\s+\/u\s+\/logfile=/ ascii $lolbin8 = /psexec(\.exe)?\s+(-accepteula\s+)?-w\s+https?:/ ascii nocase
$syncapp = /syncapp(\.exe)?\s+\/compattrack/ ascii
$psexec = /psexec(\.exe)?\s+(-accepteula|-i\s+\d+|-w\s+https?:)/ ascii
// 命令执行 API(用于组合条件)
$exec1 = "CreateProcess" ascii $exec1 = "CreateProcess" ascii
$exec2 = "ShellExecute" ascii $exec2 = "ShellExecute" ascii
$exec3 = "WinExec" ascii $exec3 = "WinExec" ascii
condition: condition:
// 至少 1 个 LOLBin 调用 + 命令执行 API uint16(0) == 0x5A4D
any of ($certutil*, $mshta*, $rundll32*, $regsvr32*, $bitsadmin*, $wmic*, $msiexec*, $installutil*, $syncapp*, $psexec*) and any of ($lolbin*)
and any of ($exec*) and any of ($exec*)
} }
// ============================================================================= // =============================================================================
// C8-1 勒索加密 API 组合(CryptEncrypt + BCrypt* + 文件遍历 + 删除卷影) // C1-7 PE 高熵 + 网络 API(疑似加壳下载器)
// 需要「多个网络 API + 明确壳节名」同时成立
// =============================================================================
rule C1_PE_EntropyHighNetwork {
meta:
risk_level = "medium"
category = "C1"
false_positive_rate = "low"
reference = "binary.py HighEntropySection 联动"
description = "PE 加壳节名 + 多个下载 API = 加壳下载器"
strings:
$net1 = "InternetOpenUrl" ascii
$net2 = "URLDownloadToFile" ascii
$net3 = "WinHttpOpenRequest" ascii
$net4 = "WSAConnect" ascii
condition:
uint16(0) == 0x5A4D
and 2 of ($net*)
and (
for any i in (0..pe.number_of_sections - 1): (
pe.sections[i].name matches /UPX0/ or pe.sections[i].name == "UPX1"
or pe.sections[i].name matches /\.vmp[0-9]/ or pe.sections[i].name matches /Themida|\.themida/
)
)
}
// =============================================================================
// C8-1 勒索加密链(加密 API + 卷影删除/勒索信,勒索样本常以文本形态投递,不加 PE 头)
// ============================================================================= // =============================================================================
rule C8_Ransomware_Encryption { rule C8_Ransomware_Encryption {
@@ -261,9 +339,9 @@ rule C8_Ransomware_Encryption {
category = "C8" category = "C8"
false_positive_rate = "low" false_positive_rate = "low"
reference = "MITRE ATT&CK T1486 Data Encrypted for Impact" reference = "MITRE ATT&CK T1486 Data Encrypted for Impact"
description = "勒索加密链:CryptEncrypt/BCryptEncrypt + 大量文件遍历 + 卷影删除 = 勒索" description = "勒索加密链:加密 API + 卷影删除命令或勒索信文件名"
strings: strings:
// 加密 API(至少出现两个) // 加密 API
$enc1 = "CryptEncrypt" ascii $enc1 = "CryptEncrypt" ascii
$enc2 = "CryptDeriveKey" ascii $enc2 = "CryptDeriveKey" ascii
$enc3 = "BCryptEncrypt" ascii $enc3 = "BCryptEncrypt" ascii
@@ -276,25 +354,23 @@ rule C8_Ransomware_Encryption {
$enum3 = "FindNextFileA" ascii $enum3 = "FindNextFileA" ascii
$enum4 = "FindNextFileW" ascii $enum4 = "FindNextFileW" ascii
// 卷影删除(勒索前置) // 卷影删除(完整命令,不认裸 "wmic")
$vss1 = "vssadmin" ascii $vss1 = "vssadmin" ascii nocase
$vss2 = "delete shadows" ascii $vss2 = "delete shadows" ascii nocase
$vss3 = "DeleteShadowCopies" ascii $vss3 = "DeleteShadowCopies" ascii
$vss4 = "wmic" ascii $vss4 = "shadow copies" ascii nocase
// 勒索信文件名变体(出现在二进制里) // 勒索信文件名 / 加密扩展名
$ransomnote1 = /(HOW_TO_RECOVER|RESTORE_FILES|DECRYPT_INSTRUCTIONS|README|HELP_DECRYPT|_readme|_restore)[\w.-]*\.(txt|html|png|jpg|bmp|url)/ nocase $note1 = /(HOW_TO_RECOVER|RESTORE_FILES|DECRYPT_INSTRUCTIONS|HELP_DECRYPT)[\w.-]*\.(txt|html|png|jpg|bmp|url)/ nocase
$ransomnote2 = /\.(locky|zepto|cryptolocker|tor|crpt|encrypted|virus|wallet|pay2)/ nocase $note2 = /\.(locky|zepto|cryptolocker|crpt|encrypted|pay2)\b/ nocase
$ransomnote3 = /bitcoin|bitcash|monero|ltc|BTC\b/ ascii
condition: condition:
// 加密 API + 文件遍历 = 加密工具(必须至少 2 个加密 API) any of ($enc*)
(2 of ($enc*)) and any of ($enum*) and
or (
// 加密 API + 卷影删除 = 完整勒索 ($vss1 and ($vss2 or $vss3 or $vss4))
(any of ($enc*)) and any of ($vss*) or (2 of ($enum*) and any of ($note*))
or or (2 of ($enc*) and any of ($enum*) and $vss1)
// 加密 API + 勒索信 + 加密文件扩展名 = 明确勒索 )
(any of ($enc*)) and (any of ($ransomnote1, $ransomnote2)) and $ransomnote3
} }
@@ -310,53 +386,51 @@ rule C8_Mining_Monero {
reference = "MITRE ATT&CK T1496 Resource Hijacking" reference = "MITRE ATT&CK T1496 Resource Hijacking"
description = "挖矿程序特征:xmrig 命令行参数 / stratum+tcp / 矿池域名" description = "挖矿程序特征:xmrig 命令行参数 / stratum+tcp / 矿池域名"
strings: strings:
// xmrig 命令行特征
$xmrig1 = "xmrig" ascii nocase $xmrig1 = "xmrig" ascii nocase
$xmrig2 = /--donate-level\s+\d+/ ascii $xmrig2 = /--donate-level\s+\d+/ ascii
$xmrig3 = /-o\s+stratum\+tcp/ ascii $xmrig3 = /-o\s+stratum\+tcp/ ascii
$xmrig4 = /--cpu-priority\s+\d/ ascii $xmrig4 = /--cpu-priority\s+\d/ ascii
$xmrig5 = /--max-threads-hint/ ascii $xmrig5 = /--max-threads-hint/ ascii
// 矿池域名(Monero)
$pool1 = "pool.xmrig.com" ascii nocase $pool1 = "pool.xmrig.com" ascii nocase
$pool2 = "pool.minexmr.com" ascii nocase $pool2 = "pool.minexmr.com" ascii nocase
$pool3 = "pool.supportxmr.com" ascii nocase $pool3 = "pool.supportxmr.com" ascii nocase
$pool4 = "mine.moneropool.com" ascii nocase $pool4 = "mine.moneropool.com" ascii nocase
$pool5 = "stratum+tcp://" ascii $pool5 = "stratum+tcp://" ascii
// CPU 高占用循环特征
$loop = /(while|for)\s*\(\s*\d+\s*[<>=]/ ascii
$thread = "CreateThread" ascii $thread = "CreateThread" ascii
condition: condition:
// 至少 2 个 xmrig 特征 any of ($pool1, $pool2, $pool3, $pool4)
(any of ($xmrig1, $xmrig2, $xmrig3, $xmrig4, $xmrig5)) and 2 of ($*) or ($pool5 and (any of ($xmrig*) or $thread))
or or (2 of ($xmrig*) and any of ($pool*))
// stratum+tcp 矿池协议 + 多个线程创建
$pool5 and ($thread or 2 of ($xmrig*))
} }
// ============================================================================= // =============================================================================
// C9 壳特征(UPX / Themida / VMProtect / Enigma) // C9 壳特征(UPX / Themida / VMProtect)
// 壳识别后建议升级到 L3 沙箱深检,不单独阻断 // 加壳本身不是恶意证据(大量开源工具用 UPX),只作沙箱深检触发条件
// ============================================================================= // =============================================================================
rule C9_Packer_UPX { rule C9_Packer_UPX {
meta: meta:
risk_level = "medium" risk_level = "low"
category = "C9" category = "C9"
false_positive_rate = "low" false_positive_rate = "low"
reference = "UPX Packer" reference = "UPX Packer"
description = "UPX 加壳 PE:节名 UPX0 / UPX1 + UPX 签名" description = "UPX 加壳 PE:节名 UPX0/UPX1 + UPX 签名"
strings: strings:
$upx0 = "UPX0" ascii
$upx1 = "UPX1" ascii
$upx2 = "UPX!" ascii $upx2 = "UPX!" ascii
$upx3 = /UPX\s+\d+\.\d+/ ascii // "UPX 3.96" 这类版本字符串
$upx4 = /UPX\b/ ascii
condition: condition:
// 节名(两个)+ 签名 uint16(0) == 0x5A4D
($upx0 and $upx1) or ($upx2 and any of ($upx3, $upx4)) and (
(
for any i in (0..pe.number_of_sections - 1): (
pe.sections[i].name matches /UPX0/
)
and for any j in (0..pe.number_of_sections - 1): (
pe.sections[j].name matches /UPX1/
)
)
or ($upx2 and pe.number_of_sections <= 4)
)
} }
rule C9_Packer_Themida_VMP { rule C9_Packer_Themida_VMP {
@@ -365,39 +439,18 @@ rule C9_Packer_Themida_VMP {
category = "C9" category = "C9"
false_positive_rate = "low" false_positive_rate = "low"
reference = "Themida / VMProtect" reference = "Themida / VMProtect"
description = "商业壳特征:Themida / VMProtect 产品标识字符串" description = "商业壳特征:Themida / VMProtect 节名 + 产品标识字符串"
strings: strings:
$themida1 = "Themida" ascii wide $themida1 = "Themida" ascii wide
$themida2 = "WinLicense" ascii wide $themida2 = "WinLicense" ascii wide
$themida3 = "Oreans" ascii wide $themida3 = "Oreans" ascii wide
$vmp1 = "VMProtect" ascii wide $vmp1 = "VMProtect" ascii wide
$vmp3 = "VMProtect SDK" ascii wide
condition: condition:
any of ($themida*) or any of ($vmp*) uint16(0) == 0x5A4D
} and (
for any i in (0..pe.number_of_sections - 1): (
pe.sections[i].name matches /\.themida|\.vmp[0-9]/
// ============================================================================= )
// C1-7 PE 高熵 + 大量网络 API(疑似加壳下载器) or (any of ($themida*) and any of ($vmp1))
// binary.py 的 HighEntropySection 已单独扫描,但 YARA 看"熵 + 网络"组合 )
// =============================================================================
rule C1_PE_EntropyHighNetwork {
meta:
risk_level = "medium"
category = "C1"
false_positive_rate = "medium"
reference = "binary.py HighEntropySection 联动"
description = "PE 同时出现:多个网络 API + 加壳特征(熵高) + 无签名"
strings:
$net1 = "InternetOpenUrl" ascii
$net2 = "WinHttpOpen" ascii
$net3 = "URLDownloadToFile" ascii
$net4 = "WSAConnect" ascii
// 壳特征
$upx = "UPX0" ascii
$themida = "Themida" ascii
condition:
// 网络 API + 壳特征 = 加壳下载器
any of ($net*) and ($upx or $themida)
} }

View File

@@ -126,26 +126,41 @@ rule C8_Ransomware_Note_Filename {
meta: meta:
risk_level = "medium" risk_level = "medium"
category = "C8" category = "C8"
false_positive_rate = "medium" false_positive_rate = "low"
reference = "勒索信文件名通用模式" reference = "勒索信文件名通用模式"
description = "勒索信典型文件名变体(匹配后触发需要配合加密 API 才升级高危)" description = "勒索信文件名 + (加密 API + 赎金语言)或(赎金语言 + 钱包地址)"
strings: strings:
// 只保留"文件名形态"的勒索信特征:家族专有名 + 扩展名。
// 不再用裸 README.txt / _readme.txt:任何安装包都带这两个字面量,
// 在二进制内容上做全文匹配时必然命中(Go 官方 MSI 即被这样判成恶意)。
$n1 = /HOW_TO_RECOVER[\w.-]*\.(txt|html|png|bmp|jpg|url|vbs|js)/ nocase $n1 = /HOW_TO_RECOVER[\w.-]*\.(txt|html|png|bmp|jpg|url|vbs|js)/ nocase
$n2 = /RESTORE_FILES[\w.-]*\.(txt|html|png|bmp|jpg|url|vbs|js)/ nocase $n2 = /RESTORE_FILES[\w.-]*\.(txt|html|png|bmp|jpg|url|vbs|js)/ nocase
$n3 = /DECRYPT_INSTRUCTIONS[\w.-]*\.(txt|html|png|bmp|jpg|url)/ nocase $n3 = /DECRYPT_INSTRUCTIONS[\w.-]*\.(txt|html|png|bmp|jpg|url)/ nocase
$n4 = /README[_\s-]*[\d]*[_\s-]*[\w]*\.(txt|html|png|bmp|jpg|url)/ nocase $n4 = /README[._-]*(FOR[._-]*DECRYPT|TO[._-]*RESTORE)[\w.-]*\.(txt|html|png|bmp|jpg|url)/ nocase
$n5 = /HELP_DECRYPT[\w.-]*\.(txt|html|png|bmp|jpg|url)/ nocase $n5 = /HELP_DECRYPT[\w.-]*\.(txt|html|png|bmp|jpg|url)/ nocase
$n6 = /_readme[\w.-]*\.(txt|html|png|bmp|jpg|url)/ nocase $n6 = /\b_(readme|restore)[\w.-]*\.(txt|html|png|bmp|jpg|url)/ nocase
$n7 = /_restore[\w.-]*\.(txt|html|png|bmp|jpg|url)/ nocase $n7 = /(how_to_decrypt|restore_my_files|decrypt_your_files)[\w.-]*\.(txt|html)/ nocase
$bitcoin = /bitcoin address|BTC\b|monero address|XMR\b|send money|ransom|payment/ nocase
$wallet = /1[A-Za-z0-9]{25,34}|bc1[A-Za-z0-9]{11,71}|4[0-9A-Za-z]{32}/ nocase // 赎金语言:必须是完整词组。裸 "BTC"/"XMR"/"payment"/"ransom" 在压缩流里
// 会以随机字节出现,"ransom" 还会被 "Ransomware" 子串命中。
$payment = /(bitcoin|monero|xmr|btc)\s+(address|wallet)|send\s+(the\s+)?money|pay\s+(the\s+)?(ransom|fee)|\bransom\b\s+(note|payment|demand)|your\s+files\s+(are\s+)?encrypted|decrypt\s+your\s+files/ nocase
// 钱包地址:锚定到整个 token,并用真实地址字符集/长度。
// 原写法 1[A-Za-z0-9]{25,34} 无边界,单个 56MB 安装包命中 7.6 万次。
$btc_addr = /\b[13][a-km-zA-HJ-NP-Z1-9]{25,34}\b/
$bech32 = /\bbc1q[a-z0-9]{38}\b|\bbc1q[a-z0-9]{58}\b/
$xmr_addr = /\b[48][0-9A-Za-z]{94}\b/
$encapi = /CryptEncrypt|BCryptEncrypt|CryptDeriveKey/ ascii $encapi = /CryptEncrypt|BCryptEncrypt|CryptDeriveKey/ ascii
condition: condition:
// 勒索信文件名 + 加密 API = 明确勒索 any of ($n*)
any of ($n*) and $encapi and (
or // 投递器:真加密 API + 赎金语言
// 勒索信文件名 + 比特币/XMR 地址 = 大概率勒索 ($encapi and $payment)
any of ($n*) and any of ($bitcoin, $wallet) or
// 纯文本勒索信:赎金语言 + 具体钱包地址
($payment and any of ($btc_addr, $bech32, $xmr_addr))
)
} }
@@ -199,11 +214,13 @@ rule C8_Ransomware_VolumeShadow {
$vss2 = /vssadmin\s+delete\s+shadows\s+\/all\b/ ascii $vss2 = /vssadmin\s+delete\s+shadows\s+\/all\b/ ascii
$wmic1 = /wmic\s+shadowcopy\s+delete/ ascii $wmic1 = /wmic\s+shadowcopy\s+delete/ ascii
$wmic2 = /wmic\s+shadowcopy\s+where\s+\d+/ ascii $wmic2 = /wmic\s+shadowcopy\s+where\s+\d+/ ascii
$vss3 = /ShadowCopy|VolumeShadowCopy|vss\b/i ascii // "i" flag = nocase in YARA $vss4 = /vssadmin[^\n]{0,48}delete[^\n]{0,48}(shadows|shadow\s+copies)/ ascii
$enc = /CryptEncrypt|BCryptEncrypt|CryptDeriveKey/ ascii $enc = /CryptEncrypt|BCryptEncrypt|CryptDeriveKey/ ascii
condition: condition:
// 卷影删除 + 加密 API = 完整勒索前置 // 卷影删除 + 加密 API = 完整勒索前置
(any of ($vss1, $vss2, $wmic*, $vss3)) and any of ($enc*) // 不用裸 "vss"/"ShadowCopy" 子串:任何备份/卷影库的符号表都有它们,
// 和 CryptEncrypt 同框就报勒索,是典型误报来源。
(any of ($vss1, $vss2, $wmic*, $vss4)) and any of ($enc*)
or or
// 连续多次 vssadmin delete = 正在清卷影 // 连续多次 vssadmin delete = 正在清卷影
any of ($vss1, $vss2) and any of ($wmic*) any of ($vss1, $vss2) and any of ($wmic*)

View File

@@ -57,12 +57,19 @@ rule C3_PHP_Webshell_Obfuscated {
$rot13 = "str_rot13" ascii wide $rot13 = "str_rot13" ascii wide
$assert = "assert(" ascii wide $assert = "assert(" ascii wide
$createfn = "create_function" ascii wide $createfn = "create_function" ascii wide
// PHP 上下文证据:没有 <?php / 超全局变量,就不要凭函数名定罪(vim、git 等
// 二进制里同样存在 base64_decode / eval( 字样)
$php1 = "<?php" ascii
$php2 = /\$_(GET|POST|REQUEST|COOKIE|FILES|SESSION)\b/ ascii
condition: condition:
// base64_decode + (eval|assert|create_function) = 解码后动态执行 any of ($php*)
$b64 and ($eval or $assert or $createfn) and (
or // base64_decode + (eval|assert|create_function) = 解码后动态执行
// 解码链:base64 + gzinflate / str_rot13 + 执行 ($b64 and ($eval or $assert or $createfn))
$b64 and ($gzinflate or $rot13) and ($eval or $assert) or
// 解码链:base64 + gzinflate / str_rot13 + 执行
($b64 and ($gzinflate or $rot13) and ($eval or $assert))
)
} }
// PHP Webshell —— 变量变形($$a = $_POST['cmd'];$$a($_POST['arg'])) // PHP Webshell —— 变量变形($$a = $_POST['cmd'];$$a($_POST['arg']))

View File

@@ -0,0 +1,264 @@
"""DB 归档 + 清理脚本。
职责:
1. scan_tasks 90 天前: 清 detail JSONB (节省空间, 扫描详情留 90 天)
2. scan_tasks 365 天前: 移到 scan_tasks_archive (含 ScanFinding)
3. idempotency_records: 清 7 天前的 (幂等缓存过期重扫)
4. policy_versions: 只保留最新 5 个版本 (历史版本清掉)
用法:
docker compose exec api python3 scripts/archive_old_tasks.py # dry-run, 只报告不动数据
docker compose exec api python3 scripts/archive_old_tasks.py --apply # 真正执行
建议: 每天 cron 跑一次
设计原则:
- 小批量提交 (1000 条一批), 避免长事务锁表
- 先 dry-run 再 apply, 输出清晰
- pg_repack / VACUUM ANALYZE 在脚本最后触发 (让 autovacuum 兜底也行)
"""
from __future__ import annotations
import argparse
import logging
import sys
import os
from datetime import timedelta, datetime, timezone
from pathlib import Path
sys.path.insert(0, "/app")
os.environ.setdefault("MESHSAFE_MASTER_KEY", "Dt0ktRs8SjEH8VxWZJDX125NV1qiII88HoiXZPhWers=")
from app.db import SessionLocal # noqa: E402
from app.models import ( # noqa: E402
ScanTask, ScanFinding, IdempotencyRecord, PolicyVersion,
)
from sqlalchemy import select, func # noqa: E402
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger("archive")
# ---- 配置 ----
DETAIL_RETENTION_DAYS = 90 # 90 天前清 detail JSONB
ARCHIVE_RETENTION_DAYS = 365 # 365 天前移归档
IDEM_RETENTION_DAYS = 7 # 幂等缓存 7 天 TTL
POLICY_KEEP_VERSIONS = 5 # policy_versions 只留最新 5 个
BATCH_SIZE = 1000 # 分批处理
def _cutoff(days: int) -> datetime:
return datetime.now(timezone.utc) - timedelta(days=days)
def archive_tasks(dry_run: bool) -> None:
"""归档 365 天前的 scan_tasks 到 scan_tasks_archive。"""
cutoff = _cutoff(ARCHIVE_RETENTION_DAYS)
db = SessionLocal()
try:
total = db.execute(
select(func.count(ScanTask.id)).where(ScanTask.created_at < cutoff)
).scalar() or 0
log.info(f"[归档] {ARCHIVE_RETENTION_DAYS} 天前的 scan_tasks: {total} 条")
if total == 0:
log.info(" → 无需归档")
return
if dry_run:
log.info(f" [dry-run] 将归档 {total} 条")
return
# 分批
moved = 0
while True:
batch = db.execute(
select(ScanTask).where(ScanTask.created_at < cutoff).limit(BATCH_SIZE)
).scalars().all()
if not batch:
break
for task in batch:
try:
db.execute(
__import__("sqlalchemy").text("""
INSERT INTO scan_tasks_archive
(id, channel, client_id, filename, declared_ext, size,
sha256, md5, real_type, status, mode, verdict, action, score,
reason, policy_version, has_detail, has_findings,
created_at, finished_at, duration_ms, archived_at)
VALUES
(:id, :channel, :client_id, :filename, :declared_ext, :size,
:sha256, :md5, :real_type, :status, :mode, :verdict, :action, :score,
:reason, :policy_version, :has_detail, :has_findings,
:created_at, :finished_at, :duration_ms, NOW())
ON CONFLICT (id) DO NOTHING
"""),
{
"id": task.id,
"channel": task.channel,
"client_id": task.client_id,
"filename": task.filename,
"declared_ext": task.declared_ext,
"size": task.size,
"sha256": task.sha256,
"md5": task.md5,
"real_type": task.real_type,
"status": task.status,
"mode": task.mode,
"verdict": task.verdict,
"action": task.action,
"score": task.score,
"reason": task.reason,
"policy_version": task.policy_version,
"has_detail": bool(task.detail and len(task.detail) > 0),
"has_findings": bool(task.findings),
"created_at": task.created_at,
"finished_at": task.finished_at,
"duration_ms": task.duration_ms,
},
)
# CASCADE 删 ScanFinding
db.delete(task)
moved += 1
except Exception as e:
log.warning(f" task {task.id} 归档失败: {e}")
db.rollback()
continue
db.commit()
log.info(f" 已归档 {moved}/{total}")
log.info(f" ✅ 归档完成: {moved} 条")
finally:
db.close()
def scrub_detail(dry_run: bool) -> None:
"""90~365 天之间的任务清 detail JSONB (留摘要, 省空间)。"""
db_cutoff = _cutoff(DETAIL_RETENTION_DAYS)
arc_cutoff = _cutoff(ARCHIVE_RETENTION_DAYS)
db = SessionLocal()
try:
total = db.execute(
select(func.count(ScanTask.id)).where(
ScanTask.created_at < db_cutoff,
ScanTask.created_at >= arc_cutoff,
ScanTask.detail.isnot(None),
)
).scalar() or 0
log.info(f"[清 detail] {DETAIL_RETENTION_DAYS}~{ARCHIVE_RETENTION_DAYS} 天之间有 detail 的: {total} 条")
if total == 0:
log.info(" → 无需清理")
return
if dry_run:
log.info(f" [dry-run] 将清 {total} 条 detail")
return
# 分批 UPDATE
updated = 0
while True:
batch_ids = db.execute(
select(ScanTask.id).where(
ScanTask.created_at < db_cutoff,
ScanTask.created_at >= arc_cutoff,
ScanTask.detail.isnot(None),
).limit(BATCH_SIZE)
).scalars().all()
if not batch_ids:
break
db.execute(
__import__("sqlalchemy").update(ScanTask).where(ScanTask.id.in_(batch_ids)).values(detail=None)
)
db.commit()
updated += len(batch_ids)
log.info(f" 已清 {updated}/{total}")
log.info(f" ✅ detail 清理完成: {updated} 条")
finally:
db.close()
def clean_idempotency(dry_run: bool) -> None:
"""清 7 天前的幂等缓存 (过期的可以重扫)。"""
cutoff = _cutoff(IDEM_RETENTION_DAYS)
db = SessionLocal()
try:
total = db.execute(
select(func.count(IdempotencyRecord.id)).where(IdempotencyRecord.created_at < cutoff)
).scalar() or 0
log.info(f"[幂等清理] {IDEM_RETENTION_DAYS} 天前的 idempotency_records: {total} 条")
if total == 0:
log.info(" → 无需清理")
return
if dry_run:
log.info(f" [dry-run] 将清 {total} 条")
return
db.execute(
__import__("sqlalchemy").delete(IdempotencyRecord).where(IdempotencyRecord.created_at < cutoff)
)
db.commit()
log.info(f" ✅ 幂等清理完成: {total} 条")
finally:
db.close()
def trim_policy_versions(dry_run: bool) -> None:
"""policy_versions 只留最新 N 个。"""
db = SessionLocal()
try:
all_pv = db.execute(
select(PolicyVersion).order_by(PolicyVersion.created_at.desc())
).scalars().all()
keep = all_pv[:POLICY_KEEP_VERSIONS]
drop = all_pv[POLICY_KEEP_VERSIONS:]
log.info(f"[policy] 共 {len(all_pv)} 条历史版本, 留 {len(keep)} 删 {len(drop)}")
if not drop:
log.info(" → 无需清理")
return
if dry_run:
for pv in drop:
log.info(f" [dry-run] 将删 {pv.version}")
return
for pv in drop:
db.delete(pv)
db.commit()
log.info(f" ✅ policy_versions 清理完成: 删 {len(drop)} 条")
finally:
db.close()
def main() -> None:
parser = argparse.ArgumentParser(description="MeshSafe DB 归档 + 清理")
parser.add_argument("--apply", action="store_true", help="真正执行 (默认 dry-run)")
args = parser.parse_args()
dry = not args.apply
log.info(f"模式: {'DRY-RUN (仅报告)' if dry else 'APPLY (真正执行)'}")
log.info("")
archive_tasks(dry)
log.info("")
scrub_detail(dry)
log.info("")
clean_idempotency(dry)
log.info("")
trim_policy_versions(dry)
if dry:
log.info("\n💡 确认无误后加 --apply 参数正式执行")
if __name__ == "__main__":
main()

140
scripts/cron_entrypoint.py Normal file
View File

@@ -0,0 +1,140 @@
"""MeshSafe 定时任务调度器 — 零依赖,Docker 容器前台常驻。
和 api 共享同一个镜像,容器 entrypoint 换成本脚本即可。
用法: docker run meshsafe:latest python3 scripts/cron_entrypoint.py
支持的任务:
每天 03:00 归档 + 清理 DB (archive_old_tasks.py --apply)
每周一 03:10 VACUUM ANALYZE (维护表统计信息)
"""
from __future__ import annotations
import logging
import os
import subprocess
import sys
import time
from datetime import datetime, timedelta
sys.path.insert(0, "/app")
logging.basicConfig(
level=logging.INFO,
format="%(asctime)s [%(levelname)s] %(message)s",
)
log = logging.getLogger("scheduler")
_SECRET_ENV_KEYS = ("MESHSAFE_DB_URL", "MESHSAFE_MASTER_KEY", "MESHSAFE_ADMIN_PASSWORD",
"MESHSAFE_API_KEY", "MESHSAFE_API_CLIENTS")
def _redact(text: str) -> str:
"""崩溃日志里 pydantic 会把整个 settings dict(含 DB 口令)原样打出来,落盘前抹掉。"""
out = text
for key in _SECRET_ENV_KEYS:
val = os.environ.get(key, "")
if val and len(val) >= 8:
out = out.replace(val, "***")
return out
def run(cmd: list[str], label: str) -> None:
"""跑一个命令,捕获输出并记日志。"""
log.info(f"[{label}] 开始执行: {' '.join(cmd)}")
try:
proc = subprocess.run(cmd, capture_output=True, text=True, timeout=1800)
if proc.stdout:
for line in proc.stdout.strip().splitlines():
log.info(f"[{label}] {_redact(line)}")
if proc.stderr:
for line in proc.stderr.strip().splitlines():
log.warning(f"[{label}] {_redact(line)}")
if proc.returncode == 0:
log.info(f"[{label}] ✅ 完成")
else:
log.error(f"[{label}] ❌ 返回码 {proc.returncode}")
except subprocess.TimeoutExpired:
log.error(f"[{label}] ❌ 超时 (30min)")
except Exception as e:
log.error(f"[{label}] ❌ 异常: {e}")
def run_vacuum() -> None:
"""VACUUM ANALYZE — 维护表统计信息。
VACUUM 不能在事务块里跑(psycopg2 会抛 ActiveSqlTransaction),
所以用 AUTOCOMMIT 连接而不是 engine.begin()。
"""
try:
from app.db import engine
from sqlalchemy import text
tables = [
"scan_tasks", "scan_tasks_archive",
"scan_findings", "idempotency_records",
"policy_versions", "audit_logs",
]
with engine.connect() as raw:
conn = raw.execution_options(isolation_level="AUTOCOMMIT")
for tbl in tables:
log.info(f"[VACUUM] {tbl} ...")
conn.execute(text(f"VACUUM ANALYZE {tbl}"))
log.info("[VACUUM] ✅ 全部完成")
except Exception as e:
log.error(f"[VACUUM] ❌ 异常: {_redact(str(e))}")
def seconds_until(target_hour: int, target_minute: int, weekday: int | None = None) -> float:
"""计算到下次目标时间的秒数。
weekday: 0=周一...6=周日, None=每天
"""
now = datetime.now()
candidate = now.replace(hour=target_hour, minute=target_minute, second=0, microsecond=0)
if weekday is not None:
# 找到本周目标日
days_ahead = weekday - now.weekday()
if days_ahead < 0 or (days_ahead == 0 and now >= candidate):
days_ahead += 7
candidate += timedelta(days=days_ahead)
else:
if candidate <= now:
candidate += timedelta(days=1)
return (candidate - now).total_seconds()
def main() -> None:
log.info("=" * 60)
log.info("MeshSafe 定时任务调度器启动")
log.info(f"PID={os.getpid()} TZ=Asia/Shanghai")
log.info("任务:")
log.info(" - 每天 03:00 archive_old_tasks.py --apply")
log.info(" - 每周一 03:10 VACUUM ANALYZE")
log.info("=" * 60)
while True:
# 计算两个任务下次执行时间,取更早的那个
secs_archive = seconds_until(3, 0, weekday=None) # 每天 03:00
secs_vacuum = seconds_until(3, 10, weekday=0) # 周一 03:10
if secs_archive <= secs_vacuum:
log.info(f"下一个任务: 归档 → {secs_archive/3600:.1f}h 后")
time.sleep(secs_archive)
run(
[sys.executable, "/app/scripts/archive_old_tasks.py", "--apply"],
"ARCHIVE",
)
# 跑完后清下一次时间 (避免在 03:00 这一刻两个任务重叠)
continue
else:
log.info(f"下一个任务: VACUUM → {secs_vacuum/3600:.1f}h 后")
time.sleep(secs_vacuum)
run_vacuum()
continue
if __name__ == "__main__":
main()

View File

@@ -1,3 +1,9 @@
-- 修复 real_type 字段长度:VARCHAR(128) → VARCHAR(256) -- 修复 real_type 字段长度:VARCHAR(128) → VARCHAR(256)
-- libmagic 对 ZIP/ISO 等返回的描述可能超过 128 字符 -- libmagic 对 ZIP/ISO 等返回的描述可能超过 128 字符
ALTER TABLE scan_tasks ALTER COLUMN real_type TYPE VARCHAR(256); ALTER TABLE scan_tasks ALTER COLUMN real_type TYPE VARCHAR(256);
-- 修复 real_type 字段长度:VARCHAR(256) → VARCHAR(512)
-- libmagic 对 MSI/OLE 复合文档会把 Title/Subject/Author/Creating Application 等元数据
-- 全部拼进描述串(实测 go1.25.0.windows-amd64.msi 超过 334 字符),超出列宽会让
-- 整条 scan_tasks INSERT 失败,连带的 scan_findings 一起丢失——被拒绝的文件反而不留痕。
ALTER TABLE scan_tasks ALTER COLUMN real_type TYPE VARCHAR(512);

View File

@@ -1 +0,0 @@
meshsafe-hot-reload-test-string present here

View File

@@ -1 +0,0 @@
trojan-metadata-test payload here

View File

@@ -1 +0,0 @@
powershell -enc ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwx+/==

View File

@@ -1 +0,0 @@
size_condition_test_payload small file

View File

@@ -43,7 +43,7 @@
"filename": "L0-004_pdf_normal.pdf", "filename": "L0-004_pdf_normal.pdf",
"case_id": "L0-004", "case_id": "L0-004",
"category": "L0", "category": "L0",
"expected_verdict": "pass", "expected_verdict": "suspicious",
"expected_rules": [ "expected_rules": [
"MalformedStructure", "MalformedStructure",
"PdfParseFailed" "PdfParseFailed"
@@ -55,7 +55,7 @@
"filename": "L0-005_pe_clean.exe", "filename": "L0-005_pe_clean.exe",
"case_id": "L1-008", "case_id": "L1-008",
"category": "L1", "category": "L1",
"expected_verdict": "pass", "expected_verdict": "suspicious",
"expected_rules": [ "expected_rules": [
"DangerousExecutableExtension", "DangerousExecutableExtension",
"PeParseFailed" "PeParseFailed"
@@ -1047,14 +1047,14 @@
"filename": "L5-005_injection_imports.exe", "filename": "L5-005_injection_imports.exe",
"case_id": "L5-005", "case_id": "L5-005",
"category": "L5", "category": "L5",
"expected_verdict": "reject", "expected_verdict": "review",
"expected_rules": [ "expected_rules": [
"DangerousExecutableExtension", "DangerousExecutableExtension",
"PEArchInfo", "PEArchInfo",
"SuspiciousTimestamp", "SuspiciousTimestamp",
"UnsignedExecutable" "UnsignedExecutable"
], ],
"description": "注入组合导入(需要真实 pefile 引入 API)", "description": "注入组合导入占位样本(928B 假 PE,实际没有导入表)。旧策略靠 载体45+时间戳20+未签名10+架构10 累加到 85 才判 reject,属于权重堆叠误报;权重收紧后应停在人工复核。真实注入器由 YARA C1_PE_ProcessInjection(critical=85) 直接拦截。",
"_scan_verdict": "malicious" "_scan_verdict": "malicious"
}, },
{ {