Compare commits
6 Commits
c3e37f642a
...
0d74f0405a
| Author | SHA1 | Date | |
|---|---|---|---|
| 0d74f0405a | |||
| ea69788d0c | |||
| 3bb3528ad6 | |||
| 8cf3aff4e2 | |||
| 26ca0e6ab2 | |||
| de6818ad17 |
82
.claude/skills/git-commit/SKILL.md
Normal file
82
.claude/skills/git-commit/SKILL.md
Normal file
@ -0,0 +1,82 @@
|
||||
---
|
||||
name: git-commit
|
||||
description: Generate a Conventional-Commits-style message from the repo's current staged/unstaged changes and create the commit. Use whenever the user explicitly asks to commit, save, or checkpoint their work ("提交一下", "commit这些改动", "帮我提交代码", "存一下进度", "commit this"). Never invoke this proactively — only when the user asks for a commit, not merely after finishing an edit. Does not cover pushing, PR creation, amending, or any other git operation.
|
||||
---
|
||||
|
||||
# Git 提交(规范化 commit message)
|
||||
|
||||
## 为什么需要这个技能
|
||||
|
||||
"提交一下"这句话背后其实有一套容易被跳过的步骤:先看清楚这次改动到底涉及
|
||||
哪些文件、有没有不该进版本库的东西(密钥、临时文件、误生成的二进制),再把
|
||||
分散的改动归纳成一句准确说明"为什么改"而不是"改了什么"的话——好的commit
|
||||
message省的是未来翻`git log`时重新读一遍diff的时间。这个技能把这套步骤和
|
||||
一套固定的message格式(Conventional Commits)定下来,避免每次现编、风格漂移。
|
||||
|
||||
## 工作流程
|
||||
|
||||
### 第一步:摸清这次改动的全貌
|
||||
|
||||
并行执行:
|
||||
- `git status`(不要加`-uall`,大仓库会有性能问题)
|
||||
- `git diff` 看未暂存的改动,`git diff --cached` 看已暂存的改动
|
||||
- `git log --oneline -10` 看最近的提交,判断这个仓库习惯的message风格(有的仓库不用Conventional Commits前缀,这时候跟随仓库现有风格优先于本技能默认的格式)
|
||||
|
||||
### 第二步:提交前检查——发现问题就先说清楚,不要直接提交
|
||||
|
||||
逐条过一遍将要暂存的文件,确认没有:
|
||||
- 明显的密钥/凭据文件(`.env`、`credentials.json`、私钥等)——发现了就跳过并提醒用户,不要连带提交
|
||||
- 不该进版本库的产物:打包导出的二进制(`.zip`/`.skill`等)、编辑器/系统临时文件、体积异常大的文件
|
||||
- 用`git add -A`/`git add .`扫进来的、用户可能还不想提交的无关改动——按文件名显式`git add`,不要笼统全加
|
||||
|
||||
发现可疑内容,先列出来问用户怎么处理,不要自作主张排除或包含。
|
||||
|
||||
### 第三步:生成 commit message
|
||||
|
||||
**message 正文(subject/body)用中文写**——除非第一步发现这个仓库历史上明显
|
||||
是用英文写(且不是本技能自己产生的commit),这种情况下跟随仓库现有语言。
|
||||
`type`前缀本身仍用英文标准词(`feat`/`fix`等,这是社区通用约定,不翻译),
|
||||
`scope`用原始的模块/skill名(通常本身就是英文目录名,不需要翻译)。
|
||||
|
||||
格式采用 Conventional Commits:
|
||||
|
||||
```
|
||||
<type>(<scope>): <中文subject>
|
||||
|
||||
<中文body — 可选,只在"为什么"不是一眼能看出来的时候写>
|
||||
```
|
||||
|
||||
- `type` 从改动的实际性质里选,不要凭感觉:`feat`(新功能)、`fix`(修bug)、
|
||||
`refactor`(不改行为的重构)、`docs`(纯文档)、`test`(测试)、`chore`(构建/
|
||||
依赖/杂项)、`style`(纯格式,不改逻辑)——一次提交只涉及一种性质就只写一个
|
||||
type;如果改动明显跨性质(比如"顺手"把无关重构也塞进了这次提交),提醒用户
|
||||
考虑拆成多个提交,而不是硬凑一个笼统的type。
|
||||
- `scope` 可选,涉及单一模块/skill时加上(比如`literature-search-verify`、
|
||||
`paper-writing-grounded`),改动横跨全仓库时省略。
|
||||
- `subject` 祈使/陈述语气均可、不超过~35个汉字、不以句号结尾,说清楚这次改动
|
||||
做了什么,但真正的价值在于讲清楚**为什么**要这么改——如果"为什么"不是从
|
||||
改动本身能一眼看出来的(比如修了一个不明显的bug、调整了一个非默认行为),
|
||||
写进body里,不要只重复diff里已经能看到的内容。
|
||||
- 不要在message里提及和这次改动无关的历史背景或猜测性动机。
|
||||
|
||||
### 第四步:暂存并提交
|
||||
|
||||
- 按第二步筛选后的文件列表显式`git add <file1> <file2> ...`,不要`-A`/`.`
|
||||
- 用 heredoc 传message,保证多行格式正确:
|
||||
```bash
|
||||
git commit -m "$(cat <<'EOF'
|
||||
<type>(<scope>): <subject>
|
||||
|
||||
<body>
|
||||
EOF
|
||||
)"
|
||||
```
|
||||
- 提交后跑一次`git status`确认工作区状态符合预期
|
||||
|
||||
### 第五步:红线
|
||||
|
||||
- 只在用户明确要求提交时才创建commit,不要在完成一次编辑后主动提交
|
||||
- 不新建commit时优先于`--amend`——只有用户明确要求"修改上一个commit"才用amend,而且如果上一次是因为pre-commit hook失败而"没有真正提交成功",绝不能用`--amend`(那样会改到更早的一个commit)
|
||||
- 不加`--no-verify`/`--no-gpg-sign`跳过hook或签名,除非用户明确要求;hook失败时先定位问题、修复后重新`git add`再建一个新commit
|
||||
- 不主动`git push`,除非用户在这次请求里明确说了要推送
|
||||
- 不做`reset --hard`/`checkout --`/`clean -f`等破坏性操作来"顺手"清理提交前发现的问题,发现了就报告给用户决定
|
||||
@ -1,6 +1,6 @@
|
||||
---
|
||||
name: literature-search-verify
|
||||
description: Search academic literature across arXiv, Semantic Scholar, Crossref, and other connected paper-search MCP tools, and independently verify every candidate citation before it is treated as real. Use this whenever the user asks to find papers, search literature on a topic, build a reading list, compile related-work references, check whether a citation or bibliography entry actually exists, or prepare references to import into Zotero or a .bib file — especially in academic writing contexts where a fabricated citation would be a real problem. Also covers guiding the user to the Zotero Connector browser extension for Chinese-language sources (CNKI/知网, Wanfang/万方, VIP/维普) that have no public API and cannot be reached by search tools.
|
||||
description: Search academic literature across arXiv, Semantic Scholar, Crossref, OpenAlex, and other connected paper-search MCP tools, and independently verify every candidate citation before it is treated as real. Use this whenever the user asks to find papers, search literature on a topic, build a reading list, compile related-work references, check whether a citation or bibliography entry actually exists, or prepare references to import into Zotero or a .bib file — especially in academic writing contexts where a fabricated citation would be a real problem. Also covers guiding the user to the Zotero Connector browser extension for Chinese-language sources (CNKI/知网, Wanfang/万方, VIP/维普) that have no public API and cannot be reached by search tools.
|
||||
---
|
||||
|
||||
# 文献检索 + 反幻觉引用核查
|
||||
@ -25,15 +25,34 @@ description: Search academic literature across arXiv, Semantic Scholar, Crossref
|
||||
`scripts/` 目录下已经写好了能直接跑的检索脚本,不依赖任何第三方Python包,也不需要装MCP工具:
|
||||
|
||||
```bash
|
||||
# 始终从项目根目录调用(脚本本身不关心cwd,但--bib-out等输出路径按项目根目录约定拼)。
|
||||
# 一次性搞定:检索 arXiv + Semantic Scholar + Crossref,自动去重、逐条验证,
|
||||
# 并把通过验证的条目写成BibTeX文件——这是应该默认调用的入口
|
||||
python3 scripts/literature_search.py "UAV magnetic compensation Tolles-Lawson" \
|
||||
--max-per-source 8 --bib-out refs.bib
|
||||
mkdir -p output/literature-search-verify/uav_aeromagnetic_compensation
|
||||
python3 .claude/skills/literature-search-verify/scripts/literature_search.py \
|
||||
"UAV magnetic compensation Tolles-Lawson" \
|
||||
--max-per-source 8 \
|
||||
--bib-out output/literature-search-verify/uav_aeromagnetic_compensation/refs.bib
|
||||
```
|
||||
|
||||
正常情况下**只需要跑这一条命令**,它内部会依次调用 `search_arxiv.py`、`search_semantic_scholar.py`、`search_crossref.py` 做检索,再对每条合并后的候选文献跑 `verify_citation.py` 做交叉验证,输出一份JSON报告(每条候选都带`verdict`字段)。如果只是想单独查一个来源,或者针对某一条文献单独复核,再分别调用对应的单个脚本(用法见每个脚本文件开头的docstring)。
|
||||
草稿路径(检索报告JSON、下载的PDF、中间生成的bib)统一落在项目根目录下的
|
||||
`output/literature-search-verify/`,不要写回技能自己的目录里(`.claude/skills/`
|
||||
应该只放技能代码,不放运行时产生的草稿数据)。如果项目里以后还有别的技能也
|
||||
需要草稿区,各自建 `output/<skill-name>/` 子目录,互不混放。
|
||||
|
||||
如果这些脚本因为网络原因跑不动(比如内网/代理限制导致连不上 arxiv.org、semanticscholar.org、crossref.org),`literature_search.py` 会把每个来源的报错单独记在`search_errors`里而不是直接崩溃——这时候老实告诉用户"检索脚本连不上网络,以下是报错信息",不要退回去凭记忆编文献。如果用户这边确实连不上这几个学术API域名,才退回到 web_search 工具,并在结果里明确标注"来自通用网络搜索的补充结果,未经过脚本的交叉验证流程,置信度较低"。
|
||||
**同一个技能内部也要按主题分开,不要所有检索会话都堆在同一层**:开始一个新
|
||||
主题的检索时,先按"和第六步归档时同样的规则"把主题名转成 slug(小写、非
|
||||
字母数字换成下划线,比如"UAV aeromagnetic compensation" → `uav_aeromagnetic_compensation`),
|
||||
建 `output/literature-search-verify/<topic-slug>/` 子目录,这一整个主题下
|
||||
不管跑多少轮检索、多少条不同的query,原始JSON/bib草稿都写进这一个子目录里
|
||||
(文件名可以随意区分轮次,比如`q1.json`/`q2.json`),不要用不带主题区分的
|
||||
通用文件名散落在`output/literature-search-verify/`根下。这样同一个主题的
|
||||
草稿和第六步归档产出的`references/<topic-slug>/`目录能通过同一个slug对上号,
|
||||
之后回来补充检索同一主题时也知道去哪个子目录续。
|
||||
|
||||
正常情况下**只需要跑这一条命令**,它内部会依次调用 `search_arxiv.py`、`search_semantic_scholar.py`、`search_crossref.py`、`search_openalex.py` 做检索,再对每条合并后的候选文献跑 `verify_citation.py` 做交叉验证,输出一份JSON报告(每条候选都带`verdict`字段)。如果只是想单独查一个来源,或者针对某一条文献单独复核,再分别调用对应的单个脚本(用法见每个脚本文件开头的docstring)。OpenAlex(`search_openalex.py`)是在 arXiv/Semantic Scholar/Crossref 之外新增的第四个源,免费、无需API key,覆盖面比单独的 Crossref 更广(聚合了包括 IEEE 在内的绝大多数出版商元数据),还能拿到开放获取PDF直链;可以设置`OPENALEX_MAILTO`环境变量为一个邮箱地址进入OpenAlex的"polite pool"换取更稳的限额,不设置也能正常用。
|
||||
|
||||
所有脚本的网络请求都经过 `http_utils.py` 统一封装:遇到 429(限流)或 5xx 会按指数退避自动重试几次(优先遵守服务端返回的`Retry-After`),不需要每次手动重试;`verify_citation.py`的跨源标题核查更是同时查 Semantic Scholar 和 OpenAlex 两个独立源、任一命中即算通过,不会再出现"S2一限流,整批候选的跨源核查全部退化成skipped"的情况(这是实际发生过的问题,现在已经用两个源+重试解决)。即便如此,如果这些脚本因为网络原因跑不动(比如内网/代理限制导致连不上这几个学术API域名),`literature_search.py` 会把每个来源的报错单独记在`search_errors`里而不是直接崩溃——这时候老实告诉用户"检索脚本连不上网络,以下是报错信息",不要退回去凭记忆编文献。如果用户这边确实连不上这几个学术API域名,才退回到 web_search 工具,并在结果里明确标注"来自通用网络搜索的补充结果,未经过脚本的交叉验证流程,置信度较低"。
|
||||
|
||||
如果用户已经连了 paper-search-mcp / scholar_mcp_server 这类MCP工具,可以补充用来扩大覆盖面(比如它们能覆盖PubMed、能直接下载PDF),但**不能替代**`verify_citation.py`的交叉验证这一步——MCP搜到的候选一样要过一遍验证,不能因为是工具搜出来的就默认可信。
|
||||
|
||||
@ -43,7 +62,7 @@ python3 scripts/literature_search.py "UAV magnetic compensation Tolles-Lawson" \
|
||||
|
||||
1. **arXiv ID 独立核实**:如果有 arXiv ID,反查一次 arXiv API,确认这个ID真的存在且标题对得上——一个编造的ID在这一步会直接暴露。
|
||||
2. **DOI 独立核实**:如果有 DOI,反查一次 Crossref,确认这个 DOI 真的能解析出对应文献。
|
||||
3. **跨源标题复核**:不管有没有ID,单独拿标题去 Semantic Scholar 搜一次,要求返回的标题跟候选标题高度相似(相似度≥0.9)——这一步专门用来抓"标题作者读起来很像真的,但其实是编出来的"这种情况。
|
||||
3. **跨源标题复核(双源)**:不管有没有ID,单独拿标题分别去 Semantic Scholar 和 OpenAlex 各搜一次,要求返回的标题跟候选标题高度相似(相似度≥0.9)——这一步专门用来抓"标题作者读起来很像真的,但其实是编出来的"这种情况。两个源里任一个命中相似度达标就算通过;只有当两个源都明确没找到匹配(不是因为网络问题被跳过)时才算这一项核查失败——避免某个源覆盖不全(比如论文太新还没被其中一个索引收录)被误判成"编造"。
|
||||
|
||||
每条候选最后会带一个`verdict`:
|
||||
- **verified**:至少一项独立核查通过,而且没有任何一项核查明确失败
|
||||
@ -66,15 +85,19 @@ MCP 检索工具覆盖的是 arXiv/Semantic Scholar/Crossref 这类有公开 API
|
||||
|
||||
### 第六步:归档
|
||||
|
||||
`output/` 目录只是脚本运行时的草稿区——里面混着每一轮探索性检索的原始JSON(包括被过滤掉的噪声,比如"Tolles""Lawson"被当成人名匹配出的无关文献),不适合作为最终交付物,而且随着会话增多会越堆越乱、也不方便下次会话或用户直接翻阅。
|
||||
`<project-root>/output/literature-search-verify/` 只是脚本运行时的草稿区——里面
|
||||
混着每一轮探索性检索的原始JSON(包括被过滤掉的噪声,比如"Tolles""Lawson"被当成
|
||||
人名匹配出的无关文献),不适合作为最终交付物,而且随着会话增多会越堆越乱、也不
|
||||
方便下次会话或用户直接翻阅。
|
||||
|
||||
所以每次整理出一份**稳定可信的参考文献列表**(不管是第一轮检索还是后续多轮补充检索合并后的结果)之后,调用归档脚本把它固化到项目级目录,而不是留在技能自己的`output/`里:
|
||||
所以每次整理出一份**稳定可信的参考文献列表**(不管是第一轮检索还是后续多轮补充检索合并后的结果)之后,调用归档脚本把它固化到项目级目录,而不是留在草稿区里:
|
||||
|
||||
```bash
|
||||
python3 scripts/archive_references.py "UAV aeromagnetic compensation" \
|
||||
--bib output/uav_aeromagnetic_compensation_final.bib \
|
||||
python3 .claude/skills/literature-search-verify/scripts/archive_references.py \
|
||||
"UAV aeromagnetic compensation" \
|
||||
--bib output/literature-search-verify/uav_aeromagnetic_compensation/uav_aeromagnetic_compensation_final.bib \
|
||||
--project-root . \
|
||||
--pdfs-dir output/pdfs \
|
||||
--pdfs-dir output/literature-search-verify/uav_aeromagnetic_compensation/pdfs \
|
||||
--suspect "某条可疑文献标题|不建议引用的具体原因" \
|
||||
--notes "检索覆盖了哪些方向、哪些方向搜了但没结果、中文文献缺口提醒等"
|
||||
```
|
||||
@ -89,6 +112,21 @@ python3 scripts/archive_references.py "UAV aeromagnetic compensation" \
|
||||
- 同一个`topic`名字多次调用会往同一个归档目录里覆盖更新(bib和README会被覆盖,pdfs按文件名去重合并),所以后续检索到更多文献后可以直接对同一个topic重新跑一遍归档脚本来更新,不需要手动合并。
|
||||
- 这一步做完之后可以明确告诉用户归档目录的路径,方便他们后续在`paper-writing-grounded`阶段直接引用。
|
||||
|
||||
## 维护者备注:单元测试
|
||||
|
||||
`scripts/tests/` 下有针对核心正确性逻辑的单元测试(不发真实网络请求,全部用
|
||||
mock):`verify_citation.py` 的三档判定和双源标题核查合并逻辑、
|
||||
`archive_references.py` 的 bib 解析(含嵌套花括号、中文主题名slugify等边界
|
||||
情况)、`literature_search.py` 的候选去重合并与BibTeX生成、`http_utils.py`
|
||||
的限流重试。改动这几个脚本后应该跑一遍:
|
||||
|
||||
```bash
|
||||
python3 -m unittest discover -s .claude/skills/literature-search-verify/scripts/tests -t .claude/skills/literature-search-verify/scripts
|
||||
```
|
||||
|
||||
只用标准库`unittest`,不引入pytest等第三方测试框架,和脚本本身"不依赖第三方
|
||||
包"的原则保持一致。
|
||||
|
||||
## 和 paper-writing-grounded 技能的配合
|
||||
|
||||
这个技能负责把"真实存在、经过核实的文献"整理好并生成 BibTeX;写作阶段的 paper-writing-grounded 技能会直接消费这里产出的 citation key,正文引用只能来自这里核实过的条目,不会凭空生成新的引用。两个技能配合使用时,建议先跑完这个技能、拿到稳定的参考文献列表,再进入写作。
|
||||
|
||||
@ -13,11 +13,14 @@ what's there without re-reading the conversation that produced it.
|
||||
|
||||
No third-party dependencies; uses only the standard library.
|
||||
|
||||
CLI usage:
|
||||
python3 archive_references.py "UAV aeromagnetic compensation" \\
|
||||
--bib output/uav_aeromagnetic_compensation_final.bib \\
|
||||
CLI usage (run from the project root; scratch input lives under
|
||||
output/literature-search-verify/<topic-slug>/, matching the slug this
|
||||
script derives from the topic argument):
|
||||
python3 .claude/skills/literature-search-verify/scripts/archive_references.py \\
|
||||
"UAV aeromagnetic compensation" \\
|
||||
--bib output/literature-search-verify/uav_aeromagnetic_compensation/uav_aeromagnetic_compensation_final.bib \\
|
||||
--project-root . \\
|
||||
--pdfs-dir output/pdfs \\
|
||||
--pdfs-dir output/literature-search-verify/uav_aeromagnetic_compensation/pdfs \\
|
||||
--suspect "Some fabricated-looking title|DOI resolves but venue is topically unrelated" \\
|
||||
--notes "Kalman-filter and GA/PSO angles searched, no on-topic hits found."
|
||||
|
||||
@ -32,8 +35,12 @@ from datetime import date
|
||||
|
||||
|
||||
def slugify(text):
|
||||
"""ASCII gets lowercased; CJK characters are kept as-is (there's no
|
||||
reversible case-fold for them) so an all-Chinese topic name doesn't
|
||||
collapse to nothing and fall back to the generic "references" name,
|
||||
colliding with every other Chinese-named topic archived in this repo."""
|
||||
text = text.strip().lower()
|
||||
text = re.sub(r"[^a-z0-9]+", "_", text)
|
||||
text = re.sub(r"[^a-z0-9一-鿿]+", "_", text)
|
||||
return text.strip("_")[:60] or "references"
|
||||
|
||||
|
||||
|
||||
@ -0,0 +1,46 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Shared HTTP GET helper with exponential-backoff retry for rate limiting (429)
|
||||
and transient server errors (5xx). Stdlib only.
|
||||
|
||||
Every search_*.py / verify_citation.py network call should go through this
|
||||
instead of calling urllib.request.urlopen directly, so a single source being
|
||||
temporarily rate-limited doesn't quietly degrade a whole batch of candidates
|
||||
to "unverified" the way Semantic Scholar's 429s did before this existed.
|
||||
|
||||
Importable:
|
||||
from http_utils import fetch
|
||||
"""
|
||||
import time
|
||||
import urllib.request
|
||||
import urllib.error
|
||||
|
||||
RETRYABLE_CODES = {429, 500, 502, 503, 504}
|
||||
|
||||
|
||||
def _wait_seconds(http_error, attempt, base=1.0, cap=20.0):
|
||||
retry_after = http_error.headers.get("Retry-After") if http_error.headers else None
|
||||
if retry_after:
|
||||
try:
|
||||
return min(float(retry_after), cap)
|
||||
except ValueError:
|
||||
pass
|
||||
return min(base * (2 ** attempt), cap)
|
||||
|
||||
|
||||
def fetch(url, headers=None, timeout=20, max_retries=4):
|
||||
"""GET url, retrying on 429/5xx with exponential backoff (honoring a
|
||||
Retry-After header when the server sends one). Returns the raw response
|
||||
bytes. Raises the underlying urllib error if retries are exhausted or the
|
||||
error isn't retryable (e.g. 404)."""
|
||||
req = urllib.request.Request(url, headers=headers or {})
|
||||
attempt = 0
|
||||
while True:
|
||||
try:
|
||||
with urllib.request.urlopen(req, timeout=timeout) as resp:
|
||||
return resp.read()
|
||||
except urllib.error.HTTPError as e:
|
||||
if e.code not in RETRYABLE_CODES or attempt >= max_retries:
|
||||
raise
|
||||
time.sleep(_wait_seconds(e, attempt))
|
||||
attempt += 1
|
||||
@ -1,8 +1,9 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
End-to-end literature search: query arXiv + Semantic Scholar + Crossref,
|
||||
merge/dedupe candidates, independently verify each one, and emit both a
|
||||
human-readable report and BibTeX for the entries that passed verification.
|
||||
End-to-end literature search: query arXiv + Semantic Scholar + Crossref +
|
||||
OpenAlex, merge/dedupe candidates, independently verify each one, and emit
|
||||
both a human-readable report and BibTeX for the entries that passed
|
||||
verification.
|
||||
|
||||
This is the one script Claude should actually call for a normal literature
|
||||
search -- the individual search_*.py / verify_citation.py scripts exist
|
||||
@ -28,6 +29,7 @@ sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
|
||||
from search_arxiv import search_arxiv
|
||||
from search_semantic_scholar import search_s2
|
||||
from search_crossref import search_crossref
|
||||
from search_openalex import search_openalex
|
||||
from verify_citation import verify
|
||||
|
||||
|
||||
@ -117,7 +119,13 @@ def to_bibtex(candidate, key):
|
||||
def run(query, max_per_source=8):
|
||||
all_results = []
|
||||
errors = {}
|
||||
for name, fn in (("arxiv", search_arxiv), ("semantic_scholar", search_s2), ("crossref", search_crossref)):
|
||||
sources = (
|
||||
("arxiv", search_arxiv),
|
||||
("semantic_scholar", search_s2),
|
||||
("crossref", search_crossref),
|
||||
("openalex", search_openalex),
|
||||
)
|
||||
for name, fn in sources:
|
||||
try:
|
||||
all_results.extend(fn(query, max_per_source))
|
||||
except Exception as e:
|
||||
|
||||
@ -9,12 +9,15 @@ Importable:
|
||||
from search_arxiv import search_arxiv
|
||||
"""
|
||||
import sys
|
||||
import os
|
||||
import json
|
||||
import argparse
|
||||
import urllib.request
|
||||
import urllib.parse
|
||||
import xml.etree.ElementTree as ET
|
||||
|
||||
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
|
||||
from http_utils import fetch
|
||||
|
||||
ARXIV_API = "http://export.arxiv.org/api/query"
|
||||
NS = {"atom": "http://www.w3.org/2005/Atom"}
|
||||
|
||||
@ -28,8 +31,7 @@ def search_arxiv(query, max_results=10, timeout=20):
|
||||
"sortOrder": "descending",
|
||||
}
|
||||
url = f"{ARXIV_API}?{urllib.parse.urlencode(params)}"
|
||||
with urllib.request.urlopen(url, timeout=timeout) as resp:
|
||||
data = resp.read()
|
||||
data = fetch(url, timeout=timeout)
|
||||
root = ET.fromstring(data)
|
||||
results = []
|
||||
for entry in root.findall("atom:entry", NS):
|
||||
|
||||
@ -10,11 +10,14 @@ Importable:
|
||||
from search_crossref import search_crossref
|
||||
"""
|
||||
import sys
|
||||
import os
|
||||
import json
|
||||
import argparse
|
||||
import urllib.request
|
||||
import urllib.parse
|
||||
|
||||
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
|
||||
from http_utils import fetch
|
||||
|
||||
CROSSREF_API = "https://api.crossref.org/works"
|
||||
UA = "literature-search-verify-skill/1.0 (mailto:research-assistant@example.com)"
|
||||
|
||||
@ -22,9 +25,7 @@ UA = "literature-search-verify-skill/1.0 (mailto:research-assistant@example.com)
|
||||
def search_crossref(query, max_results=10, timeout=20):
|
||||
params = {"query": query, "rows": max_results}
|
||||
url = f"{CROSSREF_API}?{urllib.parse.urlencode(params)}"
|
||||
req = urllib.request.Request(url, headers={"User-Agent": UA})
|
||||
with urllib.request.urlopen(req, timeout=timeout) as resp:
|
||||
data = json.loads(resp.read())
|
||||
data = json.loads(fetch(url, headers={"User-Agent": UA}, timeout=timeout))
|
||||
results = []
|
||||
for item in data.get("message", {}).get("items", []) or []:
|
||||
titles = item.get("title") or []
|
||||
|
||||
@ -0,0 +1,92 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Search the OpenAlex Works API. No API key required -- set the OPENALEX_MAILTO
|
||||
environment variable to an email address to use OpenAlex's "polite pool" for
|
||||
higher and more reliable rate limits (OpenAlex's own recommendation, not a
|
||||
credential). Coverage is broader than Crossref alone (includes arXiv, IEEE,
|
||||
and most publishers' metadata in one place) and each result carries an
|
||||
open-access PDF link when one exists.
|
||||
|
||||
CLI usage:
|
||||
python3 search_openalex.py "UAV magnetic compensation" --max 10
|
||||
|
||||
Importable:
|
||||
from search_openalex import search_openalex
|
||||
"""
|
||||
import sys
|
||||
import os
|
||||
import json
|
||||
import argparse
|
||||
import urllib.parse
|
||||
|
||||
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
|
||||
from http_utils import fetch
|
||||
|
||||
OPENALEX_API = "https://api.openalex.org/works"
|
||||
SELECT_FIELDS = ",".join([
|
||||
"doi", "title", "display_name", "publication_year", "authorships",
|
||||
"primary_location", "open_access", "cited_by_count", "abstract_inverted_index",
|
||||
])
|
||||
|
||||
|
||||
def _reconstruct_abstract(inverted_index):
|
||||
"""OpenAlex returns abstracts as a word -> [positions] inverted index
|
||||
(for copyright reasons) instead of plain text; rebuild the plain text."""
|
||||
if not inverted_index:
|
||||
return None
|
||||
positions = {}
|
||||
for word, idxs in inverted_index.items():
|
||||
for i in idxs:
|
||||
positions[i] = word
|
||||
if not positions:
|
||||
return None
|
||||
return " ".join(positions[i] for i in sorted(positions))
|
||||
|
||||
|
||||
def search_openalex(query, max_results=10, timeout=20):
|
||||
params = {"search": query, "per_page": max_results, "select": SELECT_FIELDS}
|
||||
mailto = os.environ.get("OPENALEX_MAILTO")
|
||||
if mailto:
|
||||
params["mailto"] = mailto
|
||||
url = f"{OPENALEX_API}?{urllib.parse.urlencode(params)}"
|
||||
data = json.loads(fetch(url, timeout=timeout))
|
||||
results = []
|
||||
for item in data.get("results", []) or []:
|
||||
title = item.get("title") or item.get("display_name")
|
||||
authors = [
|
||||
a.get("author", {}).get("display_name")
|
||||
for a in (item.get("authorships") or [])
|
||||
if a.get("author")
|
||||
]
|
||||
doi = item.get("doi")
|
||||
if doi and doi.startswith("https://doi.org/"):
|
||||
doi = doi[len("https://doi.org/"):]
|
||||
primary_location = item.get("primary_location") or {}
|
||||
source = primary_location.get("source") or {}
|
||||
open_access = item.get("open_access") or {}
|
||||
results.append({
|
||||
"source": "openalex",
|
||||
"title": title,
|
||||
"authors": authors,
|
||||
"year": item.get("publication_year"),
|
||||
"venue": source.get("display_name"),
|
||||
"doi": doi,
|
||||
"arxiv_id": None,
|
||||
"abstract": _reconstruct_abstract(item.get("abstract_inverted_index")),
|
||||
"citation_count": item.get("cited_by_count"),
|
||||
"pdf_url": open_access.get("oa_url") or primary_location.get("pdf_url"),
|
||||
})
|
||||
return results
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
ap = argparse.ArgumentParser(description=__doc__)
|
||||
ap.add_argument("query")
|
||||
ap.add_argument("--max", type=int, default=10)
|
||||
args = ap.parse_args()
|
||||
try:
|
||||
out = search_openalex(args.query, args.max)
|
||||
print(json.dumps(out, ensure_ascii=False, indent=2))
|
||||
except Exception as e:
|
||||
print(json.dumps({"error": str(e)}, ensure_ascii=False))
|
||||
sys.exit(1)
|
||||
@ -13,9 +13,11 @@ import sys
|
||||
import os
|
||||
import json
|
||||
import argparse
|
||||
import urllib.request
|
||||
import urllib.parse
|
||||
|
||||
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
|
||||
from http_utils import fetch
|
||||
|
||||
S2_API = "https://api.semanticscholar.org/graph/v1/paper/search"
|
||||
FIELDS = "title,authors,year,venue,externalIds,abstract,citationCount"
|
||||
|
||||
@ -23,12 +25,11 @@ FIELDS = "title,authors,year,venue,externalIds,abstract,citationCount"
|
||||
def search_s2(query, max_results=10, timeout=20):
|
||||
params = {"query": query, "limit": max_results, "fields": FIELDS}
|
||||
url = f"{S2_API}?{urllib.parse.urlencode(params)}"
|
||||
req = urllib.request.Request(url)
|
||||
headers = {}
|
||||
api_key = os.environ.get("S2_API_KEY")
|
||||
if api_key:
|
||||
req.add_header("x-api-key", api_key)
|
||||
with urllib.request.urlopen(req, timeout=timeout) as resp:
|
||||
data = json.loads(resp.read())
|
||||
headers["x-api-key"] = api_key
|
||||
data = json.loads(fetch(url, headers=headers, timeout=timeout))
|
||||
results = []
|
||||
for p in data.get("data", []) or []:
|
||||
ext = p.get("externalIds") or {}
|
||||
|
||||
@ -0,0 +1,99 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Unit tests for archive_references.py's slugify() and the minimal bib parser,
|
||||
covering the edge cases that are easy to get wrong silently: CJK topic names
|
||||
and nested-brace LaTeX field values (e.g. \\ensuremath{\\epsilon}).
|
||||
|
||||
Run: python -m unittest discover -s .claude/skills/literature-search-verify/scripts
|
||||
"""
|
||||
import os
|
||||
import sys
|
||||
import tempfile
|
||||
import unittest
|
||||
|
||||
sys.path.insert(0, os.path.join(os.path.dirname(os.path.abspath(__file__)), ".."))
|
||||
import archive_references as ar
|
||||
|
||||
|
||||
class TestSlugify(unittest.TestCase):
|
||||
def test_ascii_topic(self):
|
||||
self.assertEqual(ar.slugify("UAV aeromagnetic compensation"), "uav_aeromagnetic_compensation")
|
||||
|
||||
def test_chinese_topic_is_preserved_not_collapsed_to_generic_name(self):
|
||||
slug = ar.slugify("无人机磁补偿")
|
||||
self.assertNotEqual(slug, "references")
|
||||
self.assertIn("补偿", slug)
|
||||
|
||||
def test_mixed_chinese_and_english(self):
|
||||
slug = ar.slugify("无人机磁补偿 Tolles-Lawson")
|
||||
self.assertTrue(slug.endswith("tolles_lawson"))
|
||||
|
||||
def test_blank_topic_falls_back_to_references(self):
|
||||
self.assertEqual(ar.slugify(" "), "references")
|
||||
|
||||
def test_truncated_to_60_chars(self):
|
||||
self.assertEqual(len(ar.slugify("a" * 100)), 60)
|
||||
|
||||
|
||||
class TestParseBibEntries(unittest.TestCase):
|
||||
def _write_bib(self, content):
|
||||
fd, path = tempfile.mkstemp(suffix=".bib")
|
||||
with os.fdopen(fd, "w", encoding="utf-8") as f:
|
||||
f.write(content)
|
||||
self.addCleanup(os.remove, path)
|
||||
return path
|
||||
|
||||
def test_nested_braces_in_field_value_do_not_truncate_it(self):
|
||||
bib = (
|
||||
"@article{wu2017,\n"
|
||||
" title = {Aeromagnetic gradient compensation using \\ensuremath{\\epsilon}-SVR},\n"
|
||||
" journal = {Journal of Applied Remote Sensing},\n"
|
||||
" year = {2017},\n"
|
||||
" doi = {10.1117/1.jrs.11.025012},\n"
|
||||
"}"
|
||||
)
|
||||
entries = ar.parse_bib_entries(self._write_bib(bib))
|
||||
self.assertEqual(len(entries), 1)
|
||||
self.assertIn("\\ensuremath{\\epsilon}-SVR", entries[0]["title"])
|
||||
self.assertEqual(entries[0]["year"], "2017")
|
||||
|
||||
def test_multiple_entries_and_optional_fields(self):
|
||||
bib = (
|
||||
"@article{a2020,\n"
|
||||
" title = {First paper},\n"
|
||||
" year = {2020},\n"
|
||||
"}\n\n"
|
||||
"@article{b2021,\n"
|
||||
" title = {Second paper},\n"
|
||||
" year = {2021},\n"
|
||||
" note = {some note},\n"
|
||||
"}"
|
||||
)
|
||||
entries = ar.parse_bib_entries(self._write_bib(bib))
|
||||
self.assertEqual([e["key"] for e in entries], ["a2020", "b2021"])
|
||||
self.assertNotIn("note", entries[0])
|
||||
self.assertEqual(entries[1]["note"], "some note")
|
||||
|
||||
|
||||
class TestYearSortKey(unittest.TestCase):
|
||||
def test_entries_without_a_parseable_year_sort_last(self):
|
||||
entries = [{"key": "b", "year": ""}, {"key": "a", "year": "1999"}]
|
||||
entries.sort(key=ar.year_sort_key)
|
||||
self.assertEqual([e["key"] for e in entries], ["a", "b"])
|
||||
|
||||
|
||||
class TestBuildReadme(unittest.TestCase):
|
||||
def test_omits_optional_sections_when_not_given(self):
|
||||
readme = ar.build_readme("Topic", [], 0, [], None)
|
||||
self.assertNotIn("Flagged during search", readme)
|
||||
self.assertNotIn("Search coverage notes", readme)
|
||||
|
||||
def test_includes_suspect_and_notes_when_given(self):
|
||||
readme = ar.build_readme("Topic", [], 0, ["Bad title|dubious venue"], "coverage notes here")
|
||||
self.assertIn("Bad title", readme)
|
||||
self.assertIn("dubious venue", readme)
|
||||
self.assertIn("coverage notes here", readme)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@ -0,0 +1,103 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Unit tests for http_utils.fetch's retry/backoff behavior. All network calls
|
||||
are mocked -- these never touch the real network.
|
||||
|
||||
Run: python -m unittest discover -s .claude/skills/literature-search-verify/scripts
|
||||
"""
|
||||
import os
|
||||
import sys
|
||||
import unittest
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
from email.message import Message
|
||||
from unittest import mock
|
||||
|
||||
sys.path.insert(0, os.path.join(os.path.dirname(os.path.abspath(__file__)), ".."))
|
||||
import http_utils
|
||||
|
||||
|
||||
def _http_error(code, retry_after=None):
|
||||
hdrs = Message()
|
||||
if retry_after is not None:
|
||||
hdrs["Retry-After"] = str(retry_after)
|
||||
return urllib.error.HTTPError("http://example.com", code, "err", hdrs, None)
|
||||
|
||||
|
||||
class FakeResponse:
|
||||
def __init__(self, data):
|
||||
self._data = data
|
||||
|
||||
def read(self):
|
||||
return self._data
|
||||
|
||||
def __enter__(self):
|
||||
return self
|
||||
|
||||
def __exit__(self, *exc_info):
|
||||
return False
|
||||
|
||||
|
||||
class TestFetch(unittest.TestCase):
|
||||
@mock.patch("urllib.request.urlopen")
|
||||
@mock.patch("http_utils.time.sleep", return_value=None)
|
||||
def test_succeeds_first_try_without_sleeping(self, mock_sleep, mock_urlopen):
|
||||
mock_urlopen.return_value = FakeResponse(b"ok")
|
||||
result = http_utils.fetch("http://example.com")
|
||||
self.assertEqual(result, b"ok")
|
||||
mock_sleep.assert_not_called()
|
||||
|
||||
@mock.patch("urllib.request.urlopen")
|
||||
@mock.patch("http_utils.time.sleep", return_value=None)
|
||||
def test_retries_on_429_then_succeeds(self, mock_sleep, mock_urlopen):
|
||||
mock_urlopen.side_effect = [_http_error(429), FakeResponse(b"ok")]
|
||||
result = http_utils.fetch("http://example.com", max_retries=3)
|
||||
self.assertEqual(result, b"ok")
|
||||
self.assertEqual(mock_urlopen.call_count, 2)
|
||||
mock_sleep.assert_called_once()
|
||||
|
||||
@mock.patch("urllib.request.urlopen")
|
||||
@mock.patch("http_utils.time.sleep", return_value=None)
|
||||
def test_retries_on_5xx(self, mock_sleep, mock_urlopen):
|
||||
mock_urlopen.side_effect = [_http_error(503), FakeResponse(b"ok")]
|
||||
result = http_utils.fetch("http://example.com", max_retries=3)
|
||||
self.assertEqual(result, b"ok")
|
||||
|
||||
@mock.patch("urllib.request.urlopen")
|
||||
@mock.patch("http_utils.time.sleep", return_value=None)
|
||||
def test_raises_after_exhausting_retries(self, mock_sleep, mock_urlopen):
|
||||
mock_urlopen.side_effect = [_http_error(429)] * 10
|
||||
with self.assertRaises(urllib.error.HTTPError):
|
||||
http_utils.fetch("http://example.com", max_retries=2)
|
||||
self.assertEqual(mock_urlopen.call_count, 3) # initial attempt + 2 retries
|
||||
|
||||
@mock.patch("urllib.request.urlopen")
|
||||
@mock.patch("http_utils.time.sleep", return_value=None)
|
||||
def test_non_retryable_status_raises_immediately(self, mock_sleep, mock_urlopen):
|
||||
mock_urlopen.side_effect = _http_error(404)
|
||||
with self.assertRaises(urllib.error.HTTPError):
|
||||
http_utils.fetch("http://example.com")
|
||||
self.assertEqual(mock_urlopen.call_count, 1)
|
||||
mock_sleep.assert_not_called()
|
||||
|
||||
|
||||
class TestWaitSeconds(unittest.TestCase):
|
||||
def test_honors_retry_after_header(self):
|
||||
err = _http_error(429, retry_after=5)
|
||||
self.assertEqual(http_utils._wait_seconds(err, attempt=0), 5.0)
|
||||
|
||||
def test_falls_back_to_exponential_backoff_without_header(self):
|
||||
err = _http_error(429)
|
||||
self.assertAlmostEqual(http_utils._wait_seconds(err, attempt=2, base=1.0), 4.0)
|
||||
|
||||
def test_caps_wait_time(self):
|
||||
err = _http_error(429, retry_after=999)
|
||||
self.assertEqual(http_utils._wait_seconds(err, attempt=0, cap=20.0), 20.0)
|
||||
|
||||
def test_ignores_unparseable_retry_after(self):
|
||||
err = _http_error(429, retry_after="not-a-number")
|
||||
self.assertAlmostEqual(http_utils._wait_seconds(err, attempt=0, base=1.0), 1.0)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@ -0,0 +1,67 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Unit tests for literature_search.py's candidate merging and BibTeX generation.
|
||||
|
||||
Run: python -m unittest discover -s .claude/skills/literature-search-verify/scripts
|
||||
"""
|
||||
import os
|
||||
import sys
|
||||
import unittest
|
||||
|
||||
sys.path.insert(0, os.path.join(os.path.dirname(os.path.abspath(__file__)), ".."))
|
||||
import literature_search as ls
|
||||
|
||||
|
||||
class TestMergeCandidates(unittest.TestCase):
|
||||
def test_dedupes_same_paper_across_sources_and_fills_missing_fields(self):
|
||||
items = [
|
||||
{"source": "arxiv", "title": "Deep Learning for Aeromagnetic Compensation",
|
||||
"arxiv_id": "2401.00001", "doi": None, "year": "2024"},
|
||||
{"source": "crossref", "title": "Deep Learning for Aeromagnetic Compensation",
|
||||
"arxiv_id": None, "doi": "10.1/x", "year": 2024, "venue": "Some Journal"},
|
||||
]
|
||||
merged = ls.merge_candidates(items)
|
||||
self.assertEqual(len(merged), 1)
|
||||
self.assertEqual(merged[0]["arxiv_id"], "2401.00001")
|
||||
self.assertEqual(merged[0]["doi"], "10.1/x")
|
||||
self.assertEqual(sorted(merged[0]["sources"]), ["arxiv", "crossref"])
|
||||
|
||||
def test_distinct_titles_are_not_merged(self):
|
||||
items = [
|
||||
{"source": "arxiv", "title": "Paper About Cats"},
|
||||
{"source": "crossref", "title": "Paper About Dogs"},
|
||||
]
|
||||
self.assertEqual(len(ls.merge_candidates(items)), 2)
|
||||
|
||||
|
||||
class TestBibtexKey(unittest.TestCase):
|
||||
def test_collision_gets_letter_suffix(self):
|
||||
used = set()
|
||||
key1 = ls.make_bibtex_key({"authors": ["Jane Smith"], "year": "2020"}, used)
|
||||
key2 = ls.make_bibtex_key({"authors": ["John Smith"], "year": "2020"}, used)
|
||||
self.assertEqual(key1, "smith2020")
|
||||
self.assertEqual(key2, "smith2020a")
|
||||
|
||||
def test_no_authors_falls_back_to_unknown(self):
|
||||
key = ls.make_bibtex_key({"authors": [], "year": "2020"}, set())
|
||||
self.assertEqual(key, "unknown2020")
|
||||
|
||||
|
||||
class TestToBibtex(unittest.TestCase):
|
||||
def test_arxiv_only_candidate_uses_misc_with_eprint(self):
|
||||
cand = {"authors": ["A B"], "title": "T", "year": "2024",
|
||||
"arxiv_id": "2401.00001", "venue": None, "doi": None}
|
||||
bib = ls.to_bibtex(cand, "b2024")
|
||||
self.assertTrue(bib.startswith("@misc{b2024,"))
|
||||
self.assertIn("eprint = {2401.00001}", bib)
|
||||
|
||||
def test_candidate_with_venue_uses_article_with_doi(self):
|
||||
cand = {"authors": ["A B"], "title": "T", "year": "2024",
|
||||
"arxiv_id": None, "venue": "Some Journal", "doi": "10.1/x"}
|
||||
bib = ls.to_bibtex(cand, "b2024")
|
||||
self.assertTrue(bib.startswith("@article{b2024,"))
|
||||
self.assertIn("doi = {10.1/x}", bib)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@ -0,0 +1,94 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Unit tests for verify_citation's verdict aggregation logic -- this is the
|
||||
anti-hallucination guarantee, so its three-way verified/suspect/unverified
|
||||
decision needs to stay correct independent of any real network call. All
|
||||
individual check_* functions are mocked; only the aggregation logic itself
|
||||
(verify() and _combine_title_checks()) is under test.
|
||||
|
||||
Run: python -m unittest discover -s .claude/skills/literature-search-verify/scripts
|
||||
"""
|
||||
import os
|
||||
import sys
|
||||
import unittest
|
||||
from unittest import mock
|
||||
|
||||
sys.path.insert(0, os.path.join(os.path.dirname(os.path.abspath(__file__)), ".."))
|
||||
import verify_citation as vc
|
||||
|
||||
|
||||
class TestCombineTitleChecks(unittest.TestCase):
|
||||
"""Two independent title sources (S2 + OpenAlex) shouldn't let one
|
||||
source's incomplete coverage outvote the other's confirmation."""
|
||||
|
||||
def test_both_pass_picks_higher_similarity(self):
|
||||
s2 = {"status": "pass", "similarity": 0.95, "matched_title": "A"}
|
||||
oa = {"status": "pass", "similarity": 0.99, "matched_title": "B"}
|
||||
combined = vc._combine_title_checks(s2, oa)
|
||||
self.assertEqual(combined["status"], "pass")
|
||||
self.assertEqual(combined["matched_title"], "B")
|
||||
|
||||
def test_one_pass_one_fail_still_passes(self):
|
||||
s2 = {"status": "fail", "reason": "no matching title found on Semantic Scholar"}
|
||||
oa = {"status": "pass", "similarity": 0.95, "matched_title": "B"}
|
||||
self.assertEqual(vc._combine_title_checks(s2, oa)["status"], "pass")
|
||||
|
||||
def test_one_pass_one_skipped_still_passes(self):
|
||||
s2 = {"status": "skipped", "reason": "timeout"}
|
||||
oa = {"status": "pass", "similarity": 0.95, "matched_title": "B"}
|
||||
self.assertEqual(vc._combine_title_checks(s2, oa)["status"], "pass")
|
||||
|
||||
def test_both_fail_is_fail(self):
|
||||
s2 = {"status": "fail", "reason": "x"}
|
||||
oa = {"status": "fail", "reason": "y"}
|
||||
self.assertEqual(vc._combine_title_checks(s2, oa)["status"], "fail")
|
||||
|
||||
def test_fail_plus_skipped_is_still_fail(self):
|
||||
s2 = {"status": "fail", "reason": "x"}
|
||||
oa = {"status": "skipped", "reason": "y"}
|
||||
self.assertEqual(vc._combine_title_checks(s2, oa)["status"], "fail")
|
||||
|
||||
def test_both_skipped_is_skipped_not_fail(self):
|
||||
s2 = {"status": "skipped", "reason": "x"}
|
||||
oa = {"status": "skipped", "reason": "y"}
|
||||
self.assertEqual(vc._combine_title_checks(s2, oa)["status"], "skipped")
|
||||
|
||||
|
||||
class TestVerifyVerdict(unittest.TestCase):
|
||||
def test_all_checks_pass_is_verified(self):
|
||||
with mock.patch.object(vc, "check_arxiv_id", return_value={"status": "pass"}), \
|
||||
mock.patch.object(vc, "check_doi", return_value={"status": "pass"}), \
|
||||
mock.patch.object(vc, "check_title_cross_source_s2", return_value={"status": "pass", "similarity": 1.0}), \
|
||||
mock.patch.object(vc, "check_title_cross_source_openalex", return_value={"status": "pass", "similarity": 1.0}):
|
||||
result = vc.verify(title="T", arxiv_id="1234.5678", doi="10.1/x")
|
||||
self.assertEqual(result["verdict"], "verified")
|
||||
|
||||
def test_one_failing_check_is_suspect_even_if_another_passes(self):
|
||||
with mock.patch.object(vc, "check_arxiv_id", return_value={"status": "fail", "reason": "arXiv ID not found"}), \
|
||||
mock.patch.object(vc, "check_doi", return_value={"status": "pass"}):
|
||||
result = vc.verify(arxiv_id="9999.99999", doi="10.1/x")
|
||||
self.assertEqual(result["verdict"], "suspect")
|
||||
|
||||
def test_all_checks_skipped_is_unverified_not_verified(self):
|
||||
with mock.patch.object(vc, "check_arxiv_id", return_value={"status": "skipped", "reason": "timeout"}):
|
||||
result = vc.verify(arxiv_id="1234.5678")
|
||||
self.assertEqual(result["verdict"], "unverified")
|
||||
|
||||
def test_no_identifiers_given_produces_no_checks_and_unverified(self):
|
||||
result = vc.verify(title=None, arxiv_id=None, doi=None)
|
||||
self.assertEqual(result["checks"], {})
|
||||
self.assertEqual(result["verdict"], "unverified")
|
||||
|
||||
|
||||
class TestTitleSimilarity(unittest.TestCase):
|
||||
def test_identical_titles_score_one(self):
|
||||
self.assertEqual(vc.title_similarity("Same Title", "same title"), 1.0)
|
||||
|
||||
def test_empty_or_missing_inputs_score_zero(self):
|
||||
self.assertEqual(vc.title_similarity("", "x"), 0.0)
|
||||
self.assertEqual(vc.title_similarity(None, "x"), 0.0)
|
||||
self.assertEqual(vc.title_similarity("x", None), 0.0)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@ -13,14 +13,17 @@ Importable:
|
||||
from verify_citation import verify
|
||||
"""
|
||||
import sys
|
||||
import os
|
||||
import json
|
||||
import argparse
|
||||
import difflib
|
||||
import urllib.request
|
||||
import urllib.parse
|
||||
import urllib.error
|
||||
import xml.etree.ElementTree as ET
|
||||
|
||||
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
|
||||
from http_utils import fetch
|
||||
|
||||
ATOM_NS = {"atom": "http://www.w3.org/2005/Atom"}
|
||||
|
||||
|
||||
@ -34,8 +37,7 @@ def check_arxiv_id(arxiv_id, timeout=20):
|
||||
"""Confirm an arXiv ID actually resolves to a real paper."""
|
||||
try:
|
||||
url = f"http://export.arxiv.org/api/query?id_list={urllib.parse.quote(arxiv_id)}"
|
||||
with urllib.request.urlopen(url, timeout=timeout) as resp:
|
||||
data = resp.read()
|
||||
data = fetch(url, timeout=timeout)
|
||||
root = ET.fromstring(data)
|
||||
entry = root.find("atom:entry", ATOM_NS)
|
||||
if entry is None:
|
||||
@ -51,11 +53,7 @@ def check_doi(doi, timeout=20):
|
||||
"""Confirm a DOI actually resolves via Crossref."""
|
||||
try:
|
||||
url = f"https://api.crossref.org/works/{urllib.parse.quote(doi)}"
|
||||
req = urllib.request.Request(
|
||||
url, headers={"User-Agent": "literature-search-verify-skill/1.0"}
|
||||
)
|
||||
with urllib.request.urlopen(req, timeout=timeout) as resp:
|
||||
data = json.loads(resp.read())
|
||||
data = json.loads(fetch(url, headers={"User-Agent": "literature-search-verify-skill/1.0"}, timeout=timeout))
|
||||
titles = data.get("message", {}).get("title") or []
|
||||
return {"status": "pass", "canonical_title": titles[0] if titles else None}
|
||||
except urllib.error.HTTPError as e:
|
||||
@ -66,27 +64,70 @@ def check_doi(doi, timeout=20):
|
||||
return {"status": "skipped", "reason": str(e)}
|
||||
|
||||
|
||||
def check_title_cross_source(title, timeout=20):
|
||||
"""Independently re-search by title on a different source (Semantic
|
||||
Scholar) and require a near-exact title match. This is what catches a
|
||||
plausible-sounding but entirely invented title/author combination."""
|
||||
def _best_title_match(candidates, title, title_field="title"):
|
||||
best = max(candidates, key=lambda p: title_similarity(title, p.get(title_field, "") or ""))
|
||||
sim = title_similarity(title, best.get(title_field, "") or "")
|
||||
return sim, best.get(title_field)
|
||||
|
||||
|
||||
def check_title_cross_source_s2(title, timeout=20):
|
||||
"""Independently re-search by title on Semantic Scholar and require a
|
||||
near-exact title match. This is what catches a plausible-sounding but
|
||||
entirely invented title/author combination."""
|
||||
try:
|
||||
params = {"query": title, "limit": 3, "fields": "title"}
|
||||
url = f"https://api.semanticscholar.org/graph/v1/paper/search?{urllib.parse.urlencode(params)}"
|
||||
with urllib.request.urlopen(url, timeout=timeout) as resp:
|
||||
data = json.loads(resp.read())
|
||||
data = json.loads(fetch(url, timeout=timeout))
|
||||
candidates = data.get("data", []) or []
|
||||
if not candidates:
|
||||
return {"status": "fail", "reason": "no matching title found on Semantic Scholar"}
|
||||
best = max(candidates, key=lambda p: title_similarity(title, p.get("title", "")))
|
||||
sim = title_similarity(title, best.get("title", ""))
|
||||
if sim >= 0.9:
|
||||
return {"status": "pass", "similarity": round(sim, 3), "matched_title": best.get("title")}
|
||||
return {"status": "fail", "similarity": round(sim, 3), "matched_title": best.get("title")}
|
||||
sim, matched = _best_title_match(candidates, title)
|
||||
status = "pass" if sim >= 0.9 else "fail"
|
||||
return {"status": status, "similarity": round(sim, 3), "matched_title": matched}
|
||||
except Exception as e:
|
||||
return {"status": "skipped", "reason": str(e)}
|
||||
|
||||
|
||||
def check_title_cross_source_openalex(title, timeout=20):
|
||||
"""Same idea as check_title_cross_source_s2 but against OpenAlex -- a
|
||||
second, independent source for the title cross-check so a single source
|
||||
(S2 in particular has real-world rate-limit issues) being unavailable
|
||||
doesn't drop this whole check to 'skipped' for a batch of candidates."""
|
||||
try:
|
||||
params = {"search": title, "per_page": 3, "select": "display_name"}
|
||||
url = f"https://api.openalex.org/works?{urllib.parse.urlencode(params)}"
|
||||
data = json.loads(fetch(url, timeout=timeout))
|
||||
candidates = data.get("results", []) or []
|
||||
if not candidates:
|
||||
return {"status": "fail", "reason": "no matching title found on OpenAlex"}
|
||||
sim, matched = _best_title_match(candidates, title, title_field="display_name")
|
||||
status = "pass" if sim >= 0.9 else "fail"
|
||||
return {"status": status, "similarity": round(sim, 3), "matched_title": matched}
|
||||
except Exception as e:
|
||||
return {"status": "skipped", "reason": str(e)}
|
||||
|
||||
|
||||
def _combine_title_checks(s2_result, openalex_result):
|
||||
"""Merge the two independent title cross-checks into one verdict-relevant
|
||||
check. Coverage differs between S2 and OpenAlex, so one source not
|
||||
indexing a (real) paper yet is common and must not, by itself, read as
|
||||
"actively contradicted" the way a single-source design would have taken
|
||||
it -- a pass from either source is enough; it only counts as an active
|
||||
fail once neither source could corroborate it and at least one of them
|
||||
genuinely searched (wasn't just skipped for network reasons)."""
|
||||
sources = {"semantic_scholar": s2_result, "openalex": openalex_result}
|
||||
passing = [r for r in sources.values() if r["status"] == "pass"]
|
||||
if passing:
|
||||
best = max(passing, key=lambda r: r.get("similarity", 0))
|
||||
return {"status": "pass", "sources": sources, "similarity": best.get("similarity"),
|
||||
"matched_title": best.get("matched_title")}
|
||||
non_skipped = [r for r in sources.values() if r["status"] != "skipped"]
|
||||
if non_skipped:
|
||||
return {"status": "fail", "sources": sources,
|
||||
"reason": "no title match on either Semantic Scholar or OpenAlex"}
|
||||
return {"status": "skipped", "sources": sources, "reason": "both sources unavailable"}
|
||||
|
||||
|
||||
def verify(title=None, arxiv_id=None, doi=None):
|
||||
checks = {}
|
||||
if arxiv_id:
|
||||
@ -94,7 +135,10 @@ def verify(title=None, arxiv_id=None, doi=None):
|
||||
if doi:
|
||||
checks["doi_check"] = check_doi(doi)
|
||||
if title:
|
||||
checks["title_cross_source_check"] = check_title_cross_source(title)
|
||||
checks["title_cross_source_check"] = _combine_title_checks(
|
||||
check_title_cross_source_s2(title),
|
||||
check_title_cross_source_openalex(title),
|
||||
)
|
||||
|
||||
passed = [c for c in checks.values() if c["status"] == "pass"]
|
||||
failed = [c for c in checks.values() if c["status"] == "fail"]
|
||||
|
||||
Binary file not shown.
124
CLAUDE.md
Normal file
124
CLAUDE.md
Normal file
@ -0,0 +1,124 @@
|
||||
# autoPaper — 科研辅助 Claude Skill 库
|
||||
|
||||
本仓库不是一篇具体论文的工作区,而是一套持续维护的 Claude Code skill 库,
|
||||
服务于科研工作流(文献检索 → 核实 → 归档 → 写作)。`references/` 下按主题
|
||||
归档的文献是这套 skill 实际产出的样例/沉淀,不是本仓库的主体。
|
||||
|
||||
## 目录结构
|
||||
|
||||
```
|
||||
.claude/skills/ 所有 skill 的定义,每个子目录一个 skill,只放代码/说明
|
||||
literature-search-verify/ 检索 + 反幻觉引用核查
|
||||
SKILL.md skill 说明(frontmatter name 必须与目录名一致)
|
||||
scripts/ 纯标准库 Python 脚本,无第三方依赖(含 http_utils.py 统一限流重试)
|
||||
tests/ unittest 单测,全部mock网络请求,不发真实请求
|
||||
paper-writing-grounded/ 论文写作(强制数据溯源)
|
||||
SKILL.md
|
||||
git-commit/ 规范化 commit message 生成与提交
|
||||
SKILL.md
|
||||
output/<skill-name>/<topic-slug>/ 各 skill 运行时的草稿区,.gitignore 掉,不进版本库;
|
||||
按 skill 分子目录、同一 skill 内再按主题分子目录
|
||||
(topic-slug 规则和 references/ 下的目录名一致),
|
||||
避免不同 skill、不同主题的草稿互相覆盖/混堆
|
||||
references/<topic-slug>/ literature-search-verify 归档产出的稳定文献库
|
||||
references.bib 已验证文献,写作阶段 \cite{} 直接复用这里的 key
|
||||
README.md 人可读索引,含 suspect/unverified 条目说明
|
||||
pdfs/ 开放获取的原文 PDF(如有)
|
||||
```
|
||||
|
||||
## 核心设计原则(新增/修改 skill 时必须保持)
|
||||
|
||||
1. **反幻觉是硬约束,不是建议**:任何进入正文引用或数字的内容都必须能独立核实
|
||||
或追溯到真实数据,验证不通过就必须显式标记(`suspect`/`unverified`/
|
||||
`[需要数据: ...]`),绝不能为了让输出"看起来完整"而悄悄丢弃或蒙混过关。
|
||||
2. **脚本优先于临场编 API 调用**:能写成 `scripts/` 里可重复运行的脚本就不要
|
||||
指望模型每次现场拼 HTTP 请求——后者不可复现、容易在细节上出错。脚本只用
|
||||
标准库,不引入第三方依赖,保证任何环境下拿来就能跑。
|
||||
3. **草稿区与交付物分离**:草稿(项目根目录下的 `output/<skill-name>/<topic-slug>/`)
|
||||
只是运行痕迹,不是可信的最终产物,也不放在 `.claude/skills/` 里面(那里只放
|
||||
skill 代码本身);确认稳定后要显式归档到 `references/<slug>/` 这类项目级
|
||||
目录,才算数。同一 skill 下不同主题的草稿必须分子目录,不能用无区分度的
|
||||
通用文件名（如`t1.json`）散落在同一层——这类命名冲突曾经真实发生过。
|
||||
4. **skill 之间通过约定(如 bibtex key)解耦协作**,而不是互相读对方内部状态;
|
||||
每个 SKILL.md 末尾应有一节说明它和其他 skill 的配合方式。
|
||||
5. **目录名必须和 SKILL.md frontmatter 里的 `name:` 完全一致**——skill 发现/
|
||||
调用是按目录名走的,两者不一致会导致"文档里写的名字"和"实际能唤起的名字"
|
||||
对不上(曾经出现过 `paper-wiriting-grounded` 目录名手滑打错、和 frontmatter
|
||||
里正确拼写的 `paper-writing-grounded` 不一致的问题,已修正)。
|
||||
|
||||
## 已知薄弱环节 / 待办方向
|
||||
|
||||
- 目前只覆盖"检索/核实"与"写作/溯源"两段,引用一致性核查(定稿里的
|
||||
`\cite{}` 是否都对得上 `references.bib`、有没有归档了却从未引用的条目)
|
||||
还没有对应 skill。
|
||||
- `search_openalex.py` 目前只用了 OpenAlex Works API 里比较基础的字段
|
||||
(标题/作者/年份/venue/DOI/引用数/开放获取PDF),没有用它的 concept/topic
|
||||
分类字段做更细的学科过滤——如果以后要精确限定学科方向,这是可以深挖的点。
|
||||
|
||||
## 下一步计划
|
||||
|
||||
1. ~~给核心脚本补单测 + 给网络请求加限流重试~~ **已完成**:新增
|
||||
`scripts/http_utils.py` 统一封装限流重试(429/5xx指数退避,遵守
|
||||
`Retry-After`),`search_arxiv/crossref/semantic_scholar/openalex.py` 和
|
||||
`verify_citation.py` 都已接入;`verify_citation.py` 的跨源标题核查改成
|
||||
同时查 Semantic Scholar + OpenAlex 两个独立源(任一命中即通过),不再
|
||||
单点依赖 S2;新增 `search_openalex.py` 作为第四个检索源;`archive_references.py`
|
||||
的 `slugify()` 修了中文主题名被折叠成通用"references"名的问题;
|
||||
`scripts/tests/` 下补了 37 个 unittest 单测(全部mock,不发真实请求),
|
||||
覆盖上述所有改动。
|
||||
2. **新增"引用一致性核查" skill**(`citation-consistency-check`):检查定稿里
|
||||
所有 `\cite{}` 的 key 是否都能在对应主题的 `references/<slug>/references.bib`
|
||||
里找到,以及有没有已归档但从未被正文引用的"僵尸条目",在
|
||||
literature-search-verify 和 paper-writing-grounded 之间补上这个校验环节。
|
||||
3. **新增"图表生成" skill**(`figure-from-data`,独立 skill,不并入
|
||||
paper-writing-grounded):把全局 dataviz skill 的画图能力和本项目"数据不
|
||||
能编"的红线结合起来——图上出现的每一个数字/误差棒/显著性标注都必须能
|
||||
追溯到用户提供的真实数据,追溯不到就必须显式标记,不能为了图好看而插值/
|
||||
编造。用户目前"才刚开始"做实验,这个和 reproducibility-checklist 一起
|
||||
算当前阶段实际用得上的。
|
||||
4. **新增"实验可复现性自查" skill**(`reproducibility-checklist`):检查一个
|
||||
实验结果和"跑出这个结果的环境/随机种子/超参数/代码版本"之间是否可追溯,
|
||||
在博士研究早期建立这个记录习惯,比后期(投稿前)才补收益更大。具体检查
|
||||
项和触发时机(每次跑完实验?还是准备写进论文前?)还需要进一步讨论确定。
|
||||
5. **新增"投稿格式合规检查" skill**(`submission-checklist`):投稿前检查页数
|
||||
限制、双盲匿名化处理、模板合规、supplementary材料要求等——失败代价直接
|
||||
(格式不合规可能直接被拒),但用户目前还没到投稿周期,优先级低于上面几项,
|
||||
先记录设想,不着急做。
|
||||
6. **新增"审稿意见回复" skill**(`rebuttal-writing-grounded`):和
|
||||
paper-writing-grounded 共享同一条红线——不能为了让回复显得更有说服力,
|
||||
而承诺做不到的新实验或编造补充结果;但写作场景(逐条对应审稿人意见、
|
||||
语气要求"礼貌但坚定")不同,应该做成姊妹 skill 而不是塞进
|
||||
paper-writing-grounded 里。用户目前还没进入投稿/答辩周期,优先级最低,
|
||||
先记录设想。
|
||||
7. **新增"实验设计常见陷阱清单" skill**(`experimental-design-checklist`,
|
||||
优先级低、置信度低):不强制流程,只是一份"容易漏掉的检查项"提醒(有没有
|
||||
设基线/消融、统计检验方法选得对不对、有没有偷偷用测试集调过参),因为
|
||||
"实验设计得好不好"本质是统计学/领域判断力,skill 只能提醒别漏掉常见坑,
|
||||
不能替用户判断设计是否合理——做的时候要非常克制,避免让用户误以为"清单
|
||||
过了=设计没问题"。
|
||||
8. **新增"组会/答辩/会议报告大纲" skill**(`presentation-outline`,优先级低、
|
||||
价值存疑):把已有的真实结果整理成报告大纲,同样要遵守"不编内容"的红线。
|
||||
价值有限,因为这类大纲高度依赖听众和场合,通用流程能提供的帮助有限;暂
|
||||
不确定要不要做,先记录设想。
|
||||
9. **新增"导师进展汇报" skill**(`advisor-progress-report`,优先级低、价值
|
||||
存疑):从 git log / 实验记录整理成给导师的周报。风险是"总结不当会歪曲
|
||||
实际进度"——如果做,必须严格限定为"只整理已确认的事实,不做主观进度
|
||||
评估",且这类沟通策略本身因人而异,通用 skill 能提供的价值可能不大。
|
||||
10. **讨论过但决定不做**:
|
||||
- "新颖性/查重式文献扫描"(检索这个想法是否已被做过)。结论是它和
|
||||
literature-search-verify 的检索底层高度重合,而"够不够新颖"本质是
|
||||
判断力问题,skill 顶多能帮忙把相关已有工作找全,这部分
|
||||
literature-search-verify 已经能覆盖大半,边际价值不足以单独立项。
|
||||
- "研究问题提出/创新点判断"。这是纯判断力/领域洞察力问题,不适合
|
||||
productize——skill 最多能辅助"检索现有工作看有没有人做过"(即上一条
|
||||
讨论过的新颖性扫描),但"这个问题值不值得做"必须是人的判断,做成
|
||||
强流程 skill 反而有"流程走完=判断没问题"的误导风险。
|
||||
- 以上两条如果以后发现实际需求很明确,可以重新评估这些决定。
|
||||
|
||||
## 新增 skill 时的约定
|
||||
|
||||
- SKILL.md 的 frontmatter `description` 要写清楚"什么场景下必须触发这个
|
||||
skill",因为这是 skill 被自动选中的唯一依据。
|
||||
- 正文按"为什么需要 / 工作流程 / 和其他 skill 的配合"组织,和现有两个 skill
|
||||
保持同样的结构和颗粒度。
|
||||
- 目录名 = frontmatter `name`,不要有拼写差异。
|
||||
@ -53,4 +53,4 @@ Direction covered: UAV/airborne aeromagnetic compensation (Tolles-Lawson family)
|
||||
Queried via arXiv + Crossref + Semantic Scholar (S2 was rate-limited (HTTP 429) for much of the session, so most entries only got single-channel verification -- arXiv ID or DOI resolution -- rather than the additional cross-source title check; this is noted per-entry as "unverified"/"skipped" in the raw JSON reports, not silently upgraded to double-verified).
|
||||
Searched but found no on-topic hits: Kalman-filter-based aeromagnetic compensation; genetic-algorithm/PSO-based aeromagnetic compensation.
|
||||
Not covered at all: CNKI/Wanfang/VIP (Chinese databases, no public API) -- use the Zotero Connector browser extension logged into a university account for these.
|
||||
Raw per-query JSON search/verification reports (including filtered-out noise from ambiguous keyword matches like "Tolles"/"Lawson" as surnames) are kept in .claude/skills/literature-search-verify/output/ for audit purposes and are not part of this archive.
|
||||
Raw per-query JSON search/verification reports (including filtered-out noise from ambiguous keyword matches like "Tolles"/"Lawson" as surnames) are kept in output/literature-search-verify/uav_aeromagnetic_compensation/ for audit purposes and are not part of this archive.
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user