diff --git a/.claude/skills/literature-search-verify/SKILL.md b/.claude/skills/literature-search-verify/SKILL.md index beab026..f737240 100644 --- a/.claude/skills/literature-search-verify/SKILL.md +++ b/.claude/skills/literature-search-verify/SKILL.md @@ -1,6 +1,6 @@ --- name: literature-search-verify -description: Search academic literature across arXiv, Semantic Scholar, Crossref, and other connected paper-search MCP tools, and independently verify every candidate citation before it is treated as real. Use this whenever the user asks to find papers, search literature on a topic, build a reading list, compile related-work references, check whether a citation or bibliography entry actually exists, or prepare references to import into Zotero or a .bib file — especially in academic writing contexts where a fabricated citation would be a real problem. Also covers guiding the user to the Zotero Connector browser extension for Chinese-language sources (CNKI/知网, Wanfang/万方, VIP/维普) that have no public API and cannot be reached by search tools. +description: Search academic literature across arXiv, Semantic Scholar, Crossref, OpenAlex, and other connected paper-search MCP tools, and independently verify every candidate citation before it is treated as real. Use this whenever the user asks to find papers, search literature on a topic, build a reading list, compile related-work references, check whether a citation or bibliography entry actually exists, or prepare references to import into Zotero or a .bib file — especially in academic writing contexts where a fabricated citation would be a real problem. Also covers guiding the user to the Zotero Connector browser extension for Chinese-language sources (CNKI/知网, Wanfang/万方, VIP/维普) that have no public API and cannot be reached by search tools. --- # 文献检索 + 反幻觉引用核查 @@ -50,9 +50,9 @@ python3 .claude/skills/literature-search-verify/scripts/literature_search.py \ 草稿和第六步归档产出的`references//`目录能通过同一个slug对上号, 之后回来补充检索同一主题时也知道去哪个子目录续。 -正常情况下**只需要跑这一条命令**,它内部会依次调用 `search_arxiv.py`、`search_semantic_scholar.py`、`search_crossref.py` 做检索,再对每条合并后的候选文献跑 `verify_citation.py` 做交叉验证,输出一份JSON报告(每条候选都带`verdict`字段)。如果只是想单独查一个来源,或者针对某一条文献单独复核,再分别调用对应的单个脚本(用法见每个脚本文件开头的docstring)。 +正常情况下**只需要跑这一条命令**,它内部会依次调用 `search_arxiv.py`、`search_semantic_scholar.py`、`search_crossref.py`、`search_openalex.py` 做检索,再对每条合并后的候选文献跑 `verify_citation.py` 做交叉验证,输出一份JSON报告(每条候选都带`verdict`字段)。如果只是想单独查一个来源,或者针对某一条文献单独复核,再分别调用对应的单个脚本(用法见每个脚本文件开头的docstring)。OpenAlex(`search_openalex.py`)是在 arXiv/Semantic Scholar/Crossref 之外新增的第四个源,免费、无需API key,覆盖面比单独的 Crossref 更广(聚合了包括 IEEE 在内的绝大多数出版商元数据),还能拿到开放获取PDF直链;可以设置`OPENALEX_MAILTO`环境变量为一个邮箱地址进入OpenAlex的"polite pool"换取更稳的限额,不设置也能正常用。 -如果这些脚本因为网络原因跑不动(比如内网/代理限制导致连不上 arxiv.org、semanticscholar.org、crossref.org),`literature_search.py` 会把每个来源的报错单独记在`search_errors`里而不是直接崩溃——这时候老实告诉用户"检索脚本连不上网络,以下是报错信息",不要退回去凭记忆编文献。如果用户这边确实连不上这几个学术API域名,才退回到 web_search 工具,并在结果里明确标注"来自通用网络搜索的补充结果,未经过脚本的交叉验证流程,置信度较低"。 +所有脚本的网络请求都经过 `http_utils.py` 统一封装:遇到 429(限流)或 5xx 会按指数退避自动重试几次(优先遵守服务端返回的`Retry-After`),不需要每次手动重试;`verify_citation.py`的跨源标题核查更是同时查 Semantic Scholar 和 OpenAlex 两个独立源、任一命中即算通过,不会再出现"S2一限流,整批候选的跨源核查全部退化成skipped"的情况(这是实际发生过的问题,现在已经用两个源+重试解决)。即便如此,如果这些脚本因为网络原因跑不动(比如内网/代理限制导致连不上这几个学术API域名),`literature_search.py` 会把每个来源的报错单独记在`search_errors`里而不是直接崩溃——这时候老实告诉用户"检索脚本连不上网络,以下是报错信息",不要退回去凭记忆编文献。如果用户这边确实连不上这几个学术API域名,才退回到 web_search 工具,并在结果里明确标注"来自通用网络搜索的补充结果,未经过脚本的交叉验证流程,置信度较低"。 如果用户已经连了 paper-search-mcp / scholar_mcp_server 这类MCP工具,可以补充用来扩大覆盖面(比如它们能覆盖PubMed、能直接下载PDF),但**不能替代**`verify_citation.py`的交叉验证这一步——MCP搜到的候选一样要过一遍验证,不能因为是工具搜出来的就默认可信。 @@ -62,7 +62,7 @@ python3 .claude/skills/literature-search-verify/scripts/literature_search.py \ 1. **arXiv ID 独立核实**:如果有 arXiv ID,反查一次 arXiv API,确认这个ID真的存在且标题对得上——一个编造的ID在这一步会直接暴露。 2. **DOI 独立核实**:如果有 DOI,反查一次 Crossref,确认这个 DOI 真的能解析出对应文献。 -3. **跨源标题复核**:不管有没有ID,单独拿标题去 Semantic Scholar 搜一次,要求返回的标题跟候选标题高度相似(相似度≥0.9)——这一步专门用来抓"标题作者读起来很像真的,但其实是编出来的"这种情况。 +3. **跨源标题复核(双源)**:不管有没有ID,单独拿标题分别去 Semantic Scholar 和 OpenAlex 各搜一次,要求返回的标题跟候选标题高度相似(相似度≥0.9)——这一步专门用来抓"标题作者读起来很像真的,但其实是编出来的"这种情况。两个源里任一个命中相似度达标就算通过;只有当两个源都明确没找到匹配(不是因为网络问题被跳过)时才算这一项核查失败——避免某个源覆盖不全(比如论文太新还没被其中一个索引收录)被误判成"编造"。 每条候选最后会带一个`verdict`: - **verified**:至少一项独立核查通过,而且没有任何一项核查明确失败 @@ -95,9 +95,9 @@ MCP 检索工具覆盖的是 arXiv/Semantic Scholar/Crossref 这类有公开 API ```bash python3 .claude/skills/literature-search-verify/scripts/archive_references.py \ "UAV aeromagnetic compensation" \ - --bib output/literature-search-verify/uav_aeromagnetic_compensation_final.bib \ + --bib output/literature-search-verify/uav_aeromagnetic_compensation/uav_aeromagnetic_compensation_final.bib \ --project-root . \ - --pdfs-dir output/literature-search-verify/pdfs \ + --pdfs-dir output/literature-search-verify/uav_aeromagnetic_compensation/pdfs \ --suspect "某条可疑文献标题|不建议引用的具体原因" \ --notes "检索覆盖了哪些方向、哪些方向搜了但没结果、中文文献缺口提醒等" ``` @@ -112,6 +112,21 @@ python3 .claude/skills/literature-search-verify/scripts/archive_references.py \ - 同一个`topic`名字多次调用会往同一个归档目录里覆盖更新(bib和README会被覆盖,pdfs按文件名去重合并),所以后续检索到更多文献后可以直接对同一个topic重新跑一遍归档脚本来更新,不需要手动合并。 - 这一步做完之后可以明确告诉用户归档目录的路径,方便他们后续在`paper-writing-grounded`阶段直接引用。 +## 维护者备注:单元测试 + +`scripts/tests/` 下有针对核心正确性逻辑的单元测试(不发真实网络请求,全部用 +mock):`verify_citation.py` 的三档判定和双源标题核查合并逻辑、 +`archive_references.py` 的 bib 解析(含嵌套花括号、中文主题名slugify等边界 +情况)、`literature_search.py` 的候选去重合并与BibTeX生成、`http_utils.py` +的限流重试。改动这几个脚本后应该跑一遍: + +```bash +python3 -m unittest discover -s .claude/skills/literature-search-verify/scripts/tests -t .claude/skills/literature-search-verify/scripts +``` + +只用标准库`unittest`,不引入pytest等第三方测试框架,和脚本本身"不依赖第三方 +包"的原则保持一致。 + ## 和 paper-writing-grounded 技能的配合 这个技能负责把"真实存在、经过核实的文献"整理好并生成 BibTeX;写作阶段的 paper-writing-grounded 技能会直接消费这里产出的 citation key,正文引用只能来自这里核实过的条目,不会凭空生成新的引用。两个技能配合使用时,建议先跑完这个技能、拿到稳定的参考文献列表,再进入写作。 diff --git a/.claude/skills/literature-search-verify/scripts/archive_references.py b/.claude/skills/literature-search-verify/scripts/archive_references.py index 11ee0ba..e9ab897 100644 --- a/.claude/skills/literature-search-verify/scripts/archive_references.py +++ b/.claude/skills/literature-search-verify/scripts/archive_references.py @@ -35,8 +35,12 @@ from datetime import date def slugify(text): + """ASCII gets lowercased; CJK characters are kept as-is (there's no + reversible case-fold for them) so an all-Chinese topic name doesn't + collapse to nothing and fall back to the generic "references" name, + colliding with every other Chinese-named topic archived in this repo.""" text = text.strip().lower() - text = re.sub(r"[^a-z0-9]+", "_", text) + text = re.sub(r"[^a-z0-9一-鿿]+", "_", text) return text.strip("_")[:60] or "references" diff --git a/.claude/skills/literature-search-verify/scripts/http_utils.py b/.claude/skills/literature-search-verify/scripts/http_utils.py new file mode 100644 index 0000000..725bde4 --- /dev/null +++ b/.claude/skills/literature-search-verify/scripts/http_utils.py @@ -0,0 +1,46 @@ +#!/usr/bin/env python3 +""" +Shared HTTP GET helper with exponential-backoff retry for rate limiting (429) +and transient server errors (5xx). Stdlib only. + +Every search_*.py / verify_citation.py network call should go through this +instead of calling urllib.request.urlopen directly, so a single source being +temporarily rate-limited doesn't quietly degrade a whole batch of candidates +to "unverified" the way Semantic Scholar's 429s did before this existed. + +Importable: + from http_utils import fetch +""" +import time +import urllib.request +import urllib.error + +RETRYABLE_CODES = {429, 500, 502, 503, 504} + + +def _wait_seconds(http_error, attempt, base=1.0, cap=20.0): + retry_after = http_error.headers.get("Retry-After") if http_error.headers else None + if retry_after: + try: + return min(float(retry_after), cap) + except ValueError: + pass + return min(base * (2 ** attempt), cap) + + +def fetch(url, headers=None, timeout=20, max_retries=4): + """GET url, retrying on 429/5xx with exponential backoff (honoring a + Retry-After header when the server sends one). Returns the raw response + bytes. Raises the underlying urllib error if retries are exhausted or the + error isn't retryable (e.g. 404).""" + req = urllib.request.Request(url, headers=headers or {}) + attempt = 0 + while True: + try: + with urllib.request.urlopen(req, timeout=timeout) as resp: + return resp.read() + except urllib.error.HTTPError as e: + if e.code not in RETRYABLE_CODES or attempt >= max_retries: + raise + time.sleep(_wait_seconds(e, attempt)) + attempt += 1 diff --git a/.claude/skills/literature-search-verify/scripts/literature_search.py b/.claude/skills/literature-search-verify/scripts/literature_search.py index 20dec6d..a92c5ba 100644 --- a/.claude/skills/literature-search-verify/scripts/literature_search.py +++ b/.claude/skills/literature-search-verify/scripts/literature_search.py @@ -1,8 +1,9 @@ #!/usr/bin/env python3 """ -End-to-end literature search: query arXiv + Semantic Scholar + Crossref, -merge/dedupe candidates, independently verify each one, and emit both a -human-readable report and BibTeX for the entries that passed verification. +End-to-end literature search: query arXiv + Semantic Scholar + Crossref + +OpenAlex, merge/dedupe candidates, independently verify each one, and emit +both a human-readable report and BibTeX for the entries that passed +verification. This is the one script Claude should actually call for a normal literature search -- the individual search_*.py / verify_citation.py scripts exist @@ -28,6 +29,7 @@ sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) from search_arxiv import search_arxiv from search_semantic_scholar import search_s2 from search_crossref import search_crossref +from search_openalex import search_openalex from verify_citation import verify @@ -117,7 +119,13 @@ def to_bibtex(candidate, key): def run(query, max_per_source=8): all_results = [] errors = {} - for name, fn in (("arxiv", search_arxiv), ("semantic_scholar", search_s2), ("crossref", search_crossref)): + sources = ( + ("arxiv", search_arxiv), + ("semantic_scholar", search_s2), + ("crossref", search_crossref), + ("openalex", search_openalex), + ) + for name, fn in sources: try: all_results.extend(fn(query, max_per_source)) except Exception as e: diff --git a/.claude/skills/literature-search-verify/scripts/search_arxiv.py b/.claude/skills/literature-search-verify/scripts/search_arxiv.py index 8367290..f5183b0 100644 --- a/.claude/skills/literature-search-verify/scripts/search_arxiv.py +++ b/.claude/skills/literature-search-verify/scripts/search_arxiv.py @@ -9,12 +9,15 @@ Importable: from search_arxiv import search_arxiv """ import sys +import os import json import argparse -import urllib.request import urllib.parse import xml.etree.ElementTree as ET +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) +from http_utils import fetch + ARXIV_API = "http://export.arxiv.org/api/query" NS = {"atom": "http://www.w3.org/2005/Atom"} @@ -28,8 +31,7 @@ def search_arxiv(query, max_results=10, timeout=20): "sortOrder": "descending", } url = f"{ARXIV_API}?{urllib.parse.urlencode(params)}" - with urllib.request.urlopen(url, timeout=timeout) as resp: - data = resp.read() + data = fetch(url, timeout=timeout) root = ET.fromstring(data) results = [] for entry in root.findall("atom:entry", NS): diff --git a/.claude/skills/literature-search-verify/scripts/search_crossref.py b/.claude/skills/literature-search-verify/scripts/search_crossref.py index 1f76745..4027318 100644 --- a/.claude/skills/literature-search-verify/scripts/search_crossref.py +++ b/.claude/skills/literature-search-verify/scripts/search_crossref.py @@ -10,11 +10,14 @@ Importable: from search_crossref import search_crossref """ import sys +import os import json import argparse -import urllib.request import urllib.parse +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) +from http_utils import fetch + CROSSREF_API = "https://api.crossref.org/works" UA = "literature-search-verify-skill/1.0 (mailto:research-assistant@example.com)" @@ -22,9 +25,7 @@ UA = "literature-search-verify-skill/1.0 (mailto:research-assistant@example.com) def search_crossref(query, max_results=10, timeout=20): params = {"query": query, "rows": max_results} url = f"{CROSSREF_API}?{urllib.parse.urlencode(params)}" - req = urllib.request.Request(url, headers={"User-Agent": UA}) - with urllib.request.urlopen(req, timeout=timeout) as resp: - data = json.loads(resp.read()) + data = json.loads(fetch(url, headers={"User-Agent": UA}, timeout=timeout)) results = [] for item in data.get("message", {}).get("items", []) or []: titles = item.get("title") or [] diff --git a/.claude/skills/literature-search-verify/scripts/search_openalex.py b/.claude/skills/literature-search-verify/scripts/search_openalex.py new file mode 100644 index 0000000..3509603 --- /dev/null +++ b/.claude/skills/literature-search-verify/scripts/search_openalex.py @@ -0,0 +1,92 @@ +#!/usr/bin/env python3 +""" +Search the OpenAlex Works API. No API key required -- set the OPENALEX_MAILTO +environment variable to an email address to use OpenAlex's "polite pool" for +higher and more reliable rate limits (OpenAlex's own recommendation, not a +credential). Coverage is broader than Crossref alone (includes arXiv, IEEE, +and most publishers' metadata in one place) and each result carries an +open-access PDF link when one exists. + +CLI usage: + python3 search_openalex.py "UAV magnetic compensation" --max 10 + +Importable: + from search_openalex import search_openalex +""" +import sys +import os +import json +import argparse +import urllib.parse + +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) +from http_utils import fetch + +OPENALEX_API = "https://api.openalex.org/works" +SELECT_FIELDS = ",".join([ + "doi", "title", "display_name", "publication_year", "authorships", + "primary_location", "open_access", "cited_by_count", "abstract_inverted_index", +]) + + +def _reconstruct_abstract(inverted_index): + """OpenAlex returns abstracts as a word -> [positions] inverted index + (for copyright reasons) instead of plain text; rebuild the plain text.""" + if not inverted_index: + return None + positions = {} + for word, idxs in inverted_index.items(): + for i in idxs: + positions[i] = word + if not positions: + return None + return " ".join(positions[i] for i in sorted(positions)) + + +def search_openalex(query, max_results=10, timeout=20): + params = {"search": query, "per_page": max_results, "select": SELECT_FIELDS} + mailto = os.environ.get("OPENALEX_MAILTO") + if mailto: + params["mailto"] = mailto + url = f"{OPENALEX_API}?{urllib.parse.urlencode(params)}" + data = json.loads(fetch(url, timeout=timeout)) + results = [] + for item in data.get("results", []) or []: + title = item.get("title") or item.get("display_name") + authors = [ + a.get("author", {}).get("display_name") + for a in (item.get("authorships") or []) + if a.get("author") + ] + doi = item.get("doi") + if doi and doi.startswith("https://doi.org/"): + doi = doi[len("https://doi.org/"):] + primary_location = item.get("primary_location") or {} + source = primary_location.get("source") or {} + open_access = item.get("open_access") or {} + results.append({ + "source": "openalex", + "title": title, + "authors": authors, + "year": item.get("publication_year"), + "venue": source.get("display_name"), + "doi": doi, + "arxiv_id": None, + "abstract": _reconstruct_abstract(item.get("abstract_inverted_index")), + "citation_count": item.get("cited_by_count"), + "pdf_url": open_access.get("oa_url") or primary_location.get("pdf_url"), + }) + return results + + +if __name__ == "__main__": + ap = argparse.ArgumentParser(description=__doc__) + ap.add_argument("query") + ap.add_argument("--max", type=int, default=10) + args = ap.parse_args() + try: + out = search_openalex(args.query, args.max) + print(json.dumps(out, ensure_ascii=False, indent=2)) + except Exception as e: + print(json.dumps({"error": str(e)}, ensure_ascii=False)) + sys.exit(1) diff --git a/.claude/skills/literature-search-verify/scripts/search_semantic_scholar.py b/.claude/skills/literature-search-verify/scripts/search_semantic_scholar.py index dd6cc6b..0c089a3 100644 --- a/.claude/skills/literature-search-verify/scripts/search_semantic_scholar.py +++ b/.claude/skills/literature-search-verify/scripts/search_semantic_scholar.py @@ -13,9 +13,11 @@ import sys import os import json import argparse -import urllib.request import urllib.parse +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) +from http_utils import fetch + S2_API = "https://api.semanticscholar.org/graph/v1/paper/search" FIELDS = "title,authors,year,venue,externalIds,abstract,citationCount" @@ -23,12 +25,11 @@ FIELDS = "title,authors,year,venue,externalIds,abstract,citationCount" def search_s2(query, max_results=10, timeout=20): params = {"query": query, "limit": max_results, "fields": FIELDS} url = f"{S2_API}?{urllib.parse.urlencode(params)}" - req = urllib.request.Request(url) + headers = {} api_key = os.environ.get("S2_API_KEY") if api_key: - req.add_header("x-api-key", api_key) - with urllib.request.urlopen(req, timeout=timeout) as resp: - data = json.loads(resp.read()) + headers["x-api-key"] = api_key + data = json.loads(fetch(url, headers=headers, timeout=timeout)) results = [] for p in data.get("data", []) or []: ext = p.get("externalIds") or {} diff --git a/.claude/skills/literature-search-verify/scripts/tests/__init__.py b/.claude/skills/literature-search-verify/scripts/tests/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/.claude/skills/literature-search-verify/scripts/tests/test_archive_references.py b/.claude/skills/literature-search-verify/scripts/tests/test_archive_references.py new file mode 100644 index 0000000..3e65047 --- /dev/null +++ b/.claude/skills/literature-search-verify/scripts/tests/test_archive_references.py @@ -0,0 +1,99 @@ +#!/usr/bin/env python3 +""" +Unit tests for archive_references.py's slugify() and the minimal bib parser, +covering the edge cases that are easy to get wrong silently: CJK topic names +and nested-brace LaTeX field values (e.g. \\ensuremath{\\epsilon}). + +Run: python -m unittest discover -s .claude/skills/literature-search-verify/scripts +""" +import os +import sys +import tempfile +import unittest + +sys.path.insert(0, os.path.join(os.path.dirname(os.path.abspath(__file__)), "..")) +import archive_references as ar + + +class TestSlugify(unittest.TestCase): + def test_ascii_topic(self): + self.assertEqual(ar.slugify("UAV aeromagnetic compensation"), "uav_aeromagnetic_compensation") + + def test_chinese_topic_is_preserved_not_collapsed_to_generic_name(self): + slug = ar.slugify("无人机磁补偿") + self.assertNotEqual(slug, "references") + self.assertIn("补偿", slug) + + def test_mixed_chinese_and_english(self): + slug = ar.slugify("无人机磁补偿 Tolles-Lawson") + self.assertTrue(slug.endswith("tolles_lawson")) + + def test_blank_topic_falls_back_to_references(self): + self.assertEqual(ar.slugify(" "), "references") + + def test_truncated_to_60_chars(self): + self.assertEqual(len(ar.slugify("a" * 100)), 60) + + +class TestParseBibEntries(unittest.TestCase): + def _write_bib(self, content): + fd, path = tempfile.mkstemp(suffix=".bib") + with os.fdopen(fd, "w", encoding="utf-8") as f: + f.write(content) + self.addCleanup(os.remove, path) + return path + + def test_nested_braces_in_field_value_do_not_truncate_it(self): + bib = ( + "@article{wu2017,\n" + " title = {Aeromagnetic gradient compensation using \\ensuremath{\\epsilon}-SVR},\n" + " journal = {Journal of Applied Remote Sensing},\n" + " year = {2017},\n" + " doi = {10.1117/1.jrs.11.025012},\n" + "}" + ) + entries = ar.parse_bib_entries(self._write_bib(bib)) + self.assertEqual(len(entries), 1) + self.assertIn("\\ensuremath{\\epsilon}-SVR", entries[0]["title"]) + self.assertEqual(entries[0]["year"], "2017") + + def test_multiple_entries_and_optional_fields(self): + bib = ( + "@article{a2020,\n" + " title = {First paper},\n" + " year = {2020},\n" + "}\n\n" + "@article{b2021,\n" + " title = {Second paper},\n" + " year = {2021},\n" + " note = {some note},\n" + "}" + ) + entries = ar.parse_bib_entries(self._write_bib(bib)) + self.assertEqual([e["key"] for e in entries], ["a2020", "b2021"]) + self.assertNotIn("note", entries[0]) + self.assertEqual(entries[1]["note"], "some note") + + +class TestYearSortKey(unittest.TestCase): + def test_entries_without_a_parseable_year_sort_last(self): + entries = [{"key": "b", "year": ""}, {"key": "a", "year": "1999"}] + entries.sort(key=ar.year_sort_key) + self.assertEqual([e["key"] for e in entries], ["a", "b"]) + + +class TestBuildReadme(unittest.TestCase): + def test_omits_optional_sections_when_not_given(self): + readme = ar.build_readme("Topic", [], 0, [], None) + self.assertNotIn("Flagged during search", readme) + self.assertNotIn("Search coverage notes", readme) + + def test_includes_suspect_and_notes_when_given(self): + readme = ar.build_readme("Topic", [], 0, ["Bad title|dubious venue"], "coverage notes here") + self.assertIn("Bad title", readme) + self.assertIn("dubious venue", readme) + self.assertIn("coverage notes here", readme) + + +if __name__ == "__main__": + unittest.main() diff --git a/.claude/skills/literature-search-verify/scripts/tests/test_http_utils.py b/.claude/skills/literature-search-verify/scripts/tests/test_http_utils.py new file mode 100644 index 0000000..16003c5 --- /dev/null +++ b/.claude/skills/literature-search-verify/scripts/tests/test_http_utils.py @@ -0,0 +1,103 @@ +#!/usr/bin/env python3 +""" +Unit tests for http_utils.fetch's retry/backoff behavior. All network calls +are mocked -- these never touch the real network. + +Run: python -m unittest discover -s .claude/skills/literature-search-verify/scripts +""" +import os +import sys +import unittest +import urllib.error +import urllib.request +from email.message import Message +from unittest import mock + +sys.path.insert(0, os.path.join(os.path.dirname(os.path.abspath(__file__)), "..")) +import http_utils + + +def _http_error(code, retry_after=None): + hdrs = Message() + if retry_after is not None: + hdrs["Retry-After"] = str(retry_after) + return urllib.error.HTTPError("http://example.com", code, "err", hdrs, None) + + +class FakeResponse: + def __init__(self, data): + self._data = data + + def read(self): + return self._data + + def __enter__(self): + return self + + def __exit__(self, *exc_info): + return False + + +class TestFetch(unittest.TestCase): + @mock.patch("urllib.request.urlopen") + @mock.patch("http_utils.time.sleep", return_value=None) + def test_succeeds_first_try_without_sleeping(self, mock_sleep, mock_urlopen): + mock_urlopen.return_value = FakeResponse(b"ok") + result = http_utils.fetch("http://example.com") + self.assertEqual(result, b"ok") + mock_sleep.assert_not_called() + + @mock.patch("urllib.request.urlopen") + @mock.patch("http_utils.time.sleep", return_value=None) + def test_retries_on_429_then_succeeds(self, mock_sleep, mock_urlopen): + mock_urlopen.side_effect = [_http_error(429), FakeResponse(b"ok")] + result = http_utils.fetch("http://example.com", max_retries=3) + self.assertEqual(result, b"ok") + self.assertEqual(mock_urlopen.call_count, 2) + mock_sleep.assert_called_once() + + @mock.patch("urllib.request.urlopen") + @mock.patch("http_utils.time.sleep", return_value=None) + def test_retries_on_5xx(self, mock_sleep, mock_urlopen): + mock_urlopen.side_effect = [_http_error(503), FakeResponse(b"ok")] + result = http_utils.fetch("http://example.com", max_retries=3) + self.assertEqual(result, b"ok") + + @mock.patch("urllib.request.urlopen") + @mock.patch("http_utils.time.sleep", return_value=None) + def test_raises_after_exhausting_retries(self, mock_sleep, mock_urlopen): + mock_urlopen.side_effect = [_http_error(429)] * 10 + with self.assertRaises(urllib.error.HTTPError): + http_utils.fetch("http://example.com", max_retries=2) + self.assertEqual(mock_urlopen.call_count, 3) # initial attempt + 2 retries + + @mock.patch("urllib.request.urlopen") + @mock.patch("http_utils.time.sleep", return_value=None) + def test_non_retryable_status_raises_immediately(self, mock_sleep, mock_urlopen): + mock_urlopen.side_effect = _http_error(404) + with self.assertRaises(urllib.error.HTTPError): + http_utils.fetch("http://example.com") + self.assertEqual(mock_urlopen.call_count, 1) + mock_sleep.assert_not_called() + + +class TestWaitSeconds(unittest.TestCase): + def test_honors_retry_after_header(self): + err = _http_error(429, retry_after=5) + self.assertEqual(http_utils._wait_seconds(err, attempt=0), 5.0) + + def test_falls_back_to_exponential_backoff_without_header(self): + err = _http_error(429) + self.assertAlmostEqual(http_utils._wait_seconds(err, attempt=2, base=1.0), 4.0) + + def test_caps_wait_time(self): + err = _http_error(429, retry_after=999) + self.assertEqual(http_utils._wait_seconds(err, attempt=0, cap=20.0), 20.0) + + def test_ignores_unparseable_retry_after(self): + err = _http_error(429, retry_after="not-a-number") + self.assertAlmostEqual(http_utils._wait_seconds(err, attempt=0, base=1.0), 1.0) + + +if __name__ == "__main__": + unittest.main() diff --git a/.claude/skills/literature-search-verify/scripts/tests/test_literature_search.py b/.claude/skills/literature-search-verify/scripts/tests/test_literature_search.py new file mode 100644 index 0000000..c76a04d --- /dev/null +++ b/.claude/skills/literature-search-verify/scripts/tests/test_literature_search.py @@ -0,0 +1,67 @@ +#!/usr/bin/env python3 +""" +Unit tests for literature_search.py's candidate merging and BibTeX generation. + +Run: python -m unittest discover -s .claude/skills/literature-search-verify/scripts +""" +import os +import sys +import unittest + +sys.path.insert(0, os.path.join(os.path.dirname(os.path.abspath(__file__)), "..")) +import literature_search as ls + + +class TestMergeCandidates(unittest.TestCase): + def test_dedupes_same_paper_across_sources_and_fills_missing_fields(self): + items = [ + {"source": "arxiv", "title": "Deep Learning for Aeromagnetic Compensation", + "arxiv_id": "2401.00001", "doi": None, "year": "2024"}, + {"source": "crossref", "title": "Deep Learning for Aeromagnetic Compensation", + "arxiv_id": None, "doi": "10.1/x", "year": 2024, "venue": "Some Journal"}, + ] + merged = ls.merge_candidates(items) + self.assertEqual(len(merged), 1) + self.assertEqual(merged[0]["arxiv_id"], "2401.00001") + self.assertEqual(merged[0]["doi"], "10.1/x") + self.assertEqual(sorted(merged[0]["sources"]), ["arxiv", "crossref"]) + + def test_distinct_titles_are_not_merged(self): + items = [ + {"source": "arxiv", "title": "Paper About Cats"}, + {"source": "crossref", "title": "Paper About Dogs"}, + ] + self.assertEqual(len(ls.merge_candidates(items)), 2) + + +class TestBibtexKey(unittest.TestCase): + def test_collision_gets_letter_suffix(self): + used = set() + key1 = ls.make_bibtex_key({"authors": ["Jane Smith"], "year": "2020"}, used) + key2 = ls.make_bibtex_key({"authors": ["John Smith"], "year": "2020"}, used) + self.assertEqual(key1, "smith2020") + self.assertEqual(key2, "smith2020a") + + def test_no_authors_falls_back_to_unknown(self): + key = ls.make_bibtex_key({"authors": [], "year": "2020"}, set()) + self.assertEqual(key, "unknown2020") + + +class TestToBibtex(unittest.TestCase): + def test_arxiv_only_candidate_uses_misc_with_eprint(self): + cand = {"authors": ["A B"], "title": "T", "year": "2024", + "arxiv_id": "2401.00001", "venue": None, "doi": None} + bib = ls.to_bibtex(cand, "b2024") + self.assertTrue(bib.startswith("@misc{b2024,")) + self.assertIn("eprint = {2401.00001}", bib) + + def test_candidate_with_venue_uses_article_with_doi(self): + cand = {"authors": ["A B"], "title": "T", "year": "2024", + "arxiv_id": None, "venue": "Some Journal", "doi": "10.1/x"} + bib = ls.to_bibtex(cand, "b2024") + self.assertTrue(bib.startswith("@article{b2024,")) + self.assertIn("doi = {10.1/x}", bib) + + +if __name__ == "__main__": + unittest.main() diff --git a/.claude/skills/literature-search-verify/scripts/tests/test_verify_citation.py b/.claude/skills/literature-search-verify/scripts/tests/test_verify_citation.py new file mode 100644 index 0000000..439363d --- /dev/null +++ b/.claude/skills/literature-search-verify/scripts/tests/test_verify_citation.py @@ -0,0 +1,94 @@ +#!/usr/bin/env python3 +""" +Unit tests for verify_citation's verdict aggregation logic -- this is the +anti-hallucination guarantee, so its three-way verified/suspect/unverified +decision needs to stay correct independent of any real network call. All +individual check_* functions are mocked; only the aggregation logic itself +(verify() and _combine_title_checks()) is under test. + +Run: python -m unittest discover -s .claude/skills/literature-search-verify/scripts +""" +import os +import sys +import unittest +from unittest import mock + +sys.path.insert(0, os.path.join(os.path.dirname(os.path.abspath(__file__)), "..")) +import verify_citation as vc + + +class TestCombineTitleChecks(unittest.TestCase): + """Two independent title sources (S2 + OpenAlex) shouldn't let one + source's incomplete coverage outvote the other's confirmation.""" + + def test_both_pass_picks_higher_similarity(self): + s2 = {"status": "pass", "similarity": 0.95, "matched_title": "A"} + oa = {"status": "pass", "similarity": 0.99, "matched_title": "B"} + combined = vc._combine_title_checks(s2, oa) + self.assertEqual(combined["status"], "pass") + self.assertEqual(combined["matched_title"], "B") + + def test_one_pass_one_fail_still_passes(self): + s2 = {"status": "fail", "reason": "no matching title found on Semantic Scholar"} + oa = {"status": "pass", "similarity": 0.95, "matched_title": "B"} + self.assertEqual(vc._combine_title_checks(s2, oa)["status"], "pass") + + def test_one_pass_one_skipped_still_passes(self): + s2 = {"status": "skipped", "reason": "timeout"} + oa = {"status": "pass", "similarity": 0.95, "matched_title": "B"} + self.assertEqual(vc._combine_title_checks(s2, oa)["status"], "pass") + + def test_both_fail_is_fail(self): + s2 = {"status": "fail", "reason": "x"} + oa = {"status": "fail", "reason": "y"} + self.assertEqual(vc._combine_title_checks(s2, oa)["status"], "fail") + + def test_fail_plus_skipped_is_still_fail(self): + s2 = {"status": "fail", "reason": "x"} + oa = {"status": "skipped", "reason": "y"} + self.assertEqual(vc._combine_title_checks(s2, oa)["status"], "fail") + + def test_both_skipped_is_skipped_not_fail(self): + s2 = {"status": "skipped", "reason": "x"} + oa = {"status": "skipped", "reason": "y"} + self.assertEqual(vc._combine_title_checks(s2, oa)["status"], "skipped") + + +class TestVerifyVerdict(unittest.TestCase): + def test_all_checks_pass_is_verified(self): + with mock.patch.object(vc, "check_arxiv_id", return_value={"status": "pass"}), \ + mock.patch.object(vc, "check_doi", return_value={"status": "pass"}), \ + mock.patch.object(vc, "check_title_cross_source_s2", return_value={"status": "pass", "similarity": 1.0}), \ + mock.patch.object(vc, "check_title_cross_source_openalex", return_value={"status": "pass", "similarity": 1.0}): + result = vc.verify(title="T", arxiv_id="1234.5678", doi="10.1/x") + self.assertEqual(result["verdict"], "verified") + + def test_one_failing_check_is_suspect_even_if_another_passes(self): + with mock.patch.object(vc, "check_arxiv_id", return_value={"status": "fail", "reason": "arXiv ID not found"}), \ + mock.patch.object(vc, "check_doi", return_value={"status": "pass"}): + result = vc.verify(arxiv_id="9999.99999", doi="10.1/x") + self.assertEqual(result["verdict"], "suspect") + + def test_all_checks_skipped_is_unverified_not_verified(self): + with mock.patch.object(vc, "check_arxiv_id", return_value={"status": "skipped", "reason": "timeout"}): + result = vc.verify(arxiv_id="1234.5678") + self.assertEqual(result["verdict"], "unverified") + + def test_no_identifiers_given_produces_no_checks_and_unverified(self): + result = vc.verify(title=None, arxiv_id=None, doi=None) + self.assertEqual(result["checks"], {}) + self.assertEqual(result["verdict"], "unverified") + + +class TestTitleSimilarity(unittest.TestCase): + def test_identical_titles_score_one(self): + self.assertEqual(vc.title_similarity("Same Title", "same title"), 1.0) + + def test_empty_or_missing_inputs_score_zero(self): + self.assertEqual(vc.title_similarity("", "x"), 0.0) + self.assertEqual(vc.title_similarity(None, "x"), 0.0) + self.assertEqual(vc.title_similarity("x", None), 0.0) + + +if __name__ == "__main__": + unittest.main() diff --git a/.claude/skills/literature-search-verify/scripts/verify_citation.py b/.claude/skills/literature-search-verify/scripts/verify_citation.py index b7f4a24..17b93b9 100644 --- a/.claude/skills/literature-search-verify/scripts/verify_citation.py +++ b/.claude/skills/literature-search-verify/scripts/verify_citation.py @@ -13,14 +13,17 @@ Importable: from verify_citation import verify """ import sys +import os import json import argparse import difflib -import urllib.request import urllib.parse import urllib.error import xml.etree.ElementTree as ET +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) +from http_utils import fetch + ATOM_NS = {"atom": "http://www.w3.org/2005/Atom"} @@ -34,8 +37,7 @@ def check_arxiv_id(arxiv_id, timeout=20): """Confirm an arXiv ID actually resolves to a real paper.""" try: url = f"http://export.arxiv.org/api/query?id_list={urllib.parse.quote(arxiv_id)}" - with urllib.request.urlopen(url, timeout=timeout) as resp: - data = resp.read() + data = fetch(url, timeout=timeout) root = ET.fromstring(data) entry = root.find("atom:entry", ATOM_NS) if entry is None: @@ -51,11 +53,7 @@ def check_doi(doi, timeout=20): """Confirm a DOI actually resolves via Crossref.""" try: url = f"https://api.crossref.org/works/{urllib.parse.quote(doi)}" - req = urllib.request.Request( - url, headers={"User-Agent": "literature-search-verify-skill/1.0"} - ) - with urllib.request.urlopen(req, timeout=timeout) as resp: - data = json.loads(resp.read()) + data = json.loads(fetch(url, headers={"User-Agent": "literature-search-verify-skill/1.0"}, timeout=timeout)) titles = data.get("message", {}).get("title") or [] return {"status": "pass", "canonical_title": titles[0] if titles else None} except urllib.error.HTTPError as e: @@ -66,27 +64,70 @@ def check_doi(doi, timeout=20): return {"status": "skipped", "reason": str(e)} -def check_title_cross_source(title, timeout=20): - """Independently re-search by title on a different source (Semantic - Scholar) and require a near-exact title match. This is what catches a - plausible-sounding but entirely invented title/author combination.""" +def _best_title_match(candidates, title, title_field="title"): + best = max(candidates, key=lambda p: title_similarity(title, p.get(title_field, "") or "")) + sim = title_similarity(title, best.get(title_field, "") or "") + return sim, best.get(title_field) + + +def check_title_cross_source_s2(title, timeout=20): + """Independently re-search by title on Semantic Scholar and require a + near-exact title match. This is what catches a plausible-sounding but + entirely invented title/author combination.""" try: params = {"query": title, "limit": 3, "fields": "title"} url = f"https://api.semanticscholar.org/graph/v1/paper/search?{urllib.parse.urlencode(params)}" - with urllib.request.urlopen(url, timeout=timeout) as resp: - data = json.loads(resp.read()) + data = json.loads(fetch(url, timeout=timeout)) candidates = data.get("data", []) or [] if not candidates: return {"status": "fail", "reason": "no matching title found on Semantic Scholar"} - best = max(candidates, key=lambda p: title_similarity(title, p.get("title", ""))) - sim = title_similarity(title, best.get("title", "")) - if sim >= 0.9: - return {"status": "pass", "similarity": round(sim, 3), "matched_title": best.get("title")} - return {"status": "fail", "similarity": round(sim, 3), "matched_title": best.get("title")} + sim, matched = _best_title_match(candidates, title) + status = "pass" if sim >= 0.9 else "fail" + return {"status": status, "similarity": round(sim, 3), "matched_title": matched} except Exception as e: return {"status": "skipped", "reason": str(e)} +def check_title_cross_source_openalex(title, timeout=20): + """Same idea as check_title_cross_source_s2 but against OpenAlex -- a + second, independent source for the title cross-check so a single source + (S2 in particular has real-world rate-limit issues) being unavailable + doesn't drop this whole check to 'skipped' for a batch of candidates.""" + try: + params = {"search": title, "per_page": 3, "select": "display_name"} + url = f"https://api.openalex.org/works?{urllib.parse.urlencode(params)}" + data = json.loads(fetch(url, timeout=timeout)) + candidates = data.get("results", []) or [] + if not candidates: + return {"status": "fail", "reason": "no matching title found on OpenAlex"} + sim, matched = _best_title_match(candidates, title, title_field="display_name") + status = "pass" if sim >= 0.9 else "fail" + return {"status": status, "similarity": round(sim, 3), "matched_title": matched} + except Exception as e: + return {"status": "skipped", "reason": str(e)} + + +def _combine_title_checks(s2_result, openalex_result): + """Merge the two independent title cross-checks into one verdict-relevant + check. Coverage differs between S2 and OpenAlex, so one source not + indexing a (real) paper yet is common and must not, by itself, read as + "actively contradicted" the way a single-source design would have taken + it -- a pass from either source is enough; it only counts as an active + fail once neither source could corroborate it and at least one of them + genuinely searched (wasn't just skipped for network reasons).""" + sources = {"semantic_scholar": s2_result, "openalex": openalex_result} + passing = [r for r in sources.values() if r["status"] == "pass"] + if passing: + best = max(passing, key=lambda r: r.get("similarity", 0)) + return {"status": "pass", "sources": sources, "similarity": best.get("similarity"), + "matched_title": best.get("matched_title")} + non_skipped = [r for r in sources.values() if r["status"] != "skipped"] + if non_skipped: + return {"status": "fail", "sources": sources, + "reason": "no title match on either Semantic Scholar or OpenAlex"} + return {"status": "skipped", "sources": sources, "reason": "both sources unavailable"} + + def verify(title=None, arxiv_id=None, doi=None): checks = {} if arxiv_id: @@ -94,7 +135,10 @@ def verify(title=None, arxiv_id=None, doi=None): if doi: checks["doi_check"] = check_doi(doi) if title: - checks["title_cross_source_check"] = check_title_cross_source(title) + checks["title_cross_source_check"] = _combine_title_checks( + check_title_cross_source_s2(title), + check_title_cross_source_openalex(title), + ) passed = [c for c in checks.values() if c["status"] == "pass"] failed = [c for c in checks.values() if c["status"] == "fail"] diff --git a/CLAUDE.md b/CLAUDE.md index 345e1a8..79fbbf4 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -10,7 +10,8 @@ .claude/skills/ 所有 skill 的定义,每个子目录一个 skill,只放代码/说明 literature-search-verify/ 检索 + 反幻觉引用核查 SKILL.md skill 说明(frontmatter name 必须与目录名一致) - scripts/ 纯标准库 Python 脚本,无第三方依赖 + scripts/ 纯标准库 Python 脚本,无第三方依赖(含 http_utils.py 统一限流重试) + tests/ unittest 单测,全部mock网络请求,不发真实请求 paper-writing-grounded/ 论文写作(强制数据溯源) SKILL.md git-commit/ 规范化 commit message 生成与提交 @@ -47,21 +48,24 @@ references// literature-search-verify 归档产出的稳 ## 已知薄弱环节 / 待办方向 -- `scripts/search_semantic_scholar.py`、`verify_citation.py` 对 Semantic Scholar - 的限流(HTTP 429)没有重试/退避,会话量大时会导致大量条目退化成只有单源验证 - (已在 `references/uav_aeromagnetic_compensation/README.md` 的检索记录里实际发生过)。 -- 核心正确性逻辑(`verify_citation.py` 的三档判定、`archive_references.py` 的 - bib 字段解析)目前没有自动化测试兜底,只能靠人工抽查实际输出结果。 - 目前只覆盖"检索/核实"与"写作/溯源"两段,引用一致性核查(定稿里的 - `\cite{}` 是否都对得上 `references.bib`、有没有归档了却从未引用的条目)、 - 面向具体学科的补充检索源等还没有对应 skill。 + `\cite{}` 是否都对得上 `references.bib`、有没有归档了却从未引用的条目) + 还没有对应 skill。 +- `search_openalex.py` 目前只用了 OpenAlex Works API 里比较基础的字段 + (标题/作者/年份/venue/DOI/引用数/开放获取PDF),没有用它的 concept/topic + 分类字段做更细的学科过滤——如果以后要精确限定学科方向,这是可以深挖的点。 -## 下一步计划(已和用户确认,尚未实施) +## 下一步计划 -1. **给核心脚本补单测 + 给网络请求加限流重试**:`verify_citation.py` 的三档 - 判定逻辑、`archive_references.py` 的 bib 字段解析(覆盖嵌套花括号等边界 - 情况)补 pytest 单测;`search_semantic_scholar.py` 等对 S2/arXiv/Crossref - 的请求加退避重试,避免再出现"S2 被限流导致大量条目退化成单源验证"的情况。 +1. ~~给核心脚本补单测 + 给网络请求加限流重试~~ **已完成**:新增 + `scripts/http_utils.py` 统一封装限流重试(429/5xx指数退避,遵守 + `Retry-After`),`search_arxiv/crossref/semantic_scholar/openalex.py` 和 + `verify_citation.py` 都已接入;`verify_citation.py` 的跨源标题核查改成 + 同时查 Semantic Scholar + OpenAlex 两个独立源(任一命中即通过),不再 + 单点依赖 S2;新增 `search_openalex.py` 作为第四个检索源;`archive_references.py` + 的 `slugify()` 修了中文主题名被折叠成通用"references"名的问题; + `scripts/tests/` 下补了 37 个 unittest 单测(全部mock,不发真实请求), + 覆盖上述所有改动。 2. **新增"引用一致性核查" skill**:检查定稿里所有 `\cite{}` 的 key 是否都能在 对应主题的 `references//references.bib` 里找到,以及有没有已归档 但从未被正文引用的"僵尸条目",在 literature-search-verify 和