init
This commit is contained in:
@@ -0,0 +1,94 @@
|
||||
---
|
||||
name: literature-search-verify
|
||||
description: Search academic literature across arXiv, Semantic Scholar, Crossref, and other connected paper-search MCP tools, and independently verify every candidate citation before it is treated as real. Use this whenever the user asks to find papers, search literature on a topic, build a reading list, compile related-work references, check whether a citation or bibliography entry actually exists, or prepare references to import into Zotero or a .bib file — especially in academic writing contexts where a fabricated citation would be a real problem. Also covers guiding the user to the Zotero Connector browser extension for Chinese-language sources (CNKI/知网, Wanfang/万方, VIP/维普) that have no public API and cannot be reached by search tools.
|
||||
---
|
||||
|
||||
# 文献检索 + 反幻觉引用核查
|
||||
|
||||
## 为什么需要这个技能
|
||||
|
||||
大语言模型在编造论文引用这件事上非常擅长——生成的标题、作者、期刊名读起来都很像真的,但可能根本不存在,或者张冠李戴(把A论文的结论安在B论文头上)。这在正式学术写作里是不可接受的:一篇论文只要有一条编造的引用被发现,审稿人对全篇的信任都会崩塌。
|
||||
|
||||
所以这个技能的核心不是"搜索",而是"搜索之后不轻信"——每一条打算真正拿去引用的文献,都必须经过独立交叉验证,验证不通过的必须明确标出来,而不是悄悄丢弃或者悄悄当作真的用。
|
||||
|
||||
## 工作流程
|
||||
|
||||
### 第一步:明确检索范围
|
||||
|
||||
在开始搜索前,搞清楚(不确定就直接问,一句话就够):
|
||||
- 核心关键词/研究方向(可以中英文混合,比如"UAV磁补偿 Tolles-Lawson"这类)
|
||||
- 大致的时间范围(比如"近5年"还是不限)
|
||||
- 是否需要限定顶会/顶刊,还是什么来源都要
|
||||
|
||||
### 第二步:检索——直接调用脚本,不要自己现编API调用
|
||||
|
||||
`scripts/` 目录下已经写好了能直接跑的检索脚本,不依赖任何第三方Python包,也不需要装MCP工具:
|
||||
|
||||
```bash
|
||||
# 一次性搞定:检索 arXiv + Semantic Scholar + Crossref,自动去重、逐条验证,
|
||||
# 并把通过验证的条目写成BibTeX文件——这是应该默认调用的入口
|
||||
python3 scripts/literature_search.py "UAV magnetic compensation Tolles-Lawson" \
|
||||
--max-per-source 8 --bib-out refs.bib
|
||||
```
|
||||
|
||||
正常情况下**只需要跑这一条命令**,它内部会依次调用 `search_arxiv.py`、`search_semantic_scholar.py`、`search_crossref.py` 做检索,再对每条合并后的候选文献跑 `verify_citation.py` 做交叉验证,输出一份JSON报告(每条候选都带`verdict`字段)。如果只是想单独查一个来源,或者针对某一条文献单独复核,再分别调用对应的单个脚本(用法见每个脚本文件开头的docstring)。
|
||||
|
||||
如果这些脚本因为网络原因跑不动(比如内网/代理限制导致连不上 arxiv.org、semanticscholar.org、crossref.org),`literature_search.py` 会把每个来源的报错单独记在`search_errors`里而不是直接崩溃——这时候老实告诉用户"检索脚本连不上网络,以下是报错信息",不要退回去凭记忆编文献。如果用户这边确实连不上这几个学术API域名,才退回到 web_search 工具,并在结果里明确标注"来自通用网络搜索的补充结果,未经过脚本的交叉验证流程,置信度较低"。
|
||||
|
||||
如果用户已经连了 paper-search-mcp / scholar_mcp_server 这类MCP工具,可以补充用来扩大覆盖面(比如它们能覆盖PubMed、能直接下载PDF),但**不能替代**`verify_citation.py`的交叉验证这一步——MCP搜到的候选一样要过一遍验证,不能因为是工具搜出来的就默认可信。
|
||||
|
||||
### 第三步:理解验证结果——这是最关键的一步
|
||||
|
||||
`literature_search.py`(或单独调用`verify_citation.py`)对每条候选文献做的核查是:
|
||||
|
||||
1. **arXiv ID 独立核实**:如果有 arXiv ID,反查一次 arXiv API,确认这个ID真的存在且标题对得上——一个编造的ID在这一步会直接暴露。
|
||||
2. **DOI 独立核实**:如果有 DOI,反查一次 Crossref,确认这个 DOI 真的能解析出对应文献。
|
||||
3. **跨源标题复核**:不管有没有ID,单独拿标题去 Semantic Scholar 搜一次,要求返回的标题跟候选标题高度相似(相似度≥0.9)——这一步专门用来抓"标题作者读起来很像真的,但其实是编出来的"这种情况。
|
||||
|
||||
每条候选最后会带一个`verdict`:
|
||||
- **verified**:至少一项独立核查通过,而且没有任何一项核查明确失败
|
||||
- **suspect**:至少一项核查明确失败(比如DOI查不到、跨源标题对不上)——**这种情况下不要用这条文献,即使标题看起来很合适**
|
||||
- **unverified**:所有核查项都因为网络等原因被跳过(`skipped`),不代表验证通过,只代表"没能验证"——**同样不能当成已核实的文献直接使用**,要跟用户说清楚原因
|
||||
|
||||
呈现给用户时按这三档分组说明,`suspect`和`unverified`都要明确标出来,不要因为报告里有个"看起来还行"的标题就含糊地当真的用。
|
||||
|
||||
**原则**:找不到真实存在的相关文献时,直接说"没找到符合条件的文献",不要为了凑数编一条出来。这条原则没有例外。
|
||||
|
||||
### 第四步:输出
|
||||
|
||||
按 verified / suspect / unverified 分组呈现结果,每条包含标题、作者年份、venue、标识符、一句话相关性说明。
|
||||
|
||||
`literature_search.py` 传了 `--bib-out` 参数时,会自动把所有 `verified` 的条目写成BibTeX文件,citation key 用"姓氏+年份"约定,可以直接导入 Zotero(配合 Better BibTeX 插件)。这些 key 也是后续`paper-writing-grounded`技能里`\cite{}`要用到的,两个技能之间通过这些key保持一致,不需要额外对照。
|
||||
|
||||
### 第五步:提醒中文文献的检索缺口
|
||||
|
||||
MCP 检索工具覆盖的是 arXiv/Semantic Scholar/Crossref 这类有公开 API 的英文为主的库,**知网、万方、维普这类中文数据库没有公开 API,搜不到很正常,不是技能出错**。遇到用户明显需要中文文献的场景,主动提醒:装好 Zotero Connector 浏览器插件,在浏览器里正常登录学校账号搜索、打开文献页面,点一下 Connector 图标就能把元数据和 PDF 存进 Zotero——这部分需要用户手动完成,不要尝试用检索工具"模拟"或"猜测"中文文献的存在。
|
||||
|
||||
### 第六步:归档
|
||||
|
||||
`output/` 目录只是脚本运行时的草稿区——里面混着每一轮探索性检索的原始JSON(包括被过滤掉的噪声,比如"Tolles""Lawson"被当成人名匹配出的无关文献),不适合作为最终交付物,而且随着会话增多会越堆越乱、也不方便下次会话或用户直接翻阅。
|
||||
|
||||
所以每次整理出一份**稳定可信的参考文献列表**(不管是第一轮检索还是后续多轮补充检索合并后的结果)之后,调用归档脚本把它固化到项目级目录,而不是留在技能自己的`output/`里:
|
||||
|
||||
```bash
|
||||
python3 scripts/archive_references.py "UAV aeromagnetic compensation" \
|
||||
--bib output/uav_aeromagnetic_compensation_final.bib \
|
||||
--project-root . \
|
||||
--pdfs-dir output/pdfs \
|
||||
--suspect "某条可疑文献标题|不建议引用的具体原因" \
|
||||
--notes "检索覆盖了哪些方向、哪些方向搜了但没结果、中文文献缺口提醒等"
|
||||
```
|
||||
|
||||
这会在 `<project-root>/references/<按主题自动生成的slug>/` 下生成:
|
||||
- `references.bib` —— 传入的bib文件原样拷贝过去
|
||||
- `pdfs/`(如果传了`--pdfs-dir`且里面有PDF)—— 一并拷贝过去
|
||||
- `README.md` —— 自动从bib里解析出条目列表(标题/年份/venue/DOI/note)生成索引,`--suspect`和`--notes`里的内容会分别整理进"不要引用"和"检索覆盖说明"两个小节
|
||||
|
||||
几个要点:
|
||||
- `--bib` 传的必须是**已经过滤掉无关噪声、只保留verified条目**的干净bib文件,不要把`literature_search.py`直接吐出来的、可能夹杂噪声的原始bib不加甄别地拿去归档。
|
||||
- 同一个`topic`名字多次调用会往同一个归档目录里覆盖更新(bib和README会被覆盖,pdfs按文件名去重合并),所以后续检索到更多文献后可以直接对同一个topic重新跑一遍归档脚本来更新,不需要手动合并。
|
||||
- 这一步做完之后可以明确告诉用户归档目录的路径,方便他们后续在`paper-writing-grounded`阶段直接引用。
|
||||
|
||||
## 和 paper-writing-grounded 技能的配合
|
||||
|
||||
这个技能负责把"真实存在、经过核实的文献"整理好并生成 BibTeX;写作阶段的 paper-writing-grounded 技能会直接消费这里产出的 citation key,正文引用只能来自这里核实过的条目,不会凭空生成新的引用。两个技能配合使用时,建议先跑完这个技能、拿到稳定的参考文献列表,再进入写作。
|
||||
@@ -0,0 +1,165 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Archive a finished literature-search-verify session into a permanent,
|
||||
project-level folder instead of leaving results sitting in the skill's
|
||||
own scratch output/ directory (which is easy to lose track of across
|
||||
sessions and isn't meant to be a durable deliverable location).
|
||||
|
||||
Bundles the verified BibTeX file -- and, if given, any downloaded PDFs --
|
||||
into <project-root>/references/<topic-slug>/, and writes a README.md
|
||||
index (entry list, suspect/unverified entries flagged separately, free-
|
||||
text coverage notes) so a future session or a human can find and trust
|
||||
what's there without re-reading the conversation that produced it.
|
||||
|
||||
No third-party dependencies; uses only the standard library.
|
||||
|
||||
CLI usage:
|
||||
python3 archive_references.py "UAV aeromagnetic compensation" \\
|
||||
--bib output/uav_aeromagnetic_compensation_final.bib \\
|
||||
--project-root . \\
|
||||
--pdfs-dir output/pdfs \\
|
||||
--suspect "Some fabricated-looking title|DOI resolves but venue is topically unrelated" \\
|
||||
--notes "Kalman-filter and GA/PSO angles searched, no on-topic hits found."
|
||||
|
||||
Output: prints the path of the archive directory that was created/updated.
|
||||
"""
|
||||
import argparse
|
||||
import os
|
||||
import re
|
||||
import shutil
|
||||
import sys
|
||||
from datetime import date
|
||||
|
||||
|
||||
def slugify(text):
|
||||
text = text.strip().lower()
|
||||
text = re.sub(r"[^a-z0-9]+", "_", text)
|
||||
return text.strip("_")[:60] or "references"
|
||||
|
||||
|
||||
def parse_bib_entries(bib_path):
|
||||
"""Minimal BibTeX parser -- just enough to pull key/title/year/venue/doi/note
|
||||
(plus the raw entry text, for reordering) for the README index. Not a
|
||||
general-purpose BibTeX parser."""
|
||||
with open(bib_path, encoding="utf-8") as f:
|
||||
content = f.read()
|
||||
|
||||
entries = []
|
||||
for m in re.finditer(r"@(\w+)\{([^,\n]+),(.*?)\n\}", content, re.S):
|
||||
entry_type, key, body = m.groups()
|
||||
fields = {}
|
||||
for fm in re.finditer(r"(\w+)\s*=\s*\{(.*?)\}\s*,?\s*(?=\n\s*\w+\s*=|\n\Z|\Z)", body, re.S):
|
||||
fields[fm.group(1).lower()] = re.sub(r"\s+", " ", fm.group(2)).strip()
|
||||
entries.append({"type": entry_type, "key": key.strip(), "raw": m.group(0).strip(), **fields})
|
||||
return entries
|
||||
|
||||
|
||||
def year_sort_key(entry):
|
||||
"""Chronological order, oldest first; entries with no parseable year sort last."""
|
||||
year_str = re.sub(r"[^0-9]", "", entry.get("year", "") or "")
|
||||
year = int(year_str) if year_str else 9999
|
||||
return (year, entry.get("key", ""))
|
||||
|
||||
|
||||
def build_readme(topic, entries, pdf_count, suspect, notes):
|
||||
lines = []
|
||||
lines.append(f"# {topic} — literature archive")
|
||||
lines.append("")
|
||||
lines.append(f"Archived: {date.today().isoformat()}")
|
||||
lines.append(f"Verified entries: {len(entries)}")
|
||||
lines.append(f"PDFs bundled: {pdf_count}")
|
||||
lines.append("")
|
||||
lines.append(
|
||||
"Every entry in `references.bib` passed independent verification "
|
||||
"(arXiv ID / DOI resolution and/or cross-source title match, "
|
||||
"similarity >= 0.9) via the literature-search-verify skill before "
|
||||
"being archived here. Citation keys follow the surname+year "
|
||||
"convention and are stable -- the paper-writing-grounded skill's "
|
||||
"`\\cite{}` calls should match these keys directly."
|
||||
)
|
||||
lines.append("")
|
||||
lines.append("## Entries (chronological, oldest first)")
|
||||
lines.append("")
|
||||
for e in entries:
|
||||
title = e.get("title", "?")
|
||||
year = e.get("year", "?")
|
||||
venue = e.get("journal") or e.get("booktitle") or e.get("school") or ""
|
||||
doi = e.get("doi", "")
|
||||
note = e.get("note", "")
|
||||
line = f"- **{e['key']}** ({year}) — {title}"
|
||||
if venue:
|
||||
line += f". *{venue}*"
|
||||
if doi:
|
||||
line += f". DOI: {doi}"
|
||||
lines.append(line)
|
||||
if note:
|
||||
lines.append(f" - Note: {note}")
|
||||
|
||||
if suspect:
|
||||
lines.append("")
|
||||
lines.append("## Flagged during search — NOT included above, do not cite")
|
||||
lines.append("")
|
||||
for s in suspect:
|
||||
parts = s.split("|", 1)
|
||||
title = parts[0].strip()
|
||||
reason = parts[1].strip() if len(parts) > 1 else ""
|
||||
lines.append(f"- {title}" + (f" — {reason}" if reason else ""))
|
||||
|
||||
if notes:
|
||||
lines.append("")
|
||||
lines.append("## Search coverage notes")
|
||||
lines.append("")
|
||||
lines.append(notes)
|
||||
|
||||
return "\n".join(lines) + "\n"
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
ap.add_argument("topic", help="Human-readable topic name, e.g. \"UAV aeromagnetic compensation\"")
|
||||
ap.add_argument("--bib", required=True, help="path to the curated/verified .bib file to archive")
|
||||
ap.add_argument("--project-root", default=".", help="project root; archive is written under <root>/references/<slug>/")
|
||||
ap.add_argument("--pdfs-dir", default=None, help="optional folder of open-access PDFs to copy alongside the bib")
|
||||
ap.add_argument("--suspect", action="append", default=[], help="title|reason of a suspect/unverified entry to log; repeatable")
|
||||
ap.add_argument("--notes", default=None, help="free-text notes on search coverage/gaps for the README")
|
||||
args = ap.parse_args()
|
||||
|
||||
if not os.path.isfile(args.bib):
|
||||
print(f"error: bib file not found: {args.bib}", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
|
||||
slug = slugify(args.topic)
|
||||
archive_dir = os.path.join(args.project_root, "references", slug)
|
||||
os.makedirs(archive_dir, exist_ok=True)
|
||||
|
||||
bib_dest = os.path.join(archive_dir, "references.bib")
|
||||
shutil.copyfile(args.bib, bib_dest)
|
||||
entries = parse_bib_entries(bib_dest)
|
||||
entries.sort(key=year_sort_key)
|
||||
|
||||
# Rewrite the archived .bib in chronological order (oldest first) so the
|
||||
# file itself, not just the README, reads as a timeline.
|
||||
header = f"% {args.topic} -- verified references, chronological order\n% Archived {date.today().isoformat()}\n\n"
|
||||
with open(bib_dest, "w", encoding="utf-8") as f:
|
||||
f.write(header)
|
||||
f.write("\n\n".join(e["raw"] for e in entries))
|
||||
f.write("\n")
|
||||
|
||||
pdf_count = 0
|
||||
if args.pdfs_dir and os.path.isdir(args.pdfs_dir):
|
||||
pdf_dest_dir = os.path.join(archive_dir, "pdfs")
|
||||
os.makedirs(pdf_dest_dir, exist_ok=True)
|
||||
for fn in sorted(os.listdir(args.pdfs_dir)):
|
||||
if fn.lower().endswith(".pdf"):
|
||||
shutil.copyfile(os.path.join(args.pdfs_dir, fn), os.path.join(pdf_dest_dir, fn))
|
||||
pdf_count += 1
|
||||
|
||||
readme = build_readme(args.topic, entries, pdf_count, args.suspect, args.notes)
|
||||
with open(os.path.join(archive_dir, "README.md"), "w", encoding="utf-8") as f:
|
||||
f.write(readme)
|
||||
|
||||
print(archive_dir)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,160 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
End-to-end literature search: query arXiv + Semantic Scholar + Crossref,
|
||||
merge/dedupe candidates, independently verify each one, and emit both a
|
||||
human-readable report and BibTeX for the entries that passed verification.
|
||||
|
||||
This is the one script Claude should actually call for a normal literature
|
||||
search -- the individual search_*.py / verify_citation.py scripts exist
|
||||
mainly as building blocks it can reuse for one-off / follow-up lookups.
|
||||
|
||||
CLI usage:
|
||||
python3 literature_search.py "UAV magnetic compensation Tolles-Lawson" \\
|
||||
--max-per-source 8 --bib-out refs.bib
|
||||
|
||||
Output: prints a JSON report to stdout (one entry per merged candidate,
|
||||
with its verdict), and if --bib-out is given, writes BibTeX for every
|
||||
"verified" entry to that file (never for "suspect" or "unverified" ones).
|
||||
"""
|
||||
import sys
|
||||
import os
|
||||
import json
|
||||
import argparse
|
||||
import difflib
|
||||
import re
|
||||
|
||||
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
|
||||
|
||||
from search_arxiv import search_arxiv
|
||||
from search_semantic_scholar import search_s2
|
||||
from search_crossref import search_crossref
|
||||
from verify_citation import verify
|
||||
|
||||
|
||||
def _similar(a, b, threshold=0.88):
|
||||
if not a or not b:
|
||||
return False
|
||||
return difflib.SequenceMatcher(None, a.lower().strip(), b.lower().strip()).ratio() >= threshold
|
||||
|
||||
|
||||
def merge_candidates(all_results):
|
||||
"""Dedupe candidates that are the same paper found via multiple sources,
|
||||
merging their metadata (preferring whichever source has an identifier)."""
|
||||
merged = []
|
||||
for item in all_results:
|
||||
placed = False
|
||||
for m in merged:
|
||||
if _similar(item.get("title"), m.get("title")):
|
||||
# merge: fill in any missing fields, keep track of all sources
|
||||
for key in ("doi", "arxiv_id", "abstract", "venue", "year", "citation_count", "pdf_url"):
|
||||
if not m.get(key) and item.get(key):
|
||||
m[key] = item[key]
|
||||
m["sources"] = sorted(set(m.get("sources", [m.get("source")]) + [item.get("source")]))
|
||||
placed = True
|
||||
break
|
||||
if not placed:
|
||||
item = dict(item)
|
||||
item["sources"] = [item.get("source")]
|
||||
merged.append(item)
|
||||
return merged
|
||||
|
||||
|
||||
def make_bibtex_key(candidate, used_keys):
|
||||
authors = candidate.get("authors") or []
|
||||
surname = "unknown"
|
||||
if authors:
|
||||
first_author = authors[0]
|
||||
surname = first_author.strip().split()[-1].lower()
|
||||
surname = re.sub(r"[^a-z]", "", surname) or "unknown"
|
||||
year = str(candidate.get("year") or "nd")
|
||||
base = f"{surname}{year}"
|
||||
key = base
|
||||
suffix = ord("a")
|
||||
while key in used_keys:
|
||||
key = f"{base}{chr(suffix)}"
|
||||
suffix += 1
|
||||
used_keys.add(key)
|
||||
return key
|
||||
|
||||
|
||||
def to_bibtex(candidate, key):
|
||||
authors = candidate.get("authors") or []
|
||||
author_str = " and ".join(authors) if authors else "Unknown"
|
||||
title = candidate.get("title") or ""
|
||||
year = candidate.get("year") or ""
|
||||
venue = candidate.get("venue") or ""
|
||||
doi = candidate.get("doi") or ""
|
||||
arxiv_id = candidate.get("arxiv_id") or ""
|
||||
|
||||
if arxiv_id and not venue:
|
||||
entry_type = "misc"
|
||||
fields = [
|
||||
("author", author_str),
|
||||
("title", title),
|
||||
("year", str(year)),
|
||||
("eprint", arxiv_id),
|
||||
("archivePrefix", "arXiv"),
|
||||
]
|
||||
else:
|
||||
entry_type = "article"
|
||||
fields = [
|
||||
("author", author_str),
|
||||
("title", title),
|
||||
("journal", venue),
|
||||
("year", str(year)),
|
||||
]
|
||||
if doi:
|
||||
fields.append(("doi", doi))
|
||||
|
||||
lines = [f"@{entry_type}{{{key},"]
|
||||
for k, v in fields:
|
||||
if v:
|
||||
lines.append(f" {k} = {{{v}}},")
|
||||
lines.append("}")
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def run(query, max_per_source=8):
|
||||
all_results = []
|
||||
errors = {}
|
||||
for name, fn in (("arxiv", search_arxiv), ("semantic_scholar", search_s2), ("crossref", search_crossref)):
|
||||
try:
|
||||
all_results.extend(fn(query, max_per_source))
|
||||
except Exception as e:
|
||||
errors[name] = str(e)
|
||||
|
||||
merged = merge_candidates(all_results)
|
||||
|
||||
used_keys = set()
|
||||
for cand in merged:
|
||||
result = verify(title=cand.get("title"), arxiv_id=cand.get("arxiv_id"), doi=cand.get("doi"))
|
||||
cand["verdict"] = result["verdict"]
|
||||
cand["verification_checks"] = result["checks"]
|
||||
if result["verdict"] == "verified":
|
||||
cand["bibtex_key"] = make_bibtex_key(cand, used_keys)
|
||||
|
||||
return {"query": query, "search_errors": errors, "candidates": merged}
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
ap = argparse.ArgumentParser(description=__doc__)
|
||||
ap.add_argument("query")
|
||||
ap.add_argument("--max-per-source", type=int, default=8)
|
||||
ap.add_argument("--bib-out", default=None, help="path to write BibTeX for verified entries")
|
||||
args = ap.parse_args()
|
||||
|
||||
try:
|
||||
report = run(args.query, args.max_per_source)
|
||||
except Exception as e:
|
||||
print(json.dumps({"error": str(e)}, ensure_ascii=False))
|
||||
sys.exit(1)
|
||||
|
||||
print(json.dumps(report, ensure_ascii=False, indent=2))
|
||||
|
||||
if args.bib_out:
|
||||
verified = [c for c in report["candidates"] if c["verdict"] == "verified"]
|
||||
with open(args.bib_out, "w", encoding="utf-8") as f:
|
||||
for cand in verified:
|
||||
f.write(to_bibtex(cand, cand["bibtex_key"]))
|
||||
f.write("\n\n")
|
||||
sys.stderr.write(f"Wrote {len(verified)} verified BibTeX entries to {args.bib_out}\n")
|
||||
@@ -0,0 +1,80 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Search arXiv via its public Atom API. No API key required.
|
||||
|
||||
CLI usage:
|
||||
python3 search_arxiv.py "UAV magnetic compensation" --max 10
|
||||
|
||||
Importable:
|
||||
from search_arxiv import search_arxiv
|
||||
"""
|
||||
import sys
|
||||
import json
|
||||
import argparse
|
||||
import urllib.request
|
||||
import urllib.parse
|
||||
import xml.etree.ElementTree as ET
|
||||
|
||||
ARXIV_API = "http://export.arxiv.org/api/query"
|
||||
NS = {"atom": "http://www.w3.org/2005/Atom"}
|
||||
|
||||
|
||||
def search_arxiv(query, max_results=10, timeout=20):
|
||||
params = {
|
||||
"search_query": f"all:{query}",
|
||||
"start": 0,
|
||||
"max_results": max_results,
|
||||
"sortBy": "relevance",
|
||||
"sortOrder": "descending",
|
||||
}
|
||||
url = f"{ARXIV_API}?{urllib.parse.urlencode(params)}"
|
||||
with urllib.request.urlopen(url, timeout=timeout) as resp:
|
||||
data = resp.read()
|
||||
root = ET.fromstring(data)
|
||||
results = []
|
||||
for entry in root.findall("atom:entry", NS):
|
||||
id_el = entry.find("atom:id", NS)
|
||||
title_el = entry.find("atom:title", NS)
|
||||
summary_el = entry.find("atom:summary", NS)
|
||||
published_el = entry.find("atom:published", NS)
|
||||
if id_el is None or title_el is None:
|
||||
continue
|
||||
arxiv_id_full = id_el.text.strip()
|
||||
arxiv_id = arxiv_id_full.rsplit("/", 1)[-1]
|
||||
title = " ".join(title_el.text.split())
|
||||
summary = " ".join(summary_el.text.split()) if summary_el is not None else ""
|
||||
authors = [
|
||||
a.find("atom:name", NS).text
|
||||
for a in entry.findall("atom:author", NS)
|
||||
if a.find("atom:name", NS) is not None
|
||||
]
|
||||
published = published_el.text[:10] if published_el is not None else None
|
||||
pdf_url = None
|
||||
for link in entry.findall("atom:link", NS):
|
||||
if link.attrib.get("title") == "pdf":
|
||||
pdf_url = link.attrib.get("href")
|
||||
results.append({
|
||||
"source": "arxiv",
|
||||
"arxiv_id": arxiv_id,
|
||||
"title": title,
|
||||
"authors": authors,
|
||||
"year": published[:4] if published else None,
|
||||
"published": published,
|
||||
"abstract": summary,
|
||||
"pdf_url": pdf_url,
|
||||
"doi": None,
|
||||
})
|
||||
return results
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
ap = argparse.ArgumentParser(description=__doc__)
|
||||
ap.add_argument("query")
|
||||
ap.add_argument("--max", type=int, default=10)
|
||||
args = ap.parse_args()
|
||||
try:
|
||||
out = search_arxiv(args.query, args.max)
|
||||
print(json.dumps(out, ensure_ascii=False, indent=2))
|
||||
except Exception as e:
|
||||
print(json.dumps({"error": str(e)}, ensure_ascii=False))
|
||||
sys.exit(1)
|
||||
@@ -0,0 +1,65 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Search the Crossref works API. No API key required.
|
||||
Good for journal articles / DOIs that arXiv and Semantic Scholar might miss.
|
||||
|
||||
CLI usage:
|
||||
python3 search_crossref.py "UAV magnetic compensation" --max 10
|
||||
|
||||
Importable:
|
||||
from search_crossref import search_crossref
|
||||
"""
|
||||
import sys
|
||||
import json
|
||||
import argparse
|
||||
import urllib.request
|
||||
import urllib.parse
|
||||
|
||||
CROSSREF_API = "https://api.crossref.org/works"
|
||||
UA = "literature-search-verify-skill/1.0 (mailto:research-assistant@example.com)"
|
||||
|
||||
|
||||
def search_crossref(query, max_results=10, timeout=20):
|
||||
params = {"query": query, "rows": max_results}
|
||||
url = f"{CROSSREF_API}?{urllib.parse.urlencode(params)}"
|
||||
req = urllib.request.Request(url, headers={"User-Agent": UA})
|
||||
with urllib.request.urlopen(req, timeout=timeout) as resp:
|
||||
data = json.loads(resp.read())
|
||||
results = []
|
||||
for item in data.get("message", {}).get("items", []) or []:
|
||||
titles = item.get("title") or []
|
||||
title = titles[0] if titles else ""
|
||||
authors = []
|
||||
for a in item.get("author", []) or []:
|
||||
name = " ".join(filter(None, [a.get("given"), a.get("family")]))
|
||||
if name:
|
||||
authors.append(name)
|
||||
year = None
|
||||
date_parts = (item.get("issued", {}) or {}).get("date-parts")
|
||||
if date_parts and date_parts[0]:
|
||||
year = date_parts[0][0]
|
||||
containers = item.get("container-title") or []
|
||||
results.append({
|
||||
"source": "crossref",
|
||||
"title": title,
|
||||
"authors": authors,
|
||||
"year": year,
|
||||
"venue": containers[0] if containers else None,
|
||||
"doi": item.get("DOI"),
|
||||
"arxiv_id": None,
|
||||
"abstract": None,
|
||||
})
|
||||
return results
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
ap = argparse.ArgumentParser(description=__doc__)
|
||||
ap.add_argument("query")
|
||||
ap.add_argument("--max", type=int, default=10)
|
||||
args = ap.parse_args()
|
||||
try:
|
||||
out = search_crossref(args.query, args.max)
|
||||
print(json.dumps(out, ensure_ascii=False, indent=2))
|
||||
except Exception as e:
|
||||
print(json.dumps({"error": str(e)}, ensure_ascii=False))
|
||||
sys.exit(1)
|
||||
@@ -0,0 +1,59 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Search the Semantic Scholar Graph API. No API key required for light use;
|
||||
set the S2_API_KEY environment variable for higher rate limits.
|
||||
|
||||
CLI usage:
|
||||
python3 search_semantic_scholar.py "UAV magnetic compensation" --max 10
|
||||
|
||||
Importable:
|
||||
from search_semantic_scholar import search_s2
|
||||
"""
|
||||
import sys
|
||||
import os
|
||||
import json
|
||||
import argparse
|
||||
import urllib.request
|
||||
import urllib.parse
|
||||
|
||||
S2_API = "https://api.semanticscholar.org/graph/v1/paper/search"
|
||||
FIELDS = "title,authors,year,venue,externalIds,abstract,citationCount"
|
||||
|
||||
|
||||
def search_s2(query, max_results=10, timeout=20):
|
||||
params = {"query": query, "limit": max_results, "fields": FIELDS}
|
||||
url = f"{S2_API}?{urllib.parse.urlencode(params)}"
|
||||
req = urllib.request.Request(url)
|
||||
api_key = os.environ.get("S2_API_KEY")
|
||||
if api_key:
|
||||
req.add_header("x-api-key", api_key)
|
||||
with urllib.request.urlopen(req, timeout=timeout) as resp:
|
||||
data = json.loads(resp.read())
|
||||
results = []
|
||||
for p in data.get("data", []) or []:
|
||||
ext = p.get("externalIds") or {}
|
||||
results.append({
|
||||
"source": "semantic_scholar",
|
||||
"title": p.get("title"),
|
||||
"authors": [a.get("name") for a in (p.get("authors") or [])],
|
||||
"year": p.get("year"),
|
||||
"venue": p.get("venue"),
|
||||
"doi": ext.get("DOI"),
|
||||
"arxiv_id": ext.get("ArXiv"),
|
||||
"abstract": p.get("abstract"),
|
||||
"citation_count": p.get("citationCount"),
|
||||
})
|
||||
return results
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
ap = argparse.ArgumentParser(description=__doc__)
|
||||
ap.add_argument("query")
|
||||
ap.add_argument("--max", type=int, default=10)
|
||||
args = ap.parse_args()
|
||||
try:
|
||||
out = search_s2(args.query, args.max)
|
||||
print(json.dumps(out, ensure_ascii=False, indent=2))
|
||||
except Exception as e:
|
||||
print(json.dumps({"error": str(e)}, ensure_ascii=False))
|
||||
sys.exit(1)
|
||||
@@ -0,0 +1,122 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Independently cross-verify a single candidate citation. This is the anti-
|
||||
hallucination check: it never trusts a single source. If a check cannot be
|
||||
run at all (e.g. no network), that check is reported as "skipped" -- never
|
||||
silently counted as a pass.
|
||||
|
||||
CLI usage:
|
||||
python3 verify_citation.py --title "Compensation of magnetic ..." \\
|
||||
--arxiv-id 2401.12345 --doi 10.1109/TGRS.2024.1234567
|
||||
|
||||
Importable:
|
||||
from verify_citation import verify
|
||||
"""
|
||||
import sys
|
||||
import json
|
||||
import argparse
|
||||
import difflib
|
||||
import urllib.request
|
||||
import urllib.parse
|
||||
import urllib.error
|
||||
import xml.etree.ElementTree as ET
|
||||
|
||||
ATOM_NS = {"atom": "http://www.w3.org/2005/Atom"}
|
||||
|
||||
|
||||
def title_similarity(a, b):
|
||||
if not a or not b:
|
||||
return 0.0
|
||||
return difflib.SequenceMatcher(None, a.lower().strip(), b.lower().strip()).ratio()
|
||||
|
||||
|
||||
def check_arxiv_id(arxiv_id, timeout=20):
|
||||
"""Confirm an arXiv ID actually resolves to a real paper."""
|
||||
try:
|
||||
url = f"http://export.arxiv.org/api/query?id_list={urllib.parse.quote(arxiv_id)}"
|
||||
with urllib.request.urlopen(url, timeout=timeout) as resp:
|
||||
data = resp.read()
|
||||
root = ET.fromstring(data)
|
||||
entry = root.find("atom:entry", ATOM_NS)
|
||||
if entry is None:
|
||||
return {"status": "fail", "reason": "arXiv ID not found"}
|
||||
title_el = entry.find("atom:title", ATOM_NS)
|
||||
title = " ".join(title_el.text.split()) if title_el is not None else None
|
||||
return {"status": "pass", "canonical_title": title}
|
||||
except Exception as e:
|
||||
return {"status": "skipped", "reason": str(e)}
|
||||
|
||||
|
||||
def check_doi(doi, timeout=20):
|
||||
"""Confirm a DOI actually resolves via Crossref."""
|
||||
try:
|
||||
url = f"https://api.crossref.org/works/{urllib.parse.quote(doi)}"
|
||||
req = urllib.request.Request(
|
||||
url, headers={"User-Agent": "literature-search-verify-skill/1.0"}
|
||||
)
|
||||
with urllib.request.urlopen(req, timeout=timeout) as resp:
|
||||
data = json.loads(resp.read())
|
||||
titles = data.get("message", {}).get("title") or []
|
||||
return {"status": "pass", "canonical_title": titles[0] if titles else None}
|
||||
except urllib.error.HTTPError as e:
|
||||
if e.code == 404:
|
||||
return {"status": "fail", "reason": "DOI not found in Crossref"}
|
||||
return {"status": "skipped", "reason": f"HTTP {e.code}"}
|
||||
except Exception as e:
|
||||
return {"status": "skipped", "reason": str(e)}
|
||||
|
||||
|
||||
def check_title_cross_source(title, timeout=20):
|
||||
"""Independently re-search by title on a different source (Semantic
|
||||
Scholar) and require a near-exact title match. This is what catches a
|
||||
plausible-sounding but entirely invented title/author combination."""
|
||||
try:
|
||||
params = {"query": title, "limit": 3, "fields": "title"}
|
||||
url = f"https://api.semanticscholar.org/graph/v1/paper/search?{urllib.parse.urlencode(params)}"
|
||||
with urllib.request.urlopen(url, timeout=timeout) as resp:
|
||||
data = json.loads(resp.read())
|
||||
candidates = data.get("data", []) or []
|
||||
if not candidates:
|
||||
return {"status": "fail", "reason": "no matching title found on Semantic Scholar"}
|
||||
best = max(candidates, key=lambda p: title_similarity(title, p.get("title", "")))
|
||||
sim = title_similarity(title, best.get("title", ""))
|
||||
if sim >= 0.9:
|
||||
return {"status": "pass", "similarity": round(sim, 3), "matched_title": best.get("title")}
|
||||
return {"status": "fail", "similarity": round(sim, 3), "matched_title": best.get("title")}
|
||||
except Exception as e:
|
||||
return {"status": "skipped", "reason": str(e)}
|
||||
|
||||
|
||||
def verify(title=None, arxiv_id=None, doi=None):
|
||||
checks = {}
|
||||
if arxiv_id:
|
||||
checks["arxiv_id_check"] = check_arxiv_id(arxiv_id)
|
||||
if doi:
|
||||
checks["doi_check"] = check_doi(doi)
|
||||
if title:
|
||||
checks["title_cross_source_check"] = check_title_cross_source(title)
|
||||
|
||||
passed = [c for c in checks.values() if c["status"] == "pass"]
|
||||
failed = [c for c in checks.values() if c["status"] == "fail"]
|
||||
|
||||
if failed:
|
||||
verdict = "suspect" # something actively contradicted it
|
||||
elif passed:
|
||||
verdict = "verified" # at least one independent check passed
|
||||
else:
|
||||
verdict = "unverified" # everything skipped (e.g. no network) -- NOT the same as verified
|
||||
|
||||
return {"title": title, "arxiv_id": arxiv_id, "doi": doi, "verdict": verdict, "checks": checks}
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
ap = argparse.ArgumentParser(description=__doc__)
|
||||
ap.add_argument("--title", default=None)
|
||||
ap.add_argument("--arxiv-id", default=None)
|
||||
ap.add_argument("--doi", default=None)
|
||||
args = ap.parse_args()
|
||||
if not any([args.title, args.arxiv_id, args.doi]):
|
||||
print(json.dumps({"error": "provide at least one of --title/--arxiv-id/--doi"}))
|
||||
sys.exit(1)
|
||||
result = verify(args.title, args.arxiv_id, args.doi)
|
||||
print(json.dumps(result, ensure_ascii=False, indent=2))
|
||||
@@ -0,0 +1,50 @@
|
||||
---
|
||||
name: paper-writing-grounded
|
||||
description: Draft, revise, or polish academic paper and thesis sections (abstract, introduction, methods, experiments, related work, conclusion) for LaTeX conference/journal templates or Word/WPS-ready Chinese theses, while strictly preventing fabricated numbers, invented experimental results, or unsupported quantitative claims. Use this whenever the user asks to write, draft, outline, or polish any section of a paper or dissertation, wants raw results turned into prose, needs a LaTeX draft matching a conference template, wants Word/WPS text for a Chinese-language thesis, or wants AI-sounding writing "humanized" — anywhere the draft could end up stating a number, percentage, or comparison that didn't actually come from real data.
|
||||
---
|
||||
|
||||
# 论文写作(强制数据溯源版)
|
||||
|
||||
## 为什么需要这个技能
|
||||
|
||||
LLM写论文时最大的隐患不是文笔差,而是**在没有真实数据支撑的地方,顺手编一个"看起来合理"的数字**——比如"补偿后RMSE降低了23%"这种话,读起来完全正常,但如果这个23%不是从真实实验里来的,那就是数据捏造,一旦被发现是学术诚信问题,不是文笔问题。这个技能的核心规则就一条:**任何关于"我们的方法/我们的结果"的具体数字、百分比、对比、最高级描述,必须能追溯到用户提供的真实结果,追溯不到就必须明确标记出来,绝不能编一个数填上去。**
|
||||
|
||||
## 工作流程
|
||||
|
||||
### 第一步:建立"真实结果登记表"
|
||||
|
||||
在动笔写任何会出现具体数字的段落之前(尤其是实验/结果部分),先确认这一节要用到的真实数据来源——可以是用户粘贴的数字、一份CSV/JSON格式的指标文件、SimPEG/Harmonica处理流程跑出来的输出、图表标题里的数值等等。如果对话里还没有提供,直接问用户要,而不是先写着占位数字"等下再改"——占位数字很容易被忘记改掉,最后混进定稿。
|
||||
|
||||
把这一节会用到的每个具体数字记下来,连同它的出处(来自哪个文件/哪次实验/哪张图),这就是这一节写作时唯一可信的"真实结果登记表"。
|
||||
|
||||
### 第二步:分节起草
|
||||
|
||||
根据目标产出选结构:
|
||||
- **英文期刊/会议投稿(LaTeX)**:Abstract → Introduction → Related Work → Methods → Experiments → Conclusion,先问清楚目标模板(NeurIPS/ICLR/ICML,或者你们学校的LaTeX模板),没问清楚就先按通用IMRAD结构起草,后面再套模板。
|
||||
- **中文学位论文(Word/WPS)**:按论文各章节结构(通常是绪论/文献综述/方法/实验与结果/结论)起草,产出用docx技能生成格式化的Word文档。
|
||||
|
||||
### 第三步:强制溯源规则(核心)
|
||||
|
||||
起草时,每写到一个具体数字、百分比、显著性描述("显著优于"、"最优"这类)时,问自己:这个数字能不能在第一步的"真实结果登记表"里找到出处,或者能不能在literature-search-verify技能核实过的文献里找到支持?
|
||||
|
||||
- 能追溯到 → 正常写,行文里可以顺带标注来源(比如"如表2所示")
|
||||
- 追溯不到 → **不要编数字填上去**,改成明确的占位标记:`[需要数据:补偿后与补偿前的RMSE对比数值]`,让用户知道这里缺什么、需要补什么,而不是假装写完了
|
||||
|
||||
这条规则同样适用于"我们首次提出"、"计算效率更高"这类没有具体数字但仍是实质性主张的表述——同样需要能落到某个真实依据上,落不到就标记出来,不要含糊带过。
|
||||
|
||||
### 第四步:自查
|
||||
|
||||
起草完一节后,回头把这一节里所有的数字、比较、最高级表述再过一遍,确认每一条都能对应到登记表或已核实的引用。在回复末尾列出这次自查发现的、还没解决的`[需要数据]`标记,方便用户一次性补齐,而不是散落在长文里被忽略。
|
||||
|
||||
### 第五步:去AI味润色
|
||||
|
||||
润色不等于压缩。除非用户明确要求精简,不要为了让句子读起来"更自然"而删掉具体的研究对象、数据口径、方法条件、指标定义这些内容——这些恰恰是审稿人会重点核对的地方。去掉的应该是空洞的模板化表达(比如"综上所述,本研究具有重要意义"这类没有信息量的套话)和明显的AI腔调用词,而不是数据本身的精度和限定条件。
|
||||
|
||||
### 第六步:产出格式
|
||||
|
||||
- LaTeX:正文里的`\cite{}`只使用literature-search-verify技能核实过、生成过BibTeX的citation key,不自己编新的引用键。
|
||||
- Word/WPS:用docx技能产出格式化文档;提醒用户Zotero的Word插件可以配合插入引用,但WPS对该插件兼容性一般,可能需要先用Zotero导出RTF/纯文本引用再手动整理进WPS。
|
||||
|
||||
## 和 literature-search-verify 技能的配合
|
||||
|
||||
正文里任何"related work"或背景介绍部分引用的文献,只能来自literature-search-verify技能已核实的条目;这个技能不负责验证引用真实性,只负责确保"我们自己的实验结果"这部分不被编造数据污染。两者结合,才是"文献不编、数据不编"的完整闭环。
|
||||
Binary file not shown.
Reference in New Issue
Block a user