zhoujie
ea69788d0c
新增 scripts/http_utils.py 统一封装 HTTP 请求的限流重试(429/5xx 指数退避, 优先遵守服务端 Retry-After),search_arxiv/crossref/semantic_scholar.py 和 verify_citation.py 都已接入,不再各自裸调 urllib。 新增 scripts/search_openalex.py 作为第四个检索源:免费、无需 API key,覆盖面 比单独的 Crossref 更广,还能拿到开放获取PDF直链;已接入 literature_search.py 的主检索流程。verify_citation.py 的跨源标题核查同步改为同时查 Semantic Scholar 和 OpenAlex 两个独立源、任一命中相似度达标即通过,不再单点依赖 S2—— 这是针对"S2 被限流导致整批候选退化成 unverified"这个实际发生过的问题的直接 修复,已用真实网络请求验证:复测中 S2 确实当场返回了 429,靠 OpenAlex 兜底 最终判定仍然是 verified。 顺带修了 archive_references.py 的 slugify(),之前中文主题名会被正则全部 过滤掉、退化成通用的 "references",导致不同中文主题的归档目录互相冲突。 scripts/tests/ 下补了 37 个 unittest(全部 mock 网络请求,不发真实请求), 覆盖 verify_citation 的三档判定和双源核查合并逻辑、archive_references 的 bib 解析边界情况(嵌套花括号、中文主题名)、literature_search 的候选去重 合并与 BibTeX 生成、http_utils 的重试逻辑。用标准库 unittest 而不是 pytest, 和这些脚本本身不引入第三方依赖的原则保持一致。 CLAUDE.md 的下一步计划里,这一项已标记为完成。 Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
47 lines
1.6 KiB
Python
47 lines
1.6 KiB
Python
#!/usr/bin/env python3
|
|
"""
|
|
Shared HTTP GET helper with exponential-backoff retry for rate limiting (429)
|
|
and transient server errors (5xx). Stdlib only.
|
|
|
|
Every search_*.py / verify_citation.py network call should go through this
|
|
instead of calling urllib.request.urlopen directly, so a single source being
|
|
temporarily rate-limited doesn't quietly degrade a whole batch of candidates
|
|
to "unverified" the way Semantic Scholar's 429s did before this existed.
|
|
|
|
Importable:
|
|
from http_utils import fetch
|
|
"""
|
|
import time
|
|
import urllib.request
|
|
import urllib.error
|
|
|
|
RETRYABLE_CODES = {429, 500, 502, 503, 504}
|
|
|
|
|
|
def _wait_seconds(http_error, attempt, base=1.0, cap=20.0):
|
|
retry_after = http_error.headers.get("Retry-After") if http_error.headers else None
|
|
if retry_after:
|
|
try:
|
|
return min(float(retry_after), cap)
|
|
except ValueError:
|
|
pass
|
|
return min(base * (2 ** attempt), cap)
|
|
|
|
|
|
def fetch(url, headers=None, timeout=20, max_retries=4):
|
|
"""GET url, retrying on 429/5xx with exponential backoff (honoring a
|
|
Retry-After header when the server sends one). Returns the raw response
|
|
bytes. Raises the underlying urllib error if retries are exhausted or the
|
|
error isn't retryable (e.g. 404)."""
|
|
req = urllib.request.Request(url, headers=headers or {})
|
|
attempt = 0
|
|
while True:
|
|
try:
|
|
with urllib.request.urlopen(req, timeout=timeout) as resp:
|
|
return resp.read()
|
|
except urllib.error.HTTPError as e:
|
|
if e.code not in RETRYABLE_CODES or attempt >= max_retries:
|
|
raise
|
|
time.sleep(_wait_seconds(e, attempt))
|
|
attempt += 1
|