Econ Papers Daily / 期刊论文
大型语言模型与人类判断在科学同行评审中的作用：来自1000多篇已发表经济学论文的证据
Large language models and human judgement in scientific peer review: evidence from over 1000 published economics papers
摘要
The peer‐review process in economics faces two well‐documented challenges: a shortage of willing referees, and prolonged turnaround times. We examine whether large language models (LLMs) can help to address these inefficiencies without introducing systematic bias, using two pre‐registered experiments across four LLMs, and over 29,000 evaluations of 1220 anonymized papers. LLMs reliably differentiate research quality by journal rank, and ratings are positively correlated with realized citation impact. However, model performance varies substantially: GPT and Gemma show the strongest evaluative gradients, while LLaMA assigns the maximum score to 81% of papers, rendering it uninformative for quality discrimination. GPT excels at detecting AI‐generated content; Claude‐generated papers receive the highest quality ratings and are the hardest to detect. Varying author identity in the second experiment reveals that three of the four LLMs assign significantly higher ratings to papers attributed to prominent economists and elite Western institutions, and lower ratings to papers with non‐Western affiliations, mirroring the prestige gradients documented in the human peer‐review literature. These biases persist when paper quality is held constant, highlighting the necessity for author‐blind inputs for any editorial deployment of LLM‐assisted screening.
中文摘要
经济学同行评审流程面临两个有据可查的挑战：愿意参与评审的审稿人短缺，以及评审周期过长。我们通过两项预注册实验，涵盖四种大语言模型（LLMs），对1220篇匿名论文进行了超过29000次评估，考察大语言模型能否在不引入系统性偏差的前提下帮助解决这些效率问题。大语言模型能够可靠地依据期刊等级区分研究质量，且其评分与实际引用影响力呈正相关。然而，不同模型的表现差异显著：GPT和Gemma展现出最强的评价梯度，而LLaMA对81%的论文给出最高分，使其在质量区分方面缺乏信息量。GPT在检测AI生成内容方面表现优异；由Claude生成的论文获得最高的质量评分，且最难被检测。第二项实验中改变作者身份的结果显示，四种大语言模型中有三种对归属于知名经济学家和西方精英机构的论文给出显著更高的评分，而对具有非西方机构隶属关系的论文给出更低的评分，这与人类同行评审文献中记录的声望梯度如出一辙。在论文质量保持不变的情况下，这些偏差依然存在，凸显了在大语言模型辅助筛选的任何编辑部署中采用作者盲审输入的必要性。