AI researchers are questioning if peer review still works after ICLR released its latest conference reviews. Writing on Andrew Gelman’s Columbia blog on Nov. 9, 2026, professor Jessica Hullman said half of the reviews on her papers scored over 50% on AI detector Pangram.
Some papers even drew majorities of reviews topping 90% on Pangram, alongside odd spikes in boilerplate appendix mentions. Hullman argues that ICLR’s honor-system rule, asking reviewers to disclose limited AI help, failed completely under heavy workloads.
Executive Summary
- High AI Scores: About half of reviews on Hullman’s ICLR papers scored above 50% on Pangram, with several topping 90%.
- Broken Disclosure: ICLR let reviewers use limited AI if disclosed, yet almost nobody followed the honor rule.
- Paper Overload: Huge paper submission waves forced heavy stacks onto junior reviewers, pushing tired graders toward chatbots.
- Detector Limits: Pure AI detectors carry false-alarm risks, matching why Ivy League schools reject essay detectors.
- Trust Breakdown: When bots judge papers written with bots, science loses real human error checks.
Why Honor-System AI Disclosure Failed at Scale
Conference chairs tried to find a middle road. Rather than banning chatbots completely, ICLR let reviewers use limited AI tools if they flagged it. In practice, reviewers treated disclosure as optional.
Once authors opened their scores, social feeds lit up. Readers spotted stiff robotic layouts and weird praise for minor appendix tables. Relying on honor codes under tight grading deadlines simply doesn’t hold.
Silent AI writing reaches past student reviewers too. Last month, we reported on Dartmouth’s provost stepping down over undisclosed AI writing, showing rules fail when checks don’t exist.
How Record Paper Floods Pushed Reviewers Toward Bots
Blaming lazy graders misses the math problem under major AI events. Submission totals at venues like ICLR keep surging as labs use coding agents to crank out new drafts faster than experts can check them.
Organizers recruit huge pools of junior PhD students and hand them thick stacks of papers. Short on hours, tired reviewers feed PDFs into chatbots for quick summaries and clean critique bullets.
Hullman warns this breaks the basic pact of science. Peer review only works when a skilled human reads the claims, spots flawed math, and puts their own judgment behind the grade.
Fixes Conferences Need to Test Next
Detectors alone won’t save conferences because false flags hurt honest writers. Yet program chairs can fix incentives right away.
Stronger fixes include hard caps on papers per author, stricter vetting for new reviewers, and clean automated checks for code or proofs like those in recent formal math tools. That leaves humans free to judge real ideas instead of skimming formatting.
Peer Review Gap: Rules vs. Conference Reality
Core tensions noted in Jessica Hullman’s Nov. 9 analysis of ICLR reviews:
| Review Factor | Written Policy | What Happened in Practice |
|---|---|---|
| AI Use Rule | Limited use with disclosure | Widespread non-disclosure across reviews |
| Detector Signals | Human expert evaluation | ~50% of Hullman’s reviews >50% Pangram; some >90% |
| Reviewer Load | Thoughtful deep checks | Paper flood forces heavy loads onto junior staff |
| Appendix Notes | Targeted data checks | Odd surge in boilerplate appendix callouts |
Frequently Asked Questions
What sparked the November 2026 ICLR peer review debate?
After ICLR released paper reviews, authors flagged widespread chatbot prose online, and Jessica Hullman noted half her reviews topped 50% on Pangram.
Did ICLR allow reviewers to use AI tools?
Yes, in limited ways if disclosed, though Hullman wrote the honor-based disclosure rule broke down.
Why are academic reviewers turning to chatbots?
Huge waves of paper submissions leave overworked and junior reviewers short on time to read every draft by hand.
Can AI detectors solve automated peer reviews?
Not alone. Detectors risk false flags, so conferences also need submission caps and better reviewer incentives.
