๐ Frameworks
AI generated
New technology to measure rhetorical style
## Introduction
The technology of artificial intelligence is growing rapidly and with it new concerns are being created about the 'hype' in machine learning publications. However, it remains difficult to measure rhetorical style independently of substantive content. In this context, the new LLM-based platform presents an innovative method to disentangle rhetorical style from substantive content.
The new framework operates as follows: multiple LLM configurations generate counterfactual writings based on the same substantive content, and a judge LLM compares these writings through pairwise evaluations, and the results are aggregated using the Bradley-Terry model.
We use this method to analyze 8,485 ICLR submissions sampled from 2017 to 2025, generating over 250,000 counterfactual writings and providing a large-scale quantification of rhetorical style in ML papers.
We find that visionary framing significantly predicts downstream attention, including citations and media attention, even after controlling for peer-review evaluations. In addition, we observe a sharp rise in rhetorical strength after 2023, and provide empirical evidence showing that this increase is largely driven by the adoption of LLM-based writing assistance. The reliability of our framework is validated by its robustness to the choice of personas and the high correlation between LLM judgments and human annotations. Our work demonstrates that LLMs can serve as instruments to measure and improve scientific evaluation.
๐ฌ Commenti (0)
๐ Accedi o registrati per commentare gli articoli.
Nessun commento ancora. Sii il primo a commentare!