hmmm, good point Ben. I guess it is a proofing service!

I feel like a pre-reviewed copy both reviewers get makes the most sense? But then I guess someone has to review the LLM's review ..... so now they have to comment on that too.  So almost like a 3rd reviewer which does a 1st pass to proof it?

If it's available to researchers to use prior to submission one would hope they use it so it doesn't find much - so maybe not much of an imposition particulalry if it removes a lot of the mundine.

I think LLM’s could be very useful in picking up common methodological and statistical mistakes as well as deviations from the journals style. Made even more useful if each journal trained up their own and made it available to people submitting papers. It could really boost the quality of statistics if done right! It’s exactly what LLM’s are good at, picking up common mistakes which are highly represented in the training set.

What it can’t do very well though (if at all) is provide useful commentary on new ideas or discoveries – particularly in the context of a fast moving field. There simply isn’t enough information in the training set. It would be spectacularly bad at anything that goes against conventional wisdom too, which would be disastrous for pretty much every domain I can think of. Bad in science, but imagine its impact on philosophy, history or the different interpretations of art?? What are all the art critics going to argue about!!?? ;)

So seems to me that LLM’s have the potential to be very useful in taking out the mundane from reviewing and also improving statistical methodology. Which would leave reviewers the task of focusing on the intellectual contribution of the paper. Which sounds like the fun bit really!!!

I don't really want to comment but hate being part of an anodyne forum - I could comment on that at length too - but if Chris can point me at anywhere that is not anodyne these days etc

I have strong views about the level of most people's knowledge of LLMs to use them for such important professional tasks - I could comment at lenght on that too but etc.

According to Chat GPT those two reviews had different style and appeared not to be from the same source but could possibly have been through some later review process leading to similatirites

PS Thanks to Beth for posting the clause that says we can engage in controversy

“I do not seek controversy for the sake of noise, but when truth is buried beneath the weight of silence, I have no choice but to speak—bearing only the evidence, and nothing more.”

I have heard that referees are using LLM’s to “speed up” the reviewing process. I recently received two reviews from SPL. The paper is just a silly little result which works out very well, so I wanted to send it somewhere. The reviews are attached.

These two reviews do not seem independent to me. I have annotated (and sent back to the editor). Check the attached if you are interested. Some comments indicate that the reviewer really did not read even the abstract of the paper. (Yet it looks like some human input has been added).

Would LLM’s produce identical reviews or just similar? I guess it depends a lot on the prompt and the exact model being used.

SPL (in rejecting the paper) also sent me a list of suggested journal where I might sent it and they would facilitate transfer.

• Journal of Multivariate Analysis (there is nothing multivariate in my paper).
• International Journal of Approximate Reasoning
• Informatics in Medicine Unlocked (open access, costs $3000, seems predatory)
• Results in Applied Mathematics (open access, costs $2500, seems predatory)
• Franklin Open (what the hell?) (open access, costs $3000, seems predatory)

This list was also surely generated by a LLM. No academic with a passing knowledge of statistics or data science who has seen the abstract (or even title) of my paper would suggest this.

Can LLM’s be used for good instead of evil?

An experienced colleague, who is pretty bullish on AI, recently had two LLM’s produce hostile reports of his own unpublished research. Some of it was nonsense, but it did alert him to some clear weaknesses which allowed him to revise. It also revealed some weaknesses that he could not address. (He told me he just hopes that the referees do not use the same LLM!).