How to spot academic AI abuse

The results

When it comes to the actual results in the article, there are four almost identical figures. All four show a simple bar chart, but in three of them all bars are of exactly equal height. In only one of these figures a minor difference between the bar heights is visible.

Unfortunately there is no calculation, or even some indication, to determine whether this minor difference is statistically significant or not. Moreover, while these figures appear rather uninformative, it is concluded from them that one of the methods that are compared in the article is superior to the others. In short, the results seem rather contrived and trivial, and do not appear to support the conclusions drawn from them.

The references

What really gave it away, though, were the references. Four of the references in the article are (supposedly) to my own work. These four references are combined together in the following image. However, only one of them (reference [7]) turns out to be correct.

[7] Wim Hordijk. Autocatalytic sets: From the origin of life to the economy.
BioScience, 63(11):877-881, 2013.

[18] Mike Steel and Wim Hordijk. Autocatalytic sets and the origin of life.
Journal of Systems Chemistry, 1(1):1-7, 2010.

[13] Wim Hordijk and Mike Steel. Autocatalytic sets and the origin of life.
Entropy, 12(7):1733-1742, 2010.

[18] Wim Hordijk and Mike Steel. Intersections of autocatalytic sets and their dynamics.
Acta Biotheoretica, 58(4):379-392, 2010.

Two of these references ([8] and [13]) list the same article title, but each one lists the two authors in a different order. Moreover, each one names a different journal in which the article was supposedly published. Only one of these (reference [13]) is correct, except that there is actually a third author on this particular publication, who is not listed in either reference.

Finally, the fourth reference ([18]) is an obvious AI hallucination, as it lists a completely made-up article title. It does name a journal in which we did publish an article once, but with a different title and in a different year than what is given in this particular reference.

AI hallucinations are partly a result of the way large language models are trained, where guessing (even if wrong) is rewarded over admitting ignorance. As such, hallucinations are not a bug, but a feature. A highly undesirable one, obviously.

A final notable thing regarding the references is that the author does not cite any of his own previous work. With such a highly technical article, one would expect the (single) author to have at least some relevant background and previous publications on the specific topic, but there is no evidence of this.

An AI detector

To make completely sure, I also uploaded this by now highly suspicious article to an academic AI detector. The screenshot below shows the report it generated.

In contrast, and as a check on the reliability of this report, I also uploaded one of my own articles on a related topic. In this case the verdict was “essentially human”, with only 1% of the article potentially AI-generated. That certainly leaves little doubt.

Borderline case

Interestingly, another highly mathematical article on a similar topic, and with my name again appearing in the list of references, was recently published in an actual academic journal. Two different AI detectors considered this article to be “mostly AI produced”. Although many authors these days use AI applications to edit and improve the text of their articles, nowhere in the article is this acknowledged. Normally, publishers would (or certainly should) require this to be explicitly disclosed.

Furthermore, although the authors do cite their own previous publications to show their expertise on the particular research topic, the article also contains several inconsistencies in the references. My own name appears in only one reference, but again with a completely made-up article title, and some (supposed) co-authors that I have never even published with. At least one other reference, which lists two of my actual co-authors, is also to a non-existent article. Unfortunately, all this leaves the reader wondering about the reliability and integrity of the article and its authors.

Environmental impact

Such careless (ab)use of AI technology in academia not only erodes public trust in science, it also contributes to an enormous environmental impact. A recent scientific report estimates that the total power demand of the largest AI data centers combined is approaching that of a country the size of the UK. Furthermore, the report suggests that the combined carbon footprint of these AI data centers is equivalent to that of New York City, and that their water footprint could be in the range of the global annual consumption of bottled water. These are staggering numbers indeed.

An additional and somewhat ironic problem is that these AI methods will now most likely be trained on the very output that they produced in the first place. Since many of these AI-generated academic articles are publicly available online, either on preprint servers or as actual open access publications, they can be freely included in the input data for subsequent training rounds of the AI methods. It thus literally becomes a case of “garbage in, garbage out”, but then multiplied, and with even more resources wasted.

Conclusions

Unfortunately, academia is not immune to AI abuse, given the increasing flood of AI-generated academic articles. Although most of these are posted on (non-peer-reviewed) preprint servers, some questionable practices apparently do manage to make it through peer review and get published in established academic journals.

Current AI methods are far from perfect, and similar problems appear in other areas as well, for example in tourism and health care. With a lack of proper AI regulation, it will be up to academics themselves (and also the general public) to spot and flag improper or even fraudulent use. Luckily at this point there are often still some clear give-aways, such as contrived or irrelevant results, and incorrect or even made-up references. However, with AI methods likely to become more capable over time, the question is whether it will still be possible to reliably detect academic AI abuse in the future. For now, increasing our own awareness of the problem, and knowing how to recognize such instances, seems to be the best defense.

Wim Hordijk is an independent scientist and writer, with a special interest in evolution and the historical developments of how things came to be, from the origin of life to technology and culture. More information about his research and publications can be found on his personal website.