Is This Abstract Generated by AI? A Research for the Gap between AI-generated Scientific Text and Human-written Scientific Text

by   Yongqiang Ma, et al.

BACKGROUND: Recent neural language models have taken a significant step forward in producing remarkably controllable, fluent, and grammatical text. Although some recent works have found that AI-generated text is not distinguishable from human-authored writing for crowd-sourcing workers, there still exist errors in AI-generated text which are even subtler and harder to spot. METHOD: In this paper, we investigate the gap between scientific content generated by AI and written by humans. Specifically, we first adopt several publicly available tools or models to investigate the performance for detecting GPT-generated scientific text. Then we utilize features from writing style to analyze the similarities and differences between the two types of content. Furthermore, more complex and deep perspectives, such as consistency, coherence, language redundancy, and factual errors, are also taken into consideration for in-depth analysis. RESULT: The results suggest that while AI has the potential to generate scientific content that is as accurate as human-written content, there is still a gap in terms of depth and overall quality. AI-generated scientific content is more likely to contain errors in language redundancy and factual issues. CONCLUSION: We find that there exists a “writing style” gap between AI-generated scientific text and human-written scientific text. Moreover, based on the analysis result, we summarize a series of model-agnostic or distribution-agnostic features, which could be utilized to unknown or novel domain distribution and different generation methods. Future research should focus on not only improving the capabilities of AI models to produce high-quality content but also examining and addressing ethical and security concerns related to the generation and the use of AI-generated content.


page 1

page 2

page 3

page 4


ChatGPT v Bard v Bing v Claude 2 v Aria v human-expert. How good are AI chatbots at scientific writing? (ver. 23Q3)

Historically, proficient writing was deemed essential for human advancem...

AI, write an essay for me: A large-scale comparison of human-written versus ChatGPT-generated essays

Background: Recently, ChatGPT and similar generative AI models have attr...

Towards Automatic Boundary Detection for Human-AI Hybrid Essay in Education

Human-AI collaborative writing has been greatly facilitated with the hel...

ChatGPT as the Transportation Equity Information Source for Scientific Writing

Transportation equity is an interdisciplinary agenda that requires both ...

PaperRobot: Incremental Draft Generation of Scientific Ideas

We present a PaperRobot who performs as an automatic research assistant ...

Is writing style predictive of scientific fraud?

The problem of detecting scientific fraud using machine learning was rec...

Everyone Can Be Picasso? A Computational Framework into the Myth of Human versus AI Painting

The recent advances of AI technology, particularly in AI-Generated Conte...

Please sign up or login with your details

Forgot password? Click here to reset