ADAPT Centre co-Founder, Prof Vincent Wade, and final year PhD Candidate, Mayank Soni, recently co-authored a paper evaluating the capabilities of ChatGPT on Abstractive Summarisation through the use of automated metrics and blinded human reviewers. The paper is titled “Comparing Abstractive Summaries Generated by ChatGPT to Real Summaries Through Blinded Reviewers and Text Classification Algorithms”.
ChatGPT, developed by OpenAI, is currently being widely used by businesses and customers to accomplish numerous textual-based tasks. While currently appearing to be the most popular Large Language Model (LLM) in use, it is a fairly recent addition to the group. Prof. Wade and Mayank Soni have pursued this study to contribute to the body of systematic research studies on ChatGPT.
As outlined in the paper, Prof. Wade and Mayank Soni built automatic text classifiers to detect ChatGPT generated summaries. They record that while text classification algorithms can distinguish between real and generated summaries, humans are unable to distinguish between real summaries and those produced by ChatGPT.
The full paper is available here.
