Abstract

Scambaiting is a defense that engages with scammers to waste their resources and gain information about their fraud campaigns. This defense needs automation to scale to the vast number of scams we see today. In this paper, we propose a scalable, automated scambaiting system, Puppeteer, which leverages a large language model for response generation and state machines for conversation tracking. We measure Puppeteer’s effectiveness via a user study, where participants play a role of a scammer in two scam scenarios. Puppeteer convinced more than 72% of the participants that they were interacting with a human, and was able to extract information from 68% of participants. In comparison, using the same large language model without conversation tracking convinced only 54% of the participants that they were interacting with a human and obtained information from 54% of participants. Our results show potential for real-world use of Puppeteer. To the best of our knowledge, we are also the first to systematically evaluate a large language model for a scambaiting task.

Date

January 7, 2025

Authors

Pithayuth Charnsethikul, Benjamin Crotty, Jelena Mirkovic, Jeffrey Liu, Rishit Saiya, Genevieve Bartlett