Abstract

Large Language Models (LLMs) such as GPT-4 have shown enough promise in the few-shot learning context to suggest use in the generation of "silver" data and refinement of new ontologies through iterative application and review. Such workflows become more effective with reliable confidence estimation. Unfortunately, confidence estimation is a documented weakness of models such as GPT-4, and established methods to compensate require significant additional complexity and computation. The present effort explores methods for effective confidence estimation with GPT-4 with few-shot learning for event detection in the BETTER ontology as a vehicle. The key innovation is expanding the prompt and task presented to GPT-4 to provide License to speculate when unsure and Opportunity to quantify and explain its uncertainty (L&O). This approach improves accuracy and provides usable confidence measures (0.759 AUC) with no additional machinery.

Metadata

publication

arXiv preprint arXiv:2408.00914, 2024

year

2024

publication date

2024/8/1

authors

Steven Fincke, Adrien Bibal, Elizabeth Boschee

link

https://arxiv.org/abs/2408.00914

resource_link

https://arxiv.org/pdf/2408.00914

journal

arXiv preprint arXiv:2408.00914