Metrics
9 Record Views
Conference proceeding
Reasoning over Uncertain Text by Generative Large Language Models
Proceedings of the ... AAAI Conference on Artificial Intelligence, Vol.39(23), pp.24911-24920
AAAI Conference on Artificial Intelligence
AAAI Conference on Artificial Intelligence, 39th (Philadelphia, Pennyslvania, USA, 02/25/2025–03/04/2025)
04/11/2025
Web of Science ID: WOS:001477503400088
Abstract
This paper considers the challenges Large Language Models (LLMs) face when reasoning over text that includes information involving uncertainty explicitly quantified via probability values. This type of reasoning is relevant to a variety of contexts ranging from everyday conversations to medical decision-making. Despite improvements in the mathematical reasoning capabilities of LLMs, they still exhibit significant difficulties when it comes to probabilistic reasoning. To deal with this problem, we introduce the Bayesian Linguistic Inference Dataset (BLInD), a new dataset specifically designed to test the probabilistic reasoning capabilities of LLMs. We use BLInD to find out the limitations of LLMs for tasks involving probabilistic reasoning. In addition, we present several prompting strategies that map the problem to different formal representations, including Python code, probabilistic algorithms, and probabilistic logical programming. We conclude by providing an evaluation of our methods on BLInD and an adaptation of a causal reasoning question-answering dataset. Our empirical results highlight the effectiveness of our proposed strategies for multiple LLMs.
Code and Dataset - https://github.com/HLR/BLInD
Extended Version - https://arxiv.org/abs/2402.09614
Related links
Details
- Title
- Reasoning over Uncertain Text by Generative Large Language Models
- Publication Details
- Proceedings of the ... AAAI Conference on Artificial Intelligence, Vol.39(23), pp.24911-24920
- Resource Type
- Conference proceeding
- Conference
- AAAI Conference on Artificial Intelligence, 39th (Philadelphia, Pennyslvania, USA, 02/25/2025–03/04/2025)
- Publisher
- Association for the Advancement Artificial Intelligence
- Series
- AAAI Conference on Artificial Intelligence
- Number of pages
- 10
- Grant note
- N00014-23-1-2417 / Office of Naval Research (ONR); United States Department of Defense; United States Navy; Office of Naval Research
- Copyright
- © 2025, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved.
- Identifiers
- WOS:001477503400088; 99381474385406600
- Academic Unit
- Institute for Human and Machine Cognition; Intelligent Systems and Robotics; Hal Marcus College of Science and Engineering
- Language
- English