Logo image
Back
Reasoning over Uncertain Text by Generative Large Language Models
Conference proceeding

Reasoning over Uncertain Text by Generative Large Language Models

Aliakbar Nafar, Kristen Brent Venable and Parisa Kordjamshidi
Proceedings of the ... AAAI Conference on Artificial Intelligence, Vol.39(23), pp.24911-24920
AAAI Conference on Artificial Intelligence
AAAI Conference on Artificial Intelligence, 39th (Philadelphia, Pennyslvania, USA, 02/25/2025–03/04/2025)
04/11/2025
DOI: https://doi.org/10.1609/aaai.v39i23.34674
Web of Science ID: WOS:001477503400088

Metrics

9 Record Views
1 Times Cited - Web of Science

Abstract

This paper considers the challenges Large Language Models (LLMs) face when reasoning over text that includes information involving uncertainty explicitly quantified via probability values. This type of reasoning is relevant to a variety of contexts ranging from everyday conversations to medical decision-making. Despite improvements in the mathematical reasoning capabilities of LLMs, they still exhibit significant difficulties when it comes to probabilistic reasoning. To deal with this problem, we introduce the Bayesian Linguistic Inference Dataset (BLInD), a new dataset specifically designed to test the probabilistic reasoning capabilities of LLMs. We use BLInD to find out the limitations of LLMs for tasks involving probabilistic reasoning. In addition, we present several prompting strategies that map the problem to different formal representations, including Python code, probabilistic algorithms, and probabilistic logical programming. We conclude by providing an evaluation of our methods on BLInD and an adaptation of a causal reasoning question-answering dataset. Our empirical results highlight the effectiveness of our proposed strategies for multiple LLMs. Code and Dataset - https://github.com/HLR/BLInD Extended Version - https://arxiv.org/abs/2402.09614

Details

Logo image