Logo image
Back
Learning vs Retrieval: The Role of In-Context Examples in Regression with Large Language Models
Conference proceeding   Open access

Learning vs Retrieval: The Role of In-Context Examples in Regression with Large Language Models

Aliakbar Nafar, Kristen Brent Venable and Parisa Kordjamshidi
Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), Vol.1, pp.8206-8229
Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics ( Albuquerque, New Mexico, USA, 04/29/2025–05/04/2025)
2025
Web of Science ID: WOS:001611654000098

Metrics

1 Record Views
1 Times Cited - Web of Science

Abstract

Generative Large Language Models (LLMs) are capable of being in-context learners. However, the underlying mechanism of in-context learning (ICL) is still a major research question, and experimental research results about how models exploit ICL are not always consistent. In this work, we propose a framework for evaluating in-context learning mechanisms, which we claim are a combination of retrieving internal knowledge and learning from in-context examples by focusing on regression tasks. First, we show that LLMs can solve real-world regression problems and then design experiments to measure the extent to which the LLM retrieves its internal knowledge versus learning from in-context examples. We argue that this process lies on a spectrum between these two extremes. We provide an in-depth analysis of the degrees to which these mechanisms are triggered depending on various factors, such as prior knowledge about the tasks and the type and richness of the information provided by the in-context examples. We employ three LLMs and utilize multiple datasets to corroborate the robustness of our findings. Our results shed light on how to engineer prompts to leverage meta-learning from in-context examples and foster knowledge retrieval depending on the problem being addressed.
url
Learning vs RetrievalView
Published (Version of record)link to article Open

Related links

Details

Logo image