STaRK

STaRK is a large-scale benchmark designed to evaluate retrieval systems powered by large language models (LLMs) on semi-structured knowledge bases. It addresses the challenge of assessing how effectively LLMs can extract relevant information from both textual and relational sources, with a focus on domains such as product search, academic paper search, and biomedical inquiries. The benchmark is characterized by its inclusion of diverse, natural-sounding, and contextually complex queries that reflect real-world information needs.

The platform features a collection of synthesized and human-generated queries, aiming to simulate practical user requests and provide authentic evaluation scenarios. To ensure reliable assessment, STaRK provides precisely verified ground truth answers and nodes, employing both automatic and manual filtering processes. The benchmark covers a broad spectrum of semi-structured knowledge, presenting tasks that require nuanced reasoning and the ability to navigate complex relational and textual requirements.

STaRK is intended for researchers and practitioners interested in the performance of LLM-driven retrieval systems, particularly in areas where the effectiveness of these models remains underexplored. By offering challenging benchmarks and realistic tasks, STaRK sets a new standard for evaluating retrieval capabilities and encourages further research in the field.

The description covers all explicitly supported facts without extrapolation.