Phantom: A Benchmark for Hallucination Detection in Financial Long-Context QA

Authors: Lanlan Ji, Dominic Seyler, Gunkirat Kaur, Manjunath Hegde, Koustuv Dasgupta, Bing Xiang

This is the repository containing the dataset for the submission mentioned above.

This dataset is designed for hallucination detection in language models. It includes multiple variants of the Phantom dataset with different token lengths (seed, 2k, 5K, 10K, 20K, 30K) for long context experiments , segments (beginning, middle, end), and filing types (e.g., 10K, DEF14A).

Dataset Structure

The dataset is organized into multiple files, each representing a specific dataset version. Examples include:

• 10K_seed
• 497K_seed
• def14A_5000tokens_middle
• 10K_30000tokens_end
• ... and more

Each dataset is a single CSV file and is treated as a separate dataset with only a train split.

How to Use

To load a specific dataset configuration using the 🤗 datasets library:

from datasets import load_dataset

# Example: Load the 10K 10000-tokens (middle) dataset
df = load_dataset("seyled/Phantom_Hallucination_Detection", data_files="PhantomDataset/Phantom_10K_10000tokens_middle.csv")

Cite

@inproceedings{
ji2025phantom,
title={{PHANTOM}: A Benchmark for Hallucination Detection in Financial Long-Context {QA}},
author={Lanlan Ji and Dominic Seyler and Gunkirat Kaur and Manjunath Hegde and Koustuv Dasgupta and Bing Xiang},
booktitle={The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track},
year={2025},
}

License

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

   http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.