[2602.23603] LFQA-HP-1M: A Large-Scale Human Preference Dataset for Long-Form Question Answering

Computer Science > Computation and Language arXiv:2602.23603 (cs) [Submitted on 27 Feb 2026] Title:LFQA-HP-1M: A Large-Scale Human Preference Dataset for Long-Form Question Answering Authors:Rafid Ishrak Jahan, Fahmid Shahriar Iqbal, Sagnik Ray Choudhury View a PDF of the paper titled LFQA-HP-1M: A Large-Scale Human Preference Dataset for Long-Form Question Answering, by Rafid Ishrak Jahan and 2 other authors View PDF HTML (experimental) Abstract:Long-form question answering (LFQA) demands nuanced evaluation of multi-sentence explanatory responses, yet existing metrics often fail to reflect human judgment. We present LFQA-HP-1M, a large-scale dataset comprising 1.3M human pairwise preference annotations for LFQA. We propose nine rubrics for answer quality evaluation, and show that simple linear models based on these features perform comparably to state-of-the-art LLM evaluators. We further examine transitivity consistency, positional bias, and verbosity biases in LLM evaluators and demonstrate their vulnerability to adversarial perturbations. Overall, this work provides one of the largest public LFQA preference datasets and a rubric-driven framework for transparent and reliable evaluation. Comments: Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Information Retrieval (cs.IR) Cite as: arXiv:2602.23603 [cs.CL]   (or arXiv:2602.23603v1 [cs.CL] for this version)   https://doi.org/10.48550/arXiv.2602.23603 Focus to learn more arXiv-issued DOI via D...