Academic Project Page

CapCal: Learning from Emptiness

CapCal is a training-free calibration method for listwise LLM rerankers. It estimates the reranker's content-agnostic positional prior using placeholder passages, then calibrates output probabilities so ranking decisions rely less on structural position bias.

PaperPDFProject PageCodeCitationGitHub Repo
1. Paper2. Highlights3. Method At A Glance4. Repository Structure and Implementation5. Installation6. Data Layout7. Running the Released Experiments8. Outputs9. Configuration Notes10. Experimental Highlights11. Notes For Maintainers12. Citation13. Contact

Project Page ACL 2026 arXiv Python

Official code for "Learning from Emptiness: De-biasing Listwise Rerankers with Content-Agnostic Probability Calibration".

The paper studies a core failure mode of listwise LLM rerankers: even when the candidate set is semantically identical, the model may still prefer some list positions over others. Our method, CapCal, treats this behavior as an explicit prior, estimates it from content-agnostic inputs, and subtracts it during decoding. The public release of this repository keeps the code needed to reproduce the paper's results.

1. Paper

Hang Lv, Hongchao Gu, Ruiqing Yang, Liangyue Li, Zulong Chen, Defu Lian, Hao Wang, and Enhong Chen. Learning from Emptiness: De-biasing Listwise Rerankers with Content-Agnostic Probability Calibration. ACL 2026, 2026.

Paper / PDF / Project Page / Code / Citation

CapCal is a training-free calibration method for listwise LLM rerankers. It estimates the reranker's content-agnostic positional prior using placeholder passages, then calibrates output probabilities so ranking decisions rely less on structural position bias.

2. Highlights

3. Method At A Glance

Overview of CapCal

CapCal is a training-free calibration framework for listwise reranking.

Given a query and a list of candidate passages, we run the frozen reranker twice:

  1. Standard input: the original query and candidate passages.
  2. Content-agnostic input: the same query and the same passage identifiers, but every passage body is replaced with a placeholder.

The second pass exposes the model's input-agnostic positional prior. Intuitively, if the model still prefers some document indices when the content is empty, that preference is structural bias rather than semantic relevance.

CapCal then calibrates the ranking score by subtracting the excessive prior component from the standard probability:

S(d_i) = P(d_i | x) - α · (P(d_i | x_empty) - 1 / |C_k|)

where:

This repository includes the two variants reported in the main paper:

To make decoding valid for listwise ranking, the implementation also uses constrained decoding, so the model always produces a legal permutation of document identifiers.

4. Repository Structure and Implementation

The release is intentionally narrow: it focuses on the paper's main experimental surface rather than every exploratory branch developed during the project.

.
├── code/
│   ├── dataloader/         # TREC-COVID and TREC-DL loaders
│   ├── main/               # main experiment entry points and BM25 baseline
│   ├── scripts/            # public Qwen experiment wrappers
│   └── src/                # CapCal decoding, calibration, metrics, utilities
├── docs/
│   ├── assets/             # figures used in the documentation
│   └── REPRODUCTION.md     # compact reproduction guide
├── data/                   # expected data root (not versioned)
├── results/                # generated outputs
└── requirements.txt

The most important files are:

The public experiment surface now includes only:

5. Installation

We recommend Python 3.10.

conda create -n rerank-bias python=3.10 -y
conda activate rerank-bias
pip install -r requirements.txt

requirements.txt is intentionally minimal and contains only the core runtime packages needed for the public release:

If you want a compact step-by-step runbook, see docs/REPRODUCTION.md.

6. Data Layout

All loaders resolve datasets relative to RERANK_BIAS_DATA_ROOT. If the variable is unset, the repository defaults to ./data.

export RERANK_BIAS_DATA_ROOT=/path/to/your/data
export RERANK_BIAS_HF_HOME=$RERANK_BIAS_DATA_ROOT/huggingface

Expected layout:

$RERANK_BIAS_DATA_ROOT/
├── beir/
│   └── trec-covid/
├── trec-dl-2019/
├── trec-dl-2020/
├── trec/
│   ├── trec21/
│   ├── trec22/
│   └── trec23/
└── msmarco_v2_passage/

TREC-COVID

You can download the BEIR-formatted trec-covid data with:

python code/dataloader/download_datasets.py --datasets trec-covid

TREC-DL

TREC DL data is expected locally:

For dl21, dl22, and dl23, the loader also needs the MSMARCO v2 passage collection:

export RERANK_MSMARCO_V2_PASSAGE_DIR=/path/to/msmarco_v2_passage

If this variable is unset, the loader will look for msmarco_v2_passage under RERANK_BIAS_DATA_ROOT.

7. Running the Released Experiments

1. Verify the environment and dataset path

python code/main/main.py \
  --dataset_type beir \
  --dataset_name trec-covid \
  --num_queries 10 \
  --dry_run

2. Run fixed calibration

bash code/scripts/qwen3_1_7b_fixed.sh

3. Run adaptive calibration

bash code/scripts/qwen3_1_7b_adaptive.sh

4. Override datasets or calibration strengths

NUM_QUERIES=100 \
BIAS_RATES="1.0 1.5 2.0" \
TREC_DL_DATASETS="dl21 dl22 dl23" \
BEIR_DATASETS="trec-covid" \
bash code/scripts/qwen3_4b_fixed.sh

5. Compute BM25 baselines

bash code/scripts/calculate_bm25_baseline_all.sh

8. Outputs

By default:

9. Configuration Notes

10. Experimental Highlights

CapCal main reranking results

The paper's main result table is included here to show the reported NDCG@10 gains across MS MARCO and BEIR benchmarks before the distilled evidence summary.

The paper evaluates CapCal on 10 ranking benchmarks from MS MARCO, TREC-DL, and BEIR with Qwen models from 0.6B to 8B.

Setting Reported result Takeaway
Lightweight reranker In high-bias scenarios, CapCal improves small models by more than 10 NDCG points. Position calibration can unlock useful ranking quality from 0.6B-scale rerankers.
Qwen3-0.6B fixed comparison DL19 improves from 0.4916 to 0.5454, DL20 from 0.3415 to 0.4126, DL22 from 0.4858 to 0.5457, and FiQA from 0.1807 to 0.2364. The gains appear across both TREC-DL and BEIR-style settings.
Training-free baseline comparison On Qwen3-0.6B, CapCal matches or beats permutation self-consistency while using one additional forward pass instead of 10 full reranking calls. CapCal preserves single-pass-style efficiency better than permutation aggregation.
Larger reranker On Qwen3-8B, CapCal is better than PSC on DL20, DL22, DL23, Climate-FEVER, NFCorpus, and FiQA. The calibration remains useful beyond very small models.

Conclusion: CapCal targets a concrete deployment tradeoff: reduce listwise position bias without retraining and without paying the latency cost of repeated prompt permutations.

11. Notes For Maintainers

12. Citation

If you use this repository, please cite:

@misc{lv2026capcal,
  title={Learning from Emptiness: De-biasing Listwise Rerankers with Content-Agnostic Probability Calibration},
  author={Hang Lv and Hongchao Gu and Ruiqing Yang and Liangyue Li and Zulong Chen and Defu Lian and Hao Wang and Enhong Chen},
  year={2026},
  eprint={2604.10150},
  archivePrefix={arXiv},
  primaryClass={cs.AI},
  url={https://arxiv.org/abs/2604.10150},
}

13. Contact

For paper questions, please contact:

For repository issues, please open a GitHub issue in this repository.