[2603.20208] RedacBench: Can AI Erase Your Secrets?

Computer Science > Computation and Language arXiv:2603.20208 (cs) [Submitted on 2 Mar 2026] Title:RedacBench: Can AI Erase Your Secrets? Authors:Hyunjun Jeon, Kyuyoung Kim, Jinwoo Shin View a PDF of the paper titled RedacBench: Can AI Erase Your Secrets?, by Hyunjun Jeon and 2 other authors View PDF HTML (experimental) Abstract:Modern language models can readily extract sensitive information from unstructured text, making redaction -- the selective removal of such information -- critical for data security. However, existing benchmarks for redaction typically focus on predefined categories of data such as personally identifiable information (PII) or evaluate specific techniques like masking. To address this limitation, we introduce RedacBench, a comprehensive benchmark for evaluating policy-conditioned redaction across domains and strategies. Constructed from 514 human-authored texts spanning individual, corporate, and government sources, paired with 187 security policies, RedacBench measures a model's ability to selectively remove policy-violating information while preserving the original semantics. We quantify performance using 8,053 annotated propositions that capture all inferable information in each text. This enables assessment of both security -- the removal of sensitive propositions -- and utility -- the preservation of non-sensitive propositions. Experiments across multiple redaction strategies and state-of-the-art language models show that while more advanced mode...