Making Heads and Tails of Models with Marginal Calibration for Sparse Tagsets

09/15/2021
by   Michael Kranzlein, et al.
0

For interpreting the behavior of a probabilistic model, it is useful to measure a model's calibration–the extent to which it produces reliable confidence scores. We address the open problem of calibration for tagging models with sparse tagsets, and recommend strategies to measure and reduce calibration error (CE) in such models. We show that several post-hoc recalibration techniques all reduce calibration error across the marginal distribution for two existing sequence taggers. Moreover, we propose tag frequency grouping (TFG) as a way to measure calibration error in different frequency bands. Further, recalibrating each group separately promotes a more equitable reduction of calibration error across the tag frequency spectrum.

READ FULL TEXT

Please sign up or login with your details

Forgot password? Click here to reset