research
∙
07/07/2023
On the Efficacy of Sampling Adapters
Sampling is a common strategy for generating text from probabilistic mod...
share
research
∙
07/07/2023
Testing the Predictions of Surprisal Theory in 11 Languages
A fundamental result in psycholinguistics is that less predictable words...
research
∙
06/29/2023
Tokenization and the Noiseless Channel
Subword tokenization is a key part of many NLP pipelines. However, littl...
research
∙
06/29/2023
A Formal Perspective on Byte-Pair Encoding
Byte-Pair Encoding (BPE) is a popular algorithm used for tokenizing data...
research
∙
06/06/2023
A Cross-Linguistic Pressure for Uniform Information Density in Word Order
While natural languages differ widely in both canonical word order and w...
research
∙
12/20/2022
A Measure-Theoretic Characterization of Tight Language Models
Language modeling, a central task in natural language processing, involv...
research
∙
12/19/2022
A Natural Bias for Language Generation Models
After just a few hundred training updates, a standard probabilistic mode...
research
∙
11/25/2022
On the Effect of Anticipation on Reading Times
Over the past two decades, numerous studies have demonstrated how less p...
research
∙
10/24/2022
Mutual Information Alleviates Hallucinations in Abstractive Summarization
Despite significant progress in the quality of language generated from a...
research
∙
05/31/2022
Cluster-based Evaluation of Automatically Generated Text
While probabilistic language generators have improved dramatically over ...
research
∙
05/14/2022
Naturalistic Causal Probing for Morpho-Syntax
Probing has become a go-to methodology for interpreting and analyzing de...
research
∙
04/04/2022
Estimating the Entropy of Linguistic Distributions
Shannon entropy is often a quantity of interest to linguists studying th...
research
∙
03/31/2022
On the probability-quality paradox in language generation
When generating natural language from neural probabilistic models, high ...
research
∙
03/31/2022
Analyzing Wrap-Up Effects through an Information-Theoretic Lens
Numerous analyses of reading time (RT) data have been implemented – all ...
research
∙
03/29/2022
On Decoding Strategies for Neural Text Generators
When generating text from probabilistic models, the chosen decoding stra...
research
∙
02/01/2022
Typical Decoding for Natural Language Generation
Despite achieving incredibly low perplexities on myriad natural language...
research
∙
09/30/2021
A surprisal–duration trade-off across and within the world's languages
While there exist scores of natural languages, each with its unique feat...
research
∙
09/28/2021
On Homophony and Rényi Entropy
Homophony's widespread presence in natural languages is a controversial ...
research
∙
09/23/2021
Revisiting the Uniform Information Density Hypothesis
The uniform information density (UID) hypothesis posits a preference amo...
research
∙
09/22/2021
Conditional Poisson Stochastic Beam Search
Beam search is the default decoding strategy for many sequence generatio...
research
∙
09/20/2021
A Plug-and-Play Method for Controlled Text Generation
Large pre-trained language models have repeatedly shown their ability to...
research
∙
06/14/2021
Determinantal Beam Search
Beam search is a go-to strategy for decoding neural sequence models. The...
research
∙
06/02/2021
Is Sparse Attention more Interpretable?
Sparse attention has been claimed to increase model interpretability und...
research
∙
05/31/2021
Language Model Evaluation Beyond Perplexity
We propose an alternate approach to quantifying how well language models...
research
∙
05/15/2021
A Cognitive Regularizer for Language Modeling
The uniform information density (UID) hypothesis, which posits that spea...
research
∙
02/16/2021
Searching for Search Errors in Neural Morphological Inflection
Neural sequence-to-sequence models are currently the predominant choice ...
research
∙
10/06/2020
If beam search is the answer, what was the question?
Quite surprisingly, exact maximum a posteriori (MAP) decoding of neural ...
research
∙
07/08/2020
Best-First Beam Search
Decoding for many NLP tasks requires a heuristic algorithm for approxima...
research
∙
05/02/2020
Generalized Entropy Regularization or: There's Nothing Special about Label Smoothing
Prior work has explored directly regularizing the output distributions o...
research
∙
04/22/2020
Testing Machine Translation via Referential Transparency
Machine translation software has seen rapid progress in recent years due...
research
∙
07/19/2019