User Sentiment as a Success Metric: Persistent Biases Under Full Randomization

06/26/2019
by   Ercan Yildiz, et al.
0

We study user sentiment (reported via optional surveys) as a metric for fully randomized A/B tests. Both user-level covariates and treatment assignment can impact response propensity. We propose a set of consistent estimators for the average and local treatment effects on treated and respondent users. We show that our problem can be mapped onto the intersection of the missing data problem and observational causal inference, and we identify conditions under which consistent estimators exist. We evaluate the performance of estimators via simulation studies and find that more complicated models do not necessarily provide superior performance.

READ FULL TEXT

Please sign up or login with your details

Forgot password? Click here to reset