Filling two needs with one deed: How reminders can improve reliability and validity of survey-based estimates

Authors – Ole Brüggemann, Thomas Hinz, Julia Lang, Susanne Strauss, Nick Zubanov.

Abstract :

Empirical sociology heavily relies on surveys where participation is voluntary and the response rate (RR) can be rather low. Low RR compromises both the reliability (=precision) and validity (=unbiasedness) of survey findings. Research in survey methods shows positive effects of follow-up reminders on RR and recommends using reminders as part of survey design. However, reminders neither bring RR anywhere close to 100% nor reduce differences between survey respondents and non-respondents, so validity concerns remain even when reminders are used and are effective.

Our study contributes to the survey methods literature by showing how reminders can be used to correct for sample selection bias. The common approach to selection bias correction is to augment the measurement model with proxies for the likelihood of participating in the survey (the Heckman selection model is one famous example). There is a well-known identification problem with this approach when the same factors affect both survey participation and outcomes of interest. Our solution to this problem is to use reminders as an instrument that increases the likelihood of participation but is neutral to survey responses, both of which requirements are empirically testable. Its implementation rests on random variation in the number of reminders, which is the main difference from the common practice of sending the same maximum number of reminders to everyone selected for the survey.

We apply our proposed solution in designing the large-scale “Fair: Arbeiten in Deutschland” survey of workers in Germany, with which we study organizational context specificity of pay fairness perceptions. We randomly divide our survey sample into groups that would receive a maximum of 1, 2 or 3 reminders to participate in the survey. We find that reminders do increase RR but do not affect the answers to survey questions or the quality of survey data, thus validating the use of reminders as an instrument in the measurement model with sample selection. Estimating the models with and without sample selection on our survey data produces similar results, implying that, conditional on the participants’ observed characteristics, sample selection is random with respect to their preferences that we sought to elicit.

Our study is relevant to research on inequality in that it provides a methodological enhancement to the existing methods of measuring inequality perceptions from surveys.