Measuring and testing the interview mode effect in mixed mode surveys. (English) Zbl 1453.62296

Summary: Many studies are showing an increased tendency to use more than one data collection mode for a particular survey. However, mixed data collection modes may influence responses given by interviewees and require researchers to verify if differences in responses, when present, are ascribable to the type of data collection mode. Often, random assignment is not feasible and requires researchers to solve an additional and not negligible problem, namely to verify if differences in responses are ascribable to the self selection or to the type of data collection mode being used. The aim of the present paper is to measure the mode effect on the answers using a new data driven multivariate approach, that allows to disentangle the interview mode effect on answers from the effect of self selection. We will work through the use of the new multivariate method with AlmaLaurea case concerning the evaluation of two different data collection methods: the CAWI (Computer Assisted Web Interviewing) and the CATI (Computer Assisted Telephone Interviewing). As with any new statistical method, the success of this method depends on its efficacy in relation to that of the existing methods. Therefore, results of the multivariate approach will be compared to the Propensity Score method that AlmaLaurea usually applies to identify the presence of an interview mode effect. Both methods produce similar results.


62D05 Sampling theory, sample surveys
Full Text: DOI


[1] R. E. BARLOW, D. J. BARTHOLOMEW, J. M. BREMNER, H. BRUNK (1972). Statistical Inference Under Order Restrictions. John Wiley & Sons, New York. · Zbl 0246.62038
[2] J. BETHLEHEM (2010). Selection in Web Surveys. International Statistical Review, 78(2), pp. 161-188.
[3] J.M. BORKAN (2004). Mixed Methods Studies: A Fundation for Primary Care Research. Annals of Family Medicine, 2(1), pp. 4-6.
[4] F. CAMILLO, V. CONTI, S. GHISELLI (2011). Integration of different data collection techniques using the propensity score. AlmaLAurea Working Papers n 4.
[5] F. CAMILLO, I. D’ATTOMA (2010). A New Data Mining Approach to Estimate Causal Effects of Policy Interventions. Expert Systems with Applications, 37(2010), pp. 171-181.
[6] F. CAMILLO, I. D’ATTOMA (2012).
[7] A. CAMMELLI, G. ANTONELLI, F. CAMILLO, A. DI FRANCIA, S. GHISELLI, M. SGARZI (2011). Graduates’ employment and employability after the “Bologna Process” reform. Evidence from the Italian experience and methodological issues. AlmaLAurea Working Papers.
[8] W.G. COCHRAN (1968). The Effectiveness of Adjustment by Subclassification in Removing Bias in Observational Studies. Biometrics, 24, pp. 205-213.
[9] I. D’ATTOMA, F. CAMILLO (2011). A Multivariate Strategy to measure and test global imbalance in observational studies. Expert Systems with Applications, 38, pp. 3451-3460.
[10] E. D. DE LEEUW (2005). To mix or not to mix data collection modes in surveys. Journal of Official Statistics, 21, pp. 233-255.
[11] D.A. DILLMAN, R.L. SANGSTER, J.TARNAI,T. H. ROCKWOOD (1996). Understanding Differences in People’s Answers to Telephone and Mail Surveys. New Direction for Evaluation. 70, pp. 45-61.
[12] B. ESCOFIER (1988). Analyse des correspondances multiples conditionelle. In: E. Diday (Ed.), Data Analysis and Informatics. North Holland, Amsterdam: Elsevier Science, pp. 333-342.
[13] J.D. ESTADELLA, T. ALUJA, S. THIÒ-HENESTROSA (2005). Distribution of the inter and intra inertia in conditional MCA. Computational Statistics. 20(3), pp. 449-463. · Zbl 1089.62071
[14] M.J. GREENACRE (1984). Theory and Applications of Correspondence Analysis. London: Academic Press. · Zbl 0555.62005
[15] D.E. HO, K. IMAI, G. KING, E. A. STUART (2007). Matching as nonparametric pre-processing for reducing model dependence in parametric causal inference. Political Analysis, 15, pp. 199-236.
[16] B. JANSSEN (2006). Web data collection in a mixed mode approach: an experiment. Proceedings of Q2006. European Conference on Quality in Survey Statistics.
[17] G. KING, L. ZENG (2006). The Dangers of Extreme Counterfactual. Political Analysis, 14, pp. 131-159.
[18] F. KREUTER, S. PRESSER, R.TOURANGEAU (2008). Social desirability bias in CATI, IVR, and Web Surveys. The effects mode and question sensitivity. Public Opinion Quarterly, 72(5), pp. 847-865.
[19] S. LEE (2006). Propensity Score Adjustment as a Weighting Scheme for Volunteer Panel Web Survey. Journal of Official Statistics. 22(2), pp. 329-349.
[20] M. MORA (2011). Understanding the pros and cons of mixed-mode research. Quirk’s Marketing Research Review, 50 .
[21] L.R. PECK, F. CAMILLO, I. D’ATTOMA (2010). A Promising New Approach to Eliminating Selection Bias. The Canadian Journal of Program Evaluation, 24(2), pp. 31-56.
[22] L.R. PECK, I. D’ATTOMA, F. CAMILLO, C. GUO (2012). A new strategy for Eliminating Selection Bias in Non-Experimental Evaluations: The Case of Welfare Use’s Impact on Charitable Giving. Policy Studies Journal, 40(4), pp. 601-625.
[23] P.R. ROSENBAUM, D.B. RUBIN (1983). The Central Role of Propensity Score in Observational Studies for Causal Effects. Biometrika. 70, pp. 41-55. · Zbl 0522.62091
[24] M. SCHONLAU, R. D. FRICKER, M. N. ELLIOT (2002). Conducting research survey via email and the web. Rand Corporation.
[25] M. SCHONLAU, A. VAN SOEST, A. KAPTEYN , M. P. COUPER (2006). Selection Bias in Web Surveys and the Use of Propensity Score. Rand Corporation.
[26] J. VANNIEUWENHUYZE, G. LOOSVELDT, G. MOLENBERGHS ( 2010). A Method for Evaluating Mode Effects in Mixed-Mode Surveys. Public Opinion Quartely. 74(5), pp. 1027-1045.
[27] P.M. STEINER, D. COOK (2011). Matching and Propensity Scores. In Little, T.D. (Ed.), The Oxford Handbook of Quantitative Methods, Oxford: Oxford University Press.
[28] H. F. WOLTMAN, A. G. TURNER, J. M. BUSHERY (1980). A comparison of three Mixed-Mode Interviewing Procedures in the National Crime Survey. Journal of the American Statistical Association, 75(371), pp. 534-543.
[29] K. B. WRIGHT (2005). Researching Internet-Based Populations: Advantages and Disadvantages of Online Survey Research, Online Questionnaire authoring software packages, and web survey services. Journal Of Computer-Mediated Communication, 10(3).
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. It attempts to reflect the references listed in the original paper as accurately as possible without claiming the completeness or perfect precision of the matching.