An exploration of the strengths and weaknesses of using text messaging as a tool for self-report data collection in psychological research 

How does SMS compare with other tools for data collection?

So we've seen that SMS has the capacity to be a useful tool for self-report psychological research. The previous section began to explore that capacity, and we could see that SMS can indeed be used to collect self-report data, providing acceptable response behaviour, and with some caveats, reasonable psychometric validity. Yet, capacity alone is insufficient if a researcher is considering whether or not to use SMS for their study. The current research landscape has a multitude of data collection tools available (Dillman, Smyth, & Christian, 2009). There may be other equally viable, or better, alternatives to SMS. This section examines how SMS performs in comparison to other tools used to collect self-report data.

Like SMS, software applications (apps) running on mobile telephones are also gaining traction as a data collection tool (e.g. Holloway et al., 2014; Rosser & Eccleston, 2011). The first paper, Is SMS APPropriate? Comparative properties of SMS and apps for repeated measures data collection, investigates how the two modes compare when all other aspects of the study design are held constant. The second paper, SMS = Send My Survey: Short Message Service for Longitudinal Research, takes a similar approach, but compares SMS against paper and email surveys, and digital devices.

The final paper, Applying cross-language principles to cross-mode measurement invariances, examines how SMS compares to online surveys specifically in terms of psychometric validity. This paper draws from the robust literature investigating what degree of difference is acceptable between different language versions of the same instrument, and applies it to the degree of difference found between SMS and online administrations of that instrument.

Is SMS APPropriate? Comparative properties of SMS and apps for repeated measures data collection

The ubiquity of mobile telephones worldwide offers a unique opportunity for bidirectional communication between researchers and participants. There are two ways mobile phones could be used to collect self-report data: via Short Message Service (SMS), or app (mobile telephone software applications). This study examined the comparative data quality offered by SMS and app, when mobile phone type, self-report instrument, and sampling schedule are controlled. One hundred and ten undergraduate students used their own iPhones to complete the same repeated measures instrument on twenty occasions, responding either by SMS or app. There were no differences between SMS and app respondents in terms of response rates, or response delay. However, data from those responding via SMS was significantly less complete than from app respondents. App respondents rated their respondent experience as more convenient than SMS respondents. Though findings are only generalizable to an undergraduate sample, this suggests that researchers should consider using apps rather than SMS for repeated measures self-report data collection. [click here to read the paper in full]

SMS = Send My Survey: Short Message Service for Longitudinal Research

Repeated measures research allows unique insights into psychological change, and intra-individual variability. It is also very difficult to do. One possible solution is Short Message Service (SMS), a ubiquitous bidirectional mobile telephone-based communication system that is slowly gaining traction as a practical and cost-effective mode for self-report data collection. This paper is the first to examine how SMS performs compared to other research modes in terms of the common threats to repeated measures data collection - attrition, incomplete responses, and delays in responding. 261 participants were randomly assigned to complete a ten item negative mood questionnaire twenty four times via SMS, email, post, or digital device. Some received no reminder prompts, others prompts via SMS, email, or post. SMS produced comparable data quality to postal and email questionnaires, and superior data quality to digital device. SMS performed best when sampling daily, and in combination with SMS reminder prompts. When SMS prompts were used in combination with other modes, response delays were reduced.

[I am still trying to get this published, so no full text online yet.]


Applying cross-language principles to cross-mode measurement invariances

As technology advances, researchers encounter an ever-expanding choice of different data collection modes. Administration of instruments in modes they were not developed in (for example, administering an originally paper survey online) is increasingly common, and multi-method research is on the rise. Yet there is a lack of research comparing the impact of mode on administration, perhaps due to a lack of a verified procedure and analysis for comparison. This study established how design and analytical techniques from the cross-language literature can be applied to investigate cross-mode measurement invariance. Repeated administration of the same instrument using different data collection modes allowed for within-subjects comparison of the impact of mode on responses while controlling for individual differences. Subsequent comparison examining Cronbach’s alpha, total scores, total score distributions, and multi-group factor analysis offered mutually complimentary insights into cross-mode equivalence. To demonstrate the application of this procedure and analysis, fifty five Chinese-English bilingual participants completed four variants (English/Online, English/SMS, Chinese/Online, and Chinese/SMS) of the same 20-item questionnaire in a randomised order, two days apart. Results demonstrated the feasibility and benefits of applying techniques from the cross-language translation literature to investigation of cross-mode equivalence. 

[I am still trying to get this published, so no full text online yet.]




Dillman, D. A., Smyth, J. D., & Christian, L. M. (2009). Internet, Mail, and Mixed-Mode Surveys (Third). New Jersey: John Wiley & Sons.

Holloway, I. W., Rice, E., Gibbs, J., Winetrobe, H., Dunlap, S., & Rhoades, H. (2014). Acceptability of smartphone application-based HIV prevention among young men who have sex with men. AIDS and Behavior, 18(2), 285–96.

Rosser, B. a, & Eccleston, C. (2011). Smartphone applications for pain management. Journal of Telemedicine and Telecare, 17(6), 308–12.