Abstract
The Twitter platform is appealing to researchers due to the ease of obtaining data and the ability to analyze and produce results rapidly. However, sampling Twitter data for research purposes needs to be regulated to produce unbiased results. In this paper, factors that lead to sampling bias are addressed, case studies that have been encountered are presented, and an approach is proposed to reduce sampling bias and flaws in datasets collected from Twitter. Then, experiments are conducted on two case studies, and a larger dataset is achieved by following the proposed guideline. The results indicate that using multiple Twitter application programming interfaces (APIs) for data collection is the best way to obtain a randomly sampled dataset.
L. Alkulaib and A. Alhamadani — Equally contributed to this work.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Campan, A., Atnafu, T., Truta, T.M., Nolan, J.: Is data collection through Twitter streaming API useful for academic research? In: 2018 IEEE International Conference on Big Data (Big Data) (2018). https://doi.org/10.1109/bigdata.2018.8621898
Cortes, C., Mohri, M., Riley, M., Rostamizadeh, A.: Sample selection bias correction theory. In: Freund, Y., Györfi, L., Turán, G., Zeugmann, T. (eds.) ALT 2008. LNCS (LNAI), vol. 5254, pp. 38–53. Springer, Heidelberg (2008). https://doi.org/10.1007/978-3-540-87987-9_8
Morstatter, F., Liu, H.: Discovering, assessing, and mitigating data bias in social media. Online Soc. Netw. Media 1, 1–13 (2017). https://doi.org/10.1016/j.osnem.2017.01.001
Morstatter, F., Pfeffer, J., Liu, H.: When is it biased? In: Proceedings of the 23rd International Conference on World Wide Web - WWW 14 Companion (2014). https://doi.org/10.1145/2567948.2576952
Morstatter, F., Pfeffer, J., Liu, H., Carley, K.M.: Is the sample good enough? Comparing data from Twitter’s streaming API with Twitter’s firehose (2013)
Summers, E., Summers, E.: On forgetting (November 2014). https://medium.com/on-archivy/on-forgetting-e01a2b95272
Wang, Y., Callan, J., Zheng, B.: Should we use the sample? Analyzing datasets sampled from Twitter’s stream API. ACM Trans. Web 9(3), 1–23 (2015). https://doi.org/10.1145/2746366
Woodfield, K., Ahmed, W.: Using Twitter as a data source: an overview of ethical, legal, and methodological challenges. Adv. Res. Ethics Integr. 2, 79–107 (2017). Chapter 4. Emerald
Zhang, H., Hill, S., Rothschild, D.: Addressing selection bias in event studies with general-purpose social media panels. J. Data Inf. Qual. 10(1), 1–24 (2018). https://doi.org/10.1145/3185048
Zubiaga, A.: A longitudinal assessment of the persistence of Twitter datasets (May 2018). https://onlinelibrary.wiley.com/doi/10.1002/asi.24026
Author information
Authors and Affiliations
Corresponding authors
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Alkulaib, L., Alhamadani, A., Ji, T., Lu, CT. (2020). Collect Ethically: Reduce Bias in Twitter Datasets. In: Lossio-Ventura, J.A., Condori-Fernandez, N., Valverde-Rebaza, J.C. (eds) Information Management and Big Data. SIMBig 2019. Communications in Computer and Information Science, vol 1070. Springer, Cham. https://doi.org/10.1007/978-3-030-46140-9_11
Download citation
DOI: https://doi.org/10.1007/978-3-030-46140-9_11
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-46139-3
Online ISBN: 978-3-030-46140-9
eBook Packages: Computer ScienceComputer Science (R0)