Skip to main content
SpringerLink
Log in

Identifying Duplication in Statistical Indicators: Methodic Approach

  • Conference paper
  • First Online:
Digital Transformation and Global Society (DTGS 2020)

Part of the book series: Communications in Computer and Information Science ((CCIS,volume 1242))

Included in the following conference series:

  • 974 Accesses

Abstract

Data-based and data-driven decisions are at the core of digital government transformation. However, the more the data is to be used to guide policy development, the higher are the requirements to the data accuracy and readiness. Larger reliance on data to inform policy decisions should not lead to increased reporting requirements and hence excessive administrative burden on businesses. Therefore, identifying and reducing duplication in statistical data should be performed at the early stages of the government digital transformation. Given the constantly increasing number of strategic documents and continuous amendments to the list of statistic indicators measured, there is a need for an instrument allowing for timely identification and elimination of possible duplication in statistical and other indicators.

In this paper we propose a methodic approach to identifying and evaluating possible duplication in statistical and other administrative indicators which is based on a partially automatable algorithm complemented by expert evaluation. The results of piloting this approach on a set of about 6,000 statistical indicators suggest that it could become a useful tool for data management that would allow to improve the quality of aggregated data, on the one hand, and reduce administrative reporting burden on businesses – on the other. The proposed approach could also be applied in a broader context, i.e., for the analysis of strategic planning documents, and may be of interest to practitioners from other countries where the quality of statistical data and duplication of administrative information is considered a barrier for further government digitalization.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    See: https://blogs.worldbank.org/developmenttalk/world-development-report-2021-data-development.

  2. 2.

    Approved on September 6, 2019. See: https://www.gks.ru/storage/mediabank/Strategy.pdf (in Russian).

  3. 3.

    See: http://gasu.gov.ru/stratplanning.

References

  1. Agbehadji, I.E., Yang, H., Fong, S., Millham, R.: The comparative analysis of smith-waterman algorithm with Jaro-Winkler algorithm for the detection of duplicate health related records. In: Paper Presented at the 2018 International Conference on Advances in Big Data, Computing and Data Communication Systems, icABCD 2018 (2018). https://doi.org/10.1109/icabcd.2018.8465458

  2. Alenazi, S.R., Ahmad, K., Olowolayemo, A.: A review of similarity measurement for record duplication detection. In: Proceedings of the 2017 6th International Conference on Electrical Engineering and Informatics: Sustainable Society Through Digital Innovation, ICEEI, 6 Jan 2017 (2018). https://doi.org/10.1109/iceei.2017.8312386

  3. Blewett, L.A., Call, K.T., Turner, J., Hest, R.: Data resources for conducting health services and policy research. Ann. Rev. Public Health 39, 437–452 (2018). https://doi.org/10.1146/annurev-publhealth-040617-013544

    Article  Google Scholar 

  4. CEFC Group: Research on reporting forms inventory, including statistical data, developing new approaches to reporting (2018). http://sk.ru/foundation/legal/p/09.aspx. Accessed 15 Feb 2020. (in Russian)

  5. Dmitrieva, N.E., Plaksin, S.M., Sinyatullina, L.H.: Ocenka izderzhek organizacij na sbor statisticheskoj otchetnosti, ili skol’ko stoit pokazatel’ [The Main Approaches to the Assessment of Organizations’ Costs in the Statistic Collection and Provision]. Public Administration Issues 2, 71–93 (2018). (in Russian)

    Google Scholar 

  6. Dobrolyubova, E., Alexandrov, O., Yefremov, A.: Is Russia ready for digital transformation? In: Alexandrov, D., Boukhanovsky, A., Chugunov, A., Kabanov, Y., Koltsova, O. (eds.) DTGS. CCIS, 745, 431–444. Springer, Cham (2017)

    Google Scholar 

  7. Fitzgerald, G., FitzGibbon, M.: A comparative analysis of traditional and digital data collection methods in social research in LDCs - case studies exploring implications for participation. In: Empowerment, and (mis)Understandings (2014). http://www.validnutrition.org/wp-content/uploads/2015/03/A-Comparative-Analysis-of-Traditional-and-Digital-Data-Collection-Me.thods.pdf. Accessed 05 Feb 2020

  8. Gabriel, R.A., Kuo, T.-T., McAuley, J., Hsu, C.-N.: Identifying and characterizing highly similar notes in big clinical note datasets. J. Biomed. Inform. 82, 63–69 (2018). https://doi.org/10.1016/j.jbi.2018.04.009

    Article  Google Scholar 

  9. Gokhberg, L.M.: Perspektivnaja model’ gosudarstvennoj statistiki v cifrovuju jepohu [Prospective model of public statistics in digital era]. HSE, Moscow (2018). (in Russian)

    Google Scholar 

  10. Gubanov, D., Korgin, N., Novikov, D., Raikov, A.: E-Expertise: modern collective intelligence. Springer. Series: Studies in Computational Intelligence, vol. 558, XVIII (2014). https://doi.org/10.1007/978-3-319-06770-4

  11. Hosseini, M., Faiola, A., Jones, J., Vreeman, D.J., Wu, H., Dixon, B.E.: Impact of document consolidation on healthcare providers’ perceived workload and information reconciliation tasks: a mixed methods study. J. Am. Med. Inform. Assoc. 26(2), 134–142 (2019). https://doi.org/10.1093/jamia/ocy158

    Article  Google Scholar 

  12. Janssen, M., van den Hoven, J.: Big and Open Linked Data (BOLD) in government: a challenge to transparency and privacy? Government Inf. Q. 32(4), 363–368 (2015)

    Article  Google Scholar 

  13. Jaro, M.A.: Advances in record linkage methodology as applied to the 1985 census of Tampa Florida. J. Am. Stat. Assoc. 84(406), 414–420 (1989). http://doi.org/10.1080/01621459.1989.10478785

  14. Kabukye, J.K., de Keizer, N., Cornet, R.: Elicitation and prioritization of requirements for electronic health records for oncology in low resource settings: a concept mapping study. Int. J. Med. Inform. 135, Article No. 104055 (2020). https://doi.org/10.1016/j.ijmedinf.2019.104055

  15. Keil, J.M.: Efficient bounded Jaro-Winkler similarity based search. In: Grust, T., Naumann, F., Bohm, A., Lehner, W., Harder, T., Rahm, E., Heuer, A., Klettke, M., Meyer, H. (eds.) Lecture Notes in Informatics (LNI), Proceedings - Series of the Gesellschaft fur Informatik (GI), vol. 289, pp. 205–214 (2019)

    Google Scholar 

  16. Koneru, K., Pulla, V.S.V., Varol, C.: Performance evaluation of phonetic matching algorithms on english words and street names comparison and correlation. In: DATA 2016 - Proceedings of the 5th International Conference on Data Management Technologies and Applications, 57–64 (2016)

    Google Scholar 

  17. Kozlova, M.A.: Statistika kul’’tury v formah statisticheskoj otchetnosti: tekushhee sostojanie i traektorii razvitija [Statistics of culture in statistical forms: current status and development trends]. Intellekt. Innovacii. Investicii 9, 16–19 (2017). (in Russian)

    Google Scholar 

  18. Lazareva, M.L., Tyurina, I.V.: Finansovaja statisticheskaja otchetnost’’ medicinskih organizacij: sushhestvujushhie nedostatki i napravlenija optimizacii [Financial statistics of healthcare organizations: significant drawbacks and directions of optimization]. Farmakojekonomika. Sovremennaja farmakojekonomika i farmakojepidemiologija 11(4), 61–66 (2018). (in Russian)

    Google Scholar 

  19. Levenshtein, V.I.: Binary codes capable of correcting deletions, insertions, and reversals. Soviet Phys. Doklady. 10(8), 707–710 (1966)

    MathSciNet  Google Scholar 

  20. Máchová, R., Lnénicka, M.: Evaluating the quality of open data portals on the national level. J. Theoretical Appl. Electron. Commerce Res. 12(1), 21–41 (2017). https://doi.org/10.4067/S0718-18762017000100003

    Article  Google Scholar 

  21. Mandal, A.K., Hossain, M.D., Nadim, M.: Developing an efficient search suggestion generator, ignoring spelling error for high speed data retrieval using double Metaphone algorithm. In: Proceedings of 2010 13th International Conference on Computer and Information Technology, ICCIT 2010, 317–320 (2010)

    Google Scholar 

  22. Mason-Blakley, F.: Information system hazard analysis and mitigation. In: Proceedings - 2015 IEEE International Conference on Healthcare Informatics, ICHI 2015, p. 472 (2015)

    Google Scholar 

  23. Mironenko, A.G., Kravets, A.G.: Automated methods of patent array analysis. In: IISA 2016 - 7th International Conference on Information, Intelligence, Systems and Applications (2016)

    Google Scholar 

  24. Morozov, A.N.: Al’ternativnye istochniki statisticheskoj informacii kak osnova prinjatija politicheskih reshenij [Alternative Sources of Statistical Information as the Basis for Political Decision Making]. Public Administration Issues 2, 50–70 (2018). (in Russian)

    Google Scholar 

  25. Naiman, S.M.: Upravlenie othodami i problemy statisticheskogo ucheta [Waste management and statistical accounting issues]. Vestnik Permskogo nacional’nogo issledovatel’skogo politehnicheskogo universiteta. Prikladnaja jekologija. Urbanistika. 23(3), 5–19 (2016). (in Russian)

    Google Scholar 

  26. OECD: Key Issues for Digital Transformation in the G20 (2017). http://www.oecd.org/G20/key-issues-for-digital-transformation-in-the-G20.pdf. Accessed on 05 Feb 2020

  27. OECD: Open, Useful and Re-usable data (OURdata) Index: 2019, OECD Public Governance Policy Papers, No. 01, OECD Publishing, Paris (2020). https://doi.org/10.1787/45f6de2d-en

  28. Philips, L.: Hanging on the Metaphone. Computer Language, 7(12) (1990)

    Google Scholar 

  29. Santhosh Kumar, C.N., Pavan Kumar, V., Reddy, K.S.: Similarity matching of pairs of text using CACT algorithm. Int. J. Eng. Adv. Technol. 8(6), 2296–2298 (2019)

    Article  Google Scholar 

  30. Soboleva, O.V.: Cistematizacija pokazatelej vedomstvennoj statisticheskoj otchetnosti FSIN Rossii [Systematization of departmental reporting indicators in Federal Penitentiary Service]. In: VESTNIK FKU NIIIT FSIN ROSSII, pp. 135–142 (2018). (in Russian)

    Google Scholar 

  31. Statistisches Bundesamt. Die Bestandsmessung der Bürokratiekosten der deutschen Wirtschaft nach dem Standardkosten-Modell. Statistik und Wissenschaft, 14 (2014). https://www.destatis.de/GPStatistik/receive/DEMonografie_monografie_00000236. Accessed 05 Feb 2020. (in German)

  32. UK Office of National Statistics. Data Collection Transformation Programme (2017). https://www.ons.gov.uk/aboutus/whatwedo/programmesandprojects/datacollectiontransformationprogrammedctp. Accessed 05 Feb 2020

  33. United Nations. A World That Counts: Mobilising a Data Revolution for Sustainable Development (2014). https://www.undatarevolution.org/report/. Accessed 05 Feb 2020

  34. UzZaman, N., Khan, M.: A Double Metaphone encoding for Bangla and its application in spelling checker. In: Proceedings of 2005 IEEE International Conference on Natural Language Processing and Knowledge Engineering, IEEE NLP-KE 2005, p. 705 (2005)

    Google Scholar 

  35. Winkler, W.E.: String comparator metrics and enhanced decision rules in the fellegi-sunter model of record linkage. In: Proceedings of the Section on Survey Research Methods. American Statistical Association, pp. 354–359 (1990)

    Google Scholar 

  36. World Bank. Digital Dividends. World Development Report. http://documents.worldbank.org/curated/en/896971468194972881/pdf/102725-PUBReplacement-PUBLIC.pdf. Accessed on 05 Feb 2020

  37. World Bank. Russia Digital Economy Report. Competition in the Digital Age. Implications for the Russian Federation (2018) https://openknowledge.worldbank.org/bitstream/handle/10986/30584/AUS0000158-WP-REVISED-P160805-PUBLIC-Disclosed-10-15-2018.pdf?sequence=1&isAllowed=y. Accessed on 05 Feb 2020

Download references

Author information

Authors and Affiliations

  1. Russian Academy of National Economy and Public Administration (RANEPA), Vernadskogo pr. 84, 119571, Moscow, Russia

    Elena Dobrolyubova

  2. CEFC Group, Sadovaya-Kudrinskaya 11/1 Office 412, 121242, Moscow, Russia

    Oleg Alexandrov

Authors
  1. Elena Dobrolyubova
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Oleg Alexandrov
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Elena Dobrolyubova .

Editor information

Editors and Affiliations

  1. National Research University Higher School of Economics, St. Petersburg, Russia

    Daniel A. Alexandrov

  2. ITMO University, St. Petersburg, Russia

    Alexander V. Boukhanovsky

  3. ITMO University, St. Petersburg, Russia

    Andrei V. Chugunov

  4. National Research University Higher School of Economics, St. Petersburg, Russia

    Yury Kabanov

  5. National Research University Higher School of Economics, St. Petersburg, Russia

    Olessia Koltsova

  6. National Research University Higher School of Economics, St. Petersburg, Russia

    Ilya Musabirov

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Dobrolyubova, E., Alexandrov, O. (2020). Identifying Duplication in Statistical Indicators: Methodic Approach. In: Alexandrov, D.A., Boukhanovsky, A.V., Chugunov, A.V., Kabanov, Y., Koltsova, O., Musabirov, I. (eds) Digital Transformation and Global Society. DTGS 2020. Communications in Computer and Information Science, vol 1242. Springer, Cham. https://doi.org/10.1007/978-3-030-65218-0_15

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-65218-0_15

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-65217-3

  • Online ISBN: 978-3-030-65218-0

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation