Skip to main content
SpringerLink
Log in

Large-Scale Data-Driven Traffic Sensor Health Monitoring

  • Original Paper
  • Published:
Journal of Big Data Analytics in Transportation Aims and scope Submit manuscript

Abstract

Accurate traffic data collection is essential for supporting advanced traffic management system operations. This study investigated a large-scale data-driven sequential traffic sensor health monitoring (TSHM) module that can be used to monitor sensor health conditions over large traffic networks. Our proposed module consists of three sequential steps for detecting different types of abnormal sensor issues. The first step detects sensors with abnormally high missing data rates, while the second step uses clustering anomaly detection to detect sensors reporting abnormal records. The final step introduces a novel Bayesian changepoint modeling technique to detect sensors reporting abnormal traffic data fluctuations by assuming a constant vehicle length distribution based on average effective vehicle length (AEVL). Our proposed method is then compared with two benchmark algorithms to show its efficacy. Results obtained by applying our method to the statewide traffic sensor data of Iowa show it can successfully detect different classes of sensor issues. This demonstrates that sequential TSHM modules can help transportation agencies determine traffic sensors’ exact problems, thereby enabling them to take the required corrective steps.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15

Similar content being viewed by others

References

  • Al-Deek HM, Venkata C, Chandra SR (2004) New algorithms for filtering and imputation of real-time and archived dual-loop detector data in i–4 data warehouse. Transportation Research Record 1867(1), 116–126

    Google Scholar 

  • Apache Pig (2018) https://pig.apache.org/. Accessed 20 July 2018

  • Berkhin P (2006) A survey of clustering data mining techniques. In Grouping multidimensional data (pp 25–71). Springer, Berlin

  • Bouwmans T, Zahzah EH (2014) Robust pca via principal component pursuit: A review for a comparative evaluation in video surveillance. Computer Vision and Image Understanding 122:22–34

    Article  Google Scholar 

  • Candès EJ, Li X, Ma Y, Wright J (2011) Robust principal component analysis? Journal of the ACM (JACM) 58(3):11

    Article  MathSciNet  Google Scholar 

  • Chakraborty P, Adu-Gyamfi YO, Poddar S, Ahsani V, Sharma A, Sarkar S (2018) Traffic congestion detection from camera images using deep convolution neural networks. Transportation Research Record: Journal of the Transportation Research Board 2672(45), 222–231. doi: https://doi.org/10.1177/0361198118777631

    Article  Google Scholar 

  • Chakraborty P, Sharma A, Hegde C (2018) Freeway traffic incident detection from cameras: a semi-supervised learning approach. In 2018 21st International Conference on Intelligent Transportation Systems (ITSC) 2018 21st international conference on intelligent transportation systems (itsc), pp 1840–1845

  • Chandola V, Banerjee A, Kumar V (2009) Anomaly detection: A survey. ACM Computing Surveys (CSUR) 41(3):15

    Article  Google Scholar 

  • Chen C, Kwon J, Rice J, Skabardonis A, Varaiya P (2003) Detecting errors and imputing missing data for single-loop surveillance systems. Transportation Research Record 1855(1), 160–167

    Google Scholar 

  • Erman J, Arlitt M, Mahanti A (2006) Traffic classification using clustering algorithms. In Proceedings of the 2006 SIGCOMM workshop on Mining network data, pp 281–286

  • Ester M, Kriegel H-P, Sander J, Xu X et al (1996) A density-based algorithm for discovering clusters in large spatial databases with noise. In: Kdd, Vol 96, pp 226–231

  • Fearnhead P (2006) Exact and efficient bayesian inference for multiple changepoint problems. Statistics and Computing 16(2), 203–213

    Article  MathSciNet  Google Scholar 

  • Fearnhead P, Liu Z (2007) On-line inference for multiple changepoint problems. Journal of the Royal Statistical Society: Series B (Statistical Methodology) 69(4):589–605

    Article  MathSciNet  Google Scholar 

  • Jacobson LN, Nihan NL, Bender JD (1990) Detecting erroneous loop detector data in a freeway traffic management system (No. 1287)

  • Klein LA, Mills MK, Gibson DR et al (2006) Traffic detector handbook: Volume i (Tech. Rep.) Turner-Fairbank Highway Research Center

  • Kodinariya TM, Makwana PR (2013) Review on determining number of cluster in k-means clustering. International Journal of Advance Research in Computer Science and Management Studies 1(6), 90–95

    Google Scholar 

  • Lee H, Coifman B (2011) Quantifying loop detector sensitivity and correcting detection problems on freeways. Journal of Transportation Engineering 138(7), 871–881

    Article  Google Scholar 

  • Lu Y, Yang X, Chang G-L (2014) Algorithm for detector-error screening on basis of temporal and spatial information. Transportation Research Record: Journal of the Transportation Research Board 2443:40–48

    Article  Google Scholar 

  • Ma D, Luo X, Li W, Jin S, Guo W, Wang D (2017) Traffic demand estimation for lane groups at signal-controlled intersections using travel times from video-imaging detectors. IET Intelligent Transport Systems 11(4), 222–229

    Article  Google Scholar 

  • MacQueen J et al (1967) Some methods for classification and analysis of multivariate observations. In: Proceedings of the fifth berkeley symposium on mathematical statistics and probability, vol 1, pp 281–297

  • Mori U, Mendiburu A, Álvarez M, Lozano JA (2015) A review of travel time estimation and forecasting for advanced traveller information systems. Transportmetrica A: Transport Science 11(2), 119–157

    Google Scholar 

  • Nisa KK, Andrianto HA, Mardhiyyah R (2014) Hotspot clustering using dbscan algorithm and shiny web framework. In: 2014 international conference on advanced computer science and information system, pp 129–132

  • Payne H, Thompson S (1997) Malfunction detection and data repair for induction-loop sensors using i–880 data base. Transportation Research Record: Journal of the Transportation Research Board 1570:191–201

    Article  Google Scholar 

  • Schubert E, Sander J, Ester M, Kriegel HP, Xu X (2017) Dbscan revisited, revisited: why and how you should (still) use dbscan. ACM Transactions on Database Systems (TODS) 42(3):19

    Article  MathSciNet  Google Scholar 

  • Shi Q, Abdel-Aty M (2015) Big data applications in real-time traffic operation and safety monitoring and improvement on urban expressways. Transportation Research Part C: Emerging Technologies 58:380–394

    Article  Google Scholar 

  • Sun Z, Jin W-L, Ng M (2016) Network sensor health problem. Transportation Research Part C: Emerging Technologies 68:300–310

    Article  Google Scholar 

  • Turner S, Albert L, Gajewski B, Eisele W (2000) Archived intelligent transportation system data quality: Preliminary analyses of san antonio transguide data. Transportation Research Record: Journal of the Transportation Research Board 1719:77–84

    Article  Google Scholar 

  • Turochy RE, Smith BL (2000) New procedure for detector data screening in traffic management systems. Transportation Research Record 1727(1), 127–131

    Article  Google Scholar 

  • Vanajakshi L, Rilett L (2004) Loop detector data diagnostics based on conservation-of-vehicles principle. Transportation Research Record 1870(1), 162–169

    Google Scholar 

  • Wells TJ, Smaglik EJ, Bullock DM (2008) Implementation of station health monitoring procedures for its sensors, volume 1

  • Wu L, Liu C, Huang T, Sharma A, Sarkar S (2017) Traffic sensor health monitoring using spatiotemporal graphical modeling. In: Proceedings of the 2nd acm sigkdd workshop on machine learning for prognostics and health management, pp 13–17

  • Yao B, Chen C, Cao Q, Jin L, Zhang M, Zhu H, Yu B (2017) Short-term traffic speed prediction for an urban corridor. Computer-Aided Civil and Infrastructure Engineering 32(2), 154–169

    Article  Google Scholar 

Download references

Acknowledgements

Our research results are based upon work jointly supported by the National Science Foundation Partnerships for Innovation: Building Innovation Capacity (PFI: BIC) program under Grant No. 1632116, National Science Foundation under Grants Nos. CNS-1464279, CCF-1566281, CCF-1750920, Award ATD-1830254 supported by NSF and the National Geospatial-Intelligence Agency, and Iowa DOT Office of Traffic Operations Support Grant. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.

Author information

Authors and Affiliations

  1. Civil, Construction, and Environmental Engineering Department, Iowa State University, Ames, IA, 50011, USA

    Tongge Huang & Anuj Sharma

  2. Department of Civil Engineering, Indian Institute of Technology Kanpur, Kanpur, U.P, 208016, India

    Pranamesh Chakraborty

  3. Electrical and Computer Engineering Department, New York University, New York, NY, 10003, USA

    Chinmay Hegde

Authors
  1. Tongge Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Pranamesh Chakraborty
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Anuj Sharma
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Chinmay Hegde
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Tongge Huang.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file1 (EPS 37 KB)

Supplementary file2 (EPS 37 KB)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Huang, T., Chakraborty, P., Sharma, A. et al. Large-Scale Data-Driven Traffic Sensor Health Monitoring. J. Big Data Anal. Transp. 3, 229–245 (2021). https://doi.org/10.1007/s42421-021-00049-w

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s42421-021-00049-w

Keywords

Search

Navigation