Skip to main content
SpringerLink
Log in

Toward storytelling from personal informative lifelogging

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

The authors began collecting personal informative lifelogging data in 2011. The data collected in this article is a discontinuous data set, each of which includes one or more images, a GPS message, a description of a location, and a description of the text, we call this informative lifelogging. However, for the purposes automatically building a story from a huge collection of unstructured egocentric data presents major challenges. This paper first introduces the structure and characteristics of the collected data and uses the DB-scan algorithm to classify the data. Then a model for generating a story is proposed, and a model of story generation based on a story template is proposed in the model. The author implemented a complete software system through code, described a story generation model, and gave the key algorithm to generate stories. Through this system, 418 stories were generated automatically, of which 62% of the stories were particularly accurate. The experimental results verify that it is feasible to automatically generate stories based on personal Informative lifelogging data.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17

Similar content being viewed by others

References

  1. Andrew A, Eustice K, Hickl AJ. Using location Lifelogs to make meaning of food and physical activity behaviors. Presented at PCTHPIC, (2013).

    Book  Google Scholar 

  2. Asnaoui KE, Hamid A, Brahim A, Mohammed O (2017) A survey of activity recognition in egocentric lifelogging datasets. International Conference on Wireless Technologies, IEEE

    Google Scholar 

  3. Belhadi A, Djenouri Y, Lin CWJ, Cano A (2020) Trajectory outlier detection: algorithms, taxonomies, evaluation and open challenges. ACM Transactions on Management Information Systems.

    Google Scholar 

  4. Bolaos M, Dimiccoli M, Radeva P (2017) Toward storytelling from visual lifelogging: an overview. IEEE Transactions on Human-Machine Systems 47(1):77–90

    Google Scholar 

  5. Chen DL, Dolan WB. “Collecting highly parallel data for paraphrase evaluation,” . in Proc 49th Annu Meeting Assoc Comput Linguistics, 2011, pp. 190–200.

  6. Gao L, Li X, Song J (2019) Hierarchical LSTMs with adaptive attention for visual captioning[J]. IEEE Trans Pattern Anal Mach Intell:1–1

  7. Gurrin C, Smeaton AF, Doherty AR (2014) Sources of lifelog data, in lifelogging: personal bigdata.1st ed.,vol. 8, no. 1. Dublin, Ireland: DCU, pp. 35–47.

  8. Jo H, Ryu J-H. Placegram: A diagrammatic map for personal geotagged data browsing. IEEE Educational Activities Department, (2010).

  9. Kim PH (2011) Web-based research collaboration service: crowd lifelog research case study. International Conference on Next Generation Web Services Practices, IEEE

    Google Scholar 

  10. Kybartas B, Bidarra R (2017) A survey on story generation techniques for authoring computational narratives. IEEE Transactions on Computational Intelligence & Ai in Games

    Book  Google Scholar 

  11. Li Z, Ding B, Kays R, Nye P (2010) Mining periodic behaviors for moving objects. In Proceedings of the16th ACM SIGKDD International Conference onKnowledge Discovery and Data Mining. ACM, 1099–1108.

  12. Liu X, Huet B (2016) Event-based cross media question answering. Multimedia Tools & Applications 75(3):1495–1508

    Article  Google Scholar 

  13. Mandler JM, Johnson NS (1977) Remembrance of things parsed: story structure and recall. Cogn Psychol 9(1):111–151

    Article  Google Scholar 

  14. Mann S (1997) Wearable computing: a first step toward personal imaging. Computer 30(2):25–32

    Article  Google Scholar 

  15. Martin WM, Heylighen A, Cavallin H (2005) The right story at the right time. AI & Soc 19(1):34–47

    Article  Google Scholar 

  16. Mazimpaka JD (2016) Trajectory data mining: a review of methods and applications. Journal of Spatial Information Science 13(13)

  17. Rohrbach A, Torabi A, Rohrbach M, Tandon N, Pal C, Larochelle H, Courville A, Schiele B (2017) Movie description. Int J Comput Vis 123(1):94–120

    Article  Google Scholar 

  18. Song J, Gao L, Nie F, Shen HT, Yan Y, Sebe N (2016) Optimized graph learning using partial tags and multiple features for image and video annotation. IEEE Trans Image Process 25(11):4999–5011

    Article  MathSciNet  Google Scholar 

  19. Venugopalan S, Xu H, Donahue J, Rohrbach M, Mooney RJ, Saenko K (2015) Translating videos to natural language using deep recurrent neural networks. NAACL HLT:1494–1504

  20. Venugopalan S, Rohrbach M, Donahue J, Mooney R, Darrell T, Saenko K (2015) Sequence to sequence-video to text. in Proc IEEE Int Conf Comput Vis:4534–4542

  21. Vinyals O, Toshev A, Bengio S, Erhan D (2015) Show and tell: a neural image caption generator. Proc. IEEE Conf. Comput. Vis. Pattern Recognit, In, pp 3156–3164

    Google Scholar 

  22. Wang Y, Zheng Y, Xue Y. (2014) Travel time estimation of a path using sparse trajectories. In Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, 25–34.

  23. Wang X, Gao L, Wang P, Sun X, Liu X (2018) Two-stream 3-d convnet fusion for action recognition in videos with arbitrary size and length. IEEE Transactions on Multimedia 20(3):634–644

    Article  Google Scholar 

  24. Wang J, Yuan Y, Ni T, Ma Y, Shen W (2020) Anomalous trajectory detection and classification based on difference and intersection set distance. IEEE Trans Veh Technol 69(99):2487–2500

    Article  Google Scholar 

  25. Xiao S, Li T, Wang J (2019) Optimization methods of video images processing for mobile object recognition[J]. Multimed Tools Appl

  26. Xu K, Ba J, Kiros R, Cho K, Courville AC, Salakhutdinov R, Zemel RS, Bengio Y (2015) Show, attend and tell: Neural image caption generation with visual attention. in Proc 32nd Int Conf Mach Learn:2048–2057

  27. Xu J, Mei T, Yao T, Rui Y (2016) MSR-VTT: A large video description dataset for bridging video and language. in Proc IEEE Conf Comput Vis Pattern Recognit:5288–5296

  28. Yuan J, Zheng Y, Xie X, Sun G (2013) T-Drive: enhancing driving directions with taxi drivers' intelligence. IEEE Transaction on Knowledge and Data Engineering 25(1):220–232

    Article  Google Scholar 

  29. Yuan NJ, Zheng Y, Zhang L, Xie X (2013) T-finder: a recommender system for finding passengers and vacant taxis. IEEE Transaction on Knowledge and Data Engineering 25(10):2390–2403

    Article  Google Scholar 

  30. Zhou Y, Lau BPL, Koh Z, Yuen C, Ng BKK (2020) Understanding crowd behaviors in a social event by passive wifi sensing and data mining. IEEE Internet of Things Journal

Download references

Acknowledgments

The authors would like to thank the anonymous reviewers of this paper for their insightful comments.

Author information

Authors and Affiliations

  1. Jianzhu University, Shenyang, Liaoning, China

    Guoqi Liu & Yuhou Wu

  2. Northeastern University, Shenyang, Liaoning, China

    Mujeeb Ur Rehman

Authors
  1. Guoqi Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mujeeb Ur Rehman
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yuhou Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yuhou Wu.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Liu, G., Rehman, M.U. & Wu, Y. Toward storytelling from personal informative lifelogging. Multimed Tools Appl 80, 19649–19673 (2021). https://doi.org/10.1007/s11042-020-10453-z

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-020-10453-z

Keywords

Search

Navigation