Skip to main content
SpringerLink
Log in

Are We Nearing Singularity? A Study of Language Capabilities of ChatGPT

  • Conference paper
  • First Online:
AGC 2023 (AGC 2023)

Part of the book series: Communications in Computer and Information Science ((CCIS,volume 2008))

Included in the following conference series:

  • 135 Accesses

Abstract

There has been a huge uproar about ChatGPT in recent times. There are widespread fears that generative models such as ChatGPT will affect employability and bring humanity closer towards singularity. In this paper, we assessed the capabilities and limitations of ChatGPT empirically. We subjected ChatGPT to a series of questions with different levels of lexical and syntactic language complexity and manually scored the responses. We measured the lexical and syntactic complexity of the questions using text analytics techniques. The analysis of the responses showed ChatGPT as a highly capable language model, with certain limitations. ChatGPT showed an increased likelihood of generating an inaccurate response with increase in syntactic language complexity. Though there have been sporadic efforts and reports about ChatGPT’s limitations, this paper presents a pioneering structured approach towards assessing ChatGPT’s language capabilities. This study takes a small, yet important step in research towards assessing language models. It opens avenues for future research towards strengthening the generator model and developing the discriminator dataset.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 49.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 64.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://www.thestreet.com/technology/chatgpt-is-the-new-disruptor-in-chief-but-theres-A-catch.

References

  1. Turing, A.M.: Intelligent machinery, a heretical theory*. Philos. Math. 4, 256–260 (1996). https://doi.org/10.1093/philmat/4.3.256

    Article  MathSciNet  Google Scholar 

  2. Cadwalladr, C.: Are the robots about to rise? Google’s new director of engineering thinks so…, (2014). https://www.theguardian.com/technology/2014/feb/22/robots-google-ray-kurzweil-terminator-singularity-artificial-intelligence

  3. Joy, B.: Why the future doesn’t need us (2000). http://www.wired.com/wired/archive/8.04/joy.%0Ahtml

  4. Eden, A.H., Steinhart, E., Pearce, D., Moor, J.H.: Singularity hypotheses: a scientific and philosophical assessment. Front. Collect. 1–13 (2012). https://doi.org/10.1007/978-3-642-32560-1

  5. Lee, L.-H., et al.: All one needs to know about Metaverse: a complete survey on technological singularity, virtual ecosystem, and research agenda. J. Latex Class Files. 14, 1–66 (2021). https://doi.org/10.48550/arXiv.2110.05352

  6. Trueman, C.: Tech layoffs in 2023: A timeline (2023). https://www.computerworld.com/article/3685936/tech-layoffs-in-2023-a-timeline.html

  7. Hanine, S., Dinar, B.: The challenges of human capital management in the VUCA era. J. Hum. Resour. Sustainab. Stud. 10, 503–514 (2022). https://doi.org/10.4236/jhrss.2022.103030

    Article  Google Scholar 

  8. Biswas, S.S.: Role of chat GPT in public health. Ann. Biomed. Eng. (2023). https://doi.org/10.1007/s10439-023-03172-7

    Article  Google Scholar 

  9. Rudolph, J., Tan, S., Tan, S.: ChatGPT: bullshit spewer or the end of traditional assessments in higher education? J. Appl. Learn. Teach. 6 (2023). https://doi.org/10.37074/jalt.2023.6.1.9

  10. King, M.R.: ChatGPT: a conversation on artificial intelligence, chatbots, and plagiarism in higher education. Cell. Mol. Bioeng. 16, 1–2 (2023). https://doi.org/10.1007/s12195-022-00754-8

    Article  Google Scholar 

  11. Lahat, A., Shachar, E., Avidan, B., Shatz, Z., Glicksberg, B.S., Klang, E.: Evaluating the use of large language model in identifying top research questions in gastroenterology. Sci. Rep. 13, 4164 (2023). https://doi.org/10.1038/s41598-023-31412-2

    Article  Google Scholar 

  12. Pourhoseingholi, M.A., Hatamnejad, M.R., Solhpour, A.: Does ChatGPT (or any other artificial intelligence language tools) deserve to be included in authorship list? Gastroenterol Hepatol Bed Bench (2023). https://doi.org/10.22037/ghfbb.v16i1.2747

  13. Lund, B.D., Wang, T.: Chatting about ChatGPT: how may AI and GPT impact academia and libraries? Library Hi Tech News. preprint (2023). https://doi.org/10.1108/LHTN-01-2023-0009

    Article  Google Scholar 

  14. Dwivedi, Y.K., et al.: “So what if ChatGPT wrote it?” Multidisciplinary perspectives on opportunities, challenges and implications of generative conversational AI for research, practice and policy. Int. J. Inf. Manage. 71, 102642 (2023). https://doi.org/10.1016/j.ijinfomgt.2023.102642

    Article  Google Scholar 

  15. Loconte, R., Orrù, G., Tribastone, M., Pietrini, P., Sartori, G.: Challenging ChatGPT’s “intelligence” with human tools: A neuropsychological investigation on prefrontal functioning of a large language model. SSRN. https://doi.org/10.2139/ssrn.4377371

  16. OpenAI: Models. https://platform.openai.com/docs/models/gpt-3 (2023)

  17. Flesch, R.: A new readability yardstick. J. Appl. Psychol. 32, 221–233 (1948). https://doi.org/10.1037/h0057532

    Article  Google Scholar 

  18. Bovair, S., Kieras, D.E., Poison, P.G.: The acquisition and performance of text-editing skill: a cognitive complexity analysis. Hum. Comput. Interact. 5, 1–48 (1990). https://doi.org/10.1207/s15327051hci0501_1

    Article  Google Scholar 

  19. Johnson, M.D.: Cognitive task complexity and L2 written syntactic complexity, accuracy, lexical complexity, and fluency: a research synthesis and meta-analysis. J. Second. Lang. Writ. 37, 13–38 (2017). https://doi.org/10.1016/j.jslw.2017.06.001

    Article  Google Scholar 

  20. Besharati, M., Izadi, M.: DAST Model: Deciding About Semantic Complexity of a Text (2019)

    Google Scholar 

  21. Pallotti, G.: A simple view of linguistic complexity. Second. Lang. Res. 31, 117–134 (2015). https://doi.org/10.1177/0267658314536435

    Article  Google Scholar 

  22. Lourdes, O.: Syntactic complexity measures and their relationship to L2 proficiency: a research synthesis of college level L2 writing. Appl. Linguist. 24, 492–518 (2003). https://doi.org/10.1093/applin/24.4.492

    Article  Google Scholar 

  23. Park, H.M.: Comparing group means: t-tests and one-way ANOVA using Stata, SAS, R, and SPSS (2009). https://hdl.handle.net/2022/19735

  24. Kleinbaum, D.G., Klein, M.: Logistic Regression. Springer, New York, NY (2010). https://doi.org/10.1007/978-1-4419-1742-3

  25. Hair, J.F., Black, W.C., Babin, B.J., Anderson, R.E.: Multivariate data analysis. Pearson, Noida, India (2013)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. NSHM Knowledge Campus, Kolkata, 700053, India

    Suparna Dhar

  2. IIM Ahmedabad, Vastrapur, Ahmedabad, India

    Indranil Bose

Authors
  1. Suparna Dhar
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Indranil Bose
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Suparna Dhar .

Editor information

Editors and Affiliations

  1. NSHM Institute of Computing & Analytics, NSHM Knowledge Campus Kolkata, Kolkata, India

    Suparna Dhar

  2. NSHM Knowledge Campus, Kolkata, India

    Sanjay Goswami

  3. Indian Institute of Management Bangalore, Bengaluru, India

    U. Dinesh Kumar

  4. Indian Institute of Management Ahmedabad, Ahmedabad, Gujarat, India

    Indranil Bose

  5. Liverpool Business School, Liverpool John Moores University, Liverpool, UK

    Rameshwar Dubey

  6. Department of Computer Science and Engineering, Jadavpur University, Kolkata, West Bengal, India

    Chandan Mazumdar

Appendix a: Sample Responses by ChatGPT

Appendix a: Sample Responses by ChatGPT

Question

ChatGPT response

Correctness

I was facing East and turned 180°, which direction am I facing now?

You are now facing West

Yes

How much is 135°plus 45°?

180°

Yes

I was facing East and turned 135° plus 45°, which direction am I facing now?

You are now facing South-Southeast

No

The afternoon sun is behind me - which direction am I facing?

You are facing west

No

I was facing East and turned 180° twice, which direction am I facing now?

You are now facing West

No

Name a fruit that starts with b

Banana

Yes

Name a fruit that starts with b and ends with y

Blueberry

Yes

Name a fruit that starts with b and ends with y and contains a in between

Banana

No

What is the king’s son called?

The king’s son is typically referred to as the crown prince or the prince

Yes

What is the king’s son’s mother called?

The king’s son’s mother is typically referred to as the queen or the queen mother

Yes

What is the king’s son’s mother’s son called?

The king’s son’s mother’s son is called the king’s grandson

No

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Dhar, S., Bose, I. (2024). Are We Nearing Singularity? A Study of Language Capabilities of ChatGPT. In: Dhar, S., Goswami, S., Dinesh Kumar, U., Bose, I., Dubey, R., Mazumdar, C. (eds) AGC 2023. AGC 2023. Communications in Computer and Information Science, vol 2008. Springer, Cham. https://doi.org/10.1007/978-3-031-50815-8_8

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-50815-8_8

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-50814-1

  • Online ISBN: 978-3-031-50815-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation