Abstract
People ask questions in order to efficiently learn about the world. But do people ask good questions? In this work, we designed an intuitive, game-based task that allowed people to ask natural language questions to resolve their uncertainty. Question quality was measured through Bayesian ideal observer models that considered large spaces of possible game states. During free-form question generation, participants asked a creative variety of useful and goal-directed questions, yet they rarely asked the best questions as identified by the Bayesian ideal observers (Experiment 1). In subsequent experiments, participants strongly preferred the best questions when evaluating questions that they did not generate themselves (Experiments 2 and 3). On one hand, our results show that people can accurately evaluate question quality, even when the set of questions is diverse and an ideal observer analysis has large computational requirements. On the other hand, people have a limited ability to synthesize maximally informative questions from scratch, suggesting a bottleneck in the question asking process.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Notes
In some studies, participants performed information-seeking actions, such as clicking on a certain part of an object, to obtain information about the object, which is, for our purposes, equivalent to asking information-seeking questions. For instance, participants could click on either the eye or claw of a plankton creature presented on a computer screen, to reveal the eye/claw color and then categorize the plankton based on that information (Meder and Nelson 2012), which is equivalent to asking “What is the color of the eye/claw?” Similarly, in Coenen et al. (2015), participants could click on one of three nodes in a causal network and subsequently observe which of the other nodes would turn on, which is equivalent to asking “Which nodes will turn on when I activate this node?”
We decided against paying people based on question quality. Participants would have to reason about what we, the experimenters, expect to be good questions.
An example from the small set of questions that were not formalized asked whether the purple ship was larger than the part of it that was so far revealed on the board. This question is equivalent with asking “Is the purple ship larger than n?,” where n is the number of purple tiles already revealed in the particular context. Interestingly, only a small fraction (\(\thicksim 1\%\)) of questions necessitated such dynamic reference to the partly revealed board, while all others could be answered by only accessing the information of the true underlying board. The dropped questions did not seem to be especially informative; thus, it is unlikely that leaving them out changed the results significantly. At least in principle, all of these questions can be formalized in our model given sufficient computational power.
For each estimation, we report the 95% highest density interval (HDI), based on Bayesian data analysis. In all of these estimations, broad priors as described by Kruschke (2013) were used.
The free-from questions for each context in Experiment 1 were placed in a 2D space with EIG and generation frequency as dimensions. We then sampled 1000 six-question subsets and took the sample with the largest average pairwise distance between questions in the subset.
References
Anderson, J.R. (1990). The adaptive character of thought. Hillsdale: Lawrence Erlbaum Associates.
Cakmak, M., & Thomaz, A.L. (2012). Designing robot learners that ask good questions. In Proceedings of the Seventh Annual ACM/IEEE International Conference on Human-Robot Interaction (pp. 17–24). https://doi.org/10.1145/2157689.2157693.
Chater, N., Crocker, M., Pickering, M. (1998). The rational analysis of inquiry: The case of parsing. In Oaksford, M., & Chater, N. (Eds.) Rational models of cognition (pp. 441–468). Oxford University Press.
Chin, C., & Brown, D.E. (2002). Student-generated questions: A meaningful aspect of learning in science. International Journal of Science Education, 24(5), 521–549.
Clark, H. (1979). Responding to indirect speech acts. Cognitive Psychology, 11(4), 430–477.
Coenen, A., Nelson, J.D., Gureckis, T.M. (in press). Asking the right questions about human inquiry: Nine open challenges. Psychonomic Bulletin & Review.
Coenen, A., Rehder, B., Gureckis, T.M. (2015). Strategies to intervene on causal systems are adaptively selected. Cognitive Psychology, 79, 102–133.
Davey, B., & McBride, S. (1986). Effects of question-generation training on reading comprehension. Journal of Educational Psychology, 78(4), 256–262.
Dillon, J.T. (1988). The remedial status of student questioning. Journal of Curriculum Studies, 20(3), 197–210.
Graesser, A.C., Langston, M.C., Bagget, W.B. (1993). Exploring information about concepts by asking questions. The Psychology of Learning and Motivation, 29, 411–436.
Graesser, A.C., Person, N., Huber, J. (1992). Mechanisms that generate questions. In Lauer, T.W., Peacock, E., Graesser, A.C. (Eds.) Questions and information systems (pp. 167–187). Hillsdale, Erlbaum.
Graesser, A.C., & Person, N.K. (1994). Question asking during tutoring. American Educational Research Journal, 31(1), 104–137.
Gureckis, T.M., & Markant, D.B. (2009). Active learning strategies in a spatial concept learning game. In Proceedings of the 31st Annual Conference of the Cognitive Science Society.
Gureckis, T.M., Martin, J., McDonnell, J., Rich, A.S., Markant, D., Coenen, A., Chan, P. (2016). psiTurk: an open-source framework for conducting replicable behavioral experiments online. Behavior Research Methods, 48(3), 829–842.
Hawkins, R.X.D., Stuhlmuller, A., Degen, J., Goodman, N.D. (2015). Why do you ask? Good questions provoke informative answers. In Noelle, C.D., & et al. (Eds.) Austin, TX: Cognitive Science Society.
Hendrickson, A.T., Navarro, D.J., Perfors, A. (2016). Sensitivity to hypothesis size during information search. Decision, 3(1), 62.
Kruschke, J.K. (2013). Bayesian estimation supersedes the t test. Journal of Experimental Psychology: General, 142(2), 573–603.
Markant, D.B., & Gureckis, T.M. (2012). Does the utility of information influence sampling behavior? In Proceedings of the 34th Annual Conference of the Cognitive Science Society.
Markant, D.B., & Gureckis, T.M. (2014a). Is it better to select or to receive? Learning via active and passive hypothesis testing. Journal of Experimental Psychology: General, 143(1), 94.
Markant, D.B., & Gureckis, T.M. (2014b). A preference for the unpredictable over the informative during self-directed learning. In Proceedings of the 36th Annual Conference of the Cognitive Science Society.
Marr, D.C. (1982). Vision. San Francisco: W.H. Freeman and Company.
Meder, B., & Nelson, J.D. (2012). Information search with situation-specific reward functions. Judgment and Decision Making, 7(2), 119–148.
Nelson, J.D., Divjak, B., Gudmundsdottir, G., Martignon, L.F., Meder, B. (2014). Children’s sequential information search is sensitive to environmental probabilities. Cognition, 130(1), 74–80.
Nelson, K. (1973). Structure and strategy in learning to talk. Monographs of the Society for Research in Child Development, 38(1-2, Serial No. 149), 1–135.
Oaksford, M., & Chater, N. (1994). A rational analysis of the selection task as optimal data selection. Psychological Review, 101(4), 608–631.
Rothe, A., Lake, B.M., Gureckis, T.M. (2017). Question asking as program generation. Advances Neural Information Processing Systems, 30, 1046–1055.
Ruggeri, A., & Feufel, M.A. (2015). How basic-level objects facilitate question-asking in a categorization task. Frontiers in Psychology, 6.
Ruggeri, A., Lombrozo, T., Griffiths, T.L., Xu, F. (2016). Sources of developmental change in the efficiency of information search. Developmental Psychology, 52(12), 2159–2173.
Settles, B. (2009). Active learning literature survey (Tech. Rep.). University of Wisconsin-Madison.
Settles, B. (2012). Active learning. San Rafael: Morgan & Claypool Publishers.
Shannon, C.E. (1948). A mathematical theory of communication. The Bell System Technical Journal, 27, 379–423.
Funding
This research was supported by a National Science Foundation grant BCS-1255538, the John Templeton Foundation “Varieties of Understanding” project, a John S. McDonnell Foundation Scholar Award to TMG, and the Moore-Sloan Data Science Environment at NYU.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Rothe, A., Lake, B.M. & Gureckis, T.M. Do People Ask Good Questions?. Comput Brain Behav 1, 69–89 (2018). https://doi.org/10.1007/s42113-018-0005-5
Published:
Issue Date:
DOI: https://doi.org/10.1007/s42113-018-0005-5