Encyclopedia of Big Data

Living Edition
| Editors: Laurie A. Schintler, Connie L. McNeely

Automated Modeling/Decision Making

  • Murad A. MithaniEmail author
Living reference work entry
DOI: https://doi.org/10.1007/978-3-319-32001-4_17-1

Keywords

Eventual Decision Organizational Decision Personal Bias Human Resource Department Economist Intelligence Unit 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Big data promises a significant change in the nature of information processing, and hence, decision making. The general reaction to this trend is that the access and availability of large amounts of data will improve the quality of individual and organizational decisions. However, there are also concerns that our expectations may not be entirely correct. Rather than simplifying decisions, big data may actually increase the difficulty of making effective choices. I synthesize the current state of research and explain how the fundamental implications of big data offer both a promise for improvement but also a challenge to our capacity for decision making.

Decision making pertains to the identification of the problem, understanding of the potential alternatives, and the evaluation of those alternatives to select the ones that optimally resolve the problem. While the promise of big data relates to all aspects of decision making, it more often affects the understanding, the evaluation, and the selection of alternatives. The resulting implications comprise of the dual decision model, higher granularity, objectivity, and transparency of decisions, and the bottom-up decision making in organizational contexts. I explain each of these implications in detail to illustrate the associated opportunities and challenges.

With data and information exceeding our capacity for storage, there is a need for decisions to be made on the fly. While this does not imply that all decisions have to be immediate, our inability to store large amounts of data that is often generated continuously suggests that decisions pertaining to the use and storage of data, and therefore the boundaries of the eventual decision making context, need to be defined earlier in the process. With the parameters of the eventual decision becoming an apriori consideration, big data is likely to overcome the human tendency of procrastination. It imposes the discipline to recognize the desired information content early in the process. Whether this entails decision processes that prefer immediate conclusions or if the early choices are limited to the identification of critical information that will be used for later evaluation, the dual decision model with a preliminary decision far removed from the actual decision offers an opportunity to examine the available alternatives more comprehensively. It allows decision makers to have a greater understanding of the alignment between goals and alternatives. Compare this situation to the recruitment model for a human resource department that screens as well as finalizes prospective candidates in a single round of interviews, or separates the process into two stages where the potential candidates are first identified from the larger pool and they are then selected from the short-listed candidates in the second stage. The dual decision model not only facilitates greater insights, it also eliminates the fatigue that can seriously dampen the capacity for effective decisions. Yet this discipline comes at a cost. Goals, values, and biases that are part of the early phase of a project can leave a lasting imprint. Any realization later in the project that was not deliberately or accidently situated in the earlier context becomes more difficult to incorporate into the decision. In the context of recruitment, if the skills desired of the selected candidate change after the first stage, it is unlikely that the short-listed pool will rank highly in that skill. The more unique is the requirement that emerges in the later stage, the greater is the likelihood that it will not be sufficiently fulfilled. This tradeoff suggests that an improvement in our understanding of the choices comes at the cost of limited maneuverability of an established decision context.

In addition to the benefits and costs of early decisions in the data generation cycle, big data allows access to information at a much more granular level than possible in the past. Behaviors, attitudes, and preferences can now be tracked in extensive detail, fairly continuously, and over longer periods of time. They can in turn be combined with other sources of data to develop a broader understanding of consumers, suppliers, employees, and competitors. Not only can we understand in much more depth the activities and processes that pertain to various social and economic landscapes, higher level of granularity makes decisions more informed and, as a result, more effective. Unfortunately, granularity also brings with it the potential of distraction. All data that pertains to a choice may not be necessary for the decision, and excessive understanding can overload our capacity to make inferences. Imagine the human skin which is continuously sensing and discarding thermal information generated from our interaction with the environment. What if we had to consciously respond to every signal detected by the skin? It is this loss of granularity that comes through the human mind responsive only to significant changes in temperature that saves us from being overwhelmed by data. Even though information granularity makes it possible to know what was previously impossible, information overload can lead us astray towards inappropriate choices, and at worse, it can incapacitate our ability to make effective decisions.

The third implication of big data is the potential for objectivity. When a planned and comprehensive examination of alternatives is combined with a deeper understanding of the data, the result is more accurate information. This makes it less likely for individuals to come up to an incorrect conclusion. This eliminates the personal biases that can prevail in the absence of sufficient information. Since traditional response to overcome the effect of personal bias is to rely on individuals with greater experience, big data predicts an elimination of the critical role of experience. In this vein, Andrew McAfee and Erik Brynjolfson (2012) find that regardless of the level of experience, firms that extensively rely on data for decision making are, on average, 6% more profitable than their peers. This suggests that as decisions become increasingly imbibed with an objective orientation, prior knowledge becomes a redundant element. This however does not eliminate the value of domain-level experts. Their role is expected to evolve into individuals who know what to look for (by asking the right questions) and where to look (by identifying the appropriate sources of data). Domain expertise and not just experience is the mantra to identify people who are likely to be the most valuable in this new information age. However, it needs to be acknowledged that this belief in objectivity is based on a critical assumption: individuals endowed with identical information that is sufficient and relevant to the context, reach identical conclusions. Yet anyone watching the same news story reported by different media outlets knows the fallacy of this assumption. The variations that arise when identical facts lead individuals to contrasting conclusions are a manifestation of the differences in the way humans work with information. Human cognitive machinery associates meanings to concepts based on personal history. As a result, even while being cognizant of our biases, the translation of information into conclusion can be unique to individuals. Moreover, this effect compounds with the increase in the amount of information that is being translated. While domain experts may help ensure consistency with the prevalent norms of translation, there is little reason to believe that all domain experts are generally in agreement. The consensus is possible in the domains of physical sciences where objective solutions, quantitative measurements, and conceptual boundaries leave little ambiguity. However, the larger domain of human experience is generally devoid of standardized interpretations. This may be one reason that a study by the Economist Intelligence Unit (2012) found a significantly higher proportion of data-driven organizations in the industrial sectors such as the natural resources, biotechnology, healthcare, and financial services. Lack of extensive reliance on data in the other industries is symptomatic of our limited ability for consensual interpretation in areas that challenge the positivistic approach.

The objective nature of big data produces two critical advantages for organizations. The first is transparency. A clear link between data, information, and decision implies the absence of personal and organizational biases. Interested stakeholders can take a closer look at the data and the associated inferences to understand the basis of conclusions. Not only does this promise a greater buy-in from participants that are affected by those decisions, it develops a higher level of trust between decision makers and the relevant stakeholders, and it diminishes the need for external monitoring and governance. Thus, transparency favors the context in which human interaction becomes easier. It paves the way for richer exchange of information and ideas. This in turn facilitates the quality of future decisions. But due to its very nature, big data makes replications rather difficult. The time, energy, and other resources required to fully understand or reexamine the basis of choices makes transparency not an antecedent but a consequence of trust. Participants are more likely to believe in transparency if they already trust the decision makers, and those that are less receptive to the choices remain free to accuse the process as opaque. Regardless of the comprehensiveness of the disclosed details, transparency largely remains a symbolic expression of the participants’ faith in the people managing the process.

A second advantage that arises from the objective nature of data is decentralization. Given that decisions made in the presence of big data are more objective and require lower monitoring, they are easier to delegate to people who are closer to the action. By relying on proximity and exposure as the basis of assignments, organizations can save time and costs by avoiding the repeated concentration and evaluation of information that often occurs at the various hierarchical levels as the information travels upwards. So unlike the flatter organizations of the current era which rely on the free flow of information, lean organizations of the future may decrease the flow of information altogether, replacing it with data-driven, contextually rich, and objective findings. In fact, this is imminent since the dual decision model defines the boundaries of subsequent choices. Any attempt to disengage the later decision from the earlier one is likely to eliminate the advantages of granularity and objectivity. Flatter organizations of the future will delegate not because managers have greater faith in the lower cadres of the organization but because individuals at the lower levels are the ones that are likely to be best positioned to make timely decisions. As a result, big data is moving us towards a bottom-up model of organizational decisions where people at the interface between data and findings determine the strategic priorities within which higher-level executives can make their call. Compare this with the traditional top-down model of organizational decisions where strategic choices of the higher executives define the boundaries of actions for the lower-level staff. However, the bottom-up approach is also fraught with challenges. It minimizes the value of executive vision. The subjective process of environmental scanning allows senior executives to imbibe their valued preferences into organizational choices through selective attention to information. It enables organizations to do what would be uninformed and at times, highly irrational. Yet it sustains the spirit of beliefs that take the form of entrepreneurial action. By setting up a mechanism where facts and findings run supreme, organization of the future may constrain themselves to do only what is measureable. Extensive reliance on data can impair our capacity to imagine what lies beyond the horizon (Table 1).
Table 1

Opportunities and challenges for the decision implications of big data

 

Big data implication

Opportunity

Challenge

1.

Dual decision model

Comprehensive examination of alternatives

Early choices can constrain later considerations

2.

Granularity

In-depth understanding

Critical information can be lost due to information overload

3.

Objectivity

Lack of dependence on experience

Inflates the effect of variations in translation

4.

Transparency

Free-flow of ideas

Difficult to validate

5.

Bottom-up decision making

Prompt decisions

Impairment of vision

In sum, the big data revolution promises a change in the way individuals and organizations make decisions. But it also brings with it a host of challenges. The opportunities and threats discussed in this article reflect different facets of the implications that are fundamental to this revolution. They include the dual decision model, granularity, objectivity, transparency, and the bottom-up approach to organizational decisions. The table above summarizes how the promise of big data is an opportunity as well as a challenge for the future of decision making.

Cross-References

Further Readings

  1. Boyd, D., & Crawford, K. (2012). Critical questions for big data. Information, Communication & Society, 15(5), 662–679.CrossRefGoogle Scholar
  2. Economist Intelligence Unit. (2012). The deciding factor: Big data & decision making. New York, NY, USA: Capgemini/The Economist.Google Scholar
  3. McAfee, A., & Brynjolfsson, E. (2012). Big data: The management revolution. Harvard Business Review, 90(10), 61–67.Google Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  1. 1.School of BusinessStevens Institute of TechnologyHobokenUSA