Abstract
This chapter provides a tutorial on conducting epistemic network analysis (ENA) and ordered network analysis (ONA) using R. We introduce these two techniques together because they share similar theoretical foundations, but each addresses a different challenge for analyzing large-scale qualitative data on learning processes. ENA and ONA are methods for quantifying, visualizing, and interpreting network data. Taking coded data as input, ENA and ONA represent associations between codes in undirected or directed weighted network models, respectively. Both techniques measure the strength of association among codes and illustrate the structure of connections in network graphs, and they quantify changes in the composition and strength of those connections over time. Importantly, ENA and ONA enable comparison of networks both visually and via summary statistics, so they can be used to explore a wide range of research questions in contexts where patterns of association in coded data are hypothesized to be meaningful and where comparing those patterns across individuals or groups is important.
You have full access to this open access chapter, Download chapter PDF
Keywords
1 Introduction
This chapter provides a tutorial on conducting epistemic network analysis (ENA) and ordered network analysis (ONA) using R. We introduce these two techniques together because they share similar theoretical foundations, but each addresses a different challenge for analyzing large-scale qualitative data on learning processes.
ENA and ONA are methods for quantifying, visualizing, and interpreting network data. Taking coded data as input, ENA and ONA represent associations between codes in undirected or directed weighted network models, respectively. Both techniques measure the strength of association among codes and illustrate the structure of connections in network graphs, and they quantify changes in the composition and strength of those connections over time. Importantly, ENA and ONA enable comparison of networks both visually and via summary statistics, so they can be used to explore a wide range of research questions in contexts where patterns of association in coded data are hypothesized to be meaningful and where comparing those patterns across individuals or groups is important.
In the following sections, we will (1) briefly review literature relevant to the application of ENA and ONA, (2) provide a step-by-step guide to implementing ENA and ONA in R, and (3) suggest additional resources and examples for further exploration. By the end of this chapter, readers will be able to apply these techniques in their own research.
2 Literature Review
2.1 Epistemic Network Analysis (ENA)
ENA is a method for identifying and quantifying connections in coded data and representing them in undirected weighted network models [1]. There are two key features that differentiate ENA from other networks analysis tools or multivariate analyses: (1) ENA produces summary statistics that can be used to compare the differences in the content of networks rather than just their structure; and (2) ENA network visualizations provide information that is mathematically consistent with those summary statistics, which facilitates meaningful interpretation of statistical differences [2]. These features enable researchers to analyze a wide range of phenomena in learning analytics, including complex thinking and knowledge construction [3, 4], collaborative problem solving [5, 6], socio-emotional aspects of learning [7], mentoring [8], and teacher professional development [9,9,10,11].
One key feature that makes ENA an effective method in modeling collaborative interaction is that ENA can model individuals’ unique contributions to collaborative discourse while accounting for group context, and thus both individuals and groups can be analyzed in the same model. This feature is particularly valuable in collaborative learning environments, where the interactions and contributions of each individual are related and should not be treated as a series of isolated events. For example, Swiecki et al. [6] analyzed the communications of air defense warfare teams in training exercises and found that ENA was not only able to reveal differences in individual performance identified in a qualitative analysis of the collaborative discourse, but also to test those differences statistically.
2.2 Ordered Network Analysis (ONA)
Ordered Network Analysis (ONA) extends the theoretical and analytical advantages of ENA to account for the order of events by producing directed weighted networks rather than undirected models [12]. Like ENA, ONA takes coded data as input, identifies and measures connections among coded items, and visualizes the structure of connections in a metric space that enables both statistical and visual comparison of networks. However, ONA models the order in which codes appear in the data, enabling analysis of phenomena in which the order of events is hypothesized to be important.
For example, Tan et al. [12] used ONA to model the performance of military teams learning to identify, assess, and respond to potential threats detected by radar. The findings demonstrate that ONA could detect qualitative differences between teams in different training conditions that were not detected with unordered models and show that they are statistically significant. In their work, Tan et al. [12] argued that ONA possesses an advantage over methods such as Sequential Pattern Mining (SPM), which is widely used to identify frequent sequential patterns. In contrast to SPM, which prioritizes the specific micro-sequential order of events, ONA models processes by accounting for the co-temporal order of interactions between the units of analysis in response and what they are responding to. Consequently, ONA is a more appropriate methodological choice when modeling processes in ill-formed problem-solving scenarios, where collaborative interactions do not follow a prescribed sequence of steps but where the order of activities is still important.
ONA has also been used to analyze log data from online courses. For example, Fan et al. [13] analyzed self-regulated learning tactics employed by learners in Massive Open Online Courses (MOOC) using ONA and process mining. The authors found that ONA provided more nuanced interpretations of learning tactics compared to process mining because ONA models learning tactics across four dimensions: frequency, continuity, order, and the role of specific learning actions within broader tactics.
Like ENA, ONA produces summary statistics for network comparison and mathematically consistent network visualizations that enable interpretation of statistical measures. Unlike ENA, ONA models the order in which codes appear in data, enabling researchers to investigate whether and to what extent the order of events is meaningful in a given context.
In the following sections, we provide a step-by-step guide to conducting ENA and ONA analyses in R.
3 Epistemic Network Analysis in R
In this section, we demonstrate how to conduct an ENA analysis using the rENA package. If you are not familiar with ENA as an analytic technique, we recommend that you first read Shaffer [1], Shaffer and Ruis [14], and Bowman et al. [2] to familiarize yourself with the theoretical and methodological foundations of ENA.
3.1 Install the rENA Package and Load the Library
Before installing the rENA package, be sure that you are using R version 4.1 or newer. To check your R version, type R.version in your console. To update your R version (if needed), download and install R from the official R website: https://cran.r-project.org/
First, install the rENA package and then load the rENA library after installation is complete.
We also install the other package that is required for accessing the view() function Sect. 3.7.3 in rENA.
3.2 Dataset
The dataset we will use as an example, RS.data, is included in the rENA package. Note that the RS.data file in the package is only a subset of the full dataset, and is thus intended for demonstration purposes only.
To start, pass RS.data from the rENA package to a data frame named data.
Use the head() function in R to subset and preview the first three rows present in the input data frame to familiarize yourself with the data structure.
RS.data consists of discourse from RescuShell, an online learning simulation where students work as interns at a fictitious company to solve a realistic engineering design problem in a simulated work environment. Throughout the internship, students communicate with their project teams and mentors via online chat, and these chats are recorded in the “text” column. A set of qualitative codes were applied to the data in the “text” column, where a value of 0 indicates the absence of the code and a value of 1 indicates the presence of the code in a given line.
Further details about the RS.data dataset can be found in Shaffer and Arastoopour [15]. Analyses of data from RescuShell and other engineering virtual internships can be found in Arastoopour et al. [16] and Chesler et al. [17].
3.3 Construct an ENA Model
To construct an ENA model, there is a function called ena which enables researchers to set the parameters for their model. This function wraps two other functions—ena.accumulate.data and ena.make.set—which can be used together to achieve the same result.
In the following sections, we will demonstrate how to set each parameter and explain how different choices affect the resulting ENA model.
3.3.1 Specify Units
In ENA, units can be individuals, ideas, organizations, or any other entity whose structure of connections you want to model. To set the units parameter, specify which column(s) in the data contain the variables that identify unique units.
For this example, choose the “Condition” column and the “UserName” column to define the units. The “Condition” column has two unique values: FirstGame, and SecondGame, representing novice users and relative expert users, respectively, as some students participated in RescuShell after having already completed a different engineering virtual internship. The “UserName” column includes unique user names for all students (n = 48). This way of defining the units means that ENA will construct a network for each student in each condition.
To verify that the units are correctly specified, subset and preview the unique values in the units columns. There are 48 units from two conditions, which means that the ENA model will produce 48 individual-level networks for each of the units, and each unit is uniquely associated with either the novice group (FirstGame) or the relative expert group (SecondGame).
3.3.2 Specify Codes
Next, specify the columns that contain the codes. Codes are concepts whose pattern of association you want to model for each unit. ENA represent codes as nodes in the networks and co-occurrences of codes as edges. Most researchers use binary coding in ENA analyses, where the values in the code columns are either 0 (indicating that the code is not present in that line) or 1 (indicating that the code is present in that line). RS.data contains six code columns, all of which will be used here.
To specify the code columns, enter the code column names in a vector.
To verify that the codes are correctly specified, preview the code columns selected.
3.3.3 Specify Conversations
The conversation parameter determines which lines in the data can be connected. Codes in lines that are not in the same conversation cannot be connected. For example, you may want to model connections within different time segments, such as days, or different steps in a process, such as activities.
In our example, choose the “Condition”, “GroupName”, and “ActivityNumber” columns to define the conversations. These choices indicate that connections can only happen between students who were in the same condition (FirstGame or SecondGame) and on the same project team (group), and within the same activity. This definition of conversation reflects what actually happened in the simulation: in a given condition, students only interacted with those who were in the same group, and each activity occurred on a different day.
To specify the conversation parameter, enter the column names in a vector.
To verify that the conversations are correctly specified, subset and preview the unique values in the conversation columns.
3.3.4 Specify the Window
Once the conversation parameter is specified, a window method needs to be specified. Whereas the conversation parameter specifies which lines can be related, the window parameter determines which lines within the same conversation are related. The most common window method used in ENA is called a moving stanza window, which is what will be used here.
Briefly, a moving stanza window is a sliding window of fixed length that moves through a conversation to detect and accumulate code co-occurrences in recent temporal context. The lines within a designated stanza window are considered related to each other. For instance, if the moving stanza window is 7, then each line in the conversation is linked to the six preceding lines. See Siebert-Evenstone et al. [18] and Ruis et al. [19] for more detailed explanations of windows in ENA models.
Here, set the window.size.backFootnote 1parameter equal to 7. User can specify a different moving stanza window size by passing a different numerical value to the `window.size.back` parameter.
3.3.5 Specify Groups and Rotation Method
When specifying the units, we chose a column that indicates two conditions: FirstGame (novice group) and SecondGame (relative expert group). To enable comparison of students in these two conditions, three additional parameters need to be specified: groupVar, groups, and mean.
These three parameters indicate that when building the ENA model, the first dimension will maximize the difference between the two conditions: FirstGame and SecondGame. This difference maximization is achieved through mean = TRUE, which specifies that a means rotation will be performed at the dimensional reduction stage. If the means rotation is set to FALSE or there aren’t two distinct groups in your data and you still set mean as TRUE, ENA will by default use singular value decomposition (SVD) to perform the dimensional reduction. Bowman et al. [2] provide a mathematical explanation of the methods used in ENA to perform dimensional reductions.
3.3.6 Specify Metadata
The last parameter to be specified is metadata. Metadata columns are not required to construct an ENA model, but they provide information that can be used to subset units in the resulting model.
Specify the metadata columns shown below to include data on student outcomes related to reported self-confidence before and after participating in engineering virtual internships. We will use this data to demonstrate a simple linear regression analysis that can be done using ENA outputs as predictors.
3.3.7 Construct an Model
Now that all the essential parameters have been specified, the ENA model can be constructed.
The ena function constructs the ENA model, and we recommend that you store the output in an object (in this case, set.ena).
As noted above, the ena helper function combines the functions ena.accumu late.data and ena.make.set. The following code will construct the same ENA model specified above using these two functions.
3.4 Summary of Key Model Outputs
Users can explore what is stored in the object set by typing set$ and select items from the drop down list. Here, we briefly describe the top-level items in set that are often of interest.
3.4.1 Connection Counts
Connection counts are the frequencies of unique connections a unit made. For each unit, ENA creates a cumulative adjacency vector that contains the sums of all unique code co-occurrences for that unit across all stanza windows. Here, there are 48 units in the ENA model, so there are 48 adjacency vectors. Each term in an ENA adjacency vector represents a unique co-occurrence of codes. Thus with six codes, each vector has 15 terms (n choose two). This is because ENA models are undirected and do not model co-occurrences of the same code.
To access ENA adjacency vectors, use set.ena$connection.counts.
3.4.2 Line Weights
To compare networks in terms of their relative patterns of association, researchers can spherically normalize the cumulative adjacency vectors by diving each one by its length. The resulting normalized vectors represent each unit’s relative frequencies of code co-occurrence. In other words, the sphere normalization controls for the fact that different units might have different amounts of interaction or different numbers of activities than others.
Notice that in set.ena$connection.counts, the value for each unique code co-occurrence is an integer equal or greater than 0, because they represent the raw connection counts between each unique pair of codes. In set.``ena``$line.weights, those raw counts are normalized, and therefore the values are rational numbers between 0 and 1.
To access the normalized adjacency vectors, use set.ena$line.weights.
3.4.3 ENA Points
As the product of a dimensional reduction, for each unit, ENA produces an ENA point in a two-dimensional space. Since there are 48 units, ENA produces 48 ENA points.
By default, rENA visualizes ENA points on an x-y coordinate plane defined by the first two dimensions of the dimensional reduction: for a means rotation, MR1 and SVD2, and for an SVD, SVD1 and SVD2.
To access these points, use set.ena$points.
ENA points are thus summary statistics that researchers can use to conduct statistical tests, and they can also be used in subsequent analyses. For example, statistical differences between groups in the data can be tested using ENA dimension scores, and those scores can also be used in regression analyses to predict outcome variables, which we will demonstrate later.
3.4.4 Rotation Matrix
The rotation matrix used during the dimensional reduction can be accessed through set.ena$rotation. This is mostly useful when you want to construct an ENA metric space using one dataset and then project ENA points from different data into that space, as in Sect. 5.1.
3.4.5 Metadata
set$meta.data returns a data frame that includes all the columns of the ENA set except for the columns representing code co-occurrences.
3.5 ENA Visualization
Once an ENA set is constructed, it can be visualized, which facilitates interpretation of the model. Here, we will look at the two conditions, “FirstGame” (novices) and “SecondGame” (relative experts), by plotting their mean networks.
3.5.1 Plot a Mean Network
To plot a network, use the ena.plot.network function. This function requires the network parameter (a character vector of line weights), and the line weights come from set$line.weights.
First, subset line weights for each of the two groups.
Next, calculate the mean networks for the two groups, and store the line weights as vectors.
During plotting, use a pipe | > to send the output of one function into the first parameter of the subsequent function. To distinguish the two mean networks, set the color of the FirstGame mean network to red (Fig. 1).
ENA mean network for FirstGame group
and the color of the SecondGame mean network to blue (Fig. 2).
ENA mean network for SecondGame group
As you can see from the two network visualizations above, their node positions are exactly same. All ENA networks from the same model have the same node positions, which are determined by an optimization routine that attempts to place the nodes such that the centroid of each unit’s network and the location of the ENA point in the reduced space are co-located.
Because of the fixed node positions, ENA can construct a subtracted network, which enables the identification of the most salient differences between two networks. To do this, ENA subtracts the weight of each connection in one network from the corresponding weighted connection in another network, then visualizes the differences in connection strengths. Each edge is color-coded to indicate which of the two networks contains the stronger connection, and the thickness and saturation of the edges corresponds to the magnitude of the difference.
To plot a subtracted network, first calculate the subtracted network line weights by subtracting one group’s line weights from the other. (Because ENA computes the absolute values of the differences in edge weights, the order of the two networks in the subtraction doesn’t matter.)
Then, use the ena.plot function to plot the subtracted network. If the differences are relatively small, a multiplier can be applied to rescale the line weights, improving legibility (Fig. 3).
ENA subtracted mean network for FirstGame (red) and SecondGame (blue)
Here, the subtracted network shows that on average, students in the FirstGame condition (red) made more connections with Technical.Constraints and Collaboration than students in the SecondGame condition (blue), while students in the SecondGame condition made more connections with Design.Reasoning and Performance.Parameters than students in the FirstGame condition. This is because students with more experience of engineering design practices did not need to spend as much time and effort managing the collaborative process and learning about the basic technical elements of the problem space, and instead spent relatively more time focusing on more complex analysis and design reasoning tasks.
Note that this subtracted network shows no connection between Technical.Constraints and Design.Reasoning, simply because the strength of this connection was similar in both conditions. Thus, subtraction networks should always be visualized along with with the two networks being subtracted.
3.5.2 Plot a Mean Network and its Points
The ENA point or points associated with a network or mean network can also be visualized.
To visualize the points associated with each of the mean networks plotted above, use set$points to subset the rows that are in each condition and plot each condition as a different color.
Then, plot the FirstGame mean network the same as above using ena.plot.network, use | > to pipe in the FirstGame points that we want to include, and plot them using ena.plot.points.
Each point in the space is the ENA point for a given unit. The red and blue squares on the x-axis are the means of the ENA points for each condition, along with the 95% confidence interval on each dimension. (might need to zoom in for better readability).
Since we used a means rotation to construct the ENA model, the resulting space highlights the differences between FirstGame and SecondGame by constructing a rotation that places the means of each condition as close as possible to the x-axis of the space and maximizes the distance between them (Figs. 4, 5, 6, 7, and 8).
FirstGame ENA points (dots), mean point (square), and confidence interval (box)
FirstGame ENA mean network and points
SecondGame ENA points (dots), mean point (square), and confidence interval (box)
SecondGame ENA mean network and points
ENA subtracted mean network for FirstGame (blue) and SecondGame (red)
Then, do the same for the SecondGame condition.
Lastly, do the same for subtraction as well.
Note that the majority of the red points (FirstGame) are located on the left side of the space, and the blue points (SecondGame) are mostly located on the right side of the space. This is consistent with the line weights distribution in the mean network: the FirstGame units make relatively more connections with nodes on the left side of the space, while the SecondGame units make relatively more connections with nodes on the right side of the space. The positions of the nodes enable interpretation of the dimensions, and thus interpretation of the locations of the ENA points.
3.5.3 Plot an Individual Unit Network and its Point
Plotting the network and ENA point for a single unit uses the same approach. First, subset the line weights and point for a given unit.
Then, plot the network and point for that unit (Fig. 9).
EAN network for a student from FirstGame and its corresponding ENA point
Following the exact same procedure, we can, for example, choose a unit from the other condition to plot and also construct a subtracted plot for those two units (Fig. 10).
ENA network for a student from SecondGame and its corresponding ENA point
To visually analyze the differences between the two individual networks, plot their subtracted network (Fig. 11).
ENA subtracted network showing the differences between one student from FirstGame (red) and another student from SecondGame (blue)
In this unit-level subtracted network, Unit A (red) made relatively more connections with codes such as Technical.Constraints, Data, and Collaboration, while Unit B (blue) made relatively more connections with Design.Reasoning and Performance.Parameters.
3.5.4 Plot Everything, Everywhere, All at Once
The helper function ena.plotter enables users to plot points, means, and networks for each condition at the same time. This gives the same results as above more parsimoniously. However, this approach does not enable customization of edge and point colors.
3.6 Compare Groups Statistically
In addition to visual comparison of networks, ENA points can be analyzed statistically. For example, here we might test whether the patterns of association in one condition are significantly different from those in the other condition.
To demonstrate both parametric and non-parametric approaches to this question, the examples below use a Student’s t test and a Mann-Whitney U test to test for differences between the FirstGame and SecondGame condition. For more on differences between parametric and non-parametric tests, see Kaur and Kumar [20].
First, install the lsr package to enable calculation of effect size (Cohen’s d) for the t test.
Then, subset the points to test for differences between the points of the two conditions.
Conduct the t test on the first and second dimensions.
Compute any other statistics that may be of interest. A few examples are given below.
Here, along the x axis (MR1), a two-sample t test assuming unequal variance shows that the FirstGame (mean = −0.09, SD = 0.11, N = 26) condition is statistically significantly different for alpha = 0.05 from the SecondGame condition (mean = 0.11, SD = 0.10, N = 22; t(45.31) = −6.52, p = 0.00, Cohen’s d = 1.88). Along the y axis (SVD2), a two-sample t test assuming unequal variance shows that the FirstGame condition (mean = 0.11, SD = 0.13, N = 26) is not statistically significantly different for alpha = 0.05 from the SecondGame condition (mean = 0.00, SD = 1.3, N = 22; t(43.17) = 0, p = 1.00).
The Mann-Whitney U test is a non-parametric alternative to the independent two-sample t test.
First, install the rcompanion package to calculate the effect size (r) for a Mann-Whitney U test.
Then, conduct a Mann-Whitney U test on the first and second dimensions.
Compute any other statistics that may be of interest. A few examples are given below.
Here, along the x axis (MR1), a Mann-Whitney U test shows that the FirstGame condition (Mdn = −0.08, N = 26) was statistically significantly different for alpha = 0.05 from the SecondGame condition (Mdn = −0.007, N = 22; U = 50, p = 0.00, r = 0.86). Along the y axis (SVD2), a Mann-Whitney U test shows that the FirstGame condition (Mdn = 0.13, N = 26) is not statistically significantly different for alpha = 0.05 from the SecondGame condition (Mdn = 0.00, N = 22; U = 287, p = 0.99). The absolute value of r value in Mann-Whitney U test varies from 0 to close to 1. The interpretation values for r commonly in published literature is: 0.10 - < 0.3 (small effect), 0.30 - < 0.5 (moderate effect) and > = 0.5 (large effect).
3.7 Model Evaluation
In this section, we introduce three ways users can evaluate the quality of their ENA models.
3.7.1 Variance Explained
Briefly, variance explained (also called explained variation) refers to the proportion of the total variance in a dataset that is accounted for by a statistical model or set of predictors.
In ENA, to represent high-dimensional vectors in a two-dimensional space, ENA uses either singular value decomposition or means rotation combined with SVD. For each of the reduced dimensions, the variance in patterns of association among units explained by that dimension can be computed.
Here, the first dimension is MR1 and the second dimension is SVD2. The MR1 dimension has the highest variance explained at 32%.
As with any statstical model, greater explained variance does not necessarily indicate a better model, as it may be due to overfitting, but it provides one indicator of model quality.
3.7.2 Goodness of Fit
Briefly, a model’s goodness of fit refers to how well a model fits or represents the data. A model with a high goodness of fit indicates that it accurately represents the data and can make reliable predictions.
In ENA, a good fit means that the positions of the nodes in the space—and thus the network visualizations—are consistent with the mathematical properties of the model. In other words, we can confidently rely on the network visualizations to interpret the ENA model. The process that ENA uses to achieve high goodness of fit is called co-registration. The mathematical details of co-registration are beyond the scope of this chapter and can be found in Bowman et al. [2].
To test a model’s goodness of fit, use ena.correlations. The closer the value is to 1, the higher the model’s goodness of fit is. Most ENA models have a goodness of fit that is well above 0.90.
3.7.3 Close the Interpretative Loop
Another approach to evaluate an ENA model is to confirm the alignment between quantitative model (in our case, our ENA model) and the original qualitative data. In other words, we can return to the original data to confirm that quantitative findings give a fair representation of the data. This approach is an example of what’s called as closing the interpretative loop in Quantitative Ethnography field [1].
For example, based on our visual analysis of the network of “SecondGame.samuel o” in previous section, we are interested in what the lines are in the original data that contributed to the connection between Design.Reasoning and Performance.Parameters.
Let’s first review what “SecondGame.samuel o” ENA network looks like (Fig. 12).
ENA network for Samuel O from SecondGame
To do so, we use view() function and specify required parameters as below.
This is going to activate a window shows up in your Viewer panel. If it is too small to read, you can click on the “Show in new window” button to view it in your browser for better readability.
In the Viewer panel, hover over your cursor on any of the lines that are in bold, a size of 7 lines rectangle shows up, representing that in a moving stanza window of size 7, the referent line (the line in bold) and its preceding 6 lines. The 1 and 0 in Technical.Constraints column and Design.Reasoning column shows where the connections happened (Fig. 13).
A screenshot of the view() function result. The highlighted lines represent lines within the same stanza window
For example, line 2477 Samuel shared his [Design.Reasoning] about “mindful of (the) how one device scores relative to other ones”, to reference back to what Casey said in line 2476 about [Performance.Parameters] “not one source/censor can be the best in every area so we had to sacrifice certain attributes”, as well as what Jackson said in line 2475 about safety as one of the [Performance.Parameters] “when it came to the different attributes, i think that all were important in their own way but i think safety is one of the most important”.
This is a qualitative example of a connection made between Performance.Parameters and Design.Reasoning.
3.8 Using ENA Model Outputs in Other Analyses
It is often useful to use the outputs of ENA models in subsequent analyses. The most commonly used outputs are the ENA points, i.e., set$points. For example, we can use a linear regression analysis to test whether ENA points on the first two dimensions are predictive of an outcome variable, in this case, change in confidence in engineering skills.
The results of this regression analysis show that ENA points are not a significant predictor of the students’ pre-post change in confidence (MR1: t = −0.53, p = 0.60; SVD2: t = 0.46, p = 0.65; Condition: t = −1.36, p = 0.18). The overall model was also not significant (F(3, 43) = 2.01, p = 0.13) with an adjusted r-squared value of 0.06.
Recall that the dataset we are using is a small subset of the full RS.data, and thus results that are significant for the whole dataset may not be for this sample.
4 Ordered Network Analysis with R
This section demonstrates how to conduct an ONA analysis using the ona R package. If you are new to ONA as an analytic technique, Tan et al. [12] provides a more detailed explication of its theoretical and methodological foundations.
Because ONA shares some conceptual and procedural similarities with ENA, you may also want to read the recommended papers from the ENA section [1, 2, 14].
4.1 Install the ONA Package and Load the Library
Install the ona package and load the ona library after installing.
Then, install the other package that is required for ONA analysis.
4.2 Dataset
(Refer to Sect. 3.2 for a detailed description of the dataset used here.)
Load the RS.data dataset.
4.3 Construct an ONA Model
To construct an ONA model, identify which columns in the data to use for the parameters required by the ONA modeling function. The parameters are defined identically in both ENA and ONA; see Sect. 3.3 for detailed explanations.
4.3.1 Specify Units
Select the units as in Sect. 3.3.1.
4.3.2 Specify Codes
Select the codes as in Sect. 3.3.2.
4.3.3 Specify Conversations
The parameter to specify conversations in rENA is called “conversation”; in ONA, the equivalent is called “my_hoo_rules”, where “hoo” is an abbreviation of “horizon of observation.”
Choose the combination of “Condition” column, “GroupName” column, and “ActivityNumber” column to define the conversation parameter.
The syntax to specify conversations using my_hoo_rules in ONA is slightly different from the syntax to specify conversation in ENA, but the conceptual definition is the same.
4.3.4 Specify the Window
Specify a moving stanza window size by passing a numerical value to the window_size parameter.
4.3.5 Specify Metadata
As in ENA, metadata columns can be included if desired. Metadata columns are not required to construct an ONA model, but they provide information that can be used to subset units in the resulting model.
4.3.6 Accumulate Connections
Now that all the parameters are specified, connections can be accumulated. For each unit, the ONA algorithm uses a moving stanza window to identify connections formed from a current line of data (e.g., a turn of talk), or response, to the preceding lines within the window (the ground).
Unlike in ENA, where connections among codes are recorded in a symmetric adjacency matrix, ONA accounts for the order in which the connections occur by constructing an asymmetric adjacency matrix for each unit; that is, the number of connections from code A to code B may be different than the number of connections from B to A.
To accumulate connections, pass the parameters specified to the contexts and accumulate_contexts functions, and store the output in an object (in this case, accum.ona).
4.3.7 Construct an ONA Model
After accumulation, call the model function to construct an ONA model. ONA currently implements singular value decomposition (SVD) and means rotation (MR) to perform dimensional reduction.
To create an ONA model using SVD, pass the accum.ona object to the model function.
When there are two discrete groups to compare, a means rotation can be used, as described in Sect. 3.3.5.
A means rotation is specified using rotate.using ="mean" in the model function. Additionally, the model function expects rotation.params to be a list with two named elements, each containing a logical vector representing the rows of units to be included in each group.
Here, construct the ONA model as shown below.
4.4 Summary of Key Model Outputs
Information about an ONA model is now stored in the R object set.ona.
As in rENA, users can explore the data stored in the object by typing set.ona$ and select items from the drop down list. Here, we briefly explain the top-level items in set.ona$.
4.4.1 Connection Counts
Because ONA accounts for the order in which the connections occur by constructing an asymmetric adjacency matrix for each unit, connection counts from code A to code B and from B to A, as well as self-connections for each code (from A to A) are recorded. Thus, because six codes were included in the model, the cumulative adjacency vector for each unit contains 36 terms (n^2).
4.4.2 Line Weights
To compare networks in terms of their relative patterns of association, researchers can spherically normalize the cumulative adjacency vectors by diving each one by its length. The resulting normalized vectors represent each unit’s relative frequencies of code co-occurrence. In other words, the sphere normalization controls for the fact that different units might have different amounts of interaction or different numbers of activities than others.
In set.ona$connection.counts, the value for each unique co-occurrence of codes is an integer equal or greater than 0, because they represent the directional connection counts between each pair of codes. In set.ona$line.weights, the connection counts are sphere normalized, and so the values are between 0 and 1.
4.4.3 ONA Points
For each unit, ONA produces an ONA point in a two-dimensional space formed by the first two dimensions of the dimensional reduction.
Here, the MR1 column represents the x-axis coordinate for each unit, and the SVD2 column represents the y-axis coordinate for each unit.
4.4.4 Rotation Matrix
The rotation matrix used during the dimensional reduction can be accessed through set.ona$rotation. This is mostly useful when you want to construct an ONA metric space using one dataset and then project ONA points from different data into that space, as in Sect. 5.2.
4.4.5 Metadata
set.ona$meta.data gives a data frame that includes all the columns except for the code connection columns.
4.5 ONA Visualization
Once an ONA model is constructed, ONA networks can be visualize. The plotting function in ONA is called plot, and it works similarly to the same function in ENA.
Before plotting, you can set up several global parameters to ensure consistency across plots. These parameters will be clearer in subsequent sections.
4.5.1 Plot a Mean Network
Mean ONA networks can be plotted for each of the conditions along with their subtracted network.
First, plot the mean network for the FirstGame condition. Use a pipe | > to connect the edges function and the nodes function. Users are only required to specify the weights parameter, as the remaining parameters have default values unless specified otherwise (Fig. 14).
ONA mean network for FirstGame group
Since this is the first ONA network visualization in this chapter, we briefly explain how to read an ONA network.
Node size: In ONA, the node size is proportional to the number of occurrences of that code as a response to other codes in the data, with larger nodes indicating more responses. For example, in this plot, students in the FirstGame condition responded most frequently with discourse about Technical.Constraints.
Self-connections: The color and saturation of the circle within each node is proportional to the number of self-connections for that code: that is, when a code is both what students responded to and what they responded with. Colored circles that are larger and more saturated reflect codes with more frequent self-connections.
Edges: Note that unlike most directed network visualizations, which use arrows or spearheads to indicate direction, ONA uses a “broadcast” model, where the source of a connection (what students responded to) is placed at the apex side of the triangle and the destination of a connection (what students responded with) is placed at its base.
Chevrons on edges: The chevrons point in the direction of the connection. Between any pair of nodes, if there is a bidirectional connection, the chevron only appears on the side with the stronger connection. This helps viewers differentiate heavier edges in cases such as between Technical.Constraints and Data, where the connection strengths from both directions are similar. When the connection strengths are identical between two codes, the chevron will appear on both edges.
Now, plot the mean network for SecondGame (Fig. 15).
ONA mean network for SecondGame group
Then, plot the subtracted network to show the differences between the mean networks of the FirstGame and SecondGame conditions (Fig. 16).
ONA subtracted network showing the differences between FirstGame (red) and SecondGame (blue)
4.5.2 Plot a Mean Network and its Points
Besides plotting the mean network for each condition and the subtracted network, we can also plot the individual units within each condition.
Use set.ona$points to subset the rows that are in each condition and plot the units in each condition as a different color.
The points are specified in the units function. The edges and nodes functions remain the same as above (Figs. 17, 18, 19, and 20).\r\n
ONA points (dots) and their mean point (square) for FirstGame group
ONA mean network for FirstGame group
ONA points (dots) and their mean point (square) for SecondGame group
ONA mean network and points for SecondGame
Plot the subtracted network as follows (Fig. 21).
ONA subtracted network showing the differences between FirstGame (red) and SecondGame (blue)
4.5.3 Plot an Individual Network and its Points
To plot an individual student’s network and ONA point, use set.ona$points.\r\n
Here, we choose the same two units we compared in the ENA analysis (Sect. 3.5.3) (Figs. 22 and 23).
ONA network for a student from FirstGame
ONA network for a student from SecondGame
In this case, both units make relatively strong connections between Design.Reasoning and Data. However, for Unit A (red), the connection is relatively more from Design.Reasoning to Data than the other way around. This indicates that more often this unit responded with Data. In contrast, Unit B (blue) responded more frequently to Data with Design.Reasoning.
A subtracted network can make such differences more salient (Fig. 24).
ONA subtracted network showing the differences between one student from FirstGame (red) and another student from SecondGame (blue)
The connection between Design.Reasoning and Data consists of two triangles, one in blue pointing from Data to Design.Reasoning, the other in red pointing from Design.Reasoning to Data. This indicates that although both units made strong connections between these two codes, the relative directed frequencies are different. Recall that in the ENA subtracted network for the same two units, the connections between Data and Design.Reasoning were basically the same. ONA, by accounting for the order of events, shows that while the undirected relative frequencies were similar, there was a difference in the order in which the two students made the connection.
4.6 Compare Groups Statistically
In addition to visual comparison of networks, ENA points can be analyzed statistically. For example, here we might test whether the patterns of association in one condition are significantly different from those in the other condition.
To demonstrate both parametric and non-parametric approaches to this question, the examples below use a Student’s t test and a Mann-Whitney U test to test for differences between the FirstGame and SecondGame condition.
First, install the lsr package to enable calculation of effect size (Cohen’s d) for the t test.
Then, subset the points to test for differences between the points of the two conditions.
Conduct the t test on the first and second dimensions.
Compute any other statistics that may be of interest. A few examples are given below.
Here, along the x axis (MR1), a two-sample t test assuming unequal variance shows that the FirstGame (mean = −0.05, SD = 0.09, N = 26) condition is statistically significantly different for alpha = 0.05 from the SecondGame condition (mean = 0.06, SD = 0.12, N = 22; t(41.001) = −3.77, p = 0.00, Cohen’s d = 1.1). Along the y axis (SVD2), a two-sample t test assuming unequal variance shows that the FirstGame condition (mean = −1.73, SD = 0.17, N = 26) is not statistically significantly different for alpha = 0.05 from the SecondGame condition (mean = 1,74, SD = 0.17, N = 22; t(45.45) = 0, p = 1.00, Cohen’s d = 0.00).
The Mann-Whitney U test is a non-parametric alternative to the independent two-sample t test.
First, install the rcompanion package to calculate the effect size (r) for a Mann-Whitney U test.
Then, conduct a Mann-Whitney U test on the first and second dimensions.
Compute any other statistics that may be of interest. A few examples are given below.
Here, along the x axis (MR1), a Mann-Whitney U test shows that the FirstGame condition (Mdn = −0.04, N = 26) was statistically significantly different for alpha = 0.05 from the SecondGame condition (Mdn = 0.10, N = 22 U = 130, p = 0.001, r = 0.00). Along the y axis (SVD2), a Mann-Whitney U test shows that the FirstGame condition (Mdn = 0.001, N = 26) is not statistically significantly different for alpha = 0.05 from the SecondGame condition (Mdn = 0.00, N = 22, U = 264, p = 0.66, r = 0.71). The absolute value of r value in Mann-Whitney U test varies from 0 to close to 1. The interpretation values for r commonly in published literature is: 0.10 - < 0.3 (small effect), 0.30 - < 0.5 (moderate effect) and > = 0.5 (large effect).
4.7 Model Evaluation
4.7.1 Variance Explained
Briefly, variance explained (also called explained variation) refers to the proportion of the total variance in a dataset that is accounted for by a statistical model or set of predictors.
In ONA, to represent high-dimensional vectors in a two-dimensional space, ONA uses either singular value decomposition or means rotation combined with SVD. For each of the reduced dimensions, the variance in patterns of association among units explained by that dimension can be computed.
In our example above, since we used means rotation method, the first dimension is labeled as MR1 and the second dimension is labeled as SVD2.The two dimensions in combination explained more than 40% of the variance.
Here, the first dimension is MR1 and the second dimension is SVD2. The two dimensions in combination explained more than 40% of the variance.
As with any statistical model, greater explained variance does not necessarily indicate a better model, as it may be due to overfitting, but it provides one indicator of model quality.
4.7.2 Goodness of Fit
Briefly, a model’s goodness of fit refers to how well a model fits or represents the data. A model with a high goodness of fit indicates that it accurately represents the data and can make reliable predictions.
In ONA, a good fit means that the positions of the nodes in the space—and thus the network visualizations—are consistent with the mathematical properties of the model. In other words, we can confidently rely on the network visualizations to interpret the ONA model. The process that ONA uses to achieve high goodness of fit is called co-registration, the same as the one used in ENA. The mathematical details of co-registration are beyond the scope of this chapter and can be found in Bowman et al. [2].
To test a model’s goodness of fit, use ona::correlations. The closer the value is to 1, the higher the model’s goodness of fit is. Most ENA models have a goodness of fit that is well above 0.90.
4.7.3 Close the Interpretative Loop
Another approach to evaluate an ONA model is to confirm the alignment between quantitative model (in our case, our ONA model) and the original qualitative data. In other words, we can return to the original data to confirm that quantitative findings give a fair representation of the data. This approach is an example of what’s called as closing the interpretative loop in Quantitative Ethnography field [1].
For example, based on our visual analysis of the network of “SecondGame::samuel o” in previous section, we are interested in what the lines are in the original data that contributed to the connection from Performance.Parameters to Design.Reasoning.
Let’s first review what “SecondGame::samuel o” ONA network looks like. Based on the connection direction and strength from Technical.Constraints to Performance.Parameters, we would expect to see more examples of Samuel responded with “Design.Reasoning” to “Performance.Parameters”, than the other way around (Fig. 25).
ONA network for a student from SecondGame
To do so, we use view() function and specify required parameters as below.
This is going to activate a window shows up in your Viewer panel. If it is too small to read, you can click on the “Show in new window” button to view it in your browser for better readability.
In the Viewer panel, hover over your cursor on any of the lines that are in bold, a size of 7 lines rectangle shows up, representing that in a moving stanza window of size 7, the referent line (the line in bold) and its preceding 6 lines. The 1 and 0 in Technical.Constraints column and Design.Reasoning column shows where the connections happened (Fig. 26).
A screenshot of the view() function result. The highlighted lines represent lines within the same stanza window
Notice that here we are viewing the same qualitative example as in Sect. 3.7.3 in ENA. In line 2477 Samuel shared his [Design.Reasoning] about “mindful of (the) how one device scores relative to other ones”, as a response to what Casey said in line 2476 about [Performance.Parameters] “not one source/censor can be the best in every area so we had to sacrifice certain attributes”, as well as what Jackson said in line 2475 about safety as one of the [Performance.Parameters] “when it came to the different attributes, i think that all were important in their own way but i think safety is one of the most important”.
Here, ONA was able to not only capture the occurrence between code Design.Reasoning and Performance.Parameters as ENA did, but also represent the connection direction from Design.Reasoning to Performance.Parameters.
4.8 Using ONA Model Outputs in Other Analyses
As with ENA, the outputs of ONA models can be used as inputs in other statistical models. See Sect. 3.8 for an example using ENA points.
5 Additional Features
In the sections above, we demonstrated how to do an ENA analysis and an ONA analysis. In this section, we show how to project new data into a space constructed with different data. This can be done as long as the same codes are used in both sets.
5.1 Projections in ENA
To project the ENA points from one model into a space constructed with different data, replace the rotation.set parameter of ena.make.set. In the example below, an “expert” model is developed using the SecondGame units and the FirstGame (novice) units are projected into that space. By projecting novice model’s units into expert model’s space, users can interpret the projected novice units’ networks based on the two dimensions defined by expert model’s node positions. In other words, interpreting novice’s networks in the context of experts’ space (Figs. 27 and 28).
Mean network for all units in the expert model
Mean network for novice students projected into expert space
5.2 Projections in ONA
Projection works similarly in ONA (Fig. 29).
Projecting novice data into expert space
6 Discussion
In this chapter, we introduced two techniques, ENA and ONA, for quantifying, visualizing, and interpreting networks using coded data. Through the use of a demonstration dataset that documents collaborative discourse among students collaborating to solve an engineering design problem, we provided step-by-step instructions on how to model complex, collaborative thinking using ENA and ONA in R. The chapter combines theoretical explanations with tutorials, intended to be of aid to researchers with varying degrees of familiarity with network analysis techniques and R. This chapter mainly showcased the standard and most common use of these two tools. The ENA and ONA R packages, akin to other R packages, offer flexibility to researchers to tailor their analyses to their specific needs. For example, users with advanced R knowledge can supply their own adjacency matrices and use ENA or ONA solely as a visualization tool rather than an integrated modeling and visualization tool.
Due to the technical and practical focus of this chapter, we omitted detailed explanations of the theoretical, methodological, and mathematical foundations of ENA and ONA that are crucial for informed, theory-based learning analytics research using these techniques. Consult the Further Reading section for papers that explain these aspects of ENA and ONA in greater detail.
Notes
- 1.
The ENA package also enables use of an infinite stanza window. The infinite stanza window works the same way as a moving stanza window, but there is no limit on the number of previous lines that are included in the window besides the conversation itself. The infinite stanza window is less commonly used in ENA, but is specified as follows: window.size.back = “INF”.
References
Shaffer DW (2017) Quantitative ethnography. Cathcart Press
Bowman D, Swiecki Z, Cai Z, Wang Y, Eagan B, Linderoth J, Shaffer DW (2021) The mathematical foundations of epistemic network analysis. In: Advances in Quantitative Ethnography: Second International Conference, ICQE 2020, Malibu, CA, USA, February 1–3, 2021, Proceedings 2, pp 91–105
Csanadi A, Eagan B, Shaffer DW, Kollar I, Fischer F (2018) When coding-and-counting is not enough: using epistemic network analysis (ENA) to analyze verbal data in CSCL research. Int J Comput-Support Collab Learn 13(4):419–438
Oshima J, Oshima R, Fujita W (2018) A mixed-methods approach to analyze shared epistemic agency in Jigsaw instruction at multiple scales of temporality. J Learn Anal 5(1):10–24. https://doi.org/10.18608/jla.2018.51.2
Bressler DM, Bodzin AM, Eagan B, Tabatabai S (2019) Using epistemic network analysis to examine discourse and scientific practice during a collaborative game. J Sci Educ Technol 28:553–566
Swiecki Z, Ruis AR, Farrell C, Shaffer DW (2020) Assessing individual contributions to collaborative problem solving: a network analysis approach. Comput Hum Behav 104:105876
Prieto LP, Rodríguez-Triana MJ, Ley T, Eagan B (2021) The value of epistemic network analysis in single-case learning analytics: a case study in lifelong learning. In: Advances in quantitative ethnography: second international conference, ICQE 2020, Malibu, CA, USA, February 1–3, 2021, Proceedings 2, Springer International Publishing, pp. 202–217
Zhang S, Gao Q, Sun M, Cai Z, Li H, Tang Y, Liu Q (2022) Understanding student teachers’ collaborative problem solving: Insights from an epistemic network analysis (ENA). Comput Educ 183:104485
Bauer E, Sailer M, Kiesewetter J, Shaffer DW, Schulz C, Pfeiffer J, Gurevych I, Fischer MR, Fischer F (2020) Pre-Service teachers’ diagnostic argumentation: what is the role of conceptual knowledge and epistemic activities?. In: Proceedings of the fifteenth international conference of the learning sciences, pp 2399–2400.
Fernandez-Nieto GM, Martinez-Maldonado R, Kitto K, Buckingham Shum S (2021) Modelling spatial behaviours in clinical team simulations using epistemic network analysis: methodology and teacher evaluation. In: LAK21: 11th international learning analytics and knowledge conference, April, pp 386–396
Phillips M, Trevisan O, Carli M, Mannix T, Gargiso R, Gabelli L, Lippiello S (2023, March) Uncovering patterns and (dis) similarities of pre-service teachers through Epistemic Network Analysis. In: Society for information technology & teacher education international conference. Association for the Advancement of Computing in Education (AACE), pp 1021–1030
Tan Y, Ruis AR, Marquart C, Cai Z, Knowles MA, Shaffer DW (2022, October) Ordered network analysis. In: International conference on quantitative ethnography. Springer Nature Switzerland, Cham, pp 101–116
Fan Y, Tan Y, Raković M, Wang Y, Cai Z, Shaffer DW, Gašević D (2022) Dissecting learning tactics in MOOC using ordered network analysis. J Comput Assist Learn:1–13
Shaffer DW, Collier W, Ruis AR (2016) A tutorial on epistemic network analysis: analyzing the structure of connections in cognitive, social, and interaction data. J Learn Anal 3(3):9–45
Shaffer DW, Arastoopour G (2014) Guide to RSdata.csv sample ENA data set. Madison, WI: Games and Professional Simulations Technical Report 2014-3
Arastoopour G, Shaffer DW, Swiecki Z, Ruis AR, Chesler NC (2016) Teaching and assessing engineering design thinking with virtual internships and epistemic network analysis. Int J Eng Educ 32(3B):1492–1501
Chesler NC, Ruis AR, Collier W, Swiecki Z, Arastoopour G, Shaffer DW (2015) A novel paradigm for engineering education: Virtual internships with individualized mentoring and assessment of engineering thinking. J Biomech Eng 137(2)
Siebert-Evenstone AL, Arastoopour Irgens G, Collier W, Swiecki Z, Ruis AR, Shaffer DW (2017) In search of conversational grain size: Modelling semantic structure using moving stanza windows. J Learn Anal 4(3):123–139
Ruis AR, Siebert-Evenstone AL, Pozen R, Eagan B, Shaffer DW (2019) Finding common ground: a method for measuring recent temporal context in analyses of complex, collaborative thinking. In: Lund K, Niccolai G, Lavoué E, Hmelo-Silver C, Gwon G, Baker M (eds) A wide lens: combining embodied, enactive, extended, and embedded learning in collaborative settings: 13th international conference on Computer Supported Collaborative Learning (CSCL), I, pp 136–143
Kaur A, Kumar R (2015) Comparative analysis of parametric and non-parametric tests. J Comput Math Sci 6(6):336–342
Further Reading
Brohinsky J, Marquart C, Wang J, Ruis AR, Shaffer DW (2021) Trajectories in epistemic network analysis. In Ruis AR, Lee SB (eds) Advances in quantitative ethnography: second international conference, ICQE 2020, Malibu, CA, USA, February 1–3, 2021, Proceedings, Springer, pp 106–121
Gasevic D, Greiff S, Shaffer DW (2022) Towards strengthening links between learning analytics and assessment: challenges and potentials of a promising new bond. Comput Hum Behav:1–7
Shaffer DW (2017) Quantitative ethnography. Cathcart Press
Shaffer DW (2018) Big data for thick description of deep learning. In: Millis K, Long D, Magliano J, Weimer K (eds) Deep learning: multi-disciplinary approaches. Routledge, New York, pp 262–275
Shaffer DW (2018) Epistemic network analysis: understanding learning by using big data for thick description. In: Fischer F, Hmelo-Silver CE, Goldman SR, Reimann P (eds) International handbook of the learning sciences. Routledge, New York, pp 520–531
Shaffer DW, Eagan B, Knowles M, Porter C, Cai Z (2022) Zero re-centered projection: an alternative proposal for modeling empty networks in ENA. In: Wasson B, Zörgő S (eds) Advances in quantitative ethnography: third international conference, ICQE 2021, Virtual Event, November 6–11, 2021, Proceedings, Springer, pp 66–79
Swiecki Z, Lian Z, Ruis AR, Shaffer DW (2019) Does order matter? Investigating sequential and cotemporal models of collaboration. In: Lund K, Niccolai G, Lavoué E, Hmelo-Silver C, Gwon G, Baker M (eds) A wide lens: combining embodied, enactive, extended, and embedded learning in collaborative settings: 13th international conference on Computer Supported Collaborative Learning (CSCL), I, pp 112–119
Wang Y, Swiecki Z, Ruis AR, Shaffer DW (2021) Simplification of epistemic networks using parsimonious removal with interpretive alignment. In Ruis AR, Lee SB (eds) Advances in quantitative ethnography: second international conference, ICQE 2020, Malibu, CA, USA, February 1–3, 2021, Proceedings, Springer, pp 137–151
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Open Access This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.
The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.
Copyright information
© 2024 The Author(s)
About this chapter
Cite this chapter
Tan, Y., Swiecki, Z., Ruis, A.R., Shaffer, D. (2024). Epistemic Network Analysis and Ordered Network Analysis in Learning Analytics. In: Saqr, M., López-Pernas, S. (eds) Learning Analytics Methods and Tutorials. Springer, Cham. https://doi.org/10.1007/978-3-031-54464-4_18
Download citation
DOI: https://doi.org/10.1007/978-3-031-54464-4_18
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-54463-7
Online ISBN: 978-3-031-54464-4
eBook Packages: Computer ScienceComputer Science (R0)