## Abstract

Both classic and newer antimitotics commonly induce a prolonged mitotic arrest in cell culture. During arrest, cells predominantly undergo one of two fates: cell death by apoptosis, or mitotic slippage and survival. To refine this binary description, a quantitative understanding of these cell responses is needed. Herein, we propose a quantitative description of the kinetics of colon carcinoma RKO cell fates in response to different antimitotics, using data from the single cell experiments of Gascoigne and Taylor (2008). The mathematical model is calibrated using the *in vitro* experiments of Gascoigne and Taylor (2008). We show that the time-dependent probability of cell death or slippage is universally identical for monastrol, nocodazole and two different doses of AZ138, but significantly different for taxol. Death and slippage responses across drugs can be characterized by Gamma distributions. We demonstrate numerically that these rates increase with prolonged mitotic arrest. Our model demonstrates that RKO cells exhibit a triphasic response - first, remain in mitosis, then undergo fast and slow transition, respectively- dependent on the length of mitotic arrest and irrespective of cell fate, drug type or dose.

### Similar content being viewed by others

## Introduction

Classic microtubule-targeting drugs such as taxanes and vinca alkaloids constitute a highly successful class of antimitotic drugs, with potent anti-tumor activity in many human solid tumors^{1,2,3,4}. In an effort to reduce the hematological and neuronal toxicity induced by these drugs and thus improve efficacy-to-toxicity ratios, newer antimitotic drugs such as spindle-targeting agents have been recently developed. However, these agents demonstrated limited anti-tumor activity in the clinic^{5,6,7,8,9,10,11,12}. Despite their distinct primary targets, antimitotic drugs disrupt mitotic spindle assembly, activating the spindle assembly checkpoint (SAC), and leading to a prolonged mitotic arrest in 100% of the *in vitro* cells in the study irrespective of the antimitotic drug used^{13}.

Following prolonged mitotic arrest, cancer cells predominantly undergo one of two fates: death in mitosis via intrinsic apoptosis, or slippage out of mitotic arrest following the gradual proteolysis of cyclin B1 and subsequent survival in an abnormal G1 state^{14,15,16,17}. The proportion of cells that undergo each alternative fate and the timing of these events vary significantly between different drugs and cell types^{7,13,14,18,19,20,21,22,23}. Even within identical types of cell cultures or drugs used, cells treated with antimitotics exhibit a considerable degree of heterogeneity in response to prolonged drug exposure^{9,16,24}. Such observations have been reported in multiple single cell studies involving individual cancer cells in culture in the presence of various antimitotic drugs, including paclitaxel and Eg5 kinesin inhibitors.

Additionally, it has been experimentally demonstrated that even though the death in mitosis and mitotic slippage pathways are simultaneously active, they function independently of each other during mitotic arrest^{18,25,26,27,28}. These studies confirmed Gascoigne and Taylor’s proposed “competing pathways model”, where the death in mitosis and mitotic slippage pathways are hypothesized to compete against each other (*i.e*., the fastest process to execute in an individual cell wins)^{13}. The first pathway consists of the activation of cell death pathways, where caspase-dependent cell death signals become stronger in time, simultaneously as cyclin B1 degrades^{13,15,16,18,24,26,29,30,31,32,33,34}. The second pathway involves cells that exit mitosis following a prolonged mitotic arrest, when cyclin B1 is slowly degraded and Cyclin-dependent kinase-1 (Cdk1) activity levels fall below the threshold needed to keep cells in mitosis and thus trigger mitotic exit, despite continued SAC signaling^{2,16,25,26,27,32,34,35,36,37}. For example, in the case of Gascoigne and Taylor’s *in vitro* results on the colon carcinoma RKO cell line, the competing networks model would suggest that cell death signals in RKO cells accumulate faster than cyclin B1 levels degrade. Moreover, these accumulation rates would vary across cells, as implied by the different durations of mitotic arrest^{13}.

The quantitative understanding of the cellular apoptosis and slippage rates and their dependency on the length of mitotic arrest is essential in order to decode and better understand the effect of the molecular mechanisms that govern cellular fate in response to antimitotic therapy. Furthermore, it remains to be elucidated whether any common features in the cellular responses to the different antimitotics characterizing each pathway exist.

In this paper, we propose a quantitative description of the kinetics of colon carcinoma RKO cells in response to the microtubule-targeting agents nocodazole and taxol, and the spindle-targeting Eg5 inhibitors AZ138 and monastrol. We hypothesize that the death in mitosis and mitotic slippage pathways exhibit differential cellular apoptosis and slippage rates depending on the length of mitotic arrest. Our mathematical model is calibrated using the *in vitro* observations of^{13}, wherein time-lapse microscopy data demonstrated prolonged, variable durations of mitotic arrest in RKO cells prior to subsequent cell death or slippage.

Our aim is to provide a quantitative description of the RKO cellular apoptosis and slippage rates in response to distinct antimitotic drugs. By doing so, we report that RKO cells exhibit a triphasic response under prolonged exposure to the different antimitotics, *i.e*., first, remain in mitosis, then undergo fast and slow transition, respectively. This effect is dependent on the length of mitotic arrest and irrespective of cell fate or drug.

We demonstrate numerically that these rates increase with the duration of mitotic arrest within the 72-hour experimental time window. Additionally, given that the cellular fate is known, the hazard rates are identical among the different antimitotic drugs. This result is based on a previously unrecognized fact emerging from our quantitative analysis, *i.e*., that the proportions of RKO cells that survive until time “a” in mitotic arrest and subsequently undergo death in mitosis and mitotic slippage are identical when cells are exposed to nocodazole, AZ138, and monastrol, but significantly different for taxol. Moreover, we demonstrate that RKO cells display a higher hazard of undergoing death in mitosis than mitotic slippage throughout the 72-hour experimental time-course.

Our mathematical model is one of the first studies of its kind to provide the cellular apoptosis and slippage rates and their dependency on the length of mitotic arrest for the death in mitosis and mitotic slippage pathways in the RKO cell line. Overall, our results indicate that RKO cells exhibit a triphasic response curve irrespective of cell fate or antimitotic drug type or dose. We note that for each drug, the highest dose used was the smallest one required to block cell division, see Figs S4, 5 in^{13} Interestingly, our quantitative analysis suggests that the taxol-treated RKO cells display the slowest cell death in mitosis responses across all antimitotic drugs. Despite taxol being the slowest inducer of cell death in mitosis in the RKO cell line as evidenced by its specific hazard function, its slow induction of cell death is not a good measure for predicting its likelihood to induce death in mitosis, as 98% of RKO cells exposed to taxol do undergo death in mitosis following a prolonged mitotic arrest. Further investigations are needed to establish whether this is a concentration-dependent effect.

Based on our results, we formulate hypotheses on the dynamics behind the death in mitosis and mitotic slippage pathways in RKO cells. These have the potential to expand our understanding of the mechanisms which dictate whether a cell dies or survives a prolonged mitotic arrest, if tested in more focused experiments.

## Methods

### Data and modeling calibration

In^{13} RKO cells in culture were continuously incubated with 0.03 µM AZ138, 1 µM AZ138, 100 µM monastrol, 30 ng/mL nocodazole, and 0.1 µM taxol during a 72-hour imaging period (Fig. 1A–E, respectively). We note that these drug concentrations represent equivalent, minimal saturating dosages of antimitotics required to ensure the efficient induction of mitotic arrest and a maximal induction of a 4 N DNA peak, as analyzed by flow cytometry and reported in Fig. S4B in^{13} The times spent in mitosis (Fig. S5A and C in^{13}, red bars), or in mitosis following drug addition and before slippage (Fig. S5A and C in^{13}, blue bars) were subsequently recorded. Therein, “0 min” on the x-axis of the cell fate profiles observed in Fig. S5A and C in^{13} represents the time when cells entered mitosis (K. Gascoigne, personal communication).

In^{13}, data were pooled from recordings performed on individual cells synchronized in early S phase, using a thymidine block. Thymidine was added for 16 hours, before cells were released from the block. Drug medium was subsequently added 4.5 hours later. Imaging using automated time-lapse light microscopy was started at the same time. Images of RKO cells were then collected every 5 minutes for a total duration of 4320 minutes, equivalent to 72 hours (see Fig. 1A in^{13} for a timeline of the setup). Therein, mitosis was defined as the cellular state between nuclear envelope breakdown and the onset of anaphase (Fig. S1 in^{13}). We note that the subsequent cell fates to antimitotic drugs were recorded either in the absence or presence of the pan-caspase inhibitor Boc-D-FMK, as represented in Fig. S5A in^{13} and Supplementary Fig. 1A–P in the current text.

In these experimental findings, fewer than 5% of the total number of RKO cells were reported to have successfully completed mitosis and divided into daughter cells in response to the microtubule-targeting agents nocodazole and taxol, and the spindle-targeting Eg5 inhibitors AZ138 and monastrol (see Fig. 1A–E for representative RKO cell responses). We note that in the absence of any antimitotic drugs, unsynchronized RKO cells are observed to undergo approximately three mitoses during a 72-hour imaging period^{28}. Additionally, since in the experimental setup, the cells were spatially separated, the quantitative live-cell imagining technique employed by^{13} reported individual cell behavior, independent of spatial or global density considerations. As a result, we did not consider an explicit cellular density or a spatial component in our mathematical model. Subsequent results reported below are based on the data reported in Fig. 1A–E and Supplementary Fig. 1A–P. We note that we only considered the most predominant cell fate responses (*i.e*., ≥ 20 cells per cell fate).

### Statistical tests

We chose to focus on the predominant fates experienced by the RKO cells under prolonged antimitotic drug exposure, *i.e*., the fates governed by the death in mitosis and mitotic slippage pathways. To determine any statistically significant differences between the different RKO cell responses under prolonged exposure to the specific antimitotic drugs reported above, we first use the non-parametric Kruskal-Wallis test (or one-way ANOVA test for ranks) for n = 5 independent samples^{38}. These samples correspond to the five RKO populations that undergo death in mitosis following exposure for a 72-hour period to 0.03 µM AZ138, 1 µM AZ138, 100 µM monastrol, 30 ng/mL nocodazole, and 0.01 µM taxol, respectively. We note that this test indicates whether the samples tested originate from the same distribution and identifies whether at least one of these samples is statistically significant. It does not, however, indicate in which sample(s) this dominance occurs.

We then performed pairwise non-parametric Mann-Whitney tests using all possible combinations between the groups of cells that died in mitosis (red bars) or that underwent mitotic slippage (blue bars); n.s., non-significant, *p < 0.01, **p < 0.001, ***p < 0.0001^{38,39}. We note that this test is a nonparametric test that assesses whether two independent samples have similarly ranked distributions. It does not require the assumption of normal distributions. All statistical tests were two-tailed. The Holm-Bonferroni correction for multiple comparisons was used to calculate sequential corrected p-values, with α = 0.01 set as the determined significance threshold for rejecting the null hypothesis of samples having similarly ranked distributions. This procedure is used to control the familywise Type I error rate and reduce the risk of a Type II error as compared to using the simple Bonferroni correction.

### Distribution of times spent in mitotic arrest before dying or slipping out of mitosis

We use the experimental data to obtain the empirical cumulative density function (CDF) for the times spent in mitotic arrest corresponding to the death in mitosis and mitotic slippage pathways (illustrated in Fig. 2A–C). This is done by fitting a kernel smoothing function estimate to the CDF describing the duration of mitotic arrest corresponding to each drug, describing the duration of mitotic arrest reported in Fig. S5A,B in^{13}. The procedure is performed using MATLAB’s “ksdensity” function. This function returns a cumulative density function, based on the sampled data. The amount of time a cell spends in mitotic arrest is thus assumed to be a continuous variable. The empirical CDF corresponding to each drug obtained using the kernel smoothing procedure is illustrated for cells that die in mitosis or slip out of mitosis in Figs. 2B or C, respectively.

### Polynomial fitting

To quantify the transition rates from mitotic arrest to cell death or interphase, we perform a polynomial least-squares fitting to the empirical transition rate derived from the Gamma CDF representing the fraction of RKO cells that either died or slipped after entering mitosis. The fitting procedure is done using MATLAB’s “polyfit”, “polyval” and “polyfix” functions^{40}. The first function returns the coefficients for a polynomial of a user-specified degree, that represents the best fit in the least squares sense for the input data. The second is used to evaluate the fitted polynomials on a prescribed set of gridpoints. It also obtains error estimates in the Root-Mean-Square-Error (RMSE) sense between the approximate and fitted transitions rates from mitotic arrest to death in mitosis and mitotic slippage. The third function computes the coefficients for a polynomial of a user-specified degree, that represents the best fit in the least squares sense for the input data, with the added constraint that the polynomial must pass through a user-specified value at a specific point^{40}. To best describe the corresponding transition rates from mitosis to death in mitosis for the non-taxol and taxol drugs, as well as the transition rate from mitosis to mitotic slippage, we chose to fit piecewise linear polynomials to the empirical data. Using linear polynomials, as opposed to higher-degree polynomials, enables us to easily interpret the modeling results into biologically meaningful observations that could be further tested with more focused experiments.

### Modeling approach

To study the emerging heterogeneity in RKO cell responses to prolonged antimitotic drug exposure, we model the dynamics of the RKO cancer cell population as the following system:

with initial conditions \({\bf{M}}(0)=(\begin{array}{c}{\rm{p}}\\ 1-p\,\end{array}){{\rm{M}}}_{{\rm{t}}{\rm{o}}{\rm{t}}{\rm{a}}{\rm{l}}}.\)

Herein, the vector **M**(a) denotes the mitotic compartment, where the first component denotes the number of cells that are still in mitosis at time “a” and will undergo death in mitosis later on during the experiment. The second component of **M**(a) denotes the number of cells that are still in mitosis at time “a” and will slip out of mitosis into interphase later on during the experiment. The rate of change of **M**(a) with respect to the experimental time course “a” (*i.e*., cell-cycle age) is represented by \(\frac{{\rm{d}}}{{\rm{da}}}.\) The derivative \(\frac{{\rm{d}}}{{\rm{da}}}{\bf{M}}({\rm{a}})\) implies that mitotic cells advance in cell-cycle age as time progresses.

From mitotic arrest, cells transition with time-dependent rate \({{\rm{\alpha }}}_{{\rm{MA}}}({\rm{a}})\) and probability p to intrinsic cell death (*i.e*., apoptosis) or slip out of mitosis into interphase with time-dependent rate α_{MI}(a) and probability 1−p. In doing so, we implicitly assume that the death in mitosis and mitotic slippage pathways are simultaneously active, but mechanistically independent of each other during mitotic arrest. This assumption is supported experimentally by various cancer cell studies^{3,13,14,18,26,27}, and numerically by predictive modeling in^{18}.

The total number of RKO cells exposed to 0.03 µM AZ138, 1 µM AZ138, monastrol, nocodazole, and taxol that undergo either death in mitosis or mitotic slippage, as depicted in Fig. 1A–E, is:

Thus, the initial number of RKO cells arrested in mitosis, corresponding to each drug and cell fate, as evidenced in Fig. 1A–E is: \({\bf{M}}{(0)}_{0.03{\rm{\mu }}\text{M }{\rm{A}}{\rm{Z}}138}=\,(\begin{array}{c}43\\ 43\end{array}),\) \({\bf{M}}{(0)}_{\text{1}{\rm{\mu }}\text{M }{\rm{A}}{\rm{Z}}138}=(\begin{array}{c}85\\ 0\end{array})\), \({\bf{M}}{(0)}_{{\rm{m}}{\rm{o}}{\rm{n}}{\rm{a}}{\rm{s}}{\rm{t}}{\rm{r}}{\rm{o}}{\rm{l}}}=(\begin{array}{c}49\\ 43\end{array}),\) \({\bf{M}}{(0)}_{{\rm{n}}{\rm{o}}{\rm{c}}{\rm{o}}{\rm{d}}{\rm{a}}{\rm{z}}{\rm{o}}{\rm{l}}{\rm{e}}}=(\begin{array}{c}85\\ 0\end{array})\) \({\rm{a}}{\rm{n}}{\rm{d}}\,{\bf{M}}{(0)}_{{\rm{t}}{\rm{a}}{\rm{x}}{\rm{o}}{\rm{l}}}=(\begin{array}{c}98\\ 0\end{array}),\) which yields a drug type- and dose-dependent probability p of undergoing death in mitosis following mitotic arrest of \(\frac{1}{2},\,1,\,\frac{49}{92},\,1,\,{\rm{and}}\,1\)in the case of 0.03 µM AZ138, 1 µM AZ138, monastrol, nocodazole, and taxol, respectively.

The solution of the linear system in Equation (1) is:

In order to determine the hazard functions corresponding to the RKO cells undergoing death in mitosis and slippage, we estimate the proportion of RKO cells that survive until time “a” in mitotic arrest and subsequently undergo death in mitosis or mitotic slippage as the exponentially decaying process:

where the cell death or slippage responses across drugs are characterized by the cell-cycle age “a”-dependent Gamma CDF \({{\rm{F}}}_{{\rm{MA}}}({\rm{a}};\,{\rm{k}},\,{\rm{\theta }})\) or \({{\rm{F}}}_{{\rm{MI}}}({\rm{a}};\,{\rm{k}},\,{\rm{\theta }})\), corresponding to the death in mitosis and mitotic slippage pathways, respectively. Here, the notation “~” represents “is distributed as”.

Each Gamma CDF models the fraction of RKO cells that either die or slip after entering mitosis as a function of time, and is characterized by its corresponding shape kand scale parameters θ (see Fig. 2B,C for the quantification). Herein, \(\overline{{\bf{F}}({\bf{a}})}\) denotes the vector of survival functions corresponding to each pathway, where the survival function is defined as 1-CDF.

To determine \({{\rm{\alpha }}}_{{\rm{MA}}}({\rm{a}})\) and \({{\rm{\alpha }}}_{{\rm{MI}}}({\rm{a}})\), we obtain from Equation (3):

Taking the discrete derivative of Equation (4) yields:

where “\({\rm{\Delta }}a\)” represents the discrete time-step, which is set in our numerical simulations to one minute.

We note that the right-hand side of Equation (5) is equal to \((\begin{array}{c}-\frac{{\rm{d}}}{{\rm{d}}{\rm{a}}}\,{\rm{l}}{\rm{o}}{\rm{g}}[1-{{\rm{F}}}_{{\rm{M}}{\rm{A}}}({\rm{a}};\,{\rm{k}},\,\theta )]\\ -\frac{{\rm{d}}}{{\rm{d}}{\rm{a}}}\,{\rm{l}}{\rm{o}}{\rm{g}}[1-{{\rm{F}}}_{{\rm{M}}{\rm{I}}}({\rm{a}};\,{\rm{k}},\,\theta )]\end{array})\), with \({{\rm{f}}}_{{\rm{MA}}}({\rm{a}};\,{\rm{k}},\,{\rm{\theta }})\) and \({{\rm{f}}}_{{\rm{MI}}}({\rm{a}};\,{\rm{k}},\,{\rm{\theta }})\) representing the Gamma probability distribution functions corresponding to the fraction of RKO cells that either died or slipped after entering mitosis as a function of time “a”, respectively.

### Availability of Materials and Data

All data generated or analyzed during this study are included in this published article.

## Results

### The time-dependent probability cells die in mitosis or slip is identical for all drugs except for taxol

To determine whether the type of antimitotic drug used affects the duration of mitotic arrest in RKO cells, we compare the variable durations of mitotic arrest illustrated in Fig. 1A–E corresponding to cells that either died in mitosis or exited mitosis and returned to interphase. We first analyze the statistical differences between the different RKO cell responses under prolonged exposure to a specific antimitotic drug using the non-parametric Kruskal-Wallis test for n = 5 independent samples. These correspond to the five RKO populations that undergo death in mitosis following exposure for a 72-hour period to 0.03 µM AZ138, 1 µM AZ138, 100 µM monastrol, 30 ng/mL nocodazole, and 0.01 µM taxol, respectively. The observed aggregate difference among the five samples was significant beyond the <0.0001 significance level (data not shown).

We then performed pairwise non-parametric Mann-Whitney tests using all possible combinations between the groups of cells that died in mitosis (red bars) or that underwent mitotic slippage (blue bars); n.s., non-significant, *p < 0.01, **p < 0.001, ***p < 0.0001. RKO cells exposed to taxol exhibit a markedly distinct response to the prolonged taxol exposure, *i.e*., the duration of mitotic arrest induced by taxol in RKO cells is significantly different compared to the durations of the arrest induced by nocodazole, monastrol, or AZ138. This effect achieves statistical significance beyond the < 0.001 level (Fig. 2A). This effect is however, not conserved when RKO cells are exposed to taxol in the presence of the pan-caspase inhibitor (see Supplementary Fig. 2A).

### Cell death and slippage responses across drugs can be characterized by Gamma distributions

As indicated by our statistical analysis, the time-dependent probability cells undergo death in mitosis or mitotic slippage is identical for all drugs except for taxol. To best describe the duration of mitotic arrest cells experience before dying in mitosis or slipping from mitosis and returning to interphase, we chose to represent the RKO cell death or slippage responses across drugs by corresponding Gamma distributions \({\rm{\Gamma }}({\rm{a}};\,{\rm{k}},\,{\rm{\theta }})\). Each distribution represents the fraction of RKO cells that either died or slipped after entering mitosis as a function of time (*i.e*., cell-cycle age “a”), is characterized by its corresponding shape k and scale parameters θ, as illustrated in Fig. 2B,C. The choice of the Gamma distribution to model the duration of mitotic arrest is motivated by this distribution’s asymmetry and right-skewness, and is confirmed by the excellent fit to the empirical data.

Specifically, in Fig. 2B the death in mitosis CDF for the non-taxol drugs can be represented by \({\rm{\Gamma }}(5.91,\,101.15),\)with 95% confidence intervals [5, 6.98] and [85, 120.36] for k and θ, respectively. The RMSE between the empirical CDF (*i.e*., based on the sampled data and obtained by using MATLAB’s “ksdensity” function) and the Gamma fit is equal to \(1.1\cdot {10}^{-2}\). Similarly, the death in mitosis CDF for taxol can be represented by \({\rm{\Gamma }}(4.43,\,212.9),\) with 95% confidence intervals [3.38, 5.8] and [159.9, 283.5] for k and θ, respectively. The RMSE between the empirical CDF and the Gamma fit is equal to \(1.23\cdot {10}^{-2}\). In Fig. 2C, the mitotic slippage CDF can be represented by \({\rm{\Gamma }}(3.55,\,161),\) with 95% confidence intervals [2.66, 4.72] and [118.4, 218.8] for k and θ, respectively. The RMSE between the empirical CDF and the Gamma fit is equal to \(1.2\cdot {10}^{-2}\).

We note that using the Gamma CDF instead of the empirical CDF obtained by using MATLAB’s “ksdensity” function enables us in subsequent simulations to obtain a closed-form expression for the age-dependent transition rates from mitosis to death in mitosis and mitotic slippage, *i.e*., \({{\rm{\alpha }}}_{{\rm{MA}}}({\rm{a}})\) and \({{\rm{\alpha }}}_{{\rm{MI}}}({\rm{a}})\), respectively, as demonstrated in Equations (3–5).

Interestingly, when investigating HCT116 and HT29 colon carcinoma cell line responses to the antimitotic drugs, cell death and slippage responses can also be well characterized by Gamma distributions. For each corresponding cell fate, a distinct Gamma CDF fits the cell profiles independent of the antimitotic drug used, see Supplementary Figs 4A–I and 5A–I for the HT29 and HCT116 cell line responses, respectively. This effect is conserved in the presence of the pan-caspase inhibitor for all three cell lines under study, see Supplementary Figs 3–5.

### The mitotic arrest time does not dictate cell fate

Under prolonged 0.03 µM AZ138 and 100 µM monastrol exposure, RKO cells are slightly more likely to undergo mitotic slippage (blue lines in Fig. 3A,B) rather than death in mitosis (red lines in Fig. 3A,B) for a shorter duration of mitotic arrest, *i.e*., 11.73 and 14.65 hours, respectively. However, for durations longer than 11.73 and 14.65 hours of mitotic arrest in RKO cells exposed to 0.03 µM AZ138 and 100 µM monastrol, respectively, cells are slightly more likely to undergo death in mitosis rather than mitotic slippage. We additionally report the RKO cell fate in the death in mitosis and mitotic slippage pathways under nocodazole and taxol prolonged exposure in the presence of the pan-caspase inhibitor Boc-D-FMK in Supplementary Fig. 3F,G. We also report HCT116 cell fate in the death in mitosis and mitotic slippage pathways under monastrol, nocodazole, AZ138 µM and taxol in the presence and absence of the pan-caspase inhibitor in Supplementary Fig. 1J–N. We did not observe any significant differences in the CDFs corresponding to the death in mitosis and mitotic slippage pathways.

### RKO cells exhibit a triphasic response curve irrespective of cell fate or antimitotic drug

From mitotic arrest, cells undergo death in mitosis (*i.e*., apoptosis) with probability p and age-dependent transition rate \({{\rm{\alpha }}}_{{\rm{MA}}}({\rm{a}})\). Alternatively, they can undergo mitotic slippage and return to interphase with probability 1−p and age-dependent transition rate \({{\rm{\alpha }}}_{{\rm{MI}}}({\rm{a}})\), as shown in Equation (1). We note that these functions increase with prolonged mitotic arrest, irrespective of cell fate or antimitotic drug, see Fig. 4A–D.

We subsequently fit piecewise linear polynomials to the transition rates from mitosis to apoptosis,\({{\rm{\alpha }}}_{{\rm{MA}}}({\rm{a}}),\) for the non-taxol and taxol drugs, as illustrated in Fig. 4A,B, respectively (red bars), and from mitosis to slippage into interphase, as illustrated in Fig. 4C (blue bars). Each labeled “Window” in the Fig. 4 legends corresponds to time period during the mitotic arrest RKO cells undergo one of the following: (i) remain in mitotic arrest (“Window 1” in Fig. 4A–D legend), (ii) fast transition from mitotic arrest to cell death in mitosis, or mitotic slippage (“Window 2” in Fig. 4A–D legend, respectively), or (iii) slow transition from mitotic arrest to cell death in mitosis, or mitotic slippage (“Window 3” in Fig. 4A–D legend, respectively). We note that the fast and slow transitions refer to the slope of fitted linear polynomials corresponding to “Window 2” and “Window 3”, respectively. The slopes corresponding to “Window 2” are bigger than the ones corresponding to “Window 3” throughout Fig. 4A–D.

The piecewise linear polynomials that best describe the hazard functions corresponding to the death in mitosis and mitotic slippage cell responses are reported in Table 1. The RMSE values between the empirically-derived and fitted \({{\rm{\alpha }}}_{{\rm{MA}}}({\rm{a}})\) and \({{\rm{\alpha }}}_{{\rm{MI}}}({\rm{a}})\) corresponding to the death in mitosis and mitotic slippage, respectively, can also be found in Table 1. Overall, RKO cells display a higher hazard of undergoing death in mitosis than mitotic slippage throughout the 72-hour experimental time-course (top red and blue lines in Fig. 4D). Moreover, the transition from mitotic arrest to cell death in mitosis for the non-taxol drugs (“Windows 2–3” in Fig. 4A) is overall faster than the transition from mitotic arrest to mitotic slippage (“Windows 2–3” in Fig. 4C), with a 1.3–1.7-fold difference in the slopes of the piecewise linear polynomials corresponding to the two alternative pathways.

We additionally note that the taxol-treated RKO cells display the slowest cell death in mitosis responses across all antimitotic drugs (Fig. 4B), as evidenced by 1.8–2.8-fold difference in the slopes of the piecewise linear polynomials corresponding to the non-taxol drugs and taxol, illustrated in Fig. 4D. Interestingly, despite taxol being the slowest inducer of cell death in mitosis in the RKO cell line as evidenced by its specific hazard function (illustrated in Fig. 4D), this observation is not a good measure for predicting its likelihood to induce death in mitosis, as 98% of RKO cells exposed to taxol do undergo death in mitosis following a prolonged mitotic arrest (see Fig. 1E herein and Fig. S5A in^{13}). However, this effect might be dose-dependent, as in^{13}, RKO cells were only exposed to 0.1 µM taxol during a 72-hour imaging period.

## Discussion

The mechanisms behind drug-induced prolonged mitotic arrest and cancer cell death using different antimitotic drugs have only recently begun to be elucidated using live quantitative cell imaging^{13,14,19,20,32}. Using live quantitative single cell imaging, several studies have demonstrated that individual cancer cells display widely varying responses to antimitotic drugs. These studies have expanded our understanding of the mechanisms which determine whether a cell dies in mitosis or survives a prolonged mitotic arrest by returning to interphase following exposure to antimitotics.

For example, in^{13}, the authors proposed a model where the two predominant cancer cell fates, *i.e*., mitotic slippage and death in mitosis, are governed by two independent networks. The first network involves the cell-cycle regulator cyclin B1 and its kinase partner Cdk1 as follows: an active anaphase promoting complex APC/C, an E3 ubiquitin ligase, targets cyclin B1 for proteasome degradation past the threshold necessary to maintain sufficient Cdk1 activity and promotes mitotic exit. Cells thus escape mitotic arrest without completing mitosis, which can lead to tetraploidy, senescence, or apoptosis following a subsequent mitosis^{21,41,42}. The second network involves caspase activation and signal accumulation during mitotic arrest, the destabilization of the survivin/XIAP complex, and alterations in the intracellular localization and activation status of Bcl-2 family members^{9,36}.

Several major questions regarding cancer cell fate and cell response to prolonged antimitotic therapies remain unresolved: (1) Does duration of mitotic arrest affect cell fate? (2) What are the cellular apoptosis and slippage rates corresponding to the death in mitosis and mitotic slippage? (3) Are these rates dependent on the length of mitotic arrest? and finally (4) Do any universal features in the cellular responses to the different antimitotics characterizing each pathway exist?^{13,25,26,28,29,33}.

In this paper, we provide the cellular apoptosis and slippage rates and their dependency on the length of mitotic arrest for the death in mitosis and mitotic slippage pathways in the RKO cell line. We demonstrate numerically that these rates increase with the duration of mitotic arrest. Given the cellular fate is known, they are identical among the distinct non-taxol antimitotic drugs whose effect on RKO cell fate was investigated in^{13}. Importantly, this is a previously unrecognized fact which emerges from our quantitative analysis, *i.e*., that the proportions of RKO cells that survive until time “a” in mitotic arrest and subsequently undergo death in mitosis and mitotic slippage are identical when cells are exposed to non-taxol drugs. Moreover, we demonstrate that RKO cells display a higher hazard of undergoing death in mitosis than mitotic slippage throughout the 72-hour experimental time-course. Additionally, our results indicate that RKO cells exhibit a triphasic response curve irrespective of cell fate or antimitotic drug. Interestingly, taxol induces the slowest cell death in mitosis responses across all antimitotic drugs in RKO cells. However, its slow induction of cell death is not a good measure for predicting its likelihood to induce death in mitosis, as experimentally, almost all RKO cells exposed to taxol do undergo death in mitosis following a prolonged mitotic arrest, as reported in^{13,14}.

We now briefly comment upon several aspect emerging from our quantitative modeling results. First, it is intriguing that RKO cells exposed to the microtubule-destabilizing nocodazole, and Eg5-kinesin inhibitors AZ138 and monastrol exhibit triphasic responses to prolonged antimitotic exposure. To the best of our knowledge, this is a previously unrecognized fact. Our statistical analysis indicates that the duration of mitotic arrest induced by these drugs is not statistically different between these drugs, as both cells that die in mitosis or exit mitosis and slip into interphase display the same CDFs, respectively. While death in mitosis and slippage kinetics are highly variable from cell to cell, our results suggests that the microtubule-destabilizing nocodazole and Eg5-kinesin inhibitors AZ138 and monastrol induce the same duration of mitotic arrest in RKO cells corresponding to each pathway, despite the different drug targets and pharmacokinetics. This highlights a potential functional convergence between the different non-taxol antimitotic drugs used in the study with respect to inducing similar distributions of times spent in mitotic arrest before dying or slipping out of mitosis. This intriguing observation merits further experimental investigation. Furthermore, this effect might be dose-independent, as already observed with two different doses of AZ138 where cell death or slippage responses across different doses of the same drug could be characterized by a single Gamma distribution. Further investigations of such dose-response effects are warranted.

Second, RKO cells exposed to taxol exhibit a markedly distinct response to the prolonged taxol exposure, *i.e*., the duration of mitotic arrest induced by taxol in RKO cells is significantly longer compared to the durations of the arrest induced by nocodazole, monastrol, or AZ138. This effect achieves statistical significance beyond the < 0.001 level. Our results suggest that taxol is more efficient at inducing RKO cell death compared to the kinesin-5 inhibitors and nocodazole, but requires a longer duration of mitotic arrest to induce its proapoptotic effect compared to other antimitotic drugs, an observation also pointed out in^{14}. The addition of the pan-caspase inhibitor to the taxol-treated cells rescues some RKO cells from death in mitosis, however we did not observe a statistically significant change between the death in mitosis CDFs corresponding to the taxol-treated cells in the presence or absence of the caspase inhibitor.

Third, our results indicate that the fraction of RKO cells that either die or slip after entering mitosis following continued exposure to nocodazole, monastrol, AZ138, and taxol can be well-approximated by Gamma distributions. Specifically, our results indicate that the shape parameter k of the Gamma distributions corresponding to the fraction of RKO cells that die in mitosis under non-taxol and taxol exposure, or slip back into interphase is 5.91, 4.43, or 3.95, respectively. Interestingly, this suggests the existence of a sequence of approximately six or four independent, exponentially distributed random variables, each modeling an event responsible for inducing RKO cell death by the non-taxol drugs, and taxol, respectively - see^{43} for a theoretical proof. In the case of the death in mitosis pathway, such a sequence could involve in chronological order events such as the activation of executive caspases, Bcl-xL expression levels, the depletion of the anti-apoptotic protein Mcl-1, increased microtubule stabilization leading to interference with cellular trafficking and microtubule-mediated cellular transport, and sequestration of Bax/Bak sufficient to trigger Mitochondrial Outer Membrane Permeabilization (MOMP). In the case of the mitotic slippage pathway, our results also suggest the existence of four independent exponentially distributed random variables responsible for inducing mitotic slippage and survival of the RKO cells in interphase. These events could, for example, be correlated with cyclin B1 level degradation, or the prolonged activation of Cdk1. Interestingly, when analyzing the HCT116 and HT29 colon carcinoma data in comparison to the RKO data, the shape parameters corresponding to the Gamma fits for the HT29 data were bigger than the ones corresponding to the Gamma fits for the RKO or HCT116 data (double-digit versus single-digit numbers). This suggests that the death in mitosis and mitotic slippage pathways for the HT29 cells could be governed by a bigger number of molecular events compared to the RKO or HCT116 cells.

Fourth, our results indicate that RKO cells exhibit a triphasic response curve irrespective of cell fate or antimitotic drug. To the best of our knowledge, this observation was previously unrecognized. Interestingly, RKO cells remain in mitotic arrest for periods of time shorter than 1.5 hours, then undergo a fast transition from arrest to death in mitosis or to mitotic slippage and return in interphase as long as the duration of mitotic arrest is shorter than 24 hours. If cells continue to remain in mitotic arrest for more than 24 hours, their subsequent transition to cellular death or slippage is slower compared to the first phase of transition.

In the current work, we provide an *in silico* modeling framework for studying the emerging heterogeneity in the response of the colon carcinoma RKO cell line to antimitotic drugs. Our *in silico* quantitative approach incorporates experimental results and uses mathematical models in order to better inform *in vitro* phenomena. Our modeling framework will serve as a basis for future studies of cancer cell heterogeneity *in vitro* of more complex responses in the presence of antimitotic drugs of both apoptosis-resistant and apoptosis-sensitive cell lines other than the colon carcinoma RKO, HCT116 and HT29 cell lines.

## References

Jordan, M. A. & Wilson, L. Microtubules as a target for anticancer drugs.

*Nat Rev Cancer.***4**(4), 253–65 (2004).Weaver, B. A. How Taxol/paclitaxel kills cancer cells.

*Mol Biol Cell.***25**(18), 2677–81 (2014).Gascoigne, K. E. & Taylor, S. S. How do anti-mitotic drugs kill cancer cells?

*J Cell Sci.***122**, 2579–85 (2009).Perez, E. A. Microtubule inhibitors: Differentiating tubulin-inhibiting agents based on mechanisms of action, clinical activity, and resistance.

*Mol Cancer Ther.***8**(8), 2086–95 (2009).Jackson, J. R., Patrick, D. R., Dar, M. M. & Huang, P. S. Targeted anti-mitotic therapies: can we improve on tubulin agents?

*Nat Rev Cancer.***7**(2), 107–17 (2007).Katayama, H. & Sen, S. Aurora kinase inhibitors as anticancer molecules.

*Biochim Biophys Acta.***1799**(10-12), 829–39 (2010).Shi, J., Zhou, Y., Huang, H. C. & Mitchison, T. J. Navitoclax (ABT-263) accelerates apoptosis during drug-induced mitotic arrest by antagonizing Bcl-xL.

*Cancer Res.***71**(13), 4518–26 (2011).Marzo, I. & Naval, J. Antimitotic drugs in cancer chemotherapy: promises and pitfalls.

*Biochem Pharmacol.***86**(6), 703–10 (2013).Chan, K. S., Koh, C. G. & Li, H. Y. Mitosis-targeted anti-cancer therapies: where they stand.

*Cell Death Dis.***3**, e411 (2012).Rath, O. & Kozielski, F. Kinesins and cancer.

*Nat Rev Cancer.***12**(8), 527–39 (2012).McInnes, C. & Wyatt, M. D. PLK1 as an oncology target: current status and future potential.

*Drug Discov Today.***16**(13-14), 619–25 (2011).Purcell, J. W.

*et al*. Activity of the kinesin spindle protein inhibitor ispinesib (SB-715992) in models of breast cancer.*Clin Cancer Res.***16**(2), 566–76 (2010).Gascoigne, K. E. & Taylor, S. S. Cancer cells display profound intra- and interline variation following prolonged exposure to antimitotic drugs.

*Cancer Cell.***14**(2), 111–22 (2008).Huang, H. C., Shi, J., Orth, J. D. & Mitchison, T. J. Evidence that mitotic exit is a better cancer therapeutic target than spindle assembly.

*Cancer Cell.***16**(4), 347–58 (2009).Blagosklonny, M. V. Mitotic arrest and cell fate: why and how mitotic inhibition of transcription drives mutually exclusive events.

*Cell Cycle.***6**(1), 70–4 (2007).Galán-Malo, P.

*et al*. Cell fate after mitotic arrest in different tumor cells is determined by the balance between slippage and apoptotic threshold.*Toxicol Appl Pharmacol.***258**(3), 384–93 (2012).Tao, W.

*et al*. Induction of apoptosis by an inhibitor of the mitotic kinesin KSP requires both activation of the spindle assembly checkpoint and mitotic slippage.*Cancer Cell.***8**(1), 49–59 (2005).Huang, H. C., Mitchison, T. J. & Shi, J. Stochastic competition between mechanistically independent slippage and death pathways determines cell fate during mitotic arrest.

*PLoS One.***5**(12), e15724 (2010).Orth, J. D.

*et al*. Quantitative live imaging of cancer and normal cells treated with Kinesin-5 inhibitors indicates significant differences in phenotypic responses and cell fate.*Mol Cancer Ther.***7**(11), 3480–9 (2008).Shi, J., Orth, J. D. & Mitchison, T. Cell type variation in responses to antimitotic drugs that target microtubules and kinesin-5.

*Cancer Res.***68**(9), 3269–76 (2008).Shi, J. & Mitchison, T. J. Cell death response to anti-mitotic drug treatment in cell culture, mouse tumor model and the clinic.

*Endocr Relat Cancer.***24**(9), 83–96 (2017).Colin, D. J., Hain, K. O., Allan, L. A. & Clarke, P. R. Cellular responses to a prolonged delay in mitosis are determined by a DNA damage response controlled by Bcl-2 family proteins.

*Open Biol.***5**(3), 140156 (2015).Hain, K. O., Colin, D. J., Rastogi, S., Allan, L. A. & Clarke, P. R. Prolonged mitotic arrest induces a caspase-dependent DNA damage response at telomeres that determines cell survival.

*Sci Rep.***6**, 26766 (2016).Blajeski, A. L., Phan, V. A., Kottke, T. J. & Kaufmann, S. H. G(1) and G(2) cell-cycle arrest following microtubule depolymerization in human breast cancer cells.

*J Clin Invest.***110**(1), 91–9 (2002).Bennett, A.,

*et al*Inhibition of Bcl-xL sensitizes cells to mitotic blockers, but not mitotic drivers.*Open Biol*.**6**(**8**) (2016).Sloss, O., Topham, C., Diez, M. & Taylor, S. Mcl-1 dynamics influence mitotic slippage and death in mitosis.

*Oncotarget.***7**(5), 5176–92 (2016).Topham, C. H. & Taylor, S. S. Mitosis and apoptosis: how is the balance set?

*Curr Opin Cell Biol.***25**(6), 780–5 (2013).Topham, C.

*et al*. MYC Is a Major Determinant of Mitotic Cell Fate.*Cancer Cell.***28**(1), 129–40 (2015).Bekier, M. E., Fischbach, R., Lee, J. & Taylor, W. R. Length of mitotic arrest induced by microtubule-stabilizing drugs determines cell death after mitotic exit.

*Mol Cancer Ther.***8**(6), 1646–54 (2009).Brito, D. A. & Rieder, C. L. Mitotic checkpoint slippage in humans occurs via cyclin B destruction in the presence of an active checkpoint.

*Curr Biol.***16**(12), 1194–200 (2006).Choi, H. J., Fukui, M. & Zhu, B. T. Role of cyclin B1/Cdc2 up-regulation in the development of mitotic prometaphase arrest in human breast cancer cells treated with nocodazole.

*PLoS One.***6**(8), e24312 (2011).Choi, H. J. & Zhu, B. T. Role of cyclin B1/Cdc2 in mediating Bcl-XL phosphorylation and apoptotic cell death following nocodazole-induced mitotic arrest.

*Mol Carcinog.***53**(2), 125–37 (2014).Díaz-Martínez, L. A.

*et al*. Genome-wide siRNA screen reveals coupling between mitotic apoptosis and adaptation.*EMBO J.***33**(17), 1960–76 (2014).Eichhorn, J. M., Kothari, A. & Chambers, T. C. Cyclin B1 overexpression induces cell death independent of mitotic arrest.

*PLoS One.***9**(11), e113283 (2014).Bah, N.

*et al*. Bcl-xL controls a switch between cell death modes during mitotic arrest.*Cell Death Dis.***5**, e1291 (2014).Barillé-Nion, S., Bah, N., Véquaud, E. & Juin, P. Regulation of cancer cell survival by BCL2 family members upon prolonged mitotic arrest: opportunities for anticancer therapy.

*Anticancer Res.***32**(10), 4225–33 (2012).Eichhorn, J. M., Sakurikar, N., Alford, S. E., Chu, R. & Chambers, T. C. Critical role of anti-apoptotic Bcl-2 protein phosphorylation in mitotic death.

*Cell Death Dis.***4**, e834 (2013).Lowry, R. VassarStats: Website for Statistical Computation. http://vassarstatsnet/kw5html (1998).

Lowry, R. VassarStats: Website for Statistical Computation. http://vassarstatsnet/utesthtml (1998).

Mjaavatten, A. MATLAB function polyfix(x,y,n,xfix,yfix,xder,dydx). https://wwwmathworkscom/matlabcentral/fileexchange/54207-polyfix-x-y-n-xfix-yfix-xder-dydx- (2015).

Zasadil, L. M.

*et al*. Cytotoxicity of paclitaxel in breast cancer is due to chromosome missegregation on multipolar spindles.*Sci Transl Med.***6**(229), 229ra43 (2014).Zasadil, L. M.

*et al*. High rates of chromosome missegregation suppress tumor progression but do not inhibit tumor initiation.*Mol Biol Cell.***27**(13), 1981–9 (2016).Ross, S. A First Course in Probability (8th Edition) 254–255 (Pearson, 2010).

## Acknowledgements

The work of AL was supported by the King Abdullah University of Science and Technology (KAUST) baseline and start-up funds (BAS/1/1648-01-01 and BAS/1/1648-01-02). The work of DAB was partially supported by the Intramural Research Program of the National Institutes of Health, Center for Cancer Research, National Cancer Institute as part of a seed grant from the UMD-NCI Partnership for Cancer Technology. The work of DL was supported in part by the National Science Foundation under Grant Number DMS-1713109 and the Jayne Koskinas Ted Giovanis Foundation. The funders had no role in the study design, data collection and analysis, decision to publish, or preparation of the manuscript.

## Author information

### Authors and Affiliations

### Contributions

A.L., D.-A.B. and D.L. analyzed the data, developed the mathematical models and performed simulations. A.L., D.-A.B. and D.L. wrote the paper. All authors reviewed the manuscript.

### Corresponding author

## Ethics declarations

### Competing Interests

D.-A.B. is an employee of Boehringer Ingelheim. A.L. and D.L. declare no competing interests.

## Additional information

**Publisher's note:** Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

## Electronic supplementary material

## Rights and permissions

**Open Access** This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/.

## About this article

### Cite this article

Lorz, A., Botesteanu, DA. & Levy, D. Universal response in the RKO colon cancer cell line to distinct antimitotic therapies.
*Sci Rep* **8**, 8979 (2018). https://doi.org/10.1038/s41598-018-27267-7

Received:

Accepted:

Published:

DOI: https://doi.org/10.1038/s41598-018-27267-7

- Springer Nature Limited