Using Diary Studies to Evaluate Railway Dispatching Software
- 3.1k Downloads
In this paper, we present the application of User Diaries in the context of connection dispatching. Connection dispatching is a field with quickly rising requirements which also affect the used dispatching support software. The usage of User Diaries will be motivated for this specific domain. The diary will briefly be presented as well as the results of the study. We will point out the advantages and disadvantages using User Diaries in the given context.
KeywordsDiary Study Transportation Company Conflict Detection Prototype Software Number Search
In the field of connection dispatching currently no specifically adapted software is used. Since connection dispatching has an impact on the quality of service for the customers and thus has a high visibility for them, its importance rises. Also, traffic contracts contain rules for connection assurance which are often linked to penalties in case of not achieving them. This increases the importance of connection dispatching and therefore also the requirements for the connection dispatcher. That is why a new prototype software was developed to better support dispatchers in the field of connection dispatching.
Before integrating the prototype software in the daily working environment of the dispatchers, a field study should be conducted to prove its suitability, to further improve and better adapt it to the users’ needs. During this field study, the dispatchers should be involved in evaluating this prototype software. They should test it during their work and state their opinion and improvement proposals, if any.
To get detailed and diversified information about the usage of the prototype, three evaluation methods were carried out in a row: First, Diary Studies were conducted. Then an Observation followed before ending the evaluation process with a Focus Group. In this paper, we will concentrate on the Diary Studies in the context of a field study with a prototype software for connection dispatching.
The focus of this paper will be on describing the evaluation method of Diary Studies and the reflection of its suitability for use in the field of connection dispatching rather than on the prototype software as such.
So, first an overview over related work in the field of connection dispatching will be given in Sect. 2. In Sect. 3, current problems and our motivation for conducting a field test using Diary Studies will be presented. A description of the methodology can be found in Sect. 4. The results are presented in Sect. 5 and discussed in Sect. 6. In Sect. 7, a conclusion will be drawn.
2 Related Work
Only the studies of  are real field studies, meaning that dispatchers used the software during their everyday work. All other references mentioned are rather laboratory studies. Although the study was conducted at their work place, the dispatchers had to handle a specific scenario created for the test. Only the methods Questionnaires, Interviews and Observations are dealt with in detail in these papers. Reference  only mentions the methods Collegial Verbalisation and Interactive Diary, but no further details about method, procedure or results. That is why, in this paper, we will further concentrate on Questionnaires, Interviews and Observations when comparing and discussing User Diaries in Sect. 3.2.
The second focus is past and ongoing research concerning the detection and solution of connection conflicts. Here, the focus is often on the optimization of (traveler) delays as in [5, 6, 7]. Reference  developed a system for connection dispatching which is also based on customer delay, but interacts with an infrastructural counterpart. Reference  present a system which focused on the traveler’s side of connection dispatching including a smart phone prototype application. In [10, 11], a modular dispatching support system for connection dispatching including an evaluation for connection conflicts and solutions exceeding classical waiting strategies is presented.
3 Problem Description
For a (railway) transportation company, connection dispatching is one of the dispatching tasks with the most influence on their customers and thus has a high visibility. Apart from travelers expecting more sophisticated solutions in traffic management, also, traffic contracts contain rules for connection assurance. This makes connection dispatching an important task with increasing requirements for the connection dispatcher.
Historically, connection dispatching in the German railway environment was performed by the infrastructure company, but has been assigned to the transportation company as fulfilling contractor of the traveler. First, the task was performed by staff also responsible for personnel and vehicle dispatching whereas today dedicated positions for connection dispatching and traveler information are being created.
The task of connection dispatching comprises the surveillance of connections, conflict detection, conflict resolution and customer information. As the requirements grow, the dispatcher needs to find better solutions for more connection conflicts on a dense schedule in shorter periods of time. For this, he increasingly relies on software systems that should on one hand provide the necessary information and on the other hand need to be user friendly and easy to use such that the connection dispatcher can easily gather information about the connection conflicts which currently require dedicated attention.
3.1 Connection Dispatching Software
To address the problem described in Sect. 3, a prototype software to visualize relevant information in connection dispatching has been developed in cooperation with a German railway company . An example screen shot is given in Fig. 1. The software has been designed such that the dispatcher is able to gather information about many connections at a glance. For this, a matrix interface is used on which the feeders are arranged vertically and the distributors horizontally. Whenever an interchange between a feeder and a distributor exists, the corresponding cell in the matrix contains information regarding the connection status, otherwise it is left blank. Lines or columns without any connection relation are hidden.
Another interface, shown in Fig. 2, enables the user to reduce the displayed information providing a greater (time-wise and regional) overview of the current situation.
The software first has been developed for a simulation environment, the Eisenbahnbetriebsfeld Darmstadt (EBD) , to be tested by dispatchers . Considering the results of the evaluation in the EBD  and an expert evaluation using the IsoMetrics Questionnaire  and Cogintive Walkthrough , the user interface of the existing software was adapted and improved such that a first test in the field could be performed. The aim was to obtain information about the usability of the software interface in the field.
3.2 Evaluation in the Context of Dispatching
When deciding on an appropriate evaluation method to gain information about the usability of the software interface, we first concentrated on the three evaluation methods mentioned in Sect. 2 – Questionnaires, Interviews and Observations.
We started with describing significant boundary conditions and defined evaluation criteria of the study (cf. ). First, detailed information from the dispatchers to learn more about how to further improve the software and to better adapt it to the requirements of the dispatchers had to be obtained. Second, detailed data about the opinion of the users directly from the user in the context of using the prototype over a longer period of time had to be gained. Moreover, due to time limitations and the long distance to the locations of the field test, a way which allows the dispatchers to state their opinion without the test leader being on-site all the time had to be found.
Since Observation does not allow for asking the user about their opinion, this method could not be used. Interviews could not be applied as well because they should ideally be conducted in a face-to-face situation which was not compatible with the above-mentioned time limitations. Also, personnel restrictions on the dispatchers’ side ruled out the application of interviews via telephone or internet, since it is not possible to relieve them from service to conduct an interview with a duration of one or two hours.
Questionnaires were not deemed suitable either since the objective was to obtain information about the usage of the prototype software on a daily basis, including detailed information about every operation carried out. So, the dispatchers would have been obliged to fill in questionnaires every day which can be very tedious.
detailed data about the opinion of the user,
directly from the user,
in the context of using the prototype,
over a longer period of time,
which also considers time limitations, long distances and personnel limitations and
to obtain information about every operation they carried out with the prototype.
Only User Diaries met all these requirements and were therefore chosen for this field test.
In this section, we will first present general aspects of Diary Studies. Then, we will concentrate on the structure and the layout of the User Diaries for the aforementioned prototype software. Finally, we continue with the procedure of conducting this method.
4.1 Introduction to User Diaries
context/situation of use,
frequency of use,
A huge disadvantage is the dependency on the compliance of the user to fill in the User Diary. The initiative and motivation of the users are decisive for the amount of entries and the degree of detail. Furthermore, entries are very subjective, exaggerations are possible and the data cannot be verified by the test leader. Moreover, the users are on their own during the evaluation. A functioning prototype is needed [18, 20].
Advantages of this method are its applicability during a longer period of time and at different, distant locations. No special resources or equipment are needed, so it is cost-efficient to use [18, 19].
4.2 Structure and Layout
The design and the degree of standardization of the User Diary depends on the specific context. The objective is to allow the user to fill it in rather quickly and easily. For the presented purpose, half checkboxes and half text fields are used.
The structure of the diaries is as follows: First, a general introduction to using the diary and about the procedure of conducting the evaluation is presented. Then, a general introduction into the prototype software follows. Subsequently, a short demographic questionnaire gathering age, working place, working function and professional experience is interposed. Then, the actual diary starts with an example on how to fill in the columns and rows of the diary followed by about 20 pages of diary. Every page contains three rows for describing one operation each. At the end of the diary, there are two pages for stating general remarks about the prototype software and one page with contact data of the test leader.
On the actual diary page, we asked for time, current workload, delay of entry (if any), dispatching actions, reasons for dispatching actions, how many and which dispatching decisions were made and the functions, program views and auxiliaries used.
training power users in using the prototype (one per working place)
training power users in using the User Diaries (one per working place)
introduction to the prototype software for all participating dispatchers
introduction to the User Diaries for all participating dispatchers
filling in the User Diaries
At first, power users were trained on the software and on the User Diaries to cope with one major disadvantage of the diary studies, in particular that users are dependent on themselves (step 1 and 2). Moreover, training power users was helpful to find last bugs in the software and to specifically adapt the User Diary and especially the examples used to the needs of the dispatchers.
Then, an introductory event for the dispatchers participating in the evaluation was organized to train them in using the software and the diary (step 3 and 4). All functions and program views were introduced and the structure and layout of the User Diary were explained to them. Moreover, the example for filling in the User Diary was discussed in detail.
Then, the dispatchers had about two weeks for freely exploring the prototype software, to try all the functions and program views and to get used to it. This aimed at allowing the dispatchers to establish a work process with the software (step 5).
In the following three weeks, the dispatchers filled in the user diary (step 6) before sending them back to the test leader for evaluation. The dispatchers were asked to use the User Diary for at least three times, if possible in a row.
In this section, results of the User Diaries will be presented briefly. The focus will be on general results relevant to the usage and the situation of usage which are important for the discussion in Sect. 6, to reflect the suitability of the prototype.
In total, 91 entries were handwritten by the nine participating dispatchers. These document 471 dispatching decisions, such as dispatching connections or finding alternative trains. In 63.95 %, the current situation was described as calm, 27.91 % as medium – in-between calm and stressful – and in 8.14 % as stressful. 61.54 % of the entries were registered directly after the operation, whereas 38.46 % were delayed by about 12 min in average. The time shifted entries consisted of all entries in stressful situations and half of the entries in medium situations.
Since going into detail about operations or improvement proposals requires a detailed knowledge of the prototype software, only a few examples according to operations and program views will be given.
The most frequently used function (35.8 %) is the filter adjustment function which can be seen on the left hand side in Fig. 1. The train number search function which can be seen in both figures, once without a number (cf. Fig. 1) and once with a number (cf. Fig. 2) in the upper area of the program, was used in 13.0 % of cases. The least used function was the sorting function which can be seen in Fig. 2 and which was only used in 3.25 % of cases. Concerning the program views, in 85.44 % of cases they used the standard matrix view which can be seen in Fig. 1. The program view with reduced displayed information as shown in Fig. 2 was used only in 9.70 % of cases. The third program view which shows the already dispatched trains was the least used one (4.85 %).
Summarizing, dispatching actions and the reasons for conducting them were comparable or even identical to the ones foreseen when designing the prototype. So, the use cases which aided in maintaining a clear focus during the design process are identical to the usage patterns of the dispatchers in their everyday work.
Unfortunately, different problems arose during the field test. As stated in Sect. 5, the 9 dispatchers participating in the study wrote 91 entries in total which were often redundant. So, the variance of the data was rather small.
Moreover, only 8.14 % of the entries were done in stressful situations, all of them later on with a certain delay. The problem is that during normal operations, dispatchers do not face a high workload. It increases dramatically when disturbances occur which is also when the dispatcher starts to heavily use dispatching support software. It can be stated that stressful situations are the most important and interesting ones, but in these situations, dispatchers do not seem to be able to fill in the User Diary. Thus, data from stressful situations will be reported in the User Diary with a certain delay as stated in Sect. 5. These are the most revealing and insightful situations for improving the prototype and to better adapt it to the dispatchers’ operational requirements. As we get only delayed entries in these situations, the question arises if the provided data is still complete, and thus, whether we can rely on the time-shifted entries. The dispatchers might not remember every operation they carried out with the prototype software as well as every detail. The reliability and the working environment for stressful situations are issues for further research using User Diaries in dispatching contexts.
Nonetheless, we gained a lot of useful and valuable information for improving the software, especially regarding the most used functions and views. It could be shown that, e.g., the sorting function which can be seen in Fig. 2 and which was explicitly required by an expert panel during the design process was not intensely used. The same is valid for the program view with reduced displayed information as shown in Fig. 2. This view was later implemented because it seemed to be necessary to provide the dispatchers a view which allows to have a better overview over the current situation by reducing information, but it was not intensively used either. It could be clearly proven that dispatchers prefer the standard matrix view which can be seen in Fig. 1. Moreover, it became obvious that we had to concentrate on improving the filter adjustment function which can be seen on the left hand side in Fig. 1 because of the frequent use to better support dispatchers and also to save time during operations.
Also, valuable hints for the Observation we conducted after the User Diaries were gained. It was much easier to decide on a specific focus for the Observation since we could identify the most used functions and program view with the help of the User Diaries. Moreover, the entries and especially the remarks gave us hints about problems that arose while using the prototype which we could concentrate on during the Observation.
The process of conducting Diary Studies and also the structure and the layout of the User Diary has proven of value. There were no further inquiries regarding the use of the diary and the users filled it in a correct manner. When having a closer look at the remarks given by the dispatchers, we can conclude that the dispatchers seem to feel quite comfortable about using the diary and the prototype.
User Diaries are useful to obtain meaningful information about a prototype software in the described context. Nevertheless, weaknesses of the method, especially regarding stressful situations, could be discovered during our study. To overcome these weaknesses, a carefully considered combination with other methods needs to be developed. Further investigation how Diary Studies can be adapted better to the specific situation of dispatchers and how to combine them with other methods is necessary.
We successfully used the evaluation method of Diary Studies in a field study with connection dispatchers. A lot of insightful data could be gathered which provided helpful input for further improving and better adapting the prototype software to the needs of the users, but also to decide on a specific focus for the following evaluation methods. Thus, User Diaries have proven to be useful in the context of evaluating dispatching software regarding the standard working environment.
Nevertheless, further field tests using the evaluation method of Diary Studies need to be conducted to prove its suitability for use in similar working environments and to further investigate the problematic stressful working times. This method can either specifically be adapted to the specific environments. Otherwise, an alternative evaluation method fulfilling all the criteria we were looking for in this field study with dispatchers needs to be developed.
Using the results of the User Diaries presented in this paper, our prototype software will be developed further and improved to better support dispatchers in doing their work.
- 2.Isaksson-Luttemann, G., Kauppi, A., Andersson, A.W., Sandblad, B., Erlandsson, M.: Operative tests of a new system for train traffic control. Rail Human Factors around the World: Impacts on and of People for Successful Rail Operations, pp. 424–433 (2009)Google Scholar
- 3.Isaksson-Luttemann, G.: Future Train Traffic Control: Development and deployment of new principles and systems in train traffic control. Licentiate thesis, Uppsala University, Uppsala (2012)Google Scholar
- 4.Wikström, J., Kauppi, A., Hellström, P., Andersson, A.W., Sandblad, B.: Train traffic control by re-planning in real-time. In: Allan, J., Brebbia, C., Hill, R., Sciutto, G., Sone, S. (eds.) Computers in Railways IX. Computers in Railways IX, vol. 74, pp. 733–742. WIT Press, Wessex (2004)Google Scholar
- 6.Kliewer, N.: Mathematische optimierung zur unterstützung kundenorientierter disposition im schienenverkehr. In: Chamoni, P. (ed.) Selected Papers of the International Conference on Operations Research. Operations research proceedings, vol. 2001, pp. 473–480. Springer, Heidelberg (2002)Google Scholar
- 7.Suhl, L., Biederbick, C., Kliewer, N.: Design of customer-oriented dispatching support for railways. In: Voss, S., Daduna, J.R. (eds.) Computer-aided scheduling of public transport. Lecture Notes in Economics and Mathematical Systems, vol. 505, pp. 365–386. Springer, Berlin (2001)CrossRefGoogle Scholar
- 8.Kurby, S.: Makroskopisches Echtzeitdispositionsmodell zur Lösung von Anschlusskonflikten im Eisenbahnbetrieb. Zugl. Dissertation an der TU Dresden, Technische Universität Dresden, Dresden (2012)Google Scholar
- 9.Scheier, B., Schöne, S., Dietsch, S.: Assistenz zur anschlusssicherung im intermodalen verkehr mittels echtzeitdaten. ZEV rail Glasers Annalen 138(6–7), 224–230 (2014)Google Scholar
- 10.Stelzer, A., Oetting, A.: Konzeption einer Konfliktlösung einschließlich einer Bewertungsmethode für die Anschlussdisposition. In: Fakultä Verkehrswissenschaften Friedrich List, (ed.): 24. Verkehrswissenschaftlichen Tage 2014, Fakultät Verkehrswissenschaften Friedrich List (2014)Google Scholar
- 11.Oetting, A., Stelzer, A.: Conflict resolution in connection dispatching. In: Hansen, I.A., Tomii, N., Hirai, C. (eds.) 6th International Conference on Railway Operations Modelling and Analysis (2015) (accepted for presentation)Google Scholar
- 12.Stelzer, A., Oetting, A., Chu, F.: Connection dispatching - an algorithmic and visual support for the dispatcher. In: WCTRS (ed.) Selected Proceedings WCTR 2013, Rio de Janeiro, WCTRS (2013)Google Scholar
- 13.Stelzer, A., Schütz, I., Oetting, A.: Evaluating novel user interfaces in (safety critical) railway environments. In: Kurosu, M. (ed.) HCI 2014, Part III. LNCS, vol. 8512, pp. 502–512. Springer, Heidelberg (2014) Google Scholar
- 14.Streitzig, C., Stelzer, A., Schön, S., Chu, F.: TU darmstadt - research training and more besides. EURAILmag 26(26), 152–159 (2012)Google Scholar
- 15.Willumeit, H., Gediga, G., Hamborg, K.C.: IsoMetrics: ein verfahren zur formativen evaluation von software nach ISO 9241/10. Ergonomie und Informatik 27, 5–12 (1996)Google Scholar
- 17.Stelzer, A., Schütz, I.: Field evaluation of a new railway dispatching software. In: The Eighth International Conference on Advances in Computer-Human Interactions, Lisbon, Portugal, Feb 2015 (accepted for presentation)Google Scholar
- 18.Sharp, H., Rogers, Y., Preece, J.: Interaction Design: Beyond Human-Computer Interaction. John Wiley & Sons Ltd, Chichester (2007) Google Scholar
- 20.Rubin, J., Chisnell, D.: Handbook of Usability Testing: How to Plan, Design, and Conduct Effective Tests. John Wiley & Sons, Indianapolis (2008) Google Scholar