Measuring and Evaluating the User Experience Strategy Maturity of Spatial Web-Based Projects: A Case Study of Tehran Web-Based Map
The design and implementation of spatial Web-based projects has been driving much research and development interest from the global community of practice over the past 10 years. The user centered evaluation of such projects outputs should be a vital part of the overall system development lifecycle. Continuously measuring and improving the User Experience (UX) of the output systems is a key factor driving better usage and enhanced user engagement.
Our research objective was first to track real-time and real-world usage of Tehran web based map, developed by TMICTO (Tehran Municipality ICT Organization), then to perform a usability test on selected tasks within a defined context of use and finally, to evaluate the UX strategy maturity of Tehran web based map based on existing maturity models.
We used mixed user research methods, including remote and on-site user research methods, to understand the context of use, usage behavior patterns in terms of actions and engagement, then we designed and performed usability testing and finally we evaluated the UX strategy maturity of Tehran web based map. Concrete and practical recommendations to improve the level of UX strategy maturity of Tehran web based map oriented towards short term, medium term and long term visions are formulated at the end of our research.
KeywordsUser experience (UX) UX strategy UX maturity Usability Spatial web-based projects
From a Human Computer Interaction (HCI) perspective, the availability of personal computers from the late 1970s was quite a disruptive technological innovation from the time when the only users who interacted with computers were ICT experts and professionals. Personal software and personal computer platforms made very basic literate human being a potential computer user. As non ICT professionals started to use personal computers and systems, usability and user experience (UX) challenges surfaced and became major barriers to user adoption and improved uptake .
According to the ISO 9241-11 standard, “usability is the degree to which software can be used by specified consumers to achieve quantified objectives with effectiveness, efficiency, and satisfaction in a quantified context of use”. Usability is a subjective measure because levels of usability depend on the user and the context of use .
Customized experimental design, defining usability measure and usability measurement techniques for complex products, systems and services is difficult .
Although usability is a well-established concept with proven measure and measurement methods, HCI researcher and practitioners have different level of understanding of UX operational definition, metrics and measurement methods .
The international standard on ergonomics of human system interaction, ISO 9241-210, defines user experience as “a person’s perceptions and responses that result from the use or anticipated use of a product, system or service” .
Beside the broad range of proposed definitions for UX, there is a consensus of opinion among HCI researchers and practitioners that UX as a concept is the result of the interaction between three elements: the user, the system and the context . No one can predict for certain how a system will be perceived by the end users. Informed decisions can be made about UX during each phase of the overall system development lifecycle by using a mix or relevant user research and UX design methods and tools .
The level of interest for UX designers and researchers within organizations is constantly growing in most developed countries across the globe. Hence, competitive organizations and specifically multinational corporations compete for recruiting skilled UX consultants and experts that would be able to have a holistic impact.
The “2017 UX and User Research Industry Survey Report” published by UserTesting , indicates a positive trend toward early, frequent research positive trend toward early, frequent research positive trend toward early, frequent research throughout the design, development, and optimization of digital products. The majority of 2238 surveyed UX professionals reported that their testing frequency has increased moderately or significantly.. The number of UX professionals who do user research on their competitors has more than doubled year-over-year .
How do existing and new users interact and engage with Tehran web-based map?
What is the role of UX in Tehran web-based map?
Which models and methods can be used to evaluate it’s UX strategy maturity?
How mature is Tehran web-based map in terms of UX strategy?
What key recommendations can be formulated to improve the level of Tehran web-based map UX strategy maturity?
2 Review of Literature
While performing extensive review of literature on topics related to the UX strategy maturity of Web-based map projects, we realized that much research has been performed on the connected topic of GIS2 and online maps usability.
Harding (2009) provides a research methodology which established a reference base of qualitative data on user needs for geographic information with respect to context of use. The key elements which contribute to geographical information systems usability are namely: (1) information content, (2) information quality, (3) information structure and interoperability, (4) information file format, (5) information volume, (6) information presentation, (7) information cost, (8) information delivery and (9) information selectability .
Nivala et al. (2007) studied the level of usability methods’ familiarity among map application and systems developers. The results showed that although usability engineering is slowly being incorporated into design of map applications, knowledge on how to execute the methods is still very low or almost non-existent. Many companies are actually interested to implement such methods, but one key problem is the lack of resources and applied knowledge on how to implement an approach .
Ingensand and Golay (2010) evaluated the task-oriented usability of a Web GIS. They realized there are different strategies for interacting with Web GIS and that users perform differently according to selected personal strategies. These differences in performance are related to the users’ experience and expertise with respect to similar systems usage and to specific user interface key functions and features .
Different models to design a UX strategy and assess its maturity within organizations have been proposed in the past few years. Jaime Levy (2015) suggests a quadruple tenet to design a UX strategy framework. Levy’s UX strategy framework is composed of: (1) Business Strategy, (2) Value Innovation, (3) Validated User Research and (4) Killer UX Design .
Stern (2014) has also proposed the CUBI model for achieving improved UX competitiveness. The CUBI model includes 4 main layers: (1) Content, (2) User goals, (3) Business Goals and (4) Interaction . Each layer has dedicated sub-layers.
Kalbach (2014) has designed the UX Strategy Blueprint which is a simple tool to help define a given UX strategy. The elements within the UX Strategy Blueprint are: (1) Challenges, (2) Aspirations, (3) Focus Areas, (4) Guiding Principles, (5) Activities and (6) Measurements .
Nielson (2006) also defined an eight stages process for achieving Corporate UX Maturity . In the initial stages developers and corporations show hostility towards usability. Most companies realize the value of making designs easier for human end users. In the next stages a fixed budget is dedicated and there shall be an official usability group, led by a UX manager in the company. Then, there should be an integrated user-centered design team incorporated within the corporation.
And finally the last stage should be a user-driven corporation. At this final maturity level, the usability methods will affect corporate strategy and activities .
Chapman and Plewes (2014) have developed a UX Maturity Model composed of 5 stages . This model can help understanding any organization’s UX level of maturity. The model provides six key indicators of UX maturity as follow: (1) the timing of UX (2), the UX expertise and resources, (3) the usage of appropriate techniques, (4) the leadership and culture in the company, (5) the degree of integration of UX with other corporate processes and finally (6) applying design.
(1) Beginning, (2) Awareness, (3) Adopting, (4) Realizing and (5) Exceptional .
3 Research Methods
3.1 Quantitative Data
Online real-time and real-world user’s behavior quantitative data was collected by using professional Web Analytics tools for a period of 30 days. We identified users to analyze and understand the way they interacted with Tehran online map portal3. Our analysis included segmentation of users based on key factors such as visitors, time, frequency, actions, content and engagement. This part of research, also known as web analytics, was performed over one month period. We used Google Analytics to generate specific Java Script tracking codes and embedded the codes in the proper source code with the assistance of a partner company in charge of Tehran Web-based map.
3.2 Qualitative Data
Online real-time and real-world user’s behavior qualitative data was collected by using professional remote heatmapping tools, provided by SIGCHI Iran Chapter , for a period of 30 days. Heatmaps represent where the visitors concentrated their clicks and how frequently they clicked on a given screen zone. Generally, a color scale moving from blue to red indicates the frequency of clicks. Thus, a red spot over an area of a webpage might indicate that a participants clicked on this part of a webpage much frequently than other areas . In the qualitative part of the research we analyzed selected screen heatmaps for specific visual elements to analyze and understand usage behaviors and possible design challenges as barriers to usage.
3.3 Mixed Method
List of usability testing tasks
Use the sculpture database widget to identify a statue in Region 1 “Niavaran” neighborhood and orally state the artwork’s name and used material
Use the sculpture database widget by operating the “yes or no” selector to orally state whether the “Garshasb” statue is lighted or not. Identify and orally name the lighting equipment used to light-up this specific statue
Use the sculpture database widget to identify and orally state the district of Tehran in which the “Bolbol” sculpture is located. Also identify and orally name the artist who created the “Bolbol” sculpture
Use the sculpture database widget and the distance widget to orally state the distance between the “Garshasb” statue and the closest subway station
Use the sculpture database widget and the street view widget to identify and orally state the type of urban furniture next to the “Roftegar” statue
The usability test session were conducted in a customized office of the Tehran Beautification Organization transformed into a usability lab for one full day. All usability test participants were staff members of Tehran Beautification Organization including 8 women and 10 men. A specific participant selection procedure and protocol was used to selected relevant participants based on previous knowledge and experience with Tehran online map portal and sculpture database widget. Informed consent forms, pre-test questionnaire, observer recording sheet, system usability scale (SUS) questionnaire and professional time and video recording software were used.
4.1 Existing and New User Interactions and Engagement with Tehran Web-Based Map
To understand how users interact with Tehran online web-based map, quantitative data was collected and analyzed with Google Analytics.
Quantitative Data Analysis (Using Web Analytics).
Tehran map portal visitors’ information (in a month)
No. of unique visitors
No. of new visitors
No. of returning visitors
Number of page views
Total Time on portal
1776 days 23 h
Average time per visit
8 m 7 s
Visitors accessed Tehran online web-based map via different sources. Namely, 59.1% of visitors came via search engine, 33.9% of visitors came by directly typing the <http://map.tehran.ir> URL and 7% came from links. In term of geographical origin of users, 94.2% of visitors were from Iran, 3% from the USA, 0.7% from the UK, 0.4% from The Netherland, 0.3% from Germany and the rest from other countries.
Most visitors spent less than 10 min in Tehran map portal. A total of 19.78% of visitors engaged between 9 and 10 min and 19.44% engaged less than one minute. So, the engagement time for 12% of visitors is less than one minute which can be specified by two groups of users. Regarding bounce rate of 10–30% for service sites , a 6% bounce rate indicates the most visitors are in need of information and very much willing to engage with the map.
Qualitative Data Analysis (Using Heatmaps).
Figure 4, generated over 30 days or visitor heatmapping, displays red (hot) areas that are visitors “focus zones” on the main screen and colder colors (yellow, green) display a lower level visitor interaction and engagement. The “routing” and “Tehran key places” widgets located on the left side toolbar and the “search” and “more information” widgets located on the top toolbar got very high level of user interactions. Figure 4 provides a key insight about visitors not willing to access or read the “user manual” via the “help” section and to “register” with the map located on the top bar.
Mixed Data Analysis (Using Lab Based Usability Testing).
To determine to what extent the usability test participants were familiar with online map service, a dedicated questionnaire was distributed between them before the usability test. Among a total of 18 usability test participants, 8 were women and 10 were men.
Average time of task (task duration) for usability test participants (N = 18)
Task T3 was the most failed task by participants with an average completion time of 121.17 s as the longest recorded time among all five tasks. One underlying explanation is that as most participants didn’t manage to find and use the «Tehran city neighborhoods» selector so they would need to zoom out and in to identify the neighborhoods.
Almost 28% of the usability test participants were able to successfully perform task T4 that required using the horizontal toolbar at top of the screen as well as using the «distance calculation» widget. One of the reasons participants’ failed completing task T4 is that they were didn’t understand how to use and operate this widget.
A total of 8 participants managed to successfully complete task T5 that required using «Street view» widget. One of the participants did a shortcut by using the picture of sculpture could to complete this task. Two participants also didn’t manage to figure out and understand how to use and operate the widget.
Recurrent improvement suggestion for this map designers made by most of the usability test participants was to add simple and short usage guidelines, to use different color codes to identify the widgets and use simple unambitious icons on toolbars.
The usability test results confirm that although Tehran online Web-based map is full of advanced features and widgets, users have a hard time understanding how to usage and operate them. In addition, one of our participants looked into the available “help” section to learn how to use the widgets. Some usability test participants reported that the waring messages and text used inside the dialogue box were confusing.
4.2 What Is the Role of User Experience in Tehran Web-Based Map?
A standard System Usability Scale (SUS) questionnaire was distributed to each participant right after the end of the usability test sessions. The Tehran online Web-based map obtained a SUS final score of 66.5. Based on Bangor (2009) adjective rating and acceptability ranges , 66.5 is an acceptable SUS score than is positioned in between “ok” and “good” categories. Therefore, based on our usability test, Tehran online Web-based map portal usability has still much room for usability improvements.
Apart from the usability tests we performed with the participants, we also reviewed and analyzed the initial Request for Proposal (RFP) written for Tehran online Web-based map by TMICTO. The RFP we analyzed didn’t include any minimum requirements for key human factors such as accessibility, usability and user experience factors. Our understanding is that TMICTO is either not aware of a user centered system development lifecycle method or no resources are dedicated to human factors.
4.3 Which Models and Methods Can Be Used to Evaluate It’s UX Strategy Maturity?
Different methods and patterns have been proposed for evaluating the UX strategy maturity within organizations. In this research, Chapman & Plewes (2014) model was used to evaluate the UX strategy maturity of Tehran online Web-based map portal. As discussed in literature review section, Chapman and Plewes (2014) proposed a 5 stages model that drives an organization from an “immature” toward a mature “stage” . This model can help organization with defining each stage indicators quite distinctively, so that they would be able to evaluate each maturity stage based on clear metrics.
4.4 How Mature Is Tehran Web-Based Map in Terms of UX Strategy?
Based on our findings we can affirm that the level of UX literacy and UX strategy maturity at TMICTO is very low. TMICTO would definitely benefit from learning and applying a user-centered system development lifecycle methodology. It also appears that there is a very low level of applied knowledge about usability, UX and human factors at TMICTO. Based on chapman & Plewes (2014) model, this organization is at the very beginning stage. This first stage is where the organization just thinks of UX as end-user product add-on or a service visual design enhancement.
4.5 What Key Recommendations Can Be Formulated to Improve the Level of Tehran Web-Based Map UX Strategy Maturity?
Taking advantage of professional user research experts and UX consultants outside the organization,
Learning and applying the foundations of Human-Computer Interaction (HCI) including User-Centered Design (UCD), usability and UX by all researchers, designers and managers working at TMICTO,
Understanding the importance of incorporating key human factors and metrics to each and very future Request for Proposal (RFP) related to Tehran online web-Based map and other information systems
Performing continuous rapid and rigorous user research with a wide mix of analytical and empirical methods
Defining and incorporating usability requirements and UX targets to each and every project at TMICTO,
Ensure that all users and those who will be affected by an existing or future system have their needs clearly articulated during the assessment and design of requirements; later the system must be validated to meet requirements via usability testing
Prioritizing UX as a key success and competitiveness factor in each new project,
Sponsoring user research and UX design for the very first day of each new project,
Partnering and networking with local world-class professional communities of practice such as the AM SICHI Iran Chapter  for actionable professional training, applied and basic combined experience sharing towards better interactions.
In this research we aimed at investigating the UX of spatial web based projects and the level of UX strategy maturity within projects and more specifically organizations in charge of those projects. The Tehran online Web-based map owned by TMICTO was taken as our case study.
First, we tracked real-time and real-world usage of Tehran online Web-based map by collecting quantitative and qualitative data during 30 days. Quantitative data was also collected and analyzed with dedicated web analytics tools such as Google Analytics during the same 30 days. Qualitative data was collective via dedicated remote user research tools such as heatmap tools. The “routing” and “Tehran key places” widgets located on the left side toolbar and the “search” and “more information” widgets located on the top toolbar got very high level of user interactions. Based on the collected and analyzed data from 275320 unique visitors, we defined specific tasks for a specific context of use and conducted usability testing sessions with 18 participants including 8 women and 10 men from Tehran Beautification Organization.
Based on the results of the usability testing session, the Tehran online Web-based map obtained a SUS final score of 66.5. Based on our research findings we can affirm that the level of UX literacy and UX strategy maturity at TMICTO is very low.
The comparative survey of TMICTO and Tehran online Web-based map portal with respect to Chapman & Plewes (2014) UX Maturity Model’s indicators showed the TMICTO is still at the very beginning stage in terms of UX literacy, UX strategy and UX maturity. Much has to be done to incorporate UX from the overall lifecycle.
We proposed short term, medium term and long term practical recommendations for improving the current state of TMICTO UX literacy, maturity and Tehran online web-Based map UX strategy maturity. TMICTO should prioritize UX as a key success and competitiveness factor in each new project towards improved UX maturity.
Our research findings also highlight the negative outcomes of low usability and user experience as very strong barriers to improved end-user engagement. Sponsoring user research and UX design for the very first day of each new project and programs can be key driver towards overcoming such barriers. Finally, we strongly recommend partnering and networking with local world-class professional communities of practice such as the AM SICHI Iran Chapter <http://www.sigchi.ir> for actionable professional training, applied and basic combined experience sharing towards better interactions.
One of the limitations of this research is the time constraint of 30 days (one month) during which the remote user research with web analytics and heatmap tools were performed. Extending this period of time to 6 or 12 months would provide us with richer real-time and real-world data towards improved research insights and findings. The authors also acknowledge that the context of use, selected tasks and participants with respect to the usability tests are domain specific and linked to Tehran Beautification Organization.
- 1.UX and User Research Industry Survey Report 2017 by UserTesting. (2017). http://info.usertesting.com/ux-industry-survey-2017.html. Accessed Feb 2017
- 2.Allanwood, G., Beare, P.: User Experience Design: Creating Designs Users Really Love. Bloomsbury, New York (2014)Google Scholar
- 3.Bangor, A., Kortum, P., Miller, J.: Determining what individual SUS scores mean: adding an adjective rating scale. J. Stud. 4(3), 114–123 (2009)Google Scholar
- 4.Bryan, P.: What is UX Strategy? (2014) http://www.uxstrat.com/blog/what-is-ux-strategy. Accessed Feb 2007
- 5.Myers, B.A.: The Encyclopedia of human-computer interaction. In: Soegaard, M., Dam, R.F. (eds.) The Interaction Design Foundation, 2nd edn. (2014). http://interaction-design.org. Accessed Feb 2017. ISBN 978-87-92964-00-7
- 7.Cousins, C.: Why Does User Experience Matter? (2013). http://designshack.net/articles/why-does-user-experience-matter. Accessed Feb 2017
- 9.Ingensand, J., Golay, F.: Task-oriented usability evaluation of a WebGIS for a real-world community. URISA J. 22, 41–53 (2010)Google Scholar
- 10.ISO 9241-210: Ergonomics of human-system interaction – Part 210: Human-centred design for interactive systems. ISO (2010)Google Scholar
- 11.Kalbach, J.: UX Strategy Blueprint (2014). https://experiencinginformation.wordpress.com/2014/08/12/uxstrategyblueprint. Accessed Feb 2017
- 12.Kelly, K.: What is Bounce Rate? Avoid Common Pitfalls (2012). http://www.blastam.com/blog/index.php/2012/02/what-is-bounce-rate. Accessed Feb 2017
- 14.Levy, J.: UX Strategy: How to Devise Innovative Digital Products That People Want. O’Reilly Media (2015)Google Scholar
- 16.Nielsen, J.: Corporate UX Maturity: Stages 1–4. (2006). https://www.nngroup.com/articles/usability-maturity-stages-1-4. Accessed Feb 2017
- 17.Nielsen, J.: Corporate UX Maturity: Stages 5–8 (2006). https://www.nngroup.com/articles/usability-maturity-stages-5-8. Accessed Feb 2017
- 18.U.S. Dept. of Health and Human Services. The Research-Based Web Design & Usability Guidelines, Enlarged/Expanded edition. Government Printing Office, Washington (2006) https://www.usability.gov/how-to-and-tools/methods/eye-tracking.html. Accessed Feb 2017
- 20.Stern, C.: CUBI: A User Experience Model for Project Success (2014). https://uxmag.com/articles/cubi-a-user-experience-model-for-project-successAccessed Feb 2017