1. Introduction
The TFI Leap Top-Up App (National Transport Security, 2016) is a mobile application that enables users to manage their TFI Leap Cards for public transport in Ireland, by scanning their card with an NFC-enabled device. It provides several features, such as checking the balance, topping-up credit, purchasing tickets, viewing transaction history, and tracking capping values (TFI Leap, 2015). As many users rely on the TFI Leap Top-Up App for convenient access to public transport, it is important that the application is user-friendly, efficient, and fully functional. This in-field user evaluation aims to assess the user experience, identify any difficulties users encounter, and provide recommendations for improvement.
2. Methodology
This user-evaluation combines quantitative and qualitative data from five users to research how to improve the TFI Leap Top-Up Application’s user experience. Quantitative data is collected via a Microsoft Forms questionnaire, collecting demographic data and responses to standardized questionnaires to assess the usability and user experience (UEQ-S (Schrepp, 2017)), workload (NASA-TLX (Hart, 2006)), and user satisfaction (PSSUQ (Lewis, 1992)). Qualitative data is gathered in semi-structured interviews, recorded, and summarized. Eight questions cover the user’s overall understanding (Q1&Q2), difficulties (Q3), navigation (Q4), comparison to similar applications (Q5), assessment of features (Q6&7), and visual design (Q8). Interviews, quantitative data, and qualitative data are all available in the appendices.
3. User Study
3.1. Procedure
The user study was conducted in circumstances suitable for one-on-one interviews. Three interviews were held in an apartment, and two on the street. Before starting the data collection, participants were asked for their consent to collect their data anonymously and record the interview. They were also asked about their familiarity with the TFI Leap Top-Up App. Participants unfamiliar with the application were asked to explore it and attempt to top-up a Leap Card. After addressing these introduction questions, participants completed the Microsoft Forms questionnaire to gather the quantitative data, followed by a semi-structured interview to gather qualitative data.
3.2. Apparatus
An NFC-enabled device with the TFI Leap Top-Up App and a Leap Card were available for the users during the interview. The study used a Microsoft Forms to collect demographic data and responses to the standardized questionnaires NASA-TLX, UEQ-S, and PSSUQ to collect quantitative data. Participants completed the Microsoft Forms questionnaire via a link or QR code using their own mobile device, or on a provided mobile device for participants without access to the internet. Thereafter, the interview was recorded on a mobile device. The quantitative data was subsequently processed and partially analysed using Excel.
3.3. Participants
| User 1 | User 2 | User 3 | User 4 | User 5 | Average | |
| Quantitative data | 05:29 | 03:52 | 03:38 | 04:15 | 02:08 | 03:52 |
| Qualitative data | 10:11 | 03:50 | 03:54 | 07:40 | 05:12 | 06:09 |
| Total time | 15:40 | 07:42 | 07:32 | 11:55 | 07:20 | 10:01 |
| Table 1 |
In total five users participated in the study, with each interview lasting approximately 10 minutes (see Table 1). Four participants were female, and one was male. The group was relatively young and academically inclined, with four participants under 25 years old and currently studying. All participants were familiar with mobile devices, having used smartphones for at least three years, and spending a minimum of one hour on their devices daily. Four of these participants had previously used the TFI Leap Top-Up App, with each having used it for over six months. Usage frequency of the application varied among these participants, ranging from rarely to weekly to monthly.
4. Results
4.1. Quantitative Data
4.1.1. UEQ-S
Based on the results the overall user experience was positive (0.8). Especially, pragmatic quality (1.55) scored high, indicating user satisfaction with the application’s clarity and efficiency. The hedonic quality (0.05) is neutral, suggesting neither appreciation nor dislike for the user interface.
| Dimension | Average |
| Pragmatic quality | 1.55 |
| Hedonic quality | 0.05 |
| Overall score | 0.8 |
| Table 2 |
4.1.2. NASA-TLX
The overall workload was rated positive (2.6), with especially low physical demand (1.4). The results suggest the mental load, time demand and frustrations were all limited, each scoring a 2.6. Effort (3.4), and performance (3) were rated slightly higher, indicating that it was somewhat demanding to use the application effectively.
| Dimension | Average |
| Mental Demand | 2.6 |
| Physical Demand | 1.4 |
| Temporal Demand | 2.6 |
| Effort | 3.4 |
| Frustration Level | 2.6 |
| Performance | 3 |
| Overall Workload | 2.6 |
| Table 3 |
4.2. PSSUQ
Overall user satisfaction was good with a score of 2.59, with an especially high score for system usefulness (2.17). The interface quality (2.6) scored slightly lower, suggesting room for minor improvements. The information quality (3.04) was rated the lowest, indicating that the clarity of information could be slightly improved.
| Subscale | Average Score |
| System Usefulness | 2.17 |
| Information Quality | 3.04 |
| Interface Quality | 2.60 |
| Overall PSSUQ Score | 2.59 |
| Table 4 |
4.2.1. Qualitative Data
4.2.2. Usability
The use of the application was perceived to be ‘easy’ and ‘simple’ in terms of efficiency, and users described the NFC-enabled card scanning as ‘really good’. Users did wish for more payment options, besides credit/debit card. Dependability was sometimes lacking, further discussed in Cognitive Load. The visuals were not considered exiting, however often described positively as ‘clean’ and ‘easy’.
4.2.3. Cognitive Load
Despite some frustrations, users found the application ‘easy’, ‘simple’ and ‘straightforward’. Main difficulties included struggles to scan the card, limited payment options, random downtime, and slow balance updates. Further suggestions were an automatic top-up (add credit automatically if the balance is below given threshold), and another method to access the application.
4.2.4. User Satisfaction
Users were generally satisfied with the application. They found it ‘quite easy’ to navigate. This is however slightly contradicting with the finding that most users did not know the purpose of the ‘capping’ and ‘add ticket’ pages. Suggested improvements were adding a search function and a bus location tracking.
5. Discussion
5.1. Usability
The high score for pragmatic quality (0.8), suggests that the application is supportive, easy, efficient and clear. This is confirmed by user feedback, as they would consider the application to be easy and well-functioning, aside from a few issues discussed in Cognitive Load. Therefore, efficiency improvements are not a priority. However, a suggestion to improve the usability even further, was adding more payment options. The hedonic quality was lower. This is explained by users who found the application’s layout to be simple and clean. Therefore, the layout was appreciated for ease of use, but does not stand out. Consequently, the application may not require changes to the visual aspect.
5.2. Cognitive Load
The overall workload was considered low, confirmed by users describing the application as ‘easy’ and ‘straightforward’. Effort and performance scored slightly worse than other dimensions. This is represented in the qualitative data, as users did encounter a couple of difficulties. These problems mainly included difficulty scanning the card, having limited payment options, the application being down randomly, and slow balance updates. To decrease the overall workload, these issues should be fixed. Other suggestions of users that would decrease the workload are to include an automatic top-up and another method to access their account.
5.3. User Satisfaction
The quantitative data suggested a positive user satisfaction, validated by users saying the application is good. The interface also scored well, but slightly lower. This means there is room for small improvements, considering the suggestions of users to add a search function and bus location tracking. Information quality was considered to be the worst. Qualitative data reflects this, as users found the application easy to navigate. However, there were some features on the main menu they were unfamiliar with. Therefore this aspect could be improved by enhancing overall messaging, and clarity of the ‘capping’ and ‘add ticket’ pages.
6. Design Guidelines
Given the positive reactions, which appreciated the simplicity of the visuals and layout, there is no need to alter the design to boost the UEQ-S hedonic quality score. Instead, the focus should be on maintaining a simple and clear layout.
Difficulties users encountered when using the application should be addressed. Firstly, for when there are issues with card scanning, there should be another option to access their account, for instance by filling in their card number. Additionally, more payment options should be available, including Google Pay and Apple Pay. Furthermore, although payment transaction speed improvements fall outside the scope of this study, a pop-up message to inform users of delays should be added. These adjustments would likely improve both the UEQ-S’s pragmatic quality score and NASA-TLX’s effort, performance, and frustration scores.
Introducing an automatic top-up feature, enabling automatic top-ups when credit is low, would reduce NASA-TLX’s temporal demand, but also improve scores across other NASA-TLX dimensions, USEQ-S’s pragmatic quality, and system usefulness in PSSUQ.
Finally, the information present in the application – particularly on the ‘add ticket’ and ‘capping’ pages – should be slightly improved. Also, the interface could be even further enhanced by implementing a search function. These two changes would improve the PSSUQ’s information and interface quality scores.
6.1. Limitations
Two categories of difficulties and suggestions of users are not within the limitations of this project. First, the suggestion to add a feature to track the bus is already available in the application TFI Live (National Transport Authority, 2023). Combining different TFI applications for better accessibility and efficiency is beyond the aim of this project, which is to only improve the TFI Leap Top Up App.
Secondly, resolving technical issues of the application, including the app being down and a slow payment transaction process, is unachievable to improve within this project.
7. Conclusion
Reflecting on the process, combining quantitative and qualitative data was essential to gain a deeper understanding of the user’s perspective and create comprehensive design guidelines. This user-evaluation of the TFI Leap Top-Up App reports a great user experience, with users especially valuing its simplicity and ease of navigation. The design guidelines discuss areas for improvement, such as an alternate method to access accounts besides scanning a card, adding payment options, clearer explanations for specific features, an automatic top up feature, and a search function. In the end, the findings address user-centred improvements to further elevate the user experience.
8. References APA
- Hart, G. S. (2006) NASA-TASK LOAD INDEX (NASA_TLX); 20 YEARS LATER. Moffett Field, CA. NASA-Ames Research Center. Accessed: 16/10/2024
- Lewis, J. R. (1992) PSYCHOMETRIC EVALUATION OF THE POST-STUDY SYSTEM USABILITY QUESTIONNAIRE: THE PSSUQ. Boca Raton, FL. IBM Design Center. Accessed: 16/10/2024.
- National Transport Authority (24 January 2016) Leap Top-Up (Version 0.4.2) [Mobile App]. Available at: https://play.google.com/store/apps/details?id=ie.leapcard.tnfc. Accessed: 10/10/2024.
- National Transport Authority (14 February 2023) TFI Live (Version 1.8.34) [Mobile App]. Available at: https://play.google.com/store/apps/details?id=com.trapezegroup.TFILive.nta&hl=en_IE. Accessed: 24/10/2024
- Schrepp, M. et all. (11 September 2017) Design and Evaluation of a Short Version of the User Experience Questionnaire (UEQ-S). SAP AG. UNIR. Accessed: 16/10/2024
- TFI Leap (2015) Leap Top-Up App. Available at: https://about.leapcard.ie/leap-top-up-app. Accessed: 20/10/2024.