Usability Testing Report
Table of Contents
Overview
Product
Executive Summary
Study Goals
Target Users
Heuristic Evaluation
Methodology
User Flow
Findings & Successes
Summary
Supporting data and Photos
Data Visualization
Recommendations
Next Steps
Insights and Themes
Appendix
Survey Data
Interview Questions
Overview
Product
For this project, we studied the Tesla Model S 2017 infotainment system. This model, known for its sleek and minimalistic design, aims to offer an intuitive interface and comprehensive connectivity options.
The Central infotainment and Car Dashboard both share a modern featured filled system that makes tesla unique. However there are a few safety issues that have been noted with an all touch display and some of the features lean towards unsafe versus intuitive.
Central Infotainment System
Car Dashboard
Overview
Study Goals
Determine how intuitive the infotainment system's user interface is for new and existing users.
Evaluate how the system's design influences driver safety, particularly in terms of how often drivers need to take their eyes off the road.
Examine the accessibility of features like navigation, climate control, and media through the touchscreen interface.
Gathering data on users preferences, system status, and feedback.
Collect qualitative feedback from users about their overall experience with the system, including any frustrations, likes and dislikes.
Study how quickly users are able to seamlessly switch between media and other features.
Study how quickly users can learn to use the system effectively and which features present more challenges.
Where does the car fail to integrate or distinguish between the System and the Real World?
Overview
Executive Summary
This project is focused on Tesla’s Model S 2017 infotainment system. Testing was done to evaluate the user interface (UI) and usability aspects of the climate control and other infotainment features. The purpose of this user testing study
is to assess how intuitive, responsive, and accessible the system is for drivers, particularly considering how these aspects affect the overall driving experience.
Testing methods include both qualitative and quantitative approaches; observations during use, structured interviews, and surveys filled out to gather a broad range of data on user satisfaction, ease of use, and potential areas for improvement. The specific features being tested include the navigation of menus, setting adjustments (like temperature and audio), and the ease of accessing driving-related information.
Participants would be asked to drive from A to B while using the tesla navigation system. Tester#1 would observe while tester#2 would ask questions from a script. This test was made to observe in person user testing and see the design experience. User Surveys would be posted at Tesla Chargers to get quantitate data from experienced Tesla Drivers and gather their thoughts on the product. This allows
us to contrast quantitate and qualitative data as well as experienced versus inexperienced impressions.
During usability tests, it became clear that people across all ages and tech experience levels leaned towards preferring physical buttons over touchscreen controls. While some participants easily navigated the tests, many expressed frustration over
the touchscreen's responsiveness. Concerns were also voiced about the safety implications of having to divert their gaze from the road to locate controls for climate and other features on the screen. A/B testing overwhelming supports that people use the steering wheel controls while driving.
We recommend further investigating the input responsiveness of the screen, accuracy and calibration for inputs while using the scroll wheel and touch screen
Overview
Methodology
Testing Methods
Survey
Should be done prior to other testing. QR code posters will be printed out and distributed on Tesla’s charging stations.
A/B Testing
After surveys are conducted, we’ll test the product with participants in person, by interacting with the climate control dashboard and screen options. Participants will have 5-10 minutes of testing at the LWTech parking lot, where they will interact with the screen and dashboard of the Tesla.
Guerrilla Product Testing
After surveys are conducted, we’ll test the product with participants in person, by driving and interacting with the screen. Participants will have 30-45 minutes of testing at the LWTech parking lot, where they will interact with the screen and drive the Tesla.
Overview
Target Users
Our main target is people who have purchased a Tesla or people who want a better driving experience while operating the car. Users are typically a normally older
as most users are between 40-50 with a considerable portion in the 30’s and 60’s as well. The second group would be Tesla Buyers for which we looked into the demographics of tesla sold so far.
Target Demographic Characteristics
Tesla is considered a premium electric vehicle and is marketed to households making 100k or more. Tesla owners typically tend to be older in the 40-50 age bracket but that is lowering with their new entry-level vehicles so we can aim towards 30-50. Tesla Buyers are often more concentrated in suburban or urban areas with good charging infrastructure. Tesla buyers often work in technology, finance, engineering, or entrepreneurial fields. These individuals appreciate Tesla's forward-thinking approach and are often more open to experimenting with new technologies like autonomous driving.
Participant Details
In this section, we introduce the five participants selected for our user testing. Each participant was chosen with the intent of represent a range of our target demographics and technical proficiencies. Which ensures more comprehensive insights into the usability of the system. Here's a overview of each participant:
Overview
Participant Profiles
The participants who participated in our research where given a procedure to follow before our test. Each participant was asked to start the Tesla and adjust the climate temperature and proceed to a route around the block. Then we asked our participants to type in the destination back to campus and adjust the climate while driving.
Initial Setup:
Each participant began by starting the Tesla vehicle and adjusting the climate control system to their preferred temperature.
Route Familiarization:
Participants drove a designated route around the block to familiarize themselves with the vehicle's handling and the basic controls.
Multitasking Task:
While driving, participants were asked to input a destination (returning to campus) into the navigation system. Simultaneously, they were required to adjust the climate control system to simulate multitasking while on the road.
Lonnie:
Age
Has Driven an EV
Brianna:
Age
Has not driven an EV
Samson:
Age
Driven Tesla?
Nate:
Age
Driven Tesla?
Haadi:
Age
Driven Tesla?
Overview
Methodology
We completed a user flow to get an understanding of the information architecture and understand the product a little better. We were able to conclude that A/B testing would be great for comparing the two screens in the infotainment. We noticed that there were many different ways for the user to change the climate settings with certain ones being more limited while others offering something intuitive.
Overview
Heuristic Evaluation
This section of our report dives into the heuristic evaluation of the Tesla’s infotainment system. Our goal was to analyze the system against the usability principles to find any issues that might affect user interactions. These heuristics, such as system visibility, user control, and error prevention, guide our analysis and pinpoint areas for improvement.
Through this evaluation, we aim to identify pain points of the system’s design and user expectations. The insights gained from actual user feedback are crucial, as they define our recommendations. Next, we’ll walk you through each heuristic, the challenges we found, and our analysis over our findings.
Dashboard
B
C
D
A
Visibility of System Status
Volume Button Hides the navigation for a few seconds
A
Error Prevention
B
Easy to HIt the wrong button on the dashbaord menu
Match Between the System and the Real World
C
The Scroll wheel lets you select quickly
Flexibility and Efficiency of Use
D
The icons and language for this screen communicate well to the user
Navigation Screen
D
C
C
B
A
A
Visibility of System Status
Blue line in search bar shows it is ready for a destination to be enter
Pop up keyboard indicates the system is ready for you to enter destination
A
The background map show an overview of the area
Language is simple and easy to understand
Match between system and real world
B
Error Prevention
C
No indicators of warning
or confirmation
All other icons disappear
Help and Documentation
D
Unclear where the help, settings or home option
is located
No quick access for customer support
Navigation Screen
Aesthetic and minimalist design
A
Overwhelming navigation system with too many options
Unwanted/unused music bar
Recognition rather than recall
B
Icons don’t communicate well
their actions
Error Prevention
C
All other icons disappear
No indicators of warning or confirmation
Help and Documention
D
Unclear where the help, settings or home option is located
No quick access for customer support
Climate
C
A
B
D
C
User Control and Freedom
A
Easy to miss input for where you want on the slider
Visibility of System Status
B
The Slider does not display the temperature of the
Error Prevention
C
Icons are small and close together
No error indicator or help option
Easy to hit the wrong button on the dashboard menu
Flexibility and Efficiency
of Use
D
Heuristic Evaluation
Recognition rather than recall
Error Prevention
Visibility of System Status
Heuristic
Heuristic Insight
Icons and symbols easily labeled and identifiable
Aims to prevent users from making mistakes in a system, or recover easily from mistakes
System needs to provide feedback to the user within a reasonable amount of time
Rational
The icons presented in the upper right hand corner are not easily identifable are not clear in what they do
No indicators of warning or confirmation; No back button
Volume Button Hides the navigation for a few seconds; Navigation delay
Action / Approach
Survey Question: Do you use the top right buttons often? Do you feel like the icons are easily recognizable?
Survey Question: Do you want a back button or an undo button?
Survey Question: If the volume button appears on the dashboard navigation menu
Heuristic Evaluation
Action / Approach
Rational
Heuristic Insight
Heuristic
An aesthetic and minimalist design
User Control and Freedom
Visibility of System Status
Survey Question: Asking experienced users if they like the setup
Interview Question: Asking new users
Unwanted/unused music bar; Overwhelming navigation system with too many options
Survey Question: How do you feel about the volume button obscuring the dashboard navigation?
Unwanted/unused music bar; Overwhelming navigation system with too many options
The Dashboard climate menu displays a slider
Design that focus on reducing unnecessary elements, simple layout, visual harmony of UI
Communicate clearly to users what the system’s state is — no action with consequences to users should be taken without informing them.
The Climate Slider offers a lack of information to the user on what they are changing the climate to
The volume button appears blocking the navigation when you need it
Findings & Successes
Supporting data and Photos
Summary
During our user testing and heuristic evaluation, we focused on how users from different backgrounds interact with the technology. Following next, we gathered insights from both experienced and not experienced electric vehicle users to understand how intuitive and user-friendly the system is. The testing involved
real-world scenarios where participants navigated various functions of the system, including climate controls and navigation, to identify common patterns and pinpoint areas needing improvement. Our goal was to not only test usability but also to understand the contexts in which the infotainment system excels or underperforms, ensuring a evaluation of the capabilities and limitations.
Our analysis revealed that participants with prior experience using electric vehicles (EVs) demonstrated greater proficiency in navigating the Tesla's infotainment system. They were more successful in understanding and using its functions and icons. In contrast, participants who were new to EVs encountered difficulties operating the system and adapting to its interface.
Findings & Successes
Data Visualization
20%
Do you like to make climate or navigation changes while parked or diving?
A
1
2
3
4
5
B
80%
Driving
Parked
3 out of 5 participants voiced concern to take eyes out of the road to use touch screen.
60%
40%
Do you prefer to change the temperature using the touch screen or steering wheel?
Are the icons and layout of the climate controls easy to understand?
Easy to understand
Prefers to use steering wheel
Unclear
100%
Findings & Successes
Data Visualization
We created a word map to see what words were brought up most often. We noticed that people had a lot more feedback on the climate control system rather than the navigation. Words describing the physical button interactions were more common than other ones. We noticed that people often compared the interface to apple in similarty. Spotify was mentioned several times as someones way to play music in the navigation test but they often had issues. Speed, Responsiveness, and manual were all commonly found descriptors for the test.
Findings & Successes
Affinity Mapping
With the data gathered from our user testing, we utilized the affinity mapping method to organize and interpret the data collected from our participants. This visual sorting technique allowed us to cluster user observations, comments, and feedback into groups and themes, providing a comprehensive overview of user interactions and experiences with the system. We then began to look for common themes and trends in the data.
Findings & Successes
Finding Themes and Trends
Preference for Physical FeedBack
1.Mis Inputs On Scroll Wheel and Screen
Findings & Successes
Finding Themes and Trends
3.Poor Iconogprahy
Voice Commands
Findings & Successes
Themes
Prefers Tactile Buttons
Would prefer not to use the touch screen while driving to change to temperature
Navigation screen is too big
Would like a zoomed in option for navigation while driving
Would like a higher contrast in color for some of the touch screen buttons.
Navigation feels glitchy, doesn’t keep up with real time
Controls were sensitive and misinputs were common
No clear signifier for the temperature settings
Findings & Successes
Key Themes and Analysis
Based on the affinity map feedback, participants have expressed clear preferences and concerns regarding the interaction with the infotainment system. Here’s a synthesis of the key themes and how they contribute to understanding user experience:
Preference with Physical buttons
Many participants expressed the preference of tactile buttons, while also liking the touch screen feature. They find physical buttons not only easier to navigate by feel but also less distracting, as they don’t require taking eyes off the road.
Issues with touch screen and interface
Users have mentioned that the touch screen’s sensitivity and glitchy behavior is problematic. They find that the interface can be unresponsive or overly sensitive, that can lead to accidental inputs which can be frustrating and also unsafe while driving. Theme 1
Navigation System Feedback
The feedback about the navigation was mixed, but leading towards on the negative and critical side. Users were confused by the navigation directions and found the interface either too cluttered or too zoomed out. The color and design of navigation arrows were also points of conflict, with some finding them hard to distinguish from the map background.
Wish for a improved layout, settings and menu:
Users suggested improving the layout of controls and the user interface, including higher-contrast touchscreen buttons and a simpler display. Some even expressed a desire for customization options, including the ability to personalize the placement of menus and features on the screen.
Findings & Successes
Conclusion
In conclusion, our in-depth exploration of the Tesla interface provided valuable insights and detailed feedback from all our testers. These findings have deepened our understanding of the Tesla infotainment screen, highlighting areas for improvement and offering clear direction on how to enhance the driving experience for Tesla owners.
The following areas as the most important for further research
Mis inputs on scroll wheel and touch screen interface
Navigation system UI needs clarity
Usability and distraction survey
Improved Survey Methods for testing and for gathering larger quantifiable data
Increase layout changes
Recommendations
Add Control factors to the test study to get more predictable results. Created follow up questions regarding descriptions. Increased the amount of technicality in the A/B Testing
Test Control Factors To Change
The Test Environment should be more controlled, the user should be driving in an unfamiliar area so they have to rely on navigation accounting for people using the voice feature.
The Test Environment should be more controlled, the user should be driving in an unfamiliar area so they have to rely on navigation accounting for people using the voice feature.
Changes To Test Plan
What we did well:
We worked well as a team and communicating to the testers what we where looking for and what we where expected from the tests leading to the end of this project
We were able to create 3 types of tests and had one as a backup when they survey data did not come through
We were able to analyze the quantitate data in multiple ways giving a strong start into understanding our users.
Communicated with testers what we needed from their experience and observations
Asked follow up questions
Tons of space for taking observations
We spent alot of time looking at qualitative data
What we could’ve
done better:
We've observed recurring confusion around preferred settings, similar to the glitches experienced on the large dashboard screen.
Additional time would allow us to refine our plan and perfect the questions for both the pre-drive and post-drive evaluations.
It’s challenging for testers to locate the arrow on the map. Consider increasing its size to improve visibility, while ensuring it doesn’t dominate the entire road as you zoom in.
Similar to testing in a science lab, ensure that all weather conditions remain consistent while accounting for varying traffic scenarios during every test.
Physical buttons, such as dials, were preferred over on-screen functions like sliding controls for the climate. Some testers struggled to figure out the on-screen interface.
It would be valuable to test both Tesla enthusiasts and those less fond of the brand to observe how each group interacts with the interface when completing the same tasks.
USABILITY TEST KIT
Tesla Usability Test Kit For Climate A/B Testing
Screening Questions
What is your daily method of transportation?
How comfortable are you using a touchscreen interface for your climate?
Have you considered buying a car for any reason in the past year? How does
the Tesla climate control system compare to other car climate systems you've used?
Test Compensation
25 dollars in gift cards given after the test, Thank you card printed or emailed to the participant
Tesla Interview Script
The participant will get in the car and start the testing while the car is parked.
Try set the temperature to 62 degrees and low airflow
Try to set the temperature to 75 degrees and medium airflow using a different method
Post Interview Questions
How does the Tesla climate control system compare to other car climate systems you've used?
How intuitive do you find the climate control interface? Can you quickly figure out how to adjust the temperature or airflow?
Are there any customization options you wish were available, such as saved climate profiles for different drivers or quick access to frequently used settings?
Do you find the on-screen feedback (e.g., temperature numbers, icons) clear and easy to understand?
Do you feel that the climate control system adds to your overall Tesla experience? What could be improved to make it easier or more enjoyable to use?
Do you ever experience any discomfort or challenges due to the way the climate control system operates?
Would you prefer a car that offers both touch and physical controls for climate and media settings, or do you prefer to rely on one type of control (touchscreen or physical buttons)?
Observation Notes:
Tesla Usability Test Kit For
Driving Testing
Screening Questions
What is your age?
Do you have an active driver's license?
What type of transportation do you use?
Have you considered buying a car lately?
Would you be willing to participate in a 15-minute usability testing assessment?
Would you be willing to drive for the usability testing assessment?
Test Compensation
25 dollars in gift cards given after the test, Thank you card printed or emailed to the participant
Tesla Interview Script
We will drive from LWtech to the Kirkland fire station and back.
Enter the Kirkland fire station in the navigation.
Go ahead and turn the car into drive and proceed
Arrive at the location park and enter navigation for Lake Washington Tech
Pre Interview Questions
Have you ever driven a Tesla before?
Are you familiar with an infotainment screen?
What form of navigation system do you use while driving?
In-Test Questionnaires
How easy is it to search for a location?
How easy is it for you to enter the address?
Is the touch screen consistently responsive?
How do you find the visibility and readability of the map display while driving? Is it easy to glance at for quick updates?
Are the turn-by-turn directions easy to follow?
Please change the volume on the media center
Did the navigation adjust to real-time traffic updates, road closures and construction?
Please talk us through your process
Post Interview Questions
Can you describe your initial experience with the navigation system? Was it intuitive, or did you need help getting started?
How would you rate the accuracy of the directions provided by the navigation system? Have you ever experienced incorrect or confusing directions?
How easy or difficult is it to input a destination? Do you prefer typing, using voice commands, or using the saved destinations?
How would you rate the accuracy of the directions provided by the navigation system? Have you ever experienced incorrect or confusing directions?
How does the Tesla navigation system compare to other navigation apps you've used, like Google Maps or Waze?
Overall, how satisfied are you with the navigation system, and do you feel it’s a vital part of the Tesla experience? Would you recommend improvements?
Observation Notes:
AUDIO/VIDEO RECORDING CONSENT FORM
If your study includes audio or video recording, please note that participants must consent to:
· Participate in the research
· Audio or video recording procedures.
CONSENT TO AUDIO- OR VIDEO RECORDING & TRANSCRIPTION
(Tesla Navigation and Climate Student Study)
This study involves the audio or video recording of your interview with the researcher. Neither your name nor any other identifying information will be associated with the audio or audio recording or the transcript. Only the research team will be able to listen (view) to the recordings.
The tapes will be transcribed by the researcher and erased once the transcriptions are checked for accuracy. Transcripts of your interview may be reproduced in whole or in part for use in presentations or written products that result from this study. Neither your name nor any other identifying information (such as your voice or picture) will be used in presentations or in written products resulting from the study.
By signing this form, I am allowing the researcher to audio or video tape me as part of this research. I also understand that this consent for recording is effective until the following date: _________________. On or before that date, the tapes will be destroyed.
Participant's Signature: ___________________________________________Date:___________
USER TESTING NON-DISCLOSURE AGREEMENT
This non-disclosure agreement (“Agreement”) is made on [ ], between [LW TECH STUDENT RESEARCH GROUP C] (“Company”) and the other party named on the signature page below (“Other Party”) to protect certain confidential information being disclosed between Company and the Other Party solely to perform user experience testing (“Purpose”). Company and Other Party may be referred to in this agreement individually as a “party” and collectively as “parties”. 1. Confidential Information. “Confidential Information” means the proprietary information exchanged between parties, which includes, without limitation, information (tangible or intangible) regarding patent and patent applications, trade secrets, product or service ideas or plans, techniques, sketches and designs, works of authorship, engineering designs and drawings, hardware configuration information, inventions, know-how, processes, equipment used, algorithms, software programs, tasks, questions, videos, recordings, websites, products, materials, images, text, communications, software source documents, formulae, related to the current, future, and proposed products and services, as well as information concerning disclosing party’s business,research, experimental work, development process, design details and specifications, engineering requirements, procurement requirements, employees and consultants (including, but not limited to, the names, contact information, jobs, compensation, and expertise of such employees and consultants), suppliers and customers, price lists, pricing methodologies, cost data, market share data, licences, contract information, business plans, financial forecast, historical financial data, budgets, sales plans, merchandising plans, marketing plans, clinical data and analysis, all other information that receiving party knew, or reasonably should have known, was Confidential Information of disclosing party and any information specifically marked “confidential” or “proprietary”. 2. Nondisclosure of Confidential Information. Subject to clause 16 (Permitted disclosure), the Other Party agrees to hold Confidential Information in confidence and to not use or disclose it to a third party for three (3) years from the date of initial disclosure of Confidential Information and will use Confidential Information of Company only for Purpose. Other Party will also protect such Confidential Information with at least the same degree of care that Other Party uses to protect its Confidential Information, but in no case, less than reasonable care (including reasonable security measures) to prevent the unauthorized use, dissemination, or publication of Confidential Information. Other Party must promptly notify Company of any misuse, misappropriation, or unauthorized disclosure of Confidential Information of Company which may come to Other Party’s attention. Other Party may disclose Confidential Information to its employees, agents, financial advisers, and independent contractors with a need-to-know, only as necessary to fulfill Purpose and provided such parties have signed confidentiality agreements containing, or are otherwise bound by, confidentiality
obligations at least as restrictive as those contained in this agreement. 3. Permitted disclosure. Information disclosed under this agreement will not be considered Confidential Information, if Other Party can prove that such information: (a) was in the public domain before the time of disclosure or has entered the public domain through no fault of Other Party; (b) was known to Other Party, without restriction, at the time of disclosure; (c) was independently developed by Other Party without use of or reference to Confidential Information; (d) is rightfully disclosed to Other Party by a third party without confidentiality restrictions; (e) is disclosed with the prior written approval of Company; or (f) is disclosed according to the order or requirement of a court, administrative agency, or other governmental body; provided, however, that Other Party will (i) notify Company in writing of the requirement for disclosure, unless notice is prohibited by law; and (ii) disclose only that portion of Confidential Information legally required. 4. Return of information. Upon request of the Company, the Other Party will promptly return or certify the destruction of Confidential Information and all authorized copies thereof 5. No rights to Confidential Information are granted. The company will retain all rights, title, and interest to its Confidential Information. This agreement does not grant to Other Party patent, copyright, or other intellectual property right that has been issued or that may be issued, based on Confidential Information or other rights, except the limited right to use Confidential Information for Purpose. Nothing in this agreement creates or will be deemed to create any employment, joint venture, or agency between Parties. Nothing in this agreement requires the Company to enter into any transaction with the Other Party in connection with which Confidential Information may be disclosed. 6. No reverse engineering. Other Party will not modify, reverse engineer, decompile, reproduce, create other works from, or disassemble any software programs contained in the Confidential Information of Company. Any reproduction by Other Party of any Confidential 7. Information of the Company will remain the property of the Company. 7. No warranty. All Confidential Information is provided “as is” for use by Other Party at its own risk. Company disclaims any warranties, express, implied, statutory, or otherwise, regarding Confidential Information, including without limitation, any warranties of title, merchantability, fitness for a particular purpose, or non-infringement. 8. Term. This agreement will terminate three (3) years after the date of this agreement or may be terminated by the Company at any time upon thirty (30) days' written notice to the Other Party. Other Party’s obligations under this agreement will survive termination of this agreement and will be binding upon Other Party’s heirs, successors, and assigns. 9. Remedies. Other Party agrees that due to the unique nature of the Company’s Confidential Information, any breach of this agreement may result in irreparable damage to the Company for which monetary damages would be an inadequate remedy. Therefore, in addition to any other remedies that may be available, in law, in equity or otherwise, the Company will be entitled to obtain
injunctive relief against the threatened breach of this agreement or the continuation of any such breach by the Other Party. 10. Governing law. This agreement will be governed by the laws of the State of Delaware, without giving effect to principles of conflicts of law. 11. Final provisions. (g) This agreement constitutes the entire agreement between the parties concerning its subject matter and supersedes all prior agreements between the parties regarding the same subject matter. (h) This agreement can be modified only by a written amendment signed by parties. (i) Failure to enforce any provisions of this agreement will not constitute a waiver. (j) If any provision is unenforceable, the other provisions will remain effective. (k) Parties may execute this agreement in counterparts, which taken together will constitute one instrument. (l) The Company may assign any of its rights and obligations under this agreement. Other Party may not assign, whether voluntarily or by operation of law, any of its rights and obligations under this agreement, except with the Company's prior written consent. (m)Any notice under this agreement must be in writing and delivered personally or by overnight courier or sent by email.
COMPANY
By: ______________________________
Name: ________________________
Address: ________________________
OTHER PARTY
By: ______________________________
Name: ________________________
Title: ________________________
Address: ________________________
Appendix