How do you discover what people really do with your software? Our behavior often differs from what we think we do. The human-computer interaction research method eye tracking can reveal where a user's attention goes when they use a product.
Two days after Skype launched the Skype for Windows 4.0 Beta in June 2008, Cimex, a British firm with a usability and accessibility lab, spent a day watching people use Skype. Then wrote them up. I'm including the full text of their report below the fold. From their blog:
A usability study by Cimex has revealed Skype’s new Beta release may prove difficult to use, due to weaknesses in presentation, navigation, structure and functionality.
The aim of this study was to capture the users’ expectations, needs and preferences, all of which we believe can be used as a base to create a more engaging, efficient and effective Skype application.
Caveats:
-
Only six users were tested: 25-50 years' old; British, French, Greek; three newbies, three experienced Skypers; three men, three women. This is not representative of Skype's worldwide user community. Skypers come in all ages, speak many languages and bring wildly different cultural norms to software use.
The team said "A larger sample would allow us to perform more detailed quantitative data analysis of the performance rates." A larger sample would also "allow us to create more targeted scenarios and cover all the range of Skype user profiles."
-
The study was relatively shallow since few user goals were tested in the time available, and many product features were unexplored. See the sidebar for four user goals.
Four of the twelve tasks used in the study
Could you please have a look at this welcome screen and tell me what your first impressions are? What would be the first thing you would like to do from here? (Users were allowed to explore the options of the welcome screen and then asked to start using the application.)
What are you impressions by looking at this page of the application? Do you like the way the information is presented?
Let’s have a look at the top menu (Skype, Contacts, Conversation, etc). Are these options meaningful enough to you? Without clicking on them yet, could you please tell me what do you expect to find under each of these menu options?
Imagine now, that you want to invite a friend who hasn’t got a Skype account, to join Skype. How would you do that?
Observing users with eye tracking is best applied iteratively and with a specific problem in mind. Feedback from one round can reveal a problem which, once cleared, reveals other obstacles to people doing what they set out to do.
I'd be curious how these results compare with the 3.8 release; has 4's redesign addressed many of 3.X's problems with complexity and discovery?
Unlike many other products, Skype comes in many flavors, like the mobile edition on the Skypephone or Skype for the Nokia tablets. You might learn a great deal by testing goal success (e.g. "Call this phone number", "IM a friend on your buddy list", "start a video call") across operating systems and devices.
So here's the Skype Journal Daily Question: What are the top three design goals you see in Skype 4b?
See also:
Download the Usability Study of Skype 4.0 in PDF format or read the web version below...
tags: skype, cimex, design, usability, ui, ue, ux, heat, tracking, attention, layout, report, 4, 4.0, 4b, analysis, research, uk, london
Follow Phil Wolff on Twitter or FriendFeed or on Skype.
Skype 4.0 Beta usability review
1. Introduction
This report contains the results of a short series of user testing sessions, carried out by Cimex on the new version of Skype, namely the Skype 4.0 Beta 1 release. Our brief usability study has revealed many of the difficulties users faced, due to weaknesses in the presentation, navigation, structure and functionality of this widely used application.
Figure 1. The first page of Skype application.
2. Aims and Objectives
The aim of this study was to capture the users’ expectations, needs and preferences all of which can be used as a base to create a more engaging, efficient and effective Skype application. The key objectives were to:
- Explore users’ experiences and expectations based on their previous use of Skype.
- Assess the usability of Skype 4.0 Beta 1 version by users performing typical tasks with the application.
- Assess if the information presented and the functionality are appropriate for the target audience.
- Capture the users’ reactions and preferences for style and design ideas for the application.
3. Methodology
For the purpose of this study, the qualified User Experience team at Cimex carried out six individual user testing sessions, which took place at the Cimex specialised usability labs in Central London. Each session lasted 60 minutes and consisted of four main areas:
Pre-session interview
The purpose of this first exercise was to make the user feels comfortable with the testing and to gather useful information about the user’s expectations and their previous experience with Skype, or other similar applications which are based on VoIP (Voice translated into data) or IM (Instant messaging) technology.
Task performance with Eye Tracking
In the next part of the session, each participant was asked to complete a set of twelve tasks on Skype, while the eye tracking device recorded their initial eye movements. These were typical tasks based on “real-life” scenarios, for example “Imagine that you want to invite a friend to join Skype. How would you do that?” The user was encouraged to think aloud and share their thoughts as they interacted with the application.
Figure 2. A user testing session at Cimex usability lab
Post-session interview
After the task performance with the eye tracking, the post-session interview was conducted to learn the user’s overall thoughts and conclusions on using the application.
Design
The final section of the session was to gather feedback on the design of the application by capturing the user’s opinion and reactions about the overall “look and feel”, style and colour of the Skype application.
4. Participants
A total of six participants took part in the current testing. The participants were representative of typical Skype users and those who were keen to learn more about Skype. In particular, two of the users had never used Skype before, two had used Skype a few times, and the other two used Skype on a regular basis. The sample included three male and three female, and they were between 25 to 40 years old.
5. Data Analysis
Qualitative analysis was used to analyse the information obtained from the user testing sessions, and gaze plot analysis was used to analyse the data collected with the eye tracker. The data analysis was based on the users’ reactions and comments during the interviews and the interaction with application, as well as on their ability to perform the given tasks.
6. Results
Pre-session Interview
People who had never used Skype before expected to find clear instructions on how to use Skype services, a simple interface and information about the costs.
Free Skype-to-Skype calls and cheap international calls were considered the most popular services by the existing Skype users, whereas conference calls and file transferring were described problematic.
Task Performance
Task performance revealed many important findings which indicated the numerous problematic areas in the presentation, navigation, structure and functionality of Skype 4.0 Beta. Some of the key findings are described below:
Figure 3. Content was overlapped when the window size was decreased.
a. Finding: The size of the application’s window was described as unnecessarily big and the content was overlapped when the users tried to decrease it.
Recommendation: Allow users to select a desirable window size without the content being overlapped.
Figure 4. The conversation window had many unclear icons.
b. Finding: The purpose of many icons was considered not obvious and unclear.
Recommendation: Provide intuitive and distinct icons, and ensure that their purpose is not misinterpreted by looking similar to other buttons.
Figure 5. The search for contacts had no option for advanced searching or sorting the search results.
c. Finding: Users wanted to be able to narrow down the search for contacts and also to sort the search results easily.
Recommendation: Ensure the contacts search facility is effective and provides intuitive advanced search and filtering methods.
d. Finding: It was not clear when a contact was finally added to the user’s list as no confirmation message was given.
Recommendation: Provide appropriate feedback through clearly defined messages to keep users informed about what is going on.
e. Finding: All the users wondered how they could view their credit balance.
Recommendation: Make important information, such as the user’s credit balance, visible and accessible at any point throughout the application. Eye Tracking The image below demonstrates an example scan path extracted by the eye tracking analysis. The blue dots are fixations that indicate where the eyes looked to process information. Larger dots show that more time has been spent looking at the elements on the screen.
Figure 6. An example of an eye tracking scan path for the first page.
f. Finding: During the first seconds of viewing the first page, all of the users’ attention has been attracted by the right area of the screen, without noticing the important options on the left.
Recommendation: Enhance the application’s presentation so that users will not spend unnecessary time and get disappointed while trying to detect the options that will allow them to start using the application.
Post-session interview
Overall the application was considered very useful and easy to use but with a potential for improvement. In general, users wanted a better interface design, clearer navigation mechanisms, and also highlighted the importance of accessing their credit balance at any time. Design Although some users supported that they did not have many design expectations from this kind of applications, a better layout with smaller window size and more intuitive icons was considered necessary by everyone.
Figure 7. Users’ comments on the design of the conversation page.
7. Conclusion
Our usability study demonstrated how the contribution of real users can reveal a number of significant findings and recommendations. The Skype 4.0 Beta 1 was overall described useful and easy to use, but with poor layout, not obvious icons, unclear navigation and information structure. For that reason, an exploration of alternative solutions to the site structure and naming conventions to provide a more intuitive and efficient user experience is considered essential. The outcomes of this study can be used as the foundation for building a more intuitive, engaging and efficient new version of the popular web application.
If you want more information about Cimex's usability lab, contact:
Andrew Japp, Business Development Manager
Cimex, 53-55 Scrutton Street, London UK, EC2A 4PJ
T: +44 (0)20 7324 7780
E: andrew.japp AT cimex DOT com
www.cimex.com
]]>
Having spent three days with Nokia personnel based in Finland, my need to make calls to Finland on my Skype Unlimited World subscription has just increased. And with all the technology development going on in Israel (for instance, iSkoot's and Fring's developers are based in Israel), Israel becomes another high demand country for inclusion in Skype's various Unlimited subscriptions.
(Note: Links in this post will probably go to the pages for Canadian users; simply change your pages via the "change region" drop down box on the leading Skype subscription page for each region.)
Over the past few months there have been a few Skype Journal posts about iSkoot and how they provide mobile device access to Skype's services. Of course, iSkoot also provides the software behind the Skypephone on the 3 network in nine countries. Recently iSkoot has been getting some accolades from third parties:
This is another in my catch-up series (first post here) - and I'm slow in getting a second one out. About three weeks ago Phil and I wrote three posts about SightSpeed's new partnership with Dell where SightSpeed is providing the infrastructure for Dell Video Chat. In my second post I discussed the role CxO's play in bringing business to technology. This triggered SightSpeed CEO Peter Csathy to write a post: Speculation about SightSpeed's Partnership Success - "How Do They Do It?" Peter talks about other partnerships SightSpeed has built over the past five years:
launched today. Anything that helps more projects survive the path from concept to adoption is fantastic. The OWF is tackling several human barriers to technical success. 
Slides from David Recordon's OSCON announcement in Portland, Oregon, this morning. 
while driving in the American South. Friends read chat room comments to him over Skype. You can follow his GPS. If nothing else, it makes the road less lonely. 
While I can find no press release from any of Skype, iSkoot or 3, it appears there was some 3 media event yesterday where it was announced that 3 will be introducing a new Skypephone 2 on August 18. Key new features include:
