July '99 WebTV Usability Lab tour and presentation
Lyne Plamondon and Elissa Darnell of WebTV

Meeting Notes from BayCHI-UE BOF
WebTV Networks, Palo Alto, Ca, 7/99.

Notes taken by Claudia Case of Alden Case Enterprises, Inc.,
ccase@aldencase.com

In a tent set up behind their labs, Lyne Plamondon and Elissa Darnell of WebTV presented the history and growth of usability at WebTV as well as suggestions for making the most of usability testing. Following the presentation, members of the WebTV usability group led small group tours through their labs. The following paragraphs describe the information presented throughout the evening.

History

WebTV Networks began product R & D in 1995, under a pseudo-company name of Artemis Research. Curious minds who came across their web site would have read that Artemis Research was doing "sleep deprivation research". Lyne and others attest to their being the subjects of these "studies", as they scrambled to get their product to market. In the ensuing years, staff has grown from 5 persons in 1995 to over 600 persons (plus several hundred contractors) in 1999. The usability group has grown from an initial staff of 1 to its current size of 11 plus 2 interns.

The first usability studies were conducted from a room in the Holiday Inn in Dec. 1995. At that time, testing focused only on basic navigation and the use of a TV remote control for navigation. Participants used a product simulator; a product prototype wouldn't be available until later. Recording conditions were, in Lyne's words, "very amateur-ish" with cameras controlled by live operators rather than positional cameras operated by remotes.

In Feb. 1996, Lyne's group conducted the second usability study. This time, their "lab" was located at Nichols Research. Participants were observed as they went through the registration process. It became clear to the usability team that they needed to do a lot more testing. They were finding that a lot of their assumptions weren't true. Also, having a user base which encompasses whole families from tots to grandparents, they found they couldn't take anything for granted regarding users' knowledge of how computers or the Internet worked.

The weekly testing machine

The usability group started weekly testing of their product in the Spring of 1996, moving the "labs" to the downtown Palo Alto facilities. Still somewhat without a real lab, they used 2 conference rooms for testing which they had to share with other employees. By July 1996, two months before their product was launched, they moved once again to a somewhat official but "funky" (Lyne's word) lab consisting of dedicated rooms with no soundproofing making it difficult to conceal observers reactions. Also, the observation room was practically a walk-in closet. They tested 6-8 participants per week, holding a weekly debriefing with the designer/developer at week's end. They wrote up a short weekly report and started producing highlights tapes, showing them at the weekly UI meetings.

Usability test participants include computer novices, Internet novices, regular Internet users and, starting in 1997, current WebTV customers. Their hardest customer to satisfy is the computer novice, as the product functionality must be very explicit. Some statistics about customers purchasing their first WebTV product:

70% are Internet novices
71% don't own a computer
21% haven't ever tried a computer
Average age is 43

Before formal testing, the usability group conducts "peer reviews" which also include an heuristic evaluation. They sit down with the UI designers and walk-thru a product or feature, trying to identify problems before it goes into the usability lab.

Starting in the Summer of 1997, they also began field testing their product in customers' homes.

Customer surveys are also an integral part of their testing strategy.

The first product, Web TV Classic, was launched in Sept. 1997 and even today in their current Charleston Ave. location with bona fide usability labs, they continue to do weekly testing.

They now pair an Usability Engineer with UI design teams; that product becomes the Usability Engineer's primary responsibility. Usability testing is conducted in teams comprised of the primary product usability engineer and another usability engineer. One tester controls the test and interacts with the participants while the other observes the test and records observations in a log.

They continue to produce weekly highlights tapes which are now aired, along with the usability tests, on their own internal cable channel. Most staffers have a TV at their workstations so they can tune in any time. The advantages of having their own usability channel are that it provides an easy way for designers to observe testing without leaving their cubes and it helps keep other people (marketing, project managers, etc.) informed.

Focus: the "whole user experience"
WebTV is concerned about the entire user experience with every aspect of their product. Therefore, testing includes:

Set-up and installation (which testers call the "Out-of-Box experience") Documentation User Interface Phone tree (1 800 Go Web TV) Remote control and keyboard

Their product line now includes a Japanese game product (Web TV for Dreamcast)and their Japanese market has a number of WebTV Classic and Plus users as well. Testing has been extended to Japan, using 2 Usability engineers based there and one based here.

In their recent field testing of the "Out-of-Box" experience for the WebTV product, testers had the objectives of:

1) getting the user's first exposure to the product, before even opening the box.

2)observing installation and setup of WebTV, and

3) assessing the effectiveness of Welcome Kit and installation materials.

They had a number of hurdles to overcome which included recruiting participants through retail channels and having to be prepared to go to a customer's home within 24 hours of a call coming in.

New testing methods

Elissa Darnell talked to the audience about new strategies and methods they are beginning to use at WebTV. One new twist is the addition of what they call "Remote Testing" and "Longitudinal Testing" to complement the current lab and field studies. With the remote testing, participants are drawn from a representative sample of current WebTV customers. Participants' service is upgraded to include the new feature which they will be testing. Tasks are mailed out via email and participants are asked to use the service regularly for about 3 weeks. Finally, online questionnaires are used to collect feedback from the participants. All aspects of this testing are handled electronically, even down to an electronic non-disclosure agreement. Though remote study has advantages of no geographical constraints on sample, long-term use of participants, users' use in a natural setting, and feedback on the content and discoverability of a new feature, it also has some disadvantages. The disadvantages are the results being totally reliant on users' self-reports, reporting of a less rich set of usability problems (most likely because of the data being self-reported), and the limits of using only existing customers. This method has still proven very useful as a complementary method to lab testing. Most recently they used it to get feedback on their latest service release.

In the Longitudinal Study, participants are studied in the lab and at home for extended periods of time. This method combines lab testing strategies with those of field testing. For their first longitudinal study, participants were recruited from among potential WebTV customers. Participants were not asked to perform explicit tasks. After setting up WebTV, they were allowed to use it however they wanted. They were asked to keep audio or written logs. The disadvantages of this approach are the limited sample size due to the duration of the test, the geographic restrictions due to users needing proximity to the lab which may not be a representative sample, and the engineers getting little feedback on infrequently used features since users were not asked to perform explicit tasks. The advantages are feedback on long-term use which one doesn't get in a typical lab study, data on realistic use by participants rather than users doing proscribed tasks, and data on the day-to-day learning process.

Usage data analysis

WebTV's usability engineers have also begun to do Usage Data Analysis. This gives them some objective input on how different features are used, on average. They may be able to use this data to begin building typical profiles of sub-population groups. It complements nicely any self-reporting techniques, such as surveys. The data shows where the user actually clicks which, in turn, reduces some of the questions needed to be asked in a self-report survey (i.e., "Do you use feature X and if so how often do you use it?")

Usage data analysis has the advantages of a large sample size, no time needed to "run" participants through a lab, and it captures actual users and their actual usage. Downsides to this method are that the usability team is unable to discern what was the user's real goal and whether the user achieved his/her goal. Another disadvantage is that the usability engineers can see "what" a customer did but not "why" they did it. This method also isn't the best for capturing usability problems.

Usability lab tour

Following the presentation, members of the WebTV usability group led small group tours through their labs. The tour included demos of their products: Classic, Plus, Satellite box and the Japanese game product. Each group spent some time in the lab "living rooms" and observation room. Lyne and Elissa gave us pointers on the advantages and disadvantages of their current labs. They also described soon-to-be-implemented improvements to the labs which will be installed when they move to new and more spacious accommodations later this year.

Original Announcement

WebTV presents the history and growth of usability at WebTV and suggestions for making the most of usability testing. The meeting includes small group tours through WebTV's usability labs.