Company Size

20+ employees

Company Location

Tel Aviv, Israel

Testing Type

Functional, Exploratory

App Type

Web

Testing Coverage

Global

AOL Answers

Agile Testing With The Crowd

In the year 2006, Avichay Nissenbaum and a small group of Israeli entrepreneurs founded Yedda, an online service that matched answer-seeking users with experts in any given field. Avichay said the group’s thesis was that “questions and answers are essentially the DNA of web content.”

As it turns out, their thesis was spot on. After years of increasing popularity, Yedda was officially acquired by AOL. Today, the company is known as AOL Answers.

During this transition, most of the company’s technology and processes would remain the same. Users would still be able to search thousands of ad hoc Q&A discussions and submit media, images and videos of their own. What would need to change was the company’s testing methodology. As an Agile shop, it was the task of Osher Frimerman to find a testing solution that could keep pace with the company’s aggressive development schedule.

“We sometimes release as often as three times per week,” said Frimerman. “So it was important that we find a way to test as quickly and efficiently as possible. That’s why we chose uTest.”

This brief case study will show how AOL Answers leveraged the uTest community for functional and exploratory testing. Subject matter includes defining the scope of testing, tester selection, project management and other aspects of the uTest experience.

Test Cycle Details and Results

Despite its simple user interface, Frimerman explained that behind the scenes, AOL Answers is actually a very sophisticated application, as it uses an advanced matching algorithm to identify experts. So while students were responsible for moderating the content, uTest would be called on to perform functional and exploratory testing on the application’s more complex features. This would ultimately include social media integration, privacy & security settings, browser compatibility, JavaScript issues, load performance and other areas of concern.

"You wouldn't believe some of the behaviors we observed on these home machines," said Shie. "So when you are testing for performance, it's imperative to know how the software runs outside of the lab environment."

As the testing progressed, Frimerman quickly realized the benefit of having a flexible group of expert testers.

“For every test cycle we asked that at least half of the testers should be new,” said Frimerman. “The reason being was that we wanted at least some of the testers to be having their first experience with our product. That way we could avoid dealing with bugs that had already been reported, while also getting some fresh perspective.”

As expected from such a mature application, testers were not able to find any “showstopper” bugs (i.e. bugs in need of immediate attention). However, Frimerman and his team would approve an average of 50 bugs per test cycle, with each test cycle lasting roughly 9 days in duration.

The Role of Project Management

Throughout the entire process, the team at AOL Answers would benefit from the guidance of their uTest project manager.

To start, Frimerman said that their PM helped them to select testers for each new release; upload written instructions; monitor bug reports and user feedback and act as a liaison between the customer and the testers.

“Our uTest project manager was very responsive,” said Frimerman. “She spent a lot of time helping us find testers, document the known bugs and making sure the testers were all on the same page. She was very helpful.”

Next Steps

While AOL Answers continues to operate within an agile framework, they have scaled back their test cycles from two to one per month since signing on as a uTest customer. Frimerman said the company continues to use uTest for most of their high-profile releases.

"There aren’t many solutions that work for a company with a release schedule that’s as tight as ours. uTest really gets the job done. We’re very happy customers."
-- Osher Frimerman, VP of Research and Development, AOL Answers
Open Announmcement
uTest and Appplause Banner

Heard the news?

In early 2014, uTest will be changing its name to Applause. With an expanded app quality offering including test services, tools, SDKs & integrated analytics, we will do even more to help companies delight their web & mobile users.

The uTest name will live on in our tester community, with an expanded offering for all facets of a tester’s professional life. This includes paid testing projects, hands-on training, networking, events, career resources & more.

While we’ve got a lot of work to do, we’re pumped to tackle both of these bigger, bolder challenges. Get the details on this news here.