Tel Aviv, Israel
Agile Testing With The Crowd
In the year 2006, Avichay Nissenbaum and a small group of Israeli entrepreneurs founded Yedda, an online service that matched answer-seeking users with experts in any given field. Avichay said the group’s thesis was that “questions and answers are essentially the DNA of web content.”
As it turns out, their thesis was spot on. After years of increasing popularity, Yedda was officially acquired by AOL. Today, the company is known as AOL Answers.
During this transition, most of the company’s technology and processes would remain the same. Users would still be able to search thousands of ad hoc Q&A discussions and submit media, images and videos of their own. What would need to change was the company’s testing methodology. As an Agile shop, it was the task of Osher Frimerman to find a testing solution that could keep pace with the company’s aggressive development schedule.
“We sometimes release as often as three times per week,” said Frimerman. “So it was important that we find a way to test as quickly and efficiently as possible. That’s why we chose uTest.”
This brief case study will show how AOL Answers leveraged the uTest community for functional and exploratory testing. Subject matter includes defining the scope of testing, tester selection, project management and other aspects of the uTest experience.
Test Cycle Details and Results
"You wouldn't believe some of the behaviors we observed on these home machines," said Shie. "So when you are testing for performance, it's imperative to know how the software runs outside of the lab environment."
As the testing progressed, Frimerman quickly realized the benefit of having a flexible group of expert testers.
“For every test cycle we asked that at least half of the testers should be new,” said Frimerman. “The reason being was that we wanted at least some of the testers to be having their first experience with our product. That way we could avoid dealing with bugs that had already been reported, while also getting some fresh perspective.”
As expected from such a mature application, testers were not able to find any “showstopper” bugs (i.e. bugs in need of immediate attention). However, Frimerman and his team would approve an average of 50 bugs per test cycle, with each test cycle lasting roughly 9 days in duration.
The Role of Project Management
Throughout the entire process, the team at AOL Answers would benefit from the guidance of their uTest project manager.
To start, Frimerman said that their PM helped them to select testers for each new release; upload written instructions; monitor bug reports and user feedback and act as a liaison between the customer and the testers.
“Our uTest project manager was very responsive,” said Frimerman. “She spent a lot of time helping us find testers, document the known bugs and making sure the testers were all on the same page. She was very helpful.”
While AOL Answers continues to operate within an agile framework, they have scaled back their test cycles from two to one per month since signing on as a uTest customer. Frimerman said the company continues to use uTest for most of their high-profile releases.
"There aren’t many solutions that work for a company with a release schedule that’s as tight as ours. uTest really gets the job done. We’re very happy customers."
-- Osher Frimerman, VP of Research and Development, AOL Answers