Web & Desktop
Testing In a Real-World Environment
Working with some of the biggest, most successful companies on the planet, VML is a full-service digital marketing agency comprised of “thinkers, dreamers, constructors and builders.” With more than 800 employees and offices around the globe, the company specializes in client solutions at the intersection of marketing and technology.
In 2011, as the company began developing a sophisticated web product for one of its high-profile clients, Senior Project Manager Nick Langworth realized that their testing efforts lacked a real-world component, prompting him to look for new ways of getting testing results outside of their internal lab.
"The software was passing all of the major tests internally," said Nick. "But we weren’t entirely pleased with the real-world results. We needed a way to test our software outside of the sterile lab environment – and that’s where uTest came into play."
This case study will detail how VML leveraged uTest for “in-the-wild” testing of a client application, including a summary of their testing requirements, testing process, the role of uTest’s project management team and other aspects of the uTest experience.
As is often the case, VML had found that their test results varied depending on the environment. While the application would pass with flying colors in the lab, the real-world results were a different story. Nick explained that certain variables (OS versions, plug-ins, etc.) would have a major effect on the application’s overall performance. After learning about uTest through one of its own partners, Nick was pleased to learn that “in-the-wild” testing was even easier than he imagined.
"The uTest community was exactly what we were looking for," said Nick. "We needed testers – lots of testers – that weren't affiliated with our partners and who could give us a fresh, objective perspective before we sent this product out to the general public. Basically, we needed to make sure our software worked on real machines."
Working with his uTest project manager, Nick drafted a simple test case that would cover the following areas:
- Exception handling
- Index verification
- Indexing time
After specifying other requirements in terms of Windows® operating systems, Nick’s project was accepted by more than 200 testers from all over the world, who immediately began executing the test case.
After nearly a full week of continuous testing, Nick had received exactly 200 complete test cases – with a 90% success rate. While there were no “showstopper bugs” (i.e. bugs in need of immediate attention), Nick and his team gained some valuable insight into their product.
"The errors discovered were mostly around Outlook® connections and idiosyncrasies with some tester’s hardware," said Nick. "This was mostly due to the various plug-ins they might have had running on their machines. There was nothing major, but it’s still something we wanted to know more about."
After resolving these issues, VML ran a second test cycle – this time with 100 testers – and ended up with a 97% success rate.
Summary and Next Steps
Throughout the entire testing process, Nick and his team would benefit from the aid of their dedicated uTest project manager. Acting as the “gatekeeper” between VML and the tester community, uTest’s project management team helps facilitate the testing process by selecting and communicating with testers, vetting bugs, drafting test cases and performing other essential day-to-day activities.
"Our project manager was great," said Nick. "He was available at all hours of the day and really streamlined the process by vetting bugs, communicating with testers and helping us draft the test cases. We were very impressed."
In the future, Nick said he expects uTest to play a pivotal role in helping them deliver value to their clients.
"This was exactly what we were looking for,” said Nick. "There's no way we could have done this on our own."
"We’re very impressed with the size of the community; it’s incredibly easy to find expert testers anywhere in the world."
-- Nick Langworth, Senior Project Manager, VML