Company Size

Large

Company Location

Global

Testing Type

Functional

App Type

Web & Desktop

Testing Coverage

United States

VML

Testing In a Real-World Environment

Working with some of the biggest, most successful companies on the planet, VML is a full-service digital marketing agency comprised of “thinkers, dreamers, constructors and builders.” With more than 800 employees and offices around the globe, the company specializes in client solutions at the intersection of marketing and technology.

In 2011, as the company began developing a sophisticated web product for one of its high-profile clients, Senior Project Manager Nick Langworth realized that their testing efforts lacked a real-world component, prompting him to look for new ways of getting testing results outside of their internal lab.

"The software was passing all of the major tests internally," said Nick. "But we weren’t entirely pleased with the real-world results. We needed a way to test our software outside of the sterile lab environment – and that’s where uTest came into play."

This case study will detail how VML leveraged uTest for “in-the-wild” testing of a client application, including a summary of their testing requirements, testing process, the role of uTest’s project management team and other aspects of the uTest experience.

Testing Challenges

As is often the case, VML had found that their test results varied depending on the environment. While the application would pass with flying colors in the lab, the real-world results were a different story. Nick explained that certain variables (OS versions, plug-ins, etc.) would have a major effect on the application’s overall performance. After learning about uTest through one of its own partners, Nick was pleased to learn that “in-the-wild” testing was even easier than he imagined.

"The uTest community was exactly what we were looking for," said Nick. "We needed testers – lots of testers – that weren't affiliated with our partners and who could give us a fresh, objective perspective before we sent this product out to the general public. Basically, we needed to make sure our software worked on real machines."

Working with his uTest project manager, Nick drafted a simple test case that would cover the following areas:

  • Installation
  • Exception handling
  • Index verification
  • Indexing time

After specifying other requirements in terms of Windows® operating systems, Nick’s project was accepted by more than 200 testers from all over the world, who immediately began executing the test case.

Testing Results

After nearly a full week of continuous testing, Nick had received exactly 200 complete test cases – with a 90% success rate. While there were no “showstopper bugs” (i.e. bugs in need of immediate attention), Nick and his team gained some valuable insight into their product.

"The errors discovered were mostly around Outlook® connections and idiosyncrasies with some tester’s hardware," said Nick. "This was mostly due to the various plug-ins they might have had running on their machines. There was nothing major, but it’s still something we wanted to know more about."

After resolving these issues, VML ran a second test cycle – this time with 100 testers – and ended up with a 97% success rate.

Summary and Next Steps

Throughout the entire testing process, Nick and his team would benefit from the aid of their dedicated uTest project manager. Acting as the “gatekeeper” between VML and the tester community, uTest’s project management team helps facilitate the testing process by selecting and communicating with testers, vetting bugs, drafting test cases and performing other essential day-to-day activities.

"Our project manager was great," said Nick. "He was available at all hours of the day and really streamlined the process by vetting bugs, communicating with testers and helping us draft the test cases. We were very impressed."

In the future, Nick said he expects uTest to play a pivotal role in helping them deliver value to their clients.

"This was exactly what we were looking for,” said Nick. "There's no way we could have done this on our own."

"We’re very impressed with the size of the community; it’s incredibly easy to find expert testers anywhere in the world."
-- Nick Langworth, Senior Project Manager, VML
Open Announmcement
uTest and Appplause Banner

Heard the news?

In early 2014, uTest will be changing its name to Applause. With an expanded app quality offering including test services, tools, SDKs & integrated analytics, we will do even more to help companies delight their web & mobile users.

The uTest name will live on in our tester community, with an expanded offering for all facets of a tester’s professional life. This includes paid testing projects, hands-on training, networking, events, career resources & more.

While we’ve got a lot of work to do, we’re pumped to tackle both of these bigger, bolder challenges. Get the details on this news here.