Microsoft® SE

“Assumptions become reality after a while,” said Shie Erlich. “And before long, you stop questioning them altogether.”

As Test Lead for Microsoft Security Essentials – the company’s no charge anti-virus software – Shie refused to assume the testing procedures in place were sufficient on their own. And as one of the more anticipated Microsoft projects in recent memory, he knew that any bugs found post-release could instantly, perhaps permanently, harm the app’s reputation with consumers.

With that mindset, and with the official release date for Microsoft Security Essentials fast-approaching, Shie set out to make expanded testing coverage a reality - signing on with uTest in January 2009.

But with tens of thousands of beta users, a rolodex of outsourcers and one of the top in-house labs on the planet, how could the Microsoft Security Essentials team possibly improve their testing coverage? Where were the dark spots? And how would uTest help shine light on them?

This case study will detail Microsoft’s use of the uTest community, demonstrating how five distinct test cycles - performed by professional testers from around the world - enabled the Microsoft Security Essentials team to successfully launch their product with an even higher degree of confidence.

Getting Started: Selecting Testers, Defining Scope
With the help of a uTest project manager (there’s one assigned to every uTest customer), Shie’s first order of business was to select a testing team and define the testing scope in a way that complimented their existing QA efforts. Since Microsoft Security Essentials would be released to a global audience, he needed to validate Microsoft Security Essentials’ use in countries like Brazil, China, India and Russia, as well as the US and Israel.

“You wouldn’t believe some of the behaviors we observed on these home machines,” said Shie. “So when you are testing for performance, it’s imperative to know how the software runs outside of the lab environment.”