Global Users, Global Testing
"Assumptions become reality after a while," said Shie Erlich. "And before long, you stop questioning them altogether."
As Test Lead for Microsoft Security Essentials – the company's no charge anti-virus software – Shie refused to assume the testing procedures in place were sufficient on their own. And as one of the more anticipated Microsoft projects in recent memory, he knew that any bugs found post-release could instantly, perhaps permanently, harm the app's reputation with consumers.
With that mindset, and with the official release date for Microsoft Security Essentials fast-approaching, Shie set out to make expanded testing coverage a reality - signing on with uTest in January 2009.
But with tens of thousands of beta users, a rolodex of outsourcers and one of the top in-house labs on the planet, how could the Microsoft Security Essentials team possibly improve their testing coverage? Where were the dark spots? And how would uTest help shine light on them?
This case study will detail Microsoft's use of the uTest community, demonstrating how five distinct test cycles - performed by professional testers from around the world - enabled the Microsoft Security Essentials team to successfully launch their product with an even higher degree of confidence.
Getting Started: Selecting Testers, Defining Scope
With the help of a uTest project manager (there's one assigned to every uTest customer), Shie's first order of business was to select a testing team and define the testing scope in a way that complimented their existing QA efforts. Since Microsoft Security Essentials would be released to a global audience, he needed to validate Microsoft Security Essentials' use in countries like Brazil, China, India and Russia, as well as the US and Israel.
"You wouldn't believe some of the behaviors we observed on these home machines," said Shie. "So when you are testing for performance, it's imperative to know how the software runs outside of the lab environment."
"The great thing about Microsoft Security Essentials is the subtlety. There are no pop-ups, no messages trying to sell you something – nothing that's in your face. It was designed to be quiet and non-intrusive, which makes it drastically different than other anti-virus products on the market today." -- Shie Erlich, Testing Manager, Microsoft
For example, Shie pointed out that many Chinese users don't rely on a lone anti-virus program. Thus, the Microsoft Security Essentials team had to verify that their product functions concurrently with other such applications. In order to ensure its real-world compatibility (along with certain usability issues unique to China), he needed testers who actually resided in China.
With uTest, Shie was able to add these testers (along with dozens from other countries) who had the specific OS, browsers and plug-ins that he needed. From there, he simply uploaded a list of known bugs, and the initial testing cycle was set to kickoff. Unlike an outsourcing firm, the on-ramping period with uTest is anything but complicated.