Company Size

Enterprise

Company Location

Global

Testing Type

Functional, Exploratory

App Type

Desktop

Testing Coverage

10+ countries

Microsoft SE

Global Users, Global Testing

"Assumptions become reality after a while," said Shie Erlich. "And before long, you stop questioning them altogether."

As Test Lead for Microsoft Security Essentials – the company's no charge anti-virus software – Shie refused to assume the testing procedures in place were sufficient on their own. And as one of the more anticipated Microsoft projects in recent memory, he knew that any bugs found post-release could instantly, perhaps permanently, harm the app's reputation with consumers.

With that mindset, and with the official release date for Microsoft Security Essentials fast-approaching, Shie set out to make expanded testing coverage a reality - signing on with uTest in January 2009.

But with tens of thousands of beta users, a rolodex of outsourcers and one of the top in-house labs on the planet, how could the Microsoft Security Essentials team possibly improve their testing coverage? Where were the dark spots? And how would uTest help shine light on them?

This case study will detail Microsoft's use of the uTest community, demonstrating how five distinct test cycles - performed by professional testers from around the world - enabled the Microsoft Security Essentials team to successfully launch their product with an even higher degree of confidence.

Getting Started: Selecting Testers, Defining Scope

With the help of a uTest project manager (there's one assigned to every uTest customer), Shie's first order of business was to select a testing team and define the testing scope in a way that complimented their existing QA efforts. Since Microsoft Security Essentials would be released to a global audience, he needed to validate Microsoft Security Essentials' use in countries like Brazil, China, India and Russia, as well as the US and Israel.

"You wouldn't believe some of the behaviors we observed on these home machines," said Shie. "So when you are testing for performance, it's imperative to know how the software runs outside of the lab environment."

"The great thing about Microsoft Security Essentials is the subtlety. There are no pop-ups, no messages trying to sell you something – nothing that's in your face. It was designed to be quiet and non-intrusive, which makes it drastically different than other anti-virus products on the market today." -- Shie Erlich, Testing Manager, Microsoft

For example, Shie pointed out that many Chinese users don't rely on a lone anti-virus program. Thus, the Microsoft Security Essentials team had to verify that their product functions concurrently with other such applications. In order to ensure its real-world compatibility (along with certain usability issues unique to China), he needed testers who actually resided in China.

With uTest, Shie was able to add these testers (along with dozens from other countries) who had the specific OS, browsers and plug-ins that he needed. From there, he simply uploaded a list of known bugs, and the initial testing cycle was set to kickoff. Unlike an outsourcing firm, the on-ramping period with uTest is anything but complicated.

Open Announmcement
uTest and Appplause Banner

Heard the news?

In early 2014, uTest will be changing its name to Applause. With an expanded app quality offering including test services, tools, SDKs & integrated analytics, we will do even more to help companies delight their web & mobile users.

The uTest name will live on in our tester community, with an expanded offering for all facets of a tester’s professional life. This includes paid testing projects, hands-on training, networking, events, career resources & more.

While we’ve got a lot of work to do, we’re pumped to tackle both of these bigger, bolder challenges. Get the details on this news here.