Testing the Limits of Mobile Innovation
For nearly thirty years, USA TODAY has been at the forefront of innovation in the news & media industry across print, web and now mobile platforms. The first national paper to introduce color and one of the first publications to establish an active web presence in the mid-1990s, USA TODAY was also among the first in its class to launch native applications for the iPhone, iPad, Android and the other mobile operating systems.
The company’s dedication to mobile quality has clearly paid off: As of December 2011, USA TODAY saw aggregated mobile page views exceed web page views for the first time in its history.
But with greater success, comes greater responsibility. As USA TODAY began to support more mobile platforms and devices in an increasingly fragmented environment, key players within the company realized the need for an efficient, scalable way to increase their testing capabilities. Tim Carlson - Director of Mobile Product Development and Operations – began this search with some key criteria in mind.
"We needed a QA and testing solution that could be a part of our in-house team," said Tim, who started his IT career as a tester. "We have such a close knit team and we wanted to keep that culture, so traditional outsourcing was not even an option. Of course, we also needed mobile experts who could hit the ground running. Mobile is a fast-growing space, but there are surprisingly few people who specialize in the testing side of the equation. Those were the main reasons why we chose uTest."
Mobile Testing Challenges
In the world of mobile, the meaning of “quality” can often be ambiguous. This is not the case with USA TODAY, which takes a very quantitative approach to the term.
"App store ratings are extremely important to us. You can launch a beautifully designed native application, but if it crashes, then it will receive a poor rating and users will go elsewhere. Our goal is to launch nothing short of a 4.5 star app every time - no exceptions. Anyone can read your app store rating. There’s no way to hide poor quality in the world of mobile."
-- Michael Croghan, Mobile Solutions Architect, USA TODAY
Another fact of life in the world of mobile, as Tim explained, is tight deadlines. Case in point: In the summer of 2011 USA TODAY began developing a native application for the soon-to-be released Kindle Fire. Their internal deadline coincided with the device’s official launch date, meaning their timeframe for developing, testing and deploying was extremely short. This is a common, but welcomed challenge for Tim and his team.
"It’s important for us to have a presence on new mobile platforms from day one,” explained Tim. “With uTest, we were able to start testing the Kindle app, for instance, with testers all over the world almost immediately. Making a deadline like that without uTest would have been quite a challenge."
In addition to uTest – which would test their application with professionals all over the globe – Tim also hired two in-house mobile testing experts (one of whom happened to be a top member of the uTest community).
Mobile Testing Challenges
With the help of their dedicated uTest project manager (there’s one assigned to every customer account) USA TODAY kicked things off by specifying the objectives of their testing project. (Note: Details included here reflect numerous test cycles for Android Kindle Fire, iPhone and iPad applications.) Common criteria included:
- Tester Location: USA TODAY routinely leverages testers from all over the globe, including Greece, Singapore, India, Canada, Spain, Israel, UK, US and other locations.
- Test Team Size: Some test cycles with broader objectives, such as the iPad application, involved as many as 60 testers. Other projects, where the scope was much tighter, required smaller teams of 10-12 testers.
- Device Coverage: Where fragmentation is an issue (i.e. Android devices) USA TODAY works with a project manager to ensure that testing is extended to a wide-range of the most popular devices and operating systems.
- Testing Scope: USA TODAY would specify the testing scope via detailed test cases. In the absence of test cases, uTest would conduct “exploratory testing” where testers were free to search for issues at their own discretion
Real World Results
Almost immediately, the team at USA TODAY recognized that uTest would forever change the way they approached testing and quality assurance. This was attributed to:
- The quality of testers in the uTest community
- The flexibility of the uTest platform
Quality of Testers
As alluded to earlier, a major challenge for USA TODAY (indeed, for all companies with a significant mobile presence) is the lack of mobile testing talent. With uTest’s community of 50,000+ testers from 180 countries, finding talented testers that matched their specific needs was anything but problematic. Aside from the coverage, USA TODAY was impressed with the quality of bug reporting and user feedback.
'The video [bug reports] were great," said Tim. "If you can see a tester actually walk-through a bug on video, then it’s much easier to diagnose and fix. And with this method, it’s much easier to prove that an issues exists to the developer, since there is no ambiguity with video. It’s a smarter, more transparent way to diagnose and fix issues. The quality of the bug reports exceeded my highest expectations."
The uTest Platform
"Members of our team could log-in anytime to see the latest bug reports, user reviews, test cases and test scripts," said Tim. "That was instrumental in helping us all stay involved in the testing process. This became even more critical for us as we started to test more applications on a much more frequent basis."
In less than four years, USA TODAY went from managing a single iPhone application to now managing dozens of native apps (and versions) for iOS, Android, Windows Phone, tablets and eReaders, in addition to the mobile web. In other words, they don’t expect testing to get any easier. As such, they expect uTest to continue playing a role in helping them launch world-class mobile and web applications.
"uTest always provides a fresh perspective on our applications," said Tim. "There’s a lot of value in this model of testing, so we’re already discussing ways in which uTest can help in areas besides mobile. It’s been a great experience all around."
"At the end of the day, uTest gives our organization a much better sense of where we stand in terms of application stability and performance. For a group that iterates as often as we do, that type of insight is critical."