Palo Alto, CA
Test, Test, Release
It didn't take long for Tapulous to conquer the iPhone with Tap Tap Revenge. In less than two years, they had become a household name among iPhone users and the envy of mobile app companies worldwide.
There was definitely some luck involved, said VP of Engineering Jess Kahn, but their success has been anything but a fluke. The Dartmouth grad, who spent ten years at Apple as a software engineer, credits their good fortune to a combination of talented personnel, calculated execution and impeccable timing.
Of course, there were plenty of challenges along the way. Like other mobile app companies, Tapulous would have to find an efficient way to consistently test their latest versions. With tight deadlines and limited in-house resources, using standard testing methods would ultimately prove to be an exercise in futility.
“We wanted to avoid falling into the trap of testing ourselves,” said Jess. “Otherwise, it’s just another task piled on to an 80-hour work week. And managing an offshoring firm, for a company our size, was totally out of the question.”
This case study will demonstrate why Tapulous turned to uTest for their mobile app testing needs - covering the results of their exploratory and functional test cycles; the role of uTest’s project manager; the limits of beta testing and more.
Tapped Out: The Limits of Beta Testing
“Our target users are teenage boys with no formal tester training,” explained Jess. “So as beta testers, they are likely to give us defects that don’t contain very much detail. They tend to point out nuances with the in-game experience – timing issues, high score discrepancies and things like that. It’s certainly valuable feedback, and we are lucky to have it, but unfortunately it’s not a substitute for professional testing.”
Since offshoring was “out of the question” Jess would need to find a way to ensure testing coverage that would not burden her in-house resources any further. And with another version of Tap Tap Revenge schedule for release in the weeks ahead, she would need to find such a solution quickly.
“I decided to spend a day doing research,” said Jess. “And it didn’t take me long to find uTest. They were headquartered in the US, and didn’t require us to spend much time or money to get started, so we gave it a shot.”
Getting Started with uTest
After signing up, Tapulous was immediately assigned a dedicated project manager who would introduce them to the uTest platform and procedures. In preparation of their initial exploratory testing cycle, their uTest PM assisted them in uploading testing requirements by OS, device, tester experience and other criteria.
“It only took us about a half day to get started with uTest,” said Jess. “There’s no way we could have done this in-house. Not with our headcount and certainly not with our testing matrix.”
As for their matrix, it was particularly important for Jess to expand their testing coverage by device – a task the uTest platform made incredibly easy via a database that is constantly updated.
"What I really love about uTest is that they can help us test on devices we don’t have in-house. Everyone in our office always has the latest and greatest iPhone devices, with the latest software. uTest supplying testing on the rest of the matrix allows us to discover bugs that would have otherwise gone unnoticed."
-- Jess Kahn, VP of Engineering, Tapulous
Multi-Player: The Role of uTest’s Project Managers
“uTest’s project management team has played an enormous role in our success,” said Jess. “They are available on IM at all hours of the day and they are always there when you need them.”
As the constant point of contact, uTest’s project managers are there to assist customers with every step of the testing process. This includes – but is NOT limited to – tasks such as:
- Tester selection/communication
- Defining requirements/test case creation
- Bug-tracking integration
- Bug approval/rejection
“No matter what type of notice I give in advance, uTest is always ready to go,” said Jess.
During their initial exploratory test cycles - each lasting anywhere from three days to a month - testers were asked to search Tap Tap Revenge at their own discretion. This step is common for new uTest customers, as it allows testers to familiarize themselves with the application before more in-depth testing is performed.
The next step for Tapulous was to perform functional test cycle, where the uTest community would test specific features of their application. As a result, Jess was able to compile a ‘known bugs’ list and work towards resolving a variety of critical issues before their next release.
“Could we have found these bugs ourselves?” asked Jess. “Yes, but we don’t have the time or a team to do so as efficiently as the uTest community. As our applications progress, so too does our team of testers. In fact, I don’t think we’ve lost a tester yet.”
Although Jess plans on hiring an in-house testing manager, she said one of their duties will be to act as the primary liaison with uTest. Indeed, she sees uTest as a major part of their success moving forward.