This view can limit the testing and fails to grasp the point that testing a mobile app is in essence ‘product testing’. The rational for this limitation is based on the fact that the mobile devices both the hardware and OS) are consumer products and have been tested and work, so it follows that no testing is needed regarding the device itself. As for the software application, it may currently work in a client server environment with networked PC’s so has to be ‘ported’ to a mobile environment. Once migrated to the mobile environment, the testing should just focus on functionality plus some basic performance testing, once done, then the app is deemed to be working OK, in other words ‘job done’, simple eh!
It is correct to assume that functional tests will need to be run and so will some performance tests but this mind-set is limited as it reduces the device to being a mere portal and can result in a ‘tested’ system may failing to meet the user’s needs despite passing all the defined functional and non-functional tests.
The correct approach is go beyond seeing the mobile device as ‘variant’ on a desktop PC and consider the combination of the application and the device as a single ‘product’ which a real person will use almost as a tool to carry out a task of some sort in a variety of changing environments.
As an example, I travel around the UK frequently, primarily by train so make use of one of the train booking applications on an iPhone. Often the environment I am in may be changing, typically I may be trying to book something, change something or just check which platform to use as I am getting in and out of taxis whilst juggling a laptop and suitcase at the same time. In this real environment I will be using the app possibly while in a hurry to check some aspect of my journey on a train which may be departing in 5 minutes time. In this scenario I assume functionality is fine, but what about Usability (typing with my thumb) and needing key information on a single screen, viewing the screen at odd angles, hoping I have enough battery left (and that the app minimises battery drain) as well as intermittent connectivity. I may get a text, a notification (alert) from another app or even a phone call which I want to ignore. The key thing is that none of these and many other ‘interrupts’ should negatively impact the running app and cause it to freeze, lock up or just crash as I search for key information under pressure.
Another example may be an app which uses the phone’s hardware (motion sensors and GPS) to track how far I have walked and provides me with ‘coaching information’ and a ‘motivational message’ whilst I am walking’ as well as playing my favourite music. Any motivational audios which cut in at key points in time or distance (after 10 minutes of walking say or after 5K of distance) should result in a fade out of music whilst the motivational message plays and a fade back in of music after the message ends.
The underlying architecture of mobile devices can generate a plethora of interrupts from hardware (such as motion, light and sound sensors) and software (the OS or indeed other apps); testing how the app behaves whilst interacting with the device and other apps is critical to the app being usable.
In essence when viewing the app with the device as a ‘product’ then the product has to be tested for the variety of input methods and keyboards (taking photos is a type of input method). Testers will need to test from an orientation perspective (as when out and about I may frequently change between landscape and portrait) and how good a ‘neighbour’ the app is (does it excessively use the battery, memory or other ‘shared resources’) plus many other aspects. UI and Usability testing will have to feature strongly in testing any mobile app. Field Testing where specific tests are designed to test the app whilst on the move may need to be designed and carried out as well as testing for a variety of types of networks connectivity and switching between these.
Thus along with all the usual functional and non-functional tests for a mobile app there are a variety of other test types which really test the combination of the app as a piece of software interacting with the hardware and software of the device it is installed on (and other apps on device which may generate interrupts or interfere with a shared resource such as a camera).
The real danger of limiting testing to just functionality via a mobile portal, mentioned at the start of this blog is that not only is it a partial test but it will result in insufficient time and resource being planned for testing, particularly if the limited viewpoint is one held by line managers or project managers who are directly involved in provide estimates for the work.
It should come as no surprise, that the ISTQB Foundation Course for Software Testing describes four test types whilst the Mobile Apps Foundation Course talks about 16 test types (including such things as battery testing and installation testing). The sum of the parts does not equal the whole and a holistic approach to mobile testing where the app plus device are viewed as an integrated product is the right way forward when planning and undertaking mobile apps testing.
Steve Helsby, Senior Trainer
To discuss mobile app testing, or to book Certified Mobile Apps Professional (CMAP) course, please contact us.