Monday, 27 August 2018
Any organisation involved deploying an IT System to solve a business problem will, at some point, have to consider how much testing and indeed what testing is needed. Often, at the lower levels of testing such as Component Testing or Unit Testing or even at a higher levels such as System Testing, the answer is fairly clear, including how formal it should be. At these levels we may find it easier to determine the level of formality needed but when it comes to User Acceptance Testing how formal should this be?
We know that User Acceptance Testing (UAT) is taking us into an area where ‘rubber meets the road’ as they say. Beyond this point comes actual deployment, which may initially be on a small scale followed by a rapid ramp up across the user base.
The attraction of a formal process for UAT, with software proven to comply largely to a set of formal requirements and which also integrates with a (possibly new) business process, is attractive as it provides a sense of control. However, is this what is really required and how achievable is it?
UAT variety
The answer to the question of how much formality is needed is one of context. I have been involved in systems where UAT is formally defined in a contract with a supplier, with milestone payments based on everything from the production and agreement of formal UAT specs to actually running of tests to demonstrate the system works as agreed and agreement on the system being ‘fit for purpose’ (ie, suitable for training staff on and deploying across an enterprise).
I have equally seen systems accepted with users spending an afternoon having a quick run through the main features and giving the system the ‘thumbs up’ or ‘thumbs down’.
One system I worked on was quite complex, took nine months to develop but was put through an informal acceptance process and rejected outright as not suitable; although implemented based on an agreed Requirements Spec it was not was wanted!
A more formal approach may well have have been better in this case. What then is the key to how much formality should be applied?
Context, purpose and more
The answer lies in what may be called the ‘Context’; by this I am talking about what the app or software does (its purpose), how it is used and by whom. Serious consideration must be given (before development starts) to the implications of any kind of failure of the system in the live operational environment. These implications may range from no impact or a minor irritation to a user, to a financial impact on a revenue stream for a business, or worse, death or injury of a person!
Compare the context of local authority website advertising activities to do over the summer holidays and the implication of the software failing, with the failure of software used by pilots on a commercial airline. Clearly the latter demands significant formality with process, documentation, change control, and traceability, which can be subjected to an audit to demonstrate compliance to specific industry standards.
Risk matters
Risk and the management of risk also play a major part in the formality required in UAT. If the impact of failure once the system is live is high, from a business or personal perspective, then the way forward should be a risk averse approach with a controlled process driven UAT phase.
The reality is, this is only normally achievable if a risk averse mindset of the business stakeholders has been applied throughout the project lifecycle; if not, UAT will be a major gating process and the system may never be deployed without major rework or may just never reach deployment!
The importance of documentation
When talking formality, the expectation is that there are written requirements (including User Stories), which have been reviewed by the UAT team in advance. Formal Test Plans are produced with agreement on specific Acceptance Criteria (including performance aspects) which must be met.
All testing is documented and there is clear change control and traceability from UAT results to requirements and business process. Adopting this approach normally requires trained testers, including staff from the business who may need to undertake some formal UAT training.
Reducing maintenance costs through test formality
From a cost perspective, it has been estimated that the 20% of the lifetime cost of a system is expended during development whilst 80% occurs post ‘go live’ (often called the ‘Maintenance Phase’). Thus, even if the system is not safety critical, building some formality such as implementing elements of the Fundamental Test Process and associated artefacts such as test plans, test cases and test scripts and incident report can help reduce the seriously reduce ‘maintenance burden’ and reduce the lifetime cost of a system.
Informality and quality risks
I realise that some systems may be developed with little formality and UAT can be carried out informally; this may be perfectly adequate provided the stakeholders are content with the lower level of quality this affords and an informal approach is adequate for the context.
However it is realistic to expect a higher level of support once a system is live plus the possibility of irritation from real users discovering the bugs (by indirectly testing) the system. If an informal approach to UAT is adopted, it is always good idea to have multiple acceptance stages.
The first stage should be partial acceptance, followed by a limited release to users (effectively a form of beta testing). It is important to ensure all user classes are covered and the users attempt to do their actual work with this release of the system. This approach can de-risk the system significantly.
After the period of running with actual users, a business and IT review should then be held and all problems raised by users discussed (particularly from the perspective of ‘impact on businesses'). This review can be the basis for a ‘go/no go’ meeting to agree acceptance and subsequent roll out (or not).
Formal vs informal?
So, when it comes to formality should UAT be ‘black tie’ or ‘jeans and T shirt’, the answer is that it all depends on context and risk.
Steve Helsby,
Senior Trainer
