Test documentation - IEEE 829 the standard for Software testing Documentation
Standards?
“Why standards? The use of standards simplifies communication, promotes consistency and uniformity, and eliminates the need to invent yet another (often different and even incompatible) solution to the same problem. Standards, whether ‘official’ or merely agreed upon, are especially important when we’re talking to customers and suppliers, but it’s easy to underestimate their importance when dealing with different departments and disciplines within our own organisation. They also provide vital continuity so that we are not forever reinventing the wheel. They are a way of preserving proven practices above and beyond the inevitable staff changes within organisations.” [Ed Kit - Software Testing in the Real World]
That paragraph neatly and [quite] succinctly describes WHY standards in general. How does that affect testing practitioners who live, as in the title of Ed Kit’s book, in the real world? Anything that promotes better project communication has to be good for testers. So my first point is that standards have to be effective, and produce recognisable (and measurable?) gains, while not adding disproportionate overheads. I once worked for a large organisation that had an internal (and mandatory) standard for almost all documents. This was such that a document of 200 real words was turned into 18 pages, when all the necessary parts (‘glossary’, ‘associated documents’) were added. Perhaps this was counterproductive and unnecessary in this instance.
IEEE 829 – Overview
There have been diverse document types used in software testing, developed in many cases for the needs of a particular organisation. IEEE 829 (1983) is the Standard for Software Test Documentation, and this was an attempt to pull sources together and present some best practice ideas. The standard was revisited, and revised in 1998. Note that the standard applies to any level of testing that may take place [including Acceptance testing], although there may be less obvious application in agile development methodologies. It is usual to have ‘a full set’ of IEEE 829 documents for each testing stage that is being undertaken.
IEEE 829 is often thought of as being the standard for a High Level Test Plan or Master Test Plan (HLTP or MTP). It is more than this, as the standard describes EIGHT documents that can be produced, as part of the testing effort. These documents are sometimes distributed between different categories, although there is no consensus on the subdivisions. I find the following partitioning helpful:
Test Planning
Test Plan
Test Specification
Test design specification
Test case specification
Test procedure specification
Test Reporting
Test Item transmittal report
Test log
Test incident summary
Test summary
The Eight parts
Most of these eight document types are well known; I will provide a very brief summary, before returning to the test plan.
Test Plan A high level view of how testing will proceed; WHAT is to be tested, by WHOM, HOW, in what TIME frame, to what QUALITY level.
Test Design Spec Details the test conditions to be exercised, with the expected outcome (in general terms).
Test Case Spec Specific data requirements to run tests, based upon the test conditions identified
Test Procedure Spec Describes how the tester will physically run the test, including set up procedures. The standard defines ten procedure steps that may be applied when running a test.
Test Item Transmittal The recording of when individual items to be tested have been passed from one stage of testing to another. This includes where to find such items, what is new about them, and is in effect a warranty of ‘fit for test’.
Test Log Details of what tests were run, by whom, and whether individual tests passed or failed.
Test Incident Summary Details of instances where a test ‘failed’ for a specific reason.
Test Summary The Test Summary brings together all pertinent information about the testing, including the number of incidents raised and outstanding, and crucially an assessment about the quality of the system. Also recorded for use in future project planning is details of what was done, and how long it took.
This document is important in deciding whether the quality of the system is good enough to allow it to proceed to another stage. This assessment is based upon detailed information that was documented in the Test Plan.
Test planning revisited
Test planning is a key activity in any software testing project, and for that reason many would associate IEEE 829 ONLY with test planning. “To fail to plan is to plan to fail”. The standard defines 16 items that should be considered for an MTP. This includes the key activities of estimation (as ‘schedule’ is one of the 16) and risk, both of which are large topics in their own right.
The 16 are given below, for completeness, with a well-known mnemonic for remembering the list although textbooks, and web sites that are commonly used have the 16 items listed, with more detail for each.
S Scope Test items, What to test, what not to test
P People Training, Responsibilities, Schedule
A Approach The approach that will be taken to testing
C Criteria Entry / Exit Criteria, Suspension / Resumption Criteria
E Environment Environment needs
D Deliverables What is being delivered as part of the test process
I Incidentals Introduction, Identification (of the document), Approval authorities
R Risks Risks and contingencies
T Tasks The test tasks that are involved in the testing process.
It is worth noting at this point that the standard lists as ‘deliverables’ the seven other document types that perform part of the standard. Some organisations add to this basic list, including key items such as ‘glossary’, and ‘references to other documents’. I usually keep MTP documents, both from other projects within the same firm, and from previous firms, so that I can look back and see specific details that were included.
MTP is a LIVING document
The document specifies what is going to be done, and how it is going to be done. It needs to be published, to appropriate people, to make others aware of what is going to be tested, and what is not going to be tested. However, don’t wait for everything to be completed, before the document is circulated for comment and / or review. The MTP will change during the life of the project. This does not mean that it is not necessary to get individual and departmental sign-off; sign-off is achieved on the basis of what is known at a point in time. In one organisation, sign-off is achieved by stating that unless this is received by a specified (and realistic) date, it will be assumed. It is remarkable how that concentrates the minds of those concerned.
Two areas that indicate the dynamic nature of the MTP concern schedules, and risks. During the testing phase, good news and bad news can occur, and this can change priories. Does that mean that the original MTP was wrong? No; the MTP is what its name suggests, just a plan. At the time, it was based on the best available information, incomplete though this was. Information will be improved as testing progresses, and what was time a critical risk has now been addressed (by third-party security testing, for example). The risk is now answered, and will possibly require no further action.
Review the document
The MTP needs to be reviewed, with review taking place face-to-face. If it is contentious, points of conflict need to be talked through. The MTP is not just “owned” by the testing team(s), but developments groups and users can significantly contribute to clarification, and suggest the addition of other items. What is to be tested, and what not to be tested are two KEY elements in the MTP. One example dates back to work undertaken in October 2002, where testing was pushed for time (as always). The MTP specified that significant testing would concentrate on the retail system, with respect to 53-week year processing (2002 – 2003 is a 53-week year). The development team had not realised the significance of 53-week years (that it was THIS year), and merely the insertion of the testing intention resulted in better code (development extended unit test coverage, found some problems and implemented fixes).
It is usual for the detail listed in the MTP to be used as a basis for deciding whether the software under test is suitable for the next stage of testing, deployment to production, etc. Therefore key individuals need to see this detail, and agree, before the crunch implementation meeting!
Face Reality
The MTP is one place where testing faces reality. The MTP is not free-standing, but fits into the overall Test Strategy. In some ways, it is not a prescriptive approach, but more of a check list, to remind those responsible what should be CONSIDERED to go into the MTP. The only prescriptive thing about it is the use of the 16 point “check-list”. It is perfectly OK to not include one of the 16 points – as long as the reason why that has been excluded are listed (and agreed by the reviewers of the MTP). Risks and assumptions are also included in the MTP; sometimes the explicit stating of a risk or assumption can promote lively discussion, and even resolution!
Relationship to other standards
IEEE 829 as a standard is not so much about how to test, but how to document that you have tested. There is interplay between other standards, and other documents on the project. Just because IEEE 829 is being adhered to does not mean that the testing project will be a success. These are some of the other standards that may be referred to when documenting according to IEEE 829:
IEEE 1008 (Standard for Unit testing),
IEEE 1028 (Standard for Software Reviews),
IEEE 1044 (Standard Classification for Software Anomalies),
IEEE 1044-1 (Guide to Classification for Software Anomalies)
BSS 7925-1 (Vocabulary of Terms in Software Testing),
BSS 7925-2 (Standard for Software Component Testing),
Conclusion
IEEE 829 should be used as a standard appropriately, not blindly. Testing is a service organisation. In themselves, testers add nothing to the output of the project team. A tester does not make better software (testers are not allowed to change the code, are they?). Therefore, we need to slay the “documentation for documentation’s sake” myth. Is the output enabling the test team to do a better job, helping them to present the information found during testing in a clearer way, or showing to an outside agency (auditors?) that testing has been completed? In itself, merely incorporating IEEE 829 will not make the project a success. It can help to make a success, by giving guidelines and pointing the way to the light.