The topic of web software testing, how to address this topic and where to start is a difficult question? Everyone that has been a part of the testing process or who has managed this part of a web development project has an opinion on what works and what does not work. The only conclusion that can be agreed to is that we all agree that our way is the best way. These conclusions are both right and wrong.
This topic does not have a shortage on good resources out there for the practitioner to research, investigate and try. This paper will explore what I have seen as the good, the bad, and the ugly as it relates to testing. I will then pull together what has work for me (e.g., as a former analyst and development manager) regarding a web development methodology.
The question before the development effort
The answer to the question prior to the development effort is a step often over looked in web development projects. In my view, the question and its answer is often all that needs to be accomplished in the initial stages. In a successful web development project, the later steps come after the question. My daughter who knows nothing about software development has a good thought that fits what I am trying to explain. When I ask her to clean her room (e.g., plan, develop, implement and maintain the project to clean her room) she usually says, “Let’s noodle on the question and answer before we jump into the fire of life.”
As a former developer, I know the need to take a step back and try to frame the business question and its initial answer before diving into the other tasks of web development.
Common scenario
It’s late Friday afternoon. Your boss comes to you with a high priority task for the CEO. You are informed this is needed for a Monday morning meeting at the CEO’s office. You look out the window and it’s already dark and your staff has left for the weekend. You look at the task request and respond, “Can do!” As your boss walks away, your smile fades and you are asking yourself, “why did I just say that?” I don’t want to work all weekend. But, I don’t have to work all weekend. I have an ace up my sleeve.
As a former manager of an information systems group, I emphasized the need for good documentation and repeatable processes. I knew we had answered my boss’s questions before. All I had to do was find the documentation, read it and follow the steps. This could help to turn the “Can do” response into reality, avoid re-inventing the wheel and get me some weekend time off.
The practice of requiring good documentation and the use of repeatable processes is called a methodology. Webster defines a methodology as a system of methods and principles used in a specific branch of knowledge. It should be reliable and repeatable.
From a user’s perspective it should be well documented and easy to use. From a manager’s perspective it should produce results that are timely and verifiable (e.g., through metrics).
There are several methodologies being used to address the question dilemma. A poll conducted by KDnuggets indicated one used more than others. This is the CRISP-DM. This methodology has a cyclical component which highlights the possibility of improvement each time that it’s applied to a particular project. Each pass through the methodology involves six phases. The sequences of the phases are flexible. Users are encouraged to move back and forth between the phases and each phase of the methodology benefits from the previous one.
http://www.kenningresearch.com/crispdm.htm
For a methodology to be useful it must be adapted to the environment of its intended use. Below follows a brief outline of the phases of the CRISP-DM methodology adapted to fit the need of an entity (e.g., one way to answer the question before the big dive):
Business Understanding
This first phase focuses on understanding the expectations of the business user, then translating this understanding into the business question to be answered. The first step is a preliminary plan to address the business question and meet the expectations of the business user. This understanding includes an agreement on validation, presentation, and documentation processes associated to the question being asked.
Data Understanding
The second phase focuses on looking at the initial data to discover first insights into the data or to detect interesting subsets to form a hypothesis for unknown information of interest. Any data quality issues should be discussed with the business user.
Data Preparation
The third phase focuses on collecting and formulating the data to be used from the initial data set. This phase is benefited by exploring the knowledge of past questions asked. Tasks include selecting the final data to be used for analysis, describing the decisions and actions taken to address the data quality problems reported during the data understanding phase, deriving new data as a result of calculations or the combining of multiple data elements into one data element, and reformatting of the data for future use.
Modeling
The fourth phase focuses on techniques used to model the results of the data preparation phase. Modeling is an art and selection of a “right for the task” approach. This will increase the probability for success. To ensure the “right” choice, the following three factors should be considered: (a) understanding of the business processes associated to the question being asked, (b) characteristics and constraints or limitations of the modeling environment selected for answering the question, (c) intended use of the model after it is completed. The final outcome should assist the user in getting some insight into why a certain modeling technique and certain parameter settings lead to the results obtained (e.g., an answer to the question asked).
Evaluation
The fifth phase focuses on a thorough evaluation of what you have done so far along with some possible adjustments. Before proceeding to the final phase of the question answering methodology, steps to shape the data to the user’s expectations, preliminary presentation of results, and the formal walkthrough need to be finalized. A formal review process will help ensure the reliability and repeatability of results obtained (e.g., the answer to the question).
Deployment (presenting the answer to the question)
This final phase focuses on tasks needed to complete the cyclical part of this methodology. Documentation is approved, final presentation tasks are completed, possible process improvement actions for a possible next go-around of the question are documented and any clean-up and/or maintenance tasks are completed. This phase concludes with the presentation of findings and signed-off of the results from the business user.
So, how did the business question story end? Because my staff followed a methodology and I knew where the past question folders were kept that addressed the initial question asked, I was able to find the previous iteration of the question asked. I was confident I could provide the answer to the question asked for the Monday morning meeting in a matter of hours and not days.
While the above Friday afternoon scenario maybe a little exaggerated, the facts are when you follow a sound methodology you can save yourself time and be confident your results will be consistent across iterations. This could be the difference in having either a good or bad day at the CEO’s office or in the boss’s office.
Changing gears (addressing the testing aspect prior to development)
[Why testing is needed, launch link below]
http://www.youtube.com/watch?v=FJ4A0aaaOAw
The good
The older is sometimes the better when it comes to reference material. As an analyst, it is important to be well rounded in the views of the early pioneers of testing methodologies. One of the first to really define the ins and outs of testing was Glenford J. Meyers (The Art of Software Testing, first released in 1946). In chapter two the discussion of economics and psychology is framed as the most important aspect of what can be learned about testing any development project or effort. The bottom line is that you cannot test everything. Focusing on the core functionality associated to the initial question asked by the customer is a good first focus area in any testing effort.
In my view the second area that needs to be addressed prior to actually testing anything is how will your effort’s success or lack of success be measured. In chapter seven, Rick D. Craig and Stefan P Jaskiel (Systematic Software Testing, 2002) discuss three areas that need to be measured against:
· Customer satisfaction (e.g., gathered by analyzing surveys and feedback from calls to your organization’s incident collection area)
· Defects (e.g., focusing on what was found, severity, age, density and distribution)
· Coverage (e.g., during the requirement gathering, code and designing phases)
Learning from the past through testing metrics, adjusting these metrics to the current effort and monitoring the current effort is often over looked in today’s tight schedules. In my view, this is a mistake.
James A. Whittaker in chapter one (How to Break Software, 1965), talks about how many forms of testing share common elements such as requiring that the testing be done in a real or close to real environment as possible. The point that stood out to me was the one that indicated that all forms of testing differ mainly in intent and in how the details are handled. The intent is focused on the initial question asked and the results defined.
The view I have with the good area is that the basics tend to stand the test of time. Yes, the new technologies require thinking out of the box at time such as in Robert Culbertson’s Rapid Testing (2002) to properly cover specific functionality at a breakneck pace but just because the reference is dated doesn’t mean that they should not be explored for potential use.
The bad
Some references in my view hit the middle ground in some areas regarding their views on testing. Edward Kit in chapter eight (Software Testing in the Real World, 1995) indicates the need for more small test cases versus a relatively a few large ones as a better approach to designing for test execution. This view is an “it just depends.” In large complex systems, end-to-end process type test scenarios that focus on the happy path and implemented error handling in my view tend to produce the best results. I have no issues with many small test cases to test boundary and edge type test conditions. I view this area as the middle ground that is dated in the object oriented area of testing (e.g., having to test objects as they flow from interface to interface) and therefore a candidate for the bad rating.
Another reference with a similar viewpoint as above is by David Marks in chapter five (Testing Very Big Systems, 1992), this reference indicates that simple tests are better than complex ones. Again, I do not fully agree with statement. This reference did indicate to test where the problems are. I agree with this statement. In my experience, bugs tend to migrate to the same areas (e.g., if you find edge type bugs or error condition type bugs, there is a good probability that you will find other edge and error condition type bugs). Again, I view this area as the middle ground and also a candidate for the bad rating.
The ugly
In my view, some references that may have had good intentions miss the mark in some of their viewpoints regarding testing. Brian Marick in chapter one (The Craft of Software Testing, Subsystem Testing Including Object-based and Object-oriented Testing, 1995) defines some assumptions about testing that I do not agree with. One assumption noted was that at every stage of testing, mistakes are inevitable. This point I agree with but the reference then goes on to say that the later stages should compensate for them. Errors found later in the Software Development Life-Cycle (SDLC) are more expensive. Focusing on prevention versus detection earlier in the SDLC is a more efficient use of resources.
Web development
After a good understanding of the question asked and a plan of action of the testing approach that best suits the question asked, in my view, you are now ready to explore the development approach need to make all the parties happy. We can agree to agree that every developer has an opinion to the right and wrong way to engage the development question. We can also agree to agree that every former software development manager has a preference to what works and what does not work.
In my view, it just depends. Whether wrong or right, I will offer a way (e.g., a methodology) of doing web software development that has worked for me.
Determine project size
Prior to starting any web development effort, I try to frame the project in terms of project size and effort. Based upon metrics, our organization has defined three types of web projects:
· Fast track
· Small project
· All others
|
|
Description |
TASKS (T) |
FAST TRACK (FT) |
SMALL PROJECTS (SP) |
ALL OTHER PROJECTS (P) |
|
1 |
Time frames are “lapsed time” from the start of the project until it goes live. |
Day-to-day items that don’t require extensive research / documentation
|
One to two days
|
Two days to 2 months
|
More than 2 months from start to finish
|
|
2 |
Very little to new functionality |
X |
X |
|
|
|
3 |
Follows Web Project Matrix |
X |
X |
X |
X |
|
4 |
Has specific due date |
|
X |
X |
X |
|
5 |
PMO Leads Project |
|
X |
X |
X |
|
6 |
Non PMO Leads Project |
X |
|
|
|
|
7 |
Follows Web Project Matrix |
X |
X |
X |
X |
|
8 |
Creation of Templates |
# |
|
# |
# |
|
9 |
All resources needed for project remain focused on the project until it is complete. |
|
X |
|
|
|
10 |
Project Requirements |
# |
|
X |
X |
|
11 |
Analyst |
# |
# |
X |
X |
X = Preferred # = As Needed Blank = Not Applicable
Suggested web development iterations
Once the project size has been framed, the team is ready for around of prototyping that ties into the web development methodology.
|
1. Whiteboard |
|
Site laid out on a whiteboard using |
|
2. Paper Prototype |
|
Hand written sketch of site on
paper. No computer or software used to create or view |
|
3. Non-Functional Wireframe |
|
Pixel perfect grayscale |
|
4. Functional Wireframe |
|
Functional version of the site
without graphics that can be viewed and clicked on using a browser |
|
5. Paper
Composition |
|
Graphic representation as close |
|
6. Test Site |
|
Website developed and ready |
|
7. Live Site |
|
Site or page published to the web accessible to the public
|
LEGEND
PROJECT SIZES:
T = Tasks FT = Fasttrack SP = Small Project P = Project
(day to day) (2 days max) (2 days to 2 months) (2 months >)
X = Preferred # = As Needed Blank = Not Applicable
|
T |
FT |
SP |
P |
Step # |
Step Title |
Purpose |
Participants |
Owns |
Outputs |
|
X |
X |
X |
X |
1
|
Project
|
Discuss and gather project requirements, objectives, due dates, sponsor identification, etc. |
Sponsor Analyst Lead |
Lead |
1. Project Brief § Sponsor(s) Identified § Project Objectives Defined § Targeted Go Live Date Identified § Success Metrics § Target Audience Identified |
|
# |
|
X |
X |
2 |
Project
|
To review and finalize the Project Brief |
Sponsor Analyst Lead |
Lead |
1. Project Brief Approval/Finalize 2. High Level Project Requirements 3. Team member are identified by Dept. Mgrs. 4. Business Requirements 5. Project Charter 6. High Level Project Schedule |
|
X |
X |
X |
X |
3 |
Team Alignment / Creative Brainstorm
|
Team meets to review and discuss high level Project Requirements followed by white board session. |
Project
Team Lead |
Lead |
Without User Testing 1. High Level Estimates 2. Electronic file (digital photo) of Whiteboard 3. 1-3 Non-Functional Wireframes (PSD/jpg) |
|
With User Testing 1. High Level Estimates 2. Electronic file (digital photo) of Whiteboard 3. Paper Prototype(s) (hand drawn) for User Test Lab |
|||||||||
|
|
# |
X |
X |
3.5 |
User
Test |
First session (morning) 3-5 users test navigation using paper prototype.
Second session (afternoon) test group recommends navigation. |
Test
Team Lead |
Lead |
1. 1-3 Non-Functional Wireframes (PSD/jpg) |
|
|
# |
X |
X |
4 |
Non-Functional Wireframe Approval
|
Sponsor reviews High Level Estimates and non-functional wireframes.
Any final requirements are identified and recorded. |
Sponsor Analyst Lead |
Lead |
1. Approved Wireframe (Functionality Freeze) 2. Functional Wireframe Created § Accurate dimensional layout
(Creative Team Agreement) |
T |
FT |
SP |
P |
Step # |
Step Title |
Purpose |
Participants |
Owns |
Outputs |
|
|
# |
X |
X |
5 |
Functional Wireframe Approval
|
Sponsor reviews functional wireframe
Any final requirements are identified and recorded. |
Sponsor Analyst Lead |
Lead |
1. Approved Functional Wireframe 2. Finalized Project Requirements (Requirements Freeze) 3. Finalize Project Schedule 4. Project Requirements Walkthrough Team Meeting 5. 1-3 Compositions Created, Creative Brief and Sitemap 6. QA of Final PSDs 7. Final PSD (web site without approved graphics)
START EXECUTION OF APPROVED WORK |
|
# |
X |
X |
X |
6 |
Creative Presentation
|
Present compositions, creative brief and site map to the sponsor(s). |
Sponsor Project
Team Lead
|
Lead |
1. Approved Composition 2. QA of Final Videos, Images, and Photos 3. Composition finalized (Graphical Freeze) a) Placement of approved graphics b) Resource gallery adjustments c) Code completion
|
|
X |
X |
X |
X |
7 |
Quality Assurance |
Quality Assurance is performed on website |
Quality Assurance |
QA |
1. Test Link to PMO 2. QA Test Plan |
|
# |
# |
X |
X |
8 |
User Review |
May Include:
§ User Test Lab #2 § PMO Review § Department Review § Sponsor Review |
Test
Team PM Sponsor |
Lead |
1. Test Team consist of 3-5 users (1 session) 2. PMO reviews the site against Project Requirements. 3. Departments final review and feedback 4. Sponsor final review and feedback 5. PMO summarizes items for User Review Report 6. Final adjustments (if necessary) are made based on feedback |
|
X |
X |
X |
X |
9 |
Site
Live |
|
-- |
IT |
|
|
X |
X |
X |
X |
10 |
Post Production Quality Assurance |
Quality Assurance is performed on live website |
Q/A
Team |
Lead |
1. QA Test Plan Updates 2. User Acceptance Review Report Updates 3. Website launch email notification sent to sponsors/Team |
|
|
# |
X |
X |
11 |
Closure |
Closure report includes lessons learned and analytics measuring key success factors as outlined in Charter. |
Project
Team |
Lead |
1. Lessons Learned 2. Success Metrics 3. Project Closure Report 4. Project Closed |
As with any good intention or methodology, there is always better ways to do it. Based on metrics and repetition, I would suggest giving this one a trial run.
Enjoy!