מתודולוגיית בדיקות אוטומטיות

מיועד למי שמעוניין להקים ולהטמיע מערך אוטמציה בחברתו. לחברת טסטפרו ניסיון רב בתחום זה והיא תוכל לתרום רבות ולייעל הטמעה מסוג זה

Disclaimer

The information contained in this document is proprietary to TestPro Ltd., any unauthorized reproduction, use or disclosure of any part thereof is strongly prohibited.

TestPro Ltd. reserves the right to make any modifications to this document at any time without prior notice. The information contained in this document is believed to be accurate and reliable. However, no responsibility is assumed by TestPro Ltd. for its use.

Introduction

The writer of this article is Test Pro’s Automation manager with more then 12 years of experience in Quality Assurance market. He had participated in more then 80 projects most of which were Automation projects.

What is Automated Testing?

On the first look the meaning of "Automated Testing" is simple :Automating current manual testing process. But, from the other side, this may be very complicated process. The main problems are:

Documents for automated testing should be much more specific then ones for manual testing.
Special testing environment (including automation tools) should be established.
Automated tests are often harder to maintain then manual ones.
Test automation can add a lot of complexity and cost to a test team's effort.

Also, what if there is no manual testing? Even so you still can go with automation. But in this case great attention should be paid when designing test documents.

Rest of this document will explain why to use automation, when to do so and when to do not and how to do it right.

Why to use automated testing?

Automation isn't always a good idea – the key is to know what the benefits and costs really are, then to make an informed decision about what is the best for your circumstances. The main benefits of automation are:

Repeatability – It means that automated tests can be executed more than once, consistently each time. In order to realize this benefit, the system under test must be stable enough that the same tests can be repeated without excessive maintenance.
Leverage – True leverage from automated tests comes not only from repeating a test that was captured while performed manually, but from executing tests that were never performed manually at all. Enjoying this benefit requires the proper test case design.
Accumulation – The most critical benefit for the long term. It is a fact that applications change and gain complexity over their life. Constant modifications and enhancements are typical. Therefore, the number of tests which are needed for coverage also constantly increasing. But, if the automated tests are not designed to be maintainable as the application changes, the test library will be fighting just to stay even. Therefore, it is critical to adopt an approach to test library design that supports maintainability over the life of the application.

When not to use automation?

The corner stone of test automation is the premise that the expected application behavior is known. When this is not the case, it is usually better not to automate. Following are some examples when not to use automation:

Unstable design – There are certain applications that are inherently unstable by design. For example, a weather-mapping system will not demonstrate sufficiently predictable for automation. Applications whose data is not stable enough to produce consistent results are not good candidates for automation. The investment required to develop and maintain the automated tests will not be offset by the benefits, since repeatability will be doubtful.
Inexperienced testers – If person(s) automating the test are not sufficiently experienced with the application to know the expected behavior, automating their tests is also of doubtful value. Their test may not accurately reflect the correct behavior, causing later confusion and wasted effort. Remember, an automated test is only as good as the person who created it.
Insufficient time, resources – If you don't have enough time or resources to get testing done manually in the short term, don't expect a tool to help you. The initial investment for planning, training and implementation will take more time in the short term than the tool can save you. Get through current crisis and then look at automation for the longer term.

The automated testing lifecycle.

Decision to Automate Test

Test Tool Acquisition

Automated Testing Introduction Process

Test Planning, D&D

Execution and Management of Tests

Test Program Review

By using the systematic approach present in previous diagram it is possible to organize and execute test activities in such a way as to maximize test coverage within the limits of testing resources. This structured test methodology involves a multi-stage process, supporting the detailed and interrelated activities that are required to introduce and utilize an automated test tool:

Develop test design.
Develop and execute test cases.
Develop and manage test data and the test environment.
Document, track, and obtain closure on issue/trouble reports.

Let's take a closer look at the main phases of the lifecycle.

1. Decision to Automate Testing

This phase covers the entire process that goes into the automated testing decision. During this phase, it's important for the test team to manage automated testing expectations and to outline the potential benefits of automated testing when implemented correctly. A test tool proposal needs to be outlined, which will be helpful in acquiring management support.

2. Test Tool Acquisition

This phase guides the test engineer through the entire test tool evaluation and selection process, starting with confirmation of management support. Since a tool should support most of the organizations' testing requirements, whenever feasible the test engineer will need to review the system's engineering environment and other organizational needs and come up with a list of tool evaluation criteria. The test engineer then needs to define an evaluation domain to pilot the test tool. Finally, after all those steps have been completed, the test engineer can make vendor contact to bring in the selected tool(s).

3. Automated Testing Introduction

This phase outlines the steps necessary to successfully introduce automated testing to a new project, which are summarized in the following sections.

Test Process Analysis - Test process analysis ensures that an overall test process and strategy are in place and are modified, if necessary, to allow automated testing to be introduced in a successful fashion. The test engineers define and collect test process metrics in order to allow for process improvement. Here test goals/objectives and strategies need to be defined and test process needs to be documented and communicated to the test team. In this phase, the kinds of testing applicable for the technical environment will be defined, and tests are defined that can be supported by automated tools.
Test Tool Consideration - The test tool consideration process includes steps that investigate whether incorporation of automated test tools that have been brought into the company without a specific project in mind now would be beneficial to a specific project, given the project testing requirements, available test environment, personnel resources, user environment, platform, and product features of the application under test.

4. Test Planning, Design and Development

The test planning stage represents the need to review long–lead-time test planning activities. During this phase, the test team identifies test procedure creation standards and guidelines; hardware, software, and network required to support test environment; test data requirements; a preliminary test schedule; performance measure requirements; a procedure to control test configuration and environment; as well as defect-tracking procedure(s) and associated tracking tool(s).

The test design component addresses the need to define the number of tests to be performed, the ways that testing will be approached (paths, functions), and the test conditions that need to be exercised. Test design standards need to be defined and followed.

5. Execution and Management of Tests

At this stage, the test team has addressed test design and test development. Test procedures are now ready to be executed in support of exercising the application under test. Also, test environment setup planning and implementation was addressed consistent with the test requirements and guidelines provided within the test plan.

With the test plan in hand and the test environment now operational, it's time to execute the tests defined for the test program. When executing test procedures, the test team must comply with a test procedure execution schedule, as discussed earlier. The test procedure execution schedule implements the strategy defined within the test plan. Plans for unit, integration, system, and user acceptance testing are executed. Together, these testing phases make up the steps that are required to test the system as a whole.

6. Test Program Review and Assessment

Test program review and assessment activities need to be conducted throughout the testing lifecycle, to allow for continuous improvement activities. Throughout the testing lifecycle and following test execution activities, metrics need to be evaluated and final review and assessment activities need to be conducted to allow for process improvement.

Automation methodologies.

Following is a brief descritpion of most popular and time proven implementations of automated testing methodologies.

1. Functional Decomposition Method

The main concept behind the "Functional Decomposition" script development methodology is to reduce all test cases to their most fundamental tasks, and write User-Defined Functions, Business Function Scripts, and "Sub-routine" or "Utility" Scripts which perform these tasks independently of one another.

In order to accomplish this, it is necessary to separate Data from Function. This allows an automated test script to be written for a Business Function, using data-files to provide the both the input and the expected-results verification. A hierarchical architecture is employed, using a structured or modular design.

The highest level is the Driver script, which is the engine of the test. The Driver begins a chain of calls to the lower level components of the test. Drivers may perform one or more test case scenarios by calling one or more Main scripts. The Main scripts contain the test case logic, calling the Business Function scripts necessary to do the application testing. All utility scripts and functions are called as needed by Drivers, Main, and Business Function scripts. Lets see the advantages and disadvantages of this method:

Advantages:

Utilizing a modular design, and using files or records to both input and verify data, reduces redundancy and duplication of effort in creating automated test scripts.
Scripts may be developed while application development is still in progress. If functionality changes, only the specific "Business Function" script needs to be updated.
Since scripts are written to perform and test individual Business Functions, they can easily be combined in a "higher level" test script in order to accommodate complex test scenarios.
Data input/output and expected results is stored as easily maintainable text records. The user's expected results are used for verification, which is a requirement for System Testing.
Functions return "TRUE" or "FALSE" values to the calling script, rather than aborting, allowing for more effective error handling, and increasing the robustness of the test scripts. This, along with a well-designed "recovery" routine, enables "unattended" execution of test scripts.

Disadvantages:

Requires proficiency in the tool's Scripting Language;
Multiple data-files are required for each Test Case. There may be any number of data-inputs and verifications required, depending on how many different screens are accessed. This requires data-files to be kept in separate directories by Test Case.
Tester must not only maintain the Detail Test Plan with specific data, but must also re-enter this data in the various data-files.
If a simple "text editor" such as Notepad is used to create and maintain the data-files, careful attention must be paid to the format required by the scripts/functions that process the files, or processing-errors will occur.

2. Test Plan Driven Method

The "Test Plan Driven" method preserves most of the advantages of the "Functional Decomposition" method, while eliminating most of the disadvantages. In this method, the entire testing process is data-driven, including functionality. The Detail Test Plan is written in a specific format, then saved in a particular record-format which the pre-written "Utility" scripts use to control the entire processing of the Automated Test.

The architecture of the "Test Plan Driven" method looks similar to that of the "Functional Decomposition" method, but in fact, they are substantially different:

Driver Script –

Performs initialization (if required);
Calls the Application-Specific "Controller" Script, passing to it the file-name of the Test Case file (which has been saved in a "tab-delimited" format from the tester-generated Detail Test Plan spreadsheet);

Controller Script –

Reads and processes the "tab-delimited" file, using a "switch-case" statement to break on each Key Word;
Builds a list of "processing actions" from the data within the rows / columns within the Key-Word group;
Calls a specific "Utility" script associated with the Key Word, passing to it the created "processing list";

Utility Scripts –

Process input-list received from the "Controller" script;
Perform specific tasks (e.g. press a key or button, enter data, verify data, etc.), calling "User Defined Functions" if required;
Report any errors to a customized Test Report for the Test Case;
Return to "Controller" script;

Functions –

General and Application-Specific functions may be called by any of the above script-types in order to perform specific tasks;