Support Statement: Verifying a Migration Solution

A convenient means of planning and estimating functional testing is by running gmStudio's GUI Report. The GUI Report is an organized listing of all the controls that allow user input. A typical enterprise application will have hundreds of forms and thousands of active controls. The set of input controls is a crude measure of the number of function points in the application and it correlates with the number of tests needed to fully exercise the app through its User Interface. In theory, if we ran test cases that exercised all of these UI inputs in a "meaningful" way, we would invoke/test all the functions of the system. Bear in mind that most of these controls support many possible valid and invalid inputs, so the effective number of use/test scenarios is much, much larger -- at least one or two orders of magnitude larger than the number of controls. Clearly, the number of permutations and combinations possible is much too large a number to expect a team to define, let alone run them all.

When thinking about testing the results of a tool-assisted rewrite, there are a couple important points to consider.

First, the new code is being generated from source code. Furthermore, that source code is assumed to be a complete, formal specification of the legacy functionality that has been verified through thousands of hours of use. In theory, the expected results for any and every conceivable use case performed in the new system, is to get the same results as produced for that use case in the legacy system.
Second, the new code is produced by a systematic transformation process. This transformation process is a collection of many different types of code translation and re-engineering operations. These operations can be referred to as migration features.

A migration feature is a set of rules for rewriting a particular technical element of the legacy code: for example, replacing a specific member in a COM API or rewriting a line of code that depends on a specific VB6 language feature. Some features are general and some are client-specific. Some are mature and some are being used for the first time. Some features are very simple and are expected to be semantically accurate; others are more complex and might alter the behavior of the application. There are many thousands of migration features used in a typical migration solution. The more noteworthy migration features planned for a project are defined, albeit at a high-level, in the re-engineering plan. The implementation details of features are in the migration solution files, the default metalanguage configuration files, and the gmBasic tool itself.

Each migration feature can be said to have a correctness quotient: the likelihood that it will produce functionally correct code. The correctness quotient can be determined by an understanding of the expected semantic similarity of the legacy code and the migrated code based on the language specifications and empirical data (prior test results), and on the expected reliability of the process that implements the feature. The correctness quotient of a feature can be improved by improving its implementation and verifying the results. The vast majority of migration features have a high correctness quotient and are not likely to result in defects. However, complex rules, new rules, and to some degree, the interaction of multiple rules can produce defects. These migration features have a lower correctness quotient and are termed "risky".

In order to test the migration solution efficiently, the team must identify a set of test cases that exercise code impacted by risky migration features. We can determine the location of migration features through static analysis of the legacy code and of the migration solution files. We can then trace these impacted codes up to specific UI elements. Once we know which UI elements depend on risky migration features, we will ask the client to develop or select existing test cases that exercise that area of the UI and consequently the critical sections of code. The test results can be used to improve the correctness quotient of the entire migration solution by reducing the probability of functional defects. When the probability of functional defects is "low enough", the migration process can be considered "complete".

TODO: talk about conformance quotient: how well the code conforms to technical standards.

See also /wiki/spaces/GMI/pages/1742183