Stress Tests and Virtualized Environments
Virtualization Adds More Intricacies to Software Testing
Oct 5, 2009
These days virtualization is everywhere and even though there is no doubt that this is a great technology with tons of benefits, when stress tests are concerned, virtualization is a nightmare. In a sense, no matter how precise stress tests are, testers can never be sure that the results they have obtained are reliable. Additionally, reproducing a particular test environment where several virtualized applications run on the same machine and repeating exactly the same tests on a different machine is next to impossible.
A Virtualized Stress Test Environment Is Not Always Easily Reproduced
When virtualization is used, there is one physical machine, a couple of operating systems that run simultaneously on the same hardware and a couple of applications that run inside each of the operating systems. If only one of these applications/operating systems is changed, the test environment becomes very different.
For instance, here is an example with only two operating systems, uniquely identified by a checksum and a build number/release date. One of these operating systems is Windows (let's say Windows 2008, build XYZ) and the other is Linux (let's say Red Hat Enterprise 5.4, released on XYZ date).
The first question is which of them is the host operating system in which the virtual machine with the second OS is deployed. In terms of performance it does make a difference if Windows is running from inside Linux or the other way round. It also matters how much resources the guest operating system is allowed to use but at least this can be easily determined – i.e. 1GB of RAM, 2GB of RAM, etc.
In addition to the multiple operating systems, another factor that matters with stress tests in virtualized environments is the combination of applications that run on these operating systems and their loads. If one parameter is changed (i.e. one application is replaced with a different one), the equation becomes different because different applications will have different demands for resources. Well, this is true for non-virtualized environments as well, so nobody should blame virtualization (at least) for that.
Virtualization Itself Is the First Stress Test
When applications are run in a virtualized environment, these applications share the same hardware and software resources. It is quite logical that the load on a machine is heavier when 10 applications under 3 different operating systems are running on it than when only 1 application is running in the sole operating system on the machine. So, virtualization itself is like the first stress test for the machine – when multiple operating systems are running simultaneously, this loads the machine.
Now, in addition to this stress test (i.e. the load of virtualization), testers bombard the application with client requests. In other words, formally the stress test starts now, though in fact it has already started by loading the machine with the multiple operating systems and the applications inside them. In that aspect in terms of load the tested application is in more unfavorable situation when run in a virtualized environment.
As a result of all this, when a stress test is performed on an application in a non-virtualized environment and later this application is used in a virtualized environment, the results will be very different because of the very different test setup. That is why, always issue a warning that the stress test for an application has been performed on a dedicated machine and that if it is planned to run the application in a virtualized environment, the output will be very different. Only this way it is possible to achieve correct results.
 |
