Visual Studio Test Suite 2008: Unit Tests Pass but Load Tests Fail

Boyan Tsolov — Thu, 03 Jun 2010 15:53:00 GMT

This is a problem I recently ran into:

- I am using Visual Studio Test Suite 2008 to create load tests that run multiple unit tests

- I created unit tests that run by getting configuration settings from an app.config file

- If I run the unit test by itself, they pass

- I placed the unit tests in a load test and ran that: the load test fails at the line of code that tries to access the app.config:

if (System.Configuration.ConfigurationManager.AppSettings["xxx"].Equals("xxx"))

Fixing this is extremely simple. In fact, this blog post describes it perfectly: http://www.apexa.net/Blog/web_design_Blog_20080602.aspx.

View the properties of the load test's Run Settings and set the property: "Run unit tests in application domain" to TRUE.

Done!

Performance Testing: The Performance Test Plan

Boyan Tsolov — Wed, 11 Nov 2009 03:57:00 GMT

I have always had a problem doing performance testing (aka load testing). I am quite adept at recording web tests and collecting counters, but I still find it troublesome interpreting the counters that I gather from load tests. The problem is that I need a plan that will produce relevant results with which I can conclude with confidence: "This system is a juggernaught! It can withstand more hits than Google.com." If you are like me and you have trouble planning your load tests, then maybe this post will help you. The following post describes how I like to run my own plan and what conclusions I can make with this plan.

Special thanks goes out to Jeff Dunmall, our co-CEO, whose article "Real-World Load Testing Tips to Avoid Bottlenecks When Your Web App Goes Live" got me to write something more about load testing here, and Randar Puust who uses this plan (and thus now I use it too).

There are a number of facets to load testing:

- setting up a load testing rig and setting up your environment in Visual Studio

- recording the web tests

- planning the load tests

- collecting performance counters and figuring out what the recorded result mean

Unless you have a plan, planning the load tests and reading the results will be meaningless. A performance test plan will keep you on track throughout the last two phases mentioned above. The results from a test plan will allow you to determine:

- maximum number of users that can hit the site before it becomes unresponsive

- can the system recover from overload

- are there memory leaks in the code

A performance test plan consists of four types of tests that need to be run in the following order:

1. Performance Tests

2. Load Tests

3. Stress Tests

4. Endurance Tests

1. Performance Tests

These are the benchmark tests. By this point you should have recorded web tests which cover the most-used parts of your web application. Now, you will create a load test that gathers and runs all of the web tests that you have recorded. Run the load test with settings similar to these:

- load: low end of the expected number of users for the system. I like to start with 5 or 10.

- time: 10 minutes

- think time: 5 seconds

- warmup: 30 seconds to 1 minute is more than enough

A load test like this should not stress the web server, it should be low-impact. The goal of this test is for it to be run later on in this plan and be used to compare against (i.e. a benchmark). Collect and save the result counters from the web server and from the client. Web Server counters to collect are: "CPU %", "Used Memory". Client counters to collect are: "Respose time per page", "Number of Requests per second". Other counters you might like to save are "pages that take the longest time to load", "number of errors", "network usage" - these are useful for neat reports that you might want to compile.

2. Load Tests

The load tests are going to stress the web application and try and find its breaking point. The goal of these test will be to find the lucky number X. This number is the number of users that are concurrently hitting the system and that cause the system to be unresponsive or response times to be unacceptable. For this part of the plan you will need to run a number of load tests. In each case, increase the number of users and collect the counters:

- load: 10 users, 15 users, 20 users, 50 users.... until the response times get too large, or server CPU % gets dangerously high.

- time, think time, warmup: same as for the performance test. Basically we are running the performance test from step 1but with more and more users.

By the end of this step you will know what is the maximum number of users that can hit the system.

3. Stress Tests

Stress tests check that the web application can recover from errors induced by overload.

- load: X + 20% (where X is the magic number from step 2 which causes the system to be unstable)

- time: 1 minute

- think time: 5 seconds

After this test is run the system should become unresponsive. If it is not unresponsive, then increase the 20%. One way to check if it is unresponsive is to check that response times are suddenly exponentially far worse than the results in step 2 and CPU % usage on the server is close to 100%.

Immediately after this test runs run the performance test from step 1. If the performance test runs fine and the results are comparable to the test that was run at step 1, then your system passes! It can safely recover from overload issues.

4. Endurance Tests

The endurance tests are unrelated to the other 3 tests. These tests are run over a long period of time, which causes many users to hit the site. If there are memory leaks, then over the course of this endurance test the free memory on the web server should decrease dramatically.

- load: X - 20% (where X is the magic number from step 2 which causes the system to be unstable). We'll use -(minus) 20% because we don't want to overload the system

- time: 12 hours

- think time: 5 seconds

Remember to record the "Free" or "Used Memory" on the web server. The goal at the end of this step is to make sure that the memory usage on the web server has not decreased dramatically and thus there are no memory leaks in your code. If that is the case with your results then tip of the hat to you!

At the end of these four types of tests you should have a better idea of where your system stands in terms of load that it can handle. You will know:

- what is the maximum number of users it can handle before performance is unacceptable

- can it recover after an overload

- are there any memory leaks

Happy testing!

Setting up a load test database store with Visual Studio Test Edition

Boyan Tsolov — Tue, 12 Aug 2008 22:39:00 GMT

When setting up Visual Studio to run load tests you might have noticed a nasty warning message such as this one:

"The connection string for the database has not been set. To store your results in a database for post-run analysis, set the connections string to a valid load test results store."

This happens because Visual Studio is trying to store your test results in a database store. This way you can load it later (in case you close your current open Visual Studio application). This post will show you how you can configure Visual Studio to stop bugging you.

We will get rid of that message by actually listening to it and configuring a database for Visual Studio to use. You will need access to an SQL database server and access to the Management Studio - you will need to run a SQL Query.

1. Open SQL Management Studio

2. Find the SQL Query that will create your database store

This file is C:\Program Files\Visual Studio 9\Common7\IDE\loadtestresultsrepository.sql
This is the directory for Visual Studio 2008, but this file should be there for other versions as well.

3. Run the SQL Query file

Hopefully you will get the "Command(s) completed successfully" message

4. Link Visual Studio to this new database

Open Visual Studio.

Go to Test > Administer Test Controllers...

And enter in the LoadTest database name in the dialog that appears

Et Voila! No more error message!

Boyan's blog : load test

Visual Studio Test Suite 2008: Unit Tests Pass but Load Tests Fail

Performance Testing: The Performance Test Plan

Setting up a load test database store with Visual Studio Test Edition