Reliability Testing or Endurance Testing..

Definition:
Reliability testing– is the probability of failure-free operation of a computer program in a specified environment for a specified time.

Informally software reliability is about how well an application accurately provides without failure the services that were defined in the original specification. In addition to how long the application runs before failure, reliability engineering is about providing correct results and handling error detection and recovery in order to avoid failures.

The main intention is to find out the memory leak with the application, unexpected errors, any crash if we run it for long hours.

In real Business Day based scenarios cases usually application run for week /month/more. We need to simulate the application under workload like as a real case. That the reason we must do for reliability test. For a short duration we could not able to find the memory leak, unexpected behavior with the application.


The objective of Reliability test is to measure:

Memory leaks
JVM Heap
Concurrency
Will the application performance be consistent over time
Unhandled exceptions - crashes, hangs
Number of user Number of hits/sec
%CPU utilization on all serversMemory utilization on all server
GC collection (if any)

Steps:

To execute a scalability tests, follow the following steps1. Finalize the business scenarios and decide the Key Performance Indicators (KPI) for each operation, memory, CPU etc.
2. Verify the scenarios manually, load the volumes of data required.
3. Create the load scripts using any load testing tool; customize the scripts to make it robust.
4. Check the environment settings and ensure that all the setting is as per the load testing recommendation.
5. Create the load test scenario as with Business Day scenarios

Design the scenario for Business Day based scenarios to simulate real application usage cases like peak hours run (8 hrs) then idle time for 30 minutes then ramp up and run for another 8 hrs then idle time for 30 minutes
Continue this cycle around 8 -10 times.

6. Add all the performance counters required.
7. Execute the load test for pre-defined number of users, by slowly ramping up the users
8. Run the test for about 48 or more hours.9. Collect the results and analyze the results by comparing with the KPI / previous build and release results.

Results:
Plot the graph for private bytes
Put the all the performance monitor counters in the result sheet
Response time
Throughput

Main Observation:

Monitor Memory -RSS, VSZ on UNIX
Private Bytes and Virtual Bytes on Windows
Observe that all the transaction have been passed successfully.
(If you have more failed transaction there would be chance of application hangs, crashing.)

Tips:
--> Set granularity = 5 secs for all the counters.
-->Verify that the measurement scale setting to 1 for all the counters

Thanks
Senthil