Scalability Test Methodology

This is to explain the Scalability concepts, methodology and the root cause analysis

Scalability Testing:

Scalability is a Performance testing focused on ensuring the application under test gracefully handles increases in work load. Its is the testing of a software application for measuring its capability to scale up or scale out in terms of any of its non-functional capability be it the user load supported, the number of transactions, the data volume etc


The objective of scalability test is to measure:

1) Response time
2) Number of user the application
3) Number of hits/sec
4) Slow SQLs
5) Throughput
6) %CPU utilization on all servers
7) Memory utilization on all server
8) GC collection (if any)


Prerequisite to start Scalability test execution:

Restart Application/web server services
clear cache and log files
Observe the CPU and memory usage before starting the execution and analyze what is the CPU and memory usage prior the application is run
Verify the DB memory and CPU Utilization
Baseline the DB size. If production DB has 10k records then 4 times of the data should be made available in the testing environment DB
Run DB index fragmentation
IE caching depends on the type of application being tested. Ex: If the application is banking system based, then caching need not be enabled because of frequency of the user logins in rare. When the frequency of user login in a given working day is more, then caching needs to be on.


Key Process for Scalability Testing:

Identify Performance Acceptance Criteria
Identify Key Scenarios
Create a Workload Model
Identify Target Load Levels
Identify Metrics
Run Tests
Analyze the Results

Scalability Testing Methodology:

To execute a scalability tests, follow the following steps

1. Finalize the business scenarios and decide the Key Performance Indicators (KPI) for each operation, memory, CPU etc.

2. Verify the scenarios manually, load the volumes of data required.
3. Create the load scripts using any load testing tool; customize the scripts to make it robust.
4. Run the script for warm up run, at least two runs
5. Take SQL trace;
6. Find the list of SQLs for all the operations in the scenario that exceed KPI; Run the SQLs
against the Database and find the time taken by each SQL
7. If you find any slow SQLs, report to DB Team
8. Create the load test scenario, add all the performance counters required and execute the test
9. Check the environment settings and ensure that all the setting is as per the load testing recommendation
10. Execute the load test for pre-defined number of users, by slowly ramping up the users and then running the test for about 3 hours.
11. Once the test is completed, filter the results for steady state (Between 2- 3 hrs)
12. Collect the results and analyze the results by comparing with the KPI / previous build and release results.
13. If the results do not meet the KPI, log the defect and continue analyzing the root cause.


Take the CPU Profile in case of CPU being more than KPI
Take Memory profile if the memory is higher than KPI.