Bug Counts..

Bug Count are also accountable to tester!!!

There was a discussion/debate about the Bug Count in the industry,
I feel that bug count numbers is also one factor to measure tester performance indicator. I feel it’s important to the tester efficiency / creditability.

Why its being?....

In the team How the efficient tester finding more bugs compare to other team members , not only simply executing the well written test cases,

Thinking capability
ideas to exploratory testing
His in depth Domain knowledge and the complexity of the product
his intention to find more bugs ( off course everyone wishes J)
his productivity hours
finding the bugs in a stable products
severity and priority of the bugs


Bug count also depend on other factors as Alan mentioned in his sites
http://blogs.msdn.com/imtesty/archive/2006/06/26/647628.aspx


I am trying to emphasis that bug count is also one factor to measure tester performance.

Thanks
Senthil

Scalability Test Methodology

This is to explain the Scalability concepts, methodology and the root cause analysis

Scalability Testing:

Scalability is a Performance testing focused on ensuring the application under test gracefully handles increases in work load. Its is the testing of a software application for measuring its capability to scale up or scale out in terms of any of its non-functional capability be it the user load supported, the number of transactions, the data volume etc


The objective of scalability test is to measure:

1) Response time
2) Number of user the application
3) Number of hits/sec
4) Slow SQLs
5) Throughput
6) %CPU utilization on all servers
7) Memory utilization on all server
8) GC collection (if any)


Prerequisite to start Scalability test execution:

Restart Application/web server services
clear cache and log files
Observe the CPU and memory usage before starting the execution and analyze what is the CPU and memory usage prior the application is run
Verify the DB memory and CPU Utilization
Baseline the DB size. If production DB has 10k records then 4 times of the data should be made available in the testing environment DB
Run DB index fragmentation
IE caching depends on the type of application being tested. Ex: If the application is banking system based, then caching need not be enabled because of frequency of the user logins in rare. When the frequency of user login in a given working day is more, then caching needs to be on.


Key Process for Scalability Testing:

Identify Performance Acceptance Criteria
Identify Key Scenarios
Create a Workload Model
Identify Target Load Levels
Identify Metrics
Run Tests
Analyze the Results

Scalability Testing Methodology:

To execute a scalability tests, follow the following steps

1. Finalize the business scenarios and decide the Key Performance Indicators (KPI) for each operation, memory, CPU etc.

2. Verify the scenarios manually, load the volumes of data required.
3. Create the load scripts using any load testing tool; customize the scripts to make it robust.
4. Run the script for warm up run, at least two runs
5. Take SQL trace;
6. Find the list of SQLs for all the operations in the scenario that exceed KPI; Run the SQLs
against the Database and find the time taken by each SQL
7. If you find any slow SQLs, report to DB Team
8. Create the load test scenario, add all the performance counters required and execute the test
9. Check the environment settings and ensure that all the setting is as per the load testing recommendation
10. Execute the load test for pre-defined number of users, by slowly ramping up the users and then running the test for about 3 hours.
11. Once the test is completed, filter the results for steady state (Between 2- 3 hrs)
12. Collect the results and analyze the results by comparing with the KPI / previous build and release results.
13. If the results do not meet the KPI, log the defect and continue analyzing the root cause.


Take the CPU Profile in case of CPU being more than KPI
Take Memory profile if the memory is higher than KPI.

Linux system resource monitor using VMSTAT

Linux resource monitor using VMSTAT-Vmstat reports virtual memory statistics of processes, virtual memory, disk, trap, and CPU activity. Vmstat is useful to correlate memory, processes, and CPU utilization of a system. On multi-processor systems, vmstat averages the number of CPUs into the output.


vmstat interval count
options -- Options are used to get information about specific activity. A complete set of options can be seen by referring to man vmstat on your system.

interval -- This is the time period in seconds between two samples. Too small an interval (< 5 sec) may add command overhead to the output results, and the results may not reflect the true system state.

count -- This is the number of times the data is needed. The first line of vmstat gives the average of data since the time of system reboot and can be ignored. Without the count option, the command produces continuous samples until terminated by ^c. As a common practice, the samples are observed for some time without count argument to get a feel of system state.

Procs
r: The number of processes waiting for run time.
b: The number of processes in uninterruptable sleep.
w: The number of processes swapped out but otherwise runnable. This field is calculated, but Linux never desperation swaps.

Swap
si: Amount of memory swapped in from disk (kB/s).
so: Amount of memory swapped to disk (kB/s).

IO
bi: Blocks sent to a block device (blocks/s).
bo: Blocks received from a block device (blocks/s).

System
in: The number of interrupts per second, including the clock.
cs: The number of context switches per second.

CPU
These are percentages of total CPU time.
us: user time
sy: system time
id: idle time

Run for a particular time & Redirect to file
Eg: Run the vmstat for 1 hour

vmstat -n 5 750 > filename.log &

750 --> Number of times at the interval of 5 sec. How --> 5*750/60*60 =~ 1 Hr


Bottleneck Idetification:
1. If the number of processes in the run queue column "r" under procs is consistently greater than the number of CPUs on the system, it will slow down the system because there are more processes than available CPUs.
2. If this number is more than four times the number of available CPUs in the system, then the system is facing a shortage of CPU resources.
3. If the idle time (cpu id) is consistently 0 and the system time (cpu sy) is double the user time (cpu us), the system is facing a shortage of CPU resources

LoadRunner Page Size Capture

LoadRunner Page Size Capture


The biggest hole in Mercury LoadRunner is its lack of page size monitoring. LoadRunner can monitor anything else imaginable, including transaction counts, transaction times, errors, and all Windows Performance Monitor metrics. However, monitoring page size, download times, and HTTP Return codes are only available through programming.


//Define the data types in int Section of the script
int iPageSize;
char *grouname;
char filename[1024];
long file;
int nItr;

//Create a unique file name
strcpy(filename, "C:\\tmp.txt");

// OPen the file as write mode ie: "w"
if ((file=fopen(filename,"w"))==NULL

{
lr_output_message("unable to open the file",filename);
}


// write the iteration number in the action section top
fprintf(file, "\nCurrent iteration #: %d\n", nIt++);

Some transaction.
..
..
web_url.....
web_submit_data(...
// After every web_url(…) script when you want to capture the page size
// or At the end of transacation, add the following code snippet:

iPageSize = web_get_int_property(HTTP_INFO_DOWNLOAD_SIZE,);
fprintf(file, "Transaction name: %d Bytes\n", iPageSize);

//like that we can get the download time, return code


iRetcode=web_get_int_property(HTTP_INFO_RETURN_CODE);
downloadtime=web_get_int_property(HTTP_INFO_DOWNLOAD_TIME);

fprintf(file, "Transaction name: %d\n", iRetcode);
fprintf(file, "Transaction name: %d MilliSeconds\n", downloadtime);

//finally close the file
//Close the file after you finished all the page size capture and writing:
fclose(file);

When should do we stop our testing?

At one of time we should need stop our testing activities for the release. Here are some criteria to go for release..

1. The number of bugs found is close enough to the number you expect to find.
2. All the tests have been run.
3. All the code has been exercised.
4. Every feature has been shown to work (with every other).
5. Every use case scenario has been exercised.
6. Code agitation has fallen enough.
7. Feature stability is sufficient.
8. The system can be shown to be within some Reliability, Availability, and Maintainability Profiles defined in the test plan.
9. The number and severity of outstanding bugs has fallen to a satisfactory level.10. When all regression, smoke, and confidence tests pass.



Thanks
Senthil

First Blogging...

I got inspired from much tester community in the industry to write a blog about testing and my experience in this field.

Main purpose to write this blog
1) Own thoughts!
2) Experience
3) Sharing the knowledge which I read learned from others, etc

Thanks for reading my blog....

Regards,
Senthilnathan