Paraphrased Question on the JMeter mailing list
While working on JMeter I prepared some Test Plans in JMeter and executed
them with different combinations and I put forward the Results,graphs etc.
so please guide me what kind of details/screens/graphs/files should my
presentation include and what should not?
The first Shakespearean to be or not to be dilemma I have when I read the above is should I try to answer this question ?. On one hand I'd like to help the person asking the question, and this is more of a theoretical exercise where I can blab away without fear of consequence or effort needed to actually do what I'm saying. On the other hand the question is so generic that much of what I say will have to have so many caveats to be accurate, there will be a bunch of follow up questions I will have to answer, that my blabbing away might mislead said person because I don't really know what the person wants and as an engineer I hate to speculate without knowing. Perhaps a brusque "what are your requirements?" with an additional dose of the sad state of software testing where people don't know their goals or their requirements , if I'm in bad mood. with a slight guilty conscience since I've been here and done this and asked such questions. Should I feed the person or teach the person to fish? Since the tester seems to be a beginner will my long drawn caveat ridden, opinionated, biased opinion help the tester or make the matter worse?
Any Road.
The single most important thing to know is when you ran the tests what did you want to know about the system? What did you expect to prove? What would you consider a successful test? What would you consider a failure? or in other words What were your requirements?
If you have ever read a technical work on Testing systems , you should know that there are various types of tests and indeed there are even various types of Performance tests. Unless you know this you don't know what type of tests you need to run and you don't know how to interpret the results, indeed you probably don't even know which system to look at.
To illustrate with some examples
a. You want to know , for a specified load, whether all your operations succeed (an operation could be anything like placing an order or searching a value).
b. You want to know, for a specified load, whether your system performs acceptably
c. You want to know, for a specified load, which has been kept on the system for a sufficiently long time, your system shows acceptable characteristics.
You will notice there are a lot of weasel words in the examples above. What is "specified" load and how do we determine that?. What is "performs acceptably" and how do we determine that? What is "acceptable characteristics" and how do we determine that?
And the answer is - it depends , hence the weasel words.
Take specified load for example - How many users? How many of them hitting the site concurrently? What think time?
And answer goes back to "why are you running your tests?".
Suppose you have changed some functionality - say you refactored some code. then usually the specified load is the same as the one you see in production. i.e. you are trying to find out if your change has no impact. Success is determined if the system performs the same or better with your change.
Suppose you are doing capacity planning - Then the specified load is the number of users you expect to access your site (perhaps sometime in the future as your site grows). Success is determined by the response time and other system characteristics
Say you are going live with a new site. Then specified load is the maximum load you expect the site to see. Success is determined by the response time being lower than those specified and system characteristics being acceptable
Say you want to check if your site can take the added shopping load when Thanksgiving arrives. Then specified load is the peak burst you expect to see and so on.
It isn't easy. Most testers don't even think this is part of their job - I guess it isn't. But asking these questions should be.
Now lets complicate it some more. Lets say you like the large majority of testers are interested in the "response time" for some load. Lets break this up into two parts
a. The accuracy of the simulation and
b. The limitation of the tool under test.
a) When you run any load test, you are running a simulation (except in a very rare number of cases). The accuracy of the simulation depends on how good your test script is, but it also depends on your test environment. This is it's own blog post but unless you know your simulation is reasonably accurate or you have safety margins built in , your tests aren't any good. This isn't particularly earth shattering - but how many testers actually validate that their simulation is accurate? Many don't. If you run a JMeter with 500 threads on a single windows machine, with no think time - and you dont get an out of memory error , is your simulation accurate? In most cases the answer would be no. The JMeter machine will not be able to generate the same load as 500 windows machines each running a single user. Having 500 machines is not a practical scenario for most people (perhaps this will change with the cloud) - but the key here is if I can live with some margin of error , what is the number of machines I should have and how do I determine this? The answer is again - it depends. It depends on the application, It depends on your test, It depends on the machine you are running from, It depends on the network etc etc.
One of the ways to validate this is run the test , see some of the values you are getting from Jmeter. Now while the test is running (this is important!) , using some other machine , that is as independent from the Jmeter machine as possible , access the site under test with a browser (or even a single user JMeter thread) and see the value you get. if this is close enough to the average Jmeter value - then great!. If it varies wildly then you might have an issue. Another way is half the load on the Jmeter client machine and add another Jmeter client machine with half the load. Do you see substantial changes in response times? Do you see many more concurrent threads on the server? if so you have a problem. Repeat this till you see no significant difference. Or you can monitor the jmeter client machine - is memory/cpu pegged at highest value? If so probably the test isn't reliable.
b) Jmeter is not a browser yadda yadda. Do not expect to be able to tell browser render times without some additional tests. Running tests from an intranet is probably not going to give you the same time as when a user from some far off country accesses your site and gets a page of 500KB. Remember to factor in the network (either guesstimate or have different machines with different locations if possible)
Remember however that you can never have 100% accuracy (well perhaps you could - it depends :). An important part of engineering is knowing what tradeoffs to make when you have limited time and resources.
While working on JMeter I prepared some Test Plans in JMeter and executed
them with different combinations and I put forward the Results,graphs etc.
so please guide me what kind of details/screens/graphs/files should my
presentation include and what should not?
The first Shakespearean to be or not to be dilemma I have when I read the above is should I try to answer this question ?. On one hand I'd like to help the person asking the question, and this is more of a theoretical exercise where I can blab away without fear of consequence or effort needed to actually do what I'm saying. On the other hand the question is so generic that much of what I say will have to have so many caveats to be accurate, there will be a bunch of follow up questions I will have to answer, that my blabbing away might mislead said person because I don't really know what the person wants and as an engineer I hate to speculate without knowing. Perhaps a brusque "what are your requirements?" with an additional dose of the sad state of software testing where people don't know their goals or their requirements , if I'm in bad mood. with a slight guilty conscience since I've been here and done this and asked such questions. Should I feed the person or teach the person to fish? Since the tester seems to be a beginner will my long drawn caveat ridden, opinionated, biased opinion help the tester or make the matter worse?
Any Road.
The single most important thing to know is when you ran the tests what did you want to know about the system? What did you expect to prove? What would you consider a successful test? What would you consider a failure? or in other words What were your requirements?
If you have ever read a technical work on Testing systems , you should know that there are various types of tests and indeed there are even various types of Performance tests. Unless you know this you don't know what type of tests you need to run and you don't know how to interpret the results, indeed you probably don't even know which system to look at.
To illustrate with some examples
a. You want to know , for a specified load, whether all your operations succeed (an operation could be anything like placing an order or searching a value).
b. You want to know, for a specified load, whether your system performs acceptably
c. You want to know, for a specified load, which has been kept on the system for a sufficiently long time, your system shows acceptable characteristics.
You will notice there are a lot of weasel words in the examples above. What is "specified" load and how do we determine that?. What is "performs acceptably" and how do we determine that? What is "acceptable characteristics" and how do we determine that?
And the answer is - it depends , hence the weasel words.
Take specified load for example - How many users? How many of them hitting the site concurrently? What think time?
And answer goes back to "why are you running your tests?".
Suppose you have changed some functionality - say you refactored some code. then usually the specified load is the same as the one you see in production. i.e. you are trying to find out if your change has no impact. Success is determined if the system performs the same or better with your change.
Suppose you are doing capacity planning - Then the specified load is the number of users you expect to access your site (perhaps sometime in the future as your site grows). Success is determined by the response time and other system characteristics
Say you are going live with a new site. Then specified load is the maximum load you expect the site to see. Success is determined by the response time being lower than those specified and system characteristics being acceptable
Say you want to check if your site can take the added shopping load when Thanksgiving arrives. Then specified load is the peak burst you expect to see and so on.
It isn't easy. Most testers don't even think this is part of their job - I guess it isn't. But asking these questions should be.
Now lets complicate it some more. Lets say you like the large majority of testers are interested in the "response time" for some load. Lets break this up into two parts
a. The accuracy of the simulation and
b. The limitation of the tool under test.
a) When you run any load test, you are running a simulation (except in a very rare number of cases). The accuracy of the simulation depends on how good your test script is, but it also depends on your test environment. This is it's own blog post but unless you know your simulation is reasonably accurate or you have safety margins built in , your tests aren't any good. This isn't particularly earth shattering - but how many testers actually validate that their simulation is accurate? Many don't. If you run a JMeter with 500 threads on a single windows machine, with no think time - and you dont get an out of memory error , is your simulation accurate? In most cases the answer would be no. The JMeter machine will not be able to generate the same load as 500 windows machines each running a single user. Having 500 machines is not a practical scenario for most people (perhaps this will change with the cloud) - but the key here is if I can live with some margin of error , what is the number of machines I should have and how do I determine this? The answer is again - it depends. It depends on the application, It depends on your test, It depends on the machine you are running from, It depends on the network etc etc.
One of the ways to validate this is run the test , see some of the values you are getting from Jmeter. Now while the test is running (this is important!) , using some other machine , that is as independent from the Jmeter machine as possible , access the site under test with a browser (or even a single user JMeter thread) and see the value you get. if this is close enough to the average Jmeter value - then great!. If it varies wildly then you might have an issue. Another way is half the load on the Jmeter client machine and add another Jmeter client machine with half the load. Do you see substantial changes in response times? Do you see many more concurrent threads on the server? if so you have a problem. Repeat this till you see no significant difference. Or you can monitor the jmeter client machine - is memory/cpu pegged at highest value? If so probably the test isn't reliable.
b) Jmeter is not a browser yadda yadda. Do not expect to be able to tell browser render times without some additional tests. Running tests from an intranet is probably not going to give you the same time as when a user from some far off country accesses your site and gets a page of 500KB. Remember to factor in the network (either guesstimate or have different machines with different locations if possible)
Remember however that you can never have 100% accuracy (well perhaps you could - it depends :). An important part of engineering is knowing what tradeoffs to make when you have limited time and resources.
