I need to run N000 users concurrently and I have a Windows/Unix/Mac with N GB RAM , will that work? or how do I run N000 users from JMeter or Im running N000 users concurrently against my server, am I performance testing this correctly?
And the correct answer is (like all things related to performance testing) , test it out and see for yourself. This doesn't mean that there aren't rule of thumb's that you can follow but that no one can give you a definitive answer.
My rules of thumb
- Know the load your client machines can generate approximately(for your test) - On my Windows PC (with Vista and 3GB RAM, running JMeter with a heap of 1 GB, dual core 2.4GHZ) if I run more than 100 threads, my machine starts to hang. I can't use any other applications so I normally wouldn't run a single JMeter instance with > 100 threads for a machine configured like mine (But my tests usually don't have think times and delays). But this is a rule of thumb not a commandment. if I run Reliability and Performance monitor and check out the health of my PC , and I find it reasonably healthy then I can increase the threads, if not I'll reduce them. I also verify visually that the response time that JMeter seems to be giving me (by directly checking the JTL or CSV file being generated) is approximately the same as if I accessed the page(just the html using a proxy or browser sniffer) from a browser running on a different machine. We did have a case where a tester used LoadRunner to generate loads for 2000 users from a single machine which started giving him average response times of > 2 minutes but people who accessed the site saw response time of 15-20 seconds. The machine which was generating the load couldn't handle these many threads/sockets and was the bottleneck rather than the application being tested. This is also dependent on your test scripts and your application under test. Test scripts that have delays between requests will obviously be able to handle larger number of threads (because there wont be as many concurrent threads).
- Run separate instances of JMeter (instead of master-slave) - Jmeter allows you to run a test in master slave mode (With 1..n slaves). The advantage of this that you get all the results at the master as well as the start times etc can be more or less synchronized. However the overhead has now increased , as well as the problem that supposedly this bit of JMeter is not designed too well. Its better to run separate instances of JMeter and combine the results (if CSV is chosen as the output file format, then this is as simple as file concatenation, if XML you need to do a little more work, but its easy as well). You could split up the test itself (e.g. 100 users browsing 100 users logging in could be split up into two jmeter tests) or run the same test with some threads per machine.
- Run in command line mode, Disable all Listeners and preferably use CSV as your output format. Understand JVM options. JMeter is a java application , if you want the most out of your JMeter client you should run it light and you should know how to tune a java application (the Java Heap especially).
- Understand your actual requirements. Far too often testers don't know the difference between logged in users and concurrent users. Or think times. Or how http works. Or how their app works. Or why input data to the test should be varied (our App cached data and try as we might we couldn't get the tester to understand that if all the concurrent users browsed the exact same page, everyone except the first user's will get really fast times). Second normal users take a lot of time to read a page, fill in form fields, pause between requests, drink a cup of coffee whatever, The number of user's with an active session is always greater than the number of user's actually doing something at the same time (and in a lot of cases far greater). Check your site with realistic (plus safety margin )numbers. If there are 100 concurrent users at peak times in your system then check with numbers close to it first before you want to test out 10,000 concurrent users
- Performance testing goes hand in hand with tuning. Sometimes questions are asked of the sort 'I increased the load , I get SocketExceptions or the Applications responds very slowly , help!'. Well this is what you expect your test to tell you right? You now need to check why your application is responding slowly , using logs, profilers whatever. Rule of thumb tune , test, tune, test, tune, test, tune as often as you can.
