I need to run N000 users concurrently and I have a Windows/Unix/Mac with N GB RAM , will that work? or how do I run N000 users from JMeter or Im running N000 users concurrently against my server, am I performance testing this correctly?
And the correct answer is (like all things related to performance testing) , test it out and see for yourself. This doesn't mean that there aren't rule of thumb's that you can follow but that no one can give you a definitive answer.
My rules of thumb
- Know the load your client machines can generate approximately(for your test) - On my Windows PC (with Vista and 3GB RAM, running JMeter with a heap of 1 GB, dual core 2.4GHZ) if I run more than 100 threads, my machine starts to hang. I can't use any other applications so I normally wouldn't run a single JMeter instance with > 100 threads for a machine configured like mine (But my tests usually don't have think times and delays). But this is a rule of thumb not a commandment. if I run Reliability and Performance monitor and check out the health of my PC , and I find it reasonably healthy then I can increase the threads, if not I'll reduce them. I also verify visually that the response time that JMeter seems to be giving me (by directly checking the JTL or CSV file being generated) is approximately the same as if I accessed the page(just the html using a proxy or browser sniffer) from a browser running on a different machine. We did have a case where a tester used LoadRunner to generate loads for 2000 users from a single machine which started giving him average response times of > 2 minutes but people who accessed the site saw response time of 15-20 seconds. The machine which was generating the load couldn't handle these many threads/sockets and was the bottleneck rather than the application being tested. This is also dependent on your test scripts and your application under test. Test scripts that have delays between requests will obviously be able to handle larger number of threads (because there wont be as many concurrent threads).
- Run separate instances of JMeter (instead of master-slave) - Jmeter allows you to run a test in master slave mode (With 1..n slaves). The advantage of this that you get all the results at the master as well as the start times etc can be more or less synchronized. However the overhead has now increased , as well as the problem that supposedly this bit of JMeter is not designed too well. Its better to run separate instances of JMeter and combine the results (if CSV is chosen as the output file format, then this is as simple as file concatenation, if XML you need to do a little more work, but its easy as well). You could split up the test itself (e.g. 100 users browsing 100 users logging in could be split up into two jmeter tests) or run the same test with some threads per machine.
- Run in command line mode, Disable all Listeners and preferably use CSV as your output format. Understand JVM options. JMeter is a java application , if you want the most out of your JMeter client you should run it light and you should know how to tune a java application (the Java Heap especially).
- Understand your actual requirements. Far too often testers don't know the difference between logged in users and concurrent users. Or think times. Or how http works. Or how their app works. Or why input data to the test should be varied (our App cached data and try as we might we couldn't get the tester to understand that if all the concurrent users browsed the exact same page, everyone except the first user's will get really fast times). Second normal users take a lot of time to read a page, fill in form fields, pause between requests, drink a cup of coffee whatever, The number of user's with an active session is always greater than the number of user's actually doing something at the same time (and in a lot of cases far greater). Check your site with realistic (plus safety margin )numbers. If there are 100 concurrent users at peak times in your system then check with numbers close to it first before you want to test out 10,000 concurrent users
- Performance testing goes hand in hand with tuning. Sometimes questions are asked of the sort 'I increased the load , I get SocketExceptions or the Applications responds very slowly , help!'. Well this is what you expect your test to tell you right? You now need to check why your application is responding slowly , using logs, profilers whatever. Rule of thumb tune , test, tune, test, tune, test, tune as often as you can.
47 comments:
Hi,
As per my perf specification it is given that there are 5000 concurrent users and a user perform 2 actions per second. This is the peak load and it will continue for 45 minutes.
We are planning to use the distribute mode in Jmeter we have 5 client machines with a separate Jmeter console.
1. As per my understanding the we have to simulate 10000 requests per second. Can Jmeter support that load with the given number of machine.
2. How can we use constant throughput in this case.
Thanks
t is given that there are 5000 concurrent users and a user perform 2 actions per second.
Know any real users who can do this :) ? So you don't have any think times in your script?
We are planning to use the distribute mode in Jmeter we have 5 client machines with a separate Jmeter console.
It isnt what I would do. I would run instances separately and combine the results at the end.
If your test is exactly as you state then you are planning on running 1000 threads per machine. Unless you have server class machines (which i doubt) , the test might work , but I think your margin of error (i.e. what time does jmeter see as opposed to a single user on a different machine for the same request) will be quite large. However if you have significant think times in your script then the usage is actually less than 1000 threads. As mentioned in the post my experience is a normal windows PC can run about 100 threads with acceptable margins of error. Unix machines you can probably get some more.
>2. How can we use constant >throughput in this case.
Well it depends. you state that a user makes a 1 request every 2 seconds - but in how much time does the response come? you seem to want a user to make 30 requests per minute and that depends on how fast does the response actually come back.
Thanks for you reply. Please see my comments
1. "So you don't have any think times in your script?"
Comment -Yes. I have given think times
"It isnt what I would do. I would run instances separately and
combine the results at the end"
Comment - So what you mean here is to divide these 5000 users among few machines but without distributing. In this case how many Jmeter machines would you need? How many threads will a Jmeter machine support before saturate? Also in the current situation we are restricted to have only 5 - 6 machines max.
As you said there is no way that we can take a server class machine.
Do you know any other open source tool which supports this much of a load requirement?
2. When you say throughput my idea was that number of transactions per second/minute.
Is there a way that you can use constant through put timer of jmeter? Also is there a way to know how fast the response is coming back?
>Comment -Yes. I have given think times
So then how does a user make two actions in a second? Usually you spend a lot of time reading or filling in a form. In which case what you have is 5000 sessions but a much smaller concurrent footprint. You need to calculate the concurrent footprint. Say on an average a request takes 1 second and on an average a user waits 9 seconds. So a user can only make a request every 10 seconds. In which case on an average 5000 users make 5000 requests in 10 seconds and so 5000 active users make 500 concurrent requests per second. (assume you have already added in a safety margin) and this is what your jmeter instances need to be able to do. What does your value come upto?
>How many threads will a Jmeter >machine support before saturate?
Operating system dependent and the amount of resources you have on the machine. On reasonably sized winodws you can get 100 threads running concurrently and still have some accuracy(the number of active threads can be larger). On *nix you can probably get better.
Do you know any other open source tool which supports this much of load without any overhead on itself?
A tool that adds no overhead can only do nothing :).
I suppose you are asking for tools with minimal overhead - I would guess that a well written C/C++ tool can probably outperform any Java tool.
But a windows/linux machine is cheap and its far simpler to add a machine when you want to increase load than try to extract the most performance from the tool
Hi,
I one of our System, Whie an attempt to run 100 concurrent threads of services accessing MainFrame,heap dump OutOfMemory error has been displayed.
From the next time, when opening/Booting jmeter itself OutOfMemory error s displayed.
Any idea of How to start the jmeter . ? get rid of the heapdump
?
what is the exact error message you get while starting up Jmeter?
For your test running out of memory thats been discussed multiple times on the mailing list - usually you are running gui mode with memory intensive listeners. There are recommendations on the wiki - http://jmeter.apache.org/usermanual/best-practices.html#lean_mean
In some cases you are trying to do too much with a single jmeter instance and you need to use more instances or use a bigger instance.
Hi ,
as per my requirement i have 10,000 concurrent sessions which includes 3 actions.ramp up period is 30 minutes.do i need to run instances seperately and combine the result in single machine.or do i need additional machine for this requirement? should i use delays ?
Thanks
Assuming you are using the latest versions of JMeter
a. When you say 3 actions , you mean each thread is to perform 3 actions and then quit? i.e. the number of iterations is 1?
If yes then 30 minutes =1800 seconds. So assuming you have an average time of 10 seconds per request that mean you have
(10000/1800)*(3request * 10 seconds) = about 167 threads running concurrently which a single jmeter instance can handle
if no(i.e. you have iterations set ) then you need more machines . How many more depend on the capability of your machine - as always you have to check and see - no one can give you an answer
should i use delays ?
Only you can answer this question based on what you intend this test to do and how you expect to intepret your results.
Hi Deepak , thnaks for ur reply !
one more question..i want to pass 10000 user account details , along with the request .
which is the efficient way to do the above?
CSV Data Set Config
I would like to run a load test for 300 logins per minute
I would like to know some difference between below two thread group configuration.
Thread Group:7500
Ramp up:0
Loop count:forever
Schedular:1 hr
Synchronizing timer:7500
constant timer:${__javaScript(${__threadNum()}*200)}
the 2nd configuration
Thread group:400
Ramp up:0
loop count forever
schedular:1 hr
synchronizing timer:400
constant timer:${__javaScript(${__threadNum()}*200)}
My assumption is this both setting will run 300 calls per minute because of constant timer.
I get the following error
with 7500 thread group
java.lang.OutOfMemoryError: GC overhead limit exceeded
Dumping heap to java_pid9160.hprof ...
Heap dump file created [712333712 bytes in 11.873 secs]
Is Thread Group 7500 meant to represent 7500 threads? In which case the amount of memory you need is more , which in turn means that your heap gets over quickly , which in turn means your GC runs more often and in your case doesn't release enough memory
yes number of threads = 7500.
I have a question about distributed testing. Wondering if you could give some tips on it.
Lets say I have one master and one slave.
Requirements:
Number of logins per min = 300
Run the test for 4 hrs.
Lets say I have 8000 users on my csv File and I am reading the username using CSV Data Set Config.
Now my first question is:
How do configure my Thread Group property.
correct me if i am wrong this is what I thought of
Thread Group
Number of threads:300
Ramp up period:60
loop Count:Forever
Runtime Controller
Runtime: 14400 seconds
CSV Data Set Config:
Recycle on EOF: True
so it can loop the same user to make it run for 4 hrs)
2nd Question:
If I would like to run distributed testing. I have seen the document on apache jmeter site but on that document it is not running the script on the master machine just the slaves. But I have a master and one slave and I would like to run the test on both (master and slave). How do I achieve this?
3rd question:
Above i have mentioned I would like to run 300 logins per minute. So, since i have a master and one slave and I would like to split this. Run 150 login on master and 150 login on slave concurrently.
So on Thread Group Property
Do I put Number of Threads: 150 or 300.
@umesh
The number of requests per second has to be controlled by using constant throughput timer (or the one from jmeter-plugins). How many threads you use depends.
Say a request returns on average within 1 second and you have no think times in your test then to simulate 300 requests per minute you only need 5 threads. On the other hand if a requests needs 2 minutes to complete then even 300 threads cannot simulate 300 requests per second. So you need to give a large enough vslues of threads that can simulate atleast 300 request per second. If you configure more , then constant through put timer can throttle to the value you want but if you give less the timer cannot increase it - so you have to always give a little more than you need. The big problem however is that you dont know whether your responses are going to take 1 second or 1 minute - in which case you can do it one of 2 ways - run the test multiple times tuning the values. The other is to say that for e..g. if the page needs 1 minute to respond , then there is a problem anyway so you dont need more than 300 threads to generate your load - if this doesnt generate 300 requests per second that means responses are taking too long and this test has no use beyond proving it
2) You shouldnt do this - the master and the slave on the same machine is an inefficient setup. you can always do this though.
3)Thread group threads are scoped to Java VM so if you configure 300 threads , its 300 threads on every slave
But as I mention earlier in comments the more efficient setup is to not use master-slave and just use independent jmeter instances and merge the results
Hi,
I want to run a test for 100 users using 4 scenarios with 2%%,25%,15%, 35% respectively. I want to run all the scenarios in one thread group based on the percentage. Is it possible.
2. I tried using threadgroups for each scenario in one test plan and for each thread group i put the users , rampup and loops...
Please let know if this is possible using Jmeter. I have worked in LR but in jmeter i am not able to do the same way. This is an urgent requirement as we need to submit report to the customer. Your suggestion is valuable.
@Manonmani
use Ultimate Thread group from jmeter-plugins
http://jmeter-plugins.org/wiki/UltimateThreadGroup/?utm_source=jpgc&utm_medium=link&utm_campaign=UltimateThreadGroup
Sorry it appears I misread your question - use throughput controllers under your thread group with the % you need
http://jmeter.apache.org/usermanual/component_reference.html#Throughput_Controller
Thanks deepak. I tried this option. Please correct if this is the way it will work
Test Plan
Thread Group
cookie
cache
http request defaults
Scenario 1
Transaction Controller (percent execution (25) and uncheck Per user
Page clicks inside the transaction controller( this is to get a response of the whole click)
Is this is the way you are telling pls provide me a sample so that it will be very easy.
So there will be only 1 thread and all the 5 scenarios will be in the thread. When i give 100 threads with ramp up then the threads will be automatically divided as per the scenario percentage.
Extremely sorry for troubling. Thanks much for the support.
If the order doesnt matter to you then yes
ThreadGroup
+Throughput controller (25%)
++Sampler1 (or more)
+Throughput controller (25%)
++Sampler2
+Throughput controller (15%)
++Sampler3
+Throughput controller (35%)
++Sampler4
But you might see sampler4 executed first then sampler1 or whatever.
other examples (simulating drop off of users is here)
http://theworkaholic.blogspot.com/2013/04/simulating-abanadoned-flows-in-jmeter.html
Hi,
I have created Master-slave configuration using Jmeter with 1 master and 2 Slaves with thread number as 100 and ramp up time as 400 which means each sample will take 4 sec to run
I have two queries
1.If I ran 100 threads on Master does that mean that both slaves will run 100 each or will distribute 50 each or equal numbers.
2.The result was that the thread groups were running for more than 500 number of threads and same was being reflected in the application so from where can we know that if the threads created are duplicate or not as the number of threads are really large
@Anonymous
1 master and 2 Slaves with thread number as 100 and ramp up time as 400 which means each sample will take 4 sec to run
This is not what ramp up time means. It is the total time Jmeter takes to start up threads which has no relation to the amount of time for a sample , it only makes a difference for the initial concurrency as well as helps simulate actual scenarios (in a normal website , not everyone is doing the same action at the same time)
1.If I ran 100 threads on Master does that mean that both slaves will run 100 each or will distribute 50 each or equal numbers.
100 each
.The result was that the thread groups were running for more than 500 number of threads and same was being reflected in the application
Are you sure you are talking about number of threads? For e.g. if your test has 10 samplers and your iteration number is 10 and you use 2 threads (single Jmeter) - at any given point of time , the max number of requests in parallel that JMeter will make is 2 (= number of threads).
However the total requests made is 2(threads)*10(number of samplers) *10 (number of iterations).
The only scenario in which you will see more than 2 threads on the application server is if your request from Jmeter times out . So if JMeter started up 2 thread s ,made two requests and those requests timed out , they will still be running in your application server and since Jmeter gets an error it will move onto the next two requests so your application server will see 4 requests
2.The result was that the thread groups
Do you have more than one thread group? then if you have specified run in parallel then total threads = sum of threads of each thread group .
Thanks a lot for your answers
Yesterday I ran 300 threads on Master with 6 slaves created and it started running properly but after completing 3 slaves it stopped and got hanged.
Again when i restarted the process it not even started for single system and got hanged.So I cannot complete the same.
In error log there were some deadlocks and exceptions but
What can be the issue with jmeter?
Hi ,
Using Distributed testing , lets condiser a scenario
1. User login and reached Dashboard
2. Admin login and performs some set of actions
So how to acheive the above scenario using Distributed testing approach
Here I would like to check the performance of both of the scnearios executing simultaneously .
Even i tried with creating 2 ThreadGroups under single jmeter instance
i.e
Thread group1 : User login and reached Dashboard
Thread Group2 : Admin login and does his set of actions
Using Distributed testing is it possible to achive this scenario ??
If yes , can you please explain in brief .
Thanks in advance .
Hi
you are mixing up two things.
a) How do I write a test which has two or more distinct type of actions (in your case an admin v/s normal user)
b) Given any jmeter test , how do I run it distributed - i.e. in multiple machines(and there isnt anything specific to your example that makes it different to any other jmeter script)
it seems you have a problem with a) but the question you are asking keeps mentioning "distributed" so I am not sure.
For
a) There are two basic approaches.
If your admin screens is completely different from your user and you need different numbers of admin v/s normal users (usually the case) - then as you say use two different thread groups where one thread group is for admin user and another for normal user. If the screens are mostly common then you could use any of the functional controllers (like IF controller or the throughput controller ) to have one thread groups but different steps and actions for user v/s admin. I would most likely go with separate thread groups.
Note that this will run in parallel and work just fine and depending on the machine you are running Jmeter on , will be able to support some number of( users + admin)
Only if you need to run more concurrent users than your machine supports , that you need to worry about "distributed" - and if that is the case then you can go ahead and distribute the test as you normally do - there is nothing different about your use case
Hi,
How the result of multiple threads accessing single sample is displayed in Jmeter's View Results in Table listener?
Eg:The Number of Threads:3
The 3 threads are accessing same sample (HTTP Request) then in View Results in Table listener the same sample will be displayed thrice or a single sample will have aggregate result of 3 threads?
Please elaborate.
Thanks,
Swapnil
The View results in table the same sample will be displayed thrice as it is simply shows every sample.
However the Summary Report will show data aggregated by Sample Name - so by default it would show 1 row for 3 threads.
Its easy to test out the various listeners to see how they behave or check the docs
http://jmeter.apache.org/usermanual/component_reference.html#listeners
Hi All,
i want to do load testing of 8000 concurrent users in JMeter.
I have also made distributed system.I want to know what would be realistic scenarios.
What should we need to use for the same?
1.Thread Group
2. jp@gc Stepping Thread
3. jp@gc Ultimate Thread
i have 3 static pages as followings.
Home>About Us>Careers
Thanks in advance.
P.S. I am new in load testing please forgive if anything is not properly written.
@Arun
Its hard to answer your question because it is too broad.
>I want to know what would be realistic scenarios.
If you have an existing site that you want to load test , then you should already be able to figure out what a realistic scenario is by looking at how people are using your site. if you are building a new site , then About Us and Careers are really rarely accessed as compared to whatever your site does(unless this is a jobs site!) - 8000 concurrent is a large number.
You need to figure out what you need to test - and thats something only someone associated with your site can tell you - either what your current usage pattern is , or what its predicted to be - after that you can decided which JMeter component best serves your need.
Hi, I am stuck in in doing load testing, pls help.
I am doing load testing using JM for a portal, I recorded the script for logging in and searching order and parameterised for multiple users, but there is a tab where in users upload csv file and that creates an order, when i tried to record this JM proxy recorder, the csv file is not uploading only, is there any way of setup to be done in JM so it can record this scenario or how can i create a testplan to upload a csv file to that portal?
Are you saying the proxy recorder itself is not recording or the browser is unable to upload the file when you have the proxy ON? If your file upload tool is using something like flash / active X / java applet, thats possibly the cause since those tools do not necessarily work behind a proxy. The other possibility is your browser is bypassing the proxy for the location where the file is being uploaded so you also need to check browser settings
In order to see whats happening you can either inspect the code , speak to a dev , or use a tool that hooks into the browser (like Fiddler or Firebug) and perform your actions and see what the tools say is happening. Its always possible to write the script manually using the recorded request/response from Fiddler/FireBug etc - it depends on how your system is actually uploading the file. if its just using the standard multi-part upload then it should work in JMeter.
Hi deepak...thanks for reply...actually jmeter script recording works only in firefox right? Soo i set up firefox for manual proxy and started jmeter proxy server and recorded logging in and basic searching..and then i went to this tab where the browser opened browse window and selected the csv file too then when i click upload..it throwed an exception..when i saw the script recorded there was an instance id recorded at that point..when same scenario is done notmal browser that is not while jmeter proxy is set up, upload is successful...soo what are my options now..i thought of recording same in badboy tool..n one more alternative is i have asked that dev to give that url port number etc so that i can set up a post html request and try uploading..as said i dont know how to find whether it has upload tool like flash, upload x etc..how can i find it? If i use firbug to record how can i find whats happening? And yes vary much thankful for your advice :)
Hi deeepak..thanks for reply..actually when i try to upload file firefox as firefox is the only one that supports recording, its throwing me an exception..can you tell me how to see the file upload technique used?? Andwhat i have thought till now is getting that link where the file upload takes place from dev and trying with a single http request to upload and see whether the order is created in gui..
Hi
Firefox is not the only one that supports recording - Any browser has a way to configure a proxy .(For e.g. IE is Internet Options - Connection - Lan settings)
>can you tell me how to see the file upload technique used??
View source if you know HTML and javascript or you have to talk to a dev.
i will try it out...thanks mate:)
Jmeter 2.13 is not working on OS X El Capitan of MAC. Any help on this is much appreciated.
Thanks in advance.
Hi
sorry I dont have access to your configuration - however JMeter is a pure java app - so you likely have to check your java install/version and see.
Also what do you mean by not working ? you launch it and it doesnt start up? run the command line and see what error you get
Hi,
Thank you for reverting back. Application was launching, however was not able to record and playback the scripts.
Few minor configuration problem was there which i have fixed it. Now its working fine.
Thank again!
Have any of you wondered why a specific, unique thread is required to represent a load test user ? Technically isn't each thread just sitting there, waiting for the response to arrive, and thus wasting system resources ?
I think JMeter is suffering from an age old implementation in the "making http hits" part.
Shouldn't we be using non blocking io (https://en.wikipedia.org/wiki/Non-blocking_I/O_(Java)) these days so that threads don't block ?
@BinithB
>Have any of you wondered why a specific, unique thread is required to represent a load test use
It's not. And you can do this in JMeter too - The problem you will run into is how do you represent shared state (like a cookie ) - If you decide to do it yourself , you have to take care of standard concurrency. Its just easier to do it per thread , but its definitely not a requirement. Theres is nothing within JMeter that says that two requests that a thread makes need to represent the same user (and if for e.g. you dont use a cookie manager they wont)- however its upto you to figure out how you are going to manage the multiple sequence of steps that a user will take
>Shouldn't we be using non blocking io
Perhaps. It isnt quite clear that it will be more efficient though. Say 10 threads make requests and its non blocking and they move on to do other things , there still needs to be someone who is going to wait for the response - The only way to guarantee that you will receive those responses correctly is by having 10 other threads. The fact that the first 10 threads can do some other work , hasnt removed the fact that 10 threads need to wait. You also have a non deterministic number of threads needed - Say the responses take a long time to return - And your original 10 threads keep making more requests - say each make 10 more requests and their responses dont return - do you now need 110 threads ?
I believe based on what a JMeter committer replied , it would need a complete rewrite of JMeter too.
Also just because you can generate a larger load from the same Jmeter instance doesnt mean your test itself is going to give you better results - it might give you worse ones
Hi Deepak,
I have created a test plan in which i have set no of threads to 1000.
Under Listener i added 'Summary Report'
I have used Transaction controller to club the total response time for the page.
Running the script in non-gui mode.
Set Duration : 280secs
After reaching this duration Jmeter stops the execution of each thread one by one. But meanwhile Jmeter gets freezed and the for last few thread (say 25), the test doesnt stop its execution completely.
it depends what these threads are doing - for which you will have to take a thread dump.
If the thread has already begun a request and its waiting for your server to respond , then it cant be interrupted till the request returns. One way to workaround this is to specify a timeout on the HTTPrequest so it will always return within a fixed time.
Hi,
How long Jmeter can wait for the response of request if we don't add any timeouts?
JMeter wont have a problem waiting infinitely(like any java process) - but then you will also start running into OS settings . But its a good time to ask why you would want to this..
Is page reading is good or bad? For 1494 samplers count its average reading is 25519ms.
Can you please explain me that how to analyse this?
Sorry I do not understand your question. JMeter isn't an analysis tool - if you are asking why your sample is taking 26 seconds - Jmeter can only report something took time - what in your application caused it to take time will need you to analyse the application(e.g. in java you would use a profiler)
If on the other hand your question is JMeter says sample takes 25seconds but my application looks faster then you will need to check whether JMeter is out of resources or whether your application has a problem
Post a Comment