Friday, March 19, 2010

Dynamic Parameters in JMeter

Motivation
You need to send a variable number of parameters with an HTTP request i.e. the number of parameters are not known when the script is written. For example a User can register with a number of accounts , but this number varies per user. Or perhaps you want to Post process extract a number of hidden fields on the previous response and post all of them in the next Request, but you either don't know the number or there are too many fields to enter manually into the HTTPSampler.

Solution
Use a BeanShell PreProcessor to dynamically add variables to the sampler. The relevant code is sampler.addArgument(name,value)[1]

Sample
We will write a script to access a page , extract out data based on a pattern and send these as separate parameters to the next request. The script is as shown below.
We make a request to some dummy page and then we have a regex extractor to extract out multiple values. In this case the Regex extracts out all words(Match No = -1) beginning with test




Then we have a Debug Sampler to verify the regex is working, notice the results in View Results Tree Listener. The Regex has a reference variable name of inputTerms. The response of the Debug sampler is shown below

You can see from the Listeners response tab that the total number of results is 14 , available under the key inputTerms_matchNr and that each result is available as inputTerms_$resultnumber.
The matching groups which we usually do not need are available as inputTerms_$resultnumber_g$groupnumber. Armed with this information, the beanshell preprocessor script is trivial as shown below
int count = Integer.parseInt(vars.get("inputTerms_matchNr"));
for(int i=1;i<=count;i++) { //regex counts are 1 based
sampler.addArgument("hardcodedkey", vars.get("inputTerms_" + i));
}
Heres the next request to which the beanshell pre processor is attached. We have one predefined variable and the rest will be dynamically added by beanshell.

Next we check the request data we are posting using the Request tab of the View Results Tree Listener



Notice that the argument we added to the next request was sent, as were all the parameters we added dynamically. The parameter values were also encoded.
Sample Script : https://skydrive.live.com/#cid=1BD02FE33F80B8AC&id=1BD02FE33F80B8AC!268

71 comments:

Anonymous said...

Great article. Thanks!
Alex

Anonymous said...

great!!!!

RRP said...

Nice article... will help a lots of guys in need..
It helped me...

Saurabh Thakur said...

Nice short article ..would love to see more on Usage of Beanshell in Jmeter.

Anonymous said...

you cant beieve how lucky i am to have found this article. really really good. thank you so much!

Anonymous said...

i am new to jmeter. Will u plz post the step by step procedure to run this script.
Thanks in advance.

Deepak Shetty said...

@anonymous
Download script from https://skydrive.live.com/#cid=1BD02FE33F80B8AC&id=1BD02FE33F80B8AC!268

The script name is dynamic parameters. launch Jmeter and open this script file and run it.

Note that you rarely need to add parameters dynamically (typically you know what parameters and how many there are in the script so you can directly specify them)

Karthik said...

Thanks you very much mate. I've been scratching my head like hell to figure out how to add variable number of parameters dynamically. This helped me a lot.
Karthik

Anonymous said...

Awesome Work buddy. That's what I was looking for.

Thanks Again.

siva said...

hi, i have multiple select tags and i wanna get option value based on select tag id attribute value using regular expression.

Deepak Shetty said...

@Siva
XPath Post processor is usually easier in such cases (but less memory efficient and slower)
It might still be possible to do it with regex but the regex itself is much more difficult. The other option is using the id extract out the Select HTML. Then write a second regex that operates on this extracted Select box to extract out the option

Unknown said...

Hi Deepak

Excellent post mate!!!

Just wanted to know that what needs to be done if I do not want to hardcode the value of the paarameter..say I want the parameters to contain values from a predefined CSV file..Could you please help me onm this?
Thanks
Nabojyoti(sarkar.nabojyoti@gmail.com)

Deepak Shetty said...

@Nabojyoti Sarkar
The API allows you to specify whatever key/value you want and JMeter/java allow you to write whatever logic you want.

For e.g. if your CSV is read using a CSV data set config and you have 4 columns(column1 = name of parameter,column2=value of the same parameter) which map to variables col1 ,col2,col3,col4

then in the Beanshell pre processor you could write

sampler.addArgument(vars.get("col1"),vars.get("col2"));

Again the main point is sampler.addArgument(key,value)
How you get key and value isnt really relevant and can be done in many different ways

Unknown said...

Thanks a lot Deepak for your response..thanks for helping me understanding the working of this API, however, I am unable to do something using this. I aim at generating random number of paramaters (successfully done that with your help) and then assign values to the parameters from a csv file. The csv file has two columns and few rows. I want the first parameter to get value from cell(row1,col1), second parameter to get value from cell(row2,col1) and like wise...can you pls provide me the code how to do this? Currently it is only populating the parameters with the values present in cell (row1,col1)

Deepak Shetty said...

@Nabojyoti
Usually your excel file should be modelled as
row1 = data for 1 case
row2 = data for 1 case
i.e. each row should have the data .
In your example you want the column to have the data - its easier if you transpose your excel.
If you can then simply use the CSV Data Set Config
Thread Group
+While Controller (with condition)
++HTTP request Sampler
+++BeanShell Preproc
++CSV Data Set config(var1,var2...)

Then for each time the while controller runs the values you declare in the variables (in the example var1, var2) will move to the next row.
Because the reading is row by row what you want is the data for one test case to be contained in the row so parameter 1 = row1,col1
parameter 2 = row1, col2

You need to specify the sharing mode on CSV data set config and you also need to provide a condition when you want to stop (for e..g. if you want to stop when there are no more lines in the CSV then you use "${var1}" != ""



Rachna said...

Awesome soved my problems in a seconds :) :) Thanks a ton

krishna said...

@deepak


HI bro, can u explain me how to read an excel file using beanshell

Deepak Shetty said...

@krishna
Not related to this post - please send jmeter questions to the jmeter mailing list.
easiest way is to export excel to CSV and use CSV in JMeter. other way is to use POI or JExcelAPI to read the file and iterate using beanshell or custom elements - however that is too much to describe in a comment

Sattish said...

Hi Deepak,

Thanks for this nice article. It helped me a lot

Sattish.

Anonymous said...

Great article.
It helps me a lot!

Anonymous said...

God bless you. Very Very Helpful Articles.

Anonymous said...

Thank you, thank you, thank you, thank you.

Anonymous said...

HI
It thorows an error while executing
ERROR - jmeter.util.BeanShellInterpreter: Error invoking bsh method: eval Sourced file: inline evaluation of: ``int count = Integer.parseInt(vars.get("inputTerms_matchNr")); for(int i=1;i<=cou . . . '' : Typed variable declaration : Method Invocation Integer.parseInt

Deepak Shetty said...

@Anonymous
Is this your full script? most likely your variable is not called inputTerms (reference name in your regex ) or your regex is messed up or something.
Did you look at the result of debug sampler to check that the values are getting extracted as you expect?

Anonymous said...

Hi Deepak
Thanks Deepak for the response.In bean shell post processor,the script what I have mentioned is int count = Integer.parseInt(vars.get("Input Terms_matchNr"));
and in the HTTP request i'm passing the parameter "${count}" in the value section.

Note:Now the error is solved.But no value is captured in ${count} when i checked the view result tree.I'm expecting the value "8" for "count".For the response for the earlier request the value shown for "inputTerms_matchNr" is "8" and it is shown in debug sampler.

Deepak Shetty said...

as mentioned you need to check your script (variable name of Input Terms with a space in it looks wrong) - steps to debug were also specified in previous response. Basically you are not returning a parseable number so Integer.parseInt is failing

subrat said...

Hi,

I am facing one issue in Jmeter.
There are some parameters which are generated dynamically in each HTTP Request.
When data is POST,one random value is generated while getting response and append in the path.(xsbipaqc.corp.xerox.com/ibi_apps/WFServlet?IBIAPP_app=XSBID&IBIF_ex=xs_card_services&random_number=1415172585774)
This value vary for each request. I want to know how to deal with. It should be different for every thread.

Thanks,
Subrat Ray

Deepak Shetty said...

@subrat
There are two possibilities
a. This is a true random number generated which has no meaning to the server other than it is random (usually HTML apps do something like this to ensure that the browser does not cache the page since every URL is effectively new)
In this case just use the standard Jmeter function (Random) append with something like threadNum function- function help is http://jmeter.apache.org/usermanual/functions.html
You can write custom java code if you need the random number to be some specific format and call that too

b. This number is something that the server generates and it is essential that the next request passes exactly the same value (for e.g. CSRF tokens or primary keys) -
In this case the server will be returning you this value somewhere and you have to add a post processor (usually regex post processor) to extract out this value from the response into a variable and then send it to the next request using ${variable}
Post processors are described here
http://jmeter.apache.org/usermanual/component_reference.html
These are automatically different for each thread since the server will usually generate different values for each request and you will extract and store it into a variable which is scoped for the thread

Which one of the two options you have to choose is based on your application.

subrat said...

Thanks Deepak,

I tried with the Regex Post Processor to deal with it. I am getting the following exception:-
Illegal character in query (xsbipaqc.corp.xerox.com/ibi_apps/WFServlet?IBIAPP_app=XSBID&IBIF_ex=xs_card_services&${test}).

Reference Name: test
Regular Expression: random_number="(.+?)"
Template: $1$
Match No.: -1


Where I am going wrong. I think there is problem with regular expression, I tried some other expressions as well but didn't get any solution.

Deepak Shetty said...

Your regular expression didnt work (and it doesnt make sense)
So test wasnt setup as a varaible (which means there is either a syntax error or you placed it in the wrong location - the regex post processor needs to be achild of the sampler that is returning the response you want to extract data from)

You need to understand how your application works before you can write a test for this - For e.g.
Did you identify , which response is returning these values ? Then you need to look at the surrounding text and write a regex that extracts the value you want - instead you have written a regular expression that says Match any character, atleast once, but as few as possible - which makes no sense.

It should be something like this
ThreadGroup
+HTTP Request 1 (assuming the text contains the random_number in its response somewhere)
++Regex Pst processor (variable=test)
+Http Request2 (${test}

The + represents indentation level
+Http

subrat said...

Thanks Deepak,

In our application, there is one parameter (request_ID in our application) which is generated from the server side how can we handle this parameter at our end. This parameter is getting changed in each request.

One more issue is there, For say, I have recorded one scenario and play it with single thread only it executes well but after 5-10 mins report data doesn't display because at server side report which is cached is cleared after some specific time. What will be the solution for this.
Can we handle this thing in Jmeter?

Deepak Shetty said...


In our application, there is one parameter (request_ID in our application) which is generated from the server side how can we handle this parameter at our end. This parameter is getting changed in each request.

if you say the server is generating this parameter and in addition it is different for every request , then every response from the server must have this parameter - and all you need to do is extract it out (into a variable ) and send it with the next request.
You can scope a Post processor so that it executes for every request in scope
For e,g
Simple Controller
+Request 1
++Regex extractor1
+Request2
+regex extractor2
In this case regex extractor1 will only get applied to the response of request 1 (since it is a child of request 1) but regex extractor2 gets applied to both request1 and Request2 (since it is a child of simple controller)

For say, I have recorded one scenario and play it with single thread only it executes well but after 5-10 mins report data doesn't display because at server side report which is cached is cleared after some specific time. What will be the solution for this.
Other than something is wrong with your test (assuming it works fine for a browser) - there is nothing I can say - Most likely you have something like a session id that has expired (when you recorded , it recorded the id and then when you play it , it is using the same id, rather than requesting a new one) -- it is not possible to remotely debug problem with the level of information that you provide.
To repeat - you have to understand how your application works , first, then replicate the same in Jmeter.
One way is compare what the browser (or your application) does with whatever you are sending in Jmeter.

Anonymous said...

Hi Deepak,

I need to create a dynamic POST request in JMeter. The request contains an array list. The list data is available in a .csv file. If you can give some suggestion using beanshell preprocessor how I can achieve this will be helpful.

Deepak Shetty said...

@Anonymous
" The request contains an array list. "
There isn't any such thing as an "array list" in HTTP - all you have is String - Determine what you actually need passed - eg. you might be passing a single parameter with comma delimited string or you might be passing multiple parameters with the same name - once you know that this is easy to achieve in JMeter.

Unknown said...
This comment has been removed by the author.
Unknown said...
This comment has been removed by the author.
Unknown said...
This comment has been removed by the author.
Deepak Shetty said...

Im not sure exactly what you mean by "next iteration" but here is one way

Sampler Method1(returns 1000 ids)
+Extract out all ids into a variable( specifying -1 in the regex post processors)
While Controller (using __counter function to compare )
+Sampler Method 2
++ BeanShell pre processor (pick out 100 ids and set them into Sampler - use the counter variable to determine which iteration you are on and the code in this blog post as an example)

Unknown said...
This comment has been removed by the author.
Deepak Shetty said...

Hi
Your code snippet is correct - You are extracting all the IDs - However what you are probably having a problem with is how to access the 1000 elements?
In which case look at the example in the post
Adding a debug sampler shows how Jmeter extracts the variables (so varName_1 and varName_2 ...
The beanshell script shows you how to loop over the entire set.

So verify your regex is correct by debug sampler - are you only seeing one value in the response of the debug sampler - in which case the regex is wrong (most likely because its greedy and your data seems to be in a single line)

OrgId":{([^}])*}

I havent tested it - please use any regex tester

Unknown said...
This comment has been removed by the author.
Unknown said...
This comment has been removed by the author.
Unknown said...
This comment has been removed by the author.
Unknown said...
This comment has been removed by the author.
Deepak Shetty said...

Hi
If your regex only picks up one value
it is wrong.
I have uploaded a sample that does some of what you want to do - I hardcoded the response to return 4 items and the next sampler picks up 2 items at a time - You can change the code to look at the total number of results and adjust it so its all dynamic
However I am out of the country after this so please send further queries to Jmeter mailing list

https://onedrive.live.com/redir?resid=1BD02FE33F80B8AC!945&authkey=!AFzctxMn9ylCsHc&ithint=file%2cjmx

Unknown said...

Thanks Deepak! I'll try that today.

Anonymous said...

Hi Deepak,

I am recording some reports through Jmeter. Lets say there are 521 pages in a single report. When I execute the recorded script, In HTML view (View Result Tree Listener)I can see only 22 of 521 pages.

My question is:- "Is there any limitaion of Jmeter that it can only record some amount of data?"

Thanks in advance

Deepak Shetty said...

@Anonymous
"Is there any limitaion of Jmeter that it can only record some amount of data?"
No. But there are valid reasons for the recorder doing things different than what you expect.
I believe your question is to do with recording (but the latter half of your question deals with the execution of the script - which is a different thing).
So assuming you are talking about recording the script - I suppose you are saying that you clicked some links 521 times but only 22 of them got recorded into the script (note you shouldnt have yet run the script)- Look at the script that got recorded - do you have 521 steps or 21?

JMeter records browser actions by configuring itself as a proxy - as such the first basic thing is JMeter will record whatever the browser requested it for. It follows that if the browser didnt request something to the proxy then Jmeter wont record it. The cases when user did something but browser didnt make the request can be due to
a. Browser cache (so ensure that you always clear cache and also that you configure browser to not cache anything - IE for example may cache per session and you can change the settings)
b. You configured browser to bypass some requests from proxy (For e.g. No Proxy for setting in Firefox)

The other reasons fall under Jmeter configuration
a. You manually excluded some URL patterns but there is a mistake in your regular expression causing some requests to not get recorded (URL Patterns to exclude)
b. You are facing some issues due to HTTPS not being setup correctly

Anonymous said...

Thanks Deepak,

Yes, I checked with the Firefox and through firebug I found that Firefox only records 22 pages out of 512 pages. So as you described, Jmeter will record in the same way the firefox is requested for.
One more doubt here, "How will I record script in Chrome using JMeter. Will it be same like with the other browsers (IE, Mozilla). I was trying to record in Chrome but application is not getting recorded."

Deepak Shetty said...

I was trying to record in Chrome but application is not getting recorded.
All browsers support proxy so all can be used along with JMeter - (But configuration steps might differ).
However you should probably see why FF is not making the requests - The two most common settings are cache or Network settings (Tools -->Options -->Advanced -->Network-->Settings )

One more possibility is that you are mistaking HTTP requests and browser clicks. Everytime you click a link in the browser does not necessarily mean that browser makes an HTTP request (for e.g. DHTML popups )

Anonymous said...

What is the maximum limit of load in jmeter?
I want to run Load test with 350 users with ramp up of 50 users every few minutes. While recording there were 47 steps being recorded. if there will be 350 users then (350*47) 16450 steps will be executed.

Please correct me if I am going wrong.

Deepak Shetty said...

>What is the maximum limit of load in jmeter?
None - You are limited by what ypur machine can do - but you can always use multiple machines

> if there will be 350 users then (350*47) 16450 steps will be executed.
Assuming users = threads and number of iterations (on thread group) =1 then yes 16450 steps.

Anonymous said...

Hi Deepak, I am confused in the below step:

After logging in, I get the homepage of my application and I fill a form, now as I click on submit button after filling the form, I find that resultant page has a new dynamically generated id under the URL, which I am not able to find out.

because before proceeding to this url I need to get it actually.

suppose after clicking on submit URL says : www.abc.com/25/mytask

Here I need to capture this id(25) so that when I put a load that time I could write the variable name in this url to get this id.

Also I want to mention that I need this id in my one previous request.

Please guide me on this.

Anonymous said...

Hi,
I want to use Monitor Results Listener in Jmeter. I want to monitor server as well of the application.
But how can I achieve this?

The monitor only works with Tomcat5 build 5.0.19 and newer. It means if application is deployed some other server, then it will not capture the server performance?

Please suggest.

Deepak Shetty said...

JMeter is not a server monitoring tool - The interface that is provided for Tomcat for e.g. can be implemented on any other server if you wish , however most Java app servers or even Java VMs have good monitoring capabilities (like JConsole) - if you are using something other than Java then you'll have to see specific to your platform.

Anonymous said...

Hi Deepak,
During recording through Jmeter, I got the following exception:-

javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake
at sun.security.ssl.SSLSocketImpl.readRecord(Unknown Source)
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(Unknown Source)
at sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source)
at sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source)
at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:436)
at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180)
at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:294)
at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:643)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:479)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
at org.apache.jmeter.protocol.http.sampler.HTTPHC4Impl.executeRequest(HTTPHC4Impl.java:481)
at org.apache.jmeter.protocol.http.sampler.HTTPHC4Impl.sample(HTTPHC4Impl.java:298)
at org.apache.jmeter.protocol.http.sampler.HTTPSamplerProxy.sample(HTTPSamplerProxy.java:74)
at org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.sample(HTTPSamplerBase.java:1105)
at org.apache.jmeter.protocol.http.proxy.Proxy.run(Proxy.java:236)
Caused by: java.io.EOFException: SSL peer shut down incorrectly
at sun.security.ssl.InputRecord.read(Unknown Source)
... 16 more


Please suggest why this is coming.

Deepak Shetty said...

Did you follow all the steps listed in
https://jmeter.apache.org/usermanual/jmeter_proxy_step_by_step.pdf
http://jmeter.apache.org/usermanual/component_reference.html#HTTP%28S%29_Test_Script_Recorder ( HTTPS recording and certificates )
HTTPS recording needs some further setup in Jmeter and browser

Anonymous said...

Deepak,
I have followed all the steps mentioned in PDF, but still getting the same exception.

Thanks

Deepak Shetty said...

And imported and setup your browser correctly ?
Ensure you are using latest jmeter / supported java version.
Otherwise you will have to ask on the Jmeter mailing group..

Anonymous said...

Deepak,

I am using Jmeter 2.11 version and java 1.7 version.

Deepak Shetty said...

is this happening for any website that you are attempting to record ? In which case probably you have a setup issue (if its only happening for your site then it likely might be a defect in JMeter or your site) -
Its difficult to say since the error message you get is a generic java thing
You could try alternatives like https://chrome.google.com/webstore/detail/blazemeter-the-load-testi/mbopgmdnpcbohhpnfglgohlbhfongabi?hl=en

Anonymous said...

Hi Deepak,

I am working on a login page where we get a CSRF token and i need to pass the token with the login credentials.I am able to extract the CSRF code from the page using Regular expression extractor but when i make the post call but the in the same sample CSRF code goes blank as the Regular expression extractor is post processor.How i can extract the code in the same sample and send in the post.


Thanks
AD

Deepak Shetty said...

>.I am able to extract the CSRF code from the page using Regular expression extractor
Did you verify this ? Immediately after the sampler in which you extract the data into a variable , add a debug sampler + view results tree and see if you did indeed extract the value
>but when i make the post call but the in the same sample
This statement doesnt make sense - The token is for the next sample.
>How i can extract the code in the same sample and send in the post.
Again - you do not seem to understand how CSRF or the browser does this. You have to first request a page or something which gives you a token , then you extract it and then you send it in the next sample

Anonymous said...

Thanks Deepak for the reply.I was able to capture the response.

-AD

Anonymous said...

Nice post!! It is very helpful

PerfTester_0321 said...

Hi Deepak,
I am using JMeter 3.0 version and creating the scripts for web application.

I have a Get request which have JSON response. After that I have POST request which send the request based on previous response values. Like In response I am getting the total records and each record numerical value (I have captured them using Regular expression extractor names like pcount and pvalues) . Here is the POST request format..
pselect_123=1&pselect_1234=1&.......
So here the number pselect_xxx=1 are based o number of pcount and xxx values are coming from pvalues. Now here I am looking for formatted request to substitute in place of request because each request have different sets of pselect_xxx=1 values.

Could you please provide me the beanshell script to resolve this ? I really appeciate all our help.

Deepak Shetty said...

Im not sure I understand.
If say pcount=5 , and pvalues looks like
pvalues_0=xxx0
pvalues_1=xxx1
pvalues_2=xxx2
pvalues_3=xxx3
pvalues_4=xxx4
And in post you want to send
pselect_xxx0=1&pselect_xxx1=1&....&pselect_xxx4=1

Deepak Shetty said...

sorry meant to ask , is this what you want to do ?

raj said...

How do I configure a parameter so that a distinct value is used for each occurrence of the parameter in a single request?


Here is an example of the request:
http://blahblahblah.com/api/symbol=${stock},${stock},${stock}


I want the request to be sent like this:
http://blahblahblah.com/api/symbol=GOOG,MSFT,CSCO

the values of the parameter "stock" are defined in the csv file. Its a single thread execution test.

Pavan Kumar said...

Hi Deepak,

How to handle the Cache buster value in j meter.
Brief explanation of cache buster is below:

[A cache-buster is a unique piece of code that prevents a browser from reusing an ad it has already seen and cached, or saved, to a temporary memory file.

What Does a Cache-Buster Do?
The cache-buster doesn’t stop a browser from caching the file, it just prevents it from reusing it. In most cases, this is accomplished with nothing more than a random number inserted into the ad tag on each page load. The random number makes every ad call look unique to the browser and therefore prevents it from associating the tag with a cached file, forcing a new call to the ad server.

Cache-busting maximizes publisher inventory, keeps the value and meaning of an impression constant, and helps minimize discrepancies between Publisher and Marketer delivery reports.]

I am getting a dynamic value for cache buster how to handle it in j meter.whenever we are sending a request to server cache buster value is generated.how to handle it .In response the cache buster value is not shown.Please help me.

Anonymous said...

>how top capture multiple dynamics values in same request(jmeter).<