Friday, March 19, 2010

Dynamic Parameters in JMeter

Motivation
You need to send a variable number of parameters with an HTTP request i.e. the number of parameters are not known when the script is written. For example a User can register with a number of accounts , but this number varies per user. Or perhaps you want to Post process extract a number of hidden fields on the previous response and post all of them in the next Request, but you either don't know the number or there are too many fields to enter manually into the HTTPSampler.

Solution
Use a BeanShell PreProcessor to dynamically add variables to the sampler. The relevant code is sampler.addArgument(name,value)[1]

Sample
We will write a script to access a page , extract out data based on a pattern and send these as separate parameters to the next request. The script is as shown below.
We make a request to some dummy page and then we have a regex extractor to extract out multiple values. In this case the Regex extracts out all words(Match No = -1) beginning with test




Then we have a Debug Sampler to verify the regex is working, notice the results in View Results Tree Listener. The Regex has a reference variable name of inputTerms. The response of the Debug sampler is shown below

You can see from the Listeners response tab that the total number of results is 14 , available under the key inputTerms_matchNr and that each result is available as inputTerms_$resultnumber.
The matching groups which we usually do not need are available as inputTerms_$resultnumber_g$groupnumber. Armed with this information, the beanshell preprocessor script is trivial as shown below
int count = Integer.parseInt(vars.get("inputTerms_matchNr"));
for(int i=1;i<=count;i++) { //regex counts are 1 based
sampler.addArgument("hardcodedkey", vars.get("inputTerms_" + i));
}
Heres the next request to which the beanshell pre processor is attached. We have one predefined variable and the rest will be dynamically added by beanshell.

Next we check the request data we are posting using the Request tab of the View Results Tree Listener



Notice that the argument we added to the next request was sent, as were all the parameters we added dynamically. The parameter values were also encoded.
Sample Script : https://skydrive.live.com/#cid=1BD02FE33F80B8AC&id=1BD02FE33F80B8AC!268

46 comments:

Anonymous said...

Great article. Thanks!
Alex

Anonymous said...

great!!!!

RRP said...

Nice article... will help a lots of guys in need..
It helped me...

Saurabh Thakur said...

Nice short article ..would love to see more on Usage of Beanshell in Jmeter.

Anonymous said...

you cant beieve how lucky i am to have found this article. really really good. thank you so much!

Anonymous said...

i am new to jmeter. Will u plz post the step by step procedure to run this script.
Thanks in advance.

Deepak Shetty said...

@anonymous
Download script from https://skydrive.live.com/#cid=1BD02FE33F80B8AC&id=1BD02FE33F80B8AC!268

The script name is dynamic parameters. launch Jmeter and open this script file and run it.

Note that you rarely need to add parameters dynamically (typically you know what parameters and how many there are in the script so you can directly specify them)

Karthik said...

Thanks you very much mate. I've been scratching my head like hell to figure out how to add variable number of parameters dynamically. This helped me a lot.
Karthik

Anonymous said...

Awesome Work buddy. That's what I was looking for.

Thanks Again.

siva said...

hi, i have multiple select tags and i wanna get option value based on select tag id attribute value using regular expression.

Deepak Shetty said...

@Siva
XPath Post processor is usually easier in such cases (but less memory efficient and slower)
It might still be possible to do it with regex but the regex itself is much more difficult. The other option is using the id extract out the Select HTML. Then write a second regex that operates on this extracted Select box to extract out the option

Nabojyoti Sarkar said...

Hi Deepak

Excellent post mate!!!

Just wanted to know that what needs to be done if I do not want to hardcode the value of the paarameter..say I want the parameters to contain values from a predefined CSV file..Could you please help me onm this?
Thanks
Nabojyoti(sarkar.nabojyoti@gmail.com)

Deepak Shetty said...

@Nabojyoti Sarkar
The API allows you to specify whatever key/value you want and JMeter/java allow you to write whatever logic you want.

For e.g. if your CSV is read using a CSV data set config and you have 4 columns(column1 = name of parameter,column2=value of the same parameter) which map to variables col1 ,col2,col3,col4

then in the Beanshell pre processor you could write

sampler.addArgument(vars.get("col1"),vars.get("col2"));

Again the main point is sampler.addArgument(key,value)
How you get key and value isnt really relevant and can be done in many different ways

Nabojyoti Sarkar said...

Thanks a lot Deepak for your response..thanks for helping me understanding the working of this API, however, I am unable to do something using this. I aim at generating random number of paramaters (successfully done that with your help) and then assign values to the parameters from a csv file. The csv file has two columns and few rows. I want the first parameter to get value from cell(row1,col1), second parameter to get value from cell(row2,col1) and like wise...can you pls provide me the code how to do this? Currently it is only populating the parameters with the values present in cell (row1,col1)

Deepak Shetty said...

@Nabojyoti
Usually your excel file should be modelled as
row1 = data for 1 case
row2 = data for 1 case
i.e. each row should have the data .
In your example you want the column to have the data - its easier if you transpose your excel.
If you can then simply use the CSV Data Set Config
Thread Group
+While Controller (with condition)
++HTTP request Sampler
+++BeanShell Preproc
++CSV Data Set config(var1,var2...)

Then for each time the while controller runs the values you declare in the variables (in the example var1, var2) will move to the next row.
Because the reading is row by row what you want is the data for one test case to be contained in the row so parameter 1 = row1,col1
parameter 2 = row1, col2

You need to specify the sharing mode on CSV data set config and you also need to provide a condition when you want to stop (for e..g. if you want to stop when there are no more lines in the CSV then you use "${var1}" != ""



Rachna said...

Awesome soved my problems in a seconds :) :) Thanks a ton

krishna said...

@deepak


HI bro, can u explain me how to read an excel file using beanshell

Deepak Shetty said...

@krishna
Not related to this post - please send jmeter questions to the jmeter mailing list.
easiest way is to export excel to CSV and use CSV in JMeter. other way is to use POI or JExcelAPI to read the file and iterate using beanshell or custom elements - however that is too much to describe in a comment

Sattish said...

Hi Deepak,

Thanks for this nice article. It helped me a lot

Sattish.

Anonymous said...

Great article.
It helps me a lot!

Anonymous said...

God bless you. Very Very Helpful Articles.

Anonymous said...

Thank you, thank you, thank you, thank you.

Anonymous said...

HI
It thorows an error while executing
ERROR - jmeter.util.BeanShellInterpreter: Error invoking bsh method: eval Sourced file: inline evaluation of: ``int count = Integer.parseInt(vars.get("inputTerms_matchNr")); for(int i=1;i<=cou . . . '' : Typed variable declaration : Method Invocation Integer.parseInt

Deepak Shetty said...

@Anonymous
Is this your full script? most likely your variable is not called inputTerms (reference name in your regex ) or your regex is messed up or something.
Did you look at the result of debug sampler to check that the values are getting extracted as you expect?

Anonymous said...

Hi Deepak
Thanks Deepak for the response.In bean shell post processor,the script what I have mentioned is int count = Integer.parseInt(vars.get("Input Terms_matchNr"));
and in the HTTP request i'm passing the parameter "${count}" in the value section.

Note:Now the error is solved.But no value is captured in ${count} when i checked the view result tree.I'm expecting the value "8" for "count".For the response for the earlier request the value shown for "inputTerms_matchNr" is "8" and it is shown in debug sampler.

Deepak Shetty said...

as mentioned you need to check your script (variable name of Input Terms with a space in it looks wrong) - steps to debug were also specified in previous response. Basically you are not returning a parseable number so Integer.parseInt is failing

subrat said...

Hi,

I am facing one issue in Jmeter.
There are some parameters which are generated dynamically in each HTTP Request.
When data is POST,one random value is generated while getting response and append in the path.(xsbipaqc.corp.xerox.com/ibi_apps/WFServlet?IBIAPP_app=XSBID&IBIF_ex=xs_card_services&random_number=1415172585774)
This value vary for each request. I want to know how to deal with. It should be different for every thread.

Thanks,
Subrat Ray

Deepak Shetty said...

@subrat
There are two possibilities
a. This is a true random number generated which has no meaning to the server other than it is random (usually HTML apps do something like this to ensure that the browser does not cache the page since every URL is effectively new)
In this case just use the standard Jmeter function (Random) append with something like threadNum function- function help is http://jmeter.apache.org/usermanual/functions.html
You can write custom java code if you need the random number to be some specific format and call that too

b. This number is something that the server generates and it is essential that the next request passes exactly the same value (for e.g. CSRF tokens or primary keys) -
In this case the server will be returning you this value somewhere and you have to add a post processor (usually regex post processor) to extract out this value from the response into a variable and then send it to the next request using ${variable}
Post processors are described here
http://jmeter.apache.org/usermanual/component_reference.html
These are automatically different for each thread since the server will usually generate different values for each request and you will extract and store it into a variable which is scoped for the thread

Which one of the two options you have to choose is based on your application.

subrat said...

Thanks Deepak,

I tried with the Regex Post Processor to deal with it. I am getting the following exception:-
Illegal character in query (xsbipaqc.corp.xerox.com/ibi_apps/WFServlet?IBIAPP_app=XSBID&IBIF_ex=xs_card_services&${test}).

Reference Name: test
Regular Expression: random_number="(.+?)"
Template: $1$
Match No.: -1


Where I am going wrong. I think there is problem with regular expression, I tried some other expressions as well but didn't get any solution.

Deepak Shetty said...

Your regular expression didnt work (and it doesnt make sense)
So test wasnt setup as a varaible (which means there is either a syntax error or you placed it in the wrong location - the regex post processor needs to be achild of the sampler that is returning the response you want to extract data from)

You need to understand how your application works before you can write a test for this - For e.g.
Did you identify , which response is returning these values ? Then you need to look at the surrounding text and write a regex that extracts the value you want - instead you have written a regular expression that says Match any character, atleast once, but as few as possible - which makes no sense.

It should be something like this
ThreadGroup
+HTTP Request 1 (assuming the text contains the random_number in its response somewhere)
++Regex Pst processor (variable=test)
+Http Request2 (${test}

The + represents indentation level
+Http

subrat said...

Thanks Deepak,

In our application, there is one parameter (request_ID in our application) which is generated from the server side how can we handle this parameter at our end. This parameter is getting changed in each request.

One more issue is there, For say, I have recorded one scenario and play it with single thread only it executes well but after 5-10 mins report data doesn't display because at server side report which is cached is cleared after some specific time. What will be the solution for this.
Can we handle this thing in Jmeter?

Deepak Shetty said...


In our application, there is one parameter (request_ID in our application) which is generated from the server side how can we handle this parameter at our end. This parameter is getting changed in each request.

if you say the server is generating this parameter and in addition it is different for every request , then every response from the server must have this parameter - and all you need to do is extract it out (into a variable ) and send it with the next request.
You can scope a Post processor so that it executes for every request in scope
For e,g
Simple Controller
+Request 1
++Regex extractor1
+Request2
+regex extractor2
In this case regex extractor1 will only get applied to the response of request 1 (since it is a child of request 1) but regex extractor2 gets applied to both request1 and Request2 (since it is a child of simple controller)

For say, I have recorded one scenario and play it with single thread only it executes well but after 5-10 mins report data doesn't display because at server side report which is cached is cleared after some specific time. What will be the solution for this.
Other than something is wrong with your test (assuming it works fine for a browser) - there is nothing I can say - Most likely you have something like a session id that has expired (when you recorded , it recorded the id and then when you play it , it is using the same id, rather than requesting a new one) -- it is not possible to remotely debug problem with the level of information that you provide.
To repeat - you have to understand how your application works , first, then replicate the same in Jmeter.
One way is compare what the browser (or your application) does with whatever you are sending in Jmeter.

Anonymous said...

Hi Deepak,

I need to create a dynamic POST request in JMeter. The request contains an array list. The list data is available in a .csv file. If you can give some suggestion using beanshell preprocessor how I can achieve this will be helpful.

Deepak Shetty said...

@Anonymous
" The request contains an array list. "
There isn't any such thing as an "array list" in HTTP - all you have is String - Determine what you actually need passed - eg. you might be passing a single parameter with comma delimited string or you might be passing multiple parameters with the same name - once you know that this is easy to achieve in JMeter.

Raheel Azhar said...
This comment has been removed by the author.
Raheel Azhar said...
This comment has been removed by the author.
Raheel Azhar said...

Deepak Shetty ,
Your blog is very helpful. Keep up the good work! I am facing a dilemma - My WS call returns 1000 dynamic IDs and I need to pass 100 of them to the next method. I can successfully extract a single ID and use it in the next method, but it uses the same ID for all the subsequent calls. Whereas my requirement is as follows - Method 1(Get) returns 1000 Ids. Second Method (Get) should use 100 of the unique Ids in first iteration and 100 in the next and so on and so forth. Any idea as to how could that be achieved? Basically, two questions: 1 - How to capture all of the dynamic Ids with the same left boundary and right boundary. 2- How to pass 100 unique Ids to the next method/call? Thanks!

Deepak Shetty said...

Im not sure exactly what you mean by "next iteration" but here is one way

Sampler Method1(returns 1000 ids)
+Extract out all ids into a variable( specifying -1 in the regex post processors)
While Controller (using __counter function to compare )
+Sampler Method 2
++ BeanShell pre processor (pick out 100 ids and set them into Sampler - use the counter variable to determine which iteration you are on and the code in this blog post as an example)

Raheel Azhar said...

Thanks! I wish I knew JMeter that well :-) I can extract 1 Id successfully but the problem is they are all returned in the same pattern and with the same boundaries. Here is an example of what the first method returns::

{"OrgId":270101},{"OrgId":273513},{"OrgId":276463}....... .
And this is the Regex Extractor setting I am using:
Reference Name: OrgID
Regural Expression: OrgId":(.+)}
Template: $1$
Match No. -1
default Value: empty

How do I extract 1000 of them?

Deepak Shetty said...

Hi
Your code snippet is correct - You are extracting all the IDs - However what you are probably having a problem with is how to access the 1000 elements?
In which case look at the example in the post
Adding a debug sampler shows how Jmeter extracts the variables (so varName_1 and varName_2 ...
The beanshell script shows you how to loop over the entire set.

So verify your regex is correct by debug sampler - are you only seeing one value in the response of the debug sampler - in which case the regex is wrong (most likely because its greedy and your data seems to be in a single line)

OrgId":{([^}])*}

I havent tested it - please use any regex tester

Raheel Azhar said...
This comment has been removed by the author.
Raheel Azhar said...
This comment has been removed by the author.
Raheel Azhar said...
This comment has been removed by the author.
Raheel Azhar said...

Thanks. This Regex successfully captures them all:
OrgId":([^"]+?)},

But here is the issue:

Method 1- Response Data from my first method is as follows:

{"Metric":[{"OrgId":116500},{"OrgId":116600},........{"OrgId":134407},


And my Debug PostProscessor is capturing the following:

OrgID=empty
OrgID_1=116500
OrgID_10=116600.........
OrgID_1000=134407

Method - 2

When I use the Beanshell script from your example, it sends only the last id i.e. 134407. I am back to square one: How do I send multiple ids in one call such as "orgIds=.,.,.,.,.,....". Of course I could hardcode them if they were static but each method shoud send "orgIds=1,.,.,....100".

Deepak Shetty said...

Hi
If your regex only picks up one value
it is wrong.
I have uploaded a sample that does some of what you want to do - I hardcoded the response to return 4 items and the next sampler picks up 2 items at a time - You can change the code to look at the total number of results and adjust it so its all dynamic
However I am out of the country after this so please send further queries to Jmeter mailing list

https://onedrive.live.com/redir?resid=1BD02FE33F80B8AC!945&authkey=!AFzctxMn9ylCsHc&ithint=file%2cjmx

Raheel Azhar said...

Thanks Deepak! I'll try that today.