Friday, May 25, 2012

JSON in JMeter

http://jmeter.512774.n5.nabble.com/Require-help-in-Regular-Expression-td5713320.html
Roughly user gets a JSON response and wants to use some of that data for the next JSON request. Using a regular expression to extract out multiple related data is sometimes painful. So the alternative I suggest is to use JMeter to parse the JSON object and then use it (The other would be to have JMeter to natively support JSON , but unless JMeter also has a mapping tool , this only saves two lines of code , so I don't think its that useful).

So first we download Douglas Crockford's reference implementation from JSON in java , compile it and generate a jar and copy it into JMeter's lib directory (please ensure you compile it with a java version that is compatible with whatever you will use to run JMeter) - and don't use this library for evil.

Next launch JMeter.


To spoof this example , we create a BeanShell sampler that will return the JSON string we are interested in. To this sampler we add a BeanShell post processor that will parse the response and form the object that we need for the next request. The code is given below

import org.json.JSONArray;
import org.json.JSONObject;

String jsonString = prev.getResponseDataAsString();
JSONArray equipmentParts = new JSONArray(jsonString);
JSONArray parts = new JSONArray();

for(int i=0;i<equipmentParts.length();i++ ){
    JSONObject equipmentPart = equipmentParts.getJSONObject(i).getJSONObject("equipmentPart");
    JSONObject allAttributes = equipmentPart.getJSONObject("allAttributes");
    JSONObject part = new JSONObject();
    part.put("partId",allAttributes.getLong("equipmentPartId"));
    part.put("partNumber",allAttributes.getString("compositePartName"));
    // add more here
    parts.put(part);
}

vars.put("jsonResponse", parts.toString());
Note we first get the response as a string. Then we just use the JSON libraries to parse this string into a JSON array (equipmentParts). After which we iterate over every object, extract out the attributes we are interested in (you could even have an array of attributes and have a simpler loop) and then we start forming the object we want to post (i.e. the parts JSON array and the part JSON object)

Next we simply convert the JSON parts array into a String to be used by the next sampler by setting it into the jmeter variable jsonResponse.


The next sampler simply uses a POST and only specifies the variable ${jsonResponse} in the value field(So that it will be used as is in the body).
We run the script. The Debug Sampler is used to verify that the value we setup is correct and that it does get posted in the next request



Sample code is available here https://skydrive.live.com/#cid=1BD02FE33F80B8AC&id=1BD02FE33F80B8AC!876

44 comments:

Anonymous said...

Here is an alternative solution using JavaScript in BSF postprocessor: http://jmeter.apache.org/usermanual/jmeter_proxy_step_by_step.pdf

Anonymous said...

Sorry, wrong link...
An alternative solution using JavaScript in BSF postprocessor:
http://www.havecomputerwillcode.com/blog/?p=500

Deepak Shetty said...

Sure.

The reason I prefer java is
a. I can do more advanced things (usually XML related)
b. If need be , when the scripts get complicated , I can pull this into a java class which exposes a single method and call that - that lets me test my code and get compile time goodness before I run my tests in JMeter
c. I have the flexibility of moving most of my code into user defined functions

in contrast the only benefit I get from using javascript is that I can use the eval method or potentially reuse code from the application (that might make me use javascript)

Neil Anderssion said...

Scenario :

1 user will login
2 search
3 if result exits it will return that row else it will return message


when i record this scenario through proxy in the view result tree,request and response are showing correctly but when the same thing I run directly in the view result tree list for the same request sub request are showing and in the response tab wrong data are showing and for that reason assertions are not working during running the test case.

This application is Ajax+json based, so can u suggest how to overcome this situation. if more clarification required i will provide

Swarnendu Mukherjee said...

hi dipak

i am facing the problem to handle json response in jmeter , can u please help me?

Deepak Shetty said...

what is your issue?

Swarnendu Mukherjee said...

Scenario :

1 user will login
2 search
3 if result exits it will return that row else it will return message


when i record this scenario through proxy in the view result tree,request and response are showing correctly but when the same thing I run directly in the view result tree list for the same request sub request are showing and in the response tab wrong data are showing and for that reason assertions are not working during running the test case.

Swarnendu Mukherjee said...

dipak can you suggest how i can overcome this problem.

Swarnendu Mukherjee said...

when i was recording through proxy at that time in in the view result tree for the search thread in the request tab search query sting is showing [ i.e&airport_name=ABD&airport_id=6 ]

and in the response tab it is showing the no record found


but after stopping the proxy and setup thread group and controller and putting the recorded threads under the controller and setting the view result tree for the search thread when i run the test case i found in the result tree for the search that there 2 sub threads under the search thread first thread containing the search query string and in the response tab it is showing blank and another thread is for users and in the response tab of that thread it is showing the login page html

so i cant able to resolve this problem . i am not getting the reason why two threads are showing under the main search thread though it was not showing during recording through proxy

Deepak Shetty said...

did you add a cookie manager?
If yes then you need to check every request/response from jmeter and compare it with the browser (using something like fiddler ). The two requests indicate that the server redirected you - so your test didnt send something the server was expecting.

Swarnendu Mukherjee said...

dipak i hevent used cookie manager in the test plan because in this project session are not controlled through cookies, will i use this?. i have firebug add-on in my mozila browser , is it enough or i have to install fiddler?

Swarnendu Mukherjee said...

i also want to add one point here that in this project ajax+json are used through out.

Deepak Shetty said...

If you arent using cookies, then possibly you are using sessionids in URL - in which case have you dynamically extracted data?

AJAX- JSON hasn't any special requirements with respect to Jmeter as Jmeter only cares about request/response so as long as the order of requests (and the dynamic data) is the same , it should work.
Most likely you arent extracting out dynamic data correctly. Again compare the jmeter request/response with browser (right from request 1)

Swarnendu Mukherjee said...

For login section i compared the browser response using firbug inspect option and jmeter response . both are same .

i used csv config option to send login credentials dynamically in the login sampler using ${uid} and ${pwd} and as the browser response and jmeter response are same in login section thats why my assertion are working correctly.which i can clearly see through the view result tree which i added under login sampler.

but when it comes to the search module the this difference in response problem between jmeter and browser is arising.

during recording through proxy response is same as browser but when i run the testcase without proxy at that time response are different.

Deepak Shetty said...

As before you have to compare every request (including headers) - if you cant see a difference you are missing something.
Apps with dynamic data will not work with record-replay

Swarnendu Mukherjee said...
This comment has been removed by the author.
Swarnendu Mukherjee said...

Apart from the Home page Suppose there are 2 pages i.e page A ,Page B , first step user have to login, then if that user have the permission to view page Page A ,Page B then he can go to those page else on clicking those page name from the top navigational bar user will remain in the home page.
1 login
2 Home [If Login Successful]
3 Page A [if have permission]
4 Page B [if have permission]
5 logout

Suppose there are 2 users ,i.e user1 and user 2. Now User 1 have the permission to view both page a , page b but user 2 have the pewrmission to view only page a.
so at the time of logout in the logout page for user 1 in the referer field it will show the link of page b but for user 2 at the logout page referer field value will show the link of Page A.
o can u please tell me how the referer field value can be change dynamically basd on user permission in the logout page.

Swarnendu Mukherjee said...

Deepak is there any way out to fix this problem?

Deepak Shetty said...

Jmeter doesn't automatically add referer - you have to do it yourself (or it will show whatever the value was when the script was recorded). if you need referer to change dynamically you can always do that by adding the header element and specifying the value yourself.
if this is referring to your original problem , I doubt the referrer makes a difference (unless your application code actually checking the referrer - which is a silly check since it is trivial to spoor referrer)

Swarnendu Mukherjee said...

hi deepak

i have used assertion under the page a sampler , page b sampler to check weather correct threads are accessing this pages or not that is system is allowing only the authorized person or not.

in the thread group i have opted if error occur then " stop thread " , thats why if any thread fail to assert then it is not getting counted in the future , thats why in the logout sampler only 1 thread is showing, and that thread is for user 1 because this user have the permission to view both page a ,b . and user 2 and 3 are not considered as they have fail assertion in the previous stages.

so is there any way out that in the logout page i can send all those thread which have cleared the login section successfully irrespective of there view page permission.

Deepak Shetty said...

Hi
you can always setup variables and use If controllers .
For e.g you would add a regular expression extravtor under Home Page and extract out URL for Page A and only make a request for it if it exists

Home
+Regex extractor for page A into variableA (default value notfound)
+Regex extractor for page B into variableB(default value notfound)
IF controller (check variableA is not notfound)
+Request Page A
IF controller (check variableB is not notfound)
+Request Page B

OR
if you have user1 and user 2 in CSV add additional columns (e.g. hasPageAAccess which has values Y or N) and then add if controllers to your script that check if the variable has value Y then only access the page

siva said...
This comment has been removed by the author.
siva said...

Hi Deepak, i am new to jmeter. I have to send http request with key and value contains a map object. I am unable to send params as key with value as map object. Please help me as soon as possible urgently to solve it.(value itself has a key, value pair)
Thanks

siva said...

Hi Deepak, I am able to write regex extractor for select tag, but how to write regex extractor for multiple select tags based on select tag id attribute value. I have tried using xpath but i need to get using regex only not xpath extractor please tell me as soon as possible.

Deepak Shetty said...

@Siva - HTTP doesnt have the concept of a "map" and it depends on how your application is positing it - if its JSON then just populate the object and post as I have shown

For multiple seletc tags use multiple different regex

Ravi said...

hello deepak,

i have an issue with the regex expression with jmeter. i have a JSON like that:
{"key":"prod","id":"p2301d","objects":[{"id":"102955","key":"member",... ,"type":"list"}], "features":"product_features"}

i'm trying to exlude everything between [{...}] and only extract the other part: "key":"prod","id":"p2301d","features":"product_features".
Do you have a solution with the regex of jmeter? thanks you.

Deepak Shetty said...

Your requirements arent clear - but you have to note that [ and ] have special meaning in regex

http://www.regexplanet.com/advanced/java/index.html

Regex
(.*)?"([^"]*)":(\[[^\]]*\]),(.*)

Input 1
{"key":"prod","id":"p2301d","objects":[{"id":"102955","key":"member",... ,"type":"list"}], "features":"product_features"}

Then group1 and group4 give what you say you want.
In JMeter in the template field you would give
$1$$4$ to concatenate the matching group

To understand the regex , it basically says match anything till you find a " , then keep matching till you find another ":[ , then match everything thats not a ] , then match a ], then match a comma , then match everything else.

Please send jmeter queries to jmeter list - This is unrelated to this post .

Anonymous said...

Hi Deepak,

my app is giving JSON responce so want to use regular exp for automatically created ID & want to use same session ID/cookie mgr for the session

Kind of json responce for Auto genarated ID

{"jsonrpc":"2.0","method":"call","params":{"model":"purchase.order","id":${myId},"signal":"purchase_confirm","session_id":"e1eae166043a42638206646b783adaf4","context":{"lang":"en_US","tz":false,"uid":7}},"id":"r118"}

Kind of session ID responce
{"jsonrpc":"2.0","method":"call","params":{"session_id":"","context":{}},"id":"r0"}

Deepak Shetty said...

@anonymous
Im not sure what your issue is
Note that session id does not always mean cookie so you need to distinguish between the two.
Extract whatever id you want and use whenever its needed - what issue do you have? Note you have many regex testers that will let you run your regex against sample responses so you can see whether your regex is correct or not as well as in Jmeter use a debug sampler to see what you extract and what you send

monalisa said...

hi deepak

last time give u wrong data,i need to create a regular expression extractor for auto generated id. For following example "id":57 is when auto generated id. which i need to extract when next id created.

{"jsonrpc":"2.0","method":"call","params":{"model":"purchase.order","id":57,"signal":"purchase_confirm","session_id":"36be3d29e6c84c9698a5006b0b837ccb","context":{"lang":"en_US","tz":false,"uid":7}},"id":"r110"}

Deepak Shetty said...

@monalisa
whats the issue? you dont know what regex to use?
"id":([^,]+),
i.e searchj for "id": , then start capturing (the round bracket) , then match everything that is not a ',' any number of times , then stop capturing, then ','
And use $1$ in the template.

Thats assuming you dont have spaces etc otherwise
"id"\s*:\s*([^,]+?)\s*,

again plenty of regex testers to test your regex against and use debug sampler to verify that your expression / response is working correctly

monalisa said...

thanks deepak

ya i was trying different reg exp but not able to extract the auto generated id. let me try with yours need to solve it urgently

Dmitri T said...

JMeter is now able to execute JSONPath requests and parse JSON with a plugin - see Using XPath and JSONPath guide

Anonymous said...

How do you compile the JSON java code to a JAR. It would've helped if you had posted the JAR to use instead of us having to debug the java code.

Deepak Shetty said...

@anonymous
Since it's someone else's code and I dont have time to go through their licenses to see whether I can redistribute jars , I havent provided a compiled version .(besides it wouldnt help you if I compiled it with a higher java version than you have). if you intend to write BeanShell code in Jmeter its well worth spending sometime understanding how java works and how to compile code in Java.

Shadows said...

Deepak, I have a response from a page and i need to add another property to each of the objects with a value. Using Bean Shell Post Processor I am not able to do the same. I guess the jar file am using does not support. The file is jmeter-plugins-1.0.0.jar. Will work?

Deepak Shetty said...

@Shadows
not enough details to comment - Im not sure how your question relates to JMeter plugins (dont know what sampler or what object you are referring to) - As far as I know you should be able to perform variable substitution in most cases.
In any case questions related to JMeter plugins are best asked on Jmeter plugins group

Anonymous said...

Hi,

i am facing a problem
problem:- i have a webservice from which i am extracting all the information using jmeter.
Now at the output of View Results Tree i am getting the response data in the the xml format.
xml format contains &lt and &gt,i want to replace these (&lt and &gt) with the < and > sign.
how can i do this using jmeter.

Deepak Shetty said...

Anonymous
BeanShell post processor(or equivalent) - use prev.getSamplerData() to get the String response.
Use java replaceAll to replace any String with any other String

you seem to have XML (or HTML ) within your webservice that you are trying to send into the next request - in which case the technically correct answer (though expensive in processing terms) is use any java parser to parse the XML - in which case the parser will convert it based on all XML rules

http://jmeter.apache.org/usermanual/component_reference.html#BeanShell_PostProcessor

http://jmeter.apache.org/api/org/apache/jmeter/samplers/SampleResult.html

man9ar00 said...

On previous posts about sharing precompiled JAR for the JSON library, it would at least be useful for the Java/Beanshell novices to provide steps on how to pull the Java code off the GitHub page then compile and generate a JAR. That way, saves them time figuring that out if they aren't already familiar.

Deepak Shetty said...
This comment has been removed by the author.
Shachi Prasad said...

I have json org jar file in my Jmeter lib and ext folder (both places) but the beanshell script is not able to find this jar.

I am getting this error:

2014/12/21 23:47:06 ERROR - jmeter.util.BeanShellInterpreter: Error invoking bsh method: eval Sourced file: inline evaluation of: ``import org.json.JSONArray; import org.json.JSONObject; String jsonString = prev . . . '' : Typed variable declaration : Object constructor

This is my script:
import org.json.JSONArray;
import org.json.JSONObject;

String jsonString = prev.getResponseDataAsString();
System.out.println("String ->" + jsonString);

JSONArray response = new JSONArray(jsonString);
JSONArray registeredApp = new JSONArray();


for(int i = 0; i < response.length(); i++) {
System.out.println("Value of i: " + i);
System.out.println("Inside for loop");
JSONObject application = response.getJSONObject(i).getJSONObject("application");
JSONObject name = application.getJSONObject(i).getJSONObject("name");
JSONObject riskAttributes = name.getJSONObject("riskattributes");
JSONObject risk = new JSONObject();

risk.put("riskposture",riskAttributes.getString("riskposture"));
risk.put("Security Controls",riskAttributes.getString("Security Controls"));
risk.put("Threats",riskAttributes.getString("Threats"));
risk.put("Policy Alerts",riskAttributes.getString("Policy Alerts"));
risk.put("Incidents",riskAttributes.getString("Incidents"));

System.out.println("Risk posture :" + risk.get("riskposture");
registeredApp.put(risk);


}

vars.put("jsonResponse", registeredApp.toString());

Deepak Shetty said...

HI
I dont have access to a working instance to check out your code so I can only say some generic stuff

a. First remove all the JSON related code and only have
String jsonString = prev.getResponseDataAsString();
System.out.println("String ->" + jsonString);

See that this works (if you use System.out.println then you need to run jmeter in console mode to see the responses)
If this doesnt work then it means you have some issue with where you placed your BeanShell post processor

b. If this does work then just add a few lines of code first
Just the imports
and JSONArray response = new JSONArray(jsonString);

If this doesnt work , then there is a problem with the jar you are using.Verify that if you open the jar file in something like winzip or unjar it - then you can see a folder org , then a folder json and then JSONArray
Also verify that the version of java under which you compiled your jar is the same as the version of java you are running JMeter on (or JMeter must be on a newer version)

c. If this works and the rest doesnt , then there is probably a syntactic error somewhere.
Its easiest to paste your code in some Java editor like eclipse , get it work , then copy it back into beanshell

Also note that this blog post is dated -There are some plugins that allow you to do some JSON without need to write that much code

http://jmeter-plugins.org/wiki/JSONPathExtractor/

Deepak Shetty said...

@Shachi
Also to confirm - you have written this code in a beanshell post processor that is created as a child of the sampler that is return the JSON response right?