Thursday, December 31, 2009

Graphs for JMeter (parsing JMeter result logs)

Edit : Latest experiments with JMeter and graphs http://theworkaholic.blogspot.com/2015/05/graphs-for-jmeter-using-elasticsearch.html
One of the few features lacking in JMeter is when you run the tests from the command line, the out of box reports are restricted to a stylesheet that generates a summary report.
There are workarounds, you could load the result into Excel (small files) , or you could parse the log file and use JFreeChart to generate the graphs which is what I did. See examples
The following is an explanation of the mechanisms I used. These are probably not going to be out of the box , but hopefully they will be useful to someone who can customise it. The samples are also meant to be used by developers, so if you are a Tester with little or no coding experience, get a developer from your team to help.

I haven't looked closely at the JMeter parsing details, but you don't need to the details to use the JMeter classes (which in my opinion is a hallmark of a well designed system). There are two important files , the saveservice.properties and jmeter.properties which I have copied to a different location from the JMeter home so that I could modify them if I needed to.

The basic code for parsing using JMeter classes(using the JMeter API) is
SaveService.loadTestResults(FileInputStream, ResultCollectorHelper);
where ResultCollectorHelper is passed a Visualizer. The Visualizer has one method that is important to us
add (SampleResult sampleResult)
The Visualizer interface is a simple strategy that can be implemented as we want. Since we also want to write some graphs, I created a new interface called OfflineVisualizer which adds a single method
public Object writeOutput() throws IOException
Here is the class diagram (generated using FUJABA)



Visualizer is a simple strategy pattern. I've some sample implementations for LineChartVisualizer, StackedBarChartVisualizer, MinMaxAvgGraphVisualizer respectively to draw a line chart for each response, a Stacked chart (latency plus response) or a line chart showing Min,Max, Avg along with the response time.

If we take a quick look at the LineChartVisualizer code , its pretty straightforward, it simply uses the JFreeChart API and populates the data from the SampleResult. Note that the line chart objects would use memory proportional to the number of samples
//adds a sample. JFreechart uses a TimeSeries object into which we set each data item
public void add(SampleResult sampleResult) {
String label = sampleResult.getSampleLabel();
TimeSeries s1 = map.get(label);
if (s1 == null) {
   s1 = new TimeSeries(label);
   map.put(label, s1);
}
long responseTime = sampleResult.getTime();
Date d = new Date(sampleResult.getStartTime());
s1.addOrUpdate(new Millisecond(d), responseTime);
}
//uses JFreeChartAPI to write the data into an image file
public Object writeOutput() throws IOException {
TimeSeriesCollection dataset = new TimeSeriesCollection();
for (Map.Entry<String, TimeSeries> entry : map.entrySet()) {
   dataset.addSeries(entry.getValue());
}
JFreeChart chart = createChart(dataset);
FileOutputStream fos = null;
try {
   fos = new FileOutputStream(fileName);
   ChartUtilities.writeChartAsPNG(fos, chart, WIDTH, HEIGHT);
} finally {
   if (fos != null) {
       fos.close();
   }
}
return null;
}
//use the JFreeChart API to generate a Line Chart
private static JFreeChart createChart(XYDataset dataset) {
JFreeChart chart = ChartFactory.createTimeSeriesChart("Response Chart", // title
       "Date", // x-axis label
       "Time(ms)", // y-axis label
       dataset, // data
       true, // create legend?
       true, // generate tooltips?
       false // generate URLs?
       );

chart.setBackgroundPaint(Color.white);
XYPlot plot = (XYPlot) chart.getPlot();
plot.setBackgroundPaint(Color.lightGray);
plot.setDomainGridlinePaint(Color.white);
plot.setRangeGridlinePaint(Color.white);
plot.setAxisOffset(new RectangleInsets(5.0, 5.0, 5.0, 5.0));
plot.setDomainCrosshairVisible(true);
plot.setRangeCrosshairVisible(true);
XYItemRenderer r = plot.getRenderer();
if (r instanceof XYLineAndShapeRenderer) {
   XYLineAndShapeRenderer renderer = (XYLineAndShapeRenderer) r;
   renderer.setBaseShapesVisible(true);
   renderer.setBaseShapesFilled(true);
   renderer.setDrawSeriesLineAsPath(true);
}
DateAxis axis = (DateAxis) plot.getDomainAxis();
axis.setDateFormatOverride(new SimpleDateFormat("dd-MMM-yyyy HH:mm"));
return chart;
}

We can change the data some graphs show by using the Decorator pattern. One decorator LabelFilterVisualizer is shown.

/**
* decorates the visualizer by filtering out labels
*/
public void add(SampleResult sampleResult) {
 boolean allow = labels.contains(sampleResult.getSampleLabel());
 if (!pass) {
     allow = !allow;
 }
 if (allow) {
     visualizer.add(sampleResult);
 }
}

/**
* delegates to the decorated visualizer
*
* @return whatever the decorated visualizer returns
*/
public Object writeOutput() throws IOException {
 return visualizer.writeOutput();
}

This class filters out labels and only delegates those that satisfy the criteria. The writing of the image is delegated to the decorated OfflineVisualizer
We can also use the composite pattern(CompositeVisualizer) to have multiple graphs generated with a single pass through the result log file.

/**
* adds the sample to each of the composed visualizers
*/
public void add(SampleResult sampleResult) {
  for (OfflineVisualizer visualizer : visualizers) {
      visualizer.add(sampleResult);
  }

}

/**
* @return a List of each result from the composed visualizer
*/
public Object writeOutput() throws IOException {
  List<Object> result = new ArrayList<Object>();
  for (OfflineVisualizer visualizer : visualizers) {
      result.add(visualizer.writeOutput());
  }
  return result;
}


Finally we can use all the above to process multiple files , for e.g. when we want to show trends across multiple runs with varying thread counts.

/**
* parses each file
*
* @throws Exception
*/
public void parse() throws Exception {
  // One day we might multithread this
  for (String file : files) {
      ResultCollector rc = new ResultCollector();
      TotalThroughputVisualizer ttv = new TotalThroughputVisualizer();
      visualizers.add(ttv);
      ResultCollectorHelper rch = new ResultCollectorHelper(rc, ttv);
      XStreamJTLParser p = new XStreamJTLParser(new File(file), rch);
      p.parse();
  }
}

/**
* Gets the resulting throughput from each file and combines them
*
* @return always returns null
* @throws IOException
*/
public Object writeOutput() throws IOException {
  XYSeries xyseries = new XYSeries("throughput");
  for (AbstractOfflineVisualizer visualizer : visualizers) {
      Throughput throughput = (Throughput) visualizer.writeOutput();
      xyseries.add(throughput.getThreadCount(), throughput
              .getThroughput());
  }
  XYSeriesCollection dataset = new XYSeriesCollection();
  dataset.addSeries(xyseries);
  JFreeChart chart = createChart(dataset);
  FileOutputStream fos = null;
  try {
      fos = new FileOutputStream(fileName);
      ChartUtilities.writeChartAsPNG(fos, chart, WIDTH, HEIGHT);
  } finally {
      if (fos != null) {
          fos.close();
      }
  }
  return null;
}



Here's a sample that I ran. A single thread hits 3 pages on the apache website in a loop.


Response times are plotted against each label (without considering the thread).
File f = new File(JMETER_RESULT_FILE);
ResultCollector rc = new ResultCollector();
LineChartVisualizer v = new LineChartVisualizer(OUTPUT_GRAPH_DIR + "/LineChart.png");
ResultCollectorHelper rch = new ResultCollectorHelper(rc, v);//this is the visualizer we want
XStreamJTLParser p = new XStreamJTLParser(f, rch);
p.parse();
v.writeOutput(); //write the output


The next example filters out only the Component reference request and plots the response time, the minimum time, the maximum time and the average time for this request. You could extend this to indicate the median or the 90th percentile.

The code for this graph is
File f = new File(JMETER_RESULT_FILE);
ResultCollector rc = new ResultCollector();
MinMaxAvgGraphVisualizer v = new MinMaxAvgGraphVisualizer(OUTPUT_GRAPH_DIR + "/MinMaxAvg.png");
String[] labels = {"Component reference"}; //we only want this label
LabelFilterVisualizer lv= new  LabelFilterVisualizer(Arrays.asList(labels), v);//decorate the MinMaxAvgGraphVisualizer
ResultCollectorHelper rch = new ResultCollectorHelper(rc, lv);//use the decorated visualizer
XStreamJTLParser p = new XStreamJTLParser(f, rch);
p.parse();
lv.writeOutput();//write it out

The next chart shows a stacked chart which splits the response time for the Component reference into latency and the rest of the time.

File f = new File(JMETER_RESULT_FILE);
ResultCollector rc = new ResultCollector();
StackedBarChartVisualizer v = new StackedBarChartVisualizer(OUTPUT_GRAPH_DIR + "/StackedBarChart.png");
String[] labels = {"Component reference"};//we only want this label
LabelFilterVisualizer lv= new  LabelFilterVisualizer(Arrays.asList(labels), v);//Decorate the StackedBarChartVisualizer
ResultCollectorHelper rch = new ResultCollectorHelper(rc, lv);
XStreamJTLParser p = new XStreamJTLParser(f, rch);
p.parse();
lv.writeOutput(); //write the output


We could also run all these graphs at the same time using the Composite

File f = new File(JMETER_RESULT_FILE);
ResultCollector rc = new ResultCollector();
LineChartVisualizer lcv = new LineChartVisualizer(OUTPUT_GRAPH_DIR + "/AllLineChart.png");
StackedBarChartVisualizer sbv = new StackedBarChartVisualizer(OUTPUT_GRAPH_DIR + "/AllStackedBarChart.png");
MinMaxAvgGraphVisualizer mmav = new MinMaxAvgGraphVisualizer(OUTPUT_GRAPH_DIR + "/AllMinMaxAvg.png");
String[] labels = {"Component reference"};
LabelFilterVisualizer lv= new  LabelFilterVisualizer(Arrays.asList(labels), sbv);//decorate
LabelFilterVisualizer lv2= new  LabelFilterVisualizer(Arrays.asList(labels), mmav);//decorate
OfflineVisualizer[] vs = {lcv, lv,lv2};//use these 3 visualizers
CompositeVisualizer cv = new CompositeVisualizer(Arrays.asList(vs));//create a composite
ResultCollectorHelper rch = new ResultCollectorHelper(rc, cv);
XStreamJTLParser p = new XStreamJTLParser(f, rch);
p.parse();
cv.writeOutput();//the composite will delegate to each visualizer


I also reran the same test for 1, 3,5,7 and 10 threads. Using the classes above and a new throughput visualizer (where I calculate throughput as total number of requests / total time the test ran) and plotted the throughput v/s the number of threads


String [] files = {JMETER_RESULT_DIR + "/OfflineGraphs-dev-200912311310.jtl", JMETER_RESULT_DIR + "/OfflineGraphs-dev-200912311312.jtl",JMETER_RESULT_DIR + "/OfflineGraphs-dev-200912311315.jtl",JMETER_RESULT_DIR + "/OfflineGraphs-dev-200912311316.jtl",JMETER_RESULT_DIR + "/OfflineGraphs-dev-200912311318.jtl"};
MultiFileThroughput mft = new MultiFileThroughput(Arrays.asList(files),OUTPUT_GRAPH_DIR + "/Throughput.png");
mft.parse();
mft.writeOutput();


The above examples are not exhaustive and probably wont work for you (for e.g. threads are ignored, thread groups are ignored, and these might have meaning for your test). However you should be able to use this to write your own implementation.

Running the code.
a. Download the code. This is an eclipse workspace. To get this to compile, you need to define two variables in eclipse (JMETER_HOME and JFREECHART_HOME) for the classpath. Modify config.properties to whatever is applicable for your system. Use GraphClient to see the samples
I created an additional dummy directory for Jmeter home and created a bin directory under it and copied jmeter.properties and saveservice.properties.
b. Change the client to use the visualizers you want. The sample client should give you some idea. Or create a new visualizer
c. Compile and run! If you use a different IDE or want to use ANT it should be pretty straight forward. The source code has been written and tested on Java 1.5 . There isn't any 1.5 feature I use except generics and the new for loop syntax. You could change this to be 1.4 compatible.

Further work
a. Combining results from multiple files into a single run.
b. Making the visualizers configurable
c. Canned HTML reports
d. Threads/ThreadGroups
e. Determine limits for the graphs.
f. Support custom attributes


If there are specific graph requests , I might take a look into it, day job and wife willing.

11 comments:

Quasar said...

Hi Depak!
I'd like to thank you for your post. It's been very useful to me and I was able to create my own customised charts using JFreeChart library.
Beer's on me!

Deepak Shetty said...

@Quasar - Glad to be of help!

bespin said...

Your code was extremely helpful. Thanks for sharing and keep up the great work!!

gggeek said...

A great idea, but still too much of a hassle to set up for a non-java coder like me.

The questions I have are:

1. why the Jmeter devs did not yet add an 'import log file' button in the GUI that would allow to execute headless tests on one machine and visualize the reports on another one after copying over the logs

2. would you be willing to sponsor such an effort?

Thanks
gaetano

Deepak Shetty said...

@gggeek
1. The feature you are asking for already exists.you can copy the result file generated from any JMeter run and load it into a listener (See the field mentioned Filename read from file / write to file)

2. I'm not a Swing developer so it would be tough for me to do this within JMeter. I can write command line utilities though :). In any case you should raise bugzilla enhancement requests if there are features that you would like to see in JMeter. You can also use the mailing lists to see if anyone has done anything similar

Senthil said...

Hi Deepak,

All your posts on Jmeter is very helpful and thought provoking.

I am looking for some guidance in improving the performance of the Hudson performance plugin.

I am in need of a parser that would parse the Aggregate/Summary report JTL file to have a summarized entry in a single line. I have configured Hudson performance plugin to display the graphs but the performance of the plugin is very bad that the memory of the Hudson box is drained since the plugin parses all the entries for each transaction in the JTL file for all the builds ! My idea is to have an summarized aggregate entry in the JTL file of each build which will make the plugin to parse efficiently. Any advise will be helpful

cheers !

Deepak Shetty said...

@Senthil
I haven't seen how to write plugins for Hudson - but if an individual file isnt that big , you should be able to use the XSLT to get the table. If you actually need to parse the file it still should be fairly easy.

The other thing is what is your Hudson's memory. default java has a very low heap so you might be able to fix your problem just by giving a larger -Xmx.

Pallavi said...

Hi Deepak, what is your suggestion on http://code.google.com/p/jmeter-plugins/wiki/JMeterPluginsCMD??

Deepak Shetty said...

@Pallavi Jmeter Plugins is much more user friendly (but I think depends on Java 1.6). if it has the graphs that you want then I suggest you use it.
if Jmeter Plugins doesn't have the graphs you want then use whichever you are comfortable with. I do not have any current intention of making these easier to use - you have to have a good deal of java familiarity. This is written as a how could I customise the reports things (for e.g. I could generate a PDF) - Plugins is more out of the box. However if you are willing to invest time to understand its code then I think thats the better option

Frank Cohen said...

Hi Deepak: Great article. I love all the code examples. I found your blog from the JMeter user email list. I am going to design a new reports interface for TestMaker (http://www.pushtotest.com). Your blog has lots of good ideas. I'm going to use JasperReports instead of JFreeChart. Jasper lets you export to PDF, HTML, Excel, and csv files. It also lets me offer a "print" function without coding it. I really like some of the charts in TIBCO ActiveMatrix. I'm thinking of Box Plot style charts (http://en.wikipedia.org/wiki/Box_plot). I hope to contribute the work back to the open source testing space, including JMeter. We event have a JMeter ScriptRunner in TestMaker now. We need people to test it, though! -Frank

Deepak Shetty said...

@Frank
Good luck! Unfortunately my day job doesn't allow me to test out PushToTest which i have wanted to from quite a while now - But I'd love to see the Open Source tools compete on reports with the commercial ones.