Thursday, December 31, 2009

Graphs for JMeter (parsing JMeter result logs)

Edit : Latest experiments with JMeter and graphs http://theworkaholic.blogspot.com/2015/05/graphs-for-jmeter-using-elasticsearch.html
One of the few features lacking in JMeter is when you run the tests from the command line, the out of box reports are restricted to a stylesheet that generates a summary report.
There are workarounds, you could load the result into Excel (small files) , or you could parse the log file and use JFreeChart to generate the graphs which is what I did. See examples
The following is an explanation of the mechanisms I used. These are probably not going to be out of the box , but hopefully they will be useful to someone who can customise it. The samples are also meant to be used by developers, so if you are a Tester with little or no coding experience, get a developer from your team to help.

I haven't looked closely at the JMeter parsing details, but you don't need to the details to use the JMeter classes (which in my opinion is a hallmark of a well designed system). There are two important files , the saveservice.properties and jmeter.properties which I have copied to a different location from the JMeter home so that I could modify them if I needed to.

The basic code for parsing using JMeter classes(using the JMeter API) is
SaveService.loadTestResults(FileInputStream, ResultCollectorHelper);
where ResultCollectorHelper is passed a Visualizer. The Visualizer has one method that is important to us
add (SampleResult sampleResult)
The Visualizer interface is a simple strategy that can be implemented as we want. Since we also want to write some graphs, I created a new interface called OfflineVisualizer which adds a single method
public Object writeOutput() throws IOException
Here is the class diagram (generated using FUJABA)



Visualizer is a simple strategy pattern. I've some sample implementations for LineChartVisualizer, StackedBarChartVisualizer, MinMaxAvgGraphVisualizer respectively to draw a line chart for each response, a Stacked chart (latency plus response) or a line chart showing Min,Max, Avg along with the response time.

If we take a quick look at the LineChartVisualizer code , its pretty straightforward, it simply uses the JFreeChart API and populates the data from the SampleResult. Note that the line chart objects would use memory proportional to the number of samples
//adds a sample. JFreechart uses a TimeSeries object into which we set each data item
public void add(SampleResult sampleResult) {
String label = sampleResult.getSampleLabel();
TimeSeries s1 = map.get(label);
if (s1 == null) {
   s1 = new TimeSeries(label);
   map.put(label, s1);
}
long responseTime = sampleResult.getTime();
Date d = new Date(sampleResult.getStartTime());
s1.addOrUpdate(new Millisecond(d), responseTime);
}
//uses JFreeChartAPI to write the data into an image file
public Object writeOutput() throws IOException {
TimeSeriesCollection dataset = new TimeSeriesCollection();
for (Map.Entry<String, TimeSeries> entry : map.entrySet()) {
   dataset.addSeries(entry.getValue());
}
JFreeChart chart = createChart(dataset);
FileOutputStream fos = null;
try {
   fos = new FileOutputStream(fileName);
   ChartUtilities.writeChartAsPNG(fos, chart, WIDTH, HEIGHT);
} finally {
   if (fos != null) {
       fos.close();
   }
}
return null;
}
//use the JFreeChart API to generate a Line Chart
private static JFreeChart createChart(XYDataset dataset) {
JFreeChart chart = ChartFactory.createTimeSeriesChart("Response Chart", // title
       "Date", // x-axis label
       "Time(ms)", // y-axis label
       dataset, // data
       true, // create legend?
       true, // generate tooltips?
       false // generate URLs?
       );

chart.setBackgroundPaint(Color.white);
XYPlot plot = (XYPlot) chart.getPlot();
plot.setBackgroundPaint(Color.lightGray);
plot.setDomainGridlinePaint(Color.white);
plot.setRangeGridlinePaint(Color.white);
plot.setAxisOffset(new RectangleInsets(5.0, 5.0, 5.0, 5.0));
plot.setDomainCrosshairVisible(true);
plot.setRangeCrosshairVisible(true);
XYItemRenderer r = plot.getRenderer();
if (r instanceof XYLineAndShapeRenderer) {
   XYLineAndShapeRenderer renderer = (XYLineAndShapeRenderer) r;
   renderer.setBaseShapesVisible(true);
   renderer.setBaseShapesFilled(true);
   renderer.setDrawSeriesLineAsPath(true);
}
DateAxis axis = (DateAxis) plot.getDomainAxis();
axis.setDateFormatOverride(new SimpleDateFormat("dd-MMM-yyyy HH:mm"));
return chart;
}

We can change the data some graphs show by using the Decorator pattern. One decorator LabelFilterVisualizer is shown.

/**
* decorates the visualizer by filtering out labels
*/
public void add(SampleResult sampleResult) {
 boolean allow = labels.contains(sampleResult.getSampleLabel());
 if (!pass) {
     allow = !allow;
 }
 if (allow) {
     visualizer.add(sampleResult);
 }
}

/**
* delegates to the decorated visualizer
*
* @return whatever the decorated visualizer returns
*/
public Object writeOutput() throws IOException {
 return visualizer.writeOutput();
}

This class filters out labels and only delegates those that satisfy the criteria. The writing of the image is delegated to the decorated OfflineVisualizer
We can also use the composite pattern(CompositeVisualizer) to have multiple graphs generated with a single pass through the result log file.

/**
* adds the sample to each of the composed visualizers
*/
public void add(SampleResult sampleResult) {
  for (OfflineVisualizer visualizer : visualizers) {
      visualizer.add(sampleResult);
  }

}

/**
* @return a List of each result from the composed visualizer
*/
public Object writeOutput() throws IOException {
  List<Object> result = new ArrayList<Object>();
  for (OfflineVisualizer visualizer : visualizers) {
      result.add(visualizer.writeOutput());
  }
  return result;
}


Finally we can use all the above to process multiple files , for e.g. when we want to show trends across multiple runs with varying thread counts.

/**
* parses each file
*
* @throws Exception
*/
public void parse() throws Exception {
  // One day we might multithread this
  for (String file : files) {
      ResultCollector rc = new ResultCollector();
      TotalThroughputVisualizer ttv = new TotalThroughputVisualizer();
      visualizers.add(ttv);
      ResultCollectorHelper rch = new ResultCollectorHelper(rc, ttv);
      XStreamJTLParser p = new XStreamJTLParser(new File(file), rch);
      p.parse();
  }
}

/**
* Gets the resulting throughput from each file and combines them
*
* @return always returns null
* @throws IOException
*/
public Object writeOutput() throws IOException {
  XYSeries xyseries = new XYSeries("throughput");
  for (AbstractOfflineVisualizer visualizer : visualizers) {
      Throughput throughput = (Throughput) visualizer.writeOutput();
      xyseries.add(throughput.getThreadCount(), throughput
              .getThroughput());
  }
  XYSeriesCollection dataset = new XYSeriesCollection();
  dataset.addSeries(xyseries);
  JFreeChart chart = createChart(dataset);
  FileOutputStream fos = null;
  try {
      fos = new FileOutputStream(fileName);
      ChartUtilities.writeChartAsPNG(fos, chart, WIDTH, HEIGHT);
  } finally {
      if (fos != null) {
          fos.close();
      }
  }
  return null;
}



Here's a sample that I ran. A single thread hits 3 pages on the apache website in a loop.


Response times are plotted against each label (without considering the thread).
File f = new File(JMETER_RESULT_FILE);
ResultCollector rc = new ResultCollector();
LineChartVisualizer v = new LineChartVisualizer(OUTPUT_GRAPH_DIR + "/LineChart.png");
ResultCollectorHelper rch = new ResultCollectorHelper(rc, v);//this is the visualizer we want
XStreamJTLParser p = new XStreamJTLParser(f, rch);
p.parse();
v.writeOutput(); //write the output


The next example filters out only the Component reference request and plots the response time, the minimum time, the maximum time and the average time for this request. You could extend this to indicate the median or the 90th percentile.

The code for this graph is
File f = new File(JMETER_RESULT_FILE);
ResultCollector rc = new ResultCollector();
MinMaxAvgGraphVisualizer v = new MinMaxAvgGraphVisualizer(OUTPUT_GRAPH_DIR + "/MinMaxAvg.png");
String[] labels = {"Component reference"}; //we only want this label
LabelFilterVisualizer lv= new  LabelFilterVisualizer(Arrays.asList(labels), v);//decorate the MinMaxAvgGraphVisualizer
ResultCollectorHelper rch = new ResultCollectorHelper(rc, lv);//use the decorated visualizer
XStreamJTLParser p = new XStreamJTLParser(f, rch);
p.parse();
lv.writeOutput();//write it out

The next chart shows a stacked chart which splits the response time for the Component reference into latency and the rest of the time.

File f = new File(JMETER_RESULT_FILE);
ResultCollector rc = new ResultCollector();
StackedBarChartVisualizer v = new StackedBarChartVisualizer(OUTPUT_GRAPH_DIR + "/StackedBarChart.png");
String[] labels = {"Component reference"};//we only want this label
LabelFilterVisualizer lv= new  LabelFilterVisualizer(Arrays.asList(labels), v);//Decorate the StackedBarChartVisualizer
ResultCollectorHelper rch = new ResultCollectorHelper(rc, lv);
XStreamJTLParser p = new XStreamJTLParser(f, rch);
p.parse();
lv.writeOutput(); //write the output


We could also run all these graphs at the same time using the Composite

File f = new File(JMETER_RESULT_FILE);
ResultCollector rc = new ResultCollector();
LineChartVisualizer lcv = new LineChartVisualizer(OUTPUT_GRAPH_DIR + "/AllLineChart.png");
StackedBarChartVisualizer sbv = new StackedBarChartVisualizer(OUTPUT_GRAPH_DIR + "/AllStackedBarChart.png");
MinMaxAvgGraphVisualizer mmav = new MinMaxAvgGraphVisualizer(OUTPUT_GRAPH_DIR + "/AllMinMaxAvg.png");
String[] labels = {"Component reference"};
LabelFilterVisualizer lv= new  LabelFilterVisualizer(Arrays.asList(labels), sbv);//decorate
LabelFilterVisualizer lv2= new  LabelFilterVisualizer(Arrays.asList(labels), mmav);//decorate
OfflineVisualizer[] vs = {lcv, lv,lv2};//use these 3 visualizers
CompositeVisualizer cv = new CompositeVisualizer(Arrays.asList(vs));//create a composite
ResultCollectorHelper rch = new ResultCollectorHelper(rc, cv);
XStreamJTLParser p = new XStreamJTLParser(f, rch);
p.parse();
cv.writeOutput();//the composite will delegate to each visualizer


I also reran the same test for 1, 3,5,7 and 10 threads. Using the classes above and a new throughput visualizer (where I calculate throughput as total number of requests / total time the test ran) and plotted the throughput v/s the number of threads


String [] files = {JMETER_RESULT_DIR + "/OfflineGraphs-dev-200912311310.jtl", JMETER_RESULT_DIR + "/OfflineGraphs-dev-200912311312.jtl",JMETER_RESULT_DIR + "/OfflineGraphs-dev-200912311315.jtl",JMETER_RESULT_DIR + "/OfflineGraphs-dev-200912311316.jtl",JMETER_RESULT_DIR + "/OfflineGraphs-dev-200912311318.jtl"};
MultiFileThroughput mft = new MultiFileThroughput(Arrays.asList(files),OUTPUT_GRAPH_DIR + "/Throughput.png");
mft.parse();
mft.writeOutput();


The above examples are not exhaustive and probably wont work for you (for e.g. threads are ignored, thread groups are ignored, and these might have meaning for your test). However you should be able to use this to write your own implementation.

Running the code.
a. Download the code. This is an eclipse workspace. To get this to compile, you need to define two variables in eclipse (JMETER_HOME and JFREECHART_HOME) for the classpath. Modify config.properties to whatever is applicable for your system. Use GraphClient to see the samples
I created an additional dummy directory for Jmeter home and created a bin directory under it and copied jmeter.properties and saveservice.properties.
b. Change the client to use the visualizers you want. The sample client should give you some idea. Or create a new visualizer
c. Compile and run! If you use a different IDE or want to use ANT it should be pretty straight forward. The source code has been written and tested on Java 1.5 . There isn't any 1.5 feature I use except generics and the new for loop syntax. You could change this to be 1.4 compatible.

Further work
a. Combining results from multiple files into a single run.
b. Making the visualizers configurable
c. Canned HTML reports
d. Threads/ThreadGroups
e. Determine limits for the graphs.
f. Support custom attributes


If there are specific graph requests , I might take a look into it, day job and wife willing.

Wednesday, December 16, 2009

JMeter and SLA's

One of the current issues on our site is that while we profiled and performance and load tested the important pages before we went live, we haven't done it for subsequent builds and releases. There are various excuses for this (lack of time, lack of representative environments, restrictions on the actions that may be performed because the site is live), none of them really justified. However no matter how much you test the site before hand, the site may still malfunction, perhaps transiently on production. For e.g. we sometimes got timeout errors between 6:00 to 10:00 a.m. (it was eventually determined to be a Database index compacting job that was creating trouble). The problem is that we had to be reactive, look at the logs, there was a timeout, run around like headless chickens because the site was working fine now, no access to the environment to see whats happening etc. Now we could have configured logs to automatically notify us when there are errors but this would only work if there was a timeout (in our 60 seconds for any remote operation). If a page that normally takes 2 seconds to load took anything under 60 seconds we would not see errors.
In previous projects , we had OVIS, which I believe is expensive, but my current project has no such commerical tool. Open source tools all seem to solve parts of the problem , but there didn't seem to be any tool that did everything I wanted.
Briefly
a. Flexible schemes to measure response times. We needed to be able to simulate accessing stand alone urls, login flows, checkout flows, search flows.
b. Ability to store the data and view trend graphs
c. Ability to run the tests on a schedule
d. Ability to specify thresholds for each page (again with a fair degree of flexibility) and mark responses as failed
e. Flexible notification schemes

The choice of technologies I used were based more on things I wanted to learn or refresh rather than the best there is, so keep in mind this is more of a toy than I would have liked
a. JMeter for response times. I'm not really interested in loading the site, nor do I want exact browser render times, I'm just looking for ballpark numbers and deviations, especially after builds or at odd hours.
b. Hudson for scheduling Jmeter builds. I chose hudson because I haven't used it before.
c. STAX for parsing JTL. I chose stax because I wanted to be able to parse large files, and i already know SAX , but I've never used STAX.
d. Tomcat with JSP + Spring. I've loved Spring JDBC ever since I've used it (take that Hibernate, JPA, JDO, EJB). It removes all the redundant code while not sacrificing the power of SQL , and there is no learning curve beyond Spring. I chose JSP over any of the MVC framework because of shortage of time. While people may insist how their preferred framework saves them tons of time , this only applies in the long run
e. JQuery for all the javascript stuff
f. JFreeChart for the chart related functionality. I've used this before and found it to be a solid library.
g. Derby for the database, this is something I have not used before, I wanted a reasonably stable database , non embedded.

Most of the things Ive written aren't really reusable, in addition this was quick and dirty, so don't expect this to work for you
a. The JMeter Script
b. Parsing the JMeter Script and loading it into the database
c. Scheduling JMeter to run in Hudson
d. Writing a UI around this
e. Allowing administrators to configure thresholds and notifications
f. Notifications