Monday, October 12, 2009

Detecting missing files with JMeter

I have lately found that JMeter is becoming my tool of choice for almost all the normal mundane programming tasks. Case in point.
On my current website's we have a bunch of PDF's (150K) which are accessible only via search and have an entry in some table for that purpose. Each PDF is linked to a language and multiple countries so the total number of rows in the database are much more than the number of files. Now years later, due to human error and other causes some of these records exist in the database but there aren't corresponding PDF files on the webserver , which allows the user to see a link when he searches for the data but a 404 error when the user actually clicks it. I had to generate a report listing all these files.

Constraints
a. Administrators wont let you run a program on the web server.
b. You could ask them to copy files to a separate directory but it takes about a week to get approval for anything related to production except for a Database copy (which is available immediately)

I initially thought of asking for a recursive file name print of all the files from the webserver to compare against the database (but writing this Java program would have taken half a day to iron out the bugs) . So I settled on JMeter

Run query to get a list of files and save it to a CSV (Squirrel SQL client)
Thread Group (10 in parallel)
CSV Data Set
Http Request (HEAD) , the web link to the PDF is read from the CSV

Run from command line with a sample_variables property set to fields from the CSV.
Time to run test = 1.5 hrs. JMeter's sample HTML was good enough to be shown to the users to fix the issues.
Whats great is that when the missing files are uploaded I can verify the data easily again.

Now I could have used JMeter's JDBC sampler to eliminate the Squirrel client.

No comments: