When you ask a J2EE guy something about performance tuning you'd probably get something that includes JVM tuning heap space, survivor ratio's , types of GC or you might get the don't use EJB , minimise the number of remote calls , or the use Hibernate , use Prepared Statements, use Cache etc etc. Interestingly enough for my current project we had implemented most of the above, tested it locally and seemed to get <4 second times for most pages. And we missed a big problem (hint it's a web based system) , can you guess?
The system responds in under 4 seconds when accessed locally. When accessed through a browser based in New York some (Can you guess?) requests take 20+ seconds. Luckily we did run external tests as soon as the production systems were available so we found this out a couple of months before the system was actually released.
In any case , in hindsight as soon as we knew the problem existed it was relatively easy to guess and verify(YSlow) the problems. We only had few small images so we knew the problem wasn't there.
a. Whats good for development isn't necessarily good for deployment. e.g. We do normally split CSS files , Javascript files and include them separately. Yes a browser will cache these files ( you do add Etags don't you?), but the first request and HTTPS will be slow if the browser must make these connections (normally at the most two at a time). - We used Yahoo and ANT to combine the CSS into one file and the Javascript into another (at build time) drastically reducing the response times
b. GZIP. Creating Gzipped versions of the files and dynamically gzipping all the content (a flag on the webserver) also brought down the times.
Just doing the above brought down the times to under 4 seconds even when the site was accessed remotely on limited bandwidth clients.
Moral of the story : Always test , never guess when it comes to performance tuning.
Sunday, August 23, 2009
Wednesday, August 19, 2009
Testing
There are various kinds of test's that can be performed on a system , but it looks like most engineers don't agree on the definition of the test. For e.g. in my current project a 'unit test' tests out the end to end functionality but it must be performed by a developer i.e. any test that a developer performs is a unit test!
In any case this is my usage
Unit Test - Generally meant to represent that only a small subset of the code is under test. You might find a few fanatics who argue to the effect that anything that goes to the database isn't a unit test, anything that needs an external interface isn't a unit test etc etc. Ignore them. The key is that the test might not exercise the flow as it finally would. There might be mocked interfaces, hardcoded data. Generally written by developers and generally easier to automate. Unit tests are also low hanging fruit, and contrary to what most agile engineers will tell you, are pretty much useless expect to impress some manager with 'We have automated unit tests!' , 'We have continuous builds with JUnit reports!','We have 99% code coverage!'. The quality of these tests written is mostly poor (they are written by developers after all), the test data provided rarely covers boundary conditions, invalid data, exceptional conditions. Yes Yes I know you should have high code coverage. It's quite easy to game the high code coverage (which is what happens when metrics are imposed by management) but even if it wasn't, the code coverage is only as good as the code and the test.
e.g. function int add(int one, int two) {
return 4;
}
Test function with add(2,2), add(-1,5) , 100% code coverage , automated unit tests all green , mission accomplished!?
i.e. If the developer doesn't account for the boundary conditions in code, he isn't likely to test out those scenarios either.
Integration Test - Generally tests out the interface between multiple systems, though the data might be faked. If there are only two types of test that you can carry out on your system then this is one of them.
These tests are extremely hard to write (or atleast make it repeatable), and are very very useful. The earlier you can have these tests up and running on your system, the better the quality of your system. These tests are hard to write not because of technical problems(which exist) but because of people issues. Different systems are normally run by different teams (sometimes even external to your organisation) have their own schedules , develop at their own pace and make their own assumptions and are notoriously non co-operative. Technical limitations are normally due to non repeatable time sensitive data which either needs the data reset to a known state or data created from scratch.
Functional Test (End to End test). - Tests out functionality from a user's perspective. This is the second type of test that must be run and the earlier you can run these tests in your system the better the quality of the system is. Normally nothing is faked , actual data is used for the tests. When these tests are performed by a business user / stake holder these become Acceptance tests. These tests are extremely important and difficult to fully automate. e.g. These would also include UI tests (this page doesnt work on my browser!).
Performance/Load - Functional tests run in parallel (normally a good mixture) with varying concurrent users. Easy to do if the functional tests can be automated. In most cases difficult to simulate (especially if the system is already live, easier to do the first time). Depending on the duration you run this is also by different names. Some organizations refer to long running tests as smoke tests. (used to smoke out memory leaks) whereas some organisations use smoke tests to refer to some important functional tests that are run to check that no major errors exist.
System Tests - Used to check that all systems are up and running. Not really a test category by itself , but useful to abort test runs.
In any case this is my usage
Unit Test - Generally meant to represent that only a small subset of the code is under test. You might find a few fanatics who argue to the effect that anything that goes to the database isn't a unit test, anything that needs an external interface isn't a unit test etc etc. Ignore them. The key is that the test might not exercise the flow as it finally would. There might be mocked interfaces, hardcoded data. Generally written by developers and generally easier to automate. Unit tests are also low hanging fruit, and contrary to what most agile engineers will tell you, are pretty much useless expect to impress some manager with 'We have automated unit tests!' , 'We have continuous builds with JUnit reports!','We have 99% code coverage!'. The quality of these tests written is mostly poor (they are written by developers after all), the test data provided rarely covers boundary conditions, invalid data, exceptional conditions. Yes Yes I know you should have high code coverage. It's quite easy to game the high code coverage (which is what happens when metrics are imposed by management) but even if it wasn't, the code coverage is only as good as the code and the test.
e.g. function int add(int one, int two) {
return 4;
}
Test function with add(2,2), add(-1,5) , 100% code coverage , automated unit tests all green , mission accomplished!?
i.e. If the developer doesn't account for the boundary conditions in code, he isn't likely to test out those scenarios either.
Integration Test - Generally tests out the interface between multiple systems, though the data might be faked. If there are only two types of test that you can carry out on your system then this is one of them.
These tests are extremely hard to write (or atleast make it repeatable), and are very very useful. The earlier you can have these tests up and running on your system, the better the quality of your system. These tests are hard to write not because of technical problems(which exist) but because of people issues. Different systems are normally run by different teams (sometimes even external to your organisation) have their own schedules , develop at their own pace and make their own assumptions and are notoriously non co-operative. Technical limitations are normally due to non repeatable time sensitive data which either needs the data reset to a known state or data created from scratch.
Functional Test (End to End test). - Tests out functionality from a user's perspective. This is the second type of test that must be run and the earlier you can run these tests in your system the better the quality of the system is. Normally nothing is faked , actual data is used for the tests. When these tests are performed by a business user / stake holder these become Acceptance tests. These tests are extremely important and difficult to fully automate. e.g. These would also include UI tests (this page doesnt work on my browser!).
Performance/Load - Functional tests run in parallel (normally a good mixture) with varying concurrent users. Easy to do if the functional tests can be automated. In most cases difficult to simulate (especially if the system is already live, easier to do the first time). Depending on the duration you run this is also by different names. Some organizations refer to long running tests as smoke tests. (used to smoke out memory leaks) whereas some organisations use smoke tests to refer to some important functional tests that are run to check that no major errors exist.
System Tests - Used to check that all systems are up and running. Not really a test category by itself , but useful to abort test runs.
Thursday, June 25, 2009
Wednesday, June 10, 2009
Fixing URLs
Motivation
For some tests, the URL to visit is extracted from the previous request using one of the PostProcessors (usually the Regex PostProcessor). However a URL containing parameters should normally have ampersands(&) escaped . This causes problems because JMeter will not automatically unescape these URL's
Solution
Use a javascript function to unescape the urls.
Sample
Assume that the Regex Post processor has extracted the url into a variable named returnUrl, then in the next HTTPSampler (Path field), instead of using ${returnUrl} use the function below.
${__javaScript('${returnUrl}'.replace(/amp;/gi\,''))}
This will simply replace amp; with a blank string so that a URL of the form http://www.yoursite.com/page?param1=value1¶m2=value2 becomes
http://www.yoursite.com/page?param1=value1¶m2=value2
Note that we don't need to do this for encoded values like %2F or whatever because that is taken care by the webserver
For some tests, the URL to visit is extracted from the previous request using one of the PostProcessors (usually the Regex PostProcessor). However a URL containing parameters should normally have ampersands(&) escaped . This causes problems because JMeter will not automatically unescape these URL's
Solution
Use a javascript function to unescape the urls.
Sample
Assume that the Regex Post processor has extracted the url into a variable named returnUrl, then in the next HTTPSampler (Path field), instead of using ${returnUrl} use the function below.
${__javaScript('${returnUrl}'.replace(/amp;/gi\,''))}
This will simply replace amp; with a blank string so that a URL of the form http://www.yoursite.com/page?param1=value1¶m2=value2 becomes
http://www.yoursite.com/page?param1=value1¶m2=value2
Note that we don't need to do this for encoded values like %2F or whatever because that is taken care by the webserver
Tuesday, June 09, 2009
Data DrivenTesting(from a database)
Motivation
Data to a test needs to be varied , however the input needs to be constrained to a set of values(normally in some database table). E.g. check the prices of the top 10 items. Here we need to provide 10 item ids , however if these values are kept in a CSV file , then the file needs to be repeatedly updated. There needs to be a way to get these values to the test at runtime
Solution
Use the JDBC Request Sampler to fetch the data at runtime. Use either the Save Responses to a file or a BeanShell post processor to write the data to a file. Finally use the CSV Data Set Config to read this data.
Sample
Data to a test needs to be varied , however the input needs to be constrained to a set of values(normally in some database table). E.g. check the prices of the top 10 items. Here we need to provide 10 item ids , however if these values are kept in a CSV file , then the file needs to be repeatedly updated. There needs to be a way to get these values to the test at runtime
Solution
Use the JDBC Request Sampler to fetch the data at runtime. Use either the Save Responses to a file or a BeanShell post processor to write the data to a file. Finally use the CSV Data Set Config to read this data.
Sample
Sunday, June 07, 2009
Varying the data to the test
Motivation
The same test's need to be run, but the data we pass to it must be varied.
In addition further tests may need to change their behavior depending on the data. E.g. A user registering may provide different data , and subsequent screens may change depending on what the user has entered or may be skipped altogether. A common scenario is when a user registers to a site has various options he can choose from , and we need to test the behavior of the site for different combinations
Solution
JMeter provides multiple ways to vary the data
a. Use of User Parameters [1]
b. Use of Variables [2]
c. CSV Data Set Config [3]
The solution we will use is Option c.
The advantage of using CSV Data Set Config is that the data is externalised from the test , and can be updated by any user including a non technical business person. By making the assertion a part of the data, users can add more tests without needing the test itself to be modified. The other advantage of a CSV Data Set Config over a User Parameters pre processor is that the number of items that will be tested is fixed independent of the number of threads you will run (assuming you write the test in that fashion) OR can be made dependent on the number of threads. User Parameters is more closely tied to the number of threads.
e.g. If you wanted to create 10 distinct users , you'd only have 10 rows in your CSV data set config and you could use 1 to 10 threads. But if you needed to do this with User Parameters you'd probably need to specify exactly 10 threads.
So the solution takes the form of
a. Create a CSV Data Set Config element and point it to the CSV files.
b. Create your tests to use this data
If you want as many tests as you have rows in your CSV file, then you can either end the thread or use a Loop Controller an check for the special value ''<EOF>"
Sample
References
[1] User Parameters
[2] Variables
[3] CSV Data Set Config
The same test's need to be run, but the data we pass to it must be varied.
In addition further tests may need to change their behavior depending on the data. E.g. A user registering may provide different data , and subsequent screens may change depending on what the user has entered or may be skipped altogether. A common scenario is when a user registers to a site has various options he can choose from , and we need to test the behavior of the site for different combinations
Solution
JMeter provides multiple ways to vary the data
a. Use of User Parameters [1]
b. Use of Variables [2]
c. CSV Data Set Config [3]
The solution we will use is Option c.
The advantage of using CSV Data Set Config is that the data is externalised from the test , and can be updated by any user including a non technical business person. By making the assertion a part of the data, users can add more tests without needing the test itself to be modified. The other advantage of a CSV Data Set Config over a User Parameters pre processor is that the number of items that will be tested is fixed independent of the number of threads you will run (assuming you write the test in that fashion) OR can be made dependent on the number of threads. User Parameters is more closely tied to the number of threads.
e.g. If you wanted to create 10 distinct users , you'd only have 10 rows in your CSV data set config and you could use 1 to 10 threads. But if you needed to do this with User Parameters you'd probably need to specify exactly 10 threads.
So the solution takes the form of
a. Create a CSV Data Set Config element and point it to the CSV files.
b. Create your tests to use this data
If you want as many tests as you have rows in your CSV file, then you can either end the thread or use a Loop Controller an check for the special value '
Sample
References
[1] User Parameters
[2] Variables
[3] CSV Data Set Config
Friday, June 05, 2009
Testing multiple environments with JMeter
Motivation
Most projects have a number of environments through which the code moves. e.g. A Development environment, A Test environment, A User Acceptance test environment, A reference environment and finally the Production environment. Hence the same test script is to be targeted to multiple environments. In theory all of this is automated , and anything that succeeds/fails in one environment would succeed/fail consistently in all other environments. However environmental differences and human errors almost always cause one to hear "But it works on my machine"
Whats needed is a way to run the same test against different environments easily
Solution
An assumption we are going to make here is that these tests are automated and would be run from the command line, in our case using ANT.
The solution has to address two basic requirements
a. Parameterization of the various environments
b. You should still be able to run the test in GUI mode (when you are modifying/extending the test)
To implement this we will use JMeter properties[1] and use normal ANT [2],[3] features.
While writing the HTTP tests, add an HTTP Request Default element
If the Server name or IP is to be varied then enter
${__property(run.server,,yourdevserver.com)}
This will look for a property named run.server , but will use yourdevserver.com if no property is specified. For the third parameter, use the server against which you want to run the test while running in GUI mode.
[1] JMeter Properties
[2] JMeter Ant task
[3] Ant Manual
Most projects have a number of environments through which the code moves. e.g. A Development environment, A Test environment, A User Acceptance test environment, A reference environment and finally the Production environment. Hence the same test script is to be targeted to multiple environments. In theory all of this is automated , and anything that succeeds/fails in one environment would succeed/fail consistently in all other environments. However environmental differences and human errors almost always cause one to hear "But it works on my machine"
Whats needed is a way to run the same test against different environments easily
Solution
An assumption we are going to make here is that these tests are automated and would be run from the command line, in our case using ANT.
The solution has to address two basic requirements
a. Parameterization of the various environments
b. You should still be able to run the test in GUI mode (when you are modifying/extending the test)
To implement this we will use JMeter properties[1] and use normal ANT [2],[3] features.
While writing the HTTP tests, add an HTTP Request Default element
If the Server name or IP is to be varied then enter
${__property(run.server,,yourdevserver.com)}
This will look for a property named run.server , but will use yourdevserver.com if no property is specified. For the third parameter, use the server against which you want to run the test while running in GUI mode.
Finally in your Build Script that you use to run Jmeter
<jmeter jmeterhome=".." testplan="${run.test.plan}"
resultlog="${report.dir}/${run.test.report}-${run.env}-${DSTAMP}${TSTAMP}.jtl">
<property name="jmeter.save.saveservice.output_format" value="xml"/>
<property name="run.server" value="${run.server}" />
</jmeter>
where the ANT property run.server can be varied to run this against different environment.
Sample
TODO attach ant and jmx file.
[1] JMeter Properties
[2] JMeter Ant task
[3] Ant Manual
JMeter Prologue
I've been testing my existing work application using JMeter for the last 6 months , so what follows are a series of posts that I think isn't covered in most documentation or available online easily or in the same place (I had to figure out these solutions myself). However there is a caveat to that, the solution may not be the best possible one , may not be efficient and there may be more elegant ways to do what the solution does. In which case please inform me. However the solutions do have one thing in common, they worked for me. Your mileage may vary.
The rest of the post is a rant , feel free to move on
A few observations on a Friday(with a tip of the hat to BusyBee)
That people who keep asking for Unit Tests don't have a clue about testing web based applications.
That data driven integration testing between multiple systems is damn hard to do , but boy is it satisfying when you actually find a defect due to it, and boy is it worth your time to develop these tests.
That open source tools are so much better than some of the paid commercial tools. Except when it comes to presentation of the reports. Which surprisingly is also the difference between a Techie and a Manager.
That there still is no tool to truly perform visual tests on a website.
And That someone who could develop such a tool could make a lot of money.
And that someone wont be me.
That there is no time to test all the normal cases, much less the weird corner ones. and that there would always have been plenty of time in hindsight, if we had just gotten our act together earlier
That a successful test is the one that fails in local,development, test, QA , UAT environments. That a test thats succeeds in the above environments but fails on the production environment is a pain.
That a web based test should never underestimate the ignorance of a user.
That developers do not make good testers. And that most developers are better testers than the official testing team, especially when we test other developers code. And that scares me.
The rest of the post is a rant , feel free to move on
A few observations on a Friday(with a tip of the hat to BusyBee)
That people who keep asking for Unit Tests don't have a clue about testing web based applications.
That data driven integration testing between multiple systems is damn hard to do , but boy is it satisfying when you actually find a defect due to it, and boy is it worth your time to develop these tests.
That open source tools are so much better than some of the paid commercial tools. Except when it comes to presentation of the reports. Which surprisingly is also the difference between a Techie and a Manager.
That there still is no tool to truly perform visual tests on a website.
And That someone who could develop such a tool could make a lot of money.
And that someone wont be me.
That there is no time to test all the normal cases, much less the weird corner ones. and that there would always have been plenty of time in hindsight, if we had just gotten our act together earlier
That a successful test is the one that fails in local,development, test, QA , UAT environments. That a test thats succeeds in the above environments but fails on the production environment is a pain.
That a web based test should never underestimate the ignorance of a user.
That developers do not make good testers. And that most developers are better testers than the official testing team, especially when we test other developers code. And that scares me.
Thursday, April 30, 2009
Saturday, February 02, 2008
Prototype
Name : PrototypeContext : Obtain a copy
Consequences :
1)Add remove products at runtime
2)Specify new objects by varying value or structure
3)Reduced Subclassing
2)Specify new objects by varying value or structure
3)Reduced Subclassing
4)Configure an application with class dynamically
Examples
Anything implementing clone(),
Class
Factory method
Name : Factory MethodContext : Hide the implementation class but defer the instantiation to subclass
Consequence
a) Provides hooks for subclass
a) Provides hooks for subclass
Examples
Calendar.getInstance()
Connection conn = java.sql.DriverManager.getConnection(uri, name, password);
Friday, February 01, 2008
Abstract Factory
Name : Abstract FactoryContext : Creating Related family of objects
Example: Toolkit(awt)
java.sql.Connection (sort of creates Statements and PreparedStatements and CallableStatements where each is specific to the database)
Consequence
1) Exchange Product Families
2) Supporting new kinds of products
3) Factories are normally singletons! But they need not be.
Consequence
1) Exchange Product Families
2) Supporting new kinds of products
3) Factories are normally singletons! But they need not be.
4)Note that every factory must support every type of object to be created. This is sometimes problematic.
Singleton (Creational)
Name : SingletonComment : Probably the best known (atleast if you go by answers you get in interviews to the question 'Patterns you have used') of the design patterns and probably one of the text book cases of people ignoring understand the context and understand the consequences when it comes to this pattern
Context : Need only 1(or a fixed or controllable number of instances) (e.g. memory consumption , or heavy initialization like reading a file,a requirement - there is only one Neo, a pre java 1.5 enum such that == can be used )
Examples : Single Instance (java.awt.Toolkit, Inspector)
Consequences :
a) Controlled access to state
b) Permits a variable number of instances (e.g. access to any object pool)!
c) Watch out for real world multiple managed servers, clusters etc(older versions of Expresso framework used an in memory primary key generator as an incremented singleton - no use in a cluster. Real world naive code that uses a timestamp plus ip address as unique key through a singleton)
d) Watch out for synchronized access for loading the singleton, thread safety, shared state.
e) Watch out for serializable (might break singletoness if you dont follow effective java)
f) Watch out for static block errors during initialization(if initialising eagerly). Will cause NoClassDefFoundErrors.
g) Watch out for being unable to reset a singleton (e.g. a singleton that reads a file and stores it.If your not careful , changing the file needs a bounce of the server)
h) Watch out for singletonitis. i.e. Making classes singleton even when there is no shared state, no performance or initialisation penalty.
i) Watch out for problems testing the singleton
With all the Watch out's is this pattern worth the trouble? read the context!
Design Patterns Course
The following series of posts deal with a design pattern course I conducted for a few colleagues of my team who insisted that I knew enough to conduct such a course(I don't). One of the problems I had faced while attending the design patterns course conducted by probably the best Technical Manager I've worked under (Homi Bharda) was that practical examples were hard to come by, where you could see the usefullness and though the Design patterns book by Erich Gamma did point to some examples they almost always dealt with GUI heavy , stateful , highly object oriented programs while at that stage I dealt with mostly stateless CRUD web based applications.
I resolved that any course I took would show practical examples and what better example than the Java API which was quite familiar to my colleagues. I also had a lot of problem differentiating between patterns. Some of the Factory patterns looked alike. Some of the patterns like Command and Strategy looked alike. I resolved to have some comparisons at least.
What these notes dont have is my bitterly cynical comments e.g. of people who code such that all their classes have a suffix which is pattern e.g. ObjectCommand uses an ObjectFacade uses an ObjectFactory creates an ObjectService which uses an ObjectDAO and returns an ObjectAdapter which wraps an ObjectDecorator which wraps a ObjectDTO (Disclaimer I have done this too). This literally is the developer's cry look Im a designer , I use patterns , I even know their names! But ask them Give me some examples of design patterns in the Java SDK and we have a) Blank Stares b) Singleton c) A very very few Decorator.
So anyway other than the above rant , very few cynical comments
General Notes
What is a pattern ?
A Name, A Recurring problem with context, A recommended Solution(or way ahead), Consequences and Alternatives
The bit a lot of people miss out is the Context and the Consequence. A pattern cannot be used irrespective of the context . This causes problems like 'Singletonitis'. A pattern always has some consequence. A pattern probably does have alternatives which are to be evaluated.
A pattern need not always be up front design, you can arrive at many patterns with a judicious use of refactoring and some simple principles notably DRY.
Patterns are also sometimes subsumed by the language or the platform (e.g. Factory in Visual Basic COM and Iterator in java) and some people refer to patterns as deficiency in the language / platform ( I dont agree!).
Anyone who does anything with Design Patterns has to refer to Erich Gamma's et al book , and thats what I have based this on including the organisation of the patterns as Creational, Behavioral and Structural. Its quite difficult for novices (like me) to differentiate what between what constitutes a behavioral or structural pattern so dont bother right now.
Also note that any Object oriented system has available to it the mechanisms of Polymorphism and Inheritance (Implementation of Interface). Therefore all diagrams of Design patterns will look similar in a lot of case. it is necessary to understand the nuance and differences and context.
And lastly in a web project , differentiate between building a framework and solving a problem at hand. Its very easy to get carried away and over engineer and its as easy to under engineer, and it's also simple to shout TDD, refactoring , agile. Ultimately experience is what you use to decide whether you need a pattern upfront or later, Refactor or not. And anyone who tells you otherwise is either a manager or a consultant.
I resolved that any course I took would show practical examples and what better example than the Java API which was quite familiar to my colleagues. I also had a lot of problem differentiating between patterns. Some of the Factory patterns looked alike. Some of the patterns like Command and Strategy looked alike. I resolved to have some comparisons at least.
What these notes dont have is my bitterly cynical comments e.g. of people who code such that all their classes have a suffix which is pattern e.g. ObjectCommand uses an ObjectFacade uses an ObjectFactory creates an ObjectService which uses an ObjectDAO and returns an ObjectAdapter which wraps an ObjectDecorator which wraps a ObjectDTO (Disclaimer I have done this too). This literally is the developer's cry look Im a designer , I use patterns , I even know their names! But ask them Give me some examples of design patterns in the Java SDK and we have a) Blank Stares b) Singleton c) A very very few Decorator.
So anyway other than the above rant , very few cynical comments
General Notes
What is a pattern ?
A Name, A Recurring problem with context, A recommended Solution(or way ahead), Consequences and Alternatives
The bit a lot of people miss out is the Context and the Consequence. A pattern cannot be used irrespective of the context . This causes problems like 'Singletonitis'. A pattern always has some consequence. A pattern probably does have alternatives which are to be evaluated.
A pattern need not always be up front design, you can arrive at many patterns with a judicious use of refactoring and some simple principles notably DRY.
Patterns are also sometimes subsumed by the language or the platform (e.g. Factory in Visual Basic COM and Iterator in java) and some people refer to patterns as deficiency in the language / platform ( I dont agree!).
Anyone who does anything with Design Patterns has to refer to Erich Gamma's et al book , and thats what I have based this on including the organisation of the patterns as Creational, Behavioral and Structural. Its quite difficult for novices (like me) to differentiate what between what constitutes a behavioral or structural pattern so dont bother right now.
Also note that any Object oriented system has available to it the mechanisms of Polymorphism and Inheritance (Implementation of Interface). Therefore all diagrams of Design patterns will look similar in a lot of case. it is necessary to understand the nuance and differences and context.
And lastly in a web project , differentiate between building a framework and solving a problem at hand. Its very easy to get carried away and over engineer and its as easy to under engineer, and it's also simple to shout TDD, refactoring , agile. Ultimately experience is what you use to decide whether you need a pattern upfront or later, Refactor or not. And anyone who tells you otherwise is either a manager or a consultant.
Friday, January 25, 2008
Testing
You really appreciate the value of tests (especially regression) when you have to fix some code which is complex and is used in multiple places and you aren't sure of the various conditions under which it gets invoked (or even what the result should be). Unfortunately it is sometimes difficult to convince management. We asked and were denied initially anyway , for a separate environment to create and run tests. The separate environments were needed because of the extreme complexity of the business needed, in my opinion, an environment where the initial state is exactly known , otherwise there is no way to accurately predict the outcome of running time sensitive, old data sensitive scenarios. However the cost of maintaing said environment was deemed to be too high. Im not so sure. anyway we are getting an environment so yay, let the games begin.
It's also at time like these that you realise , how out of touch with reality , the TDD/Agile/ JUnit testing people that you see are , because they never seem to discuss anything but unit testing.
I cant see unit test's adding much value to the system I work in , but oh what wouldn't I give for a set of repeatable integration and functional tests.
Other than the really good Java Next Generation Testing book I havent seen the Testing books deal with real world stuff. (Did anyone else think that the Junit cookbook example of Money was representative of what people test in an automated way in real life)
So here are the things that make the testing complicated
a. The system run's on a cluster. Problems have been reported on the cluster , some of which have found to be environment related and some are code related. Would anything other than in container testing on representative environments find this out?. This deals directly with the people advocating 'mocking' interfaces, not running in container, having fast running tests etc.
b. One of the defects was caused when two unrelated processes ran at exactly the same time and on the same server of the cluster. The processes shared infrastructure code but were business wise and code wise unrelated. How would a test have found this out , other than sheer luck? Note that junit as a framework is really bad. Again this is not a complaint about JUnit, just that you dont see people discussing concurrent testing of unrelated data.
c. A complex Architecture. A portal sends data asynchronously (JMS) to a WLI system which has multiple processes , which in turn needs a lot of system data (histories) to perform calculations, makes its business decisions using a set of business rules which are User Configurable(using a totally different system) and then persists this data to a database.
Assuming you want to test that everything works end to end, is there any other way other than restoring the system to a known point in time(so you know what the out come should be?). The catch again is that running the same test on the exact same data might not still return the same results because the results are time sensitive (e.g. there is a different contract for 2007 as compared to 2008 so running the test in 2008 will give you different results).
d. The results depend on the sequence of tests. This isn't a code problem , this is exactly how the business works. Other than creating different data for different cases , this is difficult to solve. In addition you do want to also test what happens when a particular sequence of events occur ( a claim is in , it is paid, the cheque is sent, the claim is adjusted, a new check is sent , the first one is returned, now assert). How is it that the unit testing folks shout themselves hoarse that tests should be independent of each other. I know I know , what I have described isn't a unit test. But you see , the individual bits work. The sequence fails.
Oh I can go on. But there is an India v/s Australia cricket match. Go India!
It's also at time like these that you realise , how out of touch with reality , the TDD/Agile/ JUnit testing people that you see are , because they never seem to discuss anything but unit testing.
I cant see unit test's adding much value to the system I work in , but oh what wouldn't I give for a set of repeatable integration and functional tests.
Other than the really good Java Next Generation Testing book I havent seen the Testing books deal with real world stuff. (Did anyone else think that the Junit cookbook example of Money was representative of what people test in an automated way in real life)
So here are the things that make the testing complicated
a. The system run's on a cluster. Problems have been reported on the cluster , some of which have found to be environment related and some are code related. Would anything other than in container testing on representative environments find this out?. This deals directly with the people advocating 'mocking' interfaces, not running in container, having fast running tests etc.
b. One of the defects was caused when two unrelated processes ran at exactly the same time and on the same server of the cluster. The processes shared infrastructure code but were business wise and code wise unrelated. How would a test have found this out , other than sheer luck? Note that junit as a framework is really bad. Again this is not a complaint about JUnit, just that you dont see people discussing concurrent testing of unrelated data.
c. A complex Architecture. A portal sends data asynchronously (JMS) to a WLI system which has multiple processes , which in turn needs a lot of system data (histories) to perform calculations, makes its business decisions using a set of business rules which are User Configurable(using a totally different system) and then persists this data to a database.
Assuming you want to test that everything works end to end, is there any other way other than restoring the system to a known point in time(so you know what the out come should be?). The catch again is that running the same test on the exact same data might not still return the same results because the results are time sensitive (e.g. there is a different contract for 2007 as compared to 2008 so running the test in 2008 will give you different results).
d. The results depend on the sequence of tests. This isn't a code problem , this is exactly how the business works. Other than creating different data for different cases , this is difficult to solve. In addition you do want to also test what happens when a particular sequence of events occur ( a claim is in , it is paid, the cheque is sent, the claim is adjusted, a new check is sent , the first one is returned, now assert). How is it that the unit testing folks shout themselves hoarse that tests should be independent of each other. I know I know , what I have described isn't a unit test. But you see , the individual bits work. The sequence fails.
Oh I can go on. But there is an India v/s Australia cricket match. Go India!
Debugging
Due to circumstances beyond our control , we've had to take up fixing someone else's code. The business rules are complex (which means the code must be so). There are no existing tests (this is a consequence of both the complex rules and a complex architecture , the latter could be simplified if it were not for the fact that the project is years late). Now the TDD guys are probably jumping up and down at this stage , pointing out that the lack of tests is the reason why everything is so complex , just ignore them , we'll get to these folks later. The bottom line is there are no tests, we are trying to create some regression tests, that will take time , meanwhile there are still bugs to be resolved. I was assigned one of them.
It took me 2 hours to setup the environment . Ones a weblogic portal domain and Ones a weblogic integration domain. The build for the portal takes about 30 minutes after which weblogic workshop duly crashes. I have to point my servers to a configuration other than their default which needs me to mess around manually. It takes another hour to understand what the defect is. it takes some more time to verify that the tester is indeed right, the system is doing something wrong. It takes some time to identify the areas of code that are possibly the cause. it takes a couple of hours to understand the nuances in the mostly undocumented code.
I then have to go and ask another dev, whether the testers expected results are correct, and it turns out they aren't. There is a defect and what the system currently does is wrong , but what should the system do is undocumented and unknown. We could make a guess but this impacts other systems so we need them to confirm. A meeting is scheduled.
The meeting lasts 2 hours. Some issues are resolved, Some new ones are raised. Some have to be followed up with other people. Armed with some more knowledge I look through the code. I get a sudden inspiration, I test out two scenarios , it looks like I'm right.
The defect is fixed by modifying all of two lines of code. Total time to fix defect ? 1.5 days.
It took me 2 hours to setup the environment . Ones a weblogic portal domain and Ones a weblogic integration domain. The build for the portal takes about 30 minutes after which weblogic workshop duly crashes. I have to point my servers to a configuration other than their default which needs me to mess around manually. It takes another hour to understand what the defect is. it takes some more time to verify that the tester is indeed right, the system is doing something wrong. It takes some time to identify the areas of code that are possibly the cause. it takes a couple of hours to understand the nuances in the mostly undocumented code.
I then have to go and ask another dev, whether the testers expected results are correct, and it turns out they aren't. There is a defect and what the system currently does is wrong , but what should the system do is undocumented and unknown. We could make a guess but this impacts other systems so we need them to confirm. A meeting is scheduled.
The meeting lasts 2 hours. Some issues are resolved, Some new ones are raised. Some have to be followed up with other people. Armed with some more knowledge I look through the code. I get a sudden inspiration, I test out two scenarios , it looks like I'm right.
The defect is fixed by modifying all of two lines of code. Total time to fix defect ? 1.5 days.
Wednesday, February 14, 2007
Weblogic Portal interview questions
The following are the Portal interview questions that ive used or kept or have been asked(in no particular order , and no answers either :) )
I do not include questions (e.g. what is a nested pageflow) that can be answered with google.
Also see Weblogic Portal interview questions - II
To be continued...
I do not include questions (e.g. what is a nested pageflow) that can be answered with google.
Also see Weblogic Portal interview questions - II
- In what situations can you not use a UUP ?. What factors determine choosing a UUP?
- What are the different techniques you can use for Inter Portlet Communications?. What are the pros and cons of each technique?.
- Would you choose 1 desktop with entitlements or multiple desktops? Why?
- What are the options for security in Weblogic Portal?
- Can you change a LookAndFeel at runtime ?, if so what are the alternatives?
- What is the impact of using AJAX in a portlet (out of the box or otherwise)?
- Why should pageflow variables be serializable?
- If your using a file repository for content and you have a webserver in front of the application servers who serves the static files?
- When do you use the refresh action?
- What happens if the validate method of your form bean throws an exception
- How do you use the struts validation framework with BEA pageflows?
- The BEA pageflow documentation mentions that there is only 1 current pageflow. And that the current pageflow is destroyed when a new one is accessed. is that accurate?
- What all features should a screen scraping portlet possess?
To be continued...
Tuesday, February 13, 2007
Weblogic debug flags
Reproduced from Hussein Badakhchani's Blog and comments
weblogic debug flags(8.1)
IIOP
weblogic.iiop.ots
weblogic.iiop.transport
weblogic.iiop.marshal
weblogic.iiop.startup
JDBC/Data Sources
weblogic.JDBCConn
weblogic.JDBCSQL
weblogic.JDBCConnStackTrace
JMX/Deployment
weblogic.commoAdmin
weblogic.commoProxy
weblogic.deployerRuntime
weblogic.MasterDeployer
weblogic.deployTask
weblogic.deployHelper
weblogic.MasterDeployer
weblogic.OamDelta
weblogic.OamVersion
weblogic.slaveDeployer.semaphore
weblogic.slaveDeployer
weblogic.ConfigMBean
weblogic.ConfigMBeanEncrypt
weblogic.ConfigMBeanSetAttribute
weblogic.management.DynamicMBeanImpl
weblogic.management.DynamicMBeanImpl.setget
weblogic.mbeanProxyCache
weblogic.mbeanDelete
weblogic.mbeanQuery
weblogic.MBeanInteropList
weblogic.mbeanProxy
weblogic.registerMBean
weblogic.getMBeanInfo
weblogic.getMBeanAttributes
weblogic.addDependenciesRecursively
weblogic.MBeanListener
weblogic.application
weblogic.deployer
weblogic.appPoller
weblogic.appManager
weblogic.BootstrapServlet
weblogic.fileDistributionServlet
Application Deployment
weblogic.J2EEApplication
weblogic.application
weblogic.appPoller
weblogic.appManager
JTA
weblogic.JTAGateway
weblogic.JTAGatewayStackTrace
weblogic.JTA2PC
weblogic.JTA2PCStackTrace
weblogic.JTAHealth
weblogic.JTAPropagate
weblogic.JTARecovery
weblogic.JTAXA
weblogic.JTAXAStackTrace
weblogic.JTAResourceHealth
weblogic.JTAMigration
weblogic.JTARecoveryStackTrace
weblogic.JTANaming
weblogic.JTATLOG
weblogic.JTALifecycle
EJB
weblogic.ejb.cache.debug
weblogic.ejb.cache.verbose
ejb.enableCacheDump
weblogic.ejb20.cmp.rdbms.debug
weblogic.ejb20.cmp.rdbms.verbose
weblogic.ejb20.persistence.debug
weblogic.ejb20.persistence.verbose
weblogic.ejb20.compliance.debug
weblogic.ejb20.compliance.verbose
weblogic.ejb.deployment.debug
weblogic.ejb.deployment.verbose
weblogic.ejb20.dd.xml
weblogic.ejb.deployer.debug
weblogic.ejb.deployer.verbose
weblogic.ejb.verbose.deployment
weblogic.ejb20.ejbc.debug
weblogic.ejb20.ejbc.verbose
weblogic.ejb.runtime.debug
weblogic.ejb.runtime.verbose
weblogic.ejb20.jms.poll.debug
weblogic.ejb20.jms.poll.verbose
weblogic.ejb20.security.debug
weblogic.ejb20.security.verbose
weblogic.ejb.locks.debug
weblogic.ejb.locks.verbose
weblogic.ejb.bean.manager.debug
weblogic.ejb.bean.manager.verbose
weblogic.ejb.pool.InstancePool.debug
weblogic.ejb.pool.InstancePool.verbose
weblogic.ejb.swap.debug
weblogic.ejb.swap.verbose
weblogic.j2ee.dd.xml
General
weblogic.debug
weblogic.kernel.debug
weblogic.debug.DebugConnection
weblogic.debug.DebugRouting
weblogic.debug.DebugMessaging
weblogic.debug.isLogRemoteExceptionsEnabled
weblogic.StdoutDebugEnabled
WLI
wlc.debug.signature
wli.bpm.client.security.debug
wli.bpm.studio.timeprocessor.debug
wli.bpm.studio.debug
wli.bpm.server.common.timedevent.debug
wli.bpm.server.common.xmltemplate.debug
wli.bpm.server.eventprocessor.addrmsgdebug
wli.bpm.server.eventprocessor.debug
wli.bpm.server.jms.debug
wli.bpm.server.plugin.debug
wli.bpm.server.workflow.debug
wli.bpm.server.businesscalendar.debug
wli.bpm.server.busop.debug
wli.bpm.server.workflow.action.taskduedate.debug
wli.bpm.server.workflow.timedevent.debug
wli.bpm.server.xml.debug
wli.bpm.server.xslt.debug
wli.bpm.server.workflow.start.debug
wli.bpm.server.workflowprocessor.debug
wli.common.server.errorlistener.debug
Messaging Bridge
-Dweblogic.debug.DebugMessagingBridgeStartup=true -Dweblogic.debug.DebugMessagingBridgeRuntime=true And two others for stdout and stderr : -Dweblogic.Stdout= -Dweblogic.Stderr=
SSL
-Dweblogic.security.SSL.verbose=true -Dssl.debug=true
-9.x available via console plus
webservices: -Dweblogic.wsee.verbose=*
6.x,7.x
http://wldj.sys-con.com/read/42733.htm
RMI debug flags
java.rmi.server.logCalls=true
sun.rmi.loader.logLevel=[BRIEF|VERBOSE]
sun.rmi.server.exceptionTrace
sun.rmi.server.logLevel=[BRIEF|VERBOSE]
weblogic debug flags(8.1)
IIOP
weblogic.iiop.ots
weblogic.iiop.transport
weblogic.iiop.marshal
weblogic.iiop.startup
JDBC/Data Sources
weblogic.JDBCConn
weblogic.JDBCSQL
weblogic.JDBCConnStackTrace
JMX/Deployment
weblogic.commoAdmin
weblogic.commoProxy
weblogic.deployerRuntime
weblogic.MasterDeployer
weblogic.deployTask
weblogic.deployHelper
weblogic.MasterDeployer
weblogic.OamDelta
weblogic.OamVersion
weblogic.slaveDeployer.semaphore
weblogic.slaveDeployer
weblogic.ConfigMBean
weblogic.ConfigMBeanEncrypt
weblogic.ConfigMBeanSetAttribute
weblogic.management.DynamicMBeanImpl
weblogic.management.DynamicMBeanImpl.setget
weblogic.mbeanProxyCache
weblogic.mbeanDelete
weblogic.mbeanQuery
weblogic.MBeanInteropList
weblogic.mbeanProxy
weblogic.registerMBean
weblogic.getMBeanInfo
weblogic.getMBeanAttributes
weblogic.addDependenciesRecursively
weblogic.MBeanListener
weblogic.application
weblogic.deployer
weblogic.appPoller
weblogic.appManager
weblogic.BootstrapServlet
weblogic.fileDistributionServlet
Application Deployment
weblogic.J2EEApplication
weblogic.application
weblogic.appPoller
weblogic.appManager
JTA
weblogic.JTAGateway
weblogic.JTAGatewayStackTrace
weblogic.JTA2PC
weblogic.JTA2PCStackTrace
weblogic.JTAHealth
weblogic.JTAPropagate
weblogic.JTARecovery
weblogic.JTAXA
weblogic.JTAXAStackTrace
weblogic.JTAResourceHealth
weblogic.JTAMigration
weblogic.JTARecoveryStackTrace
weblogic.JTANaming
weblogic.JTATLOG
weblogic.JTALifecycle
EJB
weblogic.ejb.cache.debug
weblogic.ejb.cache.verbose
ejb.enableCacheDump
weblogic.ejb20.cmp.rdbms.debug
weblogic.ejb20.cmp.rdbms.verbose
weblogic.ejb20.persistence.debug
weblogic.ejb20.persistence.verbose
weblogic.ejb20.compliance.debug
weblogic.ejb20.compliance.verbose
weblogic.ejb.deployment.debug
weblogic.ejb.deployment.verbose
weblogic.ejb20.dd.xml
weblogic.ejb.deployer.debug
weblogic.ejb.deployer.verbose
weblogic.ejb.verbose.deployment
weblogic.ejb20.ejbc.debug
weblogic.ejb20.ejbc.verbose
weblogic.ejb.runtime.debug
weblogic.ejb.runtime.verbose
weblogic.ejb20.jms.poll.debug
weblogic.ejb20.jms.poll.verbose
weblogic.ejb20.security.debug
weblogic.ejb20.security.verbose
weblogic.ejb.locks.debug
weblogic.ejb.locks.verbose
weblogic.ejb.bean.manager.debug
weblogic.ejb.bean.manager.verbose
weblogic.ejb.pool.InstancePool.debug
weblogic.ejb.pool.InstancePool.verbose
weblogic.ejb.swap.debug
weblogic.ejb.swap.verbose
weblogic.j2ee.dd.xml
General
weblogic.debug
weblogic.kernel.debug
weblogic.debug.DebugConnection
weblogic.debug.DebugRouting
weblogic.debug.DebugMessaging
weblogic.debug.isLogRemoteExceptionsEnabled
weblogic.StdoutDebugEnabled
WLI
wlc.debug.signature
wli.bpm.client.security.debug
wli.bpm.studio.timeprocessor.debug
wli.bpm.studio.debug
wli.bpm.server.common.timedevent.debug
wli.bpm.server.common.xmltemplate.debug
wli.bpm.server.eventprocessor.addrmsgdebug
wli.bpm.server.eventprocessor.debug
wli.bpm.server.jms.debug
wli.bpm.server.plugin.debug
wli.bpm.server.workflow.debug
wli.bpm.server.businesscalendar.debug
wli.bpm.server.busop.debug
wli.bpm.server.workflow.action.taskduedate.debug
wli.bpm.server.workflow.timedevent.debug
wli.bpm.server.xml.debug
wli.bpm.server.xslt.debug
wli.bpm.server.workflow.start.debug
wli.bpm.server.workflowprocessor.debug
wli.common.server.errorlistener.debug
Messaging Bridge
-Dweblogic.debug.DebugMessagingBridgeStartup=true -Dweblogic.debug.DebugMessagingBridgeRuntime=true And two others for stdout and stderr : -Dweblogic.Stdout= -Dweblogic.Stderr=
SSL
-Dweblogic.security.SSL.verbose=true -Dssl.debug=true
-9.x available via console plus
webservices: -Dweblogic.wsee.verbose=*
6.x,7.x
http://wldj.sys-con.com/read/42733.htm
RMI debug flags
java.rmi.server.logCalls=true
sun.rmi.loader.logLevel=[BRIEF|VERBOSE]
sun.rmi.server.exceptionTrace
sun.rmi.server.logLevel=[BRIEF|VERBOSE]
Subscribe to:
Posts (Atom)