Performance Tuning tales

When you ask a J2EE guy something about performance tuning you'd probably get something that includes JVM tuning heap space, survivor ratio's , types of GC or you might get the don't use EJB , minimise the number of remote calls , or the use Hibernate , use Prepared Statements, use Cache etc etc. Interestingly enough for my current project we had implemented most of the above, tested it locally and seemed to get <4 second times for most pages. And we missed a big problem (hint it's a web based system) , can you guess?
The system responds in under 4 seconds when accessed locally. When accessed through a browser based in New York some (Can you guess?) requests take 20+ seconds. Luckily we did run external tests as soon as the production systems were available so we found this out a couple of months before the system was actually released.
In any case , in hindsight as soon as we knew the problem existed it was relatively easy to guess and verify(YSlow) the problems. We only had few small images so we knew the problem wasn't there.
a. Whats good for development isn't necessarily good for deployment. e.g. We do normally split CSS files , Javascript files and include them separately. Yes a browser will cache these files ( you do add Etags don't you?), but the first request and HTTPS will be slow if the browser must make these connections (normally at the most two at a time). - We used Yahoo and ANT to combine the CSS into one file and the Javascript into another (at build time) drastically reducing the response times
b. GZIP. Creating Gzipped versions of the files and dynamically gzipping all the content (a flag on the webserver) also brought down the times.
Just doing the above brought down the times to under 4 seconds even when the site was accessed remotely on limited bandwidth clients.
Moral of the story : Always test , never guess when it comes to performance tuning.