There are enough comparisons on Internet that compare exactly features.
But it is hard to compare: what to compare? : features, usability, compare for some platform (php, ruby, java, etc) or lightweight, or only commercial CMSs?
An important note: in our company we use only open-source software.

I use my current experience: most wonderful CMSs [enough lightweight, mature, good api] now written in php, although I prefer Java over any other software language:
joomla - user-friendly, good api, and documentation
drupal - super lightweight, good documentation
typo3 - extra features, good documentation

Sorry if I missed your favorite one, but world is big .. and probably very soon I will like something in Ruby on Rails.

>>Which tool did you use to measure duration? (milliseconds per page request)
I have used Jmeter.

Please, find my complete answer in my new post:
http://blogs.helion-prime.com/alexshapovalov/2008/07/02/battle-n2-typo3-joomla-and-drupal-cmss-performance-testing.html

Hope I answered your questions..

>>Surely cache is good as well as better CPU, bandwidth, and cluster when
>>solution is already got all benefits of good software design.
NOOO… I can do more with cache, than you could do with a cluster. I have a site running right now on a single box that would take at least 8 to 10 servers in a cluster if it wasn’t for caching. If caching is designed well, there are no negatives.

Also, purposely benchmarking without opcode caching is fine, as long as you realize nobody in their right mind would run a high volume site without it. It’s that important. So I do agree with Dimitry, in that “No one runs systems like that in real life”, but only for reasonably high volume sites. Your tests may be valid for a site hosted in a shared environment, but for the rest of the world, we use things like opcode caches, and memory resident sessions, and tune the daylights out of MySQL, php and (cough)apache(cough)

>>No one runs systems like that in real life.
>>Every real web server is optimized for performance of the system it runs.
I fully agree with you no one use systems like that in production environment.

But I make tests to test effectiveness of pure systems rather then caching sub-system.
I think that developers should think about effectiveness of their components and
don’t pass care about performance entirely to caching or other technics.

Surely cache is good as well as better CPU, bandwidth, and cluster when
solution is already got all benefits of good software design.

Do you agree?

>>Where the installations weighed in any way to get a roughly similar template output.
>>For example, did you just use the default installs for each CMS,
>>or did you tune each of them to ensure they are outputting the same page (within reason of course).
Sure I made special pages and redesigned template to have similar pages with approximately same sizes for every CMSs.

>>Second question is do Drupal and Typo3 have similar on-the-fly translation engines
>> to compare with? That would be good to see.
I have not worked with these engines. I will think about such testing.

>>Finally, do you have a PHP cache or accelerator running?
>>J!1.5 performs better with one.
No. I used only pure GNU/Linux Debian php5 package with default settings.
Anyway any improvement will provide same advantages for every CMSs.
And without php and internal CMSs caches we have fair comparison.

Hope I answered your questions..