In response to criticism
of its Linux versus Windows NT benchmark comparisons commissioned by Microsoft,
Mindcraft is withholding the publication
of its second Linux and Windows NT Server benchmark results pending an
Open Benchmark
Invitation.
Mindcraft believes that the best way to continue to show our name as a
credible source of information is to have an Open Benchmark. Therefore, we
welcome the opportunity to perform a benchmark of Linux and Windows NT
Server that is open to the best experts in the Linux community. Mindcraft
will participate in this benchmark at its own expense.
Linux Today applauds this invitation by Mindcraft. Free Software and
Linux stand much more to gain than they could ever lose in a truly
fair and objective comparison between Linux and Windows NT. In fact, it is
quite likely that any deficiencies in the software would be quickly
addressed by a swarm of Free Software developers.
But Samba development team member, Jeremy Allison, takes issue with one
benchmark requirement stipulated by Mindcraft, in particular that:
... the tests will use Windows 9x clients (these can be
in any mix including all clients using the same Windows 9x OS). The same
client set up will be used for both Linux and Windows NT Server.
In a letter to Mindcraft's Bruce Weiner, Jeremy stipulates some requirements of his own:
In order to participate, I must insist that Windows NT
clients are *also* benchmarked.
I feel that it is very much in the interest of consumers
to be able to choose between clients, and surely Microsoft
would be interested in publishing the results of their
flagship workstation operating system, running against
their flagship server, using their flagship filesystem.
I insist that we also have NT clients running against
NT server and Samba servers, using an NTFS filesystem
on NT, and an equivalent secure filesystem (ext2fs) on
Linux added to the mix.
There are no problems running Win9x clients against NT
using whatever file system you wish as well, but not
also showing the NT client data is *extremely* disingenuous,
as I'm sure you already know.
Your original benchmark was funded by, and on behalf of
Microsoft, run in their labs (as you have now revealed).
This new benchmark is different and is a joint effort
between Microsoft and the Open Source community to publish
meaningful numbers to allow consumer choice. In order to
be able to use our names on this we must have some input
into the benchmarking specifications.
If you refuse to consider this important addition to the
matrix I will have no option but to publicly refuse to
participate and publish our own Win9x client to NT or Samba
server, and WinNT client to NT or Samba server benchmark
matrix, and let our own credibility be assessed by the public.
Linux Today endorses the view of Jeremy Allison that a truly 'Open'
benchmark must allow both sides an equal opportunity to stipulate
testing conditions -- particularly in this case where the Allison proposal
does not restrict the Mindcraft benchmark platform in any way but merely
adds an additional platform against which to test.
The Jeremy Allison proposal should, in fairness to truth, be accepted and
then... let the benchmark proceed!
