After the project proposal was accepted, it struck me that doing the benchmarks on a dual-boot machine was probably not a very good idea. So I'm asking my place of work to let me have some resources to do the testing at work, and to do rather more sophisticated tests than I had originally envisaged. Here's what I've asked for:

MSc proposal for benchmarking .NET implementations

Introduction

The project aims to determine whether or not the .NET platform can be used to develop distributed, cross-platform, n-tier, secure applications (Enterprise-class applications) with an acceptable performance. The results of the project will tell C# developers what steps they need to take to write platform agnostic code and what performance implications these may have.

(From the Project Proposal)

To this end, it is proposed to develop some benchmark tests that will help developers determine the sorts of performance differences that might be expected in real-life situations between code running under the Microsoft CLR and the same code running under the mono CLR.

Hypothesis

Real-life industrial-grade code running in a distributed environment is subject to many latencies the most obvious of which are network latencies and database latencies. It is expected that any slight differences that there might be between the two .NET Framework implementations will be dwarfed by these other factors.

Experiment design

Hardware

It was originally envisaged that a single dual-boot machine could be used; however this gives rise to the following problems:

• Network latencies will not be present
• A database instance would be required on both partitions. Since the performance of the same database might vary depending on the operating system platform, any measurements might be measuring nothing but this difference in database performance.
• Similarly, there will be differences in web server performance under different operating systems
• The test application would not be widely or conveniently available to testers.

A better solution would be to use a dedicated database server accessed by two web servers over a LAN. The test program would be accessed by testers using workstations running a web browser, located elsewhere on the LAN. Such a solution will not isolate differences in web server performance, but will address the other issues. It could, in any case, be argued that web server integration was a part of the overall .NET implementation.

It is proposed that the following hardware/software combinations be used for the servers:

• Database server running MySQL or PostgreSQL under either Windows or Linux. The operating system platform is not important.
• Web/Application Server running Apache2 on Windows 2000 (or better) and the Microsoft .NET Framework v1.1.4.x
• Web/Application Server running Apache2 on Linux (Red Hat Fedora or SuSE Linux Pro v9.3 or better) and the mono .NET implementation v1.1.9.x

None of these servers need be particularly powerful. Reasonably modern PCs would be perfectly adequate. It is however important that the two web servers are running on hardware that is largely similar (identical if possible).

Test software

It is proposed to develop a test ASP.NET application based around the IMS specification for a person. A person is probably best thought of as a student in this context. The program would time testers performing the task of entering details for a reasonable number of students. The expected duration of each test would be around 10 minutes.

The application would connect, from its front page, to one of the web servers at random thus ensuring that a more-or-less equal number of tests would be carried out for each configuration. Users would be unaware of which web server was being used.

Each tester would be timed for the overall task.

In addition, the program would provide facilities for uploading student details from an XML file, in bulk. The times taken for bulk uploads would also be compared.

It is hoped that testers could be found from work colleagues. It makes no difference whether the testers are fast or slow at data entry, or that they complete the task any number of times – as long as a sufficient number of tests is carried out with a sufficient number of people.