Benchmarking Standards

[Joshua Dickerson]

As has been said before, such as on ForumInsider.com there is really no good way to benchmark a forum. Since SMF is such a high-end software (as opposed to low like the filesystem) it has too many variables to get an accurate account all of the time. There are huge differences between one person refreshing a page as fast as they can and two and more people refreshing a page with equal speed. There are a lot of differences with each server configuration.

The only way to get the most accurate and best benchmark would be to create a large database of random usage. Random being exactly what this forum would be constituted as. Use the best configured server we can create. You can only really test one thing at a time unless you can come up with a way to do it in the background. I would suggest opening up a socket to do that. Even then, you are running a socket so now there is another variable. Memory usage would have to be checked via Apache and MySQL processes. There is no other way. Not sure the best way to check CPU usage. Speed is the only accurate information you can possible expect to receive from this benchmark.

The other thing is to benchmark certain areas of the forum at a time. It would be MUCH easier to compare the topic creation function to another forum's topic creation function. That is, if it were organized. This would be the best way to do it, but most users aren't going to understand a lot of what is going on. I say it would be best because you can't make arguments that this software has more features than the next software.

To develop a standard, you need to develop a standard platform. Certain things need to be taken in to account. We aren't going to get a reputable benchmark by doing it by ourself. If you wanted to benchmark IPB against SMF, you should get IPB to add their best server configuration for the server that is being planned to use. Also, the best configuration for their software. Same with SMF. That is the best way to compare. Although, comparative benchmarks are only good for market share. The developers should be more concerned with the speed, memory usage, cpu usage, and bandwidth of the software without worrying about competition.

After one benchmark is made, the tools used should be shared. All of the variables should be included if the benchmark is to be published.

[Joshua Dickerson]

I just took a few minutes because I was bored to write up something. I know everyone has their own method and code for benchmarking. I find that most of those don't include things like load, processes, memory usage, operating system, apache version, php version, modules included, etc. Just a start and I don't want to go ahead an add anything more if it shouldn't be used.

Code Select Expand

<?php

/* NOTE:
Computers cannot accurately calculate
micro time. Thus, these results are little
more than an estimate. Use this multiple
times to get a more accurate result.

*/

// Number of iterations for the for loop
define('NUM_ITERATIONS', 10000);

// Save the raw data (recommended)
define('SAVE_RAW_DATA', true);

// The following are all depended upon SAVE_RAW_DATA. If it is set to false, no need to edit these.
// Location to save the raw data including the file name (only needed if SAVE_RAW_DATA is on)
define('RAW_DATA_LOC', './my_first_benchmark.csv');

// How often to check load in seconds (Every minute recommended. 0 to disable)
define('LOAD_CHECK_INTERVAL', 60);
// end

// This function is where you will place your bechmark code.
function benchmark_code()
{

}

### Don't edit past this point ###

// Define some variables we are going to use
$time_check = 0;
$raw_data = array();

if (SAVE_RAW_DATA)
{
$raw_data['start']['load'] = get_load();
$raw_data['start']['start'] = microtime();
}

for ($i = NUM_ITERATIONS; $i > 0; $i--)
{
if (SAVE_RAW_DATA)
$raw_data[$i]['start'] = microtime();
else
echo microtime(), ', ';

benchmark_code();

if (SAVE_RAW_DATA)
$raw_data[$i]['end'] = microtime();
else
echo microtime(), ', ';

if (LOAD_CHECK_INTERVAL !== 0 && time() - $time_check > LOAD_CHECK_INTERVAL)
{
if (SAVE_RAW_DATA)
$raw_data[$i]['load'] = get_load();
else
echo get_load(), ', ';
$time_check = time();
}
if (!SAVE_RAW_DATA)
echo "\n";
}

if (SAVE_RAW_DATA)
{
$raw_data['end']['load'] = get_load();
$raw_data['end']['end'] = microtime();
}

// We've collected all of the data we need. Time to save it to a file.
if (SAVE_RAW_DATA)
{
// There are a lot of better ways to do this, but we want to use something that all PHP versions have.
$fh = fopen (RAW_DATA_LOC, 'w');

// Write the headers
fwrite($fh, '### start_time, end_time, load ###' . "\n");

// Add the raw data
foreach ($raw_data as $pos => $values)
{
fwrite($fh, $values['start'] . ',' . $values['end'] . (isset($values['load']) ? ',\'' . serialize($values['load']) . '\'' ) . "\n");
unset($raw_data['pos']);
}
unset($raw_data);

// Now grab the system variables
fwrite($fh, str_repeat('#', 60));

// See ya later.
fclose ($fh);
}

function get_load()
{

}


?>