Need ideas on how to best optimize forum performance during surge periods

Started by vii, March 04, 2018, 02:16:19 PM

Previous topic - Next topic

vii

(by 'optimize forum performance' I mean optimize via software changes, so this is coding-related)

So I'm a bit lost on exactly how to optimize forum functionality. Most days it runs fine, but a few times a year (give or take) we have huge "surge periods" where the user activity will quadruple or more, and the activity focus is generally on a few threads, but the entire forum as a whole usually sees more activity. The problem is that the forum has grown over the past few years, and these surge days cripple the server to the point where refreshing a thread can take anywhere from a couple minutes to completely stalling out.

FYI:


  • Forum is running 1.1.21 (I know it's an older version, but there are too many custom changes to the forum to upgrade right now)
  • There are around 73 boards (it's a lot, more than would probably be recommended, but this can't be changed right now)
  • There over 5 million messages
  • Surge periods usually see activity go from around 250 - 300 regular users to anywhere from 600 - 1000 with substantial activity from most of them
  • Upgrading server hardware isn't an option right now
  • Caching (level 1) and InnoDB already done a long time ago


Here are some of my ideas that I plan on implementing:

1. Modifying BoardIndex's SQL query to stop pulling all data except the board/category info for Archived boards, which are read only and since there are around 24 or 25 archived boards, I'm hoping that performance could be helped if the query skips those boards when pulling last post / member / log_boards data.

2. Disabling log_mark_read table queries during surge periods, at least for the boards being hit with massive activity spikes

3. Disabling Search functionality during surge periods (at least for most users - it is already restricted to registered users)


-

So basically I'm asking: Will the 3 things above help? What else can I do?

LiroyvH

Caching could perhaps do with an improvement. What cacher(s) are you using?
Have you tuned InnoDB for better performance? Just switching to InnoDB isn't usually some magical fix (although it can help solve some problems), it often requires a bit o' tweaking.

Of course, if this all comes down to a lack of resources in the end (eg: not enough cpu/memory/IO capacity) then there's not much else you can do other than throwing more hardware at it - but first things first.
((U + C + I)x(10 − S)) / 20xAx1 / (1 − sin(F / 10))
President/CEO of Simple Machines - Server Manager
Please do not PM for support - anything else is usually OK.

vii

Quote from: CoreISP on March 04, 2018, 02:23:36 PM
Caching could perhaps do with an improvement. What cacher(s) are you using?
Have you tuned InnoDB for better performance? Just switching to InnoDB isn't usually some magical fix (although it can help solve some problems), it often requires a bit o' tweaking.

Of course, if this all comes down to a lack of resources in the end (eg: not enough cpu/memory/IO capacity) then there's not much else you can do other than throwing more hardware at it - but first things first.

Yup InnoDB and caching (APC) are already used. I don't think there is anything more to be done outside of tweaking the forum code.

I guess part of what I'm wondering is maybe, what are the top 5 or top 10 most "expensive" forum operations performed on a regular basis when a user visits a board/page? I'd like to create a forum setting that I can enable during surge days that would disable some functionality.

drewactual

nothing, and i mean nothing is more effective than OPCache for taking the load off a server.

it is a cache function outside of SMF- a server height install.  the basic concept of it's function is that it runs a script (php file) one time, and stores that function in RAM- not calling it and processing it all over again each and every time the file is called.  My server went from peak traffic of CPU usage @90+% to under 10% in the same peak traffic periods.   if you don't run your own server or if you're on a shared server, you obviously have to ask the managers to install it.  it's very popular, so you aren't going to have an issue... a matter of fact, it is likely already on your server and you just don't know it.  I'd wager 2:1 this is the case, actually.

to implement it's function, you're going to want to add this to your php.ini, your user.ini, or your htaccess depending on how your server is arranged... it may be better if you present this to your server manager and have them do so as they'll know where to park it for you:

zend_extension=opcache.so
opcache.memory_consumption=264
opcache.max_accelerated_files=4000
opcache.revalidate_freq=60
opcache.fast_shutdown=1
opcache.enable_cli=1
opcache.use_cwd=1
opcache.load_comments=1
opcache.save_comments=1
opcache.revalidate_path=1
opcache.validate_timestamps=1


you'll be astounded the speed this will add and the sheer drop in load it will cause.

be aware:  the only drawback to using OPCache is that you will have to 'invalidate' files when you change them.  this can be done by shutdown/restart, but that is clumsy and impact ALL files in the cache.. instead, what i rec you do is make a file and call it something only you can recall, and hide it from robots in your robots.txt... it's drafted by the maker of OPCache and a well respected coder of PHP.. .one page, here is the code:


<?php
define
('THOUSAND_SEPARATOR',true);
if (!
extension_loaded('Zend OPcache')) {
    echo 
'<div style="background-color: #F2DEDE; color: #B94A48; padding: 1em;">You do not have the Zend OPcache extension loaded, sample data is being shown instead.</div>';
    require 
'data-sample.php';
}
class 
OpCacheDataModel
{
    private 
$_configuration;
    private 
$_status;
    private 
$_d3Scripts = array();
    public function 
__construct()
    {
        
$this->_configuration opcache_get_configuration();
        
$this->_status opcache_get_status();
    }
    public function 
getPageTitle()
    {
        return 
'PHP ' phpversion() . " with OpCache {$this->_configuration['version']['version']}";
    }
    public function 
getStatusDataRows()
    {
        
$rows = array();
        foreach (
$this->_status as $key => $value) {
            if (
$key === 'scripts') {
                continue;
            }
            if (
is_array($value)) {
                foreach (
$value as $k => $v) {
                    if (
$v === false) {
                        
$value 'false';
                    }
                    if (
$v === true) {
                        
$value 'true';
                    }
                    if (
$k === 'used_memory' || $k === 'free_memory' || $k === 'wasted_memory') {
                        
$v $this->_size_for_humans(
                            
$v
                        
);
                    }
                    if (
$k === 'current_wasted_percentage' || $k === 'opcache_hit_rate') {
                        
$v number_format(
                                
$v,
                                
2
                            
) . '%';
                    }
                    if (
$k === 'blacklist_miss_ratio') {
                        
$v number_format($v2) . '%';
                    }
                    if (
$k === 'start_time' || $k === 'last_restart_time') {
                        
$v = ($v date(DATE_RFC822$v) : 'never');
                    }
                    if (
THOUSAND_SEPARATOR === true && is_int($v)) {
                        
$v number_format($v);
                    }
                    
$rows[] = "<tr><th>$k</th><td>$v</td></tr>\n";
                }
                continue;
            }
            if (
$value === false) {
                
$value 'false';
            }
            if (
$value === true) {
                
$value 'true';
            }
            
$rows[] = "<tr><th>$key</th><td>$value</td></tr>\n";
        }
        return 
implode("\n"$rows);
    }
    public function 
getConfigDataRows()
    {
        
$rows = array();
        foreach (
$this->_configuration['directives'] as $key => $value) {
            if (
$value === false) {
                
$value 'false';
            }
            if (
$value === true) {
                
$value 'true';
            }
            if (
$key == 'opcache.memory_consumption') {
                
$value $this->_size_for_humans($value);
            }
            
$rows[] = "<tr><th>$key</th><td>$value</td></tr>\n";
        }
        return 
implode("\n"$rows);
    }
    public function 
getScriptStatusRows()
    {
        foreach (
$this->_status['scripts'] as $key => $data) {
            
$dirs[dirname($key)][basename($key)] = $data;
            
$this->_arrayPset($this->_d3Scripts$key, array(
                
'name' => basename($key),
                
'size' => $data['memory_consumption'],
            ));
        }
        
asort($dirs);
        
$basename '';
        while (
true) {
            if (
count($this->_d3Scripts) !=1) break;
            
$basename .= DIRECTORY_SEPARATOR key($this->_d3Scripts);
            
$this->_d3Scripts reset($this->_d3Scripts);
        }
        
$this->_d3Scripts $this->_processPartition($this->_d3Scripts$basename);
        
$id 1;
        
$rows = array();
        foreach (
$dirs as $dir => $files) {
            
$count count($files);
            
$file_plural $count 's' null;
            
$m 0;
            foreach (
$files as $file => $data) {
                
$m += $data["memory_consumption"];
            }
            
$m $this->_size_for_humans($m);
            if (
$count 1) {
                
$rows[] = '<tr>';
                
$rows[] = "<th class=\"clickable\" id=\"head-{$id}\" colspan=\"3\" onclick=\"toggleVisible('#head-{$id}', '#row-{$id}')\">{$dir} ({$count} file{$file_plural}{$m})</th>";
                
$rows[] = '</tr>';
            }
            foreach (
$files as $file => $data) {
                
$rows[] = "<tr id=\"row-{$id}\">";
                
$rows[] = "<td>" $this->_format_value($data["hits"]) . "</td>";
                
$rows[] = "<td>" $this->_size_for_humans($data["memory_consumption"]) . "</td>";
                
$rows[] = $count "<td>{$file}</td>" "<td>{$dir}/{$file}</td>";
                
$rows[] = '</tr>';
            }
            ++
$id;
        }
        return 
implode("\n"$rows);
    }
    public function 
getScriptStatusCount()
    {
        return 
count($this->_status["scripts"]);
    }
    public function 
getGraphDataSetJson()
    {
        
$dataset = array();
        
$dataset['memory'] = array(
            
$this->_status['memory_usage']['used_memory'],
            
$this->_status['memory_usage']['free_memory'],
            
$this->_status['memory_usage']['wasted_memory'],
        );
        
$dataset['keys'] = array(
            
$this->_status['opcache_statistics']['num_cached_keys'],
            
$this->_status['opcache_statistics']['max_cached_keys'] - $this->_status['opcache_statistics']['num_cached_keys'],
            
0
        
);
        
$dataset['hits'] = array(
            
$this->_status['opcache_statistics']['misses'],
            
$this->_status['opcache_statistics']['hits'],
            
0,
        );
        
$dataset['restarts'] = array(
            
$this->_status['opcache_statistics']['oom_restarts'],
            
$this->_status['opcache_statistics']['manual_restarts'],
            
$this->_status['opcache_statistics']['hash_restarts'],
        );
        if (
THOUSAND_SEPARATOR === true) {
            
$dataset['TSEP'] = 1;
        } else {
            
$dataset['TSEP'] = 0;
        }
        return 
json_encode($dataset);
    }
    public function 
getHumanUsedMemory()
    {
        return 
$this->_size_for_humans($this->getUsedMemory());
    }
    public function 
getHumanFreeMemory()
    {
        return 
$this->_size_for_humans($this->getFreeMemory());
    }
    public function 
getHumanWastedMemory()
    {
        return 
$this->_size_for_humans($this->getWastedMemory());
    }
    public function 
getUsedMemory()
    {
        return 
$this->_status['memory_usage']['used_memory'];
    }
    public function 
getFreeMemory()
    {
        return 
$this->_status['memory_usage']['free_memory'];
    }
    public function 
getWastedMemory()
    {
        return 
$this->_status['memory_usage']['wasted_memory'];
    }
    public function 
getWastedMemoryPercentage()
    {
        return 
number_format($this->_status['memory_usage']['current_wasted_percentage'], 2);
    }
    public function 
getD3Scripts()
    {
        return 
$this->_d3Scripts;
    }
    private function 
_processPartition($value$name null)
    {
        if (
array_key_exists('size'$value)) {
            return 
$value;
        }
        
$array = array('name' => $name,'children' => array());
        foreach (
$value as $k => $v) {
            
$array['children'][] = $this->_processPartition($v$k);
        }
        return 
$array;
    }
    private function 
_format_value($value)
    {
        if (
THOUSAND_SEPARATOR === true) {
            return 
number_format($value);
        } else {
            return 
$value;
        }
    }
    private function 
_size_for_humans($bytes)
    {
        if (
$bytes 1048576) {
            return 
sprintf('%.2f&nbsp;MB'$bytes 1048576);
        } else {
            if (
$bytes 1024) {
                return 
sprintf('%.2f&nbsp;kB'$bytes 1024);
            } else {
                return 
sprintf('%d&nbsp;bytes'$bytes);
            }
        }
    }
    
// Borrowed from Laravel
    
private function _arrayPset(&$array$key$value)
    {
        if (
is_null($key)) return $array $value;
        
$keys explode(DIRECTORY_SEPARATORltrim($keyDIRECTORY_SEPARATOR));
        while (
count($keys) > 1) {
            
$key array_shift($keys);
            if ( ! isset(
$array[$key]) || ! is_array($array[$key])) {
                
$array[$key] = array();
            }
            
$array =& $array[$key];
        }
        
$array[array_shift($keys)] = $value;
        return 
$array;
    }
}
$dataModel = new OpCacheDataModel();
?>

<!DOCTYPE html>
<meta charset="utf-8">
<html>
<head>
    <style>
        body {
            font-family: "Helvetica Neue",Helvetica,Arial,sans-serif;
            margin: 0;
            padding: 0;
        }
        #container {
            width: 1024px;
            margin: auto;
            position: relative;
        }
        h1 {
            padding: 10px 0;
        }
        table {
            border-collapse: collapse;
        }
        tbody tr:nth-child(even) {
            background-color: #eee;
        }
        p.capitalize {
            text-transform: capitalize;
        }
        .tabs {
            position: relative;
            float: left;
            width: 60%;
        }
        .tab {
            float: left;
        }
        .tab label {
            background: #eee;
            padding: 10px 12px;
            border: 1px solid #ccc;
            margin-left: -1px;
            position: relative;
            left: 1px;
        }
        .tab [type=radio] {
            display: none;
        }
        .tab th, .tab td {
            padding: 8px 12px;
        }
        .content {
            position: absolute;
            top: 28px;
            left: 0;
            background: white;
            border: 1px solid #ccc;
            height: 450px;
            width: 100%;
            overflow: auto;
        }
        .content table {
            width: 100%;
        }
        .content th, .tab:nth-child(3) td {
            text-align: left;
        }
        .content td {
            text-align: right;
        }
        .clickable {
            cursor: pointer;
        }
        [type=radio]:checked ~ label {
            background: white;
            border-bottom: 1px solid white;
            z-index: 2;
        }
        [type=radio]:checked ~ label ~ .content {
            z-index: 1;
        }
        #graph {
            float: right;
            width: 40%;
            position: relative;
        }
        #graph > form {
            position: absolute;
            right: 60px;
            top: -20px;
        }
        #graph > svg {
            position: absolute;
            top: 0;
            right: 0;
        }
        #stats {
            position: absolute;
            right: 125px;
            top: 145px;
        }
        #stats th, #stats td {
            padding: 6px 10px;
            font-size: 0.8em;
        }
        #partition {
            position: absolute;
            width: 100%;
            height: 100%;
            z-index: 10;
            top: 0;
            left: 0;
            background: #ddd;
            display: none;
        }
        #close-partition {
            display: none;
            position: absolute;
            z-index: 20;
            right: 15px;
            top: 15px;
            background: #f9373d;
            color: #fff;
            padding: 12px 15px;
        }
        #close-partition:hover {
            background: #D32F33;
            cursor: pointer;
        }
        #partition rect {
            stroke: #fff;
            fill: #aaa;
            fill-opacity: 1;
        }
        #partition rect.parent {
            cursor: pointer;
            fill: steelblue;
        }
        #partition text {
            pointer-events: none;
        }
        label {
            cursor: pointer;
        }
    </style>
    <script src="//cdnjs.cloudflare.com/ajax/libs/d3/3.0.1/d3.v3.min.js"></script>
    <script src="//cdnjs.cloudflare.com/ajax/libs/jquery/1.11.0/jquery.min.js"></script>
    <script>
        var hidden = {};
        function toggleVisible(head, row) {
            if (!hidden[row]) {
                d3.selectAll(row).transition().style('display', 'none');
                hidden[row] = true;
                d3.select(head).transition().style('color', '#ccc');
            } else {
                d3.selectAll(row).transition().style('display');
                hidden[row] = false;
                d3.select(head).transition().style('color', '#000');
            }
        }
    </script>
    <title><?php echo $dataModel->getPageTitle(); ?></title>
</head>
<body>
    <div id="container">
        <h1><?php echo $dataModel->getPageTitle(); ?></h1>
        <div class="tabs">
            <div class="tab">
                <input type="radio" id="tab-status" name="tab-group-1" checked>
                <label for="tab-status">Status</label>
                <div class="content">
                    <table>
                        <?php echo $dataModel->getStatusDataRows(); ?>
                    </table>
                </div>
            </div>
            <div class="tab">
                <input type="radio" id="tab-config" name="tab-group-1">
                <label for="tab-config">Configuration</label>
                <div class="content">
                    <table>
                        <?php echo $dataModel->getConfigDataRows(); ?>
                    </table>
                </div>
            </div>
            <div class="tab">
                <input type="radio" id="tab-scripts" name="tab-group-1">
                <label for="tab-scripts">Scripts (<?php echo $dataModel->getScriptStatusCount(); ?>)</label>
                <div class="content">
                    <table style="font-size:0.8em;">
                        <tr>
                            <th width="10%">Hits</th>
                            <th width="20%">Memory</th>
                            <th width="70%">Path</th>
                        </tr>
                        <?php echo $dataModel->getScriptStatusRows(); ?>
                    </table>
                </div>
            </div>
            <div class="tab">
                <input type="radio" id="tab-visualise" name="tab-group-1">
                <label for="tab-visualise">Visualise Partition</label>
                <div class="content"></div>
            </div>
        </div>
        <div id="graph">
            <form>
                <label><input type="radio" name="dataset" value="memory" checked> Memory</label>
                <label><input type="radio" name="dataset" value="keys"> Keys</label>
                <label><input type="radio" name="dataset" value="hits"> Hits</label>
                <label><input type="radio" name="dataset" value="restarts"> Restarts</label>
            </form>

            <div id="stats"></div>
        </div>
    </div>

    <div id="close-partition">&#10006; Close Visualisation</div>
    <div id="partition"></div>

    <script>
        var dataset = <?php echo $dataModel->getGraphDataSetJson(); ?>;
        var width = 400,
            height = 400,
            radius = Math.min(width, height) / 2,
            colours = ['#B41F1F', '#1FB437', '#ff7f0e'];
        d3.scale.customColours = function() {
            return d3.scale.ordinal().range(colours);
        };
        var colour = d3.scale.customColours();
        var pie = d3.layout.pie().sort(null);
        var arc = d3.svg.arc().innerRadius(radius - 20).outerRadius(radius - 50);
        var svg = d3.select("#graph").append("svg")
                    .attr("width", width)
                    .attr("height", height)
                    .append("g")
                    .attr("transform", "translate(" + width / 2 + "," + height / 2 + ")");
        var path = svg.selectAll("path")
                      .data(pie(dataset.memory))
                      .enter().append("path")
                      .attr("fill", function(d, i) { return colour(i); })
                      .attr("d", arc)
                      .each(function(d) { this._current = d; }); // store the initial values
        d3.selectAll("input").on("change", change);
        set_text("memory");
        function set_text(t) {
            if (t === "memory") {
                d3.select("#stats").html(
                    "<table><tr><th style='background:#B41F1F;'>Used</th><td><?php echo $dataModel->getHumanUsedMemory()?></td></tr>"+
                    "<tr><th style='background:#1FB437;'>Free</th><td><?php echo $dataModel->getHumanFreeMemory()?></td></tr>"+
                    "<tr><th style='background:#ff7f0e;' rowspan=\"2\">Wasted</th><td><?php echo $dataModel->getHumanWastedMemory()?></td></tr>"+
                    "<tr><td><?php echo $dataModel->getWastedMemoryPercentage()?>%</td></tr></table>"
                );
            } else if (t === "keys") {
                d3.select("#stats").html(
                    "<table><tr><th style='background:#B41F1F;'>Cached keys</th><td>"+format_value(dataset[t][0])+"</td></tr>"+
                    "<tr><th style='background:#1FB437;'>Free Keys</th><td>"+format_value(dataset[t][1])+"</td></tr></table>"
                );
            } else if (t === "hits") {
                d3.select("#stats").html(
                    "<table><tr><th style='background:#B41F1F;'>Misses</th><td>"+format_value(dataset[t][0])+"</td></tr>"+
                    "<tr><th style='background:#1FB437;'>Cache Hits</th><td>"+format_value(dataset[t][1])+"</td></tr></table>"
                );
            } else if (t === "restarts") {
                d3.select("#stats").html(
                    "<table><tr><th style='background:#B41F1F;'>Memory</th><td>"+dataset[t][0]+"</td></tr>"+
                    "<tr><th style='background:#1FB437;'>Manual</th><td>"+dataset[t][1]+"</td></tr>"+
                    "<tr><th style='background:#ff7f0e;'>Keys</th><td>"+dataset[t][2]+"</td></tr></table>"
                );
            }
        }
        function change() {
            // Filter out any zero values to see if there is anything left
            var remove_zero_values = dataset[this.value].filter(function(value) {
                return value > 0;
            });
            // Skip if the value is undefined for some reason
            if (typeof dataset[this.value] !== 'undefined' && remove_zero_values.length > 0) {
                $('#graph').find('> svg').show();
                path = path.data(pie(dataset[this.value])); // update the data
                path.transition().duration(750).attrTween("d", arcTween); // redraw the arcs
            // Hide the graph if we can't draw it correctly, not ideal but this works
            } else {
                $('#graph').find('> svg').hide();
            }
            set_text(this.value);
        }
        function arcTween(a) {
            var i = d3.interpolate(this._current, a);
            this._current = i(0);
            return function(t) {
                return arc(i(t));
            };
        }
        function size_for_humans(bytes) {
            if (bytes > 1048576) {
                return (bytes/1048576).toFixed(2) + ' MB';
            } else if (bytes > 1024) {
                return (bytes/1024).toFixed(2) + ' KB';
            } else return bytes + ' bytes';
        }
        function format_value(value) {
            if (dataset["TSEP"] == 1) {
                return value.toString().replace(/\B(?=(\d{3})+(?!\d))/g, ",");
            } else {
                return value;
            }
        }
        var w = window.innerWidth,
            h = window.innerHeight,
            x = d3.scale.linear().range([0, w]),
            y = d3.scale.linear().range([0, h]);
        var vis = d3.select("#partition")
                    .style("width", w + "px")
                    .style("height", h + "px")
                    .append("svg:svg")
                    .attr("width", w)
                    .attr("height", h);
        var partition = d3.layout.partition()
                .value(function(d) { return d.size; });
        root = JSON.parse('<?php echo json_encode($dataModel->getD3Scripts()); ?>');
        var g = vis.selectAll("g")
                   .data(partition.nodes(root))
                   .enter().append("svg:g")
                   .attr("transform", function(d) { return "translate(" + x(d.y) + "," + y(d.x) + ")"; })
                   .on("click", click);
        var kx = w / root.dx,
                ky = h / 1;
        g.append("svg:rect")
         .attr("width", root.dy * kx)
         .attr("height", function(d) { return d.dx * ky; })
         .attr("class", function(d) { return d.children ? "parent" : "child"; });
        g.append("svg:text")
         .attr("transform", transform)
         .attr("dy", ".35em")
         .style("opacity", function(d) { return d.dx * ky > 12 ? 1 : 0; })
         .text(function(d) { return d.name; })
        d3.select(window)
          .on("click", function() { click(root); })
        function click(d) {
            if (!d.children) return;
            kx = (d.y ? w - 40 : w) / (1 - d.y);
            ky = h / d.dx;
            x.domain([d.y, 1]).range([d.y ? 40 : 0, w]);
            y.domain([d.x, d.x + d.dx]);
            var t = g.transition()
                     .duration(d3.event.altKey ? 7500 : 750)
                     .attr("transform", function(d) { return "translate(" + x(d.y) + "," + y(d.x) + ")"; });
            t.select("rect")
             .attr("width", d.dy * kx)
             .attr("height", function(d) { return d.dx * ky; });
            t.select("text")
             .attr("transform", transform)
             .style("opacity", function(d) { return d.dx * ky > 12 ? 1 : 0; });
            d3.event.stopPropagation();
        }
        function transform(d) {
            return "translate(8," + d.dx * ky / 2 + ")";
        }
        $(document).ready(function() {
            function handleVisualisationToggle(close) {
                $('#partition, #close-partition').fadeToggle();
                // Is the visualisation being closed? If so show the status tab again
                if (close) {
                    $('#tab-visualise').removeAttr('checked');
                    $('#tab-status').trigger('click');
                }
            }
            $('label[for="tab-visualise"], #close-partition').on('click', function() {
                handleVisualisationToggle(($(this).attr('id') === 'close-partition'));
            });
            $(document).keyup(function(e) {
                if (e.keyCode == 27) handleVisualisationToggle(true);
            });
        });</script>
</body>
</html>



vii

Thanks drewactual, I hadn't considered that. I will look into it.

Are you familiar with the settings "opcache.revalidate_freq" and "opcache.validate_timestamps" (I got them from this link). It seems like it can already auto-check the source files for modifications and then re-cache if the file changed. Wouldn't they make the extra code from your post unnecessary?

aegersz

plesse describe your hosting configuration  - do you run a virtual private server or are you on a shared host ?

it may be possible to automatically be given more resources on demand but such systems are probably not cheap.

SMF can display how long each page takes to build so that can be a starting point.

there is a mod that i run thst displays CPU load at the bottom of the page.

do you have restrictions on the search feature ?

we need to see where the bottleneck is first so we know where your slowdown is occurring.

my own system doesn't ever get over 1% CPU but hardly anybody uses it.

do you run on Windows or Linux ? what versions ?

do you run under VMware ?

I have spent a lot of my time making mainframes perform so i am interested in helping; please describe your hardware environment as best as you can.
The configuration of my Linux VPS (SMF 2.0 with 160+ mods & some assorted manual tweaks) can be found here and notes on my mods can be found here (warning: those links will take you to a drug related forum). My (House) music DJ dedication page is here

drewactual

Quote from: Virginiaz on March 06, 2018, 02:11:44 PM
Thanks drewactual, I hadn't considered that. I will look into it.

Are you familiar with the settings "opcache.revalidate_freq" and "opcache.validate_timestamps" (I got them from this link). It seems like it can already auto-check the source files for modifications and then re-cache if the file changed. Wouldn't they make the extra code from your post unnecessary?

you don't usually have to define those, and a matter of fact i considered leaving them out of my post... revalidate-timestamps can slow things down as it looks at the file in RAM (the cache) and asks "is the one on the hard drive newer than the one in RAM?" before handing over the output to the requesting browser.  the other one, reval freq, does just that.. it tells OPCache to dump the files at a specified interval...  you don't require it either.  they're both a good idea in a development environment, though.   

what i would consider altering is the memory set aside for this... at first i used 512mb and 17k files... and then watching it (by using the page i offered for managing) realized 128mb and 4k files was more than adequate.  the percentage of files in RAM that aren't being often used are set to dump out of the cache when they reach 10%- i didn't provide that statement either- but, only because the default is 10% and is fine with my version of SMF (and thinking it will be the same with yours).

what @aegersz offers is also pertinent.  if you're not running linux (some flavor) or Apache, this is all a moot discussion. also, you'll want to know how much resources you'll want to dedicate to this which will be contingent on how much is available.

Kindred

you really should consider upgrading.

2.0.x has load balancing and several other things to reduce load.

Additionally, there are several security fixes in 2.0.x that were made after 1.1.x stopped receiving updates.

Finally, 2.0.15 supports php7 which may help your issue as well.
Слaва
Украинi

Please do not PM, IM or Email me with support questions.  You will get better and faster responses in the support boards.  Thank you.

"Loki is not evil, although he is certainly not a force for good. Loki is... complicated."

aegersz

Quote from: Kindred on March 06, 2018, 04:11:22 PM

Finally, 2.0.15 supports php7 which may help your issue as well.

why would that be; does php 7 offer considerable performance benefits ?
The configuration of my Linux VPS (SMF 2.0 with 160+ mods & some assorted manual tweaks) can be found here and notes on my mods can be found here (warning: those links will take you to a drug related forum). My (House) music DJ dedication page is here

Kindred

I think so... but I am not enough of a server side expert to say for sure.
Слaва
Украинi

Please do not PM, IM or Email me with support questions.  You will get better and faster responses in the support boards.  Thank you.

"Loki is not evil, although he is certainly not a force for good. Loki is... complicated."

vii

aegersz Unfortunately I do not have access to most of that data. I'm a Moderator on the forum that is also making a series of upgrades since the owner is just too busy to keep up for the time being. I do know that he has already made a lot of effort in the past to maximize performance through hardware upgrades, addons like APC and other changes. At this point I'm trying to make changes to the forum code itself to enable better performance during surge periods.

Per your question:

1. It's a dedicated server
2. Guests are prohibited from using the Search feature
3. Not using VM

The forum runs smoothly most days of the year, but we have maybe a dozen or more days a year where we see spikes in activity so large that the forum can take forever to reload pages/make new posts. I know Search is a problem, so I intend to disable that during those periods. But a lot of it seems to just be increased user activity in viewing/posting on threads. This is part of why I wish to maybe disable log_mark_read / log_topic queries as well, assuming it'd be helpful in exchange for users temporarily not being brought to where they last left off.


Kindred - Unfortunately there are way too many custom changes (our own, not necessarily 3rd party mods) to do it right now. I figured that once 2.1 is released, we could move to that. It'll take a lot of effort to get things the way we want it once we upgrade, so it's preferable to wait for a version that provides maximum benefits.

This is why my request was to identify something like a top-10 list of the most resource-intensive operations the forum does, so I may see what I can selectively enable/disable during surge periods where user activity shoots through the stratosphere. In a way I guess I'm trying to create something similar to 2.x's load balancing, but perhaps more extensive than what it does.

Kindred

thing is...   1.1.x is basically at end of life. Support is minimal at this time because of that....  2.0.x fixed so many things including load and throttling that were never backported to 1.1.x

The most expensive query is  "show new posts"
Слaва
Украинi

Please do not PM, IM or Email me with support questions.  You will get better and faster responses in the support boards.  Thank you.

"Loki is not evil, although he is certainly not a force for good. Loki is... complicated."

Advertisement: