General Community > Scripting Help

Random slowness in SMF's OB callback & performance questions

(1/2) > >>

nend:
Hi,

I have been getting random slowness in the OB Callback. Sometimes the lag is around a minute, this slowness comes and goes. Most of the time it isn't present.

I have been tracking this slowness and have narrowed it down the Pretty Url's. At first I was thinking maybe it was caused by the database connection in Pretty Url's. If anyone that has browsed the source there knows it has one connection to load all the topic URL's. So I basically disabled the callback but the lag is still there.

I am starting to think it is the RegEx routine in there, maybe my fault or the mods fault since I can never keep my hands off code. Basically allot of stuff that was once in there that I believed it didn't need I ripped out or I edited and made better or maybe worse. :whistle:

I do have other concerns about the Pretty's rewrite. One of them is do you really need to save the boards in the settings table? would it be cheaper to do a join on this in the topics filter? I don't think it will be cheaper but it is a question to ask, cheapest IMHO is tearing out the board from the topic URL.

I do have questions SMF related, like the board index query, does it really need to load all that information? Would it increase performance if it didn't?

However back on topic. Here is my PrettyUrls-Filters.php, it is somewhat similar to the mod, but different. You will see comments marking out some test code I put in there, the cache system that is, it isn't that good but may be beneficial with a true cache system.


--- Code: ---<?php
//   NEnd's custom Pretty URLs OB rewrite
// Based on Version 1.0 PrettyUrls-Filters.php

if (!defined('SMF'))
die('Hacking attempt...');

// Rewrite the buffer with Pretty URLs!
function pretty_rewrite_buffer($buffer)
{
global $boardurl, $context, $modSettings, $smcFunc;

// Remove the script tags now
$context['pretty']['scriptID'] = 0;
$context['pretty']['scripts'] = array();
$buffer = preg_replace_callback('~<script.+?</script>~s', 'pretty_scripts_remove', $buffer);

// Find all URLs in the buffer
$context['pretty']['search_patterns'][] = '~(<a[^>]+href=|<link[^>]+href=|<form[^>]+?action=)(\"[^\"#]+|\'[^\'#]+)~';
$urls = array();
$context['pretty']['cached_urls'] = array();
foreach ($context['pretty']['search_patterns'] as $pattern) {
preg_match_all($pattern, $buffer, $matches, PREG_PATTERN_ORDER);
foreach ($matches[2] as $match)
{
// Rip out everything that shouldn't be cached
$match = preg_replace(array('~^[\"\']|PHPSESSID=[^;]+|(se)?sc=[^;]+|' . $context['session_var'] . '=[^;]+~', '~\"~', '~;+|=;~', '~\?;~', '~\?$|;$|=$~'), array('', '%22', ';', '?', ''), $match);

// Absolutise relative URLs
if (!preg_match('~^[a-zA-Z]+:|^#|@~', $match) && SMF != 'SSI')
$match = $boardurl . '/' . $match;

if (substr($match,0,7) == 'mailto:')
continue;
if (substr($match,0,10) == 'javascript')
continue;

// if (($data = cache_get_data('pretty'.$match)) != null) {
// $context['pretty']['cached_urls'][$match] = $data;
// } else {
$urls[$match] = array('url' => $match);
// }
}
}

// Procede only if there are actually URLs in the page
if (count($urls) != 0)
{
// Run each filter callback function on each URL
$filter_callbacks = unserialize($modSettings['pretty_filter_callbacks']);
foreach ($filter_callbacks as $callback)
$urls = call_user_func($callback, $urls);

// Fill the cached URLs array
foreach ($urls as $url_id => $url)
{
if (!isset($url['replacement'])) {$url['replacement'] = $url['url'];}
$url['replacement'] = str_replace("\x12", '\'', $url['replacement']);
$url['replacement'] = preg_replace(array('~\"~', '~;+|=;~', '~\?;~', '~\?$|;$|=$~'), array('%22', ';', '?', ''), $url['replacement']);
$context['pretty']['cached_urls'][$url_id] = $url['replacement'];
// cache_put_data('pretty'.$url_id, $url['replacement'], 1200);
}
}

// Put the URLs back into the buffer
$context['pretty']['replace_patterns'][] = '~(<a[^>]+href=|<link[^>]+href=|<form[^>]+?action=)(\"[^\"]+\"|\'[^\']+\')~';
foreach ($context['pretty']['replace_patterns'] as $pattern)
$buffer = preg_replace_callback($pattern, 'pretty_buffer_callback', $buffer);

// Restore the script tags
if ($context['pretty']['scriptID'] > 0)
$buffer = preg_replace_callback("~\x14([0-9]+)\x14~", 'pretty_scripts_restore', $buffer);

// Return the changed buffer.
return $buffer;
}

// Remove and save script tags
function pretty_scripts_remove($match)
{
global $context;

$context['pretty']['scriptID']++;
$context['pretty']['scripts'][$context['pretty']['scriptID']] = $match[0];
return "\x14" . $context['pretty']['scriptID'] . "\x14";
}

// A callback function to replace the buffer's URLs with their cached URLs
function pretty_buffer_callback($matches)
{
global $boardurl, $context;

// Is this URL in an attribute, and so will need new quotes?
$addQuotes = preg_match('~^[\"\']~', $matches[2]);

// Remove those annoying quotes
$matches[2] = preg_replace('~^[\"\']|[\"\']$~', '', $matches[2]);

// Store the parts of the URL that won't be cached so they can be inserted later
preg_match('~PHPSESSID=[^;#&]+~', $matches[2], $PHPSESSID);
preg_match('~(se)?sc=[^;#]+~', $matches[2], $sesc);
preg_match('~' . $context['session_var'] . '=[^;#]+~', $matches[2], $session_var);
preg_match('~#.*~', $matches[2], $fragment);

// Rip out everything that won't have been cached
$url_id = preg_replace(array('~PHPSESSID=[^;#]+|(se)?sc=[^;#]+|' . $context['session_var'] . '=[^;#]+|#.*$~', '~\"~', '~;+|=;~', '~\?;~', '~\?$|;$|=$~'), array('', '%22', ';', '?', ''), $matches[2]);

// Absolutise relative URLs
if (!preg_match('~^[a-zA-Z]+:|@~', $url_id) && !($url_id == '' && isset($fragment[0])) && SMF != 'SSI')
$url_id = $boardurl . '/' . $url_id;

// Stitch everything back together, clean it up and return
$replacement = isset($context['pretty']['cached_urls'][$url_id]) ? $context['pretty']['cached_urls'][$url_id] : $url_id;
$replacement .= (strpos($replacement, '?') === false ? '?' : ';') . (isset($PHPSESSID[0]) ? $PHPSESSID[0] : '') . ';' . (isset($sesc[0]) ? $sesc[0] : '') . (isset($session_var[0]) ? $session_var[0] : '') . (isset($fragment[0]) ? $fragment[0] : '');
$replacement = preg_replace(array('~;+|=;~', '~\?;~', '~\?#|;#|=#~', '~\?$|&amp;$|;$|#$|=$~'), array(';', '?', '#', ''), $replacement);
$replacement = str_replace('index.php', '', $replacement);// Remove index.php, who needs it.
return $matches[1] . ($addQuotes ? '"' : '') . $replacement . ($addQuotes ? '"' : '');
}

// Put the script tags back
function pretty_scripts_restore($match)
{
global $context;

return $context['pretty']['scripts'][(int) $match[1]];
}

// Filter miscellaneous action urls
function pretty_urls_actions_filter($urls)
{
global $boardurl, $context, $modSettings, $scripturl;

$skip_actions = array();
if (isset($modSettings['pretty_skipactions']))
$skip_actions = explode(",",$modSettings['pretty_skipactions']);

$pattern = '`' . $scripturl . '(.*)action=([^;]+)`S';
$replacement = $boardurl . '/$2/$1';
foreach ($urls as $url_id => $url)
if (!isset($url['replacement']))
if (preg_match($pattern, $url['url'], $matches))
{
// Don't rewrite these actions
if (!empty($skip_actions))
if (in_array($matches[2],$skip_actions))
continue;

if (in_array($matches[2], $context['pretty']['action_array']))
$urls[$url_id]['replacement'] = preg_replace($pattern, $replacement, $url['url']);
}
return $urls;
}

// Filter topic urls
function pretty_urls_topic_filter($urls)
{
global $context, $modSettings, $scripturl, $smcFunc, $sourcedir;

$pattern = '`' . $scripturl . '(.*[?;&])topic=([.a-zA-Z0-9]+)(.*)`S';
$query_data = array();
foreach ($urls as $url_id => $url)
{
// Get the topic data ready to query the database with
if (!isset($url['replacement']))
if (preg_match($pattern, $url['url'], $matches))
{
if (strpos($matches[2], '.') !== false)
list ($urls[$url_id]['topic_id'], $urls[$url_id]['start']) = explode('.', $matches[2]);
else
{
$urls[$url_id]['topic_id'] = $matches[2];
$urls[$url_id]['start'] = '0';
}
$urls[$url_id]['topic_id'] = (int) $urls[$url_id]['topic_id'];
$urls[$url_id]['match1'] = $matches[1];
$urls[$url_id]['match3'] = $matches[3];
$query_data[] = $urls[$url_id]['topic_id'];
}
}

// Query the database with these topic IDs
if (count($query_data) != 0)
{
// Look for existing topic URLs
$query_data = array_keys(array_flip($query_data));
$topicData = array();
$unpretty_topics = array();

$query = $smcFunc['db_query']('', '
SELECT t.id_topic, t.id_board, p.pretty_url
FROM {db_prefix}topics AS t
LEFT JOIN {db_prefix}pretty_topic_urls AS p ON (t.id_topic = p.id_topic)
WHERE t.id_topic IN ({array_int:topic_ids})',
array('topic_ids' => $query_data)
);

while ($row = $smcFunc['db_fetch_assoc']($query)) {
if (isset($row['pretty_url'])) {
$topicData[$row['id_topic']] = array(
'pretty_board' => (isset($context['pretty']['board_urls'][$row['id_board']]) ? $context['pretty']['board_urls'][$row['id_board']] : $row['id_board']),
'pretty_url' => $row['pretty_url'],
);
} else {
$unpretty_topics[] = $row['id_topic'];
}
}

$smcFunc['db_free_result']($query);
$context['pretty']['db_count']++;

// Generate new topic URLs if required *This should be its own function stupid having it here, just confusing to others.
if (count($unpretty_topics) != 0) {
require_once($sourcedir . '/Subs-PrettyUrls.php');

// Get the topic subjects
$new_topics = array();
$new_urls = array();
$query_check = array();
$existing_urls = array();
$add_new = array();

$query = $smcFunc['db_query']('', '
SELECT t.id_topic, t.id_board, m.subject
FROM {db_prefix}topics AS t
INNER JOIN {db_prefix}messages AS m ON (m.id_msg = t.id_first_msg)
WHERE t.id_topic IN ({array_int:topic_ids})',
array('topic_ids' => $unpretty_topics)
);

while ($row = $smcFunc['db_fetch_assoc']($query))
$new_topics[] = array(
'id_topic' => $row['id_topic'],
'id_board' => $row['id_board'],
'subject' => $row['subject'],
);
$smcFunc['db_free_result']($query);
$context['pretty']['db_count']++;

// Generate URLs for each new topic
foreach ($new_topics as $row)
{
$pretty_text = substr(pretty_generate_url($row['subject']), 0, 80);
// A topic in the recycle board doesn't deserve a proper URL
if (($modSettings['recycle_enable'] && $row['id_board'] == $modSettings['recycle_board']) || $pretty_text == '')
// Use 'tID_TOPIC' as a pretty url
$pretty_text = 't' . $row['id_topic'];
// No duplicates and no numerical URLs - that would just confuse everyone!
if (in_array($pretty_text, $new_urls) || is_numeric($pretty_text))
// Add suffix '-ID_TOPIC' to the pretty url
$pretty_text = substr($pretty_text, 0, 70) . '-' . $row['id_topic'];
$query_check[] = $pretty_text;
$new_urls[$row['id_topic']] = $pretty_text;
}

// Find any duplicates of existing URLs
$query = $smcFunc['db_query']('', '
SELECT pretty_url
FROM {db_prefix}pretty_topic_urls
WHERE pretty_url IN ({array_string:new_urls})',
array('new_urls' => $query_check)
);
while ($row = $smcFunc['db_fetch_assoc']($query)) {
$existing_urls[] = $row['pretty_url'];
}
$smcFunc['db_free_result']($query);
$context['pretty']['db_count']++;

// Finalise the new URLs ...
foreach ($new_topics as $row) {
$pretty_text = $new_urls[$row['id_topic']];
// Check if the new URL is already in use
if (in_array($pretty_text, $existing_urls))
$pretty_text = substr($pretty_text, 0, 70) . '-' . $row['id_topic'];
$add_new[] = array($row['id_topic'], $pretty_text);
// Add to the original array of topic URLs
$topicData[$row['id_topic']] = array(
'pretty_board' => (isset($context['pretty']['board_urls'][$row['id_board']]) ? $context['pretty']['board_urls'][$row['id_board']] : $row['id_board']),
'pretty_url' => $pretty_text,
);
}
// ... and add them to the database!
$smcFunc['db_insert']('',
'{db_prefix}pretty_topic_urls',
array('id_topic' => 'int', 'pretty_url' => 'string'),
$add_new,
array()
);
$context['pretty']['db_count']++;
}

// Build the replacement URLs
foreach ($urls as $url_id => $url) {
if (isset($url['topic_id']) && isset($topicData[$url['topic_id']])) {
$start = $url['start'] != '0' || is_numeric($topicData[$url['topic_id']]['pretty_url']) ? $url['start'] . '/' : '';
$urls[$url_id]['replacement'] = $modSettings['pretty_root_url'] . '/' . $topicData[$url['topic_id']]['pretty_board'] . '/' . $topicData[$url['topic_id']]['pretty_url'] . '/' . $start . $url['match1'] . $url['match3'];
}
}
}
return $urls;
}

// Filter board urls
function pretty_urls_board_filter($urls)
{
global $scripturl, $modSettings, $context;

$pattern = '`' . $scripturl . '(.*[?;&])board=([.0-9]+)(.*)`S';
foreach ($urls as $url_id => $url)
// Split out the board URLs and replace them
if (!isset($url['replacement']))
if (preg_match($pattern, $url['url'], $matches))
{
if (strpos($matches[2], '.') !== false)
list ($board_id, $start) = explode('.', $matches[2]);
else
{
$board_id = $matches[2];
$start = '0';
}
$board_id = (int) $board_id;
$start = $start != '0' ? $start . '/' : '';
$urls[$url_id]['replacement'] = $modSettings['pretty_root_url'] . '/' . (isset($context['pretty']['board_urls'][$board_id]) ? $context['pretty']['board_urls'][$board_id] : $board_id) . '/' . $start . $matches[1] . $matches[3];
}
return $urls;
}

// Filter profiles
function pretty_profiles_filter($urls)
{
global $boardurl, $scripturl, $smcFunc, $context;

$pattern = '`' . $scripturl . '(.*)action=profile;u=([0-9]+)(.*)`S';
$query_data = array();
foreach ($urls as $url_id => $url)
{
// Get the profile data ready to query the database with
if (!isset($url['replacement']))
if (preg_match($pattern, $url['url'], $matches))
{
$urls[$url_id]['profile_id'] = (int) $matches[2];
$urls[$url_id]['match1'] = $matches[1];
$urls[$url_id]['match3'] = $matches[3];
$query_data[] = $urls[$url_id]['profile_id'];
}
}

// Query the database with these profile IDs
if (count($query_data) != 0)
{
$query = $smcFunc['db_query']('', '
SELECT id_member, member_name
FROM {db_prefix}members
WHERE id_member IN ({array_int:member_ids})',
array('member_ids' => $query_data));

$memberNames = array();
while ($row = $smcFunc['db_fetch_assoc']($query))
$memberNames[$row['id_member']] = rawurlencode($row['member_name']);
$smcFunc['db_free_result']($query);
$context['pretty']['db_count']++;

// Build the replacement URLs
foreach ($urls as $url_id => $url)
if (isset($url['profile_id']))
if (strpos($memberNames[$url['profile_id']], '%2F') !== false)
$urls[$url_id]['replacement'] = $boardurl . '/profile/' . $url['match1'] . 'user=' . $memberNames[$url['profile_id']] . $url['match3'];
else
$urls[$url_id]['replacement'] = $boardurl . '/profile/' . $memberNames[$url['profile_id']] . '/' . $url['match1'] . $url['match3'];
}
return $urls;
}

// Filter Aeva Media URLs
function pretty_aeva_filter($urls)
{
global $boardurl;

$pattern = array(
'~.*[?;&]action=media;sa=media;in=([0-9]+);(thumba?|preview)(.*)~S',
'~.*[?;&]action=media;sa=(album|item|media);in=([0-9]+)(.*)~S',
'~.*[?;&]action=media(.*)~S',
);
$replacement = array(
$boardurl . '/media/$2/$1/?$3',
$boardurl . '/media/$1/$2/?$3',
$boardurl . '/media/?$1',
);
foreach ($urls as $url_id => $url)
if (!isset($url['replacement']) && strpos($url['url'], 'action=media') !== false)
$urls[$url_id]['replacement'] = preg_replace($pattern, $replacement, $url['url']);
return $urls;
}

?>

--- End code ---

nend:
Started adding debugging in there and notice it isn't Pretty Urls. It is something in between of the execution of the footer and the ob callback.

nend:
Found the lag, but don't know how to fix it.

It seems the lag is when the callback is called and doesn't seem code related because there is no code in the script to edit there, PHP handles that portion of the buffer getting sent to the callback. Maybe there is some setting causing the lag.

Well this has turned into a support question.  :-\

Arantor:
Have you tried tracing execution with a decent profiler? Xdebug and WinGrind tend to work pretty well for me nailing down such things.

nend:
Found the cause finally, didn't realize there was more ob callbacks.


--- Code: --- if (isset($modSettings['integrate_buffer']))
$buffers = array_merge(explode(',', $modSettings['integrate_buffer']), $buffers);

if (!empty($buffers))
foreach ($buffers as $function)
{
$function = trim($function);
$call = strpos($function, '::') !== false ? explode('::', $function) : $function;

// Is it valid?
if (is_callable($call))
ob_start($call);
}

--- End code ---

These are set by mods, so off to look in the database to see what mods are doing their own ob callback and see what is up with there code that is causing the problem. Will keep you updated as to what I find.  ;D

Navigation

[0] Message Index

[#] Next page