Simple Machines Community Forum

Righty,

Thought that with a few minutes time on my hand I'd make a bit of a post about the recent changes we've made to 2.0 – specifically to the database engine (99% of the forum population may wish to turn away now to save themselves five minutes of boredom).

Right, I'll start with the motivation behind this. There are two real motivations, one of them is security and the other improved abstraction. In my mind security has been the real driver for this and as such I'll explain why. You see, there are two main ways of someone "exploiting" web based software (Or certainly we've had no others ever affect SMF), they are:
1) SQL Injection attacks – where through some means someone is able to run arbitrary SQL commands on your box, revealing passwords, destroying data etc.
2) XSS (Cross Site Scripting) – where through injection of javascript (Or otherwise) a user may be able to steal your session and "take over" your account.

SMF obviously has always protected against both of these and continues to do so. Whilst both can have major implications in general SQL attacks are the ones that, in my mind, are the cause for most concern. So, how did SMF used to protect against these, well – in two ways:
1) Strings are "escaped" (the means by which variables are made safe) before they are inserted into a database. As the normal source for an injection is malicious data coming into the script from a URL/Form etc SMF used to take the approach of "escape everything" whereby anything coming into the script was made safe – and we'd only make it "dangerous" if we needed to do some manipulation of it – and would make it safe again afterwards.
2) The database query function did some additional checks to ensure if someone *did* manage to exploit SMF that they were limited into the extent of the damage they could cause. This is what I'd call the "second line" of defence – or damage limitation.

When developing 2.0 we'd started to become concerned about our approach to the first of these protection mechanisms. SMF 2.0 has generally become cleverer, more complex. We do more things with incoming variables before we put them in the database and in general that meant making them dangerous more and more often. Every time you make them "dangerous" there is a risk that you forget to make them safe again. We decided that this approach was no longer appropriate as it was beginning to increase as opposed to decrease the security risks in 2.0. So – like all good developers – we changed it.

Compuart, the clever chap that he is, came up with a new scheme for doing this security. The problem with our old security model was all the security was on the "gates". Once something was "in" SMF it was assumed to be secure – we wanted to change this. Now, instead of the security being on the gates it stands directly in front of the database itself. You can no longer talk to the database without doing through Bertie the Bouncer (Or whatever he is called). To get something into (Or for that matter out of) the database you need to tell it exactly what it is you're doing – and he'll make it secure for you.

So, now if I want to set my name to 'Grudge – King of the Newts' when I send it to the database I also tell it 'It's a string'. If it's not a string security won't let it if. Also, security – knowing how dangerous strings are – will clean it up before it gives it to the database. The point is that we can do whatever we want to variables within SMF – I can feel with all kinds of dangerous things – but when it comes to feeding it into the database it takes care of security there and then. Now, I'll move on from the silly examples and go into a it of practical stuff – those who vomit at the site of a it of PHP best look away now.

So, here's the old way to update my name: (If you're confused by the $smfFunc stuff see one of my previous blogs on databases)

$smfFunc['db_query']('', "
UPDATE {$db_prefix}members
SET member_name = 'grudge'", __FILE__, __LINE__);


Now, things would look quite different:

$smfFunc['db_query']('', '
UPDATE {db_prefix}members
SET member_name = {string:name}',
array(
'name' => 'grudge',
)
);


So, let's look at things bit by bit:
1)   __FILE__ and __LINE__ are gone. We know use backtracing functions in the case of an error. It doesn't work properly on very old versions of PHP so debugging won't be quite so hot on 4.3.1 but we thought that they are in the absolute majority and the new method makes queries much cleaner.
2)   No single quotes (') within the query. In fact, if I were to put a single quote in the query – anywhere – you'd get a big error message.
3)   Instead the place the string is meant to go has an inject name which references the key of the array passed as the next parameter. When the query is processed every item in the array is reinserted into the query before being passed to the database. Variables are cast and cleaned as required. This means if you inserted an int it will be cast as an int. If you insert a string it will be escaped and surrounded by quotes.

Another examples:

$new_location = "England's Home Land";
$smfFunc['db_query']('', '
UPDATE {db_prefix}members
SET location = {string:new_location}
WHERE id_member IN ({array_int:my_friends})',
array(
'new_location' => $new_location,
'my_friends' => array(3,6,7,3),
)
);


As for the first example, the string $new_location is tested to be a string, escaped, surrounded with quotes and fills the gap of {string:new_location}. Meanwhile the other type array_int looks at the array element of my_friends, checks it's an array, check's all it's contents are ints, then explodes into a list of numbers into the query.

Of course it's much easier to see this by actually viewing the code in real examples but I'm hoping this will help. So, how will this change impact people?

Well – for "normal" users – you won't see a thing. Everything will work as usual – but you should benefit from adding security that will (hopefully) also improve the security of mods as mod authors now can rely on SMF for their database security and not have to think too much about their own protection for SQL issues.

For "mod" users things aren't too much of a problem. They simply remove (I suspect) all calls to escaping functions like addslashes etc that they currently use. They need to rewrite their queries (In addition all inserts should now use a new function db_insert but I've spoken about that before) – but it's not *that* much work as all mods need to be rewritten for 2.0. Besides which – even though Compuart wrote a great tool for converting the main SMF files I still spent about 6 days over Christmas checking, and in many cases rewriting, every query in SMF so no-one can moan to me about the length of the task as I've done it first hand ;)

The main people this will impact is people who integrate SMF deeply within their site. People who use SMF for integrations (For example including SSI.php) and then doing their own queries need to be aware that variables like $_POST and $_GET are no longer escaped by SMF. This means if you previously had this code.

$_POST['test'] = 'test\'s';
require_once('SSI.php');
mysql_query("update smf_members SET member_name = $_POST[test]");


In the past that would have worked. Now that will create a very dangerous security hole. Therefore anyone using SMF for an integration needs to make one of three decisions, either:
1)   Convert their queries to use the SMF query function (Highly recommended)
2)   Escape $_GET, $_POST type variables individually before they are used in their queries
3)   Escape all $_GET, $_POST type variables straight after including SSI.php by doing something like:

require_once('SSI.php');
$_GET = escape__recursive($_GET);
... etc for $_POST


I really wouldn't however recommend doing this. Certainly if you ever use any SMF functions *after* the latter example there is a risk that data will be "double escaped" resulting in lots of slashes appearing in posts, member names etc unintentionally.

There are a few more things about the system but I've covered off the bulk of it here. I very much believe that these changes will benefit everyone, particularly users who should always be looking for better and better security. We should be upgrading our site here this weekend so I'll soon be able to report first hand how easy or otherwise converting an integration is ;)

Grudge

i cant believe i read that, i cant also believe i understood over 50% of it, im curious when smf2 becomes stable will the ssi help pages be updated to work with the new db  strings.

A very good explanatory post.
I see can it taking a bit of time to get used to, but overall it seems very positive.

LMAO, Bertie the Bouncer.

Earl > Bertie :P

are the types the same as php's resource types, or is there a list of them available?

I'll list them out

int
string
text
array_int
array_string
date
float
identifier
raw

In addition {prefix}, {query_see_board}, {query_wanna_see_board} are all magically replaced with $db_prefix, $user_info['query_see_board'], $user_info['query_wanna_see_board'] respectfully.

I appreciate everythings a work in progress.

Is there a way to find out whether a certain mysql command would be supported.
Obviously I want to avoid functions that are mysql only in my current mods, to avoid making things harder when it comes to update for 2.0.

Specifically the command I'm referring to is INSERT SELECT.
http://dev.mysql.com/doc/refman/5.0/en/insert-select.html

It is supported by PostGreSql.

AFAIK INSERT SELECT is generic SQL command that should be supported by most (if not all) database engines.  Heck even MS Access has INSERT SELECT.

Thanks for the Info, Thantos.

Pretty much the same as php resource types in terms of type casting.  (well, it is the same... duhh *slaps self*) :P

Thanks. Thats good to know.
INSERT SELECT has saved me 1000's of queries in a mod i'm working on.

So if I got the stuff right, We will no longer be needing to use the "htmlunspecialchar" functions and other functions to secure the POST data submitted?

Correct.  All data will be cleaned by the query function.

Pretty cool, Are there any commonly used commands(Which are commonly used in MySQL) which wont be supported?

Quote2)   No single quotes (') within the query. In fact, if I were to put a single quote in the query – anywhere – you'd get a big error message.

I forgot about this while making my mods work with the latest SMF beta on my site tonight. I did a work around which I would not suggest users do, but it was a dirty fix to get the mod to install.

I plan on making my mods use the new method that the developers are working on. I like this new method. It is very clean, and infact.. {int:user_id} is much more clear than ' . $user_info['id'] . ' and infact it is nicer.

Yeah,
$smfFunc['db_query'](',',
"SELECT ID_MEMBER, realName FROM {$db_prefix}members WHERE ID_MEMBER = {int:user_id}",
array(
'user_id' => $user_info['id']
)
);
Seems pretty nice and cool.

Quote from: Dragooon on January 12, 2008, 12:02:11 AM
So if I got the stuff right, We will no longer be needing to use the "htmlunspecialchar" functions and other functions to secure the POST data submitted?

Actually, this does nothing to change the way htmlspecialchars works. I'd have loved to do something like this but it would have required a context cleaning function so mod authors and developers still need to decide how to clean the HTML and make sure they make it secure.

As for MySQL commands. SMF 2.0 does not actually *police* commands - so you can use MySQL specific ones in queries - they will just obviously fail in PostgreSQL which would be a shame. I don't however think we will enforce mods being generic

Any changes for getting data out? All the examples were update queries...

Very impressive discussion about coding and I have no idea how you did it!

Dannii,

Selects are generally easier as they have less variables to insert, below is the query for loading a meber:

$request = $smfFunc['db_query']('', '
SELECT mem.*, IFNULL(a.id_attach, 0) AS id_attach, a.filename, a.attachment_type
FROM {db_prefix}members AS mem
LEFT JOIN {db_prefix}attachments AS a ON (a.id_member = {int:id_member})
WHERE mem.id_member = {int:id_member}
LIMIT 1',
array(
'id_member' => $id_member,
)
);


And for interest, here's an example of inserting data

$result = $smfFunc['db_insert']('ignore',
'{db_prefix}sessions',
array('session_id' => 'string', 'data' => 'string', 'last_update' => 'int'),
array($session_id, $data, time()),
array('session_id')
);


Grudge, the function will return the same data structure as before (or reference rather)?

Oooohh, $smfFunc['db_insert'], that's new!

Yes, the way for retrieving data is the same - only the way for doing a query is different. The other main change is that db_insert_id needs to be passed the table name and primary column to allow it to work on PostgreSQL, i.e:

$topicID = $smfFunc['db_insert_id']('{db_prefix}topics', 'id_topic');


Okay thanks.
Can I ask what we should use instead of mysql_num_rows()?

$smfFunc['db_num_rows']()

For the most part you can take any mysql_* or db_* function and it'll follow the $smfFunc['db_*'] format.  I can't think of any exceptions off the top of my head.

Great post! Very well written and relatively easy to understand.

Mod authors and people who plan to integrate SMF 2.0 via SSI etc. should definitely check this out when SMF 2.0 comes out.

Single quotes are not allowed in queries....... Okay... But are double quotes allowed?
I've converted some stuff to the latest version and it uses a lot of queries with strings, so I just left them as they were, just replacing single quotes with double quotes, and double quotes with single quotes.

Something like,

$smfFunc['db_query']('', 'SELECT something FROM {db_prefix}hello WHERE stuff="' . $yo . '"');

I think it does work, though...

Well you can simply change it to
$smfFunc['db_query']('', 'SELECT something FROM {db_prefix}hello WHERE stuff={string:yo}', array('yo' => $yo)); and then not have to worry about it.

Not sure what the results of that would be Nao. MySQL normally expects strings to be enclosed in single quotes (In terms of mysql_real_escape_string etc) so you may end up getting odd results. Personally I'd just do as Thantos suggests...

Looking good, I'm gonna have to update my mods soon :D
As for $smfFunc['db_insert'], how long has that been there for? I never noticed it :o

Quote from: Daniel15 on January 14, 2008, 01:55:46 AM
As for $smfFunc['db_insert'], how long has that been there for? I never noticed it :o

Since SMF 2.0 Alpha, if I recall correctly. It's been there for quite a while now, in any case. ;)

That has been there since the early days of 2.0.  Back when the abstraction stuff started.

Oh... My bad :P
I've been using SMF 2.0 since the Alpha, but never went through the abstraction code in great detail...

Quote from: Grudge on January 13, 2008, 06:46:13 PM
Not sure what the results of that would be Nao. MySQL normally expects strings to be enclosed in single quotes (In terms of mysql_real_escape_string etc) so you may end up getting odd results.

Hmm that's curious indeed. Never had any problems since last week at least... (When the new format was introduced.)
And I do check most of my pages regularly.
I'll try to change these to the new format anyway, progressively at least.
I will also have to make sure my strings are not escaped before they're passed to db_query... Or maybe I should just use mysql_query for these, I don't know.

I haven't personally yet did any code with the new query style yet, but from the looks of it it looks pretty nice and clean in my opinion. :) I love that it makes it more likely to be secure (lowers chances of an oversight). I've always been concerned myself about my mods having some hole in them. ;)

Yeah... It's great, indeed. Takes a while to get used to, but it really makes the programmer feel better after a while ;)
And when you need speed, you can use the "security_disable" flag, or just rely on mysql_query (if you're using MySQL ;))... This is really good policy.

Apart from that $smfFunc['db_query']('you_bet') and then $smcFunc['db_query']('my_ass'), of course..........

thanks.

Quote from: Thantos on January 11, 2008, 12:40:59 PM
Earl > Bertie :P

Umm I just noticed the phpBB's mascot is named Bertie.  http://www.phpbb.com/shop/

See again, more proof that Earl > Bertie :P

Converted my custom stuff a few days back, the one that got me was {$db_prefix} to {db_prefix}, looked for ages and couldn't see where my error was coming from. As with most things leave it for an hour or so then come back and you spot it straight away.

Now it's just a shame I can't do the actual upgrade for a little while due to the amount of traffic I'm getting at the mo.

Ben,

I have to say it's going to be a bit scary when you upgrade your site to SMF 2.0. I think it will be the first proper comparison of 2.0 vs 1.1 performance as your site probably runs up to 99% capacity so if SMF 2.0 is only 5% slower than 1.1 it's really going to show.

Will certainly be interesting :D

Where's your confidence, man?!? :P

What would you do if it turns out there is a (significant) performance regression between 1.1x and 2.0?

Try and identify the problem areas and improve them.

Even if the only solution is to remove a feature or two? In that case, I think I'll root for some unexpected slowness associated with Karma... O:)

On a more serious note, is there anything in particular which makes you (or Grudge, since he's the one who seemed a bit concerned) worry about 2.0's performance?

Most likely he is just wondering about a really big smf site using the new database abstraction method in 2.0. Bens big site will show greatly the performance gain or loss.

sorry for the offtopic, just wanted to ask Ben if they (the supporters) are really intend to put money and buy the club from the investors?

and ontopic: I spotted that demonoid.com started a forum and use SMF for it. It's a very dynamic one, >180k posts with more 100k users in less then 3 month (started 14 nov). Using 1.1.4 so when the time came for upgrading to 2.0 they also will have an idea about speed and performance progress or regress

Quote from: rommul on February 01, 2008, 10:16:16 PM
and ontopic: I spotted that demonoid.com started a forum and use SMF for it. It's a very dynamic one, >180k posts with more 100k users in less then 3 month (started 14 nov). Using 1.1.4 so when the time came for upgrading to 2.0 they also will have an idea about speed and performance progress or regress

Very fast current speed apparently... Would be curious to know if they're on a classic Apache setup. Or maybe they just have a lot of servers running the thing ;)

thank you

sorry guys I am very new with this .. if i ask can you please do pictures (img) explain. I will be so glad i am still learning or any1 can help me? I read all but make me confuse because I didn't understand:(
regards

Unless you're writing a mod, you shouldn't need to worry about any of this.

So SMF 2.0 should have around the same requirements as 1.1.4 correct? Cause you talked about older versions of PHP...

Quote from: lordtron on March 03, 2008, 05:04:33 AM
So SMF 2.0 should have around the same requirements as 1.1.4 correct? Cause you talked about older versions of PHP...

We have not changed the PHP nor the MySQL version requirements.  Everytime we've needed to use a PHP 5 function we've found a way to write a version of it ourselves for those using PHP 4.

How many people are still using PHP4?

Quote from: Dannii on March 03, 2008, 09:26:25 AM
How many people are still using PHP4?

Enough to warrant the extra effort.  Such support doesn't have a negative effect on those using PHP 5.

Yeah not all hosting companies have updated to PHP 5 yet, which sucks. Cause there is so much software out there that uses PHP 5 and more is coming out. But thank god you guys did not goto PHP 5 yet. Cause I am the lucky one that has a hosting company that has not updated just yet. So thanks

Theres also a substantial amount of software that has not updated for php5 yet either though, and some are bigger name softwares. 

Sadly, i don't see that changing anytime soon

Will 2.0 have a query debug feature like 1.0 Beta had? It was a nice benchmarking tool. Or maybe it was in YSE... can't remember, been a while...

Set $db_show_debug to true in settings.php and it'll show up.

I still see many softwares upgrading to php5. Its a bit more efficent than php4?

http://php.net/manual/en/migration5.php

Quote from: Thantos on March 03, 2008, 08:12:12 PM
Set $db_show_debug to true in settings.php and it'll show up.

Thanks :). Didn't know this was made hidden.

Well glad there is a lot involved in 2.0 Can't wait for it to be released. Hopefully I can run it with my host.

GoPHP5.org (http://gophp5.org/)

Birger :)

Question, will these changes mean that I no longer need to use addslashes() before updateSettings()?

Dannii, that is correct.

Thanks. That's a fairly easy change to make thankfully.

SMF 2.0 doesnt understand my chat integration. How do i change this:

$chatrequest = db_query("
            SELECT COUNT(*) AS numb
            FROM  fc_connections
            WHERE userid IS NOT NULL", __FILE__, __LINE__);
         list ($chatcount) = mysql_fetch_row($chatrequest);   
         mysql_free_result($chatrequest);
     
if ( $chatcount == "1" ) {
$singularplural2 = "";
} else {
$singularplural2 = "ä";
}

$chatrequest = $smcFunc['db_query']('', '
            SELECT COUNT(*) AS numb
            FROM  fc_connections
            WHERE userid IS NOT NULL');
         list ($chatcount) = $smcFunc['db_fetch_row']($chatrequest);   
         $smcFunc['db_free_result']($chatrequest);
     
if ( $chatcount == "1" ) {
$singularplural2 = "";
} else {
$singularplural2 = "ä";
}


"Fatal error: Function name must be a string in "

Owdy, do you have $smcFunc globaled?

?? I have no idea what it is.

I have another one

// array for the member ID's
$team = array();

// get all members of group 1 (administrators)
$group = 2;
$res = db_query("SELECT ID_MEMBER
                FROM members
                WHERE ID_GROUP = $group ", __FILE__, __LINE__);
while ($row = mysql_fetch_assoc($res))
        $team[] = $row['ID_MEMBER'];
mysql_free_result($res);


global $smcFunc;

$chatrequest = $smcFunc['db_query']('', '
            SELECT COUNT(*) AS numb
            FROM  fc_connections
            WHERE userid IS NOT NULL');
         list ($chatcount) = $smcFunc['db_fetch_row']($chatrequest);   
         $smcFunc['db_free_result']($chatrequest);
     
if ( $chatcount == "1" ) {
$singularplural2 = "";
} else {
$singularplural2 = "ä";
}


Try that (same code as what thantos posted, but with $smfFunc globaled.

For the second query, you could try something like this:

// array for the member ID's
$team = array();

// get all members of group 1 (administrators)
$group = 2;
$res = $smcFunc['db_query']('', 'SELECT id_member
                FROM {db_prefix}members
WHERE ID_GROUP = {int:group}',
array(
'group' => $group,
)
);
while ($row = $smcFunc['db_fetch_assoc']($res))
        $team[] = $row['id_member'];
$smcFunc['db_free_result']($res);

Edited: Fixed mistake causing it not to work.

First one worked :)

Second one:

Fatal error: Function name must be a string in /var/www/fs1/0/public_html/foorumi/Sources/Load.php(1857) : eval()'d code on line 45

Please start a topic in the SMF Coding Discussion (http://www.simplemachines.org/community/index.php?board=60.0) board.  Thanks

Okay. Sorry T for hijacking this topic.