Simple Machines Community Forum

Customizing SMF => Modifications and Packages => Topic started by: aldo on December 29, 2008, 12:32:23 PM

Title: cURL fetch_web_data
Post by: aldo on December 29, 2008, 12:32:23 PM
Link to Mod (http://custom.simplemachines.org/mods/index.php?mod=1569)

The function in Subs-Package.php fetch_web_data uses fsockopen() in order to obtain the data from the page specified. However not all hosts (usually free hosts) allow the usage of the fsockopen() function. 

So I have fixed that. This modification will allow the fetch_web_data(); function to use cURL (http://php.net/curl) to obtain the data instead. 

Of course not all hosts support cURL, but most hosts with PHP5 do have cURL installed. But don't worry, if your host doesn't support cURL but does fsockopen it will use fsockopen instead, but if thats the case you don't really need this mod.

NOTE: If you have no troubles downloading packages from external servers or getting SMF news data and such you don't need this mod as this is really a fix for servers that support cURL but have problems downloading packages and such.

Changelog:

NOTE: If you have no troubles downloading packages from external servers or getting SMF news data and such you don't need this mod as this is really a fix for servers that support cURL but have problems downloading packages and such.
Title: Re: cURL fetch_web_data
Post by: M-DVD on December 29, 2008, 01:05:40 PM
Interesting.

Yesterday I had this situation  :) (and was about to do so)

Thanks.
Title: Re: cURL fetch_web_data
Post by: aldo on December 29, 2008, 07:24:48 PM
Your very welcome :)
Title: Re: cURL fetch_web_data
Post by: karlbenson on December 29, 2008, 08:00:58 PM
Nice one.

Although in my experience its more likely thats smf's function will work than cUrl.
From what I've read/seen curl is disabled on alot more hosts - especially at the cheap end, than fsockopen is.

However if someone people are able to use it, then its worthwhile. :)
Title: Re: cURL fetch_web_data
Post by: aldo on December 30, 2008, 03:41:48 AM
Yeah but I don't know if thats exactly true... I see a lot of free hosts that have fsockopen disabled, yet cURL enabled :)
Title: Re: cURL fetch_web_data
Post by: Nao 尚 on December 30, 2008, 04:53:50 AM
Is it the season, the weather or something? ;)

This is exactly what I came up with in SMG 1.5 and Aeva 5.x :)
And yes Karl, fsockopen() is definitely more widely supported by safe-mode servers than curl(), but there are also some cases where curl() can be useful.

I feel the SMF team would do well by rewriting the fetch_web_data() function, not only to feature a similar "second choice" like aldo's mod or my implementation of curl() (check out Subs-Aeva.php in Aeva or MGallery-Embed.php in SMG), but also to use an actual preg_match on the response code, instead of the test for " 200 " and " 201 ", because most servers are definitely returning more "valid" codes than these. Basically, anything in the 2xx and 3xx ranges is good.

For instance, DailyMotion thumbnails are a ****** (I'm sure I mentioned this somewhere over at smf-media or in the SMG topic), it sends redirect responses and then when it's good it still sends a 3xx header, so my curl() stuff has to enter a redirect loop, but then again it works, while I don't think fetch_web_data() could retrieve a DailyMotion thumbnail easily.
The regexp I came up with does the job fairly:

if (preg_match('~^HTTP/.+\s+[23][0-9][0-9]\s~i', $response, $reply) != 1)
Title: Re: cURL fetch_web_data
Post by: glennk on January 03, 2009, 04:42:39 AM
What is the purpose of this mod ??
Title: Re: cURL fetch_web_data
Post by: Nao 尚 on January 03, 2009, 05:01:15 AM
To make fetch_web_data work on a wider array of web servers.
If you don't have a Safe Mode or Openbasedir'd server, this is of no concern to you.
If your server is already capable of downloading packages from external servers, then you don't need it at all.
Title: Re: cURL fetch_web_data
Post by: aldo on January 03, 2009, 10:21:56 PM
Thank you Nao, added that to the mod info.
Title: Re: cURL fetch_web_data
Post by: Nao 尚 on January 04, 2009, 04:04:34 AM
Ah yes, good idea, eh ;)
Title: Re: cURL fetch_web_data
Post by: aldo on February 08, 2009, 12:36:37 AM
Sorry peoples... Forgot to update this. It now supports SMF 2.0 RC1.
Title: Re: cURL fetch_web_data
Post by: Mystery? on February 08, 2009, 02:08:39 AM
What does this mod do? I don't know what cURL fetch_web_data.
Title: Re: cURL fetch_web_data
Post by: aldo on February 08, 2009, 02:13:09 AM
www.php.net/curl

It is recommended to use this modification if and when you cannot use fsockopen on your host (You know, like it being disabled) but yet you have cURL enabled, so this can make fetch_web_data() work :)
Title: Re: cURL fetch_web_data
Post by: pjr on April 11, 2009, 12:45:17 PM
How can you tell if fsockopen is disabled? And how can you enable it if you have full admin rights?
Title: Re: cURL fetch_web_data
Post by: Nao 尚 on April 11, 2009, 02:42:08 PM
Quote from: pjr on April 11, 2009, 12:45:17 PM
How can you tell if fsockopen is disabled?
IIRC, automatic package downloading (for SMF updates) won't work, for instance.
Title: Re: cURL fetch_web_data
Post by: aldo on April 15, 2009, 10:36:28 AM
Quote from: Nao 尚 on April 11, 2009, 02:42:08 PM
Quote from: pjr on April 11, 2009, 12:45:17 PM
How can you tell if fsockopen is disabled?
IIRC, automatic package downloading (for SMF updates) won't work, for instance.
Yeah, the news display shouldn't work either, since SMF uses fsockopen to get the JS files which displays the latest version, news, packages, etc.

Not sure about the enabling thing... should be in your php.ini I suppose. But I am no server administrator ;)
Title: Re: cURL fetch_web_data
Post by: flutter on April 14, 2010, 08:39:36 PM
Hoping this thread is still being monitored. 

This mod, would it also cover people downloading from my site?  I'm having issues with people not being able to download at times and I'm using download pro, VBGamer can't work out what's going on.  I'm wondering if this sock and culr thing could be the culprit, excuse my stupidity, this hole thread looks like a foreign language to me
Title: Re: cURL fetch_web_data
Post by: Nao 尚 on April 15, 2010, 09:00:11 AM
It's unrelated. It's for helping your *server* to fetch files from other servers, not to send them to users.
Title: Re: cURL fetch_web_data
Post by: grimm on February 09, 2011, 09:01:52 AM
Update for RC4 please?^^
Title: Re: cURL fetch_web_data
Post by: Cadal on February 11, 2011, 11:11:50 AM
Tried to install via advanced RC1-1 RC1 on test site, no dice. Don't really want to risk enabling fsockopen() on my domain. It's not urgent but an update would be appreciated :)
Title: Re: cURL fetch_web_data
Post by: Monzetti on December 28, 2012, 03:33:26 PM
I need this for 2.0.3!!!

Please!!!!

CURL is open on my server:

cURL support    enabled
cURL Information    7.19.7
Age    3
Features
AsynchDNS    No
Debug    No
GSS-Negotiate    Yes
IDN    Yes
IPv6    Yes
Largefile    Yes
NTLM    Yes
SPNEGO    No
SSL    Yes
SSPI    No
krb4    No
libz    Yes
CharConv    No
Protocols    tftp, ftp, telnet, dict, ldap, ldaps, http, file, https, ftps
Host    x86_64-pc-linux-gnu
SSL Version    OpenSSL/0.9.8k
ZLib Version    1.2.3.3

How I can check fsockopen() ???

phpinfo say: Sockets Support: enabled

Regards,

Monzetti