cURL fetch_web_data

Started by aldo, December 29, 2008, 12:32:23 PM

Previous topic - Next topic

aldo

Link to Mod

The function in Subs-Package.php fetch_web_data uses fsockopen() in order to obtain the data from the page specified. However not all hosts (usually free hosts) allow the usage of the fsockopen() function. 

So I have fixed that. This modification will allow the fetch_web_data(); function to use cURL to obtain the data instead. 

Of course not all hosts support cURL, but most hosts with PHP5 do have cURL installed. But don't worry, if your host doesn't support cURL but does fsockopen it will use fsockopen instead, but if thats the case you don't really need this mod.

NOTE: If you have no troubles downloading packages from external servers or getting SMF news data and such you don't need this mod as this is really a fix for servers that support cURL but have problems downloading packages and such.

Changelog:

  • 1.1 - Added support for SMF 2.0 RC1
  • 1.0 - Initially added

NOTE: If you have no troubles downloading packages from external servers or getting SMF news data and such you don't need this mod as this is really a fix for servers that support cURL but have problems downloading packages and such.

M-DVD

#1
Interesting.

Yesterday I had this situation  :) (and was about to do so)

Thanks.

aldo


karlbenson

Nice one.

Although in my experience its more likely thats smf's function will work than cUrl.
From what I've read/seen curl is disabled on alot more hosts - especially at the cheap end, than fsockopen is.

However if someone people are able to use it, then its worthwhile. :)

aldo

Yeah but I don't know if thats exactly true... I see a lot of free hosts that have fsockopen disabled, yet cURL enabled :)

Nao 尚

Is it the season, the weather or something? ;)

This is exactly what I came up with in SMG 1.5 and Aeva 5.x :)
And yes Karl, fsockopen() is definitely more widely supported by safe-mode servers than curl(), but there are also some cases where curl() can be useful.

I feel the SMF team would do well by rewriting the fetch_web_data() function, not only to feature a similar "second choice" like aldo's mod or my implementation of curl() (check out Subs-Aeva.php in Aeva or MGallery-Embed.php in SMG), but also to use an actual preg_match on the response code, instead of the test for " 200 " and " 201 ", because most servers are definitely returning more "valid" codes than these. Basically, anything in the 2xx and 3xx ranges is good.

For instance, DailyMotion thumbnails are a ****** (I'm sure I mentioned this somewhere over at smf-media or in the SMG topic), it sends redirect responses and then when it's good it still sends a 3xx header, so my curl() stuff has to enter a redirect loop, but then again it works, while I don't think fetch_web_data() could retrieve a DailyMotion thumbnail easily.
The regexp I came up with does the job fairly:

if (preg_match('~^HTTP/.+\s+[23][0-9][0-9]\s~i', $response, $reply) != 1)
I will not make any deals with you. I've resigned. I will not be pushed, filed, stamped, indexed, briefed, debriefed or numbered.

Aeva Media rocks your life.

glennk

What is the purpose of this mod ??

Nao 尚

To make fetch_web_data work on a wider array of web servers.
If you don't have a Safe Mode or Openbasedir'd server, this is of no concern to you.
If your server is already capable of downloading packages from external servers, then you don't need it at all.
I will not make any deals with you. I've resigned. I will not be pushed, filed, stamped, indexed, briefed, debriefed or numbered.

Aeva Media rocks your life.

aldo

Thank you Nao, added that to the mod info.

Nao 尚

I will not make any deals with you. I've resigned. I will not be pushed, filed, stamped, indexed, briefed, debriefed or numbered.

Aeva Media rocks your life.

aldo

Sorry peoples... Forgot to update this. It now supports SMF 2.0 RC1.

Mystery?

What does this mod do? I don't know what cURL fetch_web_data.
Join my NEW website that I just created and be a big part of it (July 30, 2010): http://www.gamerstatic.net

aldo

www.php.net/curl

It is recommended to use this modification if and when you cannot use fsockopen on your host (You know, like it being disabled) but yet you have cURL enabled, so this can make fetch_web_data() work :)

pjr

How can you tell if fsockopen is disabled? And how can you enable it if you have full admin rights?

Nao 尚

Quote from: pjr on April 11, 2009, 12:45:17 PM
How can you tell if fsockopen is disabled?
IIRC, automatic package downloading (for SMF updates) won't work, for instance.
I will not make any deals with you. I've resigned. I will not be pushed, filed, stamped, indexed, briefed, debriefed or numbered.

Aeva Media rocks your life.

aldo

Quote from: Nao 尚 on April 11, 2009, 02:42:08 PM
Quote from: pjr on April 11, 2009, 12:45:17 PM
How can you tell if fsockopen is disabled?
IIRC, automatic package downloading (for SMF updates) won't work, for instance.
Yeah, the news display shouldn't work either, since SMF uses fsockopen to get the JS files which displays the latest version, news, packages, etc.

Not sure about the enabling thing... should be in your php.ini I suppose. But I am no server administrator ;)

flutter

Hoping this thread is still being monitored. 

This mod, would it also cover people downloading from my site?  I'm having issues with people not being able to download at times and I'm using download pro, VBGamer can't work out what's going on.  I'm wondering if this sock and culr thing could be the culprit, excuse my stupidity, this hole thread looks like a foreign language to me

Nao 尚

It's unrelated. It's for helping your *server* to fetch files from other servers, not to send them to users.
I will not make any deals with you. I've resigned. I will not be pushed, filed, stamped, indexed, briefed, debriefed or numbered.

Aeva Media rocks your life.

grimm

"Closing your eyes forces you to see the darkness inside"
"Ist die Zeitkoordinate nur lang genug, sinkt die Überlebensquote für jeden auf Null..."

Chromatic-Fire Photography [nofollow] | Schwarze-Schweiz [nofollow]

Cadal

Tried to install via advanced RC1-1 RC1 on test site, no dice. Don't really want to risk enabling fsockopen() on my domain. It's not urgent but an update would be appreciated :)

Advertisement: