Advertisement:

Author Topic: cURL fetch_web_data  (Read 19716 times)

Offline aldo

  • Sophist Member
  • *****
  • Posts: 1,356
  • Gender: Male
cURL fetch_web_data
« on: December 29, 2008, 12:32:23 PM »
Link to Mod

The function in Subs-Package.php fetch_web_data uses fsockopen() in order to obtain the data from the page specified. However not all hosts (usually free hosts) allow the usage of the fsockopen() function. 

So I have fixed that. This modification will allow the fetch_web_data(); function to use cURL to obtain the data instead. 

Of course not all hosts support cURL, but most hosts with PHP5 do have cURL installed. But don't worry, if your host doesn't support cURL but does fsockopen it will use fsockopen instead, but if thats the case you don't really need this mod.

NOTE: If you have no troubles downloading packages from external servers or getting SMF news data and such you don't need this mod as this is really a fix for servers that support cURL but have problems downloading packages and such.

Changelog:
  • 1.1 - Added support for SMF 2.0 RC1
  • 1.0 - Initially added

NOTE: If you have no troubles downloading packages from external servers or getting SMF news data and such you don't need this mod as this is really a fix for servers that support cURL but have problems downloading packages and such.
« Last Edit: February 08, 2009, 12:35:40 AM by aldo »

Offline M-DVD

  • SMF Hero
  • ******
  • Posts: 1,650
  • Gender: Male
  • Step by step will update the mods to SMF 2RC2
Re: cURL fetch_web_data
« Reply #1 on: December 29, 2008, 01:05:40 PM »
Interesting.

Yesterday I had this situation  :) (and was about to do so)

Thanks.
« Last Edit: December 29, 2008, 01:08:48 PM by M-DVD »

Offline aldo

  • Sophist Member
  • *****
  • Posts: 1,356
  • Gender: Male
Re: cURL fetch_web_data
« Reply #2 on: December 29, 2008, 07:24:48 PM »
Your very welcome :)

Offline karlbenson

  • SMF Friend
  • SMF Super Hero
  • *
  • Posts: 15,629
  • Gender: Male
    • @mortonssols on Twitter
    • Criminal Solicitors
Re: cURL fetch_web_data
« Reply #3 on: December 29, 2008, 08:00:58 PM »
Nice one.

Although in my experience its more likely thats smf's function will work than cUrl.
From what I've read/seen curl is disabled on alot more hosts - especially at the cheap end, than fsockopen is.

However if someone people are able to use it, then its worthwhile. :)

Offline aldo

  • Sophist Member
  • *****
  • Posts: 1,356
  • Gender: Male
Re: cURL fetch_web_data
« Reply #4 on: December 30, 2008, 03:41:48 AM »
Yeah but I don't know if thats exactly true... I see a lot of free hosts that have fsockopen disabled, yet cURL enabled :)

Offline Nao 尚

  • SMF Friend
  • SMF Hero
  • *
  • Posts: 5,928
  • Gender: Male
    • wedgebook on Facebook
    • Wedge
Re: cURL fetch_web_data
« Reply #5 on: December 30, 2008, 04:53:50 AM »
Is it the season, the weather or something? ;)

This is exactly what I came up with in SMG 1.5 and Aeva 5.x :)
And yes Karl, fsockopen() is definitely more widely supported by safe-mode servers than curl(), but there are also some cases where curl() can be useful.

I feel the SMF team would do well by rewriting the fetch_web_data() function, not only to feature a similar "second choice" like aldo's mod or my implementation of curl() (check out Subs-Aeva.php in Aeva or MGallery-Embed.php in SMG), but also to use an actual preg_match on the response code, instead of the test for " 200 " and " 201 ", because most servers are definitely returning more "valid" codes than these. Basically, anything in the 2xx and 3xx ranges is good.

For instance, DailyMotion thumbnails are a ****** (I'm sure I mentioned this somewhere over at smf-media or in the SMG topic), it sends redirect responses and then when it's good it still sends a 3xx header, so my curl() stuff has to enter a redirect loop, but then again it works, while I don't think fetch_web_data() could retrieve a DailyMotion thumbnail easily.
The regexp I came up with does the job fairly:

Code: [Select]
if (preg_match('~^HTTP/.+\s+[23][0-9][0-9]\s~i', $response, $reply) != 1)
I will not make any deals with you. I've resigned. I will not be pushed, filed, stamped, indexed, briefed, debriefed or numbered.

Aeva Media rocks your life.

Offline glennk

  • Sr. Member
  • ****
  • Posts: 899
Re: cURL fetch_web_data
« Reply #6 on: January 03, 2009, 04:42:39 AM »
What is the purpose of this mod ??

Offline Nao 尚

  • SMF Friend
  • SMF Hero
  • *
  • Posts: 5,928
  • Gender: Male
    • wedgebook on Facebook
    • Wedge
Re: cURL fetch_web_data
« Reply #7 on: January 03, 2009, 05:01:15 AM »
To make fetch_web_data work on a wider array of web servers.
If you don't have a Safe Mode or Openbasedir'd server, this is of no concern to you.
If your server is already capable of downloading packages from external servers, then you don't need it at all.
I will not make any deals with you. I've resigned. I will not be pushed, filed, stamped, indexed, briefed, debriefed or numbered.

Aeva Media rocks your life.

Offline aldo

  • Sophist Member
  • *****
  • Posts: 1,356
  • Gender: Male
Re: cURL fetch_web_data
« Reply #8 on: January 03, 2009, 10:21:56 PM »
Thank you Nao, added that to the mod info.

Offline Nao 尚

  • SMF Friend
  • SMF Hero
  • *
  • Posts: 5,928
  • Gender: Male
    • wedgebook on Facebook
    • Wedge
Re: cURL fetch_web_data
« Reply #9 on: January 04, 2009, 04:04:34 AM »
Ah yes, good idea, eh ;)
I will not make any deals with you. I've resigned. I will not be pushed, filed, stamped, indexed, briefed, debriefed or numbered.

Aeva Media rocks your life.

Offline aldo

  • Sophist Member
  • *****
  • Posts: 1,356
  • Gender: Male
Re: cURL fetch_web_data
« Reply #10 on: February 08, 2009, 12:36:37 AM »
Sorry peoples... Forgot to update this. It now supports SMF 2.0 RC1.

Offline Mystery?

  • Jr. Member
  • **
  • Posts: 240
    • GamerStatic
Re: cURL fetch_web_data
« Reply #11 on: February 08, 2009, 02:08:39 AM »
What does this mod do? I don't know what cURL fetch_web_data.
Join my NEW website that I just created and be a big part of it (July 30, 2010): http://www.GamerStatic.net

Offline aldo

  • Sophist Member
  • *****
  • Posts: 1,356
  • Gender: Male
Re: cURL fetch_web_data
« Reply #12 on: February 08, 2009, 02:13:09 AM »
www.php.net/curl

It is recommended to use this modification if and when you cannot use fsockopen on your host (You know, like it being disabled) but yet you have cURL enabled, so this can make fetch_web_data() work :)

Offline pjr

  • Semi-Newbie
  • *
  • Posts: 27
Re: cURL fetch_web_data
« Reply #13 on: April 11, 2009, 12:45:17 PM »
How can you tell if fsockopen is disabled? And how can you enable it if you have full admin rights?

Offline Nao 尚

  • SMF Friend
  • SMF Hero
  • *
  • Posts: 5,928
  • Gender: Male
    • wedgebook on Facebook
    • Wedge
Re: cURL fetch_web_data
« Reply #14 on: April 11, 2009, 02:42:08 PM »
How can you tell if fsockopen is disabled?
IIRC, automatic package downloading (for SMF updates) won't work, for instance.
I will not make any deals with you. I've resigned. I will not be pushed, filed, stamped, indexed, briefed, debriefed or numbered.

Aeva Media rocks your life.

Offline aldo

  • Sophist Member
  • *****
  • Posts: 1,356
  • Gender: Male
Re: cURL fetch_web_data
« Reply #15 on: April 15, 2009, 10:36:28 AM »
How can you tell if fsockopen is disabled?
IIRC, automatic package downloading (for SMF updates) won't work, for instance.
Yeah, the news display shouldn't work either, since SMF uses fsockopen to get the JS files which displays the latest version, news, packages, etc.

Not sure about the enabling thing... should be in your php.ini I suppose. But I am no server administrator ;)

Offline flutter

  • Jr. Member
  • **
  • Posts: 225
Re: cURL fetch_web_data
« Reply #16 on: April 14, 2010, 08:39:36 PM »
Hoping this thread is still being monitored. 

This mod, would it also cover people downloading from my site?  I'm having issues with people not being able to download at times and I'm using download pro, VBGamer can't work out what's going on.  I'm wondering if this sock and culr thing could be the culprit, excuse my stupidity, this hole thread looks like a foreign language to me

Offline Nao 尚

  • SMF Friend
  • SMF Hero
  • *
  • Posts: 5,928
  • Gender: Male
    • wedgebook on Facebook
    • Wedge
Re: cURL fetch_web_data
« Reply #17 on: April 15, 2010, 09:00:11 AM »
It's unrelated. It's for helping your *server* to fetch files from other servers, not to send them to users.
I will not make any deals with you. I've resigned. I will not be pushed, filed, stamped, indexed, briefed, debriefed or numbered.

Aeva Media rocks your life.

Offline grimm

  • Semi-Newbie
  • *
  • Posts: 11
  • Gender: Male
    • Schwarze-Schweiz
Re: cURL fetch_web_data
« Reply #18 on: February 09, 2011, 09:01:52 AM »
Update for RC4 please?^^
"Closing your eyes forces you to see the darkness inside"
"Ist die Zeitkoordinate nur lang genug, sinkt die Überlebensquote für jeden auf Null…"

Chromatic-Fire Photography [nofollow] | Schwarze-Schweiz [nofollow]

Offline Cadal

  • Semi-Newbie
  • *
  • Posts: 30
  • Gender: Male
    • Shadowbreed MMO Community
Re: cURL fetch_web_data
« Reply #19 on: February 11, 2011, 11:11:50 AM »
Tried to install via advanced RC1-1 RC1 on test site, no dice. Don't really want to risk enabling fsockopen() on my domain. It's not urgent but an update would be appreciated :)