RSS Feeder

[SlammedDime]

np

[KahneFan]

If you're taking suggestions for a next version

It might be nice to see a column for "Feed Items" or even just "Total" in the admin area, so that you can see how many articles each feed has brought through. (hope that makes sense)

| Feed Enabled | Feed Title | Feed Total | ...

Also, when a feed is deleted, if possible, it would be nice to delete the articles of that feed. For example, you have a feed that you no longer want to associate with, for whatever reason, you delete the feed, but the articles are still there.

[SlammedDime]

The first would be easy to implement because I already have a log table that keeps track of the feed items posted, or I could add a column to the rssfeeds table to keep a count and update it when I update the last updated time of the feed.

The second would be easy to implement, again, because I already have the log table, would just have to add a column for topic id.

The only issue I see overcoming is that is someone deletes feeds already posted, or if they have the posts set to require approval, and they delete instead of approve, the feed won't be removed in the feed table, so the 'Total' wouldn't be accurate to what is actually on the forum, but to what has actually been posted, regardless of whether or not it's still there.

[Smoky "Rider" Blue]

also SlammedDime, Kahne wanted to be able to allow users to add feeds as well..

so if you can swing this, i know he would be very happy.. 

[Bigguy]

Might be nice to integrate this into the News Fader.

[KahneFan]

also SlammedDime, Kahne wanted to be able to allow users to add feeds as well..

so if you can swing this, i know he would be very happy.. 

It sounds as though this would be a tough one though. So, having members contact me is OK for now. But if it could happen...

Also, I just tried it HERE and when I ran it from admin to pull the old articles (535 and ran smoooooth), it posted them by either 1) the order in which I input them (z-newest -> a-oldest) or 2) in alphabeticle (z-a), instead of by date (originally) posted. As long as it posts them in date order from here out will be OK with me. Do you know if future updates will be posted by date order?

[Smoky "Rider" Blue]

i thought of that too BigGuy and i actually have one rss with its own db and page..

set up like the google home pages that you can move around.. happy thoughts tho, since i dont know what im doing with it, nor the time to play like i would like and actually learn, so..

but im more than willing to send my stuff to whoever that would be interested.

[SlammedDime]

It may be possible to allow this to extend to allow users to create feeds, actually might be quite easy to extend that functionality now that I think about it.  I already store the id_member of the poster in the feed table, so I would just have to create an additional profile page that allows a user to edit his/her feed and they could have the same options that an admin would have.

As for how they are imported... the script will run from the top of the feed down to the bottom of the feed, so however they are placed in the feed is how they will be imported, top being imported first, and bottom being imported last (most recent imported).  Now that I think about this, it should probably be reversed, but it's something to keep in mind for later versions

Kahne - it's nice to know it worked on that large of a feed.

[KahneFan]

It may be possible to allow this to extend to allow users to create feeds, actually might be quite easy to extend that functionality now that I think about it.  I already store the id_member of the poster in the feed table, so I would just have to create an additional profile page that allows a user to edit his/her feed and they could have the same options that an admin would have.

The only thing here would be making sure they know where to post them. If user feeds is a possibility, admin approval might be needed, which is still easier than entering them. OR, if admin could specify a single setting of: Post all user feeds in: {board}

[Sabre™]

wOOt!!
Ive only just setup a test site with smf2(bout time to move on from 1.1.7), and whats the first mod I see!?? lol
Very nice mate, thanks

[bloodfire]

Hi, thanks for this mod.. love it.

Just wanna know any1 tried using the Retrieve Full article features? Any example of the expression or how to use it?

Regards

[Sudhakar Arjunan]

SlammedDime such a excellent mod.

I will install it and comment here.

Have some suggestions.

Need a Auto filler column under each feed settings.

Say if i add the feed from cnet & reuters.

On the settins page, under feeds i will enter " Source : Cnet " or " Source : Reuters " on the new column you create.

so every time a feed is added , it will enroll the Source text ever time feed is posted on our forum.

Ex Post :

Subject : - Subject already printed here

Body : - Feed information is printer here.

Footer :-  Source - CNET.

[Lamarck]

Hi,this mod is just awesome.

However,I have a problem with spanish characters: the mod "cuts" every word with a spanish character and all the text that goes behind.

Example: "tecnología" (technology in english)---> tecnolog 

The other words that go behind a spanish character are also deleted.
Exemple: "Penélope Cruz gana del Oscar" (Penelope Cruz wins the Oscar) ----> "Pen"

This problem makes that almost every topic tittle doesn't show full,and also links and the news itself are almost deleted.
Could you fix that,please?.It's very important for people using SMF in other languages than engliss/feeds in other language.

More feedback/suggestions:

-A feature that would be so useful is the option to insert a "footer" to the rss post.I mean,for example: If I set up a rss for "news",It would be nice to have the option that allows you to insert a footer such as "What do you think about this? We want your opinion!",that automatically is inserted on every topic created by the rss "bot".
That way,you can make posts more "human",and ask for participation.
Is that possible? just a suggestion 

-Also: can you (or someone) make a mini "tutorial" to use the "Retrieve full article" feature?.

Thank you very much and good work.

[SlammedDime]

A.SK - Swell idea... although it would be much better (IMO) to utilize the existing information from the feed, and throw the source in as the Feed's title that is sent via RSS, rather than allow an admin to set the source... this way I can also hyperlink it automatically.

Lamarck - This is probably related to the Snoopy PHP Web browser that the mod utilizes to retrieve feeds and might be an easy fix.  I asked another user on the previous page for a direct link to a feed that has such characters so I can further test this.  Can you provide me with a link as well?

As for the footer - that wouldn't be a bad idea either.

Once 2.0 goes RC and I'm sure this still works as is with it, I'll look into incorporating many peoples suggestions.

As for a tutorial on the regular expression and full article... I'll see if I can throw a mini article together later today after work.

[Garou]

SlammedDime You rock man, a lot of people have been wanting this for a while.  Ill be testing it out later today and cant wait.

Ok I tested and it works great. In fact it does a much better job of reading poorly written RSS Feeds then VB's version does.

Question though can we get an example of how the regular expression in PCRE is supposed to work. Ive never had to use that for anything and I'm sure others have not either.

Definitely and Awesome mod. 

[Lamarck]

Yes,of course...here is a link:

http://www.20minutos.es/rss

Ah,the problem is not only with "´",the same thing happen with "ñ".

Thanks!.

[Nao 尚]

I'm not sure, but... Shouldn't accented data and single quotes be enclosed in CDATA blocks?

[SlammedDime]

First off, to use the retreive full article feature, the news source must have a decent design on their website that will make it easy.  If not, it won't work.  Here's a quick example of using the Retrieve full article feature from AZCentral.com (Arizona's Newspaper and NBC affiliate site), as well as the regular expression syntax to retreive it for this particular feed.

Here is the feed: http://www.azcentral.com/rss/feeds/northphoenix.xml

Here is one of the items from that feed: http://www.azcentral.com/community/phoenix/articles/2009/01/12/20090112phxpopulation0107.html

If you look at the HTML source for the feed, you'll find the article in the following HTML code...

Code Select Expand

<span fd-type="start" fd-id="default"></span>

  <h1 class="topHeadline">For first time in modern history, Phoenix may be losing people</h1>

        <h2 class="subHeadline">Population dip would further strain budget</h2>                <p class="byline vcard clearfix">
          <a href="#comments"><span class="bylinecomments" id="commentcount"></span></a>
          by <strong>Michael Clancy  and Casey Newton</strong> - Jan. 12, 2009 12:00 AM<br />
         
          <span class="org">The Arizona Republic</span>
        </p>

          <div id="articlestory"><p>For the first time in modern history, Phoenix's population could be shrinking. </p>
<p>It's an idea that would have been unimaginable just a few years ago, when Phoenix was surging up the list of the nation's most populous cities. Now, a variety of indicators suggest that fewer people are living here than a year ago.</p>
<p>No one knows for sure exactly how many people have moved in or out. But with the 2010 census about to get under way, some indicators suggest Phoenix's population may be smaller than the projected 1,636,170 people.
<span id="articleFlex1"><script type="text/javascript">OAS_AD('ArticleFlex_1')</script></span>
</p>
<p>City records show declining trends in several key areas. Among them: </p>
<p><br /> • Foreclosure numbers have skyrocketed, meaning fewer city homes are occupied. </p>
<p><br /> • Water hookups are down, suggesting the same. </p>

<p><br /> • Some aspects of trash collection have ebbed because fewer people are buying things that produce waste. </p>
<p><br /> • Crime has declined across the city while police are getting fewer calls for services, a possible indicator of fewer people. </p>
<p><br /> • Sales-tax revenues are likely to drop for the second year in a row, with this year's collections off almost 8 percent from last year. </p>
<p>Experts say each trend can be explained in part by other factors, such as the national recession and the bursting of the housing bubble. When consumer spending decreases, so do tax revenues and the amount of trash that people throw away.</p>
<p>Still, an <i>Arizona Republic </i>analysis of the trends suggests Phoenix has anywhere from a few hundred to several thousand people fewer than projected.</p>
<p>"I think the number is minor, but with all these indicators moving down, I think it is real," said City Manager Frank Fairbanks, declining to speculate of the number of people the city may have lost.</p>

<p>Several factors could be behind a population loss, Fairbanks and others say. The state law requiring employers to verify immigration status of their workers is believed to have driven many immigrants out of the state in 2008. The regional decline in construction jobs also could be behind an exodus. </p>
<p>Statewide, growth has slowed to a crawl. A population report released in December showed Arizona grew by 1.6 percent, or about 100,000 residents, in the previous fiscal year - less than half the growth rate of two years prior. </p>
<p>But the possibility that Phoenix has declined relative to other cities has leaders worried.</p>
<p><h3>Tax revenue at risk

</h3></p>
<p>Losing any fraction of the city's population could mean less revenue to Phoenix from the state.</p>
<p>Arizona shares 15 percent of its income-tax collections with cities based on those cities' population. </p>
<p>Phoenix's population now represents about 30 percent of Arizona's population. This year, Phoenix got $435 million from the state. That made up 38.7 percent of the city's general fund. </p>
<p>Assuming that the suburbs maintain or grow their current populations, Phoenix's share of revenue from the state could shrink in coming years, compounding the city's budget crisis.</p>
<p></p>

<p>Phoenix already is facing cuts of $270 million, or more than 22 percent, from its budget this year. The city is weighing cuts from community centers, senior centers, libraries, police and fire protection, and other services.</p>
<p>"It's an important, pressing issue for all of our cities here, but Phoenix probably most of all," said Rita Walton, who monitors population for the Maricopa Association of Governments. "They're the biggest and stand to lose the most."</p>
<p><h3>The evidence

</h3></p>
<p>Several indicators exist showing that the population could be declining. Among them: </p>
<p><b><br /> • Water: </b>The number of water-using accounts fell about 5,600 from fiscal 2007-08 to 2008-09. The number of accounts using no water almost doubled, on average, meaning those homes still have water connections but are probably empty. </p>
<p>Tracking population by water hookups is "a good way of watching for population change," said Steve Doig, a journalism professor at Arizona State University who used a similar method to track the return of people to southern Florida after Hurricane Andrew. </p>
<p>Evidence about the increasing number of water accounts using no water and the decreasing number of accounts generally indicates population decline. </p>
<p><b><br /> • Trash:</b> Trash collection dropped 2 percent overall from fiscal year 2006-07 to 2007-08. While the amount of recyclable material picked up increased slightly, bulk-trash pickup dropped almost 15 percent during the same period. </p>

<p>Dennis Hoffman, professor of economics at the W. P. Carey School of Business at ASU, said trash collections, just like electrical and water hookups, frequently are used to gauge more precise population shifts than a census provides.</p>
<p><b><br /> • Foreclosures: </b>Bank repossessions of homes continue to increase. The key areas of decline are in the western part of the city. Phoenix overall had a staggering 534 percent increase in foreclosures in the first half of 2008, or an increase of about 5,000 additional homes in foreclosure compared with the first half of 2007. Figures for the last half of the year are not yet available. </p>
<p><b><br /> • Taxes: </b>Sales-tax collections have declined dramatically. Although not tied to growth as directly as water hookups, increases in sales-tax collections began slowing in 2006-07, then dropped in the next two years. </p>
<p>Hoffman said that since the retail sector thrives on population growth, it is reasonable to conclude that with the retail sector in so much trouble, declining sales-tax revenue could correlate to declining population.</p>
<p><b><br /> • Crime: </b>Phoenix saw violent crime decrease 0.3 percent last year, along with a 0.3 percent decrease in total property crimes. Crime has continued to fall in 2008, records show. In addition, Phoenix police have had fewer calls for service this year than last year, Police Chief Jack Harris said. Growing populations almost always result in more calls to police.</p>
<p><h3>Hope ahead?


</h3></p>
<p>Not everyone is pessimistic. Mayor Phil Gordon expressed skepticism at the idea of a shrinking city.</p>
<p>"The growth of Phoenix, like all cities in the Valley, has slowed significantly. But Phoenix's net growth is still positive, both in jobs and population," he said. </p>
<p>Gordon said the city is poised to resume steady growth as soon as the economy emerges from the current recession.</p>
<p>"When the recovery comes, we will be better positioned to recover faster and take advantage of it than many other cities locally and throughout the U.S. The world and the nation know Phoenix.... Whether it's national or international, we're first on the list of cities to invest in as the economy turns."</p>
<p></p>
<p><b>Reporters Astrid Galvan, Rebekah Sanders, Dianna M. Náñez, Edythe Jensen, Elias C. Arnold, Lynh Bui and Gary Nelson contributed to this article.</p>
<p></b></p>
<p></p></div>


The main thing that I want is the story... I don't need the title or the author, and they make this easy because the store is wrapped in a div tag with the id of "articlestory" (find <div id="articlestory">, and then the closing </div> is at the bottom of the code block.

Perfect, so I know my article is always wrapped in this HTML, so it'll be easy to always pull it out.  This also makes the regex easy to write.

For those that aren't familiar with PCRE, there is sooooooo much to it that I can't really explain, but I'll give the regex I use, and why I've used it, and hopefully that will give you a starting place...

~<div id="articlestory">(.*)<\/div>~siU

In parts:
Each regex must start and end with a character, any characters, as long as it's the same.  I choose to use ~.  You could just as easily use any non-Meta character (see the PCRE reference) like /, or : or whatever works for you.  I like ~, its easy to remember and isn't used a lot in text, so it normally won't be used in your regex at all.

Next I have the beginning div tag - <div id="articlestory">

Then I have (.*).  This means grab EVERYTHING that follows that starting div tag until the end of the line.  (note, I'll get to it later)

The purpose of being in () is that whatever is found in this () will be captured into the first index of the array that the mod will use to post.  If you have to use () to enclose anything BEFORE this (.*), then you'll want to prevent it from being captured and being put into the array by starting it with (?:search string here)

If you have to use any parenthesis in your regex BEFORE your (.*), you will have to use (?: to start it.  In most cases, I dont think this will be required, but it is something to keep in mind if you see your feed being posted incorrectly.

The next part is the closing div... <\/div>.  All / in a regex need to be escaped using a \.

Following that is the closing ~ that I discussed earlier.

After that are a few letters, siU.  These are called pattern modifiers.  These can change the way your regex works.  Here is what the siU stand for -
s - means that .* should capture all characters including newlines and carriage returns.  Without this, .* will stop at the end of the line.  So by using the s modifier, .* will capture everything from where it starts to the end of the document (more to come on that too, since we want to stop at some point)
i - this means that the search is caseless.  So if you want to search for <DIV>, but <div> is the document, your regex will still find it.
U - this is the ungreedy modifier.  Remember how I said that .* will capture everything until the end of the document... this stops it.  This forces .* to stop when it comes across the next part of the pattern

Well, I hope I confused you enough... have fun...

[KahneFan]

Great information, thanks!

[Nao 尚]

I think it's best to use (.*?) and no ungreendy parameter, rather than the other way around. It can be confusing otherwise.
Also, / usually doesn't need to be escaped.