First off, to use the retreive full article feature, the news source must have a decent design on their website that will make it easy. If not, it won't work. Here's a quick example of using the Retrieve full article feature from AZCentral.com (Arizona's Newspaper and NBC affiliate site), as well as the regular expression syntax to retreive it for this particular feed.
Here is the feed:
http://www.azcentral.com/rss/feeds/northphoenix.xmlHere is one of the items from that feed:
http://www.azcentral.com/community/phoenix/articles/2009/01/12/20090112phxpopulation0107.htmlIf you look at the HTML source for the feed, you'll find the article in the following HTML code...
<span fd-type="start" fd-id="default"></span>
<h1 class="topHeadline">For first time in modern history, Phoenix may be losing people</h1>
<h2 class="subHeadline">Population dip would further strain budget</h2> <p class="byline vcard clearfix">
<a href="#comments"><span class="bylinecomments" id="commentcount"></span></a>
by <strong>Michael Clancy and Casey Newton</strong> - Jan. 12, 2009 12:00 AM<br />
<span class="org">The Arizona Republic</span>
</p>
<div id="articlestory"><p>For the first time in modern history, Phoenix's population could be shrinking. </p>
<p>It's an idea that would have been unimaginable just a few years ago, when Phoenix was surging up the list of the nation's most populous cities. Now, a variety of indicators suggest that fewer people are living here than a year ago.</p>
<p>No one knows for sure exactly how many people have moved in or out. But with the 2010 census about to get under way, some indicators suggest Phoenix's population may be smaller than the projected 1,636,170 people.
<span id="articleFlex1"><script type="text/javascript">OAS_AD('ArticleFlex_1')</script></span>
</p>
<p>City records show declining trends in several key areas. Among them: </p>
<p><br /> • Foreclosure numbers have skyrocketed, meaning fewer city homes are occupied. </p>
<p><br /> • Water hookups are down, suggesting the same. </p>
<p><br /> • Some aspects of trash collection have ebbed because fewer people are buying things that produce waste. </p>
<p><br /> • Crime has declined across the city while police are getting fewer calls for services, a possible indicator of fewer people. </p>
<p><br /> • Sales-tax revenues are likely to drop for the second year in a row, with this year's collections off almost 8 percent from last year. </p>
<p>Experts say each trend can be explained in part by other factors, such as the national recession and the bursting of the housing bubble. When consumer spending decreases, so do tax revenues and the amount of trash that people throw away.</p>
<p>Still, an <i>Arizona Republic </i>analysis of the trends suggests Phoenix has anywhere from a few hundred to several thousand people fewer than projected.</p>
<p>"I think the number is minor, but with all these indicators moving down, I think it is real," said City Manager Frank Fairbanks, declining to speculate of the number of people the city may have lost.</p>
<p>Several factors could be behind a population loss, Fairbanks and others say. The state law requiring employers to verify immigration status of their workers is believed to have driven many immigrants out of the state in 2008. The regional decline in construction jobs also could be behind an exodus. </p>
<p>Statewide, growth has slowed to a crawl. A population report released in December showed Arizona grew by 1.6 percent, or about 100,000 residents, in the previous fiscal year - less than half the growth rate of two years prior. </p>
<p>But the possibility that Phoenix has declined relative to other cities has leaders worried.</p>
<p><h3>Tax revenue at risk
</h3></p>
<p>Losing any fraction of the city's population could mean less revenue to Phoenix from the state.</p>
<p>Arizona shares 15 percent of its income-tax collections with cities based on those cities' population. </p>
<p>Phoenix's population now represents about 30 percent of Arizona's population. This year, Phoenix got $435 million from the state. That made up 38.7 percent of the city's general fund. </p>
<p>Assuming that the suburbs maintain or grow their current populations, Phoenix's share of revenue from the state could shrink in coming years, compounding the city's budget crisis.</p>
<p></p>
<p>Phoenix already is facing cuts of $270 million, or more than 22 percent, from its budget this year. The city is weighing cuts from community centers, senior centers, libraries, police and fire protection, and other services.</p>
<p>"It's an important, pressing issue for all of our cities here, but Phoenix probably most of all," said Rita Walton, who monitors population for the Maricopa Association of Governments. "They're the biggest and stand to lose the most."</p>
<p><h3>The evidence
</h3></p>
<p>Several indicators exist showing that the population could be declining. Among them: </p>
<p><b><br /> • Water: </b>The number of water-using accounts fell about 5,600 from fiscal 2007-08 to 2008-09. The number of accounts using no water almost doubled, on average, meaning those homes still have water connections but are probably empty. </p>
<p>Tracking population by water hookups is "a good way of watching for population change," said Steve Doig, a journalism professor at Arizona State University who used a similar method to track the return of people to southern Florida after Hurricane Andrew. </p>
<p>Evidence about the increasing number of water accounts using no water and the decreasing number of accounts generally indicates population decline. </p>
<p><b><br /> • Trash:</b> Trash collection dropped 2 percent overall from fiscal year 2006-07 to 2007-08. While the amount of recyclable material picked up increased slightly, bulk-trash pickup dropped almost 15 percent during the same period. </p>
<p>Dennis Hoffman, professor of economics at the W. P. Carey School of Business at ASU, said trash collections, just like electrical and water hookups, frequently are used to gauge more precise population shifts than a census provides.</p>
<p><b><br /> • Foreclosures: </b>Bank repossessions of homes continue to increase. The key areas of decline are in the western part of the city. Phoenix overall had a staggering 534 percent increase in foreclosures in the first half of 2008, or an increase of about 5,000 additional homes in foreclosure compared with the first half of 2007. Figures for the last half of the year are not yet available. </p>
<p><b><br /> • Taxes: </b>Sales-tax collections have declined dramatically. Although not tied to growth as directly as water hookups, increases in sales-tax collections began slowing in 2006-07, then dropped in the next two years. </p>
<p>Hoffman said that since the retail sector thrives on population growth, it is reasonable to conclude that with the retail sector in so much trouble, declining sales-tax revenue could correlate to declining population.</p>
<p><b><br /> • Crime: </b>Phoenix saw violent crime decrease 0.3 percent last year, along with a 0.3 percent decrease in total property crimes. Crime has continued to fall in 2008, records show. In addition, Phoenix police have had fewer calls for service this year than last year, Police Chief Jack Harris said. Growing populations almost always result in more calls to police.</p>
<p><h3>Hope ahead?
</h3></p>
<p>Not everyone is pessimistic. Mayor Phil Gordon expressed skepticism at the idea of a shrinking city.</p>
<p>"The growth of Phoenix, like all cities in the Valley, has slowed significantly. But Phoenix's net growth is still positive, both in jobs and population," he said. </p>
<p>Gordon said the city is poised to resume steady growth as soon as the economy emerges from the current recession.</p>
<p>"When the recovery comes, we will be better positioned to recover faster and take advantage of it than many other cities locally and throughout the U.S. The world and the nation know Phoenix.... Whether it's national or international, we're first on the list of cities to invest in as the economy turns."</p>
<p></p>
<p><b>Reporters Astrid Galvan, Rebekah Sanders, Dianna M. Náñez, Edythe Jensen, Elias C. Arnold, Lynh Bui and Gary Nelson contributed to this article.</p>
<p></b></p>
<p></p></div>
The main thing that I want is the story... I don't need the title or the author, and they make this easy because the store is wrapped in a div tag with the id of "articlestory" (find <div id="articlestory">, and then the closing </div> is at the bottom of the code block.
Perfect, so I know my article is always wrapped in this HTML, so it'll be easy to always pull it out. This also makes the regex easy to write.
For those that aren't familiar with PCRE, there is sooooooo much to it that I can't really explain, but I'll give the regex I use, and why I've used it, and hopefully that will give you a starting place...
~<div id="articlestory">(.*)<\/div>~siUIn parts:
Each regex must start and end with a character, any characters, as long as it's the same. I choose to use ~. You could just as easily use any non-Meta character (see the PCRE reference) like /, or : or whatever works for you. I like ~, its easy to remember and isn't used a lot in text, so it normally won't be used in your regex at all.
Next I have the beginning div tag - <div id="articlestory">
Then I have (.*). This means grab EVERYTHING that follows that starting div tag until the end of the line. (note, I'll get to it later)
The purpose of being in () is that whatever is found in this () will be captured into the first index of the array that the mod will use to post. If you have to use () to enclose anything BEFORE this (.*), then you'll want to prevent it from being captured and being put into the array by starting it with (?:
search string here)
If you have to use any parenthesis in your regex BEFORE your (.*), you will have to use (?: to start it. In most cases, I dont think this will be required, but it is something to keep in mind if you see your feed being posted incorrectly.
The next part is the closing div... <\/div>. All / in a regex need to be escaped using a \.
Following that is the closing ~ that I discussed earlier.
After that are a few letters, siU. These are called pattern modifiers. These can change the way your regex works. Here is what the siU stand for -
s - means that .* should capture all characters including newlines and carriage returns. Without this, .* will stop at the end of the line. So by using the s modifier, .* will capture everything from where it starts to the end of the document (more to come on that too, since we want to stop at some point)
i - this means that the search is caseless. So if you want to search for <DIV>, but <div> is the document, your regex will still find it.
U - this is the ungreedy modifier. Remember how I said that .* will capture everything until the end of the document... this stops it. This forces .* to stop when it comes across the next part of the pattern
Well, I hope I confused you enough... have fun...
