HeadlinesPlugin Feedback and Development

-- PeterThoeny - 12 Jun 2002

Further enhancements

Article State information

Have a look at http://www.bloglines.com 's "My Feeds" and "My Blogs" feature. Newsfeed reader have quite some added value vs. just presenting the raw data. Basically, I'd like to visit a page and see which feeds have new articles and how many. Once I visited them I'd like to tag articles as read, unread or mark them as important to keep them in a long-term storage. What is needed is access to particular information about the feed and a database that stores the state of each article of each feed for each user that wishes to do so. While rendering the HEADLINES tag this information is to be accessed to filter out those articles that don't match the given "state" argument.

HEADLINES{href="..." state="new"}
HEADLINES{href="..." state="unread"}
HEADLINES{href="..." state="new|unread"}
HEADLINES{href="..." state="keep"}
HEADLINES{href="..." state="new|unread|keep"}
HEADLINES{href="..." state="read"}
HEADLINES{href="..." state="all"}

Each article needs a select "Mark as ..." and a final submit button to post the article state changes on the complete feed. This needs a clear way to identify articles in a feed. Albeit this is already part of the rss specs, some news feeds don't care about specs. So best would be to md5 the article (-title).

The article state database needs to shrink when a feed doesn't provide a given article anymore. But what about articles marked to be kept. They get stored somewhere else and don't expire. Only if marked as read they get deleted from long-term storage again.

Fold multiple feeds into one

Generate a combined display of a set of feeds.

Filter feeds using regexps

Include/exclude articles from a feed based on argument and urlparam patterns.

Re-export feeds again

Generate an rss feed from multiple source rss feeds

WebSearch feeds

Allow WebSearch to retrieve data stemming from rss feeds. This would require substantial changes to the core by adding appropriate callbacks to SEARCH so that plugins can blend in their data.

-- MichaelDaum - 21 Nov 2005

See also http://www.zfilter.com: zfilter is an rss aggregator with emphasis on facilitating discussions on the latest news and events from all over the internet.

-- MichaelDaum - 26 Mar 2006

Discussions

Uploaded the final HeadlinesPlugin v2.0

-- MichaelDaum - 23 Jan 2006

Thanks Michael for publishing the latest version

Related, TWiki can now also generate ATOM feeds, see AtomSyndication.

-- PeterThoeny - 24 Jan 2006

Please consider adding the use strict; pragma to this plugin. Its use is important to ensuring the quality of TWiki plugins and avoiding unpleasant surprises. See UseStrict for more.

-- MeredithLesly - 02 Jul 2006

How do I get a nice date instead of Wed, 05 Jul 2006 11:54:13 +0000 ?

-- ArthurClemens - 06 Jul 2006

The Plugin returns the date in whatever format supplied, except for the dc:date which gets converted into ISO date format.

You would need a new date format feature. Suggested spec: Add a dateformat="" parameter and DATEFORMAT Plugin setting. If not set, no date conversion is done. If set, date conversion is done, with the same $year etc variables as in VarGMTIME2.

-- PeterThoeny - 06 Jul 2006

Why not put $percntGMTIME{\"...\"}$percnt into the format string of the HEADLINES tag. If you've got the TimeSincePlugin installed you could even have $percntTIMESINCE{\"$date\"}$percnt in there for more fancyness. I did not test this but in principal it should work out already. If %GMTIME{...}% does not get expanded an extra call TWiki::Func::expandCommnonVariables() may be added to the plugin somewhere. I don't think another dateformat argument is needed.

-- MichaelDaum - 07 Jul 2006

How do I cater for posts without a title? In my RSS reader these show up as "Untitled", but the generated list shows broken (unrendered) links.

For example:

format="[[$link][$title]]"

will generate: [[http://www.stevenberlinjohnson.com/2006/06/the_video_for_t.html][]]

Here's a patch to replace emtpy titles with 'Untitled' that also improves/fixes parsing atom feeds. -- MichaelDaum - 21 Jul 2006

-- ArthurClemens - 07 Jul 2006

Try:

format="[[$link][$percntCALC{$IF($EXACT($title, ), Untitled, $title)}$percnt]]"

(this is untested.)

• This doesn't work as is, due to rendering order? I get %CALC{$IF($EXACT(Boy and Girls and More, ), Untitled, Boy and Girls and More)}% almost verbatim in the list (with $title as 'Boy and Girls and More'). -- ArthurClemens - 11 Jul 2006

-- PeterThoeny - 07 Jul 2006

Hi today I installed HeadlinesPlugin with TWiki 4.0.4. I have to use a http Proxy for outbound request. Therefore I find out that the new configuration concept isn't used by this plugin.

It just try to locate the getPreferencesValue('PROXY(HOST|PORT)').

The internel Net.pm also try to locate $TWiki::cfg{PROXY}{HOST}

btw. is there any function to get those values ? I haven't found one similar to TWiki::Func::getPreferencesValue in Func.pm.

Thanks for your work.

Tom

-- ThomasFreudenberg - 08 Jul 2006

Thanks Michael again for that plugin. Anyhow, I have some problems with the installation. I get the following error message, please read Support.ProblemsWithHeadlinesPlugin.

I'd be thankful for any idea. Alex

-- AlexRaabe - 29 Aug 2006

After installing the most recent version of the HeadlinesPlugin on my CairoRelase I found the following error in a page containing the %HEADLINES{}% tag :

Can't locate LWP/UserAgent.pm in @INC (@INC contains: /path/to/twiki/lib  . /usr/lib/perl5/5.8.6/i586-linux-thread-multi /usr/lib/perl5/5.8.6 /usr/lib/perl5/site_perl/5.8.6/i586-linux-thread-multi /usr/lib/perl5/site_perl/5.8.6 /usr/lib/perl5/site_perl /usr/lib/perl5/vendor_perl/5.8.6/i586-linux-thread-multi /usr/lib/perl5/vendor_perl/5.8.6 /usr/lib/perl5/vendor_perl) at (eval 85) line 2.
BEGIN failed--compilation aborted at (eval 85) line 2.

-- KeithHelfrich - 10 Feb 2007

Oops, this plugin should fall back gracefully if LWP is not installed.

-- PeterThoeny - 10 Feb 2007

Keith - When you see a can't locate error like this, the first thing to try (assuming that you have either root access, or a local CPAN) is cpan LWP::UserAgent from the command line. see: http://search.cpan.org/~gaas/libwww-perl-5.805/lib/LWP/UserAgent.pm

-- SvenDowideit - 10 Feb 2007

The plugin adds a link to a stylesheet to the head even if no plugin tag is on the page.

-- ArthurClemens - 19 May 2007

There's no way to conditionally put a css file into the html head safely. There's a technical reason for that: the IMAGE tag is handled by a registered tag handler. The css file is added using the commonTagsHandler() . Once an IMAGE tag was detected, it is too late to add the css file to the head using commonTagsHandler() because the first part of the page containing the head element has already been processed. This plugin deliberately does not use the addToHEAD() API because that would break proper prioritization of the styles when other contribs/skins want to override the default css that comes with this plugin. This issue can't be resolved without a prependToHEAD() in the core.

-- MichaelDaum - 21 May 2007

Does my Twiki server needs a internetconnection or is the rss plugin working clientside?

-- MaartenDeRuiter - 27 Jul 2007

Does my Twiki server needs a internetconnection or is the rss plugin working clientside? i get this error:

HeadlinesPlugin ERROR: 500 Can't connect to rss.howardshome.com:80 (Bad hostname 'rss.howardshome.com')

-- MaartenDeRuiter - 27 Jul 2007

No , If your TWiki Server is on internet you will get RSS feed directly in desired topic just by using Headline variable . I feel your machine is behind firewall that why you are getting this error. To overcome this error manually specify PROXYHOST and PROXPORT variable in TWikiPreferences.

-- AnjaniKumar - 27 Jul 2007

i'm running the server on a intranet, the server has no connection to internet whatsoever. Can this plugin still work without the server having a internetconnection?

-- MaartenDeRuiter - 27 Jul 2007

You can point this plugin to RSS feeds on your intranet, but by definition you will not be able to connect to RSS feeds on the public internet if your intranet has no connection to it.

-- PeterThoeny - 29 Jul 2007

Hi, i solved the issue with the firewall, however now i get this error:

HeadlinesPlugin ERROR: 500 Can't connect to www.nu.nl:80 (connect: Connection refused)

-- MaartenDeRuiter - 30 Jul 2007

Is there a way to prevent Cross-site scripting (XSS) in the $description?

-- NicolasBorboen - 13 Oct 2007

It looks like the <author> tag is not parsed. The code looks like it should be retrieved using $contributor, but that value is void.

-- ArthurClemens - 29 Oct 2007

I'm trying to override the defaulkt HEADER and FORMAT values; having little luck.

So I use = * Set HEADLINES_HEADER = ...=? It doesn't seem to be changing anything.

-- VickiBrown - 30 Nov 2007

To overload the HEADER plugin setting, create a HEADLINESPLUGIN_HEADER setting in your TWiki.TWikiPreferences.

-- PeterThoeny - 02 Dec 2007

I would really like this plugin to handle a list of feed urls. At our company we have a number of people with delicious accounts that we want to syndicate on the homepage as 1 list.

-- ArthurClemens - 15 Dec 2007

Aggregating news feeds into a single feed, yes, this is a useful feature. Mini spec:

• Nameless and href parameter accept comma delimited URLs, such as href="http://feed1.example.com/feed.xml, http://feed2.example.com/feed.rdf"
• refresh parameter affects all feeds. Alternatively, if comma delimited list, it applies it in the sequence of feeds.
• limit, header, format parameters apply to all.
• All aggregated feeds are sorted by time, and limited to limit.
• Remove duplicate urls from collection: latest are kept

-- PeterThoeny - 15 Dec 2007

RSS feed links are from Japanese sites appear as (Unicode) character codes instead of Japanese/chinese characters. Any Work around to show up link in proper language . It showing up properly in RSS reader.

-- AnjaniKumar - 19 Dec 2007

I am having some troubles reading a atom feed: http://www.lostboys.nl/blog/articles.atom I am puzzled as why I don't get any value for $link and $summary

-- ArthurClemens - 08 Feb 2008

Under TWiki 4.0.4 with no header or format I used to get a nice little table for the headlines feed. Now under 4.2.0 and the latest HeadlinesPlugin I get a bulleted list, I can't seem to get a table - is there a way to get a table output with the Headlines plugin?

-- ThomasBoyda - 03 Mar 2008

Not sure why the default format has been changed, I'd prefer to stay compatible. In any case, you can control the format per feed with a format="..." parameter to %HEADLINES. You can also redefine the format on a site level: Create a HEADLINESPLUGIN_FORMAT setting in your Main.TWikiPreferences, and copy/modify the FORMAT setting of the HeadlinesPlugin.

-- PeterThoeny - 04 Mar 2008

I was able to modify the header and format items to the HeadlinesPlugin href parameter (news source) is missing directive to get the output to a table, but the table had alternate shading of the rows - and I have yet to get that bit figured out. But I will continue to plug away.

-- ThomasBoyda - 04 Mar 2008

You might want to add a %TABLE% tag at the top of the output and set the colors with TablePlugin attributes.

-- ArthurClemens - 04 Mar 2008

I'm trying to use this to pull in content from a secure site.

Is any support planned to pass both:

1. Apache-style user and password
2. Login credentials (I can't remember the name of this, but its the type used by mediawiki?)

Thanks. M.

-- MartinCleaver - 31 Mar 2008

I would like to limit the list of feeds by date (rather than by number), eg to show all feeds from the previous day or week. Is it possible to do this?

-- TamsinTweddell - 11 Apr 2008

Small RFE: In default feed, do not show newsfeed image in header in case there is no image tag in the feed.

-- PeterThoeny - 11 Apr 2008

There are cases where <link><![CDATA[...]]></link> format is used. The current version puts in $link, which is not right.

How about applying this patch?

--- lib/TWiki/Plugins/HeadlinesPlugin/Core.pm   2007-09-14 00:48:42.000000000 +0900
+++ lib/TWiki/Plugins/HeadlinesPlugin/Core.pm.patched   2008-06-25 15:04:58.000000000 +0900
@@ -512,7 +512,7 @@
       $ok = 1;
     }
     if (/<link[^>]*>\s*(.*?)\s*<\/link>/) {
-      $val = $1;
+      $val = &recode($1);
       $val =~ s/^http:\/\/.*\*(http:\/\/.*)$/$1/gos; # yahoo fix
       $line =~ s/\$(item)?link/$val/gos;
       $ok = 1;

-- HideyoImazu - 25 Jun 2008

Thanks for the patch. Have you got an news feed that this can be tested on?

-- MichaelDaum - 25 Jun 2008

e.g. http://z001.ig.com.br/ig/08/18/930322/blig/gamerbr/rssposts.xml

-- HideyoImazu - 25 Jun 2008

i am trying to display the changes of a web which requires a logged in user in a another web which has the same security settings. i used 'HeadlinesPlugin ERROR: 501 Protocol scheme 'https' is not supported (LWP::Protocol::https not installed) ' to display the feed, but it only displays '(TWiki login) WebRss < OtherWeb < TWiki'. is it possible to display a feed like this with the highlights plugin at all?

-- JoergEichhorn - 04 Aug 2008