TWiki Twitters - How Does This Work?

From Microblogging with Twitter:

We have two Twitter users: @twikiorg and @twikifriends. We also have a pair of Twitter "feeds" that post

1. recent updates to the TWiki Blog
2. "re-tweets" -- cross-postings of messages from TWiki "friends"

This page explains how our Twitter feeds are implemented.

High Level View

TWiki.org Blog Feed (Twitter user: twikiorg)

We use twitterfeed.com to filter and feed our TWiki Blog RSS updates into @twikiorg. Twitterfeed runs every two hours so, if you don't see your new blog post show up right away, please be patient.

TWiki Friends Twitter Sharing (Twitter user: twikifriends)

When you follow either of our two Twitter users — @twikiorg or @twikifriends — @twikiorg will follow you back. We scan the tweets of our followers every 10 minutes, filter for the word TWiki, and post the results back to Twitter at http://twitter.com/twikifriends;

Details

Blog Feed

This is the configuration of Twitterfeed for our "TWikiBlog to Twitter" magic. Pretty simple!

Friends' Feed

We originally used twitterfeed.com for our TWiki Friends' stream. However, twitterfeed wasn't really designed to feed Twitter to Twitter. One potential problem with this is that twitterfeed cannot be configured to run more often than every 30 minutes, but Twitter only sends back the last 20 entries in a stream. So, with twitterfeed, we might miss a friend's mention of TWiki if our friends are being talkative about other things! Also I (VickiBrown) wanted more control over the feed.

We now scan the twikiorg friends_timeline using pyTwerp, run the output through some Perl post-processing code, and push any results back to Twitter with pyTwerp. The process is automated with cron and runs every ten minutes.

Friends' Feed: Gory Implementation Details

If you want to try something like this yourself, you'll need:

• a computer capable of running Python, Perl, and cron
• an installed copy of pyTwerp, as well as two required Python libraries.
• a general understanding of how to use cron and Unix shell scripts.

This process is currently running on Mac OS X. It should run easily on Linux or other BSD flavors of Unix. Whether it will work under Windows is an open question.

Steps

1. Install and configure pyTwerp
2. Get the tweets
3. Filter and process the tweets
4. Retweet any mention of "TWiki"

Install and configure pyTwerp

Install pyTwerp

1. Download pyTwerp from http://code.google.com/p/pytwerp/
2. Download and install two required libraries:
   • setuptools -- http://cheeseshop.python.org/pypi/setuptools
     setuptools is easy to install. Follow the directions on the download page.
   • simplejson -- http://cheeseshop.python.org/pypi/simplejson
     simplejson doesn't come with instructions. Run easy_install simplejson
3. Install pyTwerp by running python setup.py install
   This will install the pyTwerp library into your Python's site-packages location. A utility script named twerp will be also be installed in (probably in /usr/local/bin); this script lets you invoke pyTwerp using twerp <options>.

Configure pyTwerp

The configuration file is created and populated the first time you run twerp. Configure twerp for your twitter account by running:

   twerp -c configfile-name -U twitter-username -P twitter password 

For the TWiki feed, I created two twerp configuration files, one for each of our Twitter accounts, twikiorg and twikifriends. One is used to get tweets from the people twikiorg is following; the other is used to post tweets to twikifriends.

   twerp -c ~/.twerprc_twikiorg -U twikiorg -P ******** 
   twerp -c ~/.twerprc_twikifriends -U twikifriends -P ******** 

pyTwerp allows you to specify a "template" to use when formatting its output. You can provide a template at runtime or edit the configuration file to set a standard template.

template = %(user_screen_name)s: %(text)s http://twitter.com/%(user_screen_name)s/statuses/%(id)s EOT

EOT

At least one Twitter posting client (twitterfeed) allows users to post tweets containing newlines. This confused my post-processing filter, which expected every tweet to be on a single line.

I added an "End of Tweet" code to the pyTwerp template, then run the output of twerp through Perl with the "record separator" set to that code. This ensures that Tweets aren't broken into multiple lines.

Get the Tweets; Process, Filter, and Retweet

I wrote a small shell script I named gotwerp. This does runs twerp, passes the output through several filters, and writes the results (if any) to a file. I then use twerp again to send the contents of that file back to Twitter to post at @twikifriends.

Show Code... Close

#!/bin/sh

export PATH=/usr/local/bin:/opt/local/bin:/usr/bin:/bin

tmpfile=/tmp/twerp.$$

# run twerp
# pass the results through paste to glue any "multi-line" tweets together
twerp -c ~/.twerprc_twikiorg -f |
  paste -s -d' ' - |
  perl -e '
    local $/ = "EOT";  # record separator marks end of each tweet
     open(LOG, ">>/homes/vlb/Projex/Twitter/twikitwitlog"); # keep a log
     print LOG `date`;
     while ($line = <>) {
         chomp $line;  # remove EOT code
         print LOG  $line, "\n";
         next if ($line =~ /^\s*$/); # ignore blank lines
         next unless ($line =~ /[Tt][Ww]ik/); # filter for TWiki
         $line =~ s/\\/\\\\/g;   # protect internal \
         $line =~ s/"/\\"/g;     # protect internal double quotes
         $line =~ s/`/\\`/g;     # protect internal back quotes
         $line =~ s/\$/\\\$/g;   # protect internal $
         $line =~ s/^ */"/;      # quote tweet (protect from shell)
         $line =~ s/ *$/"/;      # quote tweet (protect from shell)

         # prefix with call to twerp
         $line =~ s|^|twerp -c ~/.twerprc_twikifriends |;

         # and print the results
         print $line, "\n";
     }
     close(LOG);
  ' > $tmpfile

# Check the output file size. Greater than 0 bytes?
if [ -s $tmpfile ]; then
    # If any 'TWiki' tweets were found, retweet to twikifriends
    # use the restricted shell to run twerp this time
    /bin/sh -r $tmpfile  >> /tmp/twerp.out
else
    # remove any empty output files
    rm -f $tmpfile
    date >> /tmp/twerp.out
fi

Explanation

gotwerp first gets the friends_timeline for user twikiorg. The friends_timeline returns "the 20 most recent statuses posted in the last 24 hours from the authenticating user and that user's friends." This will include TWiki Blog posts (from the twikiorg user) as well as posts from all the TWiki friends we follow.

   twerp -c ~/.twerprc_twikiorg -f 

All text returned is merged together into one long string

   paste -s -d' ' - |

  perl -e '
    local $/ = "EOT";  # record separator marks end of each tweet
    # ...
    while ($line = <>) {
        chomp $line;  # remove EOT code

Processing continues only if the current line matches the pattern [[Tt][Ww]iki]]. The first two characters are case-insensitive to allow for matches to twiki.org (for example) or simple typos.

The next several lines of Perl code perform various transformations to excape special characters and protect them from the shell. FInally, the line is prefaced by a call to twerp and written to an output file, ready to be reposted to Twitter.

At the end of the processing stage, if any output has been written (that is, if any friends' tweets mentioned TWiki!) sh -r is called to activate twerp and post the messages. I use -r, the "restricted" shell, just in case anything "weird" is found in one of the tweet strings. This makes our local system administrators much happier. :-)

Ten minutes later, the cycle repeats.

-- Contributors: VickiBrown - 07 Jul 2008

Discussion

Cool. Thanks for this effort.

-- MartinSeibert - 08 Jul 2008