Feedback on SourceHighlightPlugin

-- ChrisWinters - 23 Jan 2003

Cool, so this is another implementation compared to SyntaxHighlightingPlugin, right?

-- GrantBow - 23 Jan 2003

it cannot create the required temporary files. The bug seems to be :

    my $tmp_dir  = File::Spec->tmpdir();
    my $tmp_file = File::Spec->catdir( $tmp_dir,
                                       mktemp( 'twikishp-XXXXXX' ) );
    eval { open( TMP, "> $tmp_file" ) || die $! };
    if ( $@ ) {
        return error_msg( $data,
                          "Failed to open working file",
                          "Failed to open temp file [$tmp_file] for writing: $@" );
    }

    my $tmp_file = mktemp(File::Spec->catdir( $tmp_dir, 'twikishp-XXXXXX' ) );

As it it only works if the user the web server is running as has write permissions in the current directory.

Other than that, looks nice.

-- MartinFuzzey - 19 Mar 2003

I suggest a little change so to be able to define a DIV STYLE via a TWiki plugin variable. This makes the Highlight plugin working fine with any SKIN, no need to edit .css files

Here are changes I made on plugin itself :

• added line 042 : my $FORMAT_STYLE = undef;
• added line 093 : $FORMAT_STYLE = TWiki::Prefs::getPreferencesValue( "${key}_FORMATSTYLE" );
• changed line 171 : qq(<div class="$FORMAT_CLASS" $FORMAT_STYLE>\n),
• changed line 182 : qq(<div class="$FORMAT_CLASS" $FORMAT_STYLE><pre>$data</pre></div>) );

* or just define the STYLE to use

• • Set FORMATSTYLE = STYLE="{background : #FFFFCC; border : 1px solid #CCCCCC; margin-left : 2em; margin-right : 2em; padding : 4px;}"

Really a nice plugin, hope you will appreciate my little contribution

-- PatrickNomblot - 02 Apr 2003

I have succeeded in getting this plugin to work under Windows 2K (Apache and ActiveState perl). I obtained the source-highlight binaries from the GnuWin32 project at SourceForge. There are two issues to fix to get this plugin working for this platform:

1. the regexp that checks the binary filename needs to accept an optional .exe suffix
2. the temporary filename is tainted and has to be de-tainted for open() to work.

--- SourceHighlightPlugin.pm.orig   Thu Jan 23 10:23:00 2003
+++ SourceHighlightPlugin.pm   Fri Jun 06 14:04:04 2003
@@ -70,8 +70,9 @@
     # This should be set to something like
     # /usr/local/bin/source-highlight or source-highlight-cgi
 
-    my $bin = TWiki::Prefs::getPreferencesValue( "${key}_BINARY" );
-    unless ( $bin and $bin =~ /source\-highlight(\-cgi)?$/ and -x $bin ) {
+    my $bin = TWiki::Prefs::getPreferencesValue( "${key}_BINARY" )
+      || 'c:/opt/bin/source-highlight.exe';
+    unless ( $bin and $bin =~ /source\-highlight(\-cgi)?(.exe)?$/ and -x $bin ) {
         TWiki::Func::writeWarning( "Invalid source-highlight binary [$bin]" );
         return 0;
     }
@@ -131,8 +132,11 @@
     my $tmp_dir  = File::Spec->tmpdir();
     my $tmp_file = File::Spec->catdir( $tmp_dir,
                                        mktemp( 'twikishp-XXXXXX' ) );
+    # We have to detaint this filename.
+    if ($tmp_file =~ /^(.+)$/) { $tmp_file = $1; }
     eval { open( TMP, "> $tmp_file" ) || die $! };
     if ( $@ ) {
+        $DEBUG && _w("failed to open working file: $@");
         return error_msg( $data,
                           "Failed to open working file",
                           "Failed to open temp file [$tmp_file] for writing: $@" );
@@ -146,6 +150,7 @@
     my $this_cmd = sprintf( $COMMAND, $lookup_lang );
     eval { open( CMD, "$this_cmd < $tmp_file |" ) || die $! };
     if ( $@ ) {
+        $DEBUG && _w("source-highlight error: $@");
         return error_msg( $data,
                           "Cannot open pipe to program",
                           "Failed to open pipe to command [$this_cmd]" );

-- PatThoyts - 06 Jun 2003

File::Spec is present on my intranet's server (FreeBSD 4.8-RELEASE), but File::Temp is not. I'm not root, so I can't install it. Since the only two calls for this pkg are tmpdir and mktemp, I think a substitute can be written. (Though since the plugin isn't working for me--perhaps due to another reason, I won't suggest a fix of my own.)

-- JonathanCline - 13 Oct 2003

You can install CPAN modules without being root.

-- MartinCleaver - 13 Oct 2003

checked .zip into CVS

-- WillNorris - 19 Jul 2005

Does this plugin work with TWiki 4.0.3?

-- EricHanson - 30 Jun 2006

You can remove the whole temp file requirnment using IPC::Open2;

@@ -125,47 +128,25 @@
                           "Unable to handle '$lang' syntax. Please use one of $valid_lang" );
     }

-    # Create a temporary file to hold the data; we clean it up before
-    # the routine ends
+    my @lines;
+    eval {
+        use IPC::Open2;
+        my $this_cmd = sprintf( $COMMAND, $lookup_lang );
+        my ($readFH, $writeFH);
+        my $pid     = open2($readFH, $writeFH, $this_cmd);

-    my $tmp_dir  = File::Spec->tmpdir();
-    my $tmp_file = File::Spec->catdir( $tmp_dir,
-                                       mktemp( 'twikishp-XXXXXX' ) );
-    eval { open( TMP, "> $tmp_file" ) || die $! };
-    if ( $@ ) {
-        return error_msg( $data,
-                          "Failed to open working file",
-                          "Failed to open temp file [$tmp_file] for writing: $@" );
-    }
-    print TMP $data;
-    close( TMP );
+        select((select($writeFH), $| = 1)[0]);
+        print $writeFH $data;
+        close( $writeFH );

-    # Send the temp file data to the source-highlight command,
-    # capturing the output in @lines
+        @lines = <$readFH>;
+    };

-    my $this_cmd = sprintf( $COMMAND, $lookup_lang );
-    eval { open( CMD, "$this_cmd < $tmp_file |" ) || die $! };
-    if ( $@ ) {
-        return error_msg( $data,
-                          "Cannot open pipe to program",
-                          "Failed to open pipe to command [$this_cmd]" );
-    }
-    my @lines = <CMD>;
-    close( CMD );
+    my $out = join( '', @lines );
+    $out =~ s/^[\r\n]*(.*)[\r\n]*$/$1/s;
+    $out =~ s/<tt>[\r\n]/<tt>/s;

-    unlink( $tmp_file )
-                    || TWiki::Func::writeWarning( " - SourceHighlightPlugin Error: cannot delete temp file [$tmp_file]: $!" );
-
-    # Cosmetic: Get rid of lines 1 and 2 (0-based), plus the
-    # next-to-last line
-
-    splice( @lines, 1, 2 );
-    splice( @lines, scalar @lines - 2, 1 );
-
-    return join( '',
-                 qq(<div class="$FORMAT_CLASS">\n),
-                 @lines,
-                 qq(\n</div>));
+    return "<div class='$FORMAT_CLASS'>$out</div>";
 }

 sub error_msg {

-- DanielCheng - 08 Aug 2006

When I try enableing this on the latest T4, I get:

Can't use string ("SOURCEHIGHLIGHTPLUGIN_BINARY") as a HASH ref while "strict refs" in use at /usr/local/wiki-var/lib//TWiki/Prefs.pm line 241.

Anyone had any luck getting it to run on T4?

-- EricHanson - 09 Aug 2006

I'm having the exact same problem as DanielCheng. This is after installation and setting the pref variables.

-- PaschalNee - 14 Aug 2006

To the Plugin maintainer: This Plugin does not work in TWiki 4.0. Please consider upgrading it so that it runs on Cairo and Dakar codebase. HandlingCairoDakarPluginDifferences has more.

Anyone with interest in the Plugin is invited to fix it, the Plugin has a PleaseFeelFreeToModify policy.

-- PeterThoeny - 16 Aug 2006

Not sure where to put this. This is an updated SourceHighlight.zip that will work with Twiki 4.0. It has only the following changes:

in SourceHighlightPlugin.pm changed: &TWiki::Prefs::getPreferencesValue to: TWiki::Func::getPreferencesValue

-- SotiriosTsongas - 01 Nov 2006

Thanks Sotirios! Please overwrite the zip file attached to the SourceHighlightPlugin topic (same name; do not add version number to name of zip file.)

-- PeterThoeny - 01 Nov 2006

The ability to supply the binary to execute seems like a serious security hole to me! What is to stop a malicious user from making a program that could be executed in this plugin in place of the real source-highlight?

There is no reason to allow this - for a given machine, this information should be known and should be fixed to a specific location. If necessary the installer should change the path appropriately, but it should NOT be configurable using any form of variable.

I removed this "feature" and hard-coded the path for our install.

-- DarrenElkerton - 03 Nov 2006

Also, for correct operation, I would suggest making the plugin using the beforeCommonTagsHandler instead of commonTagsHandler .

The reason is to ensure that the source highlighting is performed and the code within preserved BEFORE the contents are processed to ensure that anything that looks like a wiki tag in the embedded code segment is NOT expanded.

Ideally, a code section should be entirely preserved (i.e. NO processing should be performed on it).

-- DarrenElkerton - 03 Nov 2006

I'm using this plugin on TWiki 4.0.5 GNU source-highlighter 1.11-2 on Ubuntu Server "Dapper Drake". I find that lines are cut off from the source files. For instance, if I include the code below I don't see the line #include <stdio.h>

The code is:

%CODE{"cpp"}%

#include <stdio.h>
#define stringify(x) #x

int main() {
    int harold = 4;
    printf("%s=%d\n", stringify(harold), harold);
    return 0;
}
%ENDCODE%

I also don't see the final } in twiki/bin/view/TWiki/SourceHighlightPlugin

-- HaroldShip - 06 Nov 2006

Harold,

The default action of the plugin is to remove the first two lines and the next to last line of the output from source-highlight (supposedly to pretty it up by removing extranous output about the author).

Perhaps the default behaviour of source-highlight changed (not sure what version I'm using, but I also created a different output format for our site).

Take a look at SourceHighlightPlugin.pm around line 162 or so for the following two lines and try removing them and see what your output looks like.

    splice( @lines, 1, 2 );
    splice( @lines, scalar @lines - 2, 1 );

I heavily modified the implementation on our server to make the output suit my needs so I can't remember what it should look like, but that may give you an idea of what's going on.

Followup - I checked our installation and we are using V2.4 of source highlight - try using a later version than the one you have - the behaviour of the plugin may be customised to the version of source-highlight.

-- DarrenElkerton - 09 Nov 2006

Is this plugin better than the BeautifierPlugin?

-- MiloValenzuela - 21 Nov 2006

Hi togehter, I am actually working on SyntaxHighlightingPlugin. As Milo asked in his question "Is this better" I wonder that so many from us do the same thing on a different way. So for this I think we have BeautifierPlugin, SourceHighlightPlugin and SyntaxHighlightingPlugin. Wouldn't it be better to just work on one Plugin. I choosed SyntaxHighlightingPlugin because it use the well know enscript code formatter and if I get it right, enscript could handle a lot more code types than this and BeautifierPlugin. Could someone please give me his suggestions why I should choose this or BeautifierPlugin. Maybe we could meet once in one space

-- ThomasFreudenberg - 22 Nov 2006

Harold, I have realized, that with calling eval { run ... } there is sometimes a problem when source is send to enscript. If have recognized this while working on the latest Release of SyntaxHighlightingPlugin. Maybe you try this Plugin, its compatible in Syntax so you could call it YetAnotherSourcehighlightPlugin .

-- ThomasFreudenberg - 24 Nov 2006

I switched from BeautifierPlugin to this because I thought it produced prettier syntax highlighting. It wasn't playing well with whitespace and other twiki markup, however, so I made some changes. Here's a patch I made with SVK:

==== Patch <patch> level 1
Source: [No source]
Target: 8f3e6c68-2be0-46b6-b1c9-40c738eef9e6:/cgi-bin/lib/TWiki/Plugins:353 [local]
Log:
fixed some problems with whitespace
=== SourceHighlightPlugin.pm
==================================================================
--- SourceHighlightPlugin.pm    (revision 353)
+++ SourceHighlightPlugin.pm    (patch patch level 1)
@@ -105,7 +105,7 @@
 sub commonTagsHandler {
 
 # &TWiki::Func::writeDebug( "- SourceHighlightPlugin::commonTagsHandler( $_[2].$_[1] )" ) if $debug;
-    $_[0] =~ s/%CODE({"([a-z0-9]+)"})?%(.*?)%ENDCODE%/&render_code($2, $3)/gseo;
+    $_[0] =~ s/(?:^\s*$)*%CODE(?:{"([a-z0-9]+)"})?%\s*(.*?)\s*%ENDCODE%/&render_code($1, $2 . "\n\n")/mgseo;
 }
 
 sub render_code {
@@ -170,7 +170,7 @@
     splice( @lines, 1, 2 );
     splice( @lines, scalar @lines - 2, 1 );
 
-    return join( '', qq(<div class="$FORMAT_CLASS">\n), @lines, qq(\n</div>) );
+    return join( '', qq(\n<div class="$FORMAT_CLASS">\n), @lines, qq(</div>\n) );
 }
 
 sub error_msg {

-- DavidHoughton - 12 Mar 2008

There are a few bugs in this plugin. After some fix, it works perfect.

• First, there are one extra line before you code. This is cause by source-highlight itself. It adds

  <tt></tt>

  to a html output by default. So edit /usr/local/shard/source-highlight/html.outlang, eliminate them.
• Second, the last bracket is missing if you write %ENDCODE% right after the code. I don't know why, but I fixed it by adding a new line break in SourceHighlight.pm. Find print TMP $data; and add print TMP '\n'; right after it.
• Third, the default language is not working. The DEFAULTLANG in SourceHighlightPlugin should be changed to DEFAULTLANGUAGE. Now you can use %CODE{""}%. But the brackets are nonsense here, so I want to eliminate them and use %CODE%. Find

sub commonTagsHandler {
    # &TWiki::Func::writeDebug( "- SourceHighlightPlugin::commonTagsHandler( $_[2].$_[1] )" ) if $debug;
    $_[0] =~ s/%CODE({"([a-z0-9]+)"})?%(.*?)%ENDCODE%/&render_code($2, $3)/gseo;
}


sub render_code {
    my ( $lang, $data )  = @_;

    # Ensure the command is defined

and change it to

sub commonTagsHandler {
    # &TWiki::Func::writeDebug( "- SourceHighlightPlugin::commonTagsHandler( $_[2].$_[1] )" ) if $debug;
    $_[0] =~ s/%CODE(.*?)%(.*?)%ENDCODE%/&render_code($1, $2)/gseo;
}


sub render_code {
    my ( $lang, $data )  = @_;

    if ($lang =~ m/{"(.*)"}/) {
        $lang = $1;
    }
    else {
        $lang = $DEFAULT_LANG;
    }

    # Ensure the command is defined

-- FengZhaolin - 10 Feb 2009

Thank you Feng for updating the plugin!

-- PeterThoeny - 10 Feb 2009

I took Feng's version into SVN and updated the docs.

To-do for whoever wants to give this plugin some TLC:

• Use a configure variable for the BINARY setting (to plug security hole)
• Add a VarCODE variable doc topic

-- PeterThoeny - 2010-08-07