Tags:
create new tag
, view all tags

Bug: INCLUDE of URLs does not work with https servers

Included URLs from a secured web server are not interpreted correctly. The /s in the URL are replaced by . (dots) and the

%INCLUDE{"URL"}
statement fails with a message: "Note: Included URL: https..url.some.domain.com does not yet exists

Test case

Try to reference the same URL on a standard http and a secure https address.

Environment

TWiki version: TWikiRelease02Sep2004
TWiki plugins: DefaultPlugin, EmptyPlugin, InterwikiPlugin
Server OS: RedHat Linux 7.3.4
Web server: Apache 2.52
Perl version: v5.6.1
Client OS: RedHat Enterprise Linux
Web Browser: Any

-- NilsHoeimyr - 15 Feb 2005

Follow up

Off topic, but could I ask what form you used to create this topic? It was created using the old style WebForm and template, whereas we've now switched to ChangeProposals. I though I had tracked down and removed all the old submission forms.

(I've switched the form to ChangeProposalForm)

-- SamHasler - 15 Feb 2005

Hmm, I submitted the bug via: http://twiki.org/cgi-bin/view/Codev/BugReport

-- NilsHoeimyr - 18 Feb 2005

ok, probably your browser cached the page then.

-- SamHasler - 18 Feb 2005

Hmmm - you should never have got that far! http: is the only protocol supported for %INCLUDE, and the string "does not yet exists" is nowhere in the codebase. You should have seen the message:

Error: Unsupported protocol. (Must be 'http://domain/...')

Are you using a hacked TWiki?

-- CrawfordCurrie - 18 Feb 2005

The warning text comes from TWiki/TWikiPreferences:

TWiki/TWikiPreferences.txt: * Set INCLUDEWARNING = Note: Included topic $topic does not exist yet

-- NilsHoeimyr - 24 Mar 2005

speaking of hacking twiki.

what is stopping us from using LWP for our web gets? what is the benifit of low level socket requests?

speaking of hacking - i will have a crack at a https include method something like %INCLUDE and let you know how i go.

-- TerryRankine - 28 Apr 2005

Just the additional burden of another CPAN dependency, AFAIK. If you want to argue the toss, there are a number of other CPAN modules that could be used to replace other chunks of TWiki (for example, CPAN:CGI::Kwiki hehe! )

-- CrawfordCurrie - 28 Apr 2005

so - back to being "constructive" about the issue.

We dont want CPAN dependency? Where does that leave us with https includes? I dont know enough to do https socket level calls. I also wonder if its realy worth re-inventing the wheel - cos once you invent it you have to maintain it.

-- TerryRankine - 05 May 2005

Crawford: Why do you think CPAN modules are a burden? In my opinion it is much better to use existing and well tested code instead of wasting time with reinventing the wheel and find all pitfalls again.

Since our intranet runs on a secure server and we want to include text of attachments in some topics, we urgently need the possibility of including https://... pages.

-- JChristophFuchs - 23 Jun 2005

I wrote my own patch to fix this problem. I had to change two files in the lib directory: TWiki.pm and TWiki/Net.pm. patches are included below. It will not work for https:// URLs that include usernames/passwords because I didn't need that feature, but it could easily be added. This was a quick hack and I'm not suggesting that this is the best way this could be done -- it works in my case though.

% diff -u Net.pm Net.pm.ssl
--- Net.pm      Sun Mar 19 13:55:06 2006
+++ Net.pm.ssl  Sun Mar 19 13:54:12 2006
@@ -114,6 +114,18 @@
     return $result;
 }

+sub getSecureUrl
+{
+    my ( $theHost, $thePort, $theUrl, $theUser, $thePass, $theHeader ) = @_;
+
+    use LWP;
+    require HTTP::Request;
+    my $request = HTTP::Request->new(GET => "https://$theHost$theUrl");
+    my $ua = LWP::UserAgent->new;
+    my $response = $ua->request($request);
+    return $response->as_string;
+}
+
 # =========================
 =pod



% diff -u TWiki.pm TWiki.pm.ssl
--- TWiki.pm    Sun Mar 19 13:55:01 2006
+++ TWiki.pm.ssl        Sun Mar 19 13:54:22 2006
@@ -1909,6 +1909,7 @@
     my $path = "";
     my $user = "";
     my $pass = "";
+    my $https = 0;

     # For speed, read file directly if URL matches an attachment directory
     if( $theUrl =~ /^$urlHost$pubUrlPath\/([^\/\.]+)\/([^\/\.]+)\/([^\/]+)$/ ) {
@@ -1956,12 +1957,45 @@
         $host = $1;
         $path = $2;

+    } elsif( $theUrl =~ /https\:\/\/(.+)\:(.+)\@([^\:]+)\:([0-9]+)(\/.*)/ ) {
+       $https = 1;
+        $user = $1;
+        $pass = $2;
+        $host = $3;
+        $port = $4;
+        $path = $5;
+
+    } elsif( $theUrl =~ /https\:\/\/(.+)\:(.+)\@([^\/]+)(\/.*)/ ) {
+       $https = 1;
+        $user = $1;
+        $pass = $2;
+        $host = $3;
+        $path = $4;
+        $port = 443;
+
+    } elsif( $theUrl =~ /https\:\/\/([^\:]+)\:([0-9]+)(\/.*)/ ) {
+       $https = 1;
+        $host = $1;
+        $port = $2;
+        $path = $3;
+
+    } elsif( $theUrl =~ /https\:\/\/([^\/]+)(\/.*)/ ) {
+       $https = 1;
+        $host = $1;
+        $path = $2;
+       $port = 443;
+
     } else {
-        $text = showError( "Error: Unsupported protocol. (Must be 'http://domain/...')" );
+        $text = showError( "Error: Unsupported protocol. (Must be 'http://domain/...' or 'https://domain/...')" );
         return $text;
     }

-    $text = &TWiki::Net::getUrl( $host, $port, $path, $user, $pass );
+    if ($https) {
+       $text = &TWiki::Net::getSecureUrl( $host, $port, $path, $user, $pass );
+    } else {
+       $text = &TWiki::Net::getUrl( $host, $port, $path, $user, $pass );
+    }
+
     $text =~ s/\r\n/\n/gs;
     $text =~ s/\r/\n/gs;
     $text =~ s/^(.*?\n)\n(.*)/$2/s;
@@ -1972,10 +2006,17 @@
     }
     if( $contentType =~ /^text\/html/ ) {
         $path =~ s/(.*)\/.*/$1/; # build path for relative address
-        $host = "http://$host";   # build host for absolute address
-        if( $port != 80 ) {
-            $host .= ":$port";
-        }
+       if (!$https) {
+           $host = "http://$host";   # build host for absolute address
+           if( $port != 80 ) {
+               $host .= ":$port";
+           }
+       } else {
+           $host = "https://$host";   # build host for absolute address
+           if( $port != 443 ) {
+               $host .= ":$port";
+           }
+       }
         $text = cleanupIncludedHTML( $text, $host, $path );

     } elsif( $contentType =~ /^text\/(plain|css)/ ) {
@@ -2013,7 +2054,7 @@
     my $rev     = extractNameValuePair( $theAttributes, "rev" );
     my $warn    = extractNameValuePair( $theAttributes, "warn" );

-    if( $incfile =~ /^http\:/ ) {
+    if( $incfile =~ /^https?\:/ ) {
         # include web page
         return handleIncludeUrl( $incfile, $pattern, $theWeb, $theTopic );
     }

-- ChristopherTracy - 19 Mar 2006

Interesting discussion.

Why not use CPAN? Speed is a very good reason. Using general generic libs that can do everything will often execute slower than a small piece of optimized code that does exactly what you need.

It may make sense to use LWP for secure includes but I doubt it makes sense to do it for non-secure URLs.

-- KennethLavrsen - 19 Mar 2006

The patch above is not for Dakar, right? Ok, maybe I find time to provide a patch agains DakarRelease 4.0.1 in the next hours. I have the feeling, that some of the constructs could be written more compact and generic. smile Dakar's http code seems to be more clean than the (older) one, you have derived your https version from...

-- TobiasRoeser - 19 Mar 2006

Yep, that patch was for TWiki20040904, sorry. The host/path stuff definitely sucks. smile

-- ChristopherTracy - 20 Mar 2006

Ok, I created a patch agains the most current stable TWiki version 4.0.1. This version is only tested on my own site to include files like ChangeLogs and TODO lists from my https subversion webdav access. Feel free to copy and use it. And please, please, make this functionality part of TWiki.

Please test before using this patch on a production system (like me smile ). I didn't proof read the code a second and third time.

-- TobiasRoeser - 20 Mar 2006

Thanks Tobias, I filed Bugs:Item1933.

-- PeterThoeny - 22 Mar 2006

Fix record

Topic attachments
I Attachment History Action Size Date Who Comment
Unknown file formatpatch twiki-4.0.1-include-https.patch r1 manage 4.0 K 2006-03-20 - 11:10 TobiasRoeser Patch for TWiki-4.0.1 enabling https support for INCLUDE
Edit | Attach | Watch | Print version | History: r17 < r16 < r15 < r14 < r13 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r17 - 2006-05-01 - TobiasRoeser
 
  • Learn about TWiki  
  • Download TWiki
This site is powered by the TWiki collaboration platform Powered by Perl Hosted by OICcam.com Ideas, requests, problems regarding TWiki? Send feedback. Ask community in the support forum.
Copyright © 1999-2017 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.