Question

I'm having a problem under MacOSX 10.1.4 similar to that described in WhyDoLinesComeOutDoubleSpaced. Just for fun, I installed TWiki on my mac and imported a set of Unix text files by putting them in the appropriate web directories. After I edit them, they come out with ^M's at the end of each line when I view them in Emacs (which will recognize normal Mac, DOS and Unix format files and display them properly). It seems that somehow, between TWiki and Apache for OSX, my files have gotten into a fourth state.

TWiki doesn't show double lines or anything like that when the pages are served, so if you started fresh, you'd never know about the underlying problems just from looking. But you would know when you tried to edit the pages - I noticed that my first edits weren't checking in properly, though subsequent ones did - apparently, once the file is fully munged with that CF or LF, TWiki is happy with it. I've also had intermittent problems saving pages that I traced back to the first edit of a Unix file. When that happens, I get the preview, but the save itself doesn't take effect - the old page comes up instead.

I don't think there's anything in the error log about the problem, but I can check again. I'm hoping someone can give me some insight into how the code snippets posted in WhyDoLinesComeOutDoubleSpaced work, so that I'll know what to try tweaking.

• TWiki version: 01Dec2001
• Web server: Apache (the version included with OSX)
• Server OS: MacOSX 10.1.4
• Web browser: Opera for MacOSX
• Client OS: MacOSX 10.1.4

• Note: I installed everything in a /home/httpd/twiki directory on an HFS+ partition on the server. (Unfortunately, I don't have a UFS partition to experiment with.) I tried the cgi-bin installation approach with a user directory, but there seemed to be some sort of OSX permission issue (beyond the obvious stuff in httpd.conf) keeping me from running like that.

-- MaryDeMarco - 25 Apr 2002

Answer

I don't have a Mac, but see MacOS for some other links, and do have a go at the Sherlock plugin for TWiki at SherlockAndMozillaSearching (I'd like some feedback from Mac users!).

Also, see ForbiddenError - this rather bizarre problem was linked to using HFS instead of UFS.

-- RichardDonkin - 25 Apr 2002

HFS+ is the standard file system for OSX. A user would have to reformat their entire hard drive to get a UFS partition under MacOSX, or figure out how to get fdisk and do it that way. (I think it involves installing Linux.) Needless to say, I'm not going to reformat the drive over a few invisible linefeeds. "Use UFS" is not a workable solution for most people.

Either I have a new ForbiddenError, or the ForbiddenError is actually specific to IE. I've had no permission problems with Opera or Netscape. (It was only by chance that I opened IE at all and spotted it.)

But that's a whole other bug. I was really just curious about what exactly TWiki is supposed to be doing in those lines that add or strip LF's and CR's, so I'll know what the dangers of hacking them are.

-- MaryDeMarco - 26 Apr 2002

I wasn't saying you should switch to HFS, just pointing out a related bug... I don't really have any other suggestions, other than to try searching for binmode on Codev to find some possibly related topics.

-- RichardDonkin - 26 Apr 2002

> I installed everything in a /home/httpd/twiki directory on an HFS+ partition on the server.

Might be a bit OT, but I had to do the same. Seems to be a general Problem of MacOSX.

-- ArneBab - 13 May 2002

I've been reading up on line feed bugs in Codev and now I wonder whether this is more an Opera problem than Apache on OSX. Since IE won't even open TWiki pages on localhost, I'll have to try a few other browsers to check this out. More to follow...

later: After reading KfmBrowserSupportForEditing, OperaBrowserDoublesEndOfLines and WhyDoLinesComeOutDoubleSpaced, I tried a few browsers and decided that all the OSX browsers couldn't be buggy. Also, the Opera bug fix in the TWiki.pm code for encodeSpecialChars looked like it really ought to be working. On the other hand, it looked surprisingly like decodeSpecialCharacters was converting all my TWiki text to DOS format.

sub decodeSpecialChars
{
    my( $text ) = @_;
    
    $text =~ s/%_N_%/\r\n/go;

A short review for anyone as clueless as I was when I started having this problem: DOS (Windows) line endings are of the form \r\n. Unix uses \n and classic Mac uses \r. Mac in a Unix mood uses, appropriately, \n. This doesn't have anything to do with the volume format (HFS+ vs. UFS), but the program that creates the file.

My disturbing TWiki experience was of trying to import Unix text files into my MacOSX TWiki - TWiki would choke on them the first time it tried to edit one, then at about the second try, it succeed in munging the file into a mystery format. I now understand the mystery format, but I do not understand why the initial conversion usually failed. (The preview would show, but after the save, the unedited text was still there.)

One might expect, from decodeSpecialChars, that TWiki was saving a pure DOS file, but that was not the case. As my experience peeking at the TWikified files in Emacs revealed, TWiki had saved them in a mixed format. The actual text had DOS line endings, but the metadata at the top retained the native Unix line endings. Emacs apparently checked the first set of line endings, for the metadata, identified the file as a Unix file, then spat out the familiar ^M for each \r it subsequently encountered.

As I mentioned earlier, once all the files had been munged to its satisfaction, it ran like a normal TWiki. Someone who wasn't peeping at the files would never have noticed a problem. The reason I was peeking at the files was that I wanted to import some stuff I already had under RCS, in Unix format, that I prefer to edit using Emacs but that I thought would be easier to look at using TWiki (especially for the nice color diffs). So I didn't want TWiki munging these files.

So, the unmunge is very simple. In decodeSpecialChars, replace

$text =~ s/%_N_%/\r\n/go;

with

$text =~ s/%_N_%/\n/go;

So far, that's worked perfectly. I can now import unix files without errors, and they remain unix files after editing. Of course, this has only been tested with Mozilla and Opera, because IE doesn't work at all.

Rather than do a trick with the edit script to gradually un-DOS all the remaining data files, I used a perl-based file converter, nncp, specifically dos2unix. That fixed the RCS files as well. http://www.cs.wright.edu/~jslater/nnfc/Readme.html

I guess what I don't understand is why no one has had this problem with other Unixes. The initial file-conversion problem seems like it could be OSX specific, but the general problem of half-DOS/half-Unix files ought to be showing up in Linux, too. Or am I missing something?

-- MaryDeMarco - 29 May 2002

It turns out my fix is not working so perfectly. I still have my original problem - that the save fails on the first try with a new, imported text file. My change doesn't seem to have made it any worse, though, just 'freshened' the files somehow so that I'm seeing it again. I wonder if it's an RCS problem.

-- MaryDeMarco - 05 Jun 2002