Our Subversion line endings are wrong

Abstract

cvs2svn has a bug which creates files that have svn:eol-style set, but does not have newline (NL) endings.

Details

When svn:eol-style is set, the files in the repository must be NL only. The Working Copy (WC) lib will convert those NL:s to CRNL when you checkout, and the client lib will convert them when you export.

This confuses SVN in a number of ways.

Impact

1. repository will need to be taken offline briefly and repaired
2. developers will have to checkout new WC's

Procedure

SVN Server Repository

1. stop the server
2. create a new empty repository
3. svn dump old-repos | nice-filter | svn load new-repos
4. mv new-repos old-repos
5. start server

(clean this up and insert more details here)

Developers

1. check out new WC's
   • changes not checked in will need to be migrated to the new WC using cd new-wc; svn diff ../old-wc | patch -p0

IRC chat log with SVN developers

(12:10:52) MartinCleaver: bliss: any idea why would I get "svn: Error restoring text" 
during "svn revert"?
(12:18:27) MartinCleaver: but this is after a merge, which seems to have replaced many files with a 
back to back copy of the file: i.e., the old version followed by the new (or vice-versa)
(12:22:55) MartinCleaver: deleting the file and doing  a "svn up" I get:
(12:22:56) MartinCleaver: svn: Inconsistent line ending style

(12:23:22) bliss: was this repository converted from cvs using cvs2svn?
(12:23:27) MartinCleaver: yup
(12:23:59) bliss: we can fix this
(12:25:20) bliss: the proper fix requires a dump+load and will make you have to recheckout the wc:s :-(
(12:28:32) bliss: the contents of the files will not be changed
(12:28:46) bliss: not the ones in the wc:s or in exports
(12:28:57) bliss: only the ones in the repository and the text-base dirs
(12:29:12) bliss: and therefore the checksums (which is why you need to recheckout)

(12:31:14) bliss: when svn:eol-style is set, the files in the repository must be LF only. 
The WC lib will convert those LF:s to CRLF when you checkout, and the client lib will 
convert them when you export.
(12:32:09) bliss: the problem is that there is a bug in cvs2svn which
creates files that have svn:eol-style set, but does not have LF-endings.
(12:32:24) bliss: this confuses svn in a number of ways

(12:35:27) bliss: <CDot> Is there a way to "switch off" the svn:eol-style property?
(12:34:37) bliss: CDot: Yes, you can turn it off but that does not
help the old revs which will cause merge problems.
(12:36:23) bliss: but it will invalidate your wc:s, so you will have
to recheckout after the repos is fixed
(12:36:32) ***MartinCleaver thinks: this is gonna be messy
(12:36:35) bliss: or you will get MD5 errors
(12:37:11) MartinCleaver: better now than later though - just needs care to notify everyone

(12:37:15) bliss: you will have to do something like this:
(12:37:27) bliss: 1. stop the server
(12:37:34) bliss: 2. create a new empty repository
(12:37:57) bliss: 3. svn dump old-repos | nice-filter | svn load new-repos
(12:38:08) bliss: 4. mv new-repos old-repos
(12:38:11) bliss: 5. start server
(12:38:35) bliss: people that try to commit *may* get MD5 errors
(12:39:03) CDot: if they try to reuse existing wcs
(12:39:08) bliss: if they do, they have to checkout a new wc and
transfer their changes to the new wc using "svn diff ../old-wc | patch -p0"

(12:40:12) bliss: you need the filter, though
(12:40:26) MartinCleaver: what do you suggest?
(12:40:56) bliss: I have written a simple filter that should work, but
it's not heavily tested
(12:41:18) bliss: but I've heard of another tool that can do this too
(12:41:50) CDot: what does it do?
(12:42:15) bliss: It replaces the line endings of all files with 
svn:eol-style set with LF only
(12:42:47) CDot: ok. sounds simple enough!
(12:44:15) bliss: and you will not have to edit the script - just use it

(12:54:01) bliss: I've seen this bug strike twice now, so I'll
initiate a move to fix it and to inform the cvs2svn users.

(12:49:07) bliss: I wonder how those CRLF:s got in there. Did you use
-kb with CVS for the CRLF files?
(12:49:47) MartinCleaver: its TWikis (misguided) file format
(12:50:08) MartinCleaver: CDot and I were just debating changing it
(12:57:14) bliss: I understand that the problematic files must use CRLF EOLs.

(12:57:40) bliss: How did you make CVS do that? Did you use -kb to
force CVS to treat them as binary?
(12:57:52) bliss: e.g. using cvswrappers (or whatever that file's called)

(13:02:16) bliss: can you try an "svn dump repos | fix-eols.py > /dev/null" and see if it breaks?
(13:03:22) bliss: I just wanted an early test to verify that fix-eols.py survives.
(13:03:51) bliss: I'm about to add detection code to it so you can get
a report of which files that has a problem.

(13:07:23) MartinCleaver: Ok. It seems if I copy the .svn-base file to
replace my wc, and then run dos2unix I can then do svn up
(13:07:47) MartinCleaver: would cause me problems later?

(13:16:48) ***MartinCleaver discovers that "cp $FILE $FILE- && cat
.svn/text-base/$FILE.svn-base > $FILE && dos2unix $FILE && svn up
$FILE && svn resolved $FILE" more or less lets him proceed
(13:17:51) MartinCleaver: it does however leave the file in unix mode

(14:07:19) bliss: MartinCleaver: take a look at this:
http://queen.borg.ch:81/svn/repos/trunk/svn/svndumptool/
(14:08:13) MartinCleaver: cool - what does it do?
(14:08:54) bliss: many things, including eol fixup
(14:09:11) bliss: at least it promises to. I've never used it.

Feedback

Could you summarize on top what the issue is, and also set the classification? Thanks.

-- PeterThoeny - 10 Dec 2004

i started that task (as well as editting the irc log), but there's still more to tease out of it...

-- WillNorris - 10 Dec 2004

> Here's an update version of fix-eols.py that should fix all errnoneous
> CRNL occurances. I recommend that you run both my tool and
> svndumptool.py [1] and compare the results just to be sure. There should
> be no differences.
>
> /Tobias
>
> [1] http://queen.borg.ch:81/svn/repos/trunk/svn/svndumptool/

-- Tobias Ringström (from email to MartinCleaver)

Tobias kindly wrote for us the attached script fix-eols.py.txt. My understanding is that only someone with filesystem access to the server can address the issue and that Sven is looking after it.

-- MartinCleaver - 11 Dec 2004

Thanks. It's obviously too late for you, but you might be interested to know that I've fixed the cvs2svn bug (r1398), and that we will be making a new release soon.

There is a typo in the Wiki that is entirely my fault. During the entire discussion, I managed to talk about NLs, but there is no such thing. The proper name is of course LF (Line Feed). I don't know where I got that from.

/Tobias

• Updated conv above to refer to LF. -- MC

-- Tobias Ringström (from email to MartinCleaver)