create new tag
, view all tags
<And now a "checkpoint save" from my offline editing.>

<Now copied back offline for local editing (based on Rev. 1.4>

<Simply copying my "offline" start here for editing "off site">

I'm now on my third iteration of "thinking this through on paper". That's alright (I don't regret the effort for a number of reasons), including providing a hint of how some people (me) might approach a problem — it's sometimes misleading (IMHO) to read a book or similar where it appears the author's first cut is the right solution. So, now someone can see that is not always the case. Also, keeping a record of my thought process (or at least a fair amount of it, I am thinking offline as well may be useful if I ever need to prove that I developed this somewhat independently. Of course, this third iteration is based on going back to Word97 to see how it works for the user, and I can see that their approach might make it reasonable for the programmer (me) as well.

<The fourth iteration>

This sounds OK (text moved, refers to the third iteration below, the Word97 approach), but there are times when I might prefer to just collapse the heading I expanded (or created) a few "moments" ago. (I.e., I expand a heading, do some work, now want to collapse that heading — is there some alternate (G)UI that will make that easier (without making the programming too hard)? I'll keep this in mind, it might become the fourth iteration, but, for now, back to the third iteration.

Ok, I thought through (??) the third iteration, I believe it can work, and I believe it is worth starting there, for a few reasons, even though this fourth iteration might be the most desirable for my "near term" purposes with my askSam for Linux thingie, because:

  • working out the range of the selection during a collapse is somewhat (difficult / tricky) — if I properly implement that code below, I can reuse it in this iteration
  • doing this fourth iteration requires depending on the operator to specify the level of the collapse (and, ideally, to match that of the overall file) or to incorporate some metadata in the file to keep track of the "collapse level"

On the other hand, maybe I should start coding a few of the simpler things, (expand, the change wrap macros), and a collapse macro for either case, then revisit this. For now, I'll make a checkpoint save back on WikiLearn.

<So here's the start of the third iteration:>

Aside: Oops, and after I got home, I went and fired up Word97 to see how they did some things. Basically, they work in what I'll call a hierarchical fashion: If you click expand on a given heading, it first shows the next lower level of subheadings, then the next lower, and so on, until, when no more subheadings remain, the text is expanded. Collapse is basically the inverse. I need to think about this as well, maybe this is a more appropriate approach.

So, let's "brainstorm" the Word97 approach:

Aside on terminology: I'm going to start referring to scanning forward or scanning backward in a file. I'm not sure what the most universally intuitive meaning of those terms is, and I'd sort of like to find out, but, for now, when I'm at a given place in a file and want to scan towards the beginning of the file, I'll refer to that as scanning backward. Scanning forward will be scanning toward the end of the file. (And this seems rather intuitive to me, but I can see the potential for confusion.)


  • Scan forward in the file to find the next \n
  • Scan backward in the file to find the previous \n
  • Select that portion of the document (which should just be the current line)
  • While at the front of the line, "capture" the heading markup for that line
  • Now scan through the selection in a loop, something like this:
    • Scan for headings with markup one level "lower" (<np>---+...),
      • if found, replace the <np> before them with \n
      • if not found, repeat the scan with then next lower level of markup * finally, if the lowest level of markup is not found (there are no headings to expand, potentially just text), replace all <np>s (in the selection) with \n's


  • Scan backward in the file to find the previous heading (preceded by a \n yes, ok (I could explore some alternatives, but try this first)
  • "Capture" this (first) heading markup
  • Continue to scan backward to find the next previous heading at a higher level than the just found heading (I don't think it matters whether it is preceded by a <np> or a \n)
  • "Capture" this (higher level) heading markup
  • Mark this location as the beginning of the selection
  • Scan forward in the file to find (hmm, this is tougher, could be some alternatives — start with one) — the first heading at the same or higher level than the last captured heading markup (aside, it is presumably preceded by a \n)
  • Mark this location as the end of the selection
  • Replace all instances of \n<the first heading markup> with <np><the first heading markup>

Hmm, that sounds workable, except when there are no headings, we are just collapsing text. How do we detect that? Oh, I see, maybe, making modifications above — after determining the selection, see if there are any \n's not followed by a heading — if so, replace all of those and quit, otherwise proceed further (to find headings to collapse).

And, oops, another modification, not sure we want to look to combine this with a previous higher level heading (Word doesn't) — "force" the user to move to that higher level heading and collapse from there.

So, based on the two needed modifications listed above, let's redo the collapse:


  • Scan backward in the file to find the previous heading (preceded by a \n yes, ok (I could explore some alternatives, but try this first)
  • "Capture" this (first) heading markup
  • Mark this location as the beginning of the selection
  • Scan forward in the file to find (hmm, this is tougher, could be some alternatives — start with one) — the first heading at the same or higher level than the first captured heading markup (aside, it is presumably preceded by a \n)
  • Mark this location as the end of the selection
  • If there are any uncollapsed text lines (\n not followed by a heading), collapse those (replace those \n's (not any others) with <np>s, and quit
    • otherwise, look for any lowest level uncollapsed headings, if so collape them and quit,
    • otherwise look for any next higher level uncollapsed headings, if so collapse them and quit
    • repeat above until you've done the level of the first captured heading (or one less??)

The logic (as written above) isn't immediately conducive to writing a loop, so I'll need to refactor, but I think the logic is close to correct.

<The following is the second iteration (until there is indication that I'm now discussing the first iteration, but it doesn't (currently) use the words "first iteration".>

Oops, just realized I need to think through (or test through) another point.

Background, I had just about decided to force the user to specify the level of folding to attain for a single paragraph (when folding). (This was after considering various ways of automatically determining the level of the collapse of the entire document, which is doable, but not "the simplest thing that could possibly work".)

The problem I recognize now is what do I (the macro) do if the user specifies a level that causes a need to revise the collapse level of the previous paragraph. For example, suppose:

  • The basic collapse state of the document is Level 3.
  • I insert a new paragraph (or section) with a heading markup of Level 4 (one answer is obviously, don't do that, and I need to come back and consider that answer more carefully, I mean the easiest way to allow for the program to deal with this is to expand the previous paragraph / section, insert the Level 4 paragraph heading "within" that expanded section, then recollapse the document), but, if not, what do I do:

I could do one of two things:

  • collapse only that new section, to essentially, Level 4, and leave the document in this (different than the previously discussed) inconsistent state — perhaps the problem is sort of a non-issue, in that, the next time I fold or unfold the document using a "global" macro, the heading is properly folded (or maybe it resolves itself only if I fully expand the entire document and then recollapse) — I need to do some testing
  • do a "lookback" (hmm, that would be only to the previous line), and possibly insert a <np> before this new heading so that this new heading is merged with the previous line

Is there a possibility of a similar need to merge with a subsequent line — that's not as likely as I thought just 10 seconds ago — if the next line starts with a lower level heading, it really means one of three things, either the document was inconsistent before I started this insertion, the next line is the first line in the file, or the heading I'm inserting is the first at this new higher level -- I'm tempted to say that the user should deal with these situations appropriately by (probably) using global macros.


  • I want to do some experimenting, and
  • I digressed from the issue I planned to consider — I wanted to deal with the issue of the user specifying a level of folding different than the current "default" level, instead I dealt with the user putting in a heading at a different level, so, to try to get back on point:

And, I probably want to do some "real" experimentation to sort through the issue, but, in an effort to think it through without the experiments:

Let's start with some boundary conditions (or situations):

First: Assume the document consists of a mix of headings from Level 1 thru Level 6, with multiples of each, and at least one of each lower level included in each higher level. Then assume that the "default" fold level of the document is Level 3. What do we see? We see Level 1, 2, and 3 headings, and all lower level headings are hidden within the Level 3 headings.

So, we insert a new heading of Level 3 while the document is collapsed to Level 3.

Now we see that new heading starting a new (possibly long) line.

Now suppose we ask that heading to be collapsed to Level 3 (oops, depending on the situation, we might have been in continuous wrap or none mode (did we expand a heading to insert this new one, or did we just start inserting the new heading between two existing headings? — let's try to ignore that for a moment, and get back to what happens to this line.

Since this heading is at Level 3, the document is collapsed at Level 3, and the user asked this heading to be collapsed at Level 3, things are straightforward — the macro simply replaces \n's in this new line with <np>s (\f's) and does not have to worry about joining to a previous heading nor (in this case) about "sucking up" a subsequent line.

Ok, suppose all of the above is the same, but the user asks this heading to be collapsed to Level 2? Now we have to think. Unless this line is before all other Level 1 or Level 2 lines in the file, if we really honor the user's request, this heading needs to be merged with the previous line (heading).

Should we even allow the user to request this? If he does request it, should we merge it into the previous line, or should we leave something inconsistent here? Whatever we do here, the document will be inconsistent in some sense. If we honor his request, we have a portion (a single Level 2 heading) collapsed to Level 2 while all the rest of the document is at Level 3. And, to correctly do this, we have to look at the headings following this one, any Level 3 headings immediately following this (before the next Level 2 heading) also have to be collapsed into the same (previous) Level 2 heading.

I suppose that is doable — is it worth it? (Especially assuming that a fully consistent document can be achieved by running one of the global macros in the near future?)

Suppose we didn't do that? Is there a simpler approach which simply (somehow, to be discussed next) simply collapses the "current heading"?

Ways to achieve this:

  • Require that the operator put the cursor in the heading that he wants to collapse, the macro just collapses that heading (by selecting from the beginning of that heading to the next heading at the same or higher level and doing the /n to /f replacement)? * Find the "current heading" (to be collapsed) by scanning backward in the file to find the first instance of a \f (<np>), then scanning forward to find the first heading after that, and treating that as the heading to be collapsed (this appears to assume that all headings are on a line by themselves (with a "title") but ending in a \n — I don't think that's a dangerous assumption, and finally scanning past the original cursor location to find the next heading at the same or higher level, then doing the /n to /f replacement.

Are there any scenarios (like the earlier situations) that require merging of this heading into a previous one? My first gut reaction is I don't think so, but I need to think some more Combining subsequent headings with this one? Maybe

Wait, back to the first question in the previous paragraph. Presumably, when I insert a new heading, it can be at any level, so yes, sometimes it would require merging with a previous heading. Is that so hard, maybe I should build the macro with that capability. (I think I described it above, no, maybe I didn't, OK, let's try it here:)

  • First we scan back to see the first uncollapsed heading (the current one that we want to collapse) (oops, now we're shifting to a collapse section instead of collapse a single heading — that may be OK, but we ought to think carefully about it)
  • Next, unless it is a Level 1 heading, we check the heading on the previous line to see if it is at a higher level — if it is, we need to merge this line with it.

How do we do that (merge lines)? All we do is replace the \n preceding this heading with a \f.

And, while we're at it, how do we merge subsequent lines? If subsequent lines are at a lower level line than this line (or the line we just merged into), just replace their leading \n with a \f.

Ok, now I have a scheme that can fairly easily recollapse a section, where a section is defined (and I was thinking it's easier and more important to find the front boundary of the section, but that's not true (won't try to explain right now). Maybe the recollapse section should handle multiple headings, defined as all expanded headings within a section of the document bounded by sections that are collapsed (have a \f) -- in other words, scan forward and backward to find the next \f, adjust the selection to the front of the respective lines, then collapse that entire area.

I think I'm forgetting something, how do I get the right level of collapse?? Hmm, not sure, but maybe I have to take the same approach as the global macros, collapse that entire region, then go back and reexpand that region at the desired level. And now we're back to needing to know what that desired level is (what the remainder of the document is expanded to). More thinking required.

I have a feeling the simplest is still to let the operator specify it (until we add some meta data (but maybe adding meta data isn't so hard and I want to do it now), but, starting without metadata), does it (permanently) hurt anything if this region is collapsed to some different level than the remainder of the document. I don't think so, so maybe I shouldn't worry about it. Anyway, time to take a break, I may have the hint of my approach, but I'll think (and/or experiment) a little more, then try to describe it.

What am I thinking (for reference):

  • collapse an entire region that is uncollapsed, to the level specified by the user (even if it is "wrong") and merge previous and subsequent lines (headings) as required.

<previous start> I've used the basic NmTWikiFold macros for a little while now (they seem to work well), but with my increasing use of a large offline plain text (or .twk) file for collecting notes (with TWiki markup) in anticipation of moving them to TWiki pages, I see the need for more macros, specifically, some to deal with individual headings (or a selection rather than the entire file).

Refactoring in progress (this might be a record, refactoring before I ever put the page on WikiLearn). I've decided that, at least for now:

  • I'll aim for seven "per heading" macros, analogous to the "global" macros (see NmTWikiFold), plus two additional to switch between "none" and "continuous wrap" (more on this in next bullet). I'll probably prefix the shortcut keys for these macros with the <windows> (<meta> ??) key instead of the <alt> prefix for the global macros.
  • One difficulty with these macros in Nedit is that because:
    • they fold text (create a very long line to make undesired text less conspicuous) instead of collapsing text (making undesired text invisible)
    • and I prefer to use continuous wrap (lines wrap to the width of the window without "hard returns" (\n's)) instead of hard wrap (using \n's at the end of each line instead of only at the end of each paragraph),

switching between a collapsed and expanded view of a document also involves switching between continuous wrap (expanded view) and none (collapsed view). For the "global" macros, this created no difficulties, but now for a document that might be partially collapsed (or expanded, however you want to look at it), the question becomes should the document wrap be none or continous. I have an approach, described next, but because of the potential for conflict, I'll add two additional macros to easily convert between continuous wrap and none.

  • My tentative approach to wrapping will be as follows (assumption: The "normal" state of the large document that I'm concerned with at the moment is folded, thus):
    • When I unfold (expand) any single heading, I need to switch the wrap to continuous in order to make the unfold fully effective, so expand will set the wrap to continuous
    • Usually when I refold (collapse) a single heading it will be the only heading that had been expanded, thus, it will usually be appropriate to switch the wrap to none to collapse the entire document — in a (feeble) effort to handle the exceptions, I will implement macros to switch between continuous wrap and none with a single keystroke. I recognize that will not fully resolve the problem, as, if I have several headings expanded, want to collapse a few but still see the others expanded, I will have to switch the wrap back to continuous after refolding any single heading.

  • Dangers of Sorting: I should point out (and should have pointed out even for the original (global) macros that it is sometimes convenient to sort a document while it is collapsed at some level (particularly the large document I'm currently most concerned with — see askSam for Linux). However, if the document is in a partially collapsed state (some headings not collapsed) the results of the sort will almost certainly not be what you wanted. Thus it is probably a real good idea to run one of the global macros just before sorting to ensure the document is in a consistent state. (Note that the hazard comes about because the Nedit folding mechanism changes \n (line end) characters to something else to fold the text, and sort functions on individual lines — do I need to explain more? Maybe somewhere else. But note that similar shell functions which work on the concept of lines will probably have similar issues.
  • Macros to overcome the danger: Because of the danger of undesired results when sorting a document in an inconsistent state of collapse (some paragraphs folded, others not), I will consider creating macros to sort, which include a global "fold" before running the sort, and get in the habit of using those instead of the menu sort. At some point (or if I build a "custom environment" for my poor man's askSam for Linux), I may try to disable the menu sort.
  • Automatic detection of collapse level: Although I've thought of a way to automatically detect the current collapse level of the document, it isn't quite as simple as I originally thought. (My first thought was to simply find any "<np>" (\f, or \xa0), then go to the beginning of the line and see what level heading markup is there, then I realized that, for example, in a document collapsed to Level 3, lines might start with a Level 1, Level 2, or Level 3 heading (which means I'd have to check all lines to find the lowest level heading that starts a line). There can be degenerate cases where, although collapsed to Level 3 there are no Level 3 (or neither Level 3 nor Level 2 headings) — of course, I don't think there is any major practical problem with setting the collapse level to Level 2 or Level 1 in those cases, but, see the next item. Thus, because detecting the current document collapse level is not entirely straightforward (requires a search through the entire document (which could be truncated once you find one instance of a Level 6 heading starting a line (containing an <np>)), is not "the simplest thing that could possibly work" (although I'm perhaps using that test in a way not necessarily envisioned by the original "authors" of that phrase — here I'm making the program simpler at the expense of a (slightly, I hope) bigger burden on the user, and finally, I see a better solution on the horizon, see the next item.
  • Meta data: If I continue to refine this set of macros to facilitate my poor man's askSam for Linux, I can see the value of including some meta data in each file. The details of how and where to store it can be worked out later (although I have some ideas, and might initially keep it visible and at the beginning of the file to minimize the chance for accidental deletion), but I can envision the following pieces of metadata being useful:
    • global collapse level (even if not fully consistent)
    • consistency flag (is the collapse level fully consistent)
    • counters to keep track of the number of headings not consistent with the current global collapse level (so, something like "reference counting" in OOP, we can attempt to track when operations on individual headings have possibly returned the entire document to a consistent state

Note: The first and second items of metadata will be the most useful and easiest to deal with, the counters may be more trouble than they are worth, we may just want to say that when a document becomes inconsistent it is not considered consistent again until a global fold or expand macro has changed the entire document. Note that detecting dirtiness can be a little tricky, any time any text (or at least any text including a \n or a heading marker is inserted (or deleted?), the collpase state of the document could be inconsistent.

<old document before start of refactoring>



For general background on folding and the initial macros which fold and unfold an entire file at various TWiki heading levels, see NmTWikiFold.

Heading vs. Selection?

I have a choice to make — I could make new macros that fold and unfold a selection, or that fold and unfold the heading currently containing the cursor (insertion point).

Because I previously tried to create macros that dealt with the selection, and had difficulty and confusion, and the simplicity of the "brute force" approach I eventually adopted for the "entire file" macros, I think I'll focus on the approach of folding and unfolding of the heading currently containing the cursor (insertion point).

7 vs. 2 Macros?

I have a second choice to make — should I have seven macros including six to fold the current heading at a level specified as part of the keyboard shortcut (like the original macros, and the shortcuts (<Alt>1) thru (<Alt>6), or should I try to pick up the necessary level of collapse based on the heading the cursor is in?

I think for my convenience, I prefer the second, then I might squeeze the necessary two shortcuts into the current (<Alt>) "namespace", perhaps as 7 and 8, or perhaps as something like c (collapse) and e (expand), or maybe the up and down or left and right arrows.

None vs. Continuous Wrap

Another decision — every fold macro that I've written so far includes the command to set line wrapping to "none" and the macro that unfolds the entire document completely includes the command to set line wrapping to "continuous" aka "soft wrapping".

How do I want to handle this for these macros. This actually brings up a few related points:

  • Maybe simply switching between continuous wrapping and none does everything I need? I don't think so, because I'd still being viewing lines containing s in place of \n's, which means I would not see the "real" format of the "expanded" headings. Still, macros to simply switch between continuous and none might be helpful, or useful as a work around until I get a better macro. I'll go experiment on a collapsed file to see what it looks like if just switched to continuous.

  • I may end up with a third "state" of the document, with wrap set to continuous and one (or a few) heading(s) containing \n's (while the remainder contain <np<s). There might also be a fourth state, with wrap set to none and one (or a few) heading(s) containing \n's (while the remainder contain <np<s).

I don't think those extra states will be a problem, because I'm fairly certain applying one of the "global" fold or unfold macros will bring the document to a totally "consistent" state.

I guess I don't have a choice when unfolding, I must switch from none to continuous or I won't see any line wrapping.

Tentative Definition of Two Macros

TWiki>Fold>collpase_heading (<Alt>c)
Fold (collapse) the currently selected heading and all its child headings.

TWiki>Fold>expand_heading (<Alt>c)
Unfold (expand) the currently selected heading and all its child headings.

Key to the Code

I think a key function will be something that can "identify" the heading the cursor is in, find the next heading at the same level (or the end of the document if there is no next heading at the same level) and then set the selection to include the first heading and everything from it up to but not including the next heading at that level.

Then we just convert s to \n's (for expanding) or vice versa (for collapsing) and do what with the continuous / none?

How about we set continuous if we expand any heading — then if we have multiple headings expanded we are still set to continuous?

What do we do when we collapse any heading? I guess I should not assume that all other headings are collapsed (or should I — yes maybe that is more in accordance with the way I plan to work — expand (or insert) one heading, work on it, when I'm done collapse that heading (and the entire document) in preparation to navigating to some different heading to work. At that point I'd prefer the whole document to be collapsed to make that navigation easier.

For special cases where I don't want the entire document collapsed, use one of the global macros.

So, now, how do I "identify" the heading the cursor is in, find the next heading at the same level (or the end of the document if there is no next heading at the same level) and then set the selection to include the first heading and everything from it up to but not including the next heading at that level.

And, is the approach different for the two cases (expand vs. collapse), or maybe rather, if we are looking in an expanded vs. a collapsed document? Yes, I think the second is more important — regardless, let's pick one case and get started.

Normally I'd expect to have the entire document collapsed, then pick a specific heading to expand, so let's start there.

An attempt at pseudo code for the expand heading macro

(For the first cut, ignore things like saving and restoring the cursor position)

Hmm, so some assumptions:

  • Assume all (or most) of the document is currently collapsed at some level (doesn't matter what level, I don't think)

  • Assume the cursor is in a collapsed line

Ahh, OK, this could be simple:

  • Select the entire line

  • Replace all s on that line with \n's

  • Set the wrap mode to continuous

Hmm, that doesn't have all the bells and whistles that I might have envisioned, but it sounds workable.

An attempt at pseudo code for the collapse heading macro

(For the first cut, ignore things like saving and restoring the cursor position)

Hmm, so some assumptions:

  • Assume all (or most) of the document is currently collapsed at some level (doesn't matter what level, I don't think), except for something

  • Assume the cursor is in an expanded line including a heading

Hmm, wait, we don't really want to collapse to an arbitrary level, we want to collapse to the same level as the rest of the document — this makes some simplifying assumptions possible (maybe) like:

  • Maybe we can find the current collapse level by moving up or down in the document until we find a line containing a (a collapsed line), moving to the beginning of that line, and looking at the heading markup — oops, not as simple as I first thought, for example, if we're collapsed to level 3 lines might start with any of a level 1, 2, or 3 heading, it's even possible that no lines start with a level 3 heading (of course, that might be a special case — it is presumably no different than being collapsed at whatever the lowest level (highest number) visible heading is) — so, I guess this may argue for choosing the level to collapse at rather than searching for it — yes, I think for the first cut I'll go back to 7 macros, prefixing these "single heading" macros with <ctrl> instead of <alt> as used for the "global" macros

  • Maybe we don't even need to know the current level of collapse, just do something like move back to the last collapsed line then go to the beginning of the first line following it (oops, we could get a blank line — OK, the first non-blank line following it, or the first line containing (starting?) with heading markup, then select from there to either the next collapsed line (or the end of the document) and then replace all /n's with s — oops, another problem — if several lines have been expanded, we'll now collapse them all into one long line -- re really need to know, explicitly or implicitly, what level we are collapsing at.

  • There is a(n easy) way to implicitly know the level we're collapsing at — similar to the previous bullet, go back to the last previous collapsed line (if none is found, we may have to search for the next collapsed line, but let's, for the first cut, assume we find the last previous collapsed line — then we go to the beginning of that line, copy the heading markup there (like "---++"), now move to the next occurrence of the markup (that should be the line our cursor was in when we started this exercise) and select from that heading up to but not including the next similar heading, then do the replacement within that selection. Sounds workable.

Re: the above: one simplification — you can look anywhere for any , then move to the preceding \n (one case where there wouldn't be one is the first line of the file), then "capture" the markup at the beginning of that line, then return to the original cursor location

I may want to think about the if no previous occurrence is found case, but we could treat that as a manual exception as a first cut — the user could just collapse the entire document at his preferred level.

Ahh, OK, here's an approach (to the collapse):

  • Save the cursor position

  • Search for any

  • Move to the beginning of that line

  • "Capture" the markup

  • Return to the original location of the cursor

  • Scan towards the front of the file looking for that markup, mark that as the start of selection

  • Scan towards the back of the file looking for that markup, mark that as the end of selection

  • Replace all \n's within the selection with s

  • Set the wrap mode to none?

Problem to consider: if there was more than one heading expanded and I just collapsed one, now we have a file where some other headings are "semi-expanded"; i.e., they contain \n's so wrap there, but long lines don't wrap because we are set to none

Potential resolutions:

  • Accept that as is, allow user to collapse the entire document or go to other headings and collapse those (manually) using this same macro (or make a rule/guideline that if more than one heading is expanded, the user should use one of the global macros to collapse (or expand) the entire document

  • Make this macro collapse all contiguous expanded headings — doesn't really solve the problem because there may be non-contiguous expanded headings

  • Make this macro collapse all expanded headings, OK, but is probably fairly complicated — is it worth the extra programming effort vs. simply using one of the global macros

  • Don't set the wrap mode to none, leave it at continuous, and provide a separate macro to change the wrap node to none — so now we have a sort of pseudo collapse — it doesn't really collapse but prepares the line to be collapsed when the user uses the second macro

  • Make the macro smart enough to decide for itself whether to switch to none or not — maybe if this is the last expanded heading, switch to none, otherwise leave at continuous (how could I detect whether or not it was the last? — I can't simply search for the presence of any \n's — there will be some. I could possibly search for any \n's not followed by the "target" markup.)

I'm not quite sure how to do that — given that I could do anything, I might loop through all the \n's, stopping at each one to see if there is a succeeding markup of the type the document is collapsed at — if it ever fails, I quit and no that I'm not totally collapsed, if I get through the entire loop without failing, I know that the document is totally collapsed.

Hmm, here's an alternative — after collapsing the line I'm working on (switching \n's to s, a step I should have mentioned above) then I look for any heading markup at a lower level preceded by a \n — if I find any I know I'm not totally collapsed. The advantage of this is I don't need as much loop control. For example, if the document is collapsed at Level 2, I search first for \n---+++, then \n---++++, and so on -- if any of them is successful I'm not totally collapsed. Oops, but wait, that's not sufficient, I also have to see if there are any \n's followed by something else — how would I construct an RE for that — let's try:

find: "\n[^-]" partial -- as soon as you find that the first character is not a -, you know you've found something else

Now, test the headings:

find: "\n---\+" thru "\n---\+\+\+\+\+\+" (individually, choosing only those that are longer than the "target" heading level

Now, regardless of that last decision, can't I work on the macro to do the first part (change the \n's to s in some target area, for example, yes, but I'll probably wait till tomorrow.

Some lines for testing:





  • ???

Other thoughts:

  • Explain that the approach of folding (which requires the switch between continuous wrap and none) vs. collapsing causes some additional gotchas (no way to "cleanly" have a partially expanded document unless:
    • Nedit allows a choice of continuous vs. none for portions of a document by some means (sections, rangesets, ???) (does Nedit support this??)
    • I write (or find) a macro to, while in none, manually wrap portions of the document (to the current width of the window) (and update that continuously as additional text might be added) — I'd need a lot of "hooks"

  • On a different aspect of the problem, it could be useful to maintain some metadata in a(n all or partially) collapsed document (perhaps within in something looking like HTML comment markers at the beginning of the document, just to propose something (at the end could be OK, except I have more irrational fears about having it deleted or damaged during user editing). That metadata could keep track of the following:
    • Collapse "level" (i.e., 1 thru 6) (will the macros ever support collapsing different portions of the document at different levels? maybe, I can see it being useful — just like having the document collapsed for easy navigation but portions expanded for editing, you might have (large) subsections collapsed to a different level for navigation within that subsection and then headings within that subsection fully expanded for editing — nevertheless, I'm going to exclude that from the current scope of these macros, unless I see some miraculously easy way of accomodating such)
    • Number of headings that are not at that collapsed level — a "continuously" updated count as headings are collapsed and expanded, some of the points being:
      • I often want to sort the document to maintain the records in alphabetical order, if I sort while the document is in some partially collapsed state (rather than at some fully consistent state of all headings collapsed to the same level) the sort could result in some "bad" results (portions of records being broken apart) — some alternative (almost or partial) solutions to this problem could include creating sort macros that first bring the document to a consistent collapsed state and then sort (one such macro for each level, or a dialog box to ask which level to sort at) — still, this doesn't prevent someone from sorting by some other means, like the standard Shell -> Sort menu choice. * Similarly, the choice between the (global) wrap setting might consult that metadata — as long as any part of the document is expanded, stay in continuous wrap, when a collapse of a heading (or selection) results in the entire document being in a consistent state of collapse, then switch to none

This brings up a few other points:

  • Maybe I need to protect the user from himself, maybe something like have the macros with training wheels to protect the user, and a version without training wheels to allow the user to do more things while assuming more risk. The first cut at this might be "right here" in the sense that the version 0.2 macros which only allow collapse or expansion of an entire document to a fully consistent level (with the exception of editing a portion of a heading (inserting \n's in particular)) is the version with training wheels, the next version (which allow expansion and collapse of individual headings is the (first) without training wheels version
  • And, I need user documentation for both versions which gets into this

Starting that documentation:

  • Folding is wonderful, and if you haven't experienced the advantages of folding, you need to see them — maybe I can provide (elsewhere) enough explanation so that you see the advantages without a live demo, otherwise, see if you can find a friend that uses folding (or collapsing) so that he can demonstrate it for you.
  • Like many tools, folding is not without disadvantages (you can smash a finger with a hammer, cut yourself with a saw, stab yourself with a screwdriver).
  • A specific danger (as an example): I often fold a document (a specific document, for a specific purpose (my poor man's askSam for Linux) and then sort it alphabetically. (The document serves as a free form database, and although I try to insert new records in alphabetical order, I sometimes fail or rename records which then leaves them out of order unless I manually move them, so sorting to the rescue. But, the sorting must only occur when the entire document (every heading in the document) is collapsed to the same state (level, and the appropriate level) — if I sort it at any other time, the sorting can break records apart resulting in a disaster (except if I realize it soon enough and use undo) — knock on wood, so far I haven't had the disaster, so haven't had to remember the undo. But, the first version of the collapse and expand macros helped minimize the danger because they applied to the entire document -- the only time I was in an inconsistent (dangerous) state was iff I did some editing of a record while the document was collapsed (which is convenient to do at times, and safe, if after editing I apply one of the macros that restores the entire document to a consistent level. (Maybe I should explain that sorting typically occurs by line, folding (at the appropriate level) converts an entire (arbitrarily defined) record into a single line which can then be sorted. If you sort the file while, for example, fully expanded, you totally destroy the integrity of the records (maybe I need to create a small example file and a set of example results to really explain this).

Maybe start the explanation of the joys of folding by asking the simple question, do you ever edit (or create) (fairly) long text documents? Long enough so that the traditional methods of navigation are cumbersome? (Hmm, I could even start for wordprocessing (or text processing) newbies with explaining the advantages of, for example, the PgUp, PgDn, Home, End, the <ctrl> variations, the scrollbar, etc. as opposed to the use of the simple up, down, left, right arrows. The point being, it not be till they've reached the level where the those tools are cumbersome that folding becomes attractive (on the other hand, it may occur before that for the other advantages of folding, like seeing the outline or table of contents "as they write", or (for me, in one case) allowing easy alphabetical sorting).

<"old" NmTWikiFold page for reference and cut and paste>


If you know your way around Nedit, you can use them by copying and pasting them in the appropriate places under Nedit Preferences -> Default Settings -> Customize Menus -> Macro Menu, naming them, and assigning a keyboard accelerator.

The macros range in size from 5 to 11 lines of macro code.

Hmm, maybe I need to include a warning about editing (especially deleting stuff) while folded, maybe especially so if recreated for AbiWord — maybe there needs to be something in the GUI to minimize the kind of editing that can be done in AbiWord while an outline is collapsed. Maybe I should dust off Microsoft Word and see what they do?


Principle of Folding in Nedit

Form feed markers (\f) are hex \x0c and appear as "" in a Nedit text file.

Basis for Folding TWiki Raw Files

These macros fold TWiki at any of the TWiki heading levels. TWiki headings are "marked up" with a string of characters as follows:

  • "---+" Level 1 Heading (Highest)
  • "---++" Level 2 Heading
  • "---+++" Level 3 Heading
  • "---++++" Level 4 Heading
  • "---+++++" Level 5 Heading
  • "---++++++" Level 6 Heading (Lowest)

Macro Names, Default Accelerators, and Functionality

TWiki>Fold>fold_level_1 (<Alt>1)
Display only Level 1 headings (all text and lower level headings folded)

TWiki>Fold>fold_level_2 (<Alt>2)
Display only Level 1 and 2 headings (all text and lower level headings folded)

TWiki>Fold>fold_level_3 (<Alt>3)
Display only Level 1 through 3 headings (all text and lower level headings folded)

TWiki>Fold>fold_level_4 (<Alt>4)
Display only Level 1 through 4 headings (all text and lower level headings folded)

TWiki>Fold>fold_level_5 (<Alt>5)
Display only Level 1 through 5 headings (all text and lower level headings folded)

TWiki>Fold>fold_level_6 (<Alt>6)
Display all headings (Levels 1 through 6 — all text folded)

TWiki>Fold>unfold_all (<Alt>0)
Restore document to (completely) unfolded condition — all headings and text (easily) visible

Algorithms and Code (version 0.2)

fold_level_1 Algorithm and Macro

This might appear to be a backwards approach, I fold the entire document and then unfold the headings at the level(s) I want to see, but it appeared to be simpler logic than trying to fold everything except those headings.

  • save the current location of the cursor (so it can be restored after folding — the replace functions (below) move the cursor)
  • set wrapping to "none"
  • fold the entire document by replacing all \n's with \f's —
  • unfold the Level 1 headings by finding all Level 1 heading markers "---+" and replace the preceding \f with a \n
  • restore the cursor to the original position in the file
  • move the cursor (and the "focus") to the beginning of the line it is contained in

The actual macro:

Note: this works but I notice now that I was inconsistent in "double escaping" (e.g., \\f) the characters that require escaping — I need to understand the requirement for double escaping better, make all the double escaping consistent, and then retest, but I have "published" this on Wikilearn before doing so.


replace_all("\n", "\f", "regex")

replace_all("\\f---\\+ ", "\n---+ ", "regex")


fold_level_2 Algorithm and Macro

The algorithm for fold_level_2 is almost exactly the same as that for fold_level_1 with the addition of 1 extra step: after unfolding the Level 1 headings to display them, unfold the Level 2 headings to display them as well:


replace_all("\n", "\f", "regex")

replace_all("\\f---\\+ ", "\n---+ ", "regex")
replace_all("\\f---\\+\\+ ", "\n---+ ", "regex")


fold_level_3 Algorithm and Macro

Can you guess? Just like fold_level_2, but also unfold the Level 3 headings:


replace_all("\n", "\f", "regex")

replace_all("\\f---\\+ ", "\n---+ ", "regex")
replace_all("\\f---\\+\\+ ", "\n---++ ", "regex")
replace_all("\\f---\\+\\+\\+ ", "\n---+++ ", "regex")


fold_level_4 Algorithm and Macro

... also unfold the Level 4 headings:


replace_all("\n", "\f", "regex")

replace_all("\\f---\\+ ", "\n---+ ", "regex")
replace_all("\\f---\\+\\+ ", "\n---++ ", "regex")
replace_all("\\f---\\+\\+\\+ ", "\n---+++ ", "regex")
replace_all("\\f---\\+\\+\\+\\+ ", "\n---++++ ", "regex")


fold_level_5 Algorithm and Macro

... also unfold the Level 5 headings:


replace_all("\n", "\f", "regex")

replace_all("\\f---\\+ ", "\n---+ ", "regex")
replace_all("\\f---\\+\\+ ", "\n---++ ", "regex")
replace_all("\\f---\\+\\+\\+ ", "\n---+++ ", "regex")
replace_all("\\f---\\+\\+\\+\\+ ", "\n---++++ ", "regex")
replace_all("\\f---\\+\\+\\+\\+\\+ ", "\n---+++++ ", "regex")


fold_level_6 Algorithm and Macro

... also unfold the Level 6 headings:


replace_all("\n", "\f", "regex")

replace_all("\\f---\\+ ", "\n---+ ", "regex")
replace_all("\\f---\\+\\+ ", "\n---++ ", "regex")
replace_all("\\f---\\+\\+\\+ ", "\n---+++ ", "regex")
replace_all("\\f---\\+\\+\\+\\+ ", "\n---++++ ", "regex")
replace_all("\\f---\\+\\+\\+\\+\\+ ", "\n---+++++ ", "regex")
replace_all("\\f---\\+\\+\\+\\+\\+\\+ ", "\n---++++++ ", "regex")


unfold_all Algorithm and Macro

  • save the current location of the cursor (so it can be restored after unfolding — the replace functions move the cursor)
  • unfold the entire document by replacing all \f's with \n's
  • set wrapping to "continuous" (because this is my preference for normal editing)
  • restore the cursor to the original position in the file
  • move the cursor (and the "focus") to the beginning of the line it is contained in


replace_all("\f", "\n", "regex")


Efficiency of the Algorithms

As you may have noticed, the macros above traverse the document up to seven times. That cannot be the most efficient approach.

Alternatives Tried

I did do some experimenting with search and replaces of the form:

replace_all("\\f---(\\+*)", "\\n---\1", "regex")

(in other words, using regular expression multipliers and memory )

The above may not be exactly right because I reproduced it from memory — during testing I did get it in the right form so it did (almost) work.

The problem was that it acted on (unfolded) all headings of the desired level and lower, I needed it to act on (unfold) all headings of the desired level and higher.

As I wrote this, I thought that I really should be able to get the above to work, if necessary by changing the overall premise, i.e., instead of first changing all \n's to \f's and then changing \f's to \n's in front of the headings I want to see, do the opposite — leave all \n's as is, then change the \n's to \f's in front of the headings I don't want to see. The problem is, at some point I have to change the \n's to \f's in front of the non-heading text that I don't want to see, and that makes it a little more complicated. It is probably still workable, but I think there is an even more efficient approach, which I describe next.

The Next Attempt

What sounds like the most efficient approach that I've thought of so far, and for which Nedit has appropriate commands (I don't know whether the Nedit macro language is Turing complete), goes something like this:

Initialize: do things like save the cursor position, set the wrapping to none

The Next Algorithm: see below

Finalize: do things like restore the cursor position, move to the beginning of the line

The Next Algorithm


  • fold the entire document (replace \n's with \f's)
  • visit each heading (search for "---\+*") Something like while search( ... ) not equal to -1?? (-1 indicates a failed search). Or while cursor not equal to $size_of_document (or whatever)) (I will need a temporary variable to hold the result of search(...) and then use that temporary variable for various purposes including being the exit condition of the while loop.
  • determine the length of the heading (e.g., length($search_end - search("---\\+*", "regex")) (note that search returns the position of the found string, and $search_end contains the position of the end of the found string (for the last successful search (??))
  • determine the level of the heading (level of heading = length of heading -3 (~))
  • if the level of heading found is equal to or higher than the desired fold level, move the cursor back to (at least) one character before the found heading (set_cursor_pos(search(...) - 1), and change the \f found there to a \n (might need to do a replace(...)?) (Note that the logical heading level is inversely proportional to the heading (markup) length, e.g. Level 1 heading markup = "---+" = length 4 = highest logical level; Level 6 heading markup = "---++++++" = length 9 = lowest logical level.
  • move the cursor ahead at least (~)4 characters and repeat the search for a heading (the move ahead is to ensure we don't find the same heading again and get into an infinite loop)

Future Enhancements

  • insert a collapse (fold) marker ("+"?) in front of each heading that actually contains folded text or headings (some headings at any level of folding or collapse may be empty)

  • when a way to make text invisible exists in Nedit, incorporate it so that text appears hidden (collapsed) instead of just folded

  • until then, consider appending say 80 spaces between a visible heading and the folded text it contains so that the folded text starts so far to the right that it is not visible in the window

  • add macros for other related functions, like:

    • Expand / Collapse the current heading (the heading on the current line) or selection -- need to consider the exact approach, but presumably change the state of expansion or collapse by one (heading) level

    • Promote / Demote the current heading or selection — we need to consider the exact approach (especially for subheadings) — presumably each heading is promoted or demoted one level, and when we run out of levels of heading, demote it to plain text. IIRC, Word (last time I used it, i.e., Word 97) promotes text to a Level 1 heading, I'd want to at least experiment with promoting it to a Level 6 heading (or to the heading level below (or equal to?) the lowest heading level currently used in the selection), then promoting higher from there if desired. Or maybe one command to promote to Level 1, another to promote to the next heading level below the lowest level currently used within the selection.

    • Toggle between showing no non-heading text and showing the first line of each non-heading paragraph

  • if editing files while folded frequently results in unintentionally deleted text, consider ways to deal with the problem -- possibly enforcing a prohibition on editing while folded somehow (maybe via GUI changes?)

  • make changes to the GUI to support folding / collapsible outlining

Other Comments or Observations

Origin of Folding Terminology?

I've sometimes wondered how the folding terminology originated. Now, seeing how folding is performed in these Nedit macros, it makes sense to me.

<is it clear why I say that, or do I need to explain more?>

Hmm, maybe I can (sort of) show the next heading unfolded and then folded, by using the <pre> tags — that will work in some browsers, but not all, and I forget which)

Previous Section Folded

Hmm, it shows up properly in konqueror (i.e., one long line). In browser's where it doesn't show up as one long line, it will probably show up the way it would if the fold macros didn't switch from continuous (soft) to no wrapping. The <np>s are apparently Nedit's way of displaying a \f.

---+++ Origin of Folding Terminology?<np><np>I've sometimes wondered how the folding terminology originated.  Now, seeing how folding is performed in these Nedit macros, it makes sense to me.<np><np><is it clear why I say that, or do I need to explain more?><np><np>Hmm, maybe I can (sort of) show the first this heading folded, by using the <pre> tags — that will work in some browsers, but not all, and I forget which) 

Rant: Self Documenting Programs

Whether the program above works (the first time) or not, I again think and am now convinced that it is extremely arrogant or fraudulent for a programmer to declare that any program is self documenting or self evident.

I can explain what these macros are intended to achieve ("fold" a file, by replacing selected \n's with \p's, and which \n's must be replaced under which circumstances), but the way the macros achieve those replacements follows a significantly different path than the explanation of what I'm trying to achieve (either because of limitations of the language, or limitations of my knowledge of the language).

Thus, someone who is trying to understand the macros must understand the objective (in the "real" world, or even in the computer world) and the possibly roundabout way the macros achieve that objective. At least one of those explanations will not be contained in the code (discounting comments). Will there be some cases where the real world objective can be accomplished in a straightforward parallel programming path? Yes, I'm sure of that, but as a reader of programs, I am sure that is the case far less often than most programmers think.


  • () RandyKramer - 22 Jun 2003
  • I should mention the help I've gotten so far from the discuss@neditPLEASENOSPAM.org mailing list, including (recently) Thorsten Haude, Joerg Fischer, Joor Loohuis, Joachim Lous, whoever wrote the earlier version(s) of the Nedit fold macros, people on the AbiWord mailing lists (Paul Rohr, Dom Lachowicz, F.J. Franklin, others), and other people who have helped me gradually develop an understanding of how to accomplish folding or collapsible outlining. I know I've left out some names I should mention — sorry!
  • If you edit this page: add your name here; move this to the next line; and if you've used a comment marker (your initials in parenthesis), include it before your WikiName.

Page Ratings

< original page template (empty)>





  • () RandyKramer - 29 Oct 2003
  • If you edit this page: add your name here; move this to the next line; and if you've used a comment marker (your initials in parenthesis), include it before your WikiName.

Revision Comment

  • %DATE% —

Page Ratings

Edit | Attach | Watch | Print version | History: r6 < r5 < r4 < r3 < r2 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r6 - 2005-04-06 - RandyKramer
  • Learn about TWiki  
  • Download TWiki
This site is powered by the TWiki collaboration platform Powered by PerlCopyright 1999-2017 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding WikiLearn? WebBottomBar">Send feedback
See TWiki's New Look