create new tag
, view all tags

Nedit Folding: Back End Design




This is a rough first draft of a proposal for the back end changes needed in Nedit to support folding.

(New) open issues: This morning (20080317) I woke up with some additional thoughts about various aspects of this approach (and/or the simple Easy Start / Proof of Concept) which I record very roughly under Open Issues.

The reason they are recorded very roughly is because I really have other things that I should be doing. If Nedit does get a GSoC slot, and a student does get interested in this, I'll find a way to come back and refine these.

General Approach

The general idea is this: There will be two textBufs for any NEdit tab for which folding is enabled. One of those textBufs will always contain the full text of the document (the unfolded textBuf), and one will contain only the text visible based on the current folded condition of the document (the folded textBuf).

When the folding view is enabled, the folded textBuf is attached to the display widget, in place of the unfolded textBuf.

Cursor position, scroll position, ranges, selections, editing actions (like insertion or deletion of text) and similar will be tracked and kept in sync between the two textBufs .

To keep these items in sync, translation is required. For example, the absolute position of the cursor in the unfolded textBuf will not be the same as the absolute position of the cursor in the folded textBuf, the cursor location must be corrected to account for the text missing from the folded textBuf.

(In general, there are two types of correction, an absolute numerical correction based on the missing text, and corrections to disallow the cursor from being within a folded (hidden) text area in the folded textBuf. These corrections may occur at two different times for a reason that I'll try to explain better later. However, in general, this has to do with the idea that if the unfolded textBuf is currently attached to the display widget, there is no immediate need to constrain it to unfolded areas in the folded textBuf. Furthermore, I'm guessing (but not yet sure) that the cursor movement may seem more logical if that correction is not made until actual switching of the buffers occurs.)

Aside: It has been helpful to me at times to assume that the folded textBuf comes into existence when folding is enabled for a tab--eventually I'll correct this description to reflect that.

Recap and More Detail

Re-iterating the design, and getting into some more detail, with bullet items:


  • Basic approach: for each tab which is displaying (or can display) folded text, create a 2nd gapped buffer (textBuf) for that tab. (The new buffer will be called the folded textBuf and the original, the unfolded textBuf.)

  • The unfolded textBuf will always contain all the text in the document, the folded textBuf will contain only the currently visible (unfolded) text.

  • There will be a means to switch between folded view and unfolded view, which will essentially switch which textBuf is used to provide text (i.e., is attached) to the display widget.

  • Many things will be kept synchronized between the two textBufs, among them the cursor and the tracking of folded regions.

  • Folded regions will be tracked by a rangeset-like mechanism--the one exception to the current rangeset design being that it will track ranges of zero length. (An alternate to the need to track ranges of zero length will be mentioned somewhere below.)

  • There will be a rangeset mechanism for each of the gapped buffers (that is, for the folded textBuf and the unfolded textBuf), and those two rangeset mechanisms will be kept in sync (more below).

  • To support hierarchical folding, there will be separate rangesets for each level of the hierarchy. (Aside: there may be quite a few levels in the hierarchy--I currently use 7, I'm ready to expand to 8, and my TWiki marked up files can contain anything in a record, including, for example C code, which would have its own hierarchy of folding levels, thus, what is the max--12, 15 levels?)

  • (This might be the wrong place to mention this.) One thing I haven't mentioned so far is a probable need for some sort of mechanism to correlate ranges in the rangesets for the folded textBuf with the ranges in the rangesets for the unfolded textBuf. This could be handled by keeping the ranges in both sets of rangesets in the same order (sequence), but I think something more explicit, like some sort of "database" might be easier to handle. (I know rangesets can be named / labeled, it might be helpful if the same could be done for ranges. I'm looking for an easy way, given the range in one textBuf to find the corresponding range in the other textBuf without "walking" a list or similar.)

Marking a region to be folded and then folding it

With those bullet points, lets discuss what happens when a particular region of text is marked as a folding region and is then folded:

  • It doesn't matter too much which textBuf is currently attached to the display, as we keep them both in sync anyway, but for the purposes of this discussion, lets assume that the unfolded textBuf (the one that contains all the text) is currently attached to the display. (Actually, once we get down to details, maybe we have to be using the folded textBuf and be in "folding mode" in order to designate a region for folding, but we'll deal with that later.)

  • So, we pick out some region we want to fold (we'll clarify some terminology later). Whatever mechanism we use to do that, the appropriate rangeset for the unfolded textBuf (or the textBuf that is currently attached to the display widget) gets the range of that region. Also, the equivalent rangeset for the folded textBuf (or the textBuf not currently attached to the display widget) gets the same range. By equivalent I mean accounting for any translations which have to occur due to missing text in the folded textBuf.

  • Now, let's actually fold that region. Two things have to happen (still assuming the unfolded textBuf is driving the display):

    • First, the user designates that he wants to see the folded view--when he does that, the text display widget is switched so it is attached to the folded textBuf instead of the unfolded textBuf. There are GUI changes that should occur at this time, for example, the fold mark gutter should appear. Oops, careful readers may notice that I've been inconsistent--presumably (or possibly) I could not have designated a fold region while the fold gutter was not visible--or something along those lines--really, for my purposes atm, I want to treat that as a detail to be worked out later.

    • Second, the user designates that he wants that (or any) particular folding region to actually be folded. (He might make that designation by clicking on the boxed "-" in the folding gutter.) When he does that:
      • the text in that region is deleted from the folded textBuf
      • the range in the (appropriate level) rangeset becomes zero length but remains in the rangeset (for tracking of the folded region)
      • the GUI changes appropriately to show a folded region (e.g., the appropriate boxed - becomes a boxed +, ...)


In general, unfolding a folded region (range) requires finding the equivalent region (range) in the unfolded textBuf and copying the text in that range to the folded textBuf (and then, if appropriate, making appropriate adjustments to the scroll position, selection, cursor location, etc.).

Navigation / synchronizing

I could give additional examples of the behavior as I designate and then fold additional regions, or as I switch back to the unfolded view. I'm hoping the above was enough to give the general idea. There are other things to discuss:

  • during navigation, cursor movements are "synched"--again, there are details to be described later, but lets start at a simple level:

    • The simplest case is when the cursor is moved in the folded textBuf (i.e., the folded textBuf is attached to the text display widget): there will always be a corresponding location in the unfolded textBuf, but there will have to be translation between the two cursor locations to correct for missing text in the folded textBuf (again, we'll deal with details later).

    • The more difficult case is when the cursor is moved in the unfolded textBuf (i.e., the unfolded textBuf is attached to the text display widget). In that case, there are positions of the cursor that don't correspond to positions in the folded textBuf, so translations will require a little more logic. For example (and for the moment), maybe we'll assume that anytime the unfolded cursor is in an area that corresponds to a folded region in the folded textBuf, the cursor in the folded textBuf should be positioned at the beginning of that folded region. (However, I want to think some more about that, maybe there would be good reasons to have the logic be a little more complicated. And, maybe that positioning doesn't occur "continuously" but is only done on a switch between the two textBufs.)

I should point out that some of what I described is a little misleading in that there isn't really a cursor displayed on the textBuf that is not attached to the display, but we want to keep a memory location (i.e., a variable) that keeps track of where the cursor would be if we switched textBufs.

In a similar fashion, things like the scroll position of the document, and the rangesets corresponding to folded regions, and many other things need to be kept in sync between the two textBufs.

Search and Replace and Editing

I'm running out of steam here, in future drafts I can expand on some of these points (or others can), but let me just touch on search and editing operations briefly.

In general, editing operations need to be done to both buffers if the edit affects a visible region in the folded textBuf. (Not sure this sentence is helpful, or maybe it needs to be after the following.)

Things like search and replace or editing operations (usually via nedit macros?) that might affect the entire document should have two modes of operation:

  • In one mode, the search and replace (or editing macro) is intended to affect the entire document. Thus it should operate on the unfolded textBuf and sync its changes to the folded textBuf, even if the folded textBuf is the one that is currently visible. If, as I think is the case, a special string is created to support search (and replace?) operations, that special string is created from the unfolded textBuf.

  • In the other mode, search and replace (or an editing macro that could affect the entire document) is intended to affect only the visible (non-hidden) portions of the document. Thus it should operate on the folded textBuf and sync its changes to the unfolded textBuf (even if the unfolded textBuf is the one that is currently visible). If, as I think is the case, a special string is created to support search (and replace?) operations, that special string is created from the folded textBuf.

Avoid Problems at the Boundaries

One thing to discuss is avoiding the possibility of incorrect search and replaces occurring at (or across) the boundaries of folded regions.

For example, lets use this sentence as an example.

If I would fold that sentence to look like this (where the two vertical pipes indicate the missing folded text):

For example||sentence as an example.

... and then did a search (or search and replace) on "example sentence", it should not find (or replace) the area that looks like "example||sentence". (That seems obvious and simple in this example, because I've put pipes in place of the folded text. That (showing pipes) isn't what we'd normally expect to happen if we use rangesets to mark the location of folds.)

On the other hand, adding some marks like the vertical pipe, not to be displayed (perhaps a \f?) might be a workaround (or part of a workaround) that would solve two problems--the one I just described here, and the problem that, in the current design, rangesets "evaporate" when the text range becomes zero length. Something to think about and resolve.

Arguments for this approach

At this point in time, I think this is a feasible approach, and the best I've heard or thought of. I won't think of all the arguments in favor of the approach, but there are two that are a little bit out of the ordinary.

Special Arguments

  • This is, in one sense, a selfish argument that applies only for me, but, then again, maybe not. Below, under "Ordinary Arguments", I give my opinion that translating cursor position and similar between the two textBufs will require simpler logic than navigating around text in the buffer that is not supposed to be displayed. My argument here is that fewer "new things" are required to be added to the nedit codebase to support this approach (as opposed to the remove hidden text from the textBuf approach).

For example, we don't need a new mechanism to possibly store text removed from textBuf, instead we simply create an additional instance of a data structure we already have, textBuf. Similarly, we create an additional instance of the rangeset mechanism (rather than possibly create something new). (Admittedly, the rangeset mechanism might need to be modified to handle ranges of zero length, and I am recommending some mechanism to correlate ranges between the two sets of rangesets, but somehow I just feel that this is a little less complex than creating a new mechanism to store text deleted from the textBuf.)

  • A feature that comes almost for free--basically the feature in oowriter--do they call it a navigator? I need to expound on this--the idea is that there will be two panes (possibly tabs, maybe one more like a, what do they call, it a "tear-off"?), one showing the full text of the document, one showing the folded text (maybe even a special headings only version). Another way of saying it is that both the folded textBuf and the unfolded textBuf will be attached to display widgets, and in general, the user would have them arranged so he can see both.) Navigation in the folded textBuf will move the unfolded texBuf similarly (that's part of the feature).

Ordinary Arguments

  • The two major categories of approaches I considered earlier (leaving folded text in the textBuf and navigating around it, or taking text out of the textBuf (moving it somewhere else), but no need to navigate around it) each have significant drawbacks. In this approach, we don't navigate around the text, we navigate in the textBuf that is easiest to navigate in at the moment (the one currently attached to the display widget), and translate the results of that navigation (for example, updated cursor and scroll positions) to the other buffer. I think the logic for the translation will be easier than the logic needed to navigate around hidden text, and, in general, is needed for lots of other reasons anyway.

Easy Start / Proof of Concept

I'm trying to, and think I might have thought of an easy way to start heading in this direction with a minimum of C programming. The one thing I might need is the GUI stuff, but I'm going to try to talk (think out loud) myself through this and try to see what I really do need.

The thought / concept: I can get a start on the two textBuf approach by opening the same file in two different tabs.

After doing that, I can use nedit macros and the rangeset mechanism to, for example:

  • mark a folding range (the same text in both tabs), assigning both ranges to a rangeset for that tab (is there a separate set of rangesets for each tab in a nedit instance?--it doesn't really matter, if necessary I'll just use two of the "global" rangesets if they are not per tab)

  • make one of the tabs look folded by deleting the text in that range--for now, replace it with something (unique) to show that some text was folded (deleted) there (later, with a modification to rangesets to track zero length ranges, that unique placeholder would not be required (but might be desirable). (Just thinking out loud for that unique placeholder, possibly use \f (invisible), or something like "{{{folded}}}" (I know there is some language / markup that uses the triple curly brace for something, but, at least for test purposes, that should be pretty unique

  • look at the folded tab, confirm it looks folded--smile, and contemplate the beauty

  • now work on unfolding that folded range, after (somehow--for now this might be an imaginary (e.g., user assisted) step), identifying the folding range in the appropriate folded tab rangeset ("appropriate" because I expect there will be rangesets for each level of the folding hierarchy)

  • using some mechanism or logic, identify the corresponding range in the appropriate unfolded tab rangeset

  • copy the text in that range from the unfolded tab to the corresponding range in the folded tab

  • look at the folded tab, now unfolded, confirm it looks unfolded--smile, and contemplate the beauty

I don't even think I have to do this to prove the concept--it works, although I've done some handwaving to get the corresponding ranges and to identify the range to be folded and unfolded and similar.

Basically, the above can be considered a thought experiment--although I see the possibility for some of the details to fail / be wrong, the overall thing should work just fine.

Now, can I come up with simple (G)UI of some sort that can be done with Nedit's existing macros and similar? I'll have to think about that some more. It would be nice.

Oops, wait--too far, too fast

While the above will give a sort of proof of concept, what I've ignored is keeping the two tabs in sync as far as editing and navigation go. I can't immediately think of ways to deal with that except with C code. (Unless the hooks in the Macro Hook/Event Binding System project not only tell you that changes have occurred to a tab (with the right event), but exactly what those changes were so you can duplicate them in the other tab. This is discussed again below under Four Alternate Approaches to Keep textBufs in Sync.

Open Issues

I thought of four things last night (or this morning, 20080317) to cover (and I should have written them down):

Four Alternate Approaches to Keep textBufs in Sync

  • consider (examine the code) for every user (or macro) action that can occur, and see which of them needs to occur for both textBufs (the fat, dumb, and probably not even happy approach--see the next approach)

  • look for every place that textBuf is "addressed", modify that to do the same thing to both (i.e., two) textBufs

Aside: I've been talking only about keeping the textBufs in sync, but there are some other things to deal with, like the rangesets, and, maybe, if any of the following are not stored in the textBuf, things like the cursor, mouse pointer and scroll position, the selection(s), ...

  • Is there a particular function or set of functions that handle the interface to the textBuf (I'm sort of asking if there is a textBuf object, that is, struct plus code). If so, modify the struct so there are two textBufs, and modify that code so it deals with both of those textBufs) (sort of the "why scatter these duplication functions all over the code when they can be centralized in one place" approach, maybe that place being textBuf.c)

  • Check with Bert (or someone) about the textBuf modified event (do I have the right name?)--does it tell me what has changed as well as that something has changed? I was thinking about this as a hook to use an approach like every time the textBuf is changed (and I get that event notification), check to see what has changed (worst case by comparing before and after snapshots somehow), and then making the same change to the other textBuf. This has the potential of being done in macro code rather than c.

Darn, forgot one

Even though I did try to write them down when I got out of bed (I should keep a notebook next to the bed, and should have written them down immediately).

Did this have something to do with the textBuf modified event(s)?

End-of-fold marker

One thing I've ignored so far in my duplicate textBuf approach is how I would display an end-of-fold marker.

In some outliners (kate) it is a horizontal line matched up to (or which matches) the beginning of the horizontal line in the fold mark gutter L.

Going to test kate now, but my impression is that you can neither get the cursor into that line nor modify that line...

Ahh, that is correct, or sort of correct, actually, it seems that line, although a little different than the standard underbar (longer dashes), is positioned about where the underbar / underline shows up for text which can show a real underbar (i.e., not a dashed line or similar in the next line).

Thus, saying that it can't be modified is a little misleading, I can modify the text in that line (it is actually the heading line for that folded region), but it is like it is displayed with this special underbar "style" which I can't change (by normal means of editing).

Anyway, that was a digression--if I use the two textBuf approach, how (and what) do I use for an end of fold marker:

  • one obvious choice to consider (haven't thought yet about whether it will work or not) is (now that I understand it better) the same thing kate uses--display the header line for that region with a (special) underbar style

  • another choice, which fits in with the problem of zero length rangesets, and the desire to eventually have folded regions within a line, is the thing I (and, iirc, others) have mentioned previously, some sort of special marker made of text characters, like {{{folded}}}.

The first problem I think of (or want to consider) with the 2nd option is that I don't believe this marker should be editable, for lots of reasons, but including that if code has to find the marker and it changes so much as to be non-recognizable, that's a problem. (On the other hand, if we mark fold regions with rangesets it doesn't have to be recognizable, well, or not completely--we have to recognize what is part of the marker so we can exclude it from the unfolded textBuf.)

Mouse Pointer / Scroll Position vs. Cursor Position

The reason I bring this up is perhaps two-fold.

I find some of Nedit's (and most editor's or word processor's) behavior surprising. For example, suppose I have the cursor (insertion point) at a certain line of text but scroll ahead in the document using the mouse, for example, to move the vertical scroll bar. Then, if I do certain things (pageup or pagedown are examples), that movement is not from what I consider my current location (what I'm calling the scroll position), but instead from where the cursor is located.

The reason to discuss this is to decide how this should work when we switch between two textBufs.

I guess, regardless of my surprise, that is the current behavior, and until and unless we change it, we should have the same behavior in the other textBuf.

What I'm trying to say (and didn't quite say it yet) is that, if we switch textBufs and we're in a situation where the current scroll position in the document doesn't match the cursor location, when we switch, the scroll position and cursor location should still not match, they should be essentially as they were in the other buffer. (When I say essentially, that is intended to recognize that translations (discussed elsewhere) do have to occur to account for text not present in the folded textBuf.)

Then, for the present, if I do one of those actions that causes the scroll position to jump surprisingly, that's what should happen. (Before I ask for a change to that behavior, I (or all of us) need to think about what is best--one option is to train myself better so that behavior is not so surprising.)


  • () RandyKramer - 15 Mar 2008
  • If you edit this page: add your name here; move this to the next line; and if you've used a comment marker (your initials in parenthesis), include it before your WikiName.

Revision Comment

  • R1 thru R5: rhk: wordsmithing
  • R6: rhk: added "Easy Start / Proof of Concept" section showing how this can be demonstrated with no C code whatsoever
  • R7: rhk: oops, too far, too fast
  • R8: rhk: added very rough "Open Issues"
  • R9 thru R11: rhk: wordsmithing and fixing some unintentional errors (writos)

Page Ratings

Edit | Attach | Watch | Print version | History: r11 < r10 < r9 < r8 < r7 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r11 - 2008-03-26 - RandyKramer
  • Learn about TWiki  
  • Download TWiki
This site is powered by the TWiki collaboration platform Powered by PerlCopyright 1999-2017 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding WikiLearn? WebBottomBar">Send feedback
See TWiki's New Look