I am creating a kit list using INCLUDE to pick up text files from another web site. It would be really nice if the text of the included files was searched by the TWiki search function. You get the included text displayed when you click on 'see diffs' rather than just the INCLUDE line. I presume the Search function just looks at the raw view. There is probably a good reason for the way it is, and as I'm pretty new to TWiki pardon my lack of knowledge if I have missed where it is explained. The feature would be really useful for what I'm doing though.

-- AndrewJackson - 11 Nov 2003

I have been experimenting with INCLUDE and understand how it works, at least in terms of generating dynamic data from another web site. It is generated on-the-fly as the page is built. For many situations, that is more dynamic than it needs to be, and including from the other site on a scheduled basis would be enough. For example, I was including information about countries from the CIA Factbook, which has information about each country. I didn't want to have to incorporate this information by hand as it may change occassionally, but updating once a year would be enough. I wouldn't mind having this sort of occasional INCLUDE functionality. If indeed the INCLUDE is performed on an occasional basis, then the content of the INCLUDE could be stored as part of the page, and searched using the normal search function. We would need to have the INCLUDE operator continue to exist in the source file even after the inclusion was done, so that it could be envoked at an appropriate time. A parameter of the INCLUDE could be added to determine when the included data would expire, and therefore be updated. We would need to have some delimiter around the INCLUDEd information so it would be clear what we would want to replace when the INCLUDE would be actioned again.

This concept is probably appropriate for MOST uses of INCLUDE, as it is not necessary in most cases to have completely dynamic updating. However, it certainly would require changing the syntax. Currently, the INCLUDE is replaced by the text that is accessed, and there is no change to the source file. It would be feasible to change INCLUDE to have a validseconds parameter which would provide perhaps the number of seconds from one update to the next. In the source file, the INCLUDE operator would remain but the included text could appear just after that, followed by an ENDINCLUDE marker to indicate the range of text that will be replaced.

Operation would be as follows:

• Initial file would contain only the INCLUDE operator.
  • %INCLUDE{"url" validseconds="3600"}%

• Upon first actioning the file, the INCLUDE would be actioned, and the information from the INCLUDE operator saved back into the source file, preceeded by the expiration time, and followed by the ENDINCLUDE marker.
  • %INCLUDE{"url" validseconds="3600"}%
  • %INCLUDEEXPIRATION{expiresecs="9475934759"}%
  • blah blah blah (included text)
  • %ENDINCLUDE%

• The user would see only the INCLUDED data, not the INCLUDE operator, expiration date or the ENDINCLUDE marker.
  • blah blah blah (included text)

• When the time expires, the next time the page is loaded, the INCLUDE operation would access the source for the include and compare it with the data already included. Only if the data changes, the file is updated, and new revision created.

I think this is feasible (but would appreciate other opinions) and would be a nice improvement from the operation of INCLUDE for the purposes that I have, which don't need to be updated very often. I would probably set the validseconds parameter to 1 month, but I'm sure that would vary. Even if you set it to one day, that would improve performance substantially, as reloading the page would not require additional http requests.

Secondly, the INCLUDED information is searchable. One caveat would be that the search would provide information from the last build of that page. It may be necessary to have a cron job that would investigate any pages that have expired and see if they need to be updated. I don't think that any such provision is currently available, but certainly feasible, as a search of all INCLUDE statements that have expired would not be too hard, and then updates of those pages.

I would like to see if there are others who are interested in this feature.

-- RaymondLutz - 11 Nov 2003