TWiki Security Alert Process

I discovered a security issue. Now What?

• Important: In case you think that you discovered a security issue that could potentially compromise public TWiki installations, please send an e-mail to the SecurityTeam via the TWikiSecurityMailingList at mailto:twiki-security@listsPLEASENOSPAM.sourceforge.net. We will follow up in a timely manner with a fix and will inform administrators before the issue gets public.

How can I get notified of security issues?

• Please subscribe to the TWikiAnnounceMailingList to get updates on new TWiki releases and TWiki vulnerabilities in a timely manner.

• Please list your TWiki in the TWikiInstallations directory. Make sure that the TWikiInstalledBy form field points to your TWiki.org user account, and that your TWiki.org home page has a valid e-mail address. Listing your site is especially important if you run a public TWiki site. The known TWiki site administrators will be notified 2 days before twiki-announce alert.

Security Alert Process

As a free service, the TWiki team is trying its best to provide a hotfix and to send TWikiSecurityAlerts to TWiki site administrators in a timely manner.

Current process:

• Someone sends an e-mail to the SecurityTeam via the TWikiSecurityMailingList at mailto:twiki-security@listsPLEASENOSPAM.sourceforge.net
• The SecurityTeam triages the seriousness of the issue:
  • Severity 1 issue: The web server can be compromised
    • Example: Software can be installed and executed remotely
    • Responsiveness goal: Fix and alert within 24 hours
  • Severity 2 issue: The TWiki installation is compromised
    • Example: The access control of the admin group can be cirumvented
    • Responsiveness goal: Fix and alert within 48 hours
  • Severity 3 issue: TWiki content or browser is compromised
    • Responsiveness goal: Handle as BugReport, no alert
• Action for Severity 1 and 2 issues:
  • Verify issue
  • Create hotfix for affected TWikiProductionReleases
  • Initial alert: Alert TWikiDevMailingList members and TWikiAnnounceMailingList members
  • After 2 day grace period, avoiding weekend: Issue a public security advisory
  • Create a patched TWikiProductionRelease or a HotFix for the latest production release within 7 days
• Action for Priority 3 issue:
  • File BugReport
  • Fix in development branch for upcoming TWikiProductionRelease

Contributors: -- PeterThoeny, KennethLavrsen - 26 Jul 2006

Discussions

The process worked at the recent SecurityAlertExecuteCommandsWithSearch vulnerability, but with some issues. Reviewing what happended, these items should be improved:

• Reach more TWiki site administrators
• Allow a grace period for administrators to fix their installation

Grace period: The public security advisory got sent out uncoordinated before I could inform the administrators. This is not in line with our policy (as stated in BugReport). How can we increase the likelyhood of people adhering to the policy? It might be difficult since there seem to be a "I want to be the first" race. Any idea?

In the recent issue, I created a hotfix and informed known TWiki site administrators within 6 hours of the issue. (Sidenote: It was an unthankful job, I got mostly flamed, even though I dropped everything and worked on this with highest priority)

I compiled all e-mail addresses from Codev.WebNotify, TWiki.WebNotify, Support.WebNotify, Plugins.WebNotify, and from the Main.TWikiInstallations. In total 525 e-mail addresses. Little did I know that there seem to be around 30K public TWiki sites, Google:inurl%3ATWiki%2FTWikiPreferences, e.g. I reached only a small fraction!

Possible improvements:

• Use Google search to pull the e-mail address from the WIKIWEBMASTER TWikiPreferences setting
  • Reach many potential targets (30K)
  • Does not reach behind-firewall TWiki sites
  • Crackers can set up their own TWiki sites and get a headstart (admittedly this is the least likely way - SH)
• BroadcastMessageForAllDownloads
  • Intrusive
  • Performance concerns
  • Reach behind-firewall TWiki sites
  • Crackers can check this and get a headstart
• Security mailing list
  • Opt-in
  • Crackers will get on the list and get a headstart

-- PeterThoeny - 20 Nov 2004

What about conbining multiple approach? Using Google, the security mailing list (in which you should get registered as soon as you download TWiki), notifying some (or all) security sites, publishing it on the TWiki front page and Download Page, and posting it on Slashdot (this is tonge in cheek) should reach all the TWikiInstallations in the wild.

-- RafaelAlvarez - 20 Nov 2004

A careful googling and post-filtering uncovers only 688 public sites (which is still quite a few). The 30K number is counting the same site many times.

-- CrawfordCurrie - 21 Nov 2004

wrt step 1: instead of contacting one of the core team members, it should be "contact the CoreTeam". Not all members are active all of the time.

This could be facilitated by web form or singlr group email address (e.g. don't put the burden on the alerter to locate and type/copy-paste each core team email address).

-- MattWilkie - 21 Nov 2004

Added a couple of bullet points to the list above to point out that whatever method we use can be used against us.

Whatever method is chosen it should be documented prominently in the TWiki web [of future distibutions and on the front page of TWiki.org and as appropriate throughout the site -- SH - 25 Nov 2004]. That documentation should probably also be patched into the download zips for previous releases.

I think that in future we should use an opt in approach to security mailings (I've stated my misgivings about BroadcastMessageForAllDownloads in that topic), but to publicise the existence of the mailing list we should send out a one shot [unsolicited] email asking admins to register [opt in] for the security mailings. The list that was compiled for the recent issue should be expanded to include any installations found by google to receive the one shot mail.

I remember having to fill in a form to receive a production release in the past which required an email address to send the zip location to. I don't suppose a record was kept of those email addresses? They should be sent the one shot email if they can be found.

-- SamHasler - 24 Nov 2004

http://freedesktop.org/ was compromised and there's a painful discussion on http://lwn.net. (I also commented on the latest LWN story, still subscriber-only, in which it was officially confirmed it was TWiki.)

We have lost Freedesktop as a well-known user of TWiki, and much security credibility - they are going for MoinMoin. The upshot of all this is that we absolutely must have a low-volume security alert email list that everyone who downloads is encouraged to use, and a TWikiSecurityHome? page that is visible on every left bar and has latest vulnerabilities, email list signup, good practices links, etc.

Now that the exploit is actively being used, I think we should also publish the alert to BugTraq and other security alert lists - enough bad guys already know about it, and some of the good guys still have no idea.

-- RichardDonkin - 24 Nov 2004

I agree. CERT is another good organization to notify.

-- KennethPorter - 24 Nov 2004

Mail about this vulnerability was sent to bugtraq (and full-disclosure and vulnwatch) on Saturday, November 13, 2004. I know of hosts that were broken into on that weekend. Ideally, mail should have been sent to every registered TWiki user (via harvesting their e-mail addresses from their pages) immediately after this vulnerability was known about (and preferably with instructions on how to fix it). Also, (as RichardDonkin points out), the main TWiki page should have have had a highly-visible notice about this problem, and downloads needed to be taken off-line or immediately patched.

-- ClaussStrauch - 24 Nov 2004

Added to my statements above to further clarify my possition.

Also, I'm concerned that we still haven't notified many TWiki admins about SecurityAlertExecuteCommandsWithSearch. I think we either need to make a decision on what the TWikiSecurityAlertProcess should be quickly and implement it, or we need to decide whether it is expedient to send out an interim mail to any potential TWiki admins that we can find addresses for. The latter would mean an additional one shot (unsolicited) mail so I'd prefer to get the mailing lists sorted out as quick as possible.

-- SamHasler - 25 Nov 2004

We are still getting visitors to TWikiIRC asking about the "new threat" and telling us that they were not notified. There is significant loss of confidence in the process and thus the software. I am with Clauss: we must notify ALL REGISTERED USERS not just the 500 on WebNotify. Else we all look incapable of taking decisive, corrective action.

Two lines of questions:

• Where is a page detailing the setting up the mailing list? To whom has this task been assigned? Why is there a delay?
• Why has the front page of http://twiki.org not been updated? (Indeed, why is it stored as HTML instead of being in the wiki?)

I am concerned for the credibility of this project - like many of you, I have invested a significant proportion of my working career in this software product.

If anyone received the said email, please attach it here. I will then email all registered TWiki.org users with the notice by midday tomorrow. (I will check here first to see whether anyone else has done it).

-- MartinCleaver - 25 Nov 2004

There is as yet no action on the email list... Unfortunately I can't set up the announce email list myself at SourceForge but I have emailed Peter and the CoreTeam to try to get this done ASAP. I would prefer to get the SourceForge email list set up first (with zero members) and then mailshot everyone one-off re the current hole, inviting them to the announce list. My idea is that this list should be for security alerts and new release announcements only.

If we can't get the email list set up today, I think we should do the one-shot email today anyway, and mention that we will mail them again soon about the announce list. Obviously, it's preferable to email only once but the fact is that it's been almost 2 weeks since the vulnerability was posted widely to security email lists, yet the TWiki user community not on such lists has not been informed - in this case some additional email is less important than getting the word out.

-- RichardDonkin - 25 Nov 2004

We now have a SecurityTeam.

-- SamHasler - 25 Nov 2004

Interesting history of this vulnerability from point of view of one of the people who found it - probably several sides to this story, but it was sat on for several months when it could have been fixed, rather than going out to the full disclosure lists as quickly as it did.

• And one more point of view, advisory-facts.txt: Timeline of security alert, my reply to the flamewar going on between the two parties that raced to publish the vulnerability first. -- PTh

The common vulnerability (CVE) code for this hole is CAN-2004-1037 - Google:CAN-2004-1037 will find the various reports on the web, as well as the exploit code. However, this Google finds the Nov 19th incarnation of the alert, from Roman Medina-Heigl Hernandez, who claims priority - the Nov 12th Bugtraq entry does not have the CVE code but is equivalent.

-- RichardDonkin - 25 Nov 2004

• email-sec-notice.txt: The email security alert sent out on Nov 12th.

-- MattWilkie - 25 Nov 2004

I sent 500+ e-mail to known site administrators on Fri, 12 Nov. This happended 6 hours after I got notified of the issue. I am aware that it did not reach every public TWiki installation. The security advisory got released uncoordinated and was not in line with our process, published since ages in BugReport.

It is the wish of the community to establish a TWiki announce mailing list and to invite everyone who looked at TWiki in the past. The mailing list is not ideal since crackers will get on the list, but reaching not all interested parties is also not ideal. So lets establish a mailing list. It is very important to coordinate the efforts, uncoordinated efforts just cause frustration as we have seen. Lets avoid multiple e-mails, everyone has enough of Spam. Again, nobody should harvest e-mail addresses on TWiki.org to send out an uncoordinated e-mail. The goal is to send out a mass mailer no later then Sat, 27 Nov 2004. This is to (a) invite people to subscribe to the announce mailing list, (b) alert of the SecurityAlertExecuteCommandsWithSearch issue. This is a big mass-mailer to over 20K people.

Actions to take:

-- PeterThoeny - 25 Nov 2004

Out of interest, what was the ratio of opt-in to opt-out for the download form?

• 76% Yes, 24% No (default was Yes) -- PTh

If the number of opt-outs was large I would consider sending them a mail as well as we're only going to do it this one time (unless the checkbox was opt-out. Sending to people who deliberatly opted out is different from sending mails to people who overlooked or decided not to opt-in).

I'm still interested in what the ratio is even if we decide not to mail them.

I'd think it would be a good idea to do a google search to pull the e-mail address from the WIKIWEBMASTER TWikiPreferences setting as suggested above and add these to the list as they are the most likely to be affected by any security issues (and they are publicly available addresses after all.)

-- SamHasler - 25 Nov 2004

Now that the email list is set up at TWikiAnnounceMailingList, and apparently working, I hope that the TWikiSecurityAlertEmail can be sent out on Fri 26 Nov. Good to see things moving forward.

-- RichardDonkin - 25 Nov 2004

SueLocke pointed out in TWikiIRC that TWikiInstallations would be "a first port of call as to where the public TWiki sites are" for a hacker.

I think we should consider moving them to a web of their own so we can lock them down while we are dealing with security issues.

-- SamHasler - 25 Nov 2004

I think it is overkill to remove TWikiInstallations, crackers can Google public sites as easily.

This has high priority: Who can help out and compile the e-mail addresses of public TWiki site admins? E.g. a Google search on TWikiWebPreferences, and automated extract of the e-mail address from the WIKIWEBMASTER preferences setting. This list should not be posted on TWiki.org for obvious reasons, please send it to me via e-mail.

I do not want to send out separate e-mails to TWiki.org registered users, users of legacy download form, and public site admins. The three lists will be combined/sorted/uniqued into one list so that the mass-mail does not spam people more then once.

Also, the e-mail list needs to be treated with great care, it must not get into the hands of organizations with commercial interests. We also cannot afford to be stamped as spammers.

-- PeterThoeny - 25 Nov 2004

I just posted this to CoffeeBreak:

Here's a much more conservative google search for twiki installations. (It finds around 800)

http://www.google.com/search?q=inurl:TWikiPreferences+inurl:view+-intitle:Main+-inurl:rev+-inurl:skin+-inurl:raw+-inurl:sortcol&hl=en&lr=&c2coff=1&safe=off&as_qdr=all&filter=0

inurl:TWikiPreferences inurl:view -intitle:Main -inurl:rev -inurl:skin -inurl:raw -inurl:sortcol

Note:

• It is a "repeat the search with the omitted results included" which doesn't affect the numbers but now you see multiple installations at the same site in the results (under different ~users in the url).
• inurl:view will exclude sites that use short urls.
• -inurl:rev -inurl:skin -inurl:raw could potentially exclude some sites that have these in their domain name.

I can't think of a way to easily isolate installations using ShorterURLs other than doing a -inurl:scriptname for every bin script. (haven't had time to try that yet and I'm going to bed soon)

-- SamHasler - 26 Nov 2004

People who rename their TWikiPreferences topic won't be found using any of these approaches. I don't have anything constructive about how to work around it, I just didn't want them to be completely unthought of.

There are a number of unremedied security issues I'm aware of which have already been published in one form or another on twiki.org but are not tagged as a known issue or in the alerts. Should this email a core team member process be followed to get them in the queue? Or is a BugReport a better idea?

-- MattWilkie - 26 Nov 2004

We will soon have a 'mail the SecurityTeam' email list set up - until then, if you could email any critical ones (remote holes) to a CoreTeam member that would be great. I'm hoping there are no more pending remote holes just yet...

-- RichardDonkin - 26 Nov 2004

Not sure we should worry about people renaming TWikiPreferences - does anyone really do that?

Since ShorterURLs are fairly high-tech, I'd expect more of those site admins are active on TWiki.org anyway.

I've refined the query slightly - this query gets the WIKIWEBMASTER setting in the Google preview text, so it should be possible using the Google API to grab all the email addresses without visiting every page. Of course, that means some may be out of date if we are unlucky, but seems like a good idea for the first pass.

I've now run this query, giving 619 results, and done some post-processing - have passed the results onto Peter, Sven, and a few others in hopes someone can pick this up. The email list should be passed by email and not sent to lists that are archived.

-- RichardDonkin - 26 Nov 2004

I renamed TWikiPreferences in a couple of my first largely experimental twikis to the more descriptive SystemPreferences. I gave up because of the extra hassle when upgrading. (just found at least one other with non-standard prefs topic. seems to be a dead wiki. I forwarded him the Nov12th alert already).

Also the search string uses -intitle:Main but really old installs won't have a twiki web, just a Main. And, a few collapse all their webs into a single web, which could be named anything. Can "Set WIKIWEBMASTER" be used as the canonical key search phrase instead?

I'm not suggesting at all that no message should go out until each and every twiki has been identified. I'm just trying to think of who might get missed in this first pass.

I've noticed something else suspicious: ZhiXiongkang is registered in an awful lot of twikis, doesn't post anything, and all the registrations happen in the same day (either Nov 15th or Oct 15th).

-- MattWilkie - 26 Nov 2004

'nother observation: we're gonna have to use the google cache since a fair number of those wiki sites are now "you don't have permission to access this resource".

-- MattWilkie - 26 Nov 2004

Richard already sent me the list of site admins. I am currently cleaning it up. Some manual work involved due to incomplete e-mail addresses.

-- PeterThoeny - 27 Nov 2004

A lot of work was involved in fixing the incomplete addresses, e.g. from someone@xyz to someone@xyz.edu by checking the site URL and visiting the site.

10% duplicates could be removed by combining the three lists into one, e.g. less chance of reaching s single user more then once.

-- PeterThoeny - 27 Nov 2004

The Parrot VM wiki can be added to the list of twiki's hit. Apparently around 15 Nov. The compromised server was then apparently used in an ebay scam. The server, and my entire domain, was then disabled by the web hosting provider. The twiki was googleable, but not I think listed on twiki.org. Fyi.

-- MitchellNCharity - 28 Nov 2004

It would be nice to have it on something like LWN. LWN, for example, carried recent security alerts for Apache and MoinMoin. Hint Hint.

-- TomOehser - 28 Nov 2004

It doesn't help that a notice went out November 12th - but not to me - and that I was hacked on November 15th. Probably the November 12th notice led to my being hacked...?

-- TomOehser - 28 Nov 2004

The Nov 12th announcement went to a range of security lists such as BugTraq, read by a wide range of people. Unfortunately at that point the TWiki community did not have the TWikiAnnounceMailingList for such security alerts - PeterThoeny did send email to about 500 people registered in TWikiInstallations, but inevitably many people were not notified. We have now sent out a TWikiSecurityAlertEmail to a much wider range of people, which will also help to ensure they sign up for the twiki-announce email list. Clearly, the TWiki world was not ready for this sort of large-scale attack on TWiki sites, publicised on these lists with a handy exploit script - however, we are a lot better prepared for this sort of thing now.

I've also updated the process above to reflect that emails should now go to the new TWiki SecurityTeam.

-- RichardDonkin - 28 Nov 2004

It is also worth pointing out that the public advisory got released uncoordinated, around the same time I sent the first big alert to known site administrators. This was not in line with our published process.

-- PeterThoeny - 30 Nov 2004

Great. I am glad we finally had notification, even if it was too late to save many sites.

My point has never been to create discontent - only to serve customers and get both direction and movement. As the project accelerates we need priorities to ensure we tackle big issues, but we also need listening, agreement, vision and planning.

With respect to your request: I've put the internet deployment spider up again, and I have set it to automatically update every couple of days. Here's WIKIWEBMASTER:

• http://wikiconsulting.com/PublicWikisOverview/FieldSelect?fieldname=WIKIWEBMASTER

-- MartinCleaver - 28 Nov 2004

Now that the big mailing is out we can do some final touches on the alert process. I changed the timing on the alert process, open for comments:

• Initial alert: Alert TWikiDevMailingList members and administrators of public TWiki sites by e-mail (based on Google search and TWikiInstallation directory)
• After 2 day grace period: Send alert to TWikiAnnounceMailingList members
• After 4 day grace period: Issue a public security advisory

Comments:

• The TWikiDevMailingList is public, but the probability that crackers monitor it is probably lower then the big TWikiAnnounceMailingList
• A grace period of 2 to 3 days is apparently standard practice

-- PeterThoeny - 30 Nov 2004

I appreciate what you are trying to do, but I think you are at risk of promising what you can't deliver. You can't guarantee those response times, so better not to advertise them. Also, grace periods and suchlike need to be decided on a case-by-case basis. Maybe I'm just too simple, but it seems to me that the process can be stated more simply, just hitting the key points:

• Triage the problem: verify and assess severity (SecurityTeam members act alone or in concert if all awake)
• If the problem could compromise site or data security:
  1. Mail the security mailing list immediately, so admins can take public servers down if they have to
  2. Compose a hotfix ASAP & verify it
  3. Update release packages
  4. Mail the hotfix to the security and announce lists.
  5. Generate testcase to ensure the problem can never arise again

BTW can someone please generate a testcase for the recent alert, so we can't fall prey to it again!

-- CrawfordCurrie - 30 Nov 2004

I agree with the idea of grace periods - the aim is to give people time to fix the issue (i.e. elongating step 4 by mailing people who are definitely TWiki administrators). However, we have advertised the TWikiAnnounceMailingList as getting security alerts immediately, so I think we should just send the alert to both twiki-dev and twiki-announce - since twiki-dev is archived, there's a risk Google picks it up, or someone forwards it on to X then Y then Z. As long as there's no exploit attached to the alert (I think Apache are quite good at doing this) I think the degree of hacking would be less.

Lots of people don't want to be on twiki-dev, only on the twiki-announce list, due to volume, so it seems unfair to delay the twiki-announce email.

The 'typical' timeline according to one security person I talked to is:

1. Researcher notifies security vulnerability to author
2. (2-3 days later) Author sends out alert including fix
3. Software users apply fix and/or close off vulnerable systems
4. (2-3 days later) Research sends out exploit

While having exploits makes hacking easier, it also makes it easier for administrators to test if their system is truly vulnerable or proof against the vulnerability. The real world is not as well structured as this.

Also worth noting that time from step 1 to step 4 was 10 hours in the case of the recent search hole, giving Peter little time to create and distribute fix, and TWiki admins little time to apply fix. The behaviour of the security researchers involved was not exactly pristine in this case, since the exploit came out so fast (on a Friday evening as well), leading to lots of cracked TWiki systems.

-- RichardDonkin - 30 Nov 2004

Not that it matters, but my twiki was compromised and I've written a report about the incident, of which you might want to read the second paragraph, in which I take half the blame. The report is at http://www.itia.ntua.gr/twiki/bin/view/Main/ReportOnTwikiCompromise.

But that's not the reason I'm writing. I want to give my 2 cents to the security process. I'm not a security expert, but it doesn't seem that anyone in this discussion is, so here we go:

How fast you send a public alert depends on whether the problem is already public or being actively exploited. It is important to keep in mind that twiki may be distributed by operating system vendors. The forthcoming release of the Debian OS will have a prepackaged twiki.

Now administrators who have installed their vendor's prepackaged version rather than the original version will normally not want to be subscribed to the twiki mailing list, but only to the vendor's security alert service, and they will prefer to wait for the vendor's patch. For this reason, if the discovered problem is not publicly known or actively exploited, you normally keep it secret and only notify CERT, I believe (although I'm not sure about the details). CERT has an established procedure of notifying the vendors, and each one prepares/adapts, but does not announce, the needed patch and advisory. When everyone's ready or it is thought that the problem can no longer wait, CERT tells to all interested parties "OK, release now", and they issue their advisories altogether. This way there is minimal delay between the original advisory/patch and the vendor advisory/patch.

When the problem is publicly known or exploited, I believe you announce as soon as possible, but still have to notify vendors or CERT.

The thing is, rather than reinvent the wheel, you should find some security experts to help you setup the process similarly to what must have already been done for hundreds of programs. What you do when it's secret, or when it's public, or when you don't know how many people know, or when you know the hole but don't have a patch yet, or when the researcher's behaviour hasn't been optimal, and so on and so on; it's certain there already exist good procedures and experience about all such questions. Maybe the Debian security team would help, if they have the time. SvenDowideit may know whom to talk to.

-- AntoniosChristofides - 02 Dec 2004

Antonios thank you very sharing your well written report on the comprimise. It, or a good portion of it, should be part of the documentation here on twiki.org I think.

There is a mispelling of twiki in the first line of the second paragraph (tiki). And I think a typo in the find command example. I get the message "find: paths must precede expression" when running it.

I get "Additionally, a 404 Not Found error was encountered while trying to use an ErrorDocument? to handle the request." on a failed authentication (it should go to a twiki oops page). Probably your bin/.htaccess does not the correct path for ErrorDocument 401

-- MattWilkie - 02 Dec 2004

Thanks, I fixed the typos. The other problem shall remain, I'm afraid, until my queue is empty (around 2019 :-).

-- AntoniosChristofides - 02 Dec 2004

Re my posting above on security report / fix timelines - this security alert policy for some useful guidelines on how researchers should report security alerts to maintainers.

-- RichardDonkin - 10 Jan 2005

Update on security process: Discussing with Kenneth, we felt that it is sufficient to alert twiki-dev and twiki-announce, and no longer alert the admins listed in the TWikiInstallation directory and the ones found by a Google search. This is because it is well publicised to subscribe to twiki-announce. We also change the grace period from 4 days to 3 days to make it less likely that alerts go out on a weekend.

-- PeterThoeny - 13 Jun 2006