SID-01993: Non-Admin user 4 seconds delay when loading pages

Status:	Answered	TWiki version:	6.0.1	Perl version:	v5.18.2
Category:	CategoryPerformance	Server OS:	Ubuntu 14.04.1 LTS	Last update:	10 years ago

Hi,

Non-Admin users when login twiki and changing web pages, there is at least 4 seconds delay.

TWiki Wiki-5.0.1, build 19570 Plugin API version 1.3 OS: Ubuntu 10.04.1 LTS TWiki TWiki-6.0.1, build 28207, Plugin API version 6.01 OS: Ubuntu 14.04.1 LTS

Both version has the delay issue.

_linkToolTipInfo is disabled.

Please help how to troubleshoot this issue and let me know if anything is needed.

Thanks,

JJ

-- JJ O'Sullivan - 2014-11-20

Discussion and Answer

The wording "Non-Admin users" seems to indicate that the admin (or internal login) does not incur the four seconds penalty when changing pages?

If so, a usual suspect would be your login mechanism, especially if you are using Apache login with some sort of single sign-on, e.g. with LDAP.

-- Harald Jörg - 2014-11-23

Harald,

Thanks for the response.

You are correct, Admin users are not affected in this case.

LoginManager is set as ApacheLogin, please see attached screenshot.

Is there any way to improve the performance for non-admin users?

Thanks,

JJ

-- JJ O'Sullivan - 2014-11-24

Ok, so my next guess: With ApacheLogin, performance depends on the authentication mechanism used by the web server. Examine your web server's configuration files for directives like AuthType and Require. Maybe you are using a login method which checks passwords against a central repository, e.g. an Active Directory, an LDAP server, or even a password file which is located on a NFS mount? If so, check the network configuration between the servers: Bogus DNS configuration with slow (or dead) servers or bad routing might cause a significant delay. You might also add LogLevel debug to your web server's configuration file, maybe the authentication mechanism writes meaningful stuff to Apache's error log.

-- Harald Jörg - 2014-11-24

LogLevel is set to debug, but only can see ther 4 seconds gap, see the screenshot below.

Here is the snippet of the config file:

<Directory />
                Options Indexes FollowSymLinks
                AllowOverride None
                AuthType Basic
                AuthBasicProvider ldap
                AuthUserFile /dev/null
                #AuthzLDAPAuthoritative off
                AuthName "NAI CORP Credentials (without the nai-corp\ prefix)"
                AuthLDAPURL ldap://xxxxxx.corp.nai.org/OU=NAI,DC=corp,DC=nai,DC=org?sAMAccountName
                AuthLDAPBindDN "CN=svcacct-xxxxxxxx,OU=Service Accounts,OU=NAI,DC=corp,DC=nai,DC=org"
                AuthLDAPBindPassword xxxxxxxxx
                require valid-user
        </Directory>

Where else can I get more info when loading pages?

Many thanks.

• LoginManager:
  

• apache access log:
  

-- JJ O'Sullivan - 2014-11-25

This looks like the access log to me, please check the error log (that's where debug information goes to). But anyway, your config file has some hints.

Apparently I've again guessed correctly: You are indeed using an Active Directory server via LDAP for authentication. This might be the place where the four seconds vanish. Also, this looks like some corporate LDAP server, so I assume there are quite a few users. As a word of warning, I've never used LDAP based authentication for TWiki, so this is more like look into the glass orb.

You are using AuthType Basic, causing the browser to add userid and password to every request after the initial logon. You also have authentication active for all objects (<Directory />), causing the Apache server to query Active Directory for every request. This could eat four seconds under some conditions:

• Many entries in Active Directory: In this case, try narrowing down the search by adding a filter to AuthLDAPURL like this:
  ldap://xxxxxx.corp.nai.org/OU=NAI,DC=corp,DC=nai,DC=org?sAMAccountName??(objectClass=user)
  See http://httpd.apache.org/docs/2.2/mod/mod_authnz_ldap.html#authldapurl for how this works.
• Bad routing: In a corporate network, both the Apache web server and the AD server are likely to be be in a protected subnet, but not necessarily in the same subnet. AD servers are not supposed to be queried by other servers, so maybe nobody provided a good route between the two subnets. You could check this on the command line of your Apache server with traceroute (if available).

If nothing helps, there are a few ways to improve performance for huge LDAP directories, but they require help from your AD and/or network folks and your corporate policy:

• Rely on TWiki's authentication/authorization/session mechanism instead of having every single request authenticated. Needs fiddling with {AuthScripts} in TWiki's configure script.
• Use another authentication method like mod_auth_kerb instead of LDAP (http://modauthkerb.sourceforge.net/index.html, available on Linux distributions). If configured correctly (AuthType Kerberos), the Web server does not need to contact AD for authentication. However, this is a major change since user ids look different than they do with LDAP, and you need an AD entry for your Apache server, and configuration can be a bit tricky.

-- Harald Jörg - 2014-11-25

That's great, I will follow all your suggestion and see what I can do.

Thanks a million.

-- JJ O'Sullivan - 2014-11-26

Hey Harald,

I have checked traceroute between the the Apache sever and LDAP server, there is only 1 hop, there shouldn't be any network problem between the two servers.

I have also added the filter "(objectClass=user)" in AuthLDAPRUL, it didn't make any difference.

Before we move forward to change the AuthType Kerberos, I have one question: Admin users have no delay issues, if it's LDAP issue, it would affect everyone on the system and not just non-Admins.

Please advise.

-- JJ O'Sullivan - 2014-11-27

Hey Harald

I work with JJ and just wanted to make sure we are clear on this.

All users us LDAP to login to our TWiki server but the admin users (in TWikiAdminUser group) not see any delay whereas non-admin users (NOT in TWikiAdminUser group) are seeing the 4 second delay.

Please let us know if you have any questions.

Thanks

Matt T.

-- Matt Tolson - 2014-11-27

Ok, though I had a number of lucky guesses, I've been wrong.

I had assumed that "admin users" referred to TWiki's internal admin user. This works slightly out-of-band with regard to TWiki's authorization mechanism. But if all users log in with LDAP and then TWiki's usual authorization mechanism based on TWikiUsers and TWikiGroups is processed then LDAP authentication seems to be innocent, and Kerberos will not improve matters. BTW: Did you check Apache's error log after enabling LogLevel debug, and also TWiki's own warning log?

Require valid-user from your web server configuration indicates that Apache does not do any authorization, so this is left to TWiki. I assume that you are using the "traditional" registration and TWiki user mapping to map from a sAMAccountName (Windows login name) to a TWiki user name. So, what could cause a difference between users in TWikiAdminGroup from users not in that group? I have only one guess left:

• Long access control lists for web and topics, or a very large number of TWikiGroups. I'd guess you need thousands of users and groups for this to have a four-seconds effect. The access check is done several times for any topic if it is composed of INCLUDEd components like left bars or top bars.

If you don't have any active access control in place, then I'm afraid that I am short of ideas right now. Sorry.

-- Harald Jörg - 2014-11-30

Hey Harald,

Our server has 3086 users in over 80 groups currently.

Would that be the cause of it and if so what can we do to resolve it?

Please let me know.

Thanks

Matt T.

-- Matt Tolson - 2014-12-01

With 3000 users I wouldn't expect four seconds per request, even with complicated TWikiAccessControl rules in place. I've got some more guesses to try to narrow it down:

• Do you have a web without active access control, e.g. the Sandbox web? Do you see the four seconds delay in this web, too?
• Do you have the slow response time for using plain skin (add ?skin=plain to any URL)?
• Is WebSearch extra slow in your installation?

What I'm aiming at is another factor which multiplies with slow access control, e.g. a large number of topics in a web.

One candidate would be a !%SEARCH{... scope="topic"}% in a top or left navigation, maybe to populate a dropdown field in a form. In that case TWiki would read and then perform access control for every topic.

As a next step, if all this doesn't give a clue, you could try profiling, but I admit that I haven't done this for several years now and forgotten almost everything about it. Codev/TWikiTimeDebug might give a start, i.e. setting the environment variable TWIKI_MONITOR might do the trick. Again, output goes to Apache's error log.

-- Harald Jörg - 2014-12-02

Hey Harald,

Using no active access control web(Sandbox web) still gets 4 seconds delay, however, using "?skin=plain" in the url really makes a big difference, no delay at all.

Does this indicate some cause?

Thanks,

JJ

-- JJ O'Sullivan - 2014-12-02

Hey Harald,

I did some testing with the ?skin= and other skins that we have installed and the only skin setting that really seems to work is the ?skin=plain.

My guess is because the plain skin does not show any of our webs or navigation.

Next I did a quick count and we have over 110 webs and/or sub webs on the site that need to be rendered to the navigation so I wonder if that has anything to do with it. :o(

Please let us know.

Thanks

Matt T.

-- Matt Tolson - 2014-12-03

Oh yes, we seem to be getting closer.

TWiki skins are a wonderful, powerful concept, and they're also .... incredibly complex, especially the most wonderful and powerful PatternSkin plus TopMenuSkin, which is the default in a new TWiki installation. I don't know whether you are using a customized skin, but just for reference I've created EmptyTopic on twiki.org, which contains just the text (empty). Now look at the source code: 420 lines of HTML, a total of 37kByte.

I suspect that some part in your skin causes the delay (because ?skin=plain is fast as hell), and that is somehow related to access control (because that's the usual suspect when admins see different behaviour than mortals). The tricky things are 1) to find out which one, and 2) to find a remedy. So, let's start with finding the culprit:

1. Create an empty page in your TWiki and look at the source code as an admin and as a mortal: Does it include incredibly long lists if stuff like e.g. a list of all topics?
   • If yes: Track it down to the user interface element to which it belongs. Do your users really need this element?
     • If no: Track it down to the template topic or file in which it is defined, and remove it.
   • If no: Look at TWikiSkinBrowser in your installation, and check which of the skins have the delay and which don't.
     • Edited to add: We've been adding comments simultaneously. You've already done this.
2. Less likely to help, but you can try to disable plugins which create content in your empty topic. In EmptyTopic, I see ShareMePlugin and TagMePlugin, but as far as I know they are not affected by admin vs. mortal.

Otherwise... well, that would need some more knowledge about your skin and installation.

Good luck!

Added after reading your recent comment: yes, 110 webs can slow down things, because it means 110 times opening and parsing their their preferences topic, maybe also expanding TWiki groups into user lists, to perform access control. If you're using the default skin, then have a look at TWiki.TopMenuSkinHomeMenu, it contains the variable %WEBLIST%. You can verify whether this is the culprit by creating a topic containing just this variable %WEBLIST% and viewing it with ?skin=plain as admin and mortal. You should see the same performance difference as with other skins.

-- Harald Jörg - 2014-12-03

Hey Harald,

It looks like you may have something here.

I created a blank topic that only had %WEBLIST% in the page and then used ?skin=plain to display the page for both admins and non-admins.

The admin users was extremely fast and the non-admin was the 4+ second delay.

Sounds like this is the culprit so what do we do now?

Please let us know.

Thanks

Matt T.

-- Matt Tolson - 2014-12-04

The easy way (assuming you are using TWiki's standard skin): Edit TWiki.TopMenuSkinHomeMenu and delete the following part:

---

%WEBLIST{
 "      * <span style=\"white-space:nowrap\"><a href=\"%SCRIPTURLPATH{"view"}%/$name/%HOMETOPIC%\"><img src=\"%ICONURL{web-bg}%\" border=\"0\" alt=\"\" width=\"16\" height=\"16\" style=\"background-color:%$markerVAR{\"WEBBGCOLOR\" web=$qname}$marker%\" />&nbsp;<nop>$indentedname %MAKETEXT{web}%</a></span>"
 limit="26" webs="%IF{"{NoInAllPublicWebs}" then="__NONEXISTENT__"}%"

---

Your users are not completely lost after this change: Though the list of available webs is no longer immediately visible in the top menu, there's still the link to the "Site map" topic. This topic contains the same list of webs, with somewhat more details for each web. So whenever someone needs the list of webs which are accessible to her, it's one click more to do, and then she'll still suffer from the four seconds delay in the site map. But every other page view is accelerated by four seconds, which should be worth it.

-- Harald Jörg - 2014-12-04

In case you do not use sub-webs, e.g. you have just a flat list of webs, turn off the {EnableHierarchicalWebs} configure setting. This should speed up things a bit.

The WEBLIST variable checks the NOSEARCHALL preferences setting and the view permission for each web unless the user is an admin. So yes, that can be slow with many webs.

If you have many webs read LargeSite and enable the MetadataRepository.

-- Peter Thoeny - 2014-12-04

Closing this question after more than 30 days of inactivity. Feel free to reopen if needed. Consider engaging one of the TWiki consultants if you need timely help. We invite you to get involved with the community, it is more likely you get community support if you support the open source project!

-- Peter Thoeny - 2015-12-03

      Change status to: Asked question Answered question Closed unanswered