Tags:
statistics1Add my vote for this tag create new tag
, view all tags

LogsStatisticsPlugin

Create statistics from the log files

Introduction

The LogsStatisticsPlugin handles the tag STATISTICSLOGS that inserts a summary of data from the TWiki log files based on user criteria.

Besides standard statistical usage the LogsStatisticsPlugin can be used to identify problems with oversize numbers of requests or to present usage statistics based on TWiki logs.

Syntax Rules

%STATISTICSLOGS{...}%

Argument Comment Default value Values to choose Comment on values Example
date
set time period to analyze
today
today
analyze only today's data
date="today"
yesterday
analyze only yesterday's data
date="yesterday"
month
analyze data from the current month
date="month"
last X days
analyze data from last X days
date="last 13 days"
dd MMM yyy
analyze data from a specific date
date="01 Jun 2010"
dd MMM yyy - dd MMM yyyy
analyze data from a specific period
date="31 Mar 2010 - 01 Jun 2010"
type
set type of the summary
top5 users
topX (users|webs|topics|crawlers)
returns top X users, webs, topics or crawlers
type="top10 webs"
unique (users|webs|topics|crawlers)
returns number of unique users, webs, topics or crawlers
type="unique webs"
output
set output type for the summary
table
table
returns summary as a table with headings
output="table"
list
returns summary as a list
output="list"
description
prints a line with parameters summary under the table/list
output="table, description"

It is also possible to narrow data to the records containing specific values. Different parameters of the same type should be divided with commas, spaces will be automaticly deleted.

Argument Comment Example
user
narrow data to the records coming from a specific user
user="WikiName"
user="WikiName1, WikiName2"
web
narrow data to the records coming from a specific web
web="SandBox"
web="SandBox, Main"
topic
narrow data to the records coming from a specific topic
topic="WebHome"
topic="WebHome, TestPage"
action
narrow data to the records coming from a specific TWiki actions
action="view"
action="view, edit"
ip
narrow data to the records coming from a specific ip address
ip="127.0.0.1"
ip="127.0.0.1, 192.168.1.1"

Notes and hints:

  • The more parameters, the slower it will work.
  • If the summary period covers the current day then the result data may vary every time when the LogsStatisticsPlugin is being called, this is because the log files are continuously updated.
  • One can define a period covering future dates, but the plugins will analyze data only up to the servers date.
  • If the log file for a specific month does not exists then the plugin will not report any errors.
  • Output parameter should be divided with commas, spaces will be automaticly deleted.
  • If the output attribute with the table value is used then one can customize it using for example TablePlugin or ChartPlugin.
    %TABLE{ name="tableAL1" sort="on" tableborder="0"  headerbg="#D5CCB1" headercolor="#666666" databg="#FAF0D4, #F3DFA8" headerrows="2" footerrows="1" }%%STATISTICSLOGS{date="last 10 days" type="top3 users"}%
    or
    %TABLE{ name="tableAL"} }%%STATISTICSLOGS{date="last 10 days" type="top3 webs"}%
    %CHART{ type="bar" name="bar" table="tableAL" data="R2:C2..R4:C2"  legend="R1:C2..R1:C2" xaxis="R2:C1..R4:C1" }%

Example

  • Which 5 users are making the most requests today?
    %STATISTICSLOGS{date="today" type="top5 users"}%
  • How many unique topics have been viewed in the Atlas web in the current month?
    %STATISTICSLOGS{date="month" type="unique topics" action="view" web="Atlas"}%
  • Customizing tables:
    %TABLE{ name="tableAL1" sort="on" tableborder="0"  headerbg="#D5CCB1" headercolor="#666666" databg="#FAF0D4, #F3DFA8" headerrows="2" footerrows="1" }%%STATISTICSLOGS{date="last 10 days" type="top3 users"}%

Top users Number of entires
WikiName3 63056
WikiName1 187268
WikiName2 76007

  • Creating charts from reports
    %TABLE{ name="tableAL" }%%STATISTICSLOGS{date="last 10 days" type="top3 webs"}%
    
    %CHART{ type="bar" name="bar" table="tableAL" data="R2:C2..R4:C2"  legend="R1:C2..R1:C2" xaxis="R2:C1..R4:C1" }%

Top webs Number of entires
Main 265189
TWiki 101910
Sandbox 26183

Plugin Settings

Plugin settings are stored as preferences variables. To reference a plugin setting write %<plugin>_<setting>%, for example, %LOGSSTATISTICSPLUGIN_SHORTDESCRIPTION%

  • One line description, shown in the TextFormattingRules topic:
    • Set SHORTDESCRIPTION = Create statistics from the log files

  • Set DEBUG to 1 to get debug messages in data/debug.txt. Default: 0
    • Set DEBUG = 0

  • Set a path to the log files (if no option then by default set to $TWiki::cfg{LogFileName}). Path should containt %DATE%
    • Set LOGSDIRECTORY =

  • Set a list of groups and/or users allowed to use the plugin. If the user name and the group name applys for the current user then the user name parameters will be set. Names should be divided with semicolons, spaces will be automaticly deleted. One can specify a list of webs for the user/group UserName: web="Sandox, Main".

  • Set to yes if you want the plugin to cache the results for queries which don't contain server's current date. The load generated by the plugin will be significantly reduced, if this option is enabled then the plugin will need to analyze log files only once for each unique query (which doesn't include the server's current date).
    • Set CACHEOLDRESULTS = yes

  • Set to all or months if you want the plugin to cache the results for queries which contains a server's current date. The load generated by the plugin will be significantly reduced. Cache stored in this way can be reused for the same queries for different time periods, if this option is enabled then the plugin will need to analyze every time only the current month's log files. This option can enable caching results in two different ways.
    • If the option months is enabled then the plugin will store results for all the past months.
    • If the option all is enabled then the plugin will store results for all the past months and every day of the current months besides the current server's date.
    • Set CACHECURRENTDATERESULTS = all

  • Set a directory where the plugin can store it's cache files (if no option then by default set to the plugin's working area). Files will be written only if caching is enabled
    • Set CACHEDIR =

Plugin Installation Instructions

  • Download the ZIP file from the Plugin web
  • Unzip the zip file in your root ($TWIKI_ROOT) directory.
  • Visit configure in your TWiki installation, and enable the plugin in the {Plugins} section.

Notes and hints

  • If any manual changes were made to the server's log files (editing, adding, deleting log files) and if caching results is enabled then the administrator should remove all the files from the plugin's working area.
  • It is advisable to remove daily files (starting wit letter 'd') from the plugin's working area at the beginning of month. Daily files from the previous month will not be used anymore.

Plugin Info

Plugin Author: TWiki:Main/ZbigniewGrzybek
Copyright: © 2010, TWiki:Main/ZbigniewGrzybek
License: GPL (GNU General Public License)
Plugin Version: 2010-08-11 (V1.000)
Change History:
2010-08-11: Initial version
TWiki Dependency: $TWiki::Plugins::VERSION 1.1
CPAN Dependencies: none
Other Dependencies: Date::Calc; Storable; Data::Dumper; Date::Manip; List::Util; Time::Local
Perl Version: 5.005
Benchmarks: GoodStyle nn%, FormattedSearch nn%, LogsStatisticsPlugin nn%
Plugin Home: http://TWiki.org/cgi-bin/view/Plugins/LogsStatisticsPlugin
Feedback: http://TWiki.org/cgi-bin/view/Plugins/LogsStatisticsPluginDev
Appraisal: http://TWiki.org/cgi-bin/view/Plugins/LogsStatisticsPluginAppraisal

Related Topics: TWikiPlugins, DeveloperDocumentationCategory, AdminDocumentationCategory, TWikiPreferences

Topic attachments
I Attachment History Action Size Date Who Comment
Unknown file formatmd5 LogsStatisticsPlugin.md5 r2 r1 manage 0.2 K 2010-08-31 - 05:58 PeterThoeny  
Compressed Zip archivetgz LogsStatisticsPlugin.tgz r2 r1 manage 13.1 K 2010-08-31 - 05:58 PeterThoeny  
Compressed Zip archivezip LogsStatisticsPlugin.zip r4 r3 r2 r1 manage 14.8 K 2010-08-31 - 05:58 PeterThoeny  
Unknown file formatEXT LogsStatisticsPlugin_installer r1 manage 3.8 K 2010-08-31 - 05:58 PeterThoeny  
Edit | Attach | Watch | Print version | History: r3 < r2 < r1 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r3 - 2010-08-31 - PeterThoeny
 
  • Learn about TWiki  
  • Download TWiki
This site is powered by the TWiki collaboration platform Powered by Perl Hosted by OICcam.com Ideas, requests, problems regarding TWiki? Send feedback. Ask community in the support forum.
Copyright © 1999-2015 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.