TWikiStatsPlugin

Gather and Display and Analyse TWiki usage statistics in real time

On this page:

Usage

Day to day usage

To see how your TWiki is being accessed, there are a number of Summary topics that graph the gathered information:

Data Gathering

TWikiStatsPlugin has two data gathering modes, live data, where it uses 'tick' files that are created for every web server operation (to reduce the chances of any web requests being blocked or deleyed by the logging), or by post analysing the existing Apache access log files.

To limit the database impact of the logs, only the last 3 months of log and minute-results are kept. This means that when you import and analyse historical data, you need to do so in sequential time order. (unless you set $TWiki::cfg{Plugins}{TWikiStatsPlugin}{DeleteOldData} to false)

Due to a lack of standardisation in date time operations, only log files since 2000 are supported.

Rest Handler to collate live data from this TWiki

This handler uses the Tick files created on every twiki script action, and the PubAccess? mod_perl handler (if enabled). As it imports the data into the log table, it delets the matching tick file, and after that, runs the Analysis SQL.

From twiki/bin run ./rest TWikiStatsPlugin.collate.

This is best done from a Cronjob every minute or so 

# m h  dom mon dow   command
* * * * * cd /var/twiki411_stats/bin ; ./rest TWikiStatsPlugin.collate > /var/twiki411_stats/data/TWikiStatsPlugin.log

Rest Handler to import historical data or batch import (apache access log)

Imports the specified log file into the log table, and then runs the Analysis SQL.

From twiki/bin run ./rest TWikiStatsPlugin.importApache filename /var/log/apache/access_log.20070717.

This is best run after the apache log file is 'rolled over'.

Note that no attempt is made to prevent you from re-importing the same log information more than once. Doing so will produce incorrect results (the new imported log entries will be added to the previously imported ones).

If you choose to not use the live collation method above, turn off tick file generation by turning off $TWiki::cfg{Plugins}{TWikiStatsPlugin}{UseTickFiles} in configure.

createTables restHandler

Creates the tables needed for TWikiStatsPlugin to work. If you run it after the tables exist, it will show errors for those tables (not all version of sqlite have CREATE TABLE IF EXISTS).

From twiki/bin run ./rest TWikiStatsPlugin.createTables.

runAnalysis restHandler

Re-Runs the Analysis SQL for all log data. This can take several hours, and should not be necessary except for changes to the SQL.

Note that you should really remove all the old result_ data (fastest way is to drop the resutl_* tables, and call the createTables handler)

From twiki/bin run ./rest TWikiStatsPlugin.runAnalysis.

runReport restHandler

Analyses the report topic specified, and sends an email if necessary.

From twiki/bin run ./rest TWikiStatsPlugin.runReport report TWikiStatsReport.

Best run from cron. 

# m h  dom mon dow   command
* * * * * cd /var/twiki411_stats/bin ; ./rest TWikiStatsPlugin.runReport report TWiki.TWikiStatsReport > /var/twiki411_stats/data/TWikiStatsReport.log

TWIKISTATS

Used to display gathered values derived from the log. Has the following parameters:
  • DEFAULT - the output format
  • count - the number of results to show - uses period and step to determine step size (1 by default)
  • relativetime - used to offset the initial time (goes backwards in time - so relativetime="12" period="month" would start 12 months in the past
  • period - which results table to use (and the basic interval) one of minute, hour, day, month (defaults to hour)
  • step - how many periods to increment each output value by (used to add the xaxis to the Flot Charts)
  • separator - the separator used if count is greater than one ($n by default)
  • unset - used for any value that has no entry in the database

The output will always go backwards in time from the relativetime counter.

The DEFAULT output format can contain the following

  • $count -
  • $timeKey - the datetime of the datapoint being output (in the form of the relevant substring of=YY-MM-DD HH:MM= )
  • $totalAccessCount - the count of all log entries that are filtered into the database
  • unique{METRIC} - the count of normalised (or unique) {METRIC} accessed (ie. if there are only 12 requests for Main the unquie count == 1)
  • new{METRIC} - the count of normalised (or unique) {METRIC} accessed this period that were not last period
  • total{METRIC} - the total number of {METRIC} occurring in the log
  • max{METRIC} - The the maximum value of {METRIC} (ie, if the most requested topic was Main, $maxTopicsName ($maxTopics) would return Main.WebHome (12))
  • max{METRIC}Name - The Username (if a peruser metric) or Value Name that has the maximum value of {METRIC}
  • ratio{METRIC} - Average number of Accesses per unique {METRIC} (analogous to total{METRIC} / unique{METRIC}
  • maxratio{METRIC} - the percentage of the {METRIC} total (analogous to 100% * max{METRIC} / total{METRIC}
where {METRIC} can be any one of
  • Topicsperuser - unique topic count vs user (can be used to find out who has accessed the most different topics)
  • Filesperuser - unique file count vs user
  • Agentsperuser - unique agent count vs user
  • Statusperuser - unique status count vs user
  • Users
  • Topics - TWiki Topic accesses
  • Files - direct attachment accesses
  • Agents
  • Scripts
  • Status

TWIKITOPSTATS

Lists the top name and value pairs of a metric for a specified time period. For example, show the top 10 Users.
  • DEFAULT - the metric to show - same names as can be in the DEFAULT parameter of %TWIKISTATS% above
  • format - $name, $value, $total, $percent and the values specified for the DEFAULT parameter of %TWIKISTATS% above (defaults to '   * $name - $value')
  • count - the number of name value pairs to list (10 by default)
  • relativetime - used to offset the initial time (goes backwards in time - so relativetime="12" period="month" would start 12 months in the past
  • period - which results table to use (and the basic interval) one of minute, hour, day, month (defaults to hour)
  • separator - the separator used if count is greater than one ($n by default)
  • unset - used for any value that has no entry in the database

note that this Macro is likely to be merged into TWIKISTATS at a later date

Examples

graphs using FlotChartPlugin

last 12 months

last 24 hours

last 2 hours (in minutes)

Top 10 summaries

users this month (342)

users today (105)

users last hour (17)

users this hour (9)

topics this month (1267)

topics today (708)

topics last hour (30)

topics this hour (10)

files this month ((no statistics gathered for 08-08 yet))

files today ((no statistics gathered for 08-08-15 yet))

files last hour ((no statistics gathered for 08-08-15 15:00 yet))

files this hour ((no statistics gathered for 08-08-15 16:00 yet))

scripts this month (8)

  • view : 4946 (99.60 %)
  • genpdf : 8 (0.16 %)
  • oops : 6 (0.12 %)
  • save : 2 (0.04 %)
  • viewauth : 1 (0.02 %)
  • statistics : 1 (0.02 %)
  • edit : 1 (0.02 %)
  • login : 1 (0.02 %)

scripts today (2)

  • view : 1192 (99.83 %)
  • oops : 2 (0.17 %)

scripts last hour (1)

  • view : 52 (100.00 %)

scripts this hour (1)

  • view : 16 (100.00 %)

agents this month (27)

  • Mozilla : 3051 (61.44 %)
  • msnbot : 753 (15.16 %)
  • whoisi : 205 (4.13 %)
  • BuzzTracker? : 195 (3.93 %)
  • Baiduspider : 180 (3.62 %)
  • Moreoverbot : 125 (2.52 %)
  • Liferea : 113 (2.28 %)
  • Feedfetcher : 109 (2.19 %)
  • Bloglines : 80 (1.61 %)
  • Akregator : 61 (1.23 %)

agents today (14)

  • Mozilla : 868 (72.70 %)
  • msnbot : 192 (16.08 %)
  • BuzzTracker? : 27 (2.26 %)
  • whoisi : 27 (2.26 %)
  • Liferea : 17 (1.42 %)
  • Bloglines : 16 (1.34 %)
  • Feedfetcher : 16 (1.34 %)
  • Moreoverbot : 12 (1.01 %)
  • TechnoratiSnoop? : 6 (0.50 %)
  • Python : 5 (0.42 %)

agents last hour (10)

  • Mozilla : 35 (67.31 %)
  • whoisi : 4 (7.69 %)
  • msnbot : 4 (7.69 %)
  • Moreoverbot : 2 (3.85 %)
  • TechnoratiSnoop? : 2 (3.85 %)
  • Bloglines : 1 (1.92 %)
  • Liferea : 1 (1.92 %)
  • Arphzxfuw : 1 (1.92 %)
  • Pebgkduo : 1 (1.92 %)
  • Feedfetcher : 1 (1.92 %)

agents this hour (4)

  • Mozilla : 10 (62.50 %)
  • msnbot : 4 (25.00 %)
  • BuzzTracker? : 1 (6.25 %)
  • Bloglines : 1 (6.25 %)

Installation Instructions

The gathered data is kept in a Perl DBI compatible database - I have used both SQLite3 >= 3.2.6 (for smaller systems) and MySQL? >= 4.1. SQLite3 v3.2.1 works with some degraded ability - the max/ratio/maxratio results are not useable, and there non fatal errors in the collation output.

You do not need to install anything in the browser to use this extension. The following instructions are for the administrator who installs the extension on the server where TWiki is running.

Like many other TWiki extensions, this module is shipped with a fully automatic installer script written using the BuildContrib.

  • If you have TWiki 4.2 or later, you can install from the configure interface (Go to Plugins->Find More Extensions)
  • If you have any problems, then you can still install manually from the command-line:
    1. Download one of the .zip or .tgz archives
    2. Unpack the archive in the root directory of your TWiki installation.
    3. Run the installer script ( perl <module>_installer )
    4. Run configure and enable the module, if it is a plugin.
    5. Repeat for any missing dependencies.
  • If you are still having problems, then instead of running the installer script:
    1. Make sure that the file permissions allow the webserver user to access all files.
    2. Check in any installed files that have existing ,v files in your existing install (take care not to lock the files when you check in)
    3. Manually edit LocalSite.cfg to set any configuration variables.

input data Filters

To allow you to restrict the gathering of data to a subset of webs, users, scripts etc, there are 7 input Filters These are regex tests that need to succeed for the log entry to be recorded in the log database. Any non-matching log entries are totally discarded.
  • $TWiki::cfg{Plugins}{TWikiStatsPlugin}{ScriptFilter} = qr/^.*$/;
  • $TWiki::cfg{Plugins}{TWikiStatsPlugin}{WebDotTopicFilter} = qr/^.*$/;
  • $TWiki::cfg{Plugins}{TWikiStatsPlugin}{FileNameFilter} = qr/^.*$/;
  • $TWiki::cfg{Plugins}{TWikiStatsPlugin}{UserFilter} = qr/^.*$/;
  • $TWiki::cfg{Plugins}{TWikiStatsPlugin}{RemoteHostFilter} = qr/^.*$/;
  • $TWiki::cfg{Plugins}{TWikiStatsPlugin}{RemoteAgentFilter} = qr/^.*$/;
  • $TWiki::cfg{Plugins}{TWikiStatsPlugin}{StatusFilter} = qr/^.*$/;

Internals

The database, and log files are assumed to be in 'local time' so that the day and month statics are calculated on the local time boundries.

Plugin Info

Many thanks to the following sponsors for supporting this work:

  • Acknowledge any sponsors here

Plugin Author(s): TWiki:Main.SvenDowideit - http://distributedINFORMATION.com a wikiringlogo20x20.png WikiRing founding partner
Copyright: © SvenDowideit@distributedINFORMATION.com
License: GPL (Gnu General Public License)
Plugin Version: 0 (07 Feb 2008)
Change History:  
Dependencies:
NameVersionDescription
TWiki::Plugins::FlotChartPlugin>=1.0FlotChartPlugin release 1.0
  SQLite3 >= 3.2.6 (for smaller systems) and MySQL? >= 4.1.
Plugin Home: http://twiki.org/cgi-bin/view/Plugins/TWikiStatsPlugin
Feedback: http://twiki.org/cgi-bin/view/Plugins/TWikiStatsPluginDev
Appraisal: http://twiki.org/cgi-bin/view/Plugins/TWikiStatsPluginAppraisal

Related Topics: TWikiPlugins, DeveloperDocumentationCategory, AdminDocumentationCategory, TWikiPreferences

I Attachment Action Size Date Who Comment
elserrd TWiki.rrd manage 42.7 K 03 Feb 2008 - 02:34 UnknownUser  
pngpng totalAccessCount-daily.png manage 15.2 K 03 Feb 2008 - 02:34 UnknownUser  
pngpng totalAccessCount-hourly.png manage 16.8 K 03 Feb 2008 - 02:34 UnknownUser  
pngpng totalAccessCount-monthly.png manage 14.7 K 03 Feb 2008 - 02:29 UnknownUser  
pngpng totalAccessCount-uniqueTopics-uniqueUsers-daily.png manage 15.7 K 02 Feb 2008 - 19:52 UnknownUser  
pngpng totalAccessCount-uniqueTopics-uniqueUsers-hourly.png manage 15.3 K 02 Feb 2008 - 23:20 UnknownUser  
pngpng totalAccessCount-uniqueTopics-uniqueUsers-monthly.png manage 15.7 K 02 Feb 2008 - 19:52 UnknownUser  
pngpng totalAccessCount-uniqueTopics-uniqueUsers-uniqueFiles-hourly.png manage 15.0 K 01 Feb 2008 - 04:29 UnknownUser  
pngpng totalAccessCount-uniqueTopics-uniqueUsers-weekly.png manage 14.9 K 02 Feb 2008 - 19:52 UnknownUser  
pngpng totalAccessCount-weekly.png manage 14.0 K 03 Feb 2008 - 02:06 UnknownUser  
pngpng totalAccessCount-yearly.png manage 12.1 K 02 Feb 2008 - 03:16 UnknownUser  
pngpng uniqueFiles-daily.png manage 12.4 K 03 Feb 2008 - 02:34 UnknownUser  
pngpng uniqueFiles-hourly.png manage 12.9 K 03 Feb 2008 - 02:34 UnknownUser  
pngpng uniqueFiles-monthly.png manage 13.5 K 03 Feb 2008 - 02:34 UnknownUser  
pngpng uniqueFiles-weekly.png manage 12.0 K 03 Feb 2008 - 02:06 UnknownUser  
pngpng uniqueFiles-yearly.png manage 12.0 K 02 Feb 2008 - 03:16 UnknownUser  
pngpng uniqueTopics-daily.png manage 16.5 K 03 Feb 2008 - 02:34 UnknownUser  
pngpng uniqueTopics-hourly.png manage 16.5 K 03 Feb 2008 - 02:34 UnknownUser  
pngpng uniqueTopics-monthly.png manage 13.5 K 03 Feb 2008 - 02:29 UnknownUser  
pngpng uniqueTopics-weekly.png manage 16.1 K 03 Feb 2008 - 02:06 UnknownUser  
pngpng uniqueTopics-yearly.png manage 10.9 K 02 Feb 2008 - 03:16 UnknownUser  
pngpng uniqueUsers-daily.png manage 14.9 K 03 Feb 2008 - 02:34 UnknownUser  
pngpng uniqueUsers-hourly.png manage 15.8 K 03 Feb 2008 - 02:34 UnknownUser  
pngpng uniqueUsers-monthly.png manage 12.9 K 03 Feb 2008 - 02:29 UnknownUser  
pngpng uniqueUsers-weekly.png manage 14.9 K 03 Feb 2008 - 02:06 UnknownUser  
pngpng uniqueUsers-yearly.png manage 10.8 K 02 Feb 2008 - 03:16 UnknownUser  

Edit • Attach • Print version • History: r1 • Backlinks • Raw View • Raw edit • More topic actions