Update doc

This commit is contained in:
Gregory Soutade 2014-12-23 09:18:30 +01:00
parent d2d37d9c90
commit 3dd4ddfaed
3 changed files with 352 additions and 352 deletions

View File

@ -4,7 +4,7 @@ iwla
Introduction Introduction
------------ ------------
iwla (Intelligent Web Log Analyzer) is basically a clone of [awstats](http://www.awstats.org). The main problem with awstats is that it's a very monolothic project with everything in one big PERL file. In opposite, iwla has be though to be very modular : a small core analysis and a lot of filters. It can be viewed as UNIX pipes. Philosophy of iwla is : add, update, delete ! That's the job of each filters : modify statistics until final result. It's written in Python. iwla (Intelligent Web Log Analyzer) is basically a clone of [awstats](http://www.awstats.org). The main problem with awstats is that it's a very monolothic project with everything in one big PERL file. In opposite, iwla has been though to be very modular : a small core analysis and a lot of filters. It can be viewed as UNIX pipes. Philosophy of iwla is : add, update, delete ! That's the job of each filter : modify statistics until final result. It's written in Python.
Nevertheless, iwla is only focused on HTTP logs. It uses data (robots definitions, search engines definitions) and design from awstats. Moreover, it's not dynamic, but only generates static HTML page (with gzip compression option). Nevertheless, iwla is only focused on HTTP logs. It uses data (robots definitions, search engines definitions) and design from awstats. Moreover, it's not dynamic, but only generates static HTML page (with gzip compression option).
@ -32,25 +32,25 @@ Main values to edit are :
* **display_hooks** : List of display hooks * **display_hooks** : List of display hooks
* **locale** : Displayed locale (_en_ or _fr_) * **locale** : Displayed locale (_en_ or _fr_)
Then, you can then iwla. Output HTML files are created in _output_ directory by default. To quickly see it go in output and type Then, you can launch iwla. Output HTML files are created in _output_ directory by default. To quickly see it, go into _output_ and type
python -m SimpleHTTPServer 8000 python -m SimpleHTTPServer 8000
Open your favorite web browser at _http://localhost:8000_. Enjoy ! Open your favorite web browser at _http://localhost:8000_. Enjoy !
**Warning** : The order is hooks list is important : Some plugins may requires others plugins, and the order of display_hooks is the order of displayed blocks in final result. **Warning** : The order in hooks list is important : Some plugins may requires others plugins, and the order of display_hooks is the order of displayed blocks in final result.
Interesting default configuration values Interesting default configuration values
---------------------------------------- ----------------------------------------
* **DB_ROOT** : Default database directory (default ./output_db) * **DB_ROOT** : Default database directory (default ./output_db)
* **DISPLAY_ROOT** : Default HTML output directory (default ./output) * **DISPLAY_ROOT** : Default HTML output directory (default _./output_)
* **log_format** : Web server log format (nginx style). Default is apache log format * **log_format** : Web server log format (nginx style). Default is apache log format
* **time_format** : Time format used in log format * **time_format** : Time format used in log format
* **pages_extensions** : Extensions that are considered as a HTML page (or result) in opposit to hits * **pages_extensions** : Extensions that are considered as a HTML page (or result) in opposit to hits
* **viewed_http_codes** : HTTP codes that are cosidered OK (200, 304) * **viewed_http_codes** : HTTP codes that are cosidered OK (200, 304)
* **count_hit_only_visitors** : If False, doesn't cout visitors that doesn't GET a page but resources only (images, rss...) * **count_hit_only_visitors** : If False, don't count visitors that doesn't GET a page but resources only (images, rss...)
* **multimedia_files** : Multimedia extensions (not accounted as downloaded files) * **multimedia_files** : Multimedia extensions (not accounted as downloaded files)
* **css_path** : CSS path (you can add yours) * **css_path** : CSS path (you can add yours)
* **compress_output_files** : Files extensions to compress in gzip during display build * **compress_output_files** : Files extensions to compress in gzip during display build
@ -64,7 +64,7 @@ As previously described, plugins acts like UNIX pipes : statistics are constantl
* **Post analysis plugins** : Called after basic statistics computation. They are in charge to enlight them with their own algorithms * **Post analysis plugins** : Called after basic statistics computation. They are in charge to enlight them with their own algorithms
* **Display plugins** : They are in charge to produce HTML files from statistics. * **Display plugins** : They are in charge to produce HTML files from statistics.
To use plugins, just insert their name in _pre_analysis_hooks_, _post_analysis_hooks_ and _display_hooks_ lists in conf.py. To use plugins, just insert their file name (without _.py_ extension) in _pre_analysis_hooks_, _post_analysis_hooks_ and _display_hooks_ lists in conf.py.
Statistics are stored in dictionaries : Statistics are stored in dictionaries :
@ -77,7 +77,7 @@ Statistics are stored in dictionaries :
Create a Plugins Create a Plugins
---------------- ----------------
To create a new plugin, it's necessary to create a derived class of IPlugin (_iplugin.py) in the right directory (_plugins/xxx/yourPlugin.py_). To create a new plugin, it's necessary to subclass IPlugin (_iplugin.py) in the right directory (_plugins/xxx/yourPlugin.py_).
Plugins can defines required configuration values (self.conf_requires) that must be set in conf.py (or can be optional). They can also defines required plugins (self.requires). Plugins can defines required configuration values (self.conf_requires) that must be set in conf.py (or can be optional). They can also defines required plugins (self.requires).
@ -175,34 +175,6 @@ iwla
None None
plugins.display.top_downloads
-----------------------------
Display hook
Create TOP downloads page
Plugin requirements :
post_analysis/top_downloads
Conf values needed :
max_downloads_displayed*
create_all_downloads_page*
Output files :
OUTPUT_ROOT/year/month/top_downloads.html
OUTPUT_ROOT/year/month/index.html
Statistics creation :
None
Statistics update :
None
Statistics deletion :
None
plugins.display.all_visits plugins.display.all_visits
-------------------------- --------------------------
@ -230,34 +202,6 @@ plugins.display.all_visits
None None
plugins.display.top_hits
------------------------
Display hook
Create TOP hits page
Plugin requirements :
post_analysis/top_hits
Conf values needed :
max_hits_displayed*
create_all_hits_page*
Output files :
OUTPUT_ROOT/year/month/top_hits.html
OUTPUT_ROOT/year/month/index.html
Statistics creation :
None
Statistics update :
None
Statistics deletion :
None
plugins.display.referers plugins.display.referers
------------------------ ------------------------
@ -343,151 +287,57 @@ plugins.display.top_pages
None None
plugins.post_analysis.top_downloads plugins.display.top_hits
----------------------------------- ------------------------
Post analysis hook Display hook
Count TOP downloads Create TOP hits page
Plugin requirements : Plugin requirements :
None post_analysis/top_hits
Conf values needed : Conf values needed :
None max_hits_displayed*
create_all_hits_page*
Output files : Output files :
None OUTPUT_ROOT/year/month/top_hits.html
OUTPUT_ROOT/year/month/index.html
Statistics creation : Statistics creation :
None None
Statistics update : Statistics update :
month_stats: None
top_downloads =>
uri
Statistics deletion : Statistics deletion :
None None
plugins.post_analysis.top_hits plugins.display.top_downloads
------------------------------ -----------------------------
Post analysis hook Display hook
Count TOP hits Create TOP downloads page
Plugin requirements : Plugin requirements :
None post_analysis/top_downloads
Conf values needed : Conf values needed :
None max_downloads_displayed*
create_all_downloads_page*
Output files : Output files :
None OUTPUT_ROOT/year/month/top_downloads.html
OUTPUT_ROOT/year/month/index.html
Statistics creation : Statistics creation :
None None
Statistics update : Statistics update :
month_stats:
top_hits =>
uri
Statistics deletion :
None None
plugins.post_analysis.referers
------------------------------
Post analysis hook
Extract referers and key phrases from requests
Plugin requirements :
None
Conf values needed :
domain_name
Output files :
None
Statistics creation :
None
Statistics update :
month_stats :
referers =>
pages
hits
robots_referers =>
pages
hits
search_engine_referers =>
pages
hits
key_phrases =>
phrase
Statistics deletion :
None
plugins.post_analysis.reverse_dns
---------------------------------
Post analysis hook
Replace IP by reverse DNS names
Plugin requirements :
None
Conf values needed :
reverse_dns_timeout*
Output files :
None
Statistics creation :
None
Statistics update :
valid_visitors:
remote_addr
dns_name_replaced
dns_analyzed
Statistics deletion :
None
plugins.post_analysis.top_pages
-------------------------------
Post analysis hook
Count TOP pages
Plugin requirements :
None
Conf values needed :
None
Output files :
None
Statistics creation :
None
Statistics update :
month_stats:
top_pages =>
uri
Statistics deletion : Statistics deletion :
None None
@ -550,3 +400,153 @@ plugins.pre_analysis.robots
None None
plugins.post_analysis.referers
------------------------------
Post analysis hook
Extract referers and key phrases from requests
Plugin requirements :
None
Conf values needed :
domain_name
Output files :
None
Statistics creation :
None
Statistics update :
month_stats :
referers =>
pages
hits
robots_referers =>
pages
hits
search_engine_referers =>
pages
hits
key_phrases =>
phrase
Statistics deletion :
None
plugins.post_analysis.top_pages
-------------------------------
Post analysis hook
Count TOP pages
Plugin requirements :
None
Conf values needed :
None
Output files :
None
Statistics creation :
None
Statistics update :
month_stats:
top_pages =>
uri
Statistics deletion :
None
plugins.post_analysis.reverse_dns
---------------------------------
Post analysis hook
Replace IP by reverse DNS names
Plugin requirements :
None
Conf values needed :
reverse_dns_timeout*
Output files :
None
Statistics creation :
None
Statistics update :
valid_visitors:
remote_addr
dns_name_replaced
dns_analyzed
Statistics deletion :
None
plugins.post_analysis.top_hits
------------------------------
Post analysis hook
Count TOP hits
Plugin requirements :
None
Conf values needed :
None
Output files :
None
Statistics creation :
None
Statistics update :
month_stats:
top_hits =>
uri
Statistics deletion :
None
plugins.post_analysis.top_downloads
-----------------------------------
Post analysis hook
Count TOP downloads
Plugin requirements :
None
Conf values needed :
None
Output files :
None
Statistics creation :
None
Statistics update :
month_stats:
top_downloads =>
uri
Statistics deletion :
None

View File

@ -4,7 +4,7 @@ iwla
Introduction Introduction
------------ ------------
iwla (Intelligent Web Log Analyzer) is basically a clone of [awstats](http://www.awstats.org). The main problem with awstats is that it's a very monolothic project with everything in one big PERL file. In opposite, iwla has be though to be very modular : a small core analysis and a lot of filters. It can be viewed as UNIX pipes. Philosophy of iwla is : add, update, delete ! That's the job of each filters : modify statistics until final result. It's written in Python. iwla (Intelligent Web Log Analyzer) is basically a clone of [awstats](http://www.awstats.org). The main problem with awstats is that it's a very monolothic project with everything in one big PERL file. In opposite, iwla has been though to be very modular : a small core analysis and a lot of filters. It can be viewed as UNIX pipes. Philosophy of iwla is : add, update, delete ! That's the job of each filter : modify statistics until final result. It's written in Python.
Nevertheless, iwla is only focused on HTTP logs. It uses data (robots definitions, search engines definitions) and design from awstats. Moreover, it's not dynamic, but only generates static HTML page (with gzip compression option). Nevertheless, iwla is only focused on HTTP logs. It uses data (robots definitions, search engines definitions) and design from awstats. Moreover, it's not dynamic, but only generates static HTML page (with gzip compression option).
@ -32,25 +32,25 @@ Main values to edit are :
* **display_hooks** : List of display hooks * **display_hooks** : List of display hooks
* **locale** : Displayed locale (_en_ or _fr_) * **locale** : Displayed locale (_en_ or _fr_)
Then, you can then iwla. Output HTML files are created in _output_ directory by default. To quickly see it go in output and type Then, you can launch iwla. Output HTML files are created in _output_ directory by default. To quickly see it, go into _output_ and type
python -m SimpleHTTPServer 8000 python -m SimpleHTTPServer 8000
Open your favorite web browser at _http://localhost:8000_. Enjoy ! Open your favorite web browser at _http://localhost:8000_. Enjoy !
**Warning** : The order is hooks list is important : Some plugins may requires others plugins, and the order of display_hooks is the order of displayed blocks in final result. **Warning** : The order in hooks list is important : Some plugins may requires others plugins, and the order of display_hooks is the order of displayed blocks in final result.
Interesting default configuration values Interesting default configuration values
---------------------------------------- ----------------------------------------
* **DB_ROOT** : Default database directory (default ./output_db) * **DB_ROOT** : Default database directory (default ./output_db)
* **DISPLAY_ROOT** : Default HTML output directory (default ./output) * **DISPLAY_ROOT** : Default HTML output directory (default _./output_)
* **log_format** : Web server log format (nginx style). Default is apache log format * **log_format** : Web server log format (nginx style). Default is apache log format
* **time_format** : Time format used in log format * **time_format** : Time format used in log format
* **pages_extensions** : Extensions that are considered as a HTML page (or result) in opposit to hits * **pages_extensions** : Extensions that are considered as a HTML page (or result) in opposit to hits
* **viewed_http_codes** : HTTP codes that are cosidered OK (200, 304) * **viewed_http_codes** : HTTP codes that are cosidered OK (200, 304)
* **count_hit_only_visitors** : If False, doesn't cout visitors that doesn't GET a page but resources only (images, rss...) * **count_hit_only_visitors** : If False, don't count visitors that doesn't GET a page but resources only (images, rss...)
* **multimedia_files** : Multimedia extensions (not accounted as downloaded files) * **multimedia_files** : Multimedia extensions (not accounted as downloaded files)
* **css_path** : CSS path (you can add yours) * **css_path** : CSS path (you can add yours)
* **compress_output_files** : Files extensions to compress in gzip during display build * **compress_output_files** : Files extensions to compress in gzip during display build
@ -64,7 +64,7 @@ As previously described, plugins acts like UNIX pipes : statistics are constantl
* **Post analysis plugins** : Called after basic statistics computation. They are in charge to enlight them with their own algorithms * **Post analysis plugins** : Called after basic statistics computation. They are in charge to enlight them with their own algorithms
* **Display plugins** : They are in charge to produce HTML files from statistics. * **Display plugins** : They are in charge to produce HTML files from statistics.
To use plugins, just insert their name in _pre_analysis_hooks_, _post_analysis_hooks_ and _display_hooks_ lists in conf.py. To use plugins, just insert their file name (without _.py_ extension) in _pre_analysis_hooks_, _post_analysis_hooks_ and _display_hooks_ lists in conf.py.
Statistics are stored in dictionaries : Statistics are stored in dictionaries :
@ -77,7 +77,7 @@ Statistics are stored in dictionaries :
Create a Plugins Create a Plugins
---------------- ----------------
To create a new plugin, it's necessary to create a derived class of IPlugin (_iplugin.py) in the right directory (_plugins/xxx/yourPlugin.py_). To create a new plugin, it's necessary to subclass IPlugin (_iplugin.py) in the right directory (_plugins/xxx/yourPlugin.py_).
Plugins can defines required configuration values (self.conf_requires) that must be set in conf.py (or can be optional). They can also defines required plugins (self.requires). Plugins can defines required configuration values (self.conf_requires) that must be set in conf.py (or can be optional). They can also defines required plugins (self.requires).

View File

@ -83,34 +83,6 @@ iwla
None None
plugins.display.top_downloads
-----------------------------
Display hook
Create TOP downloads page
Plugin requirements :
post_analysis/top_downloads
Conf values needed :
max_downloads_displayed*
create_all_downloads_page*
Output files :
OUTPUT_ROOT/year/month/top_downloads.html
OUTPUT_ROOT/year/month/index.html
Statistics creation :
None
Statistics update :
None
Statistics deletion :
None
plugins.display.all_visits plugins.display.all_visits
-------------------------- --------------------------
@ -138,34 +110,6 @@ plugins.display.all_visits
None None
plugins.display.top_hits
------------------------
Display hook
Create TOP hits page
Plugin requirements :
post_analysis/top_hits
Conf values needed :
max_hits_displayed*
create_all_hits_page*
Output files :
OUTPUT_ROOT/year/month/top_hits.html
OUTPUT_ROOT/year/month/index.html
Statistics creation :
None
Statistics update :
None
Statistics deletion :
None
plugins.display.referers plugins.display.referers
------------------------ ------------------------
@ -251,151 +195,57 @@ plugins.display.top_pages
None None
plugins.post_analysis.top_downloads plugins.display.top_hits
----------------------------------- ------------------------
Post analysis hook Display hook
Count TOP downloads Create TOP hits page
Plugin requirements : Plugin requirements :
None post_analysis/top_hits
Conf values needed : Conf values needed :
None max_hits_displayed*
create_all_hits_page*
Output files : Output files :
None OUTPUT_ROOT/year/month/top_hits.html
OUTPUT_ROOT/year/month/index.html
Statistics creation : Statistics creation :
None None
Statistics update : Statistics update :
month_stats: None
top_downloads =>
uri
Statistics deletion : Statistics deletion :
None None
plugins.post_analysis.top_hits plugins.display.top_downloads
------------------------------ -----------------------------
Post analysis hook Display hook
Count TOP hits Create TOP downloads page
Plugin requirements : Plugin requirements :
None post_analysis/top_downloads
Conf values needed : Conf values needed :
None max_downloads_displayed*
create_all_downloads_page*
Output files : Output files :
None OUTPUT_ROOT/year/month/top_downloads.html
OUTPUT_ROOT/year/month/index.html
Statistics creation : Statistics creation :
None None
Statistics update : Statistics update :
month_stats:
top_hits =>
uri
Statistics deletion :
None None
plugins.post_analysis.referers
------------------------------
Post analysis hook
Extract referers and key phrases from requests
Plugin requirements :
None
Conf values needed :
domain_name
Output files :
None
Statistics creation :
None
Statistics update :
month_stats :
referers =>
pages
hits
robots_referers =>
pages
hits
search_engine_referers =>
pages
hits
key_phrases =>
phrase
Statistics deletion :
None
plugins.post_analysis.reverse_dns
---------------------------------
Post analysis hook
Replace IP by reverse DNS names
Plugin requirements :
None
Conf values needed :
reverse_dns_timeout*
Output files :
None
Statistics creation :
None
Statistics update :
valid_visitors:
remote_addr
dns_name_replaced
dns_analyzed
Statistics deletion :
None
plugins.post_analysis.top_pages
-------------------------------
Post analysis hook
Count TOP pages
Plugin requirements :
None
Conf values needed :
None
Output files :
None
Statistics creation :
None
Statistics update :
month_stats:
top_pages =>
uri
Statistics deletion : Statistics deletion :
None None
@ -458,3 +308,153 @@ plugins.pre_analysis.robots
None None
plugins.post_analysis.referers
------------------------------
Post analysis hook
Extract referers and key phrases from requests
Plugin requirements :
None
Conf values needed :
domain_name
Output files :
None
Statistics creation :
None
Statistics update :
month_stats :
referers =>
pages
hits
robots_referers =>
pages
hits
search_engine_referers =>
pages
hits
key_phrases =>
phrase
Statistics deletion :
None
plugins.post_analysis.top_pages
-------------------------------
Post analysis hook
Count TOP pages
Plugin requirements :
None
Conf values needed :
None
Output files :
None
Statistics creation :
None
Statistics update :
month_stats:
top_pages =>
uri
Statistics deletion :
None
plugins.post_analysis.reverse_dns
---------------------------------
Post analysis hook
Replace IP by reverse DNS names
Plugin requirements :
None
Conf values needed :
reverse_dns_timeout*
Output files :
None
Statistics creation :
None
Statistics update :
valid_visitors:
remote_addr
dns_name_replaced
dns_analyzed
Statistics deletion :
None
plugins.post_analysis.top_hits
------------------------------
Post analysis hook
Count TOP hits
Plugin requirements :
None
Conf values needed :
None
Output files :
None
Statistics creation :
None
Statistics update :
month_stats:
top_hits =>
uri
Statistics deletion :
None
plugins.post_analysis.top_downloads
-----------------------------------
Post analysis hook
Count TOP downloads
Plugin requirements :
None
Conf values needed :
None
Output files :
None
Statistics creation :
None
Statistics update :
month_stats:
top_downloads =>
uri
Statistics deletion :
None