soutade
/
iwla
1
0
Fork 0
Code Issues Pull Requests Releases 2 Activity

Update documentation

This commit is contained in:
Grégory Soutadé 2024-03-16 09:02:06 +01:00
parent 1e09852d18
commit ca3c0eefdf
3 changed files with 88 additions and 28 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

58
docs/index.md
View File

@ -6,7 +6,7 @@ Introduction
iwla (Intelligent Web Log Analyzer) is basically a clone of [awstats](http://www.awstats.org). The main problem with awstats is that it's a very monolithic project with everything in one big PERL file. In opposite, iwla has been though to be very modular : a small core analysis and a lot of filters. It can be viewed as UNIX pipes. Philosophy of iwla is : add, update, delete ! That's the job of each filter : modify statistics until final result. It's written in Python. iwla (Intelligent Web Log Analyzer) is basically a clone of [awstats](http://www.awstats.org). The main problem with awstats is that it's a very monolithic project with everything in one big PERL file. In opposite, iwla has been though to be very modular : a small core analysis and a lot of filters. It can be viewed as UNIX pipes. Philosophy of iwla is : add, update, delete ! That's the job of each filter : modify statistics until final result. It's written in Python.
Nevertheless, iwla is only focused on HTTP logs. It uses data (robots definitions, search engines definitions) and design from awstats. Moreover, it's not dynamic, but only generates static HTML page (with gzip compression option). Nevertheless, iwla is only focused on HTTP logs. It uses data (search engines definitions) and design from awstats. Moreover, it's not dynamic, but only generates static HTML page (with gzip compression option).
Demo Demo
---- ----
@ -16,8 +16,7 @@ A demonstration instance is available [here](https://iwla-demo.soutade.fr)
Usage Usage
----- -----
./iwla [-c|--config-file file] [-C|--clean-output] [-i|--stdin] [-f FILE|--file FILE] [-d LOGLEVEL|--log-level LOGLEVEL] [-r|--reset year/month] [-z|--dont-compress] [-p] [-D|--dry-run] ./iwla [-c|--config-file file] [-C|--clean-output] [-i|--stdin] [-f FILE|--file FILE] [-d LOGLEVEL|--log-level LOGLEVEL] [-r|--reset year/month] [-z|--dont-compress] [-p] [-P|--disable-display] [-D|--dry-run]
-c : Configuration file to use (default conf.py) -c : Configuration file to use (default conf.py)
-C : Clean output (database and HTML) before starting -C : Clean output (database and HTML) before starting
-i : Read data from stdin instead of conf.analyzed_filename -i : Read data from stdin instead of conf.analyzed_filename
@ -26,6 +25,7 @@ Usage
-r : Reset analysis to a specific date (month/year) -r : Reset analysis to a specific date (month/year)
-z : Don't compress databases (bigger but faster, not compatible with compressed databases) -z : Don't compress databases (bigger but faster, not compatible with compressed databases)
-p : Only generate display -p : Only generate display
-P : Don't generate display
-d : Dry run (don't write/update files to disk) -d : Dry run (don't write/update files to disk)
Basic usage Basic usage
@ -48,6 +48,7 @@ You can also append an element to an existing default configuration list by usin
multimedia_files_append = ['xml'] multimedia_files_append = ['xml']
or or
multimedia_files_append = 'xml' multimedia_files_append = 'xml'
Will append 'xml' to current multimedia_files list Will append 'xml' to current multimedia_files list
Then, you can launch iwla. Output HTML files are created in _output_ directory by default. To quickly see it, go into _output_ and type Then, you can launch iwla. Output HTML files are created in _output_ directory by default. To quickly see it, go into _output_ and type
@ -87,7 +88,7 @@ To use plugins, just insert their file name (without _.py_ extension) in _pre_an
Statistics are stored in dictionaries : Statistics are stored in dictionaries :
* **month_stats** : Statistics of current analysed month * **month_stats** : Statistics of current analysed month
* **valid_visitor** : A subset of month_stats without robots * **valid_visitors** : A subset of month_stats without robots
* **days_stats** : Statistics of current analysed day * **days_stats** : Statistics of current analysed day
* **visits** : All visitors with all of its requests (only if 'keep_requests' is true or filtered) * **visits** : All visitors with all of its requests (only if 'keep_requests' is true or filtered)
* **meta** : Final result of month statistics (by year) * **meta** : Final result of month statistics (by year)
@ -103,6 +104,7 @@ The two functions to overload are _load(self)_ that must returns True or False i
For display plugins, a lot of code has been wrote in _display.py_ that simplify the creation on HTML blocks, tables and bar graphs. For display plugins, a lot of code has been wrote in _display.py_ that simplify the creation on HTML blocks, tables and bar graphs.
Plugins Plugins
======= =======
@ -129,6 +131,7 @@ Optional configuration values ends with *.
* plugins/display/top_pages_diff.py * plugins/display/top_pages_diff.py
* plugins/display/top_pages.py * plugins/display/top_pages.py
* plugins/display/top_visitors.py * plugins/display/top_visitors.py
* plugins/display/visitor_ip.py
* plugins/post_analysis/anonymize_ip.py * plugins/post_analysis/anonymize_ip.py
* plugins/post_analysis/browsers.py * plugins/post_analysis/browsers.py
* plugins/post_analysis/feeds.py * plugins/post_analysis/feeds.py
@ -163,6 +166,7 @@ iwla
locales_path locales_path
compress_output_files compress_output_files
excluded_ip excluded_ip
excluded_domain_name
Output files : Output files :
DB_ROOT/meta.db DB_ROOT/meta.db
@ -203,7 +207,7 @@ iwla
nb_visitors nb_visitors
visits : visits :
remote_addr => remote_ip =>
remote_addr remote_addr
remote_ip remote_ip
viewed_pages{0..31} # 0 contains total viewed_pages{0..31} # 0 contains total
@ -573,7 +577,6 @@ plugins.display.robot_bandwidth
None None
Conf values needed : Conf values needed :
display_visitor_ip*
create_all_robot_bandwidth_page* create_all_robot_bandwidth_page*
Output files : Output files :
@ -763,7 +766,33 @@ plugins.display.top_visitors
None None
Conf values needed : Conf values needed :
display_visitor_ip* None
Output files :
OUTPUT_ROOT/year/month/index.html
Statistics creation :
None
Statistics update :
None
Statistics deletion :
None
plugins.display.visitor_ip
--------------------------
Display hook
Display IP below visitor name
Plugin requirements :
None
Conf values needed :
compact_ip*
Output files : Output files :
OUTPUT_ROOT/year/month/index.html OUTPUT_ROOT/year/month/index.html
@ -823,7 +852,7 @@ plugins.post_analysis.browsers
Statistics creation : Statistics creation :
visits : visits :
remote_addr => remote_ip =>
browser browser
month_stats : month_stats :
@ -861,7 +890,7 @@ plugins.post_analysis.feeds
None None
Statistics creation : Statistics creation :
remote_addr => remote_ip =>
feed_parser feed_parser
feed_name_analysed feed_name_analysed
feed_parser_last_access (for merged parser) feed_parser_last_access (for merged parser)
@ -916,13 +945,13 @@ plugins.post_analysis.filter_users
Statistics creation : Statistics creation :
visits : visits :
remote_addr => remote_ip =>
filtered filtered
geo_location geo_location
Statistics update : Statistics update :
visits : visits :
remote_addr => remote_ip =>
keep_requests keep_requests
Statistics deletion : Statistics deletion :
@ -1014,7 +1043,7 @@ plugins.post_analysis.ip_type
Statistics creation : Statistics creation :
visits : visits :
remote_addr => remote_ip =>
ip_type ip_type
month_stats : month_stats :
@ -1045,7 +1074,7 @@ plugins.post_analysis.operating_systems
Statistics creation : Statistics creation :
visits : visits :
remote_addr => remote_ip =>
operating_system operating_system
month_stats : month_stats :
@ -1279,7 +1308,8 @@ plugins.pre_analysis.robots
None None
Conf values needed : Conf values needed :
None count_hit_only_visitors
no_referrer_domains
Output files : Output files :
None None

10
docs/main.md
View File

@ -6,7 +6,7 @@ Introduction
iwla (Intelligent Web Log Analyzer) is basically a clone of [awstats](http://www.awstats.org). The main problem with awstats is that it's a very monolithic project with everything in one big PERL file. In opposite, iwla has been though to be very modular : a small core analysis and a lot of filters. It can be viewed as UNIX pipes. Philosophy of iwla is : add, update, delete ! That's the job of each filter : modify statistics until final result. It's written in Python. iwla (Intelligent Web Log Analyzer) is basically a clone of [awstats](http://www.awstats.org). The main problem with awstats is that it's a very monolithic project with everything in one big PERL file. In opposite, iwla has been though to be very modular : a small core analysis and a lot of filters. It can be viewed as UNIX pipes. Philosophy of iwla is : add, update, delete ! That's the job of each filter : modify statistics until final result. It's written in Python.
Nevertheless, iwla is only focused on HTTP logs. It uses data (robots definitions, search engines definitions) and design from awstats. Moreover, it's not dynamic, but only generates static HTML page (with gzip compression option). Nevertheless, iwla is only focused on HTTP logs. It uses data (search engines definitions) and design from awstats. Moreover, it's not dynamic, but only generates static HTML page (with gzip compression option).
Demo Demo
---- ----
@ -16,8 +16,7 @@ A demonstration instance is available [here](https://iwla-demo.soutade.fr)
Usage Usage
----- -----
./iwla [-c|--config-file file] [-C|--clean-output] [-i|--stdin] [-f FILE|--file FILE] [-d LOGLEVEL|--log-level LOGLEVEL] [-r|--reset year/month] [-z|--dont-compress] [-p] [-D|--dry-run] ./iwla [-c|--config-file file] [-C|--clean-output] [-i|--stdin] [-f FILE|--file FILE] [-d LOGLEVEL|--log-level LOGLEVEL] [-r|--reset year/month] [-z|--dont-compress] [-p] [-P|--disable-display] [-D|--dry-run]
-c : Configuration file to use (default conf.py) -c : Configuration file to use (default conf.py)
-C : Clean output (database and HTML) before starting -C : Clean output (database and HTML) before starting
-i : Read data from stdin instead of conf.analyzed_filename -i : Read data from stdin instead of conf.analyzed_filename
@ -26,6 +25,7 @@ Usage
-r : Reset analysis to a specific date (month/year) -r : Reset analysis to a specific date (month/year)
-z : Don't compress databases (bigger but faster, not compatible with compressed databases) -z : Don't compress databases (bigger but faster, not compatible with compressed databases)
-p : Only generate display -p : Only generate display
-P : Don't generate display
-d : Dry run (don't write/update files to disk) -d : Dry run (don't write/update files to disk)
Basic usage Basic usage
@ -48,6 +48,7 @@ You can also append an element to an existing default configuration list by usin
multimedia_files_append = ['xml'] multimedia_files_append = ['xml']
or or
multimedia_files_append = 'xml' multimedia_files_append = 'xml'
Will append 'xml' to current multimedia_files list Will append 'xml' to current multimedia_files list
Then, you can launch iwla. Output HTML files are created in _output_ directory by default. To quickly see it, go into _output_ and type Then, you can launch iwla. Output HTML files are created in _output_ directory by default. To quickly see it, go into _output_ and type
@ -87,7 +88,7 @@ To use plugins, just insert their file name (without _.py_ extension) in _pre_an
Statistics are stored in dictionaries : Statistics are stored in dictionaries :
* **month_stats** : Statistics of current analysed month * **month_stats** : Statistics of current analysed month
* **valid_visitor** : A subset of month_stats without robots * **valid_visitors** : A subset of month_stats without robots
* **days_stats** : Statistics of current analysed day * **days_stats** : Statistics of current analysed day
* **visits** : All visitors with all of its requests (only if 'keep_requests' is true or filtered) * **visits** : All visitors with all of its requests (only if 'keep_requests' is true or filtered)
* **meta** : Final result of month statistics (by year) * **meta** : Final result of month statistics (by year)
@ -103,6 +104,7 @@ The two functions to overload are _load(self)_ that must returns True or False i
For display plugins, a lot of code has been wrote in _display.py_ that simplify the creation on HTML blocks, tables and bar graphs. For display plugins, a lot of code has been wrote in _display.py_ that simplify the creation on HTML blocks, tables and bar graphs.
Plugins Plugins
======= =======

48
docs/modules.md
View File

@ -19,6 +19,7 @@
* plugins/display/top_pages_diff.py * plugins/display/top_pages_diff.py
* plugins/display/top_pages.py * plugins/display/top_pages.py
* plugins/display/top_visitors.py * plugins/display/top_visitors.py
* plugins/display/visitor_ip.py
* plugins/post_analysis/anonymize_ip.py * plugins/post_analysis/anonymize_ip.py
* plugins/post_analysis/browsers.py * plugins/post_analysis/browsers.py
* plugins/post_analysis/feeds.py * plugins/post_analysis/feeds.py
@ -53,6 +54,7 @@ iwla
locales_path locales_path
compress_output_files compress_output_files
excluded_ip excluded_ip
excluded_domain_name
Output files : Output files :
DB_ROOT/meta.db DB_ROOT/meta.db
@ -93,7 +95,7 @@ iwla
nb_visitors nb_visitors
visits : visits :
remote_addr => remote_ip =>
remote_addr remote_addr
remote_ip remote_ip
viewed_pages{0..31} # 0 contains total viewed_pages{0..31} # 0 contains total
@ -463,7 +465,6 @@ plugins.display.robot_bandwidth
None None
Conf values needed : Conf values needed :
display_visitor_ip*
create_all_robot_bandwidth_page* create_all_robot_bandwidth_page*
Output files : Output files :
@ -653,7 +654,33 @@ plugins.display.top_visitors
None None
Conf values needed : Conf values needed :
display_visitor_ip* None
Output files :
OUTPUT_ROOT/year/month/index.html
Statistics creation :
None
Statistics update :
None
Statistics deletion :
None
plugins.display.visitor_ip
--------------------------
Display hook
Display IP below visitor name
Plugin requirements :
None
Conf values needed :
compact_ip*
Output files : Output files :
OUTPUT_ROOT/year/month/index.html OUTPUT_ROOT/year/month/index.html
@ -713,7 +740,7 @@ plugins.post_analysis.browsers
Statistics creation : Statistics creation :
visits : visits :
remote_addr => remote_ip =>
browser browser
month_stats : month_stats :
@ -751,7 +778,7 @@ plugins.post_analysis.feeds
None None
Statistics creation : Statistics creation :
remote_addr => remote_ip =>
feed_parser feed_parser
feed_name_analysed feed_name_analysed
feed_parser_last_access (for merged parser) feed_parser_last_access (for merged parser)
@ -806,13 +833,13 @@ plugins.post_analysis.filter_users
Statistics creation : Statistics creation :
visits : visits :
remote_addr => remote_ip =>
filtered filtered
geo_location geo_location
Statistics update : Statistics update :
visits : visits :
remote_addr => remote_ip =>
keep_requests keep_requests
Statistics deletion : Statistics deletion :
@ -904,7 +931,7 @@ plugins.post_analysis.ip_type
Statistics creation : Statistics creation :
visits : visits :
remote_addr => remote_ip =>
ip_type ip_type
month_stats : month_stats :
@ -935,7 +962,7 @@ plugins.post_analysis.operating_systems
Statistics creation : Statistics creation :
visits : visits :
remote_addr => remote_ip =>
operating_system operating_system
month_stats : month_stats :
@ -1169,7 +1196,8 @@ plugins.pre_analysis.robots
None None
Conf values needed : Conf values needed :
None count_hit_only_visitors
no_referrer_domains
Output files : Output files :
None None