Gregory Soutade df78a3f4cb [pre_analysis/robots] Don't checks for /robots.txt request, but endswith /robots.txt for robot detection

2015-04-06 17:52:31 +02:00

..

__init__.py

Add bytesToStr()

2014-11-24 13:44:04 +01:00

page_to_hit.py

Set arguments of page_to_hit facultative

2015-01-13 18:54:57 +01:00

robots.py

[pre_analysis/robots] Don't checks for /robots.txt request, but endswith /robots.txt for robot detection

2015-04-06 17:52:31 +02:00