GoAccess-一个处理web日志的工具

GoAccess介绍

GoAccess是一个开源的基于终端的快速日志分析器。
Github地址:https://github.com/allinurl/goaccess
官网地址: https://goaccess.io/
说明文档: https://goaccess.io/man

通过自己配置，可以支持日志格式，默认已经集成的log格式

  • Amazon CloudFront (Download Distribution).
  • Amazon Simple Storage Service (S3)
  • AWS Elastic Load Balancing
  • Combined Log Format (XLF/ELF) Apache | Nginx
  • Common Log Format (CLF) Apache
  • Google Cloud Storage.
  • Apache virtual hosts
  • Squid Native Format.
  • W3C format (IIS)

安装

MacOS下

使用brew来安装

1
brew install goaccess

CentOS下

  1. 使用yum安装，goaccess在epel repository下

    1
    2
    yum -y install epel-release
    yum install goaccess

  2. 下载源码编译

    1
    2
    3
    4
    5
    6
    7
    # yum install ncurses-devel geoip-devel  # 安装依赖包
    $ wget https://tar.goaccess.io/goaccess-1.3.tar.gz
    $ tar -xzvf goaccess-1.3.tar.gz
    $ cd goaccess-1.3/
    $ ./configure --enable-utf8 --enable-geoip=legacy
    $ make
    # make install

Amazon Linux

下载源码编译

1
2
3
4
5
6
7
# yum install ncurses-devel geoip-devel  # 安装依赖包
$ wget https://tar.goaccess.io/goaccess-1.3.tar.gz
$ tar -xzvf goaccess-1.3.tar.gz
$ cd goaccess-1.3/
$ ./configure --enable-utf8 --enable-geoip=legacy
$ make
# make install

参数

摘录了部分常用的参数，完整参数参见GoAccess Man

PARSE OPTIONS

  • -a –agent-list
    Enable a list of user-agents by host. For faster parsing, do not enable this flag.
  • -e –exclude-ip
    Exclude an IPv4 or IPv6 from being counted. Ranges can be included as well using a dash in between the IPs (start-end)
    • exclude-ip 127.0.0.1
    • exclude-ip 192.168.0.1-192.168.0.100
    • exclude-ip ::1
    • exclude-ip 0:0:0:0:0:ffff:808:804-0:0:0:0:0:ffff:808:808
  • -o –output=
    Write output to stdout given one of the following files and the corresponding extension for the output format:
    • /path/file.csv - Comma-separated values (CSV)
    • /path/file.json - JSON (JavaScript Object Notation)
    • /path/file.html - HTML
  • -q –no-query-string
    Ignore request’s query string. i.e., www.google.com/page.htm?query => www.google.com/page.htm
  • –444-as-404
    Treat non-standard status code 444 as 404.
  • –4xx-to-unique-count
    Add 4xx client errors to the unique visitors count.
  • –browsers-file=
    Include an additional tab delimited list of browsers/crawlers/feeds etc. See config/browsers.list.
  • –ignore-crawlers
    Ignore crawlers.

GEOLOCATION OPTIONS

  • -g –std-geoip
    Standard GeoIP database for less memory usage.
  • –geoip-database
    Specify path to GeoIP database file. i.e., GeoLiteCity.dat. File needs to be downloaded from maxmind.com. IPv4 and IPv6 files are supported as well. Note: --geoip-city-data is an alias of --geoip-database.
    Note: If using GeoIP2, you will need to download the City/Country database from MaxMind and use the option --geoip-database to specify the database. Currently cities are only shown in the hosts panel (per host).

自定义格式

参见GoAccess Man Page

例子

解析nginx默认日志

1
goaccess --log-format=COMBINED access.log -o report.html

解析AWS ELB日志

1
goaccess --log-format=AWSELB 20161231T2200Z_52.27.188.188_1d0y2vo4.log > report.html

多文件

1
goaccess access.log access.log.1

实时生成

1
tail -f access.log | goaccess --log-format=COMBINED -

报告页显示为中文

1
2
3
LANG=zh_CN.UTF-8
LC_CTYPE=zh_CN.UTF-8
goaccess --log-format=COMBINED access.log -o report.html

Reference

留言

▲