原 使用ELK搭建实时日志分析系统(3)

利用ELK分析Log4j日志

举一个处理log4j格式类型的例子。

日志格式大概如下:

属于比较常规的log4j日志.

主要是需要将Java的堆栈合并成单行。

2016-06-02 15:16:50,330 [ActiveMQ Session Task-2701] ERROR com.lehecai.engine.thread.scanner.plan.PlanNonRealTimeMessageListener - 触发方案(****)的无法开奖状态回收
2016-06-02 15:36:27,952 [贴吧*********处理线程] ERROR c.lehecai.engine.thread.ticket.impl.splitter.CommonTicketVirtualSplitterRunnable - [贴吧*********处理线程]第<20160519>期,方案(id:******)**拆票出错
net.sf.json.JSONException: null object
        at net.sf.json.JSONObject.verifyIsNull(JSONObject.java:2897) ~[json-lib-2.3.jar:na]
        at net.sf.json.JSONObject.getString(JSONObject.java:2240) ~[json-lib-2.3.jar:na]
        at com.lehecai.engine.thread.ticket.impl.splitter.CommonTicketVirtualSplitterRunnable.convertExtContent(CommonTicketVirtualSplitterRunnable.java:229) ~[CommonTicketVirtualSplitterRunnable.class:na]
        at com.lehecai.engine.thread.ticket.impl.splitter.CommonTicketVirtualSplitterRunnable.updateTicketExt(CommonTicketVirtualSplitterRunnable.java:255) ~[CommonTicketVirtualSplitterRunnable.class:na]
        at com.lehecai.engine.thread.ticket.impl.splitter.CommonTicketVirtualSplitterRunnable.execute(CommonTicketVirtualSplitterRunnable.java:131) ~[CommonTicketVirtualSplitterRunnable.class:na]
        at com.lehecai.engine.thread.ticket.AbstractTicketSplitterRunnable.executeRun(AbstractTicketSplitterRunnable.java:42) [AbstractTicketSplitterRunnable.class:na]
        at com.lehecai.core.thread.AbstractThreadRunnable.run(AbstractThreadRunnable.java:74) [pcore-1.0-SNAPSHOT-qa.jar:na]
        at java.lang.Thread.run(Thread.java:745) [na:1.7.0_60]
2016-06-02 15:36:28,264 [****彩期守护线程] ERROR com.lehecai.engine.thread.phase.impl.phase.CommonPhaseHandler - 未查找到([name: ****, value: 560])有效的当前期, 切换到(160516079)的下一期继续查找

配置客户端Log4j格式收集器

input {
    file {
        path => ["/opt/deploy/qa-engine2/apache-tomcat-7.0.55/logs/engine_test.log"]
    codec => multiline {
            pattern => "(^[a-zA-Z.]+(?:Error|Exception): .+)|(^\s+at .+)|(^\s+... \d+ more)|(^\s*Caused by:.+)"
            what => "previous"
        }
    }
}
output{
   #stdout { codec => rubydebug }
   redis {
                host => "172.16.3.145"
                data_type => "list"
                key => "logstash:redis"
   }
}

配置服务端Log4j格式处理器

input {
    redis {
                host => "172.16.3.145"
                data_type => "list"
                key => "logstash:redis"
                type => "apache_log_73"
    }
}
filter {
    if [type] == "apache_log_73" {
        grok {
            match => { "message" => "%{DATESTAMP:timestamp} \[%{GREEDYDATA:log_thread}\] %{LOGLEVEL:log_level} %{JAVACLASS:class} - %{GREEDYDATA:logmsg}" }
            add_field => [ "received_at", "%{@timestamp}" ]
            add_field => [ "received_from", "%{host}" ]
            remove_field => ["message"]
            remove_field => ["path"]
    }
    }
    #date {
    #    match => [ "@timestamp" , "dd-MM-yyyy HH:mm:ss" ]
    #}
}
output{
   #stdout { codec => rubydebug }
   elasticsearch {
        hosts => ["127.0.0.1:9200"]
   }
}

Kibana显示出来的效果

这是直接看到的效果

这是查看指定字段的效果

这是查看格式化后的效果

原文链接：使用ELK搭建实时日志分析系统(3)，转载请注明来源！