Import apache access log into elasticsearch

Debugging web application its very tedious job even when if website is full of processes and action. Checking apache logs and finding out request details is time consuming. For this problem i am recommending one solution.

We can use …

• Elasticsearch – Store Log
• Kibana – Represent log in graphical way.
• Bash script – To index data into elasticsearch.
• PHP script – Map log data into json format.
• Apache Log Parser

First Create index with mapping …

curl -XPOST localhost:9200/apache -d '{
"mappings" : {
"access" : {
"properties" : {
"host": { "index": "analyzed", "store": "yes", "type": "ip" },
"logname": { "index": "analyzed", "store": "yes", "type": "string" },
"user": { "index": "analyzed", "store": "yes", "type": "string" },
"time": { "index": "analyzed", "store": "yes", "type": "date" ,"format" : "yyyy:MM:dd HH:mm:ss"},
"method": { "index": "not_analyzed", "store": "yes", "type": "string" },
"url": { "index": "not_analyzed", "store": "yes", "type": "string" },
"protocol": { "index": "not_analyzed", "store": "yes", "type": "string" },
"status": { "index": "analyzed", "store": "yes", "type": "string" },
"sentbytes": { "index": "not_analyzed", "store": "yes", "type": "string"},
"referrer": { "index": "not_analyzed", "store": "yes", "type": "string"},
"useragent": { "index": "analyzed", "store": "yes", "type": "string" }
}}}}'

Shell Script to parse apache and put into elasticsearchwatch full xXx: Return of Xander Cage movie online

#!/bin/bash
ElasticUrl="http://localhost:9200"
Index="apache"
Type="access"

LogFile=/var/log/httpd/access_log

tail -f $LogFile | while read myline; do
JSON=$(php shipper.php "$myline")
echo curl -i \
-H "Accept: application/json" \
-H "Content-Type:application/json" \
-X POST --data "$JSON" "http://localhost:9200/apache/access"
done

PHP script to convert apache log to json format

< ?php
require_once("apache-log-parser/src/Kassner/ApacheLogParser/Factory.php");
require_once("apache-log-parser/src/Kassner/ApacheLogParser/FormatException.php");
require_once("apache-log-parser/src/Kassner/ApacheLogParser/ApacheLogParser.php");
use Kassner\ApacheLogParser\ApacheLogParser;
$mapping=array();
if(isset($argv[1])){
   $parser = new ApacheLogParser("%h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"");
   $logLine = $argv[1];
   $entry = $parser->parse($logLine);
   $method='';
   $url='';
   $protocol='';
   if(isset($entry->request)){
        $arrReq=explode(" ",$entry->request);
        if(count($arrReq)==3){
            $method=$arrReq[0];
            $url=$arrReq[1];
            $protocol=$arrReq[2];
        }
   }
   $entry->stamp=@date("Y:m:d h:i:s",$entry->stamp);
   $mapping=array(
        'host'=>$entry->host,
        'logname'=>$entry->logname,
        'user'=>$entry->user,
        'time'=>$entry->stamp,
        'method'=>$method,
        'url'=>$url,
        'protocol'=>$protocol,
        'status'=>$entry->status,
        'sentbytes'=>$entry->sentBytes,
        'referer'=>$entry->HeaderReferer,
        'useragent'=>$entry->HeaderUserAgent
   );
   echo json_encode($mapping);

Now you have to just keep running of shell script….

$sh <shell script name> & </shell>

Categories: Elasticsearch, Linux, Website Peformance

About Author: