Ask To Apps
  • Home
  • WordPress
  • Elasticsearch
  • PHP
  • Linux
  • Website Peformance

Tag : apache

Import apache access log into elasticsearch

27/01/2014 Article

Debugging web application its very tedious job even when if website is full of processes and action. Checking apache logs and finding out request details is time consuming. For this problem i am recommending one solution.

We can use …

  • Elasticsearch – Store Log
  • Kibana – Represent log in graphical way.
  • Bash script – To index data into elasticsearch.
  • PHP script – Map log data into json format.
  • Apache Log Parser

First Create index with mapping …

curl -XPOST localhost:9200/apache -d '{
"mappings" : {
"access" : {
"properties" : {
"host": { "index": "analyzed", "store": "yes", "type": "ip" },
"logname": { "index": "analyzed", "store": "yes", "type": "string" },
"user": { "index": "analyzed", "store": "yes", "type": "string" },
"time": { "index": "analyzed", "store": "yes", "type": "date" ,"format" : "yyyy:MM:dd HH:mm:ss"},
"method": { "index": "not_analyzed", "store": "yes", "type": "string" },
"url": { "index": "not_analyzed", "store": "yes", "type": "string" },
"protocol": { "index": "not_analyzed", "store": "yes", "type": "string" },
"status": { "index": "analyzed", "store": "yes", "type": "string" },
"sentbytes": { "index": "not_analyzed", "store": "yes", "type": "string"},
"referrer": { "index": "not_analyzed", "store": "yes", "type": "string"},
"useragent": { "index": "analyzed", "store": "yes", "type": "string" }
}}}}'

 

Shell Script to parse apache and put into elasticsearchwatch full xXx: Return of Xander Cage movie online

#!/bin/bash
ElasticUrl="http://localhost:9200"
Index="apache"
Type="access"

LogFile=/var/log/httpd/access_log

tail -f $LogFile | while read myline; do
JSON=$(php shipper.php "$myline")
echo curl -i \
-H "Accept: application/json" \
-H "Content-Type:application/json" \
-X POST --data "$JSON" "http://localhost:9200/apache/access"
done

PHP script to convert apache log to json format

< ?php
require_once("apache-log-parser/src/Kassner/ApacheLogParser/Factory.php");
require_once("apache-log-parser/src/Kassner/ApacheLogParser/FormatException.php");
require_once("apache-log-parser/src/Kassner/ApacheLogParser/ApacheLogParser.php");
use Kassner\ApacheLogParser\ApacheLogParser;
$mapping=array();
if(isset($argv[1])){
   $parser = new ApacheLogParser("%h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"");
   $logLine = $argv[1];
   $entry = $parser->parse($logLine);
   $method='';
   $url='';
   $protocol='';
   if(isset($entry->request)){
        $arrReq=explode(" ",$entry->request);
        if(count($arrReq)==3){
            $method=$arrReq[0];
            $url=$arrReq[1];
            $protocol=$arrReq[2];
        }
   }
   $entry->stamp=@date("Y:m:d h:i:s",$entry->stamp);
   $mapping=array(
        'host'=>$entry->host,
        'logname'=>$entry->logname,
        'user'=>$entry->user,
        'time'=>$entry->stamp,
        'method'=>$method,
        'url'=>$url,
        'protocol'=>$protocol,
        'status'=>$entry->status,
        'sentbytes'=>$entry->sentBytes,
        'referer'=>$entry->HeaderReferer,
        'useragent'=>$entry->HeaderUserAgent
   );
   echo json_encode($mapping);

Now you have to just keep running of shell script….

$sh <shell script name> & </shell>

Categories: Elasticsearch, Linux, Website Peformance

Tags: access log, apache, Elasticsearch, Import

About Author:

Recent Posts

  • Install and configure logstash-forwarder
  • Redirect request on php script through squid proxy
  • Alerting for Elasticsearch : Log watcher in elasticsearch using ES Watcher
  • Detect face from image using python script with OpenCV
  • Change mysql root password on centos
  • Search part of word in elasticsearch using nGram – auto-complete search
  • Connect VPN on centos linux using command line
  • Custom river plugin in elasticsearch
  • Backup elasticsearch with snapshot and restore api
  • PHP code to exact keywords from text.

Tags

apache Apache Lucene cache Distributed Elastic Index Elasticsearch elasticsearch performance Git Clone Git Hub Git Hub Configuration Git Hub Installation grep Import Install MySQL JSON over HTTP Linux Linux Command Linux Commands Linux search local file lsyncd md5sum Multile Domain Multisite MySQL Open Source optimization performance php performance real time search remote file Remote Git Hub remote sync RESTful Scale Schema Free Search Engine Search Index Search keyword static cache Sub Domain sync Very fast Wordpress Wordpress multisite

Copyright Ask To Apps 2022 | Proudly powered by WordPress

facebook twitter google linkedin Email Rss