Tag : access-log
Tag : access-log
Debugging web application its very tedious job even when if website is full of processes and action. Checking apache logs and finding out request details is time consuming. For this problem i am recommending one solution.
We can use …
First Create index with mapping …
curl -XPOST localhost:9200/apache -d '{ "mappings" : { "access" : { "properties" : { "host": { "index": "analyzed", "store": "yes", "type": "ip" }, "logname": { "index": "analyzed", "store": "yes", "type": "string" }, "user": { "index": "analyzed", "store": "yes", "type": "string" }, "time": { "index": "analyzed", "store": "yes", "type": "date" ,"format" : "yyyy:MM:dd HH:mm:ss"}, "method": { "index": "not_analyzed", "store": "yes", "type": "string" }, "url": { "index": "not_analyzed", "store": "yes", "type": "string" }, "protocol": { "index": "not_analyzed", "store": "yes", "type": "string" }, "status": { "index": "analyzed", "store": "yes", "type": "string" }, "sentbytes": { "index": "not_analyzed", "store": "yes", "type": "string"}, "referrer": { "index": "not_analyzed", "store": "yes", "type": "string"}, "useragent": { "index": "analyzed", "store": "yes", "type": "string" } }}}}'
Shell Script to parse apache and put into elasticsearchwatch full xXx: Return of Xander Cage movie online
#!/bin/bash ElasticUrl="http://localhost:9200" Index="apache" Type="access" LogFile=/var/log/httpd/access_log tail -f $LogFile | while read myline; do JSON=$(php shipper.php "$myline") echo curl -i \ -H "Accept: application/json" \ -H "Content-Type:application/json" \ -X POST --data "$JSON" "http://localhost:9200/apache/access" done
PHP script to convert apache log to json format
< ?php require_once("apache-log-parser/src/Kassner/ApacheLogParser/Factory.php"); require_once("apache-log-parser/src/Kassner/ApacheLogParser/FormatException.php"); require_once("apache-log-parser/src/Kassner/ApacheLogParser/ApacheLogParser.php"); use Kassner\ApacheLogParser\ApacheLogParser; $mapping=array(); if(isset($argv[1])){ $parser = new ApacheLogParser("%h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\""); $logLine = $argv[1]; $entry = $parser->parse($logLine); $method=''; $url=''; $protocol=''; if(isset($entry->request)){ $arrReq=explode(" ",$entry->request); if(count($arrReq)==3){ $method=$arrReq[0]; $url=$arrReq[1]; $protocol=$arrReq[2]; } } $entry->stamp=@date("Y:m:d h:i:s",$entry->stamp); $mapping=array( 'host'=>$entry->host, 'logname'=>$entry->logname, 'user'=>$entry->user, 'time'=>$entry->stamp, 'method'=>$method, 'url'=>$url, 'protocol'=>$protocol, 'status'=>$entry->status, 'sentbytes'=>$entry->sentBytes, 'referer'=>$entry->HeaderReferer, 'useragent'=>$entry->HeaderUserAgent ); echo json_encode($mapping);
Now you have to just keep running of shell script….
$sh <shell script name> & </shell>
Categories: Elasticsearch, Linux, Website Peformance