Visualising Squid logs with Kibana

Following on from the quick guide I did on showing ASA logs with Kibana, I thought it’d be a good idea to show off how this is also great at visualising squid logs kibana. I use squid extensively and while trawling through /var/log/squid/access.log works, having this information presented visually enables you to spot anomalies or performance and configuration problems. Questions such as “Why are there so many 404’s all of sudden?”, “what is the top URL visited in the last 24 hours?” can be displayed without the need to run the logs through dedicated software or doing a heap of awk, sed, and cat on log files.

visualising squid logs with kibana

visualising squid logs with kibana

There are a few moving parts here so I will address each individually. My ELK stack looks like the following.

Squid (Ubuntu Server) -> Logstash -> Elasticsearch. Squid talks to Logstash using the filebeat, and Logstash speaks to elasticsearch over the standard HTTP api.

Filebeat

In order to ship logs from the squid server to logstash you will need to download and install filebeat. This can be downloaded here, as I am using ubuntu installing this is as simple as download the Debian package, and installing it with ‘dpkg -i filebeat-5.0.2-amd64.deb’. This will create a configuration file under /etc/filebeat/ named filebeat.yml which you will need to edit. It’s pretty straightforward but there are some important changes which need to be made.

Edit Paths to point to the squid access log.

paths:
- /var/log/squid3/access.log

Edit your output section to send logs to logstash. Replace 1.1.1.1 with your own logstash host. It’s recommended to have at least two logstash instances in production environments for obvious reasons.

output.logstash:
# The Logstash hosts
hosts: ["1.1.1.1:5044", "1.1.1.2:5044"]

That’s it! Reload the daemon by typing ‘service filebeat restart’.

Logstash

Next in line is Logstash. This will take the logs shipped from Filebeat, format them (or add, remove, or mutate fields) and send them into their finally resting place; elasticsearch. The same installation process can be followed to install logstash on a Debian or Ubuntu system. Logstash can be downloaded here. Install it with ‘dpkg -i logstash-5.0.2.deb’. This will create a new directory /etc/logstash/. Configuration files are stored under /etc/logstash/conf.d. You can create any file here which is pertinent to your deployment – I have gone with ‘logstash-forwarder.conf’ although it can be named anything essentially. Edit this file with your favourite text editor.Inputs. The following will create two inputs, one for syslog messages (that we used previously for ASA logs), and a second input named beats.

input {
udp {
port => 10514
type => syslog
}
beats {
port => 5044
}
}

Filtering. The following is some grok syntax (essentially regex) which will pickup messages and strip out the relevant fields. That will be displayed in Kibana. (sorry about the formatting, it’s doesn’t wrap nicely in wordpress). The following could probably be achieved with less lines of grok, but I’m lazy and don’t want to spend too much time playing with the various formats squid _may_ output it’s log files in (there are subtle differences between say HTTP and HTTPS connections).

filter {
grok {
match => [ "message", "%{POSINT:timestamp}.%{POSINT:timestamp_ms}\s+%{NUMBER:response_time} %{IPORHOST:src_ip} %{WORD:squid_request_status}/%{NUMBER:http_status_code} %{NUMBER:reply_size_include_header} %{WORD:http_method} %{NOTSPACE:request_url} %{NOTSPACE:user} %{WORD:squid}/%{IP:server_ip} %{NOTSPACE:content_type}" ]
match => [ "message", "%{NUMBER:timestamp}\s+%{NUMBER:response_time} %{IPORHOST:src_ip} %{WORD:squid_request_status}/%{NUMBER:http_status_code} %{NUMBER:reply_size_include_header} %{WORD:http_method} %{URI:request_url} %{USERNAME:user} %{WORD:squid_hierarchy_status}/%{IPORHOST:server_ip_or_peer_name} (?\S+\/\S+)" ]
match => [ "message", "%{NUMBER:timestamp}\s+%{NUMBER:response_time} %{IPORHOST:src_ip} %{WORD:squid_request_status}/%{NUMBER:http_status_code} %{NUMBER:reply_size_include_header} %{WORD:http_method} %{HOSTNAME:request_url}:%{NUMBER:tcp.port} %{NOTSPACE:user} %{WORD:squid}/%{GREEDYDATA:server_ip} %{NOTSPACE:content_type}" ]
match => [ "message", "%{NUMBER:timestamp}\s+%{NUMBER:response_time} %{IPORHOST:src_ip} %{WORD:squid_request_status}/%{NUMBER:http_status_code} %{NUMBER:reply_size_include_header} %{WORD:http_method} %{URI:request_url} %{NOTSPACE:user} %{WORD:squid}/%{GREEDYDATA:server_ip} %{NOTSPACE:content_type}" ]
}
}

Finally we need to tell logstash where it needs to send it’s messages. This is configured by defining an output:

output {
elasticsearch {
hosts => ["1.1.1.3:9200","1.1.1.4:9200"]
}

Pretty straight forward, these are your elasticsearch hosts, and once again, you should have more than one!

Kibana

Finally we need to create some searches and a dashboard in Kibana. These were exported from Kibana 5 so I expect you will need to use this version or later. Download this zip file and extract it.

Under Management > Click on Saved Objects > Import. Select these files and import them one at a time. With a bit of luck, when you click ‘Dashboard’ then open you will now have a dashboard named ‘Squid Proxy’

This is a work in progress. You will see there is a section named ‘Squid Response Times’ which lists no results. I’m looking at getting this to graph average response time of proxied requests. The reason this is a little difficult is that my time_response field was originally entered as a string rather than an interger. It means kibana won’t let me graph that. I’m just looking into the best way to convert this field in order to be able to create a date histogram of this data.

That’s a wrap

Well, that’s the gist of it. Let me know if you have any questions or improvements to make.

 

7 comments

  1. Hello, thanks for your document, do you have the finally configuration and template files? I just followed your steps but got errors.

    1. What errors are you getting and when? Is it after an Elasticsearch restart, Logstash, or Kibana? I monitor a heap of different services in my setup (Switches, Firewalls, Authentication logs) so its difficult to share my config as is. Happy to help though,

      1. The issue is with the second match line, specifically the regex “(?\S+\/\S+)” which ruby fails to parse. I haven’t looked deeper into it yet. Perhaps you’re willing to.

  2. It doesn’t appear that the search has been exported, only the Dashboard and the Visualizations. Import error shows search not found.

    1. Nope, no license is required. You need to license X-Pack which gives you some security features (amongst other things).

Leave a Reply

Your e-mail address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.