Following on from the quick guide I did on showing ASA logs with Kibana, I thought it’d be a good idea to show off how this is also great at visualising squid logs kibana. I use squid extensively and while trawling through /var/log/squid/access.log works, having this information presented visually enables you to spot anomalies or performance and configuration problems. Questions such as “Why are there so many 404’s all of sudden?”, “what is the top URL visited in the last 24 hours?” can be displayed without the need to run the logs through dedicated software or doing a heap of awk, sed, and cat on log files.

visualising squid logs with kibana

visualising squid logs with kibana

There are a few moving parts here so I will address each individually. My ELK stack looks like the following.

Squid (Ubuntu Server) -> Logstash -> Elasticsearch. Squid talks to Logstash using the filebeat, and Logstash speaks to elasticsearch over the standard HTTP api.

Filebeat

In order to ship logs from the squid server to logstash you will need to download and install filebeat. This can be downloaded here, as I am using ubuntu installing this is as simple as download the Debian package, and installing it with ‘dpkg -i filebeat-5.0.2-amd64.deb’. This will create a configuration file under /etc/filebeat/ named filebeat.yml which you will need to edit. It’s pretty straightforward but there are some important changes which need to be made.

Edit Paths to point to the squid access log.

paths:
- /var/log/squid3/access.log

Edit your output section to send logs to logstash. Replace 1.1.1.1 with your own logstash host. It’s recommended to have at least two logstash instances in production environments for obvious reasons.

output.logstash:
# The Logstash hosts
hosts: ["1.1.1.1:5044", "1.1.1.2:5044"]

That’s it! Reload the daemon by typing ‘service filebeat restart’.

Logstash

Next in line is Logstash. This will take the logs shipped from Filebeat, format them (or add, remove, or mutate fields) and send them into their finally resting place; elasticsearch. The same installation process can be followed to install logstash on a Debian or Ubuntu system. Logstash can be downloaded here. Install it with ‘dpkg -i logstash-5.0.2.deb’. This will create a new directory /etc/logstash/. Configuration files are stored under /etc/logstash/conf.d. You can create any file here which is pertinent to your deployment – I have gone with ‘logstash-forwarder.conf’ although it can be named anything essentially. Edit this file with your favourite text editor.Inputs. The following will create two inputs, one for syslog messages (that we used previously for ASA logs), and a second input named beats.

input {
udp {
port => 10514
type => syslog
}
beats {
port => 5044
}
}

Filtering. The following is some grok syntax (essentially regex) which will pickup messages and strip out the relevant fields. That will be displayed in Kibana. (sorry about the formatting, it’s doesn’t wrap nicely in wordpress). The following could probably be achieved with less lines of grok, but I’m lazy and don’t want to spend too much time playing with the various formats squid _may_ output it’s log files in (there are subtle differences between say HTTP and HTTPS connections).

filter {
grok {
match => [ "message", "%{POSINT:timestamp}.%{POSINT:timestamp_ms}\s+%{NUMBER:response_time} %{IPORHOST:src_ip} %{WORD:squid_request_status}/%{NUMBER:http_status_code} %{NUMBER:reply_size_include_header} %{WORD:http_method} %{NOTSPACE:request_url} %{NOTSPACE:user} %{WORD:squid}/%{IP:server_ip} %{NOTSPACE:content_type}" ]
match => [ "message", "%{NUMBER:timestamp}\s+%{NUMBER:response_time} %{IPORHOST:src_ip} %{WORD:squid_request_status}/%{NUMBER:http_status_code} %{NUMBER:reply_size_include_header} %{WORD:http_method} %{URI:request_url} %{USERNAME:user} %{WORD:squid_hierarchy_status}/%{IPORHOST:server_ip_or_peer_name} (?\S+\/\S+)" ]
match => [ "message", "%{NUMBER:timestamp}\s+%{NUMBER:response_time} %{IPORHOST:src_ip} %{WORD:squid_request_status}/%{NUMBER:http_status_code} %{NUMBER:reply_size_include_header} %{WORD:http_method} %{HOSTNAME:request_url}:%{NUMBER:tcp.port} %{NOTSPACE:user} %{WORD:squid}/%{GREEDYDATA:server_ip} %{NOTSPACE:content_type}" ]
match => [ "message", "%{NUMBER:timestamp}\s+%{NUMBER:response_time} %{IPORHOST:src_ip} %{WORD:squid_request_status}/%{NUMBER:http_status_code} %{NUMBER:reply_size_include_header} %{WORD:http_method} %{URI:request_url} %{NOTSPACE:user} %{WORD:squid}/%{GREEDYDATA:server_ip} %{NOTSPACE:content_type}" ]
}
}

Finally we need to tell logstash where it needs to send it’s messages. This is configured by defining an output:

output {
elasticsearch {
hosts => ["1.1.1.3:9200","1.1.1.4:9200"]
}

Pretty straight forward, these are your elasticsearch hosts, and once again, you should have more than one!

Kibana

Finally we need to create some searches and a dashboard in Kibana. These were exported from Kibana 5 so I expect you will need to use this version or later. Download this zip file and extract it.

Under Management > Click on Saved Objects > Import. Select these files and import them one at a time. With a bit of luck, when you click ‘Dashboard’ then open you will now have a dashboard named ‘Squid Proxy’

This is a work in progress. You will see there is a section named ‘Squid Response Times’ which lists no results. I’m looking at getting this to graph average response time of proxied requests. The reason this is a little difficult is that my time_response field was originally entered as a string rather than an interger. It means kibana won’t let me graph that. I’m just looking into the best way to convert this field in order to be able to create a date histogram of this data.

That’s a wrap

Well, that’s the gist of it. Let me know if you have any questions or improvements to make.

 

I’ve used Cisco firewalls in their various forms for the last 10 years or so, from baby PIX 501’s through to the new ASA 5500X’s. While I’ve always been a big fan of the platform, one area which has always been deficient is their logging and reporting capability. There really is no comparison when you line up other vendors such as Pallo and Checkpoint along side Cisco when it comes to this.

I’ve recently started playing around with what people call the ELK stack, and have found it to be excellent at visualising cisco ASA logs. The ELK stack consists of three applications, ElasticSearch, Logstash, and Kibana.

ElasticSearch:

Elasticsearch is a search server based on Lucene. It provides a distributed, multitenant-capable full-text search engine with a RESTful web interface and schema-free JSON documents

Logstash:

logstash is a tool for managing events and logs. You can use it to collect logs, parse them, and store them for later use (like, for searching).

Kibana:

Kibana is a great tool for real time data analytics.

The end result is a system which is able to turn simple syslog messages into a screen which looks like my example below. Broken down are graphs to represent the top protocols, actions (ie accept, deny), destination ports, origin countries, and source and destination IP’s. Within each of these views all of the content is dynamic, say you’re only interested in dropped traffic, you can click this in the graph and the whole filter will change to only represent this data.

ASA

Click on “Deny” as the action, and “TCP” and as the protocol shows the top sources of this traffic. It immediately becomes easy to see the usual volumes of this type of traffic over any period of time you specify. Anomalies are easy to see and can be drilled down into to see more closely.

While setting up an ELK stack is outside what I’d planned on mentioning here, digital ocean have a good guide on setting this all up. The pertinent logstash config I have used for the above is as follows:

input {
udp {
port => 10514
type => syslog
}
}

filter {
if [type] == "syslog" {
grok {
patterns_dir => "./patterns"
match => { "message" => "%{CISCOFW106014}" }
match => { "message" => "%{CISCOFW106001}" }
match => { "message" => "%{CISCOFW106023}" }
match => { "message" => "%{CISCOFW313005}" }
match => { "message" => "%{CISCOFW302020_302021}" }
match => { "message" => "%{CISCOFW302013_302014_302015_302016}" }
match => { "message" => "%{CISCOFW106015}" }
match => { "message" => "%{CISCOFW106006_106007_106010}" }
match => { "message" => "%{CISCOFW106100}" }
add_tag => [ "firewall" ]
}

geoip {
source => "src_ip"
target => "geoip"
database =>"/opt/logstash/vendor/geoip/GeoLiteCity.dat"
add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}" ]
}

mutate {
convert => [ "[geoip][coordinates]", "float" ]
lowercase => [ "protocol" ]
}
}
}

output {
redis { host => "127.0.0.1"
data_type => "list"
key => "logstash" }
}


The above system will listen for syslog messages on udp 10514, run them through grok to extract the pertinent parts of the message, add geo-ip data, and forward it onto a local redis cache. Another system (the ELK receiver/indexer) polls this system and retrieves the logs and finally displays them to the user through the Kibana interface.