It is with a heavy heart that I must announce I’m about to turn off our Juniper IVE’s (aka SSL VPN). In reality, all we were using these for was publishing applications and presenting these as bookmarks on a landing page. It’s been a very capable, reliable product over the years so it will come with some hesitation when I have to hit the shutdown button on these later this week.

I started this blog back in 2011 and most of the posts were about the trials and tribulations on that platform. Being fully “Azured” we decided to go with Azure Remote App and move away from RSA/Juniper SSL combo. Commercially it makes a lot of sense; RSA has been replaced by Azure MFA which is offered essentially for free when you have AD premium users.

It’s more than a little unfortunate that they have now shitcanned the product  and wont offer it after August 31st, 2017.  We intend on using it until we have something that offers similar functionality. The announcement is here. They are pushing a Citrix solution so it would seem little ironic if we ended up on that given we retired all of our Citrix environment some time ago.

Gartner stopped doing a SSL Gateway Quadrant a number of years ago and had implied that there would no longer be dedicated devices for this purpose. Instead, the function would be rolled into other network infrastructure. I have seen this to be the case where products such as F5’s LTM have been modularised to include this (theirs is called APM). I had done a POC (a long time ago) on both APM and also Netscaler and found these to be capable but expensive products.

We need a device which will terminate SSL connections, present applications to users, and ideally also offer an RDP gateway (as this is quite frankly awful on RemoteApp). Netscaler have an Azure Marketplace image so I might see whether this is a worthy successor. I’m less excited about another IaaS instance to look after but I don’t know if any managed cloud services which fit this space currently.

Following on from the quick guide I did on showing ASA logs with Kibana, I thought it’d be a good idea to show off how this is also great at visualising squid logs kibana. I use squid extensively and while trawling through /var/log/squid/access.log works, having this information presented visually enables you to spot anomalies or performance and configuration problems. Questions such as “Why are there so many 404’s all of sudden?”, “what is the top URL visited in the last 24 hours?” can be displayed without the need to run the logs through dedicated software or doing a heap of awk, sed, and cat on log files.

visualising squid logs with kibana

visualising squid logs with kibana

There are a few moving parts here so I will address each individually. My ELK stack looks like the following.

Squid (Ubuntu Server) -> Logstash -> Elasticsearch. Squid talks to Logstash using the filebeat, and Logstash speaks to elasticsearch over the standard HTTP api.

Filebeat

In order to ship logs from the squid server to logstash you will need to download and install filebeat. This can be downloaded here, as I am using ubuntu installing this is as simple as download the Debian package, and installing it with ‘dpkg -i filebeat-5.0.2-amd64.deb’. This will create a configuration file under /etc/filebeat/ named filebeat.yml which you will need to edit. It’s pretty straightforward but there are some important changes which need to be made.

Edit Paths to point to the squid access log.

paths:
- /var/log/squid3/access.log

Edit your output section to send logs to logstash. Replace 1.1.1.1 with your own logstash host. It’s recommended to have at least two logstash instances in production environments for obvious reasons.

output.logstash:
# The Logstash hosts
hosts: ["1.1.1.1:5044", "1.1.1.2:5044"]

That’s it! Reload the daemon by typing ‘service filebeat restart’.

Logstash

Next in line is Logstash. This will take the logs shipped from Filebeat, format them (or add, remove, or mutate fields) and send them into their finally resting place; elasticsearch. The same installation process can be followed to install logstash on a Debian or Ubuntu system. Logstash can be downloaded here. Install it with ‘dpkg -i logstash-5.0.2.deb’. This will create a new directory /etc/logstash/. Configuration files are stored under /etc/logstash/conf.d. You can create any file here which is pertinent to your deployment – I have gone with ‘logstash-forwarder.conf’ although it can be named anything essentially. Edit this file with your favourite text editor.Inputs. The following will create two inputs, one for syslog messages (that we used previously for ASA logs), and a second input named beats.

input {
udp {
port => 10514
type => syslog
}
beats {
port => 5044
}
}

Filtering. The following is some grok syntax (essentially regex) which will pickup messages and strip out the relevant fields. That will be displayed in Kibana. (sorry about the formatting, it’s doesn’t wrap nicely in wordpress). The following could probably be achieved with less lines of grok, but I’m lazy and don’t want to spend too much time playing with the various formats squid _may_ output it’s log files in (there are subtle differences between say HTTP and HTTPS connections).

filter {
grok {
match => [ "message", "%{POSINT:timestamp}.%{POSINT:timestamp_ms}\s+%{NUMBER:response_time} %{IPORHOST:src_ip} %{WORD:squid_request_status}/%{NUMBER:http_status_code} %{NUMBER:reply_size_include_header} %{WORD:http_method} %{NOTSPACE:request_url} %{NOTSPACE:user} %{WORD:squid}/%{IP:server_ip} %{NOTSPACE:content_type}" ]
match => [ "message", "%{NUMBER:timestamp}\s+%{NUMBER:response_time} %{IPORHOST:src_ip} %{WORD:squid_request_status}/%{NUMBER:http_status_code} %{NUMBER:reply_size_include_header} %{WORD:http_method} %{URI:request_url} %{USERNAME:user} %{WORD:squid_hierarchy_status}/%{IPORHOST:server_ip_or_peer_name} (?\S+\/\S+)" ]
match => [ "message", "%{NUMBER:timestamp}\s+%{NUMBER:response_time} %{IPORHOST:src_ip} %{WORD:squid_request_status}/%{NUMBER:http_status_code} %{NUMBER:reply_size_include_header} %{WORD:http_method} %{HOSTNAME:request_url}:%{NUMBER:tcp.port} %{NOTSPACE:user} %{WORD:squid}/%{GREEDYDATA:server_ip} %{NOTSPACE:content_type}" ]
match => [ "message", "%{NUMBER:timestamp}\s+%{NUMBER:response_time} %{IPORHOST:src_ip} %{WORD:squid_request_status}/%{NUMBER:http_status_code} %{NUMBER:reply_size_include_header} %{WORD:http_method} %{URI:request_url} %{NOTSPACE:user} %{WORD:squid}/%{GREEDYDATA:server_ip} %{NOTSPACE:content_type}" ]
}
}

Finally we need to tell logstash where it needs to send it’s messages. This is configured by defining an output:

output {
elasticsearch {
hosts => ["1.1.1.3:9200","1.1.1.4:9200"]
}

Pretty straight forward, these are your elasticsearch hosts, and once again, you should have more than one!

Kibana

Finally we need to create some searches and a dashboard in Kibana. These were exported from Kibana 5 so I expect you will need to use this version or later. Download this zip file and extract it.

Under Management > Click on Saved Objects > Import. Select these files and import them one at a time. With a bit of luck, when you click ‘Dashboard’ then open you will now have a dashboard named ‘Squid Proxy’

This is a work in progress. You will see there is a section named ‘Squid Response Times’ which lists no results. I’m looking at getting this to graph average response time of proxied requests. The reason this is a little difficult is that my time_response field was originally entered as a string rather than an interger. It means kibana won’t let me graph that. I’m just looking into the best way to convert this field in order to be able to create a date histogram of this data.

That’s a wrap

Well, that’s the gist of it. Let me know if you have any questions or improvements to make.

 

I’ve gone a bit git mad. I’ve ‘gitted’ my powershell scripts, my network security group CSV files, and linux configuration files. Leveraging our departments visual studio online account I’ve found storing these resources in a git repository a fantastic way to keep track of how, when, and why files are changing, and by who. It also serves as a good way of backing up these files as any files can be restores simply by re-cloning the git repository.

I think with the way automation is changing the way we work it’s ever so important to understand how these tools can help. In teams that manage collaborative files, I highly recommend spending an afternoon getting the hang of git.