Sunday, 10 April 2011

When you need log data you need good timekeeping

When there is a technical error with the device or you need to troubleshoot a failure the most important thing is time. Recently we were onsite troubleshooting a denial of service attack (DoS) on a web server for a client. They informed us that it had taken place sometime between 22:00 and 22:10 on a certain date. We asked for the web logs and then for the router/firewall logs for the period concerned. We spent time looking though the logs and noticed no commonality with the customers experience and the logs we were looking at. Indeed there appeared to be some difference even with the log files we had been given. For example a very clear HTTP GET request on the web server had no matching event on the firewall...

You guessed it, not only was the time wrong on the web server but it was also wrong on the firewall. Worse still the timezone was wrong so even if the time setting was correct we were at best an hour out as we were operating in BST.

Setting the time on the web server to point to the cisco switch (Catalyst 4500) to obtain it’s time was the first stage. The timezone was setup correctly and we were to use the windows time service using NTP. Here are the steps we took to setup the Cisco switch to set it’s timezone, to setup the correct British Summer Time skew plus the NTP daemon function so that the Cisco device to collect the correct time information from the internet and also allow the windows server to collect the time information.

First we setup the timezone so that the switch knew where in the world it was:

Screen shot 2011-04-10 at 21.24.37

Now we need to setup the NTP daemon itself. We need to find an NTP source out there to deal with us. There are different tiers of NTP service called ‘strata’. We don’t really need to be hugely accurate so we chose a stratum 2 NTP source. Here are a list of the UK NTP servers we used.

Screen shot 2011-04-10 at 21.28.18

The prefer keyword is just to say to IOS “if this one is available then we’d like you to trust this one most”. We also want to make sure our log data on the switch is given a timestamp.

Screen shot 2011-04-10 at 21.24.47

So lets just make sure it’s all working. Type “show ntp associations”

Screen shot 2011-04-10 at 21.25.16

No comments:

Post a Comment