Monitoring DoS Attacks with the VIP Console and NetFlow v1.0

By Rob Thomas, robt@cymru.com, 21 May 2001

[ Documents ]       [ Home ]

Introduction

DoS attacks have become almost ubiquitous. While these attacks are often easily and quickly mitigated through the use of filters on the border routers, these filters can prevent the proper analysis of the attack by obfuscating the attack and its data. For example, an ICMP attack against a given host, from a range of source IP addresses, can be quickly blocked by an ACL that blocks all ICMP to the victim host. However, this leaves only ACL counters as an indicator of the duration and intensity of the attack. Further, the ACL (presuming it does not include the log-input keyword) does not reveal the source IP addresses of the attack. Have the source addresses changed? Are there additional hosts joining the attack?  Which source is producing the bulk of the packets? These are valid questions, and the data provided by ACL counters will not provide the answers. Clearly there must be a better way. With a Cisco VIP card and NetFlow, it is possible to both block and monitor the attack.

The Cisco VIP (Versatile Interface Processor) is a "router on a card." It contains a CPU, memory, an IOS image, and all of the necessary structures to perform packet switching. Often used in the Cisco 7513 router, the VIP improves performance by removing the burden of packet processing, in many cases, from the RSP (Route Switch Processor). Because the VIP is a "router on a card" that runs an IOS image, it also has many of the same commands and data structures as the RSP.

NetFlow is a data collection method in Cisco routers that allows router support personnel to trace the flows that pass through the router. NetFlow is often used for performance and trend analysis as well as billing.

Through the use of the undocumented VIP console and NetFlow, a DoS attack may be closely monitored and analyzed. The VIP console can also be used to perform the more routine and mundane router performance checks, such as show proc cpu.

This method has been tested, in the lab and in production (during actual DoS attacks), on a Cisco 7513 router with the VIP2-50 line card.

Note that methods exist to secure a border router prior to an attack. DoS mitigation methods and NetFlow configuration can be reviewed in the Secure IOS Template.   If BGP is used, there are additional defense methods available as documented in the Secure BGP Template.
 
 

Caveat

The if-con command used to access the VIP console is UNDOCUMENTED and UNSUPPORTED by Cisco. Cisco will not assist anyone with the use (or misuse) of this command. While this methodology has been fully "battle tested" on both lab and production equipment, and has been successfully utilized during several DoS attacks, your mileage may vary.  The author is not responsible for any damage caused by the use of this command.  The author also won't take undue credit for any benefit derived from the use of this command.  ;-)

 

Credits

My thanks to the following for input, reviews, and suggestions.  

The VIP Console

The syntax for the if-con command is if-con <SLOT>, where <SLOT> is the chassis slot location in which the VIP is inserted. The if-con command can only be run from enable mode, and there is no on-line help available for it. This command will prompt for the mode of access, either console or debug. Unless you have access to the IOS source code and a tolerance for sudden router reloads, you will likely not benefit from entering debug mode. ;-)  Here is an example: secure-router01>if-con 9
Console or Debug [C]: [ PRESS ENTER HERE ]
Entering CONSOLE for VIP2 R5K 9
Type "^C^C^C" or "if-quit" to end this session

[ PRESS ENTER HERE ]

VIP-Slot9>

Note the prompt – VIP-Slot9>. This indicates that the user is on the VIP console for the VIP in slot 9 of the chassis. This makes it quite clear that this is not the RSP console. To exit the VIP console, the syntax is simply if-quit. To wit: VIP-Slot9>if-quit

Disconnecting from slot 9 CONSOLE after 00:02:12

secure-router01>

The prompt has returned to the expected RSP CLI prompt, and the amount of time spent in the VIP console mode is noted.

Once in the VIP console, many of the RSP CLI commands will work as expected. This, combined with NetFlow, is what makes DoS monitoring possible.
 
 

Monitoring a DoS Attack

To illustrate the benefit of the VIP console during a DoS attack, presume the following scenario: A single host, IP address 1.1.1.1, is the target of a DDoS attack. The attack consists of a high rate of small UDP packets to an unused port. The source IP addresses of the attack are far too numerous, dynamic, and disparate to block with black hole routes or selective ACLs. The target host is also suffering a performance degradation due to the rate of packets flowing to the host. It is decided to block all UDP to this host with a simple ACL: access-list 105 remark Temporary ACL to block the DDoS attack against 1.1.1.1 – 20 May 2001 robt
access-list 105 deny udp any host 1.1.1.1
access-list 105 permit ip any any
Due to the high rate of packets, it is decided not to log the ACL, as this might negatively impact the performance of the router and syslog server. This ACL is applied to the external interfaces of the router, which are located on the VIP in slot 9. The external interfaces are HSSI interfaces, HSSI9/0/0 and HSSI9/1/0. The victim host is saved. However, this makes tracing the attack difficult. What source IP addresses are used in the attack? Are new source addresses entering the fray? Which source addresses are the key contributors? While ACL counters can be utilized to determine if the attack is still raging, it can be quite difficult to determine the intensity of the attack, as well as the contribution to the attack of each participating host. Fortunately, the border router utilizes VIPs and NetFlow, thus the data on the attack is still available. We move to the VIP console to learn more: secure-router01>if-con 9
Console or Debug [C]: [ PRESS ENTER HERE ]
Entering CONSOLE for VIP2 R5K 9
Type "^C^C^C" or "if-quit" to end this session

[ PRESS ENTER HERE ]

VIP-Slot9>

While ACL 105 has been applied to the two interfaces located on this VIP, the VIP actually routes any denied packets to the Null interface. NetFlow caches these flows for a short time, allowing a user to monitor them and extract copious details about them. Note that these flows are routed to the VIP Null interface, and are not sent to the RSP. Thus, the NetFlow cache on the RSP does not contain any of these flows and can not access this data.

The show proc cpu command can be used to determine if the VIP is still heavily utilized by the attacking packets. To wit:

VIP-Slot9>sh proc cpu

CPU utilization for five seconds: 80%/80%; one minute: 80%; five minutes: 70%

[ Output truncated. ]

Based on the CPU utilization statistics, it is clear that the VIP is spending most of its time switching packets. The first number in the 80%/80% entry indicates the CPU utilization. The second number indicates what percentage of CPU utilization is due to network interrupts – in other words, packet processing. Next, NetFlow is queried to see what sort of traffic is hitting this VIP. This is done with the show ip cache flow command, with referenced points highlighted: secure-router01#if-con 9
Console or Debug [C]:
Entering CONSOLE for VIP2 R5K 9
Type "^C^C^C" or "if-quit" to end this session
 

VIP-Slot9>show ip cache flow
IP packet size distribution (489639251 total packets):
   1-32   64   96  128  160  192  224  256  288  320  352  384  416  448  480
   .000 .992 .000 .003 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000

    512  544  576 1024 1536 2048 2560 3072 3584 4096 4608
   .000 .000 .000 .000 .003 .000 .000 .000 .000 .000 .000

IP Flow Switching Cache, 8913408 bytes
  5088 active, 125984 inactive, 1843766371 added
  805412120 ager polls, 0 flow alloc failures
  Active flows timeout in 30 minutes
  Inactive flows timeout in 15 seconds
  last clearing of statistics never
Protocol         Total    Flows   Packets Bytes  Packets Active(Sec) Idle(Sec)
--------         Flows     /Sec     /Flow  /Pkt     /Sec     /Flow     /Flow
TCP-Telnet       28084      0.0         1    45      0.0       0.1      11.7
TCP-FTP         172835      0.0         1    47      0.0       2.4      13.7
TCP-FTPD          2818      0.0         1    40      0.0       0.2      11.3
TCP-WWW        5551226      1.2         1    53      1.3       0.1       5.0
TCP-SMTP          4179      0.0         1    42      0.0       1.0      12.2
TCP-X             2594      0.0         1    40      0.0       0.6      11.2
TCP-BGP           2546      0.0         1    40      0.0       0.2      11.5
TCP-NNTP          2554      0.0         1    40      0.0       0.1      11.2
TCP-Frag           177      0.0         2   269      0.0       1.7      16.8
TCP-other       528636      0.1         1    40     65.5       0.6      35.5
UDP-DNS          11596      0.0         1    54      0.0       0.8      17.2
UDP-NTP            723      0.0         2    40      0.0       9.0      16.8
UDP-TFTP           763      0.0         3    37      0.0      10.2      16.9
UDP-Frag            25      0.0         1    40      0.0     251.4      15.0
UDP-other    169720402     39.5         1    40     46.2       0.6      11.3
ICMP            275131      0.0        10   759      0.6       7.7      14.2
IGMP                36      0.0      1789  1246      0.0      15.2      16.9
IP-other             7      0.0        19    64      0.0      18.9      17.5
Total:       176304332     41.0         2    44    113.9       0.6      11.2

SrcIf         SrcIPaddress    DstIf         DstIPaddress    Pr SrcP DstP  Pkts
Hs9/1/0       192.168.2.51    Null          1.1.1.1         11 04A9 0017   614K
Hs9/1/0       192.168.47.72   Null          1.1.1.1         11 05F9 0017   281K
Hs9/1/0       192.168.49.52   Null          1.1.1.1         11 08EA 0017    65K
Hs9/1/0       192.168.32.18   Null          1.1.1.1         11 08EC 0017  1463K
Hs9/1/0       192.168.208.208 Null          1.1.1.1         11 0411 0017  8351K
Hs9/1/0       192.168.77.66   Null          1.1.1.1         11 126F 0017  1763K
Hs9/1/0       192.168.184.159 Null          1.1.1.1         11 0609 0017   191K
Hs9/1/0       192.168.22.48   Null          1.1.1.1         11 0885 0017  1520K
Hs9/1/0       192.168.22.48   Null          1.1.1.1         11 0883 0017    66K
Hs9/1/0       192.168.7.44    Null          1.1.1.1         11 0F07 0017    97K
Hs9/1/0       192.168.7.44    Null          1.1.1.1         11 0F09 0017  2084K
Hs9/1/0       192.168.54.208  Null          1.1.1.1         11 040C 0017  3018K
Hs9/1/0       192.168.248.90  Null          1.1.1.1         11 0521 0017   201K
Hs9/1/0       192.168.201.177 Null          1.1.1.1         11 060C 0017   171K
Hs9/1/0       192.168.201.177 Null          1.1.1.1         11 054C 0017   107K
[ Output truncated. ]
VIP-Slot9>if-quit

Disconnecting from slot 9 CONSOLE after 00:03:25

secure-router01#

The host 1.1.1.1 is a web server, so web traffic (TCP-WWW) is expected. However, this site does not send or receive and UDP based data, so the UDP-other entry is quite troublesome. We see very few entries for UDP-fragments, so this does not appear to be a fragment attack. As noted in the 64 column of the IP packet size distribution stanza, 99.2% (.992) of the packets are 64 bytes in size. The Bytes/Pkt column of the UDP-other entry reveals that the packets are all 40 bytes in size (not counting IP header information).  Thus this is an attack of many small packets. This is the first indicator of woe, and requires further analysis.

The individual flows provide the granular detail necessary for proper analysis. Note that the listed IP addresses are directing a high rate of packets to the host 1.1.1.1. The entries indicate that the flows are entering the site through the HSSI 9/1/0 interface ("SrcIf"). None of the flows are entering through the HSSI9/0/0 interface. Due to ACL 105, the flows are sent to the Null interface ("DstIf"). The destination address is the target host, 1.1.1.1 ("DstIPaddress").

The next three columns are all in hexidecimal. The protocol ("Pr") is 11, which equates to decimal 17 – UDP. The source ports ("SrcP") are varied. The destination port ("DstP") is a consistent 17, which equates to decimal 23. UDP port 23 is an unused port on this (and most) system. Finally we know the number of packets ("Pkts") each host has generated in the poll interval. These are quite high.

So what have we learned? We know that this attack is UDP based. We know that the packets are not fragments, nor do they claim to be fragments. We know that the majority of the packets are 64 bytes in length. We know of several source addresses used in the attack, and these are likely spoofed based on the address type (RFC 1918). These can be used in an effort to trace the source (see the Tracking Spoofed IP Addresses article). We know the source interface (HSSI9/1/0), which can be traced to an upstream router or provider. This greatly narrows the field when attempting to uncover the source of the attack. Since the attacks are all coming from the same interface, it is thus possible that the sources aren’t so very disparate after all. Perhaps they are even in the same netblock or company! We know the intensity of the attack based on the packet counts. We can now monitor the duration and intensity of the attack.

In short, we have collected enough data to pursue the sources of the attack, involve the upstream ISPs in the pursuit, and possibly modify our ACL to accommodate the specific target ports.
 
 

Conclusion

While no tracking method is perfect, the VIP console combined with NetFlow adds another analysis tool to the network administrator’s toolkit. This method provides both monitoring as well as data collection – data that can be used to track the sources of the attack and take appropriate action.

[ Documents ]      [ Home ]

Rob Thomas, robt@cymru.com, http://www.cymru.com