Virtual Private Networks Troubleshooting - Part I

I've seen quite a number of posts asking why VPN users are not able to access their network even though they have been successfully connected.

Let's try to troubleshoot what happens here. A very common VPN setup is to use 2 NICs and run Routing and Remote Access on one of them. The benefits of this would be that you can isolate VPN traffic and keeping it from overloading the NIC that handles the internal network traffic. One thing to note for is that the two NICs have to be assigned IPs from different networks or subnets. For example:

1st NIC (VPN): 192.168.1.2 /24
2nd NIC(internal network): 192.168.2.2 /24

(Don't worry if you don't understand the /24. It's called the slash notation for the subnet mask which is the equivalent of 255.255.255.0)

Through the initial RRAS setup, users will be able to connect to the VPN just fine and will be assigned an IP address of one of the networks. The problem arise when they try to connect to network resources on through the 2nd NIC (the other network). This is due to the fact that Windows doesn't know how to route the packets from the 192.168.1.0 network to 192.168.2.0 network.

You can tell Windows how to route traffic between the two NICs by configuring a static route with the "Route" command. Type "Route /?" at the Command Prompt for more information.

And as usual, I have a second way to accomplish the job by turning on Routing using RIP or OSPF and add the two NICs to it.

One thing that I see lacking from most Windows Administrators is the understanding of TCP/IP and network routing. I would recommend Windows Administrators to take CCNA anytime. Even though it might seem so remote from their daily tasks, one can walk away from a CCNA class with so much helpful information that makes their job much easier to do.

Basic Load Balancing

Load Balancing normally has two modes of operation: Per-Destination and Per-Packet.

Per destination load balancing means the router distributes the data packets based on the destination address. If you have two paths going to Host A & B on the same network then all packets for Host A will travel over the first path and all packets for Host B will travel over the second path. This will preserve the packet order which is very useful in certain applications; however, it could result in unequal usage of the network links as bandwidth & load are not taken into route calculation.

Per packet load balancing means the router sends one packet over the first path and second packet over the second path; all going to the same destination. Per packet load balancing guarantees equal load across all the links; however, the packets may arrive at the destination out of order because of different delay/bandwidth may exist on different paths.

Per destination load balancing is enable by the command:

Router# config t
Router(config)# interface Ethernet 0
Router(config-if)# no ip route-cache

Now the router CPU will look at every single packet and spread them across the different path available in the routing table for the destination. This is not recommended on low end server as it could crash the router because the CPU must do all the processing and might not be able to handle it. To enable fast switching, use the following commands:

Router# config t
Router(config)# interface Ethernet 0
Router(config-if)# ip route-cache

Newer switching schemes such as Cisco Express Forwarding (CEF) allow you to do per packet and per destination load balancing more quickly but it does imply that extra resources will be needed to maintain it.

Reload

Working remotely with routers & switches you don't have the flexibility to reset the devices at will when things go wrong unless you can control power distribution to the devices.

This is where the Reload command comes in handy. Before making changes to the configuration, you can enter:

Reload in 10

which will reload the device in 10 minutes. Once the command is entered, you can proceed with the configuration changes. If things go wrong and you lost connectivity to the device, try back in about 10 minutes since the device will get reloaded with the original startup-configuration and restore connectivity.

Once you confirm that your new configuration is working properly, you can issue the command:

Reload cancel

to cancel the reload. I find this trick to be very useful when I have to telnet or SSH to a remote device and hopefully you too.

Add Redundancy With Backup Command

You can add redundancy to your network and make you network more resilient with the Backup command. What this command does is specify an interface which will act as a back up in case the primary interface fails.

Let's take a look at the following configurations:

On RouterA:

Configure terminal
Interface f0/0
Ip address 192.168.1.1 255.255.255.0
no shutdown
Interface f0/1
Ip address 192.168.2.1 255.255.255.0
no shutdown
Backup interface f0/0
Exit
Router eigrp 20
network 192.168.1.0
network 192.168.2.0

On RouterB:

Configure terminal
Interface f0/0
Ip address 192.168.2.2 255.255.255.0
no shutdown
Interface f0/1
Ip address 192.168.2.2 255.255.255.0
no shutdown
Backup interface f0/0
Interface loop 0
Ip add 192.168.8.1 255.255.255.0
Exit
Router eigrp 20
network 192.168.1.0
network 192.168.2.0
network 192.168.8.0

With these configurations loaded, both Fa0/1's will change to up/up while Fa0/0's will go into standby mode and monitor the activities on Fa0/1. In the event that Fa0/1 goes down, Fa0/0 will switch to active mode and establish neighbor relationship with the connected interface and complete the failover.

To verify this, from RouterA you can ping 192.168.8.1 with the repeat parameter:

Ping 192.168.8.1 repeat 10000

This command will ping 192.168.8.1 10k times. While it's pinging, unplug the cable to Fa0/1. you will see that the link go down, the ping will fail and then within seconds Fa0/0 will be up and resume the active role and the pinging will resume as nothing has happened. If the link on Fa0/1 becomes active again, Fa0/0 will switch back to standby mode.

This is a quick and easy way to add redundancy to your network. There are other techniques that you can use to add redundancy to your network to include both layer 2 and layer 3 redundancies which I will discuss at another opportunity.

Keyboard Shortcuts

If you use CLI to manage your Cisco routers & switches then you should to get familiarized with the keyboard shortcuts as the can save you a lot of time & typings.

Here's the shortcut list:

• Ctrl+B: Moves back one character at a time
• EscB: Moves back one word at a time
• Ctrl+A: Moves to the start of a line
• Ctrl+F: Moves forward one character at a time
• Esc+F: Moves forward one word at a time
• Ctrl+E: Moves to the end of a line
• Delete: Erases the character to the left of the cursor
• Backspace: Erases the character to the left of the cursor
• Ctrl+D: Deletes the character at the cursor
• Ctrl+K: Deletes all characters from the cursor to the end of the line
• Ctrl+U: Deletes all characters from the cursor to the beginning of the line
• Ctrl+X: Deletes all characters from the cursor to the beginning of the line
• Ctrl+W: Deletes the word to the left of the cursor
• Esc+D: Deletes from the cursor to the end of a word

If none of these commands work, enhanced editing may be turned off. You can turn on enhanced editing issuing the command at the privileged command prompt:

Router# terminal editing

Stop Logging Messages From Interrupting Your Work

One of the things that I find really annoying with the Cisco devices is that when I am configuring a router or a switch, the console messages keep popping up all over the places and interfere with my work. Although this doesn't cause any typo error but it's really difficult to see what you have typed. Luckily, we can control the logging message behavior.

To disable, use:
Router(config)# no logging console

or if you prefer to see what's going on with the router/switch then use the "logging synchronous" command:
Router(config)# line console 0
Router(config-line)# logging synchronous
Router(config)# line vt 0 4
Router(config-line)# logging synchronous

It would be nice if these were taken into consideration and set as default options when the IOS is built but it's not the case so until they change it, you will have to either put up with it or do a little extra work to change the default behavior.

Optimize Routing Updates

Dynamic routing protocols such as EIGRP, OSPF & RIP update the routing table through the use of protocol updates. These updates are sent periodically from one router to another and can create quite a bit of traffic on your network.

You can optimize your router and reduce this kind of traffic through the use of the "passive-interface" command. It works by instructing the router not to send update to specific interface(s).

The "passive-interface" command can be used in two ways:

1. Specify an interface that will not participating in routing updates.

Router(config)# router rip
Router(config-router)# passive-interface Ethernet 0/0

2. Make all interfaces passive then use "no passive-interface" on interfaces that you want to participating in routing updates.

Router(config)# router rip
Router(config-router)# passive-interface default
Router(config-router)# no passive-interface Ethernet 0/0

Note that the "passive-interface" command works on all IP routing protocols except BGP.

Start a Port Configuration In a Clean State

If you are reconfiguring or changing the port settings on a router you can remove or change the old commands one by one until the job is done. However, there's a "default interface" command that you can use to quickly reset the port's configuration to its default values:

Router(config#): default interface fa0/1

Once the "default interface"command issued, you can be certain that all previous commands set on the specified ports are gone.

Protecting Your Network Edge with TTL

Protecting your network by authenticating every EBGP session is a good measure to keep the bad guys out. While this approach can keep attackers from getting inside your network, it can't keep them from launching attacks by flooding packets at BGP port (TCP port 179) and causing tremendous authentication failures which can lead to legistimate BGP connections at the target interface to fail, BGP process failure or even system crash due to spike in system resources & CPU spike.

Fortunately, you can use TTL (Time To Live) as an additional measure to reduce such DoS attacks against BGP. The default behavior of most BGP implementations is that it will send packets to external neighbors with a TTL value of 1 and accepts packets from external neighbors with TTL of 0 or higher.

You can tighten up your network by changing this default behavior by having BGP originated packets with a TTL value of 255 and only accepting packets with TTL of 254 (measured after the local router has decremented the TTL of the packet it received) or higher.

Now even if the attackers originate packets with a maximum TTL value of 255, the packets can't get to your BGP network if the attackers are not directly connected to the target interface. This is because each router hop decrements the TTL value by 1. If they are two hops away, by the time the packets reach the port TTL will be decremented to 253 and will be rejected.

You can configure the TTL value by issuing the statement:

neighbor-address ttl-security hops 254

This will instruct your router to discard incoming BGP packets with TTL value below 254 and also set outgoing packets TTL value to 255. You will need to make sure that both neighbors are configure with the same statement.

The Day The Routers Die

Those of us that are in the networking field might find this video amusing.

Note: In the song, the performer (Gary Feldman) mentioned RIPE55 quite a few times. RIPE stands for Réseaux IP Européens (European IP Networks) which is a forum open to all parties who are interested in the technical development of internet and 55 is the meeting number during which he performed the song.

Gary sung in native English accent so it might be a bit hard to understand. Here's a to link the lyrics.

Configure DNS or Disable DNS lookup

If you mistype a command in the Privileged Mode on Cisco router, by default, the router thinks you're trying to connect to a remote host through Telnet. So it performs a DNS lookup on the information you entered.

If you haven't configured DNS on the router, the command prompt will hang until the DNS lookup timeouts. This is one of the thing that really bothers me so to resolve it, I have two options:

The first option is to disable DNS using the "domain-lookup" command as in:

Router(config)# no ip domain-lookup

The second option is to properly configure DNS and point it to a valid DNS server. Here's an example:

Router(config)# ip name-server 4.2.2.1

By configure the router using one of the two options above it will save you a bit of time and cut a bit of frustration out of your daily routines.

Configure Interface - Cisco vs. Procurve

In Cisco IOS, you can switch from one interface from another real easy. The Interface command moves you from one interface to another without having to Exit every time you configure an interface.

With Procurve switch, you don't have the same kind of flexibility. Everytime you configure an interface, you would need to Exit out to Configuration mode before you can enter another interface.

Let look at this example on a Cisco switch:

Int g0/1
desc "IBA LAN"
Int g0/2
desc "ESV LAN"

That's perfectly legal on the Cisco IOS. Now configure that on a Procurve switch:

Int g0/1
name "IBA LAN"
exit
Int g0/2
name "ESV LAN"

If you miss Exit command, you'll received an "Invalid Input" error because you can not go to a different interface from within an interface. I never like this approach and prefer the shortcut where you enter two commands on one single line. Not only it cuts out the Exit command but keeps you in the Config mode.

Int g0/1 name "IBA LAN"
Int g0/2 name "ESV LAN"

Cisco Command Aliases

Alias command allows you to define aliases for long commands. Alias command breaks down into three modes:

Alias Exec for Privileged Mode.
Alias Configure for Global Configuration Mode.
Alias Interface for Interface Configuration Mode.

Let say that you want to shorten Show Running-Configuration command to just two keystrokes, you could define it as follow:

Router(Config)# alias exec sr Show Running-config

Or if you want to use ns to perform No Shut command on an interface you could define it as:

Router(Config)# alias interface ns no shutdown

So now instead of typing out all those long commands you could type sr for show runn or ns for no shut down and still achieve the same desired effects.

Cisco IOS includes some built-in command aliases. You can view these aliases by using the "show alias" command. Here are the default command aliases:

• h - help
• lo - logout
• p - ping
• r - resume
• s - show
• u - undebug
• un - undebug
• w - where

The Alias Command reminds me of the .bat files back in the DOS days when we used it to combine multiple commands together and shorten the filename to just a few keystrokes. If used effectively, this could saving you a lot of time and typing.

Show Running-Configuration

One of the most commonly used command when troubleshooting a switch or router is the "Show Run" command. It gives you an insight into the currently running configuration.

Sometimes, this command yields so much output one would be overwhelm with the amount of text on the screen and have to keep pressing the space key to scroll down to see more information.

Well, if you know what you are looking for then you could add a command prefix " begin keyword" and it'll start showing the configuration from that where the keyword occurs.

SwitchA# Show runn | begin spanning-tree

In the example above, the switch will show the current configuration from the line where it finds the first occurrence of the word "spanning-tree".

Using this command can really save you all the trouble of spacing through all the text only to find that you go too fast and miss the relevant information and have to scroll back to look for it.

Changing IP on Cisco/ProCurve Switch

Let's say we have the IP address 10.10.10.1/24 assigned to vlan 1 and want to change it to 10.10.10.2. On a Cisco switch, you can telnet in and send these commands to the switch:

conf t
interface vlan 1
ip address 10.10.10.2 255.255.255.0

The Cisco switch will update the IP address and disconnect your session. On a ProCurve switch, if you issue the same commands it will spit out an error saying:

"The IP address (or subnet) 10.10.10.2/24 already exists."

If you were going through the console port, you can set the new IP using the following commands:

conf t
vlan 1
no ip address 10.10.10.1/24
ip address 10.10.10.2/24

However, you can't do that if you connect to the switch remotely. As soon as the "no ip address" command is received and processed by the switch, your session will be disconnected and you won't be able to get to the switch.

The trick to get around this issue is to make this IP address change through the switch's built-in menu system instead of using the plain old CLI.

1. Type "menu", hit Enter
2. Select "Switch Configuration"
3. Select "IP Configuration"
4. Navigate to Edit, hit Enter
5. Change the IP and then Save

You will be disconnected once you save it but you will be able to reconnect using the new IP.

Routing Information Protocol (RIP)

Routing Information Protocol (RIP) at one time was one of the most commonly used routing protocols on small, internal network. With the availability of other routing protocol such as IGRP, EIGRP, OSPF, IS-IS, BGP... plus the lack of support for large network the usage of RIP has been decreased and it is considered obsolete. For this reason, it's often referred to as "Rest In Peace" protocol.

Nonetheless, Cisco and other networking vendors are still making RIP & RIPv2 available today in their IOS models. I like RIP for the fact that it's very simple and easy to setup. You can get a small network running RIP up in just a few minutes. RIP is also a very good choice of protocol to get one familiarized with routing protocols.

Besides its infamous Pros and Cons, one lesser known fact about RIP is that you can also use it to load balance your network. However, beware of the fact that RIP does not take bandwidth into consideration when it installs learned routes into the routing table.

So let's say you have two ways to send a packet from A to B and the first path is going through a 128k line while the second goes through a T1 line; RIP will treat both paths equally as if they have the same bandwidth.

Suggested Readings:

http://en.wikipedia.org/wiki/Routing_Information_Protocol
http://www.cisco.com/univercd/cc/td/doc/cisintwk/ito_doc/rip.htm

Smurf Attack

A smurf attack is a Denial of Service (DoS) attacked that uses spoofed broadcast ping messages to flood the target system with ping replies. The attack method gets its name after its exploit program.

In this attack, an attacker sends out a large number of ICMP Echo Request packets to IP broadcast addresses with all requests having a spoofed source address of the intended victim. Once the routing device receives this type of request, it will broadcast it to all hosts on its network. When the hosts reply, they will overwhelm the targeted's network with so much traffic that it will effectively shut out the network from the rest of the world.

Using a Cisco router, one can issue the command:

no ip directed-broadcast

to secure their network from this type of attack.

Note that this does not prevent a network from becoming the target of a smurf attack; it just prevent the network from taking part in an attack agains other networks.