Virtual Private Networks Troubleshooting - Part I

I've seen quite a number of posts asking why VPN users are not able to access their network even though they have been successfully connected.

Let's try to troubleshoot what happens here. A very common VPN setup is to use 2 NICs and run Routing and Remote Access on one of them. The benefits of this would be that you can isolate VPN traffic and keeping it from overloading the NIC that handles the internal network traffic. One thing to note for is that the two NICs have to be assigned IPs from different networks or subnets. For example:

1st NIC (VPN): 192.168.1.2 /24
2nd NIC(internal network): 192.168.2.2 /24

(Don't worry if you don't understand the /24. It's called the slash notation for the subnet mask which is the equivalent of 255.255.255.0)

Through the initial RRAS setup, users will be able to connect to the VPN just fine and will be assigned an IP address of one of the networks. The problem arise when they try to connect to network resources on through the 2nd NIC (the other network). This is due to the fact that Windows doesn't know how to route the packets from the 192.168.1.0 network to 192.168.2.0 network.

You can tell Windows how to route traffic between the two NICs by configuring a static route with the "Route" command. Type "Route /?" at the Command Prompt for more information.

And as usual, I have a second way to accomplish the job by turning on Routing using RIP or OSPF and add the two NICs to it.

One thing that I see lacking from most Windows Administrators is the understanding of TCP/IP and network routing. I would recommend Windows Administrators to take CCNA anytime. Even though it might seem so remote from their daily tasks, one can walk away from a CCNA class with so much helpful information that makes their job much easier to do.

Switch Port Security

If you are working in a strict security environment then switch port security is a must. Configuring switch port security could become a tidious task. However, if you can control the environment when you setup the network, this little trick can help you save a lot of work.

Instead of configuring port security and manually enter MAC address for the port, you could plug all your hosts in then issue the following commands:

Switch(config)#int range f0/1-xx
Switch(config-if-range)#switchport port-security
Switch(config-if-range)#switchport port-security maximum 1
Switch(config-if)#switchport port-security violation restrict
Switch(config-if-range)#switchport port-security mac-address sticky
Switch(config-if-range)#end

The first command takes you to the interface range configuration mode; the next two turn on the port security and set a maximum number of mac addresses to 1. "Violation restrict" will not allow traffic for any host whose mac address is different than what the switch has learned for the port in question. After that, the "mac-address sticky" commands instruct the switch to learn the mac address dynamically and remembers it for the each port.

Basic Load Balancing

Load Balancing normally has two modes of operation: Per-Destination and Per-Packet.

Per destination load balancing means the router distributes the data packets based on the destination address. If you have two paths going to Host A & B on the same network then all packets for Host A will travel over the first path and all packets for Host B will travel over the second path. This will preserve the packet order which is very useful in certain applications; however, it could result in unequal usage of the network links as bandwidth & load are not taken into route calculation.

Per packet load balancing means the router sends one packet over the first path and second packet over the second path; all going to the same destination. Per packet load balancing guarantees equal load across all the links; however, the packets may arrive at the destination out of order because of different delay/bandwidth may exist on different paths.

Per destination load balancing is enable by the command:

Router# config t
Router(config)# interface Ethernet 0
Router(config-if)# no ip route-cache

Now the router CPU will look at every single packet and spread them across the different path available in the routing table for the destination. This is not recommended on low end server as it could crash the router because the CPU must do all the processing and might not be able to handle it. To enable fast switching, use the following commands:

Router# config t
Router(config)# interface Ethernet 0
Router(config-if)# ip route-cache

Newer switching schemes such as Cisco Express Forwarding (CEF) allow you to do per packet and per destination load balancing more quickly but it does imply that extra resources will be needed to maintain it.

Reload

Working remotely with routers & switches you don't have the flexibility to reset the devices at will when things go wrong unless you can control power distribution to the devices.

This is where the Reload command comes in handy. Before making changes to the configuration, you can enter:

Reload in 10

which will reload the device in 10 minutes. Once the command is entered, you can proceed with the configuration changes. If things go wrong and you lost connectivity to the device, try back in about 10 minutes since the device will get reloaded with the original startup-configuration and restore connectivity.

Once you confirm that your new configuration is working properly, you can issue the command:

Reload cancel

to cancel the reload. I find this trick to be very useful when I have to telnet or SSH to a remote device and hopefully you too.

Add Redundancy With Backup Command

You can add redundancy to your network and make you network more resilient with the Backup command. What this command does is specify an interface which will act as a back up in case the primary interface fails.

Let's take a look at the following configurations:

On RouterA:

Configure terminal
Interface f0/0
Ip address 192.168.1.1 255.255.255.0
no shutdown
Interface f0/1
Ip address 192.168.2.1 255.255.255.0
no shutdown
Backup interface f0/0
Exit
Router eigrp 20
network 192.168.1.0
network 192.168.2.0

On RouterB:

Configure terminal
Interface f0/0
Ip address 192.168.2.2 255.255.255.0
no shutdown
Interface f0/1
Ip address 192.168.2.2 255.255.255.0
no shutdown
Backup interface f0/0
Interface loop 0
Ip add 192.168.8.1 255.255.255.0
Exit
Router eigrp 20
network 192.168.1.0
network 192.168.2.0
network 192.168.8.0

With these configurations loaded, both Fa0/1's will change to up/up while Fa0/0's will go into standby mode and monitor the activities on Fa0/1. In the event that Fa0/1 goes down, Fa0/0 will switch to active mode and establish neighbor relationship with the connected interface and complete the failover.

To verify this, from RouterA you can ping 192.168.8.1 with the repeat parameter:

Ping 192.168.8.1 repeat 10000

This command will ping 192.168.8.1 10k times. While it's pinging, unplug the cable to Fa0/1. you will see that the link go down, the ping will fail and then within seconds Fa0/0 will be up and resume the active role and the pinging will resume as nothing has happened. If the link on Fa0/1 becomes active again, Fa0/0 will switch back to standby mode.

This is a quick and easy way to add redundancy to your network. There are other techniques that you can use to add redundancy to your network to include both layer 2 and layer 3 redundancies which I will discuss at another opportunity.

Keyboard Shortcuts

If you use CLI to manage your Cisco routers & switches then you should to get familiarized with the keyboard shortcuts as the can save you a lot of time & typings.

Here's the shortcut list:

• Ctrl+B: Moves back one character at a time
• EscB: Moves back one word at a time
• Ctrl+A: Moves to the start of a line
• Ctrl+F: Moves forward one character at a time
• Esc+F: Moves forward one word at a time
• Ctrl+E: Moves to the end of a line
• Delete: Erases the character to the left of the cursor
• Backspace: Erases the character to the left of the cursor
• Ctrl+D: Deletes the character at the cursor
• Ctrl+K: Deletes all characters from the cursor to the end of the line
• Ctrl+U: Deletes all characters from the cursor to the beginning of the line
• Ctrl+X: Deletes all characters from the cursor to the beginning of the line
• Ctrl+W: Deletes the word to the left of the cursor
• Esc+D: Deletes from the cursor to the end of a word

If none of these commands work, enhanced editing may be turned off. You can turn on enhanced editing issuing the command at the privileged command prompt:

Router# terminal editing

Stop Logging Messages From Interrupting Your Work

One of the things that I find really annoying with the Cisco devices is that when I am configuring a router or a switch, the console messages keep popping up all over the places and interfere with my work. Although this doesn't cause any typo error but it's really difficult to see what you have typed. Luckily, we can control the logging message behavior.

To disable, use:
Router(config)# no logging console

or if you prefer to see what's going on with the router/switch then use the "logging synchronous" command:
Router(config)# line console 0
Router(config-line)# logging synchronous
Router(config)# line vt 0 4
Router(config-line)# logging synchronous

It would be nice if these were taken into consideration and set as default options when the IOS is built but it's not the case so until they change it, you will have to either put up with it or do a little extra work to change the default behavior.

Configure DHCP on a Cisco Router or Switch

When it comes to configuring DHCP, you have a number of ways to do it. You can run DHCP service on a Windows or a Linux box or as most networking professionals prefer, run it on an existing router or a switch.

The advantage of this is that you can leverage existing hardware, provide DHCP locally at each site and reduce the dependency on the WAN link.

To configure DHCP on a Cisco router or switch, follow these steps:

1. Configure an IP address on the router's Ethernet port

Router(config)# interface e0/0
Router(config-if)#ip address 172.16.1.1 255.255.255.0
Router(config-if)# no shut

2. Create a DHCP IP address pool for the IP addresses you want to use.

Router(config)# ip dhcp pool dhcppoolname

3. Specify the network and subnet for the addresses you want to use from the pool.

Router(dhcp-config)# network 172.16.1.0 255.255.255.0

4. Specify the DNS domain name for the clients.

Router(dhcp-config)#domain-name yourdomain.com

5.Specify the DNS server.

Router(dhcp-config)#dns-server 172.16.1.2

6. Specify the defaut router (or default gateway)

Router(dhcp-config)#default-router 172.16.1.1

7. Specify the lease duration.

Router(dhcp-config)#lease 7

8. Let's say that you've decided that all IP addresses between .30 and .40 will be used for static IP devices such as mail servers and database servers and you don't want them to be available in the DHCP pool, you can exclude them using "exclude-address" command:

Router(config)#ip dhcp excluded-address 172.16.1.30 172.16.1.40

Now any device that requires an DHCP address on the 172.16.1.0 network will be able to send a DHCP request to 172.16.1.1.

Start a Port Configuration In a Clean State

If you are reconfiguring or changing the port settings on a router you can remove or change the old commands one by one until the job is done. However, there's a "default interface" command that you can use to quickly reset the port's configuration to its default values:

Router(config#): default interface fa0/1

Once the "default interface"command issued, you can be certain that all previous commands set on the specified ports are gone.

Protecting Your Network Edge with TTL

Protecting your network by authenticating every EBGP session is a good measure to keep the bad guys out. While this approach can keep attackers from getting inside your network, it can't keep them from launching attacks by flooding packets at BGP port (TCP port 179) and causing tremendous authentication failures which can lead to legistimate BGP connections at the target interface to fail, BGP process failure or even system crash due to spike in system resources & CPU spike.

Fortunately, you can use TTL (Time To Live) as an additional measure to reduce such DoS attacks against BGP. The default behavior of most BGP implementations is that it will send packets to external neighbors with a TTL value of 1 and accepts packets from external neighbors with TTL of 0 or higher.

You can tighten up your network by changing this default behavior by having BGP originated packets with a TTL value of 255 and only accepting packets with TTL of 254 (measured after the local router has decremented the TTL of the packet it received) or higher.

Now even if the attackers originate packets with a maximum TTL value of 255, the packets can't get to your BGP network if the attackers are not directly connected to the target interface. This is because each router hop decrements the TTL value by 1. If they are two hops away, by the time the packets reach the port TTL will be decremented to 253 and will be rejected.

You can configure the TTL value by issuing the statement:

neighbor-address ttl-security hops 254

This will instruct your router to discard incoming BGP packets with TTL value below 254 and also set outgoing packets TTL value to 255. You will need to make sure that both neighbors are configure with the same statement.

The Day The Routers Die

Those of us that are in the networking field might find this video amusing.

Note: In the song, the performer (Gary Feldman) mentioned RIPE55 quite a few times. RIPE stands for Réseaux IP Européens (European IP Networks) which is a forum open to all parties who are interested in the technical development of internet and 55 is the meeting number during which he performed the song.

Gary sung in native English accent so it might be a bit hard to understand. Here's a to link the lyrics.

Configure DNS or Disable DNS lookup

If you mistype a command in the Privileged Mode on Cisco router, by default, the router thinks you're trying to connect to a remote host through Telnet. So it performs a DNS lookup on the information you entered.

If you haven't configured DNS on the router, the command prompt will hang until the DNS lookup timeouts. This is one of the thing that really bothers me so to resolve it, I have two options:

The first option is to disable DNS using the "domain-lookup" command as in:

Router(config)# no ip domain-lookup

The second option is to properly configure DNS and point it to a valid DNS server. Here's an example:

Router(config)# ip name-server 4.2.2.1

By configure the router using one of the two options above it will save you a bit of time and cut a bit of frustration out of your daily routines.

Configure Interface - Cisco vs. Procurve

In Cisco IOS, you can switch from one interface from another real easy. The Interface command moves you from one interface to another without having to Exit every time you configure an interface.

With Procurve switch, you don't have the same kind of flexibility. Everytime you configure an interface, you would need to Exit out to Configuration mode before you can enter another interface.

Let look at this example on a Cisco switch:

Int g0/1
desc "IBA LAN"
Int g0/2
desc "ESV LAN"

That's perfectly legal on the Cisco IOS. Now configure that on a Procurve switch:

Int g0/1
name "IBA LAN"
exit
Int g0/2
name "ESV LAN"

If you miss Exit command, you'll received an "Invalid Input" error because you can not go to a different interface from within an interface. I never like this approach and prefer the shortcut where you enter two commands on one single line. Not only it cuts out the Exit command but keeps you in the Config mode.

Int g0/1 name "IBA LAN"
Int g0/2 name "ESV LAN"

Cisco Command Aliases

Alias command allows you to define aliases for long commands. Alias command breaks down into three modes:

Alias Exec for Privileged Mode.
Alias Configure for Global Configuration Mode.
Alias Interface for Interface Configuration Mode.

Let say that you want to shorten Show Running-Configuration command to just two keystrokes, you could define it as follow:

Router(Config)# alias exec sr Show Running-config

Or if you want to use ns to perform No Shut command on an interface you could define it as:

Router(Config)# alias interface ns no shutdown

So now instead of typing out all those long commands you could type sr for show runn or ns for no shut down and still achieve the same desired effects.

Cisco IOS includes some built-in command aliases. You can view these aliases by using the "show alias" command. Here are the default command aliases:

• h - help
• lo - logout
• p - ping
• r - resume
• s - show
• u - undebug
• un - undebug
• w - where

The Alias Command reminds me of the .bat files back in the DOS days when we used it to combine multiple commands together and shorten the filename to just a few keystrokes. If used effectively, this could saving you a lot of time and typing.

Show Running-Configuration

One of the most commonly used command when troubleshooting a switch or router is the "Show Run" command. It gives you an insight into the currently running configuration.

Sometimes, this command yields so much output one would be overwhelm with the amount of text on the screen and have to keep pressing the space key to scroll down to see more information.

Well, if you know what you are looking for then you could add a command prefix " begin keyword" and it'll start showing the configuration from that where the keyword occurs.

SwitchA# Show runn | begin spanning-tree

In the example above, the switch will show the current configuration from the line where it finds the first occurrence of the word "spanning-tree".

Using this command can really save you all the trouble of spacing through all the text only to find that you go too fast and miss the relevant information and have to scroll back to look for it.

Changing IP on Cisco/ProCurve Switch

Let's say we have the IP address 10.10.10.1/24 assigned to vlan 1 and want to change it to 10.10.10.2. On a Cisco switch, you can telnet in and send these commands to the switch:

conf t
interface vlan 1
ip address 10.10.10.2 255.255.255.0

The Cisco switch will update the IP address and disconnect your session. On a ProCurve switch, if you issue the same commands it will spit out an error saying:

"The IP address (or subnet) 10.10.10.2/24 already exists."

If you were going through the console port, you can set the new IP using the following commands:

conf t
vlan 1
no ip address 10.10.10.1/24
ip address 10.10.10.2/24

However, you can't do that if you connect to the switch remotely. As soon as the "no ip address" command is received and processed by the switch, your session will be disconnected and you won't be able to get to the switch.

The trick to get around this issue is to make this IP address change through the switch's built-in menu system instead of using the plain old CLI.

1. Type "menu", hit Enter
2. Select "Switch Configuration"
3. Select "IP Configuration"
4. Navigate to Edit, hit Enter
5. Change the IP and then Save

You will be disconnected once you save it but you will be able to reconnect using the new IP.

Cisco Switch - TFTP IOS Upgrade

Update, Copy, Backup, Restore IOS and startup-config are pretty much done in the same manner. First, you've got to configure the switch and the PC to talk then run the TFTP or FTP server and start the copy process.

Let's walk though how you can update Cisco IOS on a 6500.

Router> enable
Router# configure terminal
Router(config)# interface gig 9/1
Router(config-int)# switchport
Router(config-int)# switchport mode access
Router(config-int)# switchport access vlan 1
Router(config-int)# no shutdown
Router(config-int)# exit

At this point, we have turn port 9/1 into a layer 2 port on vlan 1. Now we need to assign an ip address to this default vlan so we can communicate to it.

Router(config)# interface vlan 1
Router(config-int)# ip address 10.10.10.1 255.255.255.0
Router(config-int)# no shutdown
Router(config-int)# exit

The next thing we need to do is to configure the network adapter on your PC or laptop and give it an address that's on the same network as the switch. In this case, I set it to be 10.10.10.2 255.255.255.0.

Plug in the cable from the PC to the port we just configured; and to make sure the PC can talk to the 6500 we can try pinging it:

Router(config)# do ping 10.10.10.2

If you have a successful ping reply then you should be good to go. Launch the TFTP server and start the copy process. To copy from data from TFTP server to the switch use:

Router# copy tftp: disk0:

After you issue either of this command, you will be prompted for the FTFP server address and filename. Enter 10.10.10.2 and the filename you want to copy, then confirm it to start the copy process.

And to copy data from flash to FTFP server use the command:

Router# copy disk0: tftp:

If you are copying a rather large file then you should use FTP instead of TFTP and issue the command:

Router# copy FTP: Disk0:

I have run into some issue while moving a large file to the Cisco 6509. See this blog for more details.

Cisco TFTP - Flash Update Problem

I was upgrading the IOS for the Cisco 6509s. I got Cisco TFTP Server setup and connected to it to download the file. The file was around 80Mb. TFTP Server threw an exception and shutdown. Upon restart, the file started to get downloaded but a few seconds later Cisco TFTP Server threw another error. This time I was able to view the error message and it said the transfer failed due to Synchronization Error.

I switched to SolarWinds TFTP Server and got a more detailed error saying the file was too large to be transferred through TFTP.

As an alternative, I used FTP Server from 3CDaemon and issued the command:

copy FTP: Disk0:

and it worked like a champ.

I had IIS running on my laptop so I turned off 3CDaemon and used IIS's FTP server. It worked out just fine.

So if you are transferring a large file you should use FTP and not TFTP to avoid potential problems and I also recommend 3CDaemon as it's got TFTP Server, FTP Server, Syslog as well as TFTP client all in one package.

OSPF - Wild Card Mask

Open Shortest Path First (OSPF) is an open standard, internal routing protocol, capable of supporting large network spanning multiple sites. Routers running OSPF are aware of their network topology; they get update of changes in the network through HELLO packets.

Some of OSPF's strengths include:

It converges quickly, compared to distance-vector protocol.
Routing update packets are small, and only reflect the change (it doesn't send the whole routing table).
It's not prone to routing loops.
It scales very well for large networks.
Bandwidth is taken into account when it selects a link to install in the routing table.
It support Variable-Length Subnet Masks (VLSM)

While many people get confused by the Wild Card Mask that is required in the Network statement, I find it's not that difficult at all.

Let's say you need to advertise the network 172.30.105.84 255.255.255.252 and wants to find the wild card mask. All you have to do is take 255 and subtract each octet of the subnet mask. So in this case you would advertise the network as:

Router(config)# router ospf {process number}
Router(config-router)# network 172.30.105.84 0.0.0.3 area 0

Where do you get the 0.0.0.3? Using the rule above, you take 255 - 255 = 0 (in the first 3 octets) and 255 - 252 = 3 (in the last octet). See, it's not that hard to find the wild card mask at all.

Where do I get 255 from, you ask? Each octet is made up of 8 bits and 255 is the maximum value that 8 bits can hold.

Here are some common OSPF commands that are very useful in checking the status of OSPF configuration:

show ip ospf
show ip ospf neighbor
show ip ospf interface
show ip route ospf

Further readings:
http://www.cisco.com/univercd/cc/td/doc/product/software/ios124/124cr/hirp_r/rte_osph.htm#wp1001842
http://en.wikipedia.org/wiki/Open_Shortest_Path_First

Updating IOS on ProCurve Switches

Updating IOS software on HP's ProCurve switches are pretty simple and straight forward. All you need to do is run a TFTP server, log into the switch, add an IP to the switch, specified a default gateway so the switch can talk to the TFTP server and download the new IOS.

SwitchA# vlan 1
SwitchA# ip address 192.168.1.9 255.255.255.0
SwitchA# exit
SwitchA# ip default-gateway 192.168.1.1
SwitchA# copy tftp flash 192.168.1.8 filename.swi

Note that when you do this the switch will download the new software and reboots itself when it's done processing the file. If you need to update both primary and secondary flash then you should find out whether the switch boots from primary flash or secondary flash and copy the other one first.

Let's say the switch boots from primary flash then the sequence of commands should be:

SwitchA# copy tftp flash 192.168.1.8 filename.swi secondary
SwitchA# y
SwitchA# copy tftp flash 192.168.1.8 filename.swi
SwitchA# y

This will let you update the IOS on both primary and secondary flashes before the switch reboots itself and save you some time. Plus you can script this and cut down the update time. If you do it otherwise, the switch will update primary flash, reboot and then you have to manually update the secondary flash.