Troubleshooting

This chapter provides information on troubleshooting the Cisco network-based IPSec VPN solution Release 1.5. It provides links to information on troubleshooting solution components as well as IPSec. It also provides solution-specific troubleshooting.

Troubleshooting Solution Components

Cisco 7200 Series Internet Router Troubleshooting

RADIUS Troubleshooting

Troubleshooting Customer Premise Equipment

This section provides references to troubleshooting information for the following customer premises equipment used with the Network-Based IPSec VPN:

Troubleshooting IP Solution Center Version 3.0

Troubleshooting Cisco IOS Software

Troubleshooting Cisco VPNS

Troubleshooting IPSec

Troubleshooting IPSec-related connectivity problems is fairly straightforward if you use a logical and methodical approach.

Table 3-1 Customer Premise Equipment Troubleshooting Information
Customer Premise Equipment	Upgrade Information
Cisco VPN 3002 hardware client	See the information at: http://www.cisco.com/en/US/products/hw/vpndevc/ps2286/products_user_guide_chapter09186a00800bcd46.html.
Cisco 800 series routers	See the information at: http://www.cisco.com/univercd/cc/td/doc/product/access/acs_fix/800/800hwins/trouble.htm.
Cisco 1700 series routers	See the information at: http://www.cisco.com/en/US/products/hw/routers/ps221/prod_alerts_troubleshooting.html
Cisco 2600 series routers	See the information at: http://www.cisco.com/warp/public/108/hwts_2600_16138.html.
Cisco 3600 series routers	See the information at: http://www.cisco.com/en/US/products/hw/routers/ps274/prod_alerts_troubleshooting.html.
Cisco 7200 series routers	See the information at: http://www.cisco.com/warp/public/63/hwts_7200_16122.html.

•

Router receives interesting traffic, which is traffic that must be encrypted.

•

Router negotiates and sets up an IKE tunnel with the remote peer specified in the configuration.

•

Router negotiates IPSec tunnels (one for the inbound direction and one for the outbound) with the remote peer.

ISAKMP and IPSec Troubleshooting Commands

Note As with all other debug commands, ISAKMP and IPSec debug commands consume CPU cycles and produce copious output.

The ISAKMP and IPSec troubleshooting commands provide detailed information about IPSec failure. See Appendix A, "Troubleshooting Commands," for more examples of these commands as well as the information they provide.

debug crypto engine

The following is sample output from the debug crypto engine command. The first sample output shows messages from a router that successfully generates RSA keys. The second sample output shows messages from a router that decrypts the RSA key during Internet Key Exchange (IKE) negotiation.

debug crypto isakmp

Use this command to view to see the Internet Security Association and Key Management Protocol (ISAKMP) phase 1 negotiations.

The following is sample output from the debug crypto isakmp command for an IKE peer that initiates an IKE negotiation. First, IKE negotiates its own security association (SA), checking for a matching IKE policy:

IKE has found a matching policy. Next, the IKE SA is used by each peer to authenticate the other peer:

Next, IKE negotiates to set up the IPSec SA by searching for a matching transform set:

A matching IPSec transform set has been found at the two peers. Now the IPSec SA can be created (one SA is created for each direction):

debug crypto ipsec

The following is sample output from the debug crypto ipsec command. In this example, security associations (SAs) have been successfully established.

IPSec requests SAs between 172.21.114.123 and 172.21.114.67, on behalf of the permit ip host 172.21.114.123 host 172.21.114.67 command. It prefers to use the transform set esp-des w/esp-md5-hmac, but it also considers ah-sha-hmac.

IKE asks for SPIs from IPSec. For inbound security associations, IPSec controls its own SPI space.

IKE asks IPSec if it accepts the SA proposal. In this case, it will be the one sent by the local IPSec:

After the proposal is accepted, IKE finishes the negotiations, generates the keying material, and then notifies IPSec of the new security associations (one security association for each direction).

The following output pertains to the inbound SA. The conn_id value references an entry in the crypto engine connection table.

The following is sample output for the debug crypto ipsec command as seen on the peer router. In this example, IKE asks IPSec to accept an SA proposal. Although the peer sent two proposals, IPSec accepts the first proposal.

IKE performs the remaining processing, completing the negotiation and generating keys. It then tells IPSec about the new SAs.

show crypto ipsec sa

The encrypted tunnel is built between 12.1.1.1 and 12.1.1.2 for traffic traveling between networks 20.1.1.0 and 10.1.1.0. You can view the two Encapsulating Security Payload (ESP) SAs built inbound and outbound. Authentication Header (AH) is not used because there are no AH SAs. Below is an example of the show crypto ipsec sa command.

show crypto engine connection active

Note Remember that phase 2 SAs are unidirectional, so each SA shows traffic in one direction only (encryptions are outbound, decryptions are inbound).

IPSec Troubleshooting Strategy

Use the following commands to clear established sessions and regenerate debugs:

10.

Use the following commands to clear established sessions and regenerate debugs:

Hardware Accelerator Card Problems

Determine if the router is using a hardware accelerator (encryption) card by issuing the following command:

If the hardware card is not being used, the `crypto engine type' is reported as `software,' otherwise, it reports the actual card type in use, such as ISA/ISM.

If the hardware card is being used, it might sometimes lapse into a limbo state and cause problems with establishment of tunnels. You can disable the card to determine if this is the case by issuing the following command:

You can then recheck for the tunnels. If the tunnels resume operation successfully, the problem resides with the accelerator card.

If you continue to experience problems with the card, you can try reloading the router. If the problem persists, try swapping out the card for a new one.

Note Each time you disable or enable the accelerator card, all existing crypto connections are torn down.

The following commands can be used to obtain statistics from the hardware card for the 7000 platforms and other platforms, respectively:

Questions for Effective Troubleshooting

IPSec connectivity fails if problems occur during any of the stages listed above. To effectively troubleshoot connectivity problems, consider the key questions in the following sections.

Is the router receiving and recognizing interesting traffic?

The number of matches for the access list should increment for each interesting data packet; if the number of matches is increasing, you can skip the following step.

Step 2

If the number of matches is not increasing, check to make sure that the source interface for the traffic is operational by using the following command:

If the interface is operational, check the `match address' section of your crypto map to make sure that the access list specified there indicates the correct source and destination addresses / networks.

Step 3

Next, check to make sure that the interesting traffic is being routed to the correct interface by issuing the following command:

If the interesting traffic is not being routed to the correct interface, configure the required route entries.

Step 4

Finally, verify that the crypto map is being applied to the outbound interface by issuing the following command:

Does the router successfully negotiate an IKE tunnel with its peer?

When the router sees interesting traffic, it attempts to negotiate an IKE tunnel with the remote peer specified in the `set peer' statement of the crypto map.

If output similar to either of the following examples appears, the peers are still attempting to negotiate IKE:

Step 2

If the IKE does not become active, ping to make sure that the remote peer (tunnel endpoint) is reachable. (Remember that the tunnel endpoints must be routable addresses). If you are unable to ping the remote peer, check your routing tables by issuing the following command:

Step 3

Configure any necessary routes. Check the status of all intervening interfaces along the routing path, and make sure that they are all operational.

Step 4

If the remote peer is reachable, check to make sure that the ISAKMP policies are the same on both peers. Issue the following command on both peers:

Step 5

The encryption algorithm, hash algorithm, authentication method, ISAKMP lifetimes and Diffie-Helman group must match each other. Correct the configuration if you see discrepancies. If preshared keys are being used, such as the authentication method, the keys must be the same on both peers. Check this by issuing the following command:

Step 6

Compare the ISAKMP policies manually on the two peers to identify discrepancies. An alternate method of identifying problems is referencing error messages that are logged to the syslog by such discrepancies. (It is a good idea to keep the syslog turned on).

Step 7

Another important safeguard is checking to make sure that the ISAKMP packets are not blocked. To implement this procedure, simply run debugs on both peers and check to see if there is any activity.

Does the router successfully negotiate IPSec tunnels with its peer?

Step 1

When the IKE is operational, IPSec tunnels (one for inbound and one for outbound traffic) are negotiated. To check, issue the following command:

If the IPSec tunnels are active, you should see output similar to the following:

Step 2

The numbers in the `encrypted' and `decrypted' columns must increment continuously depending upon whether the router is encrypting packets, decrypting packets, or both. If this is not happening, check to verify that there is at least one matching crypto IPSec transform on the two peers by issuing the following command:

Step 3

There must be at least one transform common to both peers. The first transform that matches (not necessarily the strongest common transform) is used to build the IPSec tunnels. If there is no transform common to both peers, IPSec tunnels do not come up. Be sure to configure transforms correctly.

Step 4

You can also display detailed information and statistics for the IPSec tunnels by issuing the following command:

The information provided by this command might be useful for more in-depth troubleshooting. The output from this command will be similar to the following example:

Sample IPSec Debug

Common IPSec Error Messages

Table 3-2 shows some common IPSec error messages, reasons for them, as well as possible solutions.

Table 3-2 Common IPSec Error Messages
Message	Reason	Solution
`invalid local address 1.2.3.4 no IPSEC cryptomap exists for local address 1.2.3.4`	The specified address is not one of our local IPSec addresses.	Check to see if the outgoing interfaces address has been specified on the peer as peer id. Check peer id address for error. Use the crypto map local address command.
`proxy identities not supported`	The flow identities do not match a crypto map ACL permit statement.	The address being pinged is not included in the match address access-list. Change the access-list. Use reflected access-lists on the two ends of the tunnel.
`atts are not acceptable. Next payload is 0` `no offers accepted!` `SA not acceptable!` `%CRYPTO-6-IKMP_MODE_FAILURE: Processing of Main Mode failed with peer at 155.0.0.1`	The polices and configurations on the two peers do not match.	Check to make sure that the policies configured on the two ends of the tunnel are correct and correlate.
`reserved no zero on payload 5!` `%CRYPTO-4-IKMP_BAD_MESSAGE: IKE message from 155.0.0.1 failed its sanity check or is malformed`	The encryption keys on the two ends do not match.	Check to make sure that the preshared keys are correctly configured. Regenerate RSA encrypted keys.
`Rec'd packet not an IPSEC packet.` `(ip) dest_addr= 1.2.3.4, src_addr= 2.3.4.5, prot= 1`	An incoming packet matched the match address access list but was not encrypted. Due to security concerns such packets are dropped.	Check to make sure that the routing between the two routers is correct (that is, packets are being sent between the interfaces doing encryption). Verfiy NAT is not occurring for IPSEC traffic.
		Do not use match address access list with any any. Turn off fast switching. Some Cisco IOS software versions issue a bug if partial subnets are configured in the match address access list.
`ISAKMP (0:1): SA is still budding. Attached new ipsec request to it.`	Indicates IPSec has sent another request to IKE to create SAs on its behalf; however, IKE is in the process of fulfilling a similar request. Thus, IKE attaches the new request to the SA-negotiation already in progress.	This is an informational message and does not necessarily signify an error.
`ISAKMP (0:1): deleting node-45086069 error FALSE [reason]`	Indicates that a memory structure that is not needed has been cleared during normal housekeeping. The error False portion of this message indicates that an error has not occurred; an error True portion of this message would indicate that a memory structure that is not needed has been cleared, but an error has occurred.	—
`%CRYPTO-4-RECVD_PKT_INV_SPI: decaps: rec'd IPSec packet has invalid spi for destaddr=16.0.0.3, prot=50, spi=0xdeadbeef.`	Indicates that the IPSec SAs between router A and router B are out of sync. This can occur if router A has cleared its IPSec SAs but router B has not. Thus, when router B attempts to send encrypted traffic corresponding to the SAs, router A drops the packet and reports this message.	This is an informational message and does not necessarily signify an error.
`ISAKMP (0:2): deleting SA reason "P1 delete notify (in)" state (I) CONF_ADDR (peer 10.13.1.21) input queue 0`	Indicates that the IKE SA with the connection ID 2 has been deleted. P1 delete notify indicates that the router received a phase 1 delete notification from its peer. The (I) (in 'state') means that the router was the initiator for this SA; an (R) would mean that the responder was the initiator. This message also indicates the peer IP address.	—
`ISAKMP (0:2): Notify has no hash. Rejected`	Indicates that the notify message received from the peer lacked a valid hash. This means that the notify message was not authenticated. For security reasons, this message is ignored.	—
`IPSEC(validate_proposal): invalid local address 12.2.6.2` `ISAKMP (0:3): atts not acceptable. Next payload is 0` `ISAKMP (0:3): SA not acceptable!`	This error message is attributed to one of the following two common problems: •The crypto map map-name local-address interface-id command causes the router to use an incorrect address as the identity because it forces the router to use a specified address. •Crypto map is applied to the incorrect interface or, it is not applied at all.	Check the configuration to ensure that crypto map is applied to the correct interface.
`1d00H:%CRPTO-4-IKMP_BAD_MESSAGE: IKE message from 150.150.150.1 failed its` `sanity check or is malformed`	The preshared key on the peers do not match.	Check the preshared key on both sides.
`1d00h: ISAKMP (0:1): atts are not acceptable. Next payload is 0` `1d00h: ISAKMP (0:1); no offers accepted!` `1d00h: ISAKMP (0:1): SA not acceptable!` `1d00h: %CRYPTO-6-IKMP_MODE_FAILURE: Processing of Main Mode failed with` `peer at 150.150.150.1`	This is an example of the Main Mode error message. The failure of Main Mode suggests that the phase I policy does not match on both sides.	Verify that the phase I policy is on both peers and ensure that all the attributes match, for example: `Encryption DES or 3DES` `Hash MD5 or SHA` `Diffie-Hellman Group 1 or 2` `Authentcation {rsa-sig \| rsa-encr \| pre-share`
`1d00h: IPSec(validate_transform_proposal): proxy identities not supported` `1d00h: ISAKMP: IPSec policy invalidated proposal` `1d00h: ISAKMP (0:2): SA not acceptable!`	The message appears in debugs if the access-list for IPSec traffic does not match.	The access-list on each peer should mirror each other (all entries should be reversible). See below: `Peer A` `access-list 150 permit ip 172.21.113.0 0.0.0.255 172.21.114.0 0.0.0.255` `access-list 150 permit ip host 15.15.15.1 host 172.21.114.123` `Peer B` `access-list 150 permit ip 172.21.114.0 0.0.0.255 172.21.113.0 0.0.0.255` `access-list 150 permit ip host 172.21.114.123 host 15.15.15.1`
`1d00h: IPSec (validate_proposal): transform proposal (port 3, trans 2, hmac_alg 2) not supported` `1d00h: ISAKMP (0:2) : atts not acceptable. Next payload is 0` `1d00h: ISAKMP (0:2) SA not acceptable`	Indicates the Phase II (IPSec) does not match on both sides.	Occurs if there is a mismatch in the transform-set. Verify that the transform-set matches on both sides: `crypto ipsec transform-set transform-set-name transform1` `[transform2 [transform3]]` `? ah-md5-hmac` `? ah-sha-hmac` `? esp-des` `? esp-des and esp-md5-hmac` `? esp-des and esp-sha-hmac` `? esp-3des and esp-md5-hmac` `? esp-3des and esp-sha-hmac` `? comp-lzs`
`1d00h: ISAKMP: No cert, and no keys (public or pre-shared) with` `remote peer 150.150.150.2`	Indicates that the peer address configured on the router is incorrect or has changed.	Verify that the peer address is correct and reachable.
`20:44:44: IPSEC(validate_proposal_request): proposal part #1,` `(key eng. msg.) dest= 194.70.240.150, src= 198.174.236.6,` `dest_proxy= 10.0.0.76/255.255.255.255/0/0 (type=1),` `src_proxy= 198.174.238.203/255.255.255.255/0/0 (type=1),` `protocol= ESP, transform= esp-3des esp-md5-hmac ,` `lifedur= 0s and 0kb,` `spi= 0x0(0), conn_id= 0, keysize= 0, flags= 0x4` `20:44:44: IPSEC(validate_transform_proposal): peer address 198.174.236.6 not found`	Normally appears with the corresponding Cisco VPN 3000 concentrator message: No proposal chosen(14). This is a result of the connections being host-to-host. The router configuration had the IPSec proposals ordered so that the proposal chosen for the router matched the access-list, but not the peer. The access-list had a larger network that included the host that was intersecting traffic.	Make the router proposal for this concentrator-to-router connection first in line, so that it matches the specific host first.
`21:57:57: IPSEC(validate_proposal_request): proposal part #1,` `(key eng. msg.) dest= 192.1.1.1, src= 192.1.1.2,` `dest_proxy= 10.1.1.1/255.255.255.0/0/0 (type=4),` `src_proxy= 20.1.1.1/255.255.255.0/0/0 (type=4)`	The debug command output of this proposal request shows the corresponding access-list 103 permit ip 10.1.1.0 0.0.0.255 20.1.1.0 0.0.0.255 does not match. The access-list is network specific on one end and host specific on the other.	Match access lists.
`21:57:57: IPSEC(initialize_sas): invalid proxy IDs`	Indicates that the received proxy identity does not match the configured proxy identity as per the access-list.	Check to ensure that the received proxy identity and the configured proxy identity both match by checking the output from the debug command.

Common IPSec Issues

Mismatch in Phase 1 Policy Parameters

No Preshared Key for the Peer

Incorrect Preshared Key

No Matching Peer

Mismatched Transform Sets

Crypto Map Not Applied to Interface

Incorrect Access List—Mismatched Proxies

Nonmatching VPN Group for VPN Clients

Failed Authentication—XAUTH for VPN Clients

Incorrect Pool Definition for VPN Clients

Incompatible ISAKMP Policy or Preshared Secrets

If no ISAKMP policies configured match, or if no preshared key for the negotiating peer is configured, the router tries the default policy (65535); if that does not match it fails ISAKMP negotiation.

A show crypto isakmp sa command shows the ISAKMP SA to be in MM_NO_STATE, meaning the main-mode failed:

If the preshared secrets are not the same on both sides, the negotiation fails again, with the router complaining about sanity check failed. A show crypto isakmp sa command shows the ISAKMP SA to be in MM_NO_STATE, meaning the main mode failed:

Incompatible or Incorrect Access Lists

If the access lists on the two routers do not match or at least overlap, INVALID PROXY IDS or PROXY IDS NOT SUPPORTED results. It is recommended that access lists on the two routers be `reflections' of each other. It is also recommended that you do not use the key word "any" in match address access lists.

Crypto Map on Incorrect Interface

The crypto map needs to be applied to the outgoing interface of the router. If you do not want to use the outside interface's IP as the local ID, use the crypto map local address command to specify the correct interface.

If there are physical as well as logical interfaces involved in carrying outgoing traffic, you must apply the crypto map to both.

Incorrect SA Selection by the Router

If there are multiple peers to a router, make sure that the match address access lists for each of the peers are mutually exclusive from the match address access list for the other peers.

If this is not done, the router chooses the incorrect crypto map to try to establish a tunnel with one of the peers.

Routing Establishment

A packet needs to be routed to the interface which has the crypto map configured on it before IPSec successfully starts.

Routes need to be established not only for the router to reach its peers address but also for the IP subnets in the packets after they are decrypted.

Use the debug ip packet detailed command to see if the routing is occurring correctly.

Note Different switching methods use completely different code paths. It is possible to have one switching method break IPSec and another function correctly. Try a different switching path (cef, fast switching, process switching) if you are running into an obscure problem.

Troubleshooting Tunnel Establishment

The following paragraphs provide information about troubleshooting IPSec tunnel establishment.

Interesting Traffic Received

The ping source and destination addresses matched the match address access list for the crypto map multipeer.

The `src' is the local tunnel end-point, the `dest' is the remote crypto end point as configured in the map.

The src proxy is the src interesting traffic as defined by the match address access list; The dst proxy is the destination interesting traffic as defined by the match address access list

The protocol and the transforms are specified by the active crypto map, as are the lifetimes.

Main Mode IKE Negotiation

ISAKMP Authentication

Preshared authentication has succeeded; the ISAKMP SA has been successfully negotiated.

Quick Mode Negotiation

Quick mode starts and the IPSec SA negotiations begin; ISAKMP negotiates for IPSec as well.

This is the protocol offered by the remote end in accordance with its transform set.

This is the payload authentication hash offered by the remote end in accordance with it s transform set.

The IPSec SA is now successfully negotiated. ISAKMP enters a state known as QM-IDLE.

IPSec SA Establishment

ISAMKP asks the IPSec routine to validate the IPSec proposal that it has negotiated with the remote side.

Two IPSec SAs are negotiated; an incoming SA with the SPI generated by the local machine and an outbound SA with the SPIs proposed by the remote end. Crypto engine entries have been created.

The ISAKMP routine informs the IPSec routine of the IPSec SA so that the SADB can be populated:

The SADB is updated and the IPSec SAs is initialized. The tunnel is now fully functional.

Routing Issues

Before IPSec begins functioning, a packet must be routed to the interface which has the crypto map configured on it.

Routes are necessary for the router to reach its peers address and for the IP subnets in the packets after they have been decrypted.

Use the debug ip packet command to see if the routing is occurring correctly (be careful on busy networks).

Troubleshooting Example

No Connectivity Between the Peers

Note To list routes in the global routing table use the sh ip route command. Use sh ip route vrf command for listing routes in a specific VRF. You can also use the sh ip bgp vpnv4 vrf command to check routes learned/advertised through BGP.

BGP Peer Miscommunication

•

Any kind of control information filtering based on AS_PATH, route-targets, or IP address.

Problems Forwarding Packets—Data Plane

Assuming that the route is in the VRF table, use the following checks to identify the problem:

Step 2

Verify that the CEF adjacency has been built out the correct interface.

Step 3

If this is in a controlled environment, then perform a debug mpls packet to verify MPLS packet switching on the PE in the MPLS to IP path.

Step 4

Check for any kind of data traffic filtering at the egress/ingress PEs.

Step 5

CEF related packet drops can be troubleshot using show cef drop and debug ip cef drop.

Troubleshooting Tips

•

Keep things simple (for example, mode config and xauth); use preshare; work your way up the feature list.

•

Try to establish the connection from both sides; there might be issues starting it in a particular direction.

•

In the current releases of the solution a default route (or a route to the endpoint) is needed both in the global as well as VRF routing table for endpoint reachability. Lack of global route will lead to IKE negotiation right away characterized by retransmissions in MM_SA_SETUP state.

•

Ensure that the reachability to the remote LAN segment is available in the VRF table either through statics or through GRE and a routing protocol.

•

The current release of the solution also requires a public facing interface/VPN to terminate the IPSec sessions (GRE being an exception). Thus ensure that the session is terminating on the correct interface.

•

With VPN clients connecting to a multipoint interface and using RRI to install a route in the VRF routing table, Cisco IOS software does not forward packets in the return path (MPLS to IP) out a multipoint interface due to a lack off next-hop. This can be seen in the following debug that packet is received with a VPN tag but is not untagged and forwarded.

With multipoint interfaces, avoid using RRI for route injection. Instead, use a static route for the subnet pointing to a next hop that can then be redistributed into BGP.

Table Of Contents