Routing over Nexus 7000 vPC peer-link? Yes and No.

This is a Nexus 7000 design question that comes up from time to time:

In a Nexus 7000 Vpc environment, how can I form a layer 3 adjency between the two switches. Lets say I want to run OSPF and want to create two SVIs on the two switches connected via Vpc, Will the neighborship relation be formed over the Vpc Peer link or is the peer link only designed for control traffic for Vpc.

Some people believe that in order to form an L3 adjacency between two Nexus 7000 vPC peer switches you must provision a separate link (other than the peer link) to use for L3 routing.  This is not true.  You absolutely can use the existing vPC peer link to form a routing adjacency between two vPC peer Nexus 7000′s.

I believe the source of confusion comes from a vPC design caveat that gets condensed and passed on as simply: “No L3 routing over vPC”.

Not knowing any better, if you take that statement at face value its easy to see how one might believe that you cannot form a routing adjacency between the two Nexus 7000′s over the vPC peer link, and therefore seek to provision a new “non-vPC” inter-switch link for that purpose.

Let’s take a minute to look at what works, and what doesn’t.  I’m not going to bore you with all the technical details.  Instead we’ll take a look at six simple diagrams with brief explanations.

Diagram #1 below shows two Nexus 7000′s configured as vPC peers with a single inter-switch link between them, the vPC peer link.  The two Nexus 7000′s are configured for OSPF and are using an SVI associated to a VLAN on the peer-link to form the L3 adjacency.  This VLAN for the L3 adjacency should only be forwarded on the peer-link.  Do not forward this routing VLAN on the vPC member ports (such as toward the L2 Switch shown in the diagram).

L3 peering over vPC peer-link

In the diagram above, we have a Layer 3 switch or router upstream configured for OSPF.  This L3 switch is attached to each Nexus 7000 with two normal point to point Layer 3 interfaces, no port channels, no vPC.

This design works.  This design has been discussed in official Cisco design documents and is supported by Cisco TAC.

So where does the “No L3 routing over vPC” come from anyway? Simply put, this comes from the vPC design caveat that you should NOT have an external device (firewall, router, switch) forming a routing protocol adjacency with the Nexus 7000′s over the vPC peer link.

Diagram #2 below shows an L3 switch running OSPF attached with a vPC to the two Nexus 7000′s and attempting to form an OSPF adjacency with each Nexus 7000. This design does NOT work.

Router attached via vPC – peering with Nexus 7000

This design does NOT work because some of the OSPF routed traffic from the L3 switch to the Nexus 7000′s will traverse the vPC peer-link (even when no ports or links are failed). As a result, this traffic will be dropped as a result of loop prevention logic in the Nexus 7000 hardware.

Side note: For the more vPC savvy folks out there who might be wondering “Does the new vPC Peer Gateway” feature make this design work? The answer is No.

Let’s look at Diagram #3 below. Here’s another example of an external device building a routing protocol adjacency with the Nexus 7000′s, this time its firewalls. The firewalls are singly attached (no vPC) to a VLAN that is forwarded on the Nexus 7000′s vPC peer link. The firewalls are running OSPF and attempting for form an adjacency with the each Nexus 7000.  This design too does NOT work.

Firewalls running OSPF attached to Nexus 7000’s running vPC

This design does NOT work for the same reason as Diagram #2. Each firewall will form an OSPF adjacency with both Nexus 7000′s. This means that some OSPF routed traffic will traverse the vPC peer-link (even when no ports or links are failed). As a result, this traffic will be dropped.

Each firewall see’s both 7K1 and 7K2 as directly adjacent OSPF neighbors. If Firewall-1 chooses 7K-2 as the next-hop, this traffic will traverse the peer-link with the loop prevention bit set. If 7K-2 realizes the packet is destined for a vPC member port it will drop the traffic if the corresponding vPC member port on 7K-1 is up. If the packet was not destined for a vPC member port it would be forwarded normally.

To make the above Diagram #3 work, we can provision a new inter-switch link between the Nexus 7000′s that is NOT a vPC peer-link. On this new link we will forward a VLAN that is not forwarding on the vPC peer-link. This makes it a non-vPC VLAN. From here we can attach our firewalls running OSPF to the Nexus 7000′s on the non-vPC VLAN. See Diagram #3B below.

Firewalls with OSPF attached to non-vPC VLANs

This design works because each firewall will form an adjacency with both Nexus 7000′s, however the OSPF routed traffic will not traverse the vPC peer-link and not be subject to any loop prevention logic.
Keep in mind that non-vPC VLANs cannot be forwarded on vPC member ports. So if you have another device that is vPC attached and needs to be Layer 2 adjacent to the firewalls, that will not work. You will need to get that other device attached to the non-vPC VLAN as well on a normal non-vPC connection.

If your firewalls are not running a routing protocol there is no problem at all. Simply have static routes, or a static default route, pointing to the HSRP VIP address on the Nexus 7000′s.
See Diagram 4 below. This design works.

Firewall with static routes connected to Nexus 7000’s running vPC

In this case each Nexus 7000 will locally forward traffic sent to the HSRP VIP address (even if its the “Standby” switch). Therefore no static routed traffic will ever traverse the vPC peer-link, and there will be no problems.

Finally, let’s look at Diagram #5 below where we have external devices running a routing protocol attached to the Nexus 7000′s. Moreover, these devices are attached with a vPC. That shouldn’t work, right? Well, lets take a look. This design DOES work.

Routing peers using vPC for transit only

This design works because these devices are not attempting to form an OSPF adjacency with the Nexus 7000′s. Rather, these external routing devices are forming an adjacency only with each other, and simply using the vPC topology as a transit. Under normal conditions when no links or ports are failed, there will be no traffic traversing the peer link and no problems.

Summary

The Nexus 7000 hardware has loop prevention logic that drops traffic traversing the peer link (destined for a vPC member port) when there are no failed vPC ports or links.  Normally peer-link traffic is non-existent in a normal network and this is never a problem for attaching normal Layer 2 switches or servers.  However when external devices attempt to form a routing adjacency with the Nexus 7000′s over the vPC peer-link, some traffic destined for a vPC member port can be forced over the peer-link even when the network is healthy, causing traffic to be dropped by the loop prevention logic.

Best practice:

• Attach external routers or L3 switches with L3 routed interfaces.
• It’s OK to use the vPC peer-link to form a routing adjacency between the two Nexus 7000′s.  Use a VLAN dedicated to the routing adjacency and only forward this VLAN on the peer-link, not on the vPC member ports.
• Use the ‘passive-interface default’ command in your routing protocol to prevent a routing adjacency on all the other VLANs.
• If attaching external devices on a Layer 2 port running a routing protocol with the Nexus 7000′s (e.g. firewall running OSPF), provision a new non-vPC inter-switch link, and attach the device to non-vPC VLANs.
• Use static routes to the HSRP gateway address on external devices such as firewalls and load balancers.  Do not run routing protocols on these devices unless absolutely necessary.
• Read the Cisco vPC best practices design guides