Configuring Cisco Nexus 5500 series switches with Dual-Homed FEXes

More and more Enterprises come to a decision to deploy Cisco Nexus switches in their corporate data centers. One of the main design considerations relates to Cisco 2200 Fabric Extenders (FEX) connectivity topology. To provide high availability Cisco Nexus 5500 series switches support different options to connect FEXes (all are based on a Virtual Port Channel feature):

• Straight-Through, where every FEX is connected to a single N5K (Active/Active and Active/Passive servers);
• Dual-Homed, where each FEX has one or more uplinks to two N5K switches (Active/Passive servers);
• Enhanced vPC, same as Dual-Homed but with Active/Active servers;

Each option has its own limitations and field of use, but this time we will concentrate our attention on the Dual-Homed FEX topology with Active/Standby Dual-Homed servers. Mainly because I recently deployed a pair of Cisco Nexus 5596UP switches with a number of Cisco 2248 TP-E Fabric Extenders using this approach.

Let’s take a look at the following diagram.

Dashed ellipse implies the use of Port-Channels. These links are logical and may consist of multiple physical connections. Well, they very likely do.

Each FEX is Dual-Homed to both Nexus 5596UP switches using vPC feature. Server A is using Active/Passive NIC configuration, so that NIC-B is only active when NIC-A, or FEX-01, is faulty. To make this topology work and provide High Availability, both N5K switches have to be configured using the following guidelines.

• Both switches must have Layer 2 connectivity via mgmt0 interfaces;
• Both mgmt0 interfaces must have IP addresses configured that belong to the same subnet;

  interface mgmt0
   description Out-of-Band Management Interface
   ip address 192.168.0.11/24

• Both switches must have a vPC domain configured using each peer’s mgmt0 interface’s IP address as keepalive destination;

  vpc domain 10
   role priority 100
   peer-keepalive destination 192.168.0.12

• Both switches must be interconnected using a Port Channel (2ⁿ physical links) that is also configured to be a vPC peer-link;

  interface port-channel69
   description vPC Peer-Link
   switchport mode trunk
   switchport trunk native vlan 10
   switchport trunk allowed vlan 10,150-155
   spanning-tree port type network
   speed 10000
   vpc peer-link
  !
  interface ethernet1/47
   description Link to N5K-02 (vPC peer-link)
   switchport mode trunk
   switchport trunk native vlan 10
   switchport trunk allowed vlan 10,150-155
   channel-group 69
  !
  interface ethernet1/48
   description Link to N5K-02 (vPC peer-link)
   switchport mode trunk
   switchport trunk native vlan 10
   switchport trunk allowed vlan 10,150-155
   channel-group 69

• Fabric Extenders must be pre-provisioned on both Nexus switches, otherwise the topology will not provide High Availability in one particular case – when one switch loses its connectivity to the FEX, while the other switch keeps it. By default, Cisco Nexus 5500 switches dynamically reacts to any FEX connectivity changes – they remove any configuration commands for the non-existing interfaces (this happens when FEX is not connected). In turn, this brings the relevant interfaces on the other switch into vPC inconsistent state even though the FEX is still connected to it;

  slot 101
   provision model N2K-C2248TP-E-1GE
  slot 102
   provision model N2K-C2248TP-E-1GE

• Each Dual-Homed FEX’s configuration must existon both N5K switches using identical numbering and vPC configuration;

  fex 101
   pinning max-links 1
  fex 102
   pinning max-links 1
  !
  interface port-channel101
    description Link to FEX-01
    switchport mode fex-fabric
    fex associate 101
    vpc 101
  !
  interface port-channel102
    description Link to FEX-02
    switchport mode fex-fabric
    fex associate 102
    vpc 102
  !
  interface ethernet1/1
   description Link to FEX-101
   switchport mode fex-fabric
   fex associate 101
   channel-group 101
  !
  interface ethernet1/2
   description Link to FEX-102
   switchport mode fex-fabric
   fex associate 102
   channel-group 102

• Dual-Homed FEX interfaces configuration must be identical, otherwise interface will be shutdown due to the configuration inconsistency (Remember about this requirement every time a new FEX interface is configured);

  interface ethernet101/0/10
   description Server-A-NIC-A
   switchport access vlan 150
   spanning-tree port type edge
  !
  interface ethernet102/0/10
   description Server-A-NIC-B
   switchport access vlan 150
   spanning-tree port type edge

That’s it. Considering all the requirements listed above were met, a vPC peer- and keepalive links will come up.

Original Link # http://nexp.com.ua/technologies/dc/cisco-nexus-5500-dual-homed-fex/