FCoE – Waiting for flogi…

Last week I was working on a pair of Nexus 5596s and for the life of me could not get a CNA to login to the fabric.  I kept getting the following error:

2012 Jul  6 11:36:33 ANR-N5K-A %PORT-5-IF_TRUNK_DOWN: %$VSAN 664%$ Interface vfc1, vsan 664 is down (waiting for flogi)

After reconfiguring the vfc and pulling my hair out, I contacted Cisco TAC and learned the error of my ways.  The Nexus 5500s have the FCoE class-maps configured by default; however, the Nexus 5500s lack the service-policies that actually apply them.  What I lacked was as follows:

system qos
service-policy type qos input fcoe-default-in-policy
service-policy type queuing input fcoe-default-in-policy
service-policy type queuing output fcoe-default-out-policy
service-policy type network-qos fcoe-default-nq-policy

That config is enabled by default on Nexus 5000s (i.e. 5010, 5020), but not on the 5500s (i.e. 5548, 5596).  Once applied, we were able to confirm the QoS configuration on the interface:

N5K-A# show queuing interface Ethernet 1/7
Ethernet1/7 queuing information:
TX Queuing
qos-group sched-type oper-bandwidth
0 WRR 50
1 WRR 50

RX Queuing
qos-group 0
q-size: 360960, HW MTU: 1500 (1500 configured)
drop-type: drop, xon: 0, xoff: 2256
Statistics:
Pkts received over the port : 34086995
Ucast pkts sent to the cross-bar : 34038196
Mcast pkts sent to the cross-bar : 48799
Ucast pkts received from the cross-bar : 230543
Pkts sent to the port : 543488
Pkts discarded on ingress : 0
Per-priority-pause status : Rx (Inactive), Tx (Inactive)

qos-group 1
q-size: 79360, HW MTU: 2158 (2158 configured)
drop-type: no-drop, xon: 128, xoff: 252
Statistics:
Pkts received over the port : 40197323
Ucast pkts sent to the cross-bar : 40197387
Mcast pkts sent to the cross-bar : 0
Ucast pkts received from the cross-bar : 1664478
Pkts sent to the port : 1664501
Pkts discarded on ingress : 0
Per-priority-pause status : Rx (Inactive), Tx (Inactive)

Total Multicast crossbar statistics:
Mcast pkts received from the cross-bar : 312967

QoS Group 1 above is the group for FCoE traffic.  Without that service-policy applied, your vfcs will come up, but will always “wait for flogi”.  Once we applied the service-policy, we saw the following in the logs:

2012 Jul 9 09:59:48 ANR-N5K-A %PORT-5-IF_TRUNK_DOWN: %$VSAN 664%$ Interface vfc1, vsan 664 is down (waiting for flogi)
2012 Jul 9 09:59:49 ANR-N5K-A %PORT-5-IF_TRUNK_DOWN: %$VSAN 664%$ Interface vfc1, vsan 664 is down (Initializing)
2012 Jul 9 09:59:49 ANR-N5K-A %PORT-5-IF_TRUNK_UP: %$VSAN 664%$ Interface vfc1, vsan 664 is up

Hopefully that will save you some time!