Cisco Bug: CSCvf52723-FlexConnect AP Cannot Pass Traffic

How Do You Know, It Is A Bug?

Each software has bugs. The most important thing is about a bug, how it affects your business or your life!

We are tech guys and we spent most of our time to solve issues. However, some of issues require a different perspective or knowledge level to solve it. CSCvf52723 is one of the bug that was required Cisco TAC support to solve it quickly. Did I solve it quickly with Cisco TAC, actually no! This was an interesting issue for all of us!

We Are Unable To Connect Guest Network!

I received an e-mail from my customer and he was telling that visitors could not connect to guest network at a FlexConnect AP location. My first question was: “Is there any problem on corporate SSID for employees?” The answer was really important for me. Because I need to clearly understand the issue and focus to related part of network or device. Answer was clear. Only guest network is affected!

I connected to Cisco WLC and ISE and started viewing client logs for guest network. Everything was good. Client sending an association request, WLC associate the client and applying related ACL to redirect CWA Portal. Client is opening browser, typing username/password and successfully authenticated. Everything seems like working, right?

Friendship With Cisco TAC Engineers

Yes, I have built a friendship with Cisco TAC Engineers, because I have worked with 4 different engineers for 3 nights! I have worked with 2 Cisco Wireless Engineers, 2 Cisco Security Engineers from Cisco ASA and ISE Teams.

Is That A Wireless Issue?

Wireless Engineer analyzed the WLC and collected client debug logs. Everything was working as expected. Client was connecting to SSID, getting an IP address and successfully authenticating. At this step, we have seen that client’s Radius NAC State was RUN. It was mean that all required steps are completed and client successfully joined to the network. However, still there is no connectivity. Nevertheless, we were assume that WLC is doing its job perfectly!

ISE Engineer Joined The Party!

We were assume that WLC is doing its job perfectly. If it is, where the problem was? Wireless Engineer invited an ISE Engineer to our WebEx party! We have checked both of ISE nodes and monitor logs. Normally, ISE was accepting all authentication requests and applying correct authorization policy and result. So, ISE Engineer reported that ISE is working as expected. Hold on, one more TAC engineer is here!

Can ASA Deny Our Connection Requests?

Yes, ASA can deny our connection requests, because it is a firewall. However, to block a connection request, firstly ASA need to see some requests on it. When we check the firewall logs in real time, there was no connection request from client to ASA. In this step, we perceive that only DNS requests are coming from client to ASA. At this point, I have remembered an previous test result and shared with engineers. If I change the CWA ACL on Cisco WLC as default permit, everything works fine but this time clients can connect to SSID without an username/password. After that we were focus on WLC, again.

Which Version Is Running?

After a while, we focus on WLC again. Wireless Engineer checked WLC’s firmware version and completed his analysis. Finally, he reported that our running firmware version is not a TAC recommended version and needs to be upgraded.

Urgent Upgrade Required!

I contacted with my customer and described the current situation. I planned a maintenance window and upgraded WLC from 8.2.164 to 8.2.166.

Cisco Bug Id is here! (Cisco Account Required)

Thank You Friends, Issue Resolved!

As a result, 8.2.166 firmware solved my issue. Clients could connect to SSID, authenticated via ISE and accessing to the internet. However, there is one more thing. If a client connects to guest once, ISE cannot prompt credentials second time!

How did I solve this problem? We will check that case in a different article!

Please share your similar issues and resolves in comments!