At 07:00 AM, my phone is ringing. I am answering it and my teammate saying that our customer’s users’ getting authentication fail messages on their computers. I am logging on my computer and accessing to Cisco ISE Primary Node via SSH, but an error message is there; % Error: Unable to launch ADE-OS shell. Disk full.
My story begun especially like this. I have understood that users’ getting authentication fail message because Cisco ISE Primary Node’s disk is full, so it cannot provide authentication services for our environment. Also, appliance dis-joined from Active Directory Domain. Firstly, I have disabled Cisco ISE node in Cisco Wireless LAN Controller’s Authentication Servers and prevented to forwarding AuthC requests to Cisco ISE Primary Node. I was isolated it from the infrastructure.
I have created a Cisco TAC Support Case, because I couldn’t issued any command on appliance because of the error message.
Cisco TAC Engineer sent us an ISO image to reboot appliance from this ISO. I have burned this image to DVD and connect it to our appliance via USB CD/DVD-ROM and reboot the appliance. I have configured the BIOS settings to reboot appliance from USB CD/DVD-ROM. However, appliance didn’t use it and rebooted from its primary HDD. Interestingly, after the reboot I could access to node via SSH without error message.
Cisco TAC Engineer connected to my machine via Webex and sent two different ZIP file. I uploaded these files to TFTP and TAC Engineer downloaded them from TFTP to ISE appliance. These files were patches to access Red Hat shell from ISE Application Level. TAC Engineer, installed patches and accessed to shell and checked the partitions disk usages. He found that TMP folder is using %100 of its partition size. He deleted all of TMP folder content and then disk usage was %13.
After the clean-up, disk usage is decreased but Cisco ISE couldn’t join to domain. Cisco TAC Engineer, sent a registry file and applied it to appliance. After the applying registry file, ISE successfully joined to domain. I have enabled ISE in WLC configuration and seems that everything was working fine!
Cisco sent a short summary and a bug ID. We were hitting the bug CSCuz25672. Because of this bug, Cisco ISE’s affected versions cannot delete TMP files in OS level and result of this disk usage hitting to %100 then Cisco ISE dis-joined from domain and AuthC services stopped in environment.
This bug is resolved in Cisco ISE Version 2.1 Patch 5