In the case of a perm channel, anope faces a odd problem
Since upon exit of all users the channel is removed from thus there is no channel to anope.
1. All request to anope to change items on the channel without enter would result in errors... ie. /cs topic
2. I would expect that if all users exited the channel, and an oper decided to oper override without entering the channel, anope would throw a series of errors about modes or topics being set on a nonexistant channel.
This can be a quick fix, with a check in chan_deluser() if the channel has the mode to be perm then don't delete it. Or a much more complex by going through each function to review how it handles it.
**note I have not looked at inspircd docs to see how they do perm channels, going off my unreal knowledge in how they do it **