A week ago, we decommissioned an old core switch in preparation for the upcoming scheduled upgrade this week, this left us with less redundancy on our core environment.
Today, during a work order, one of the facility techs accidentally unplugged one of the fiber cables to one of the other core switch. This caused the OSPF backbone area to go down and as a result the default route was also withdrawn from all the TORs.
Once the issue was identified, we immediately added a static route to all the TORs to resolve the outage and restore connectivity.
After which, we’ve had the facility reseat the SFP to the other core switch and the backbone area was restored.
In our new deployments, we usually force the backbone area to be up even when the rest of the links go down. However, unfortunately this old switch does not support it. (which will be upgraded soon as well)