For a short period now all connectivity has been online and no longer showing signs of issues.
At the start of connectivity issues we noticed around 5% of packets being dropped and our team worked to confirm no issues in terms of DDoS or PATH end. MTR and other tools confirmed one of our ISP's having issues and this was instantly raised as a Ticket.
Open Reach kept us in the loop with tons of notifications and on one of the final changes packet loss dropped. Speaking with Open Reach confirmed everything had been fixed and it looked to be an issue internally now. On investigation, One of the ISP BGP Routers had lost all connectivity so an attempt to reboot the router resulted in losing it's a configuration for a yet unconfirmed reason. A new BGP Router has been built to replace as a precaution, the new configuration completed and all systems are now showing as online.
An investigation into why the BGP Router lost its configuration due to an issue ISP end and a possible NIC failure is still undergoing however we're set plans into motion to ensure just like all our other networking is redundant, Per ISP BGP Router will also be redundant - This had already been created as a task for the start of 2022 but sadly we should have acted sooner!
For those in DC02-DH01 that have been affected, a huge sorry for the issues caused. Will will continue to monitor and this incident will remain open until NOC has deemed the situation closed.
Posted Dec 22, 2021 - 20:57 GMT
Update
We are currently working with Open Reach to resolve an issue outside of DC02. Open Reach Engineers are currently working on the task. During downtime NOC has also started/completed Network Upgrades planned for the start of 2022 allowing us to avoid the planned tasks future downtime.
Posted Dec 22, 2021 - 16:41 GMT
Identified
We have confirmed the location of the issue and are working to resolve connectivity blips. We have noticed an improvement however will keep this area updated with any new news.
Posted Dec 22, 2021 - 14:11 GMT
Investigating
We are currently seeing blips in connectivity within DC02-DH01 and are investigating.
Posted Dec 22, 2021 - 13:17 GMT
This incident affected: Data Centre 02 (Data Hall 01).