Incident History

View Incident History

Current Status

September 17 - Virgin NNI Outage [55 Circuits Affected] Details

Resolved [18/09/2021 19:40]

Virgin have confirmed the issue was an faulty attenuator at Telehouse West. That was replaced to resolve the fault. Circuits restored at approximately 13:08.

September 13 - Control Panel Access Details

Resolved [13/09/2021 10:13]

The issue has been identified and resolved. Apologies for any disruption caused.

August 24 - Control Panel & Shared Email Access Details

Resolved [13/09/2021 10:06]

July 02 - Network Disruption Details

Resolved [02/07/2021 13:44]

Whilst investigating a degraded performance issue on a dark fibre at our LD8 PoP, a third party engineer inadvertently disconnected another dark fibre that connects LD8 to a third location. This subsequently resulted in LD8 becoming isolated from the rest of the network for a short period, between 00:06:02 and 00:09:56.

As previously reported, during this time leased line circuits terminating at LD8 would have experienced a loss of connectivity. Broadband circuits were impacted further due to a large number of subscriber sessions that were terminating at LD8 disconnecting.

Whilst the majority of the affected broadband subscribers regained a session at another PoP relatively quickly, others whose sessions were steered to a particular aggregation router on the network failed to start. Our engineers investigated and discovered that the router was experiencing a fault condition and took it out of service. At this point the vast majority of remaining subscribers re-gained their sessions.

Apologies for the disruption this may have caused.

June 13 - Telehouse West POP Issue Details

Resolved [14/06/2021 10:00]

At 12:32:23 on 13/06/21 a supervisor in a core switch at our THW PoP experienced an inexplicable reboot. Shortly afterwards at 12:32:40 a hot standby supervisor took over the active role and restored the overall connectivity to the PoP.

The original active supervisor that rebooted was back in service as a hot standby by 12:41:52. By 12:54:47 it had brought all its line cards online following a full and successful diagnostics run. All connectivity was restored to the site by this point.

Non-resilient leased line circuits that terminate on NNIs directly connected to the rebooted supervisor would have experienced an outage between 12:32:23 and 12:54:47.

All other non-resilient leased line circuits as well as any broadband circuits that were terminating at THW would have seen a loss of connectivity between 12:32:23 and 12:32:40.

We have raised this to the vendor's TAC for further investigation. The device is currently stable and not showing any signs of issues. As such we do not deem the site to be at further risk at this time.

Apologies for the disruption this may have caused.

May 18 - TalkTalk DSL Circuit Outages Details

Resolved [18/05/2021 17:01]

The carrier has resolved the issue and the majority of affected circuits are online. A power cycle of the router may be required to force a reconnection.

May 11 - Network Performance at DC5 Data Centre Details

Resolved [11/05/2021 09:33]

The issue was traced to a peering device that we've now taken offline and full service has been restored. Apologies for the disruption that this may have caused some users this morning.

May 10 - Network Performance at DC5 Data Centre Details

Resolved [10/05/2021 17:54]

The issue has been resolved. The cause remains under investigation. Apologies for any disruption to service you may have experienced.

April 30 - Network Issue Affecting Broadband and Leased Line Circuits Details

Resolved [10/05/2021 12:58]

April 23 - Network Performance at DC5 Data Centre Details

Resolved [30/04/2021 09:21]

The cause was linked to a denial of service attack. We apologise for the disruption experienced.

March 24 - Database Maintenance 24/03/2021 Details

Completed [26/03/2021 11:29]

March 06 - Control Panel / CIX Forums Issue Details

Resolved [06/03/2021 22:04]

March 02 - Openreach COVID Update 3 Details

Resolved [28/06/2021 11:14]

February 16 - CityFibre Outage Details

Resolved [17/02/2021 17:46]

CityFibre have confirmed that all affected services have been restored and a full investigation is underway. We apologise for those customers affected by this issue.

February 12 - Network Issue Affecting DC5 Data Center Details

Resolved [12/02/2021 17:24]

We believe the reboot has resolved the issue.

February 11 - Network Maintenance 11/02/2021 9PM GMT Details

Completed [12/02/2021 14:43]

February 11 - Hosting Network Maintenance 10/02/2021 21:00 GMT Details

Completed [10/02/2021 09:50]

January 28 - Hosting Network Maintenance 28/01/2021 23:00GMT - 06:00AM. Details

Completed [10/02/2021 09:46]

January 12 - Openreach COVID Update 2 Details

Resolved [02/03/2021 12:23]

December 29 - Control Panel Maintenance 29/12/2020 Details

Completed [30/12/2020 01:25]

December 18 - Openreach API Interaction Details

Resolved [18/12/2020 16:21]

We are now receiving responses from the various affected systems. Confirmation of a resolution hasn't been announced by Openreach, so services should be considered at risk.

December 14 - TalkTalk Outage in North East of England Details

Resolved [15/12/2020 13:21]

The issue has been resolved. Control panel users can see further details -

December 14 - Core Network at DC5 Data Centre Details

Resolved [16/12/2020 11:25]

We believe yesterday's networking issues for the hosting data centre are now fully resolved. We have waited for the dust to settle before giving the all clear. Apologies again for any disruption this has caused. Once we receive a full explanation into the cause, we will provide this as soon as possible.

December 02 - WLR Diagnostic Tests Details

Resolved [02/12/2020 09:39]

This issue has now been resolved and diagnostics are working again.

November 06 - Openreach COVID Update Details

Resolved [12/01/2021 15:03]

November 03 - Control Panel Maintenance 03/11/2020 - 9PM GMT Details

Completed [03/11/2020 22:46]

This work completed.

October 29 - BT Wholesale Ethernet Interconnects Down Details

Resolved [30/10/2020 06:33]

Service was resumed at approximately 02:15. We apologise for this unexpected outage.

October 19 - Control Panel Unavailable Details

Resolved [19/10/2020 13:50]

October 14 - Unexpected Broadband Session Drops Details

Resolved [14/10/2020 11:15]

Apologies for the session drops this morning. The cause was linked to additional interconnects being patched into one of our London POPs. This caused an issue with one of our broadband LNS which dropped sessions, only for them to be able to reconnect. It would have impacted any circuits routed via that LNS across TalkTalk and BT Wholesale.

This was unexpected behaviour and should not have occurred. We will continue to monitor and will raise this with the manufacturer as a suspected bug.

October 04 - Control Panel and VPS Platform Maintenance Details

Completed [15/10/2020 12:05]

September 26 - Virgin NNI Outage [30 circuits Affected] Details

Resolved [26/09/2020 08:18]

Fault was tracked down to a power failure within a Virgin Media rack. All circuits are operational.

September 15 - Broadband network outage affecting TTB based circuits Details

Resolved [15/09/2020 16:20]

We have seen near-normal levels of sessions restore through the afternoon. Anyone unable to reconnect should be able to do so with a power cycle. If this doesn't address it try powering down for an hour and reconnect. Failure to connect still may require assistance from our support team.

We will not terminate sessions to force a reconnection back to Telehouse North, they will naturally spread out as sessions drop of their own accord.

We are reviewing this outage internally, but ultimately the cause lay with the carrier.

August 31 - TalkTalk Ethernet Outage [LD8] Details

Resolved [31/08/2020 22:47]

The root cause was TalkTalk maintained hardware failure affecting 1 Ethernet NNI of ours and 6000 other B2B clients. TalkTalk fault incident resolution states:

<-- snip -->
NOC monitoring identified an FPC10 (Flexible PIC Concentrator) failure at NGE001.LOH. This caused a total loss of service to approx. 6k B2B circuits from approx. 12:47 (31/08). The Core Network Ops team were engaged and their investigations found that the FPC10 had failed and could not be restored remotely. To restore service as of approx. 17:23 a field engineer attended site and replaced the faulty FTP10 with support from the core network ops team. This incident will now be closed with any further root cause analysis being completed via the problem management process.
<-- snip -->

Apologies for the disruption caused this afternoon.

August 30 - Problems with Internet Routing / CenturyLink Details

Resolved [30/08/2020 21:25]

The issue was resolved around 16:10. CenturyLink responded via Twitter to say:

<-- snip -->

We are able to confirm that all services impacted by today’s IP outage have been restored. We understand how important these services are to our customers, and we sincerely apologize for the impact this outage caused.

<-- snip -->

Although we and their other global customers withdrew routes and shut down peering sessions, they continued to announce them to their peers regardless. This caused black holing of any inbound traffic routed via CenturyLink. All affected customers were left powerless and it has been a case of having to wait for them resolve the issue.

Thankfully less than 10% of our overall traffic routes in via CenturyLink's network, so the impact was minimal. We know of only a small handful of destinations that were unreachable during their outage. Apologies if your access was disrupted.

August 19 - Hosting Outage Details

Resolved [04/10/2020 13:46]

August 18 - POP Affecting Issue at LD8 Details

Resolved [04/10/2020 13:45]

July 23 - Broadband Incident in the North of England Details

Resolved [23/07/2020 17:25]

Following a small fire at one of our Newcastle Upon Tyne exchanges earlier today, Openreach have now restored power to all services. All Broadband and Ethernet services should now be up and working.

July 08 - Control Panel Maintenance Details

Completed [10/07/2020 22:53]

July 04 - Server Maintenance and Rack Move in London Data Centre. Details

Completed [06/07/2020 10:37]

June 22 - Control Panel Details

Resolved [04/07/2020 21:44]

June 19 - Calling Office Number Details

Resolved [04/07/2020 21:50]

June 13 - Control Panel Details

Completed [13/06/2020 10:36]

This work completed successfully.

June 03 - Control Panel and Shared SQL Down Details

Resolved [13/06/2020 06:33]

May 15 - TalkTalk Broadband API Details

Resolved [13/06/2020 06:33]

April 22 - Control Panel Maintenance Details

Completed [05/05/2020 10:36]

April 16 - Issue with Virgin Leased Lines Details

Resolved [22/04/2020 15:07]

March 24 - Statement from Openreach regarding Covid 19 Details

Resolved [13/06/2020 06:34]

February 19 - Networking Issue Details

Resolved [19/02/2020 13:10]

The issue was related to LINX (the London Internet Exchange), which has now been resolved and would have potentially affected several Internet providers in the UK. We are awaiting a full RFO from them to confirm the cause.

October 08 - TalkTalk Business Sessions Failing to Reconnect Details

Resolved [08/10/2019 15:26]

The cause has been located and service has now stablised. If a connection hasn't returned please power cycle the router to force a reconnection attempt. Apologies for the disruption witnessed.

August 30 - Emergency DC5 Core Router Upgrade Details

Completed [30/08/2019 00:10]

The upgrade was successful and cleared the fault condition as suspected. We have been monitoring for the past hour and have not seen any further instability.

August 29 - Connectivity Outage Details

Resolved [29/08/2019 17:42]

We are seeing services restored now. If any connections remain offline please reboot the routers. The root cause is under investigation.

May 29 - Virgin Media Fibre Break Details

Resolved [30/05/2019 12:29]

The fault was resolved with all circuits restored by 13:35.

Full notes from Virgin Media Business surrounding the handling of this fault can be found here:

We apologise again for the prolonged outage which affected working hours.

April 15 - BT Wholesale Interconnect Details

Resolved [17/04/2019 17:19]

BT have confirmed that a line card needed to be reloaded in order to resolve the issue, we consider services to no longer be at risk.

Please let know if you have any further concerns.

April 10 - Network Issue Affecting DC5 Data Centre Details

Resolved [15/04/2019 15:26]

The problem was localised to 1 rack of servers following a PDU failure.

April 05 - Virgin Media Leased Line Outage [London & Surrounding Areas] Details

Resolved [05/04/2019 13:02]

The affected circuits appear to have been restored. We have received no further communication from Virgin, so please consider service to be at risk.

February 25 - Communicating With TalkTalk Business API Details

Resolved [25/02/2019 12:23]

TalkTalk's systems appear to be operational again. However, please consider them to be at risk as we have not received any communication from them to confirm that everything is back to normal.

February 14 - Control Panel Access Details

Resolved [14/02/2019 11:22]

The issue has been resolved. The cause was linked to a connectivity issue between the servers in the cluster.

January 18 - Internet Routing Issues Details

Resolved [18/01/2019 10:11]

The issue with routing has been resolved, we apologise for any inconvenience caused this morning.

If you continue to have any problems please contact the support desk with specific examples.

December 14 - Network Maintenance At Equinix LD8 Details

Completed [14/12/2018 01:31]

The maintenance is now complete.

November 15 - Broadband outage in Yorkshire Details

Resolved [15/11/2018 17:33]

October 15 - Virgin Leased Lines Details

Resolved [16/10/2018 08:59]

Virgin have supplied the following reason for outage:

"In relation to the issue identified in the London area regarding loss of service. This issue was fully restored at 20:43 yesterday evening when a faulty DC output breaker was discovered at our Hayes hubsite and services were moved away from it onto a different output breaker. All services have been stable since that time."

October 10 - Control Panel and Webmail Details

Completed [10/10/2018 23:57]

October 05 - Broadband outage in Keighley, Bradford and Bingley Details

Resolved [09/10/2018 12:05]

All services are back working now.

September 26 - BT Wholesale based connections down Details

Resolved [02/10/2018 09:47]

September 25 - TalkTalk Business portal and API issues Details

Resolved [25/09/2018 12:11]

This issue is fully resolved now.

September 09 - Webmail and Control Panel Access Details

Resolved [09/09/2018 22:02]

Earlier this evening, we experienced an issue with our core SQL cluster which prevented access to our webmail interfaces and access to our Control Panel. This was resolved around 21:30 BST.

September 06 - Network Maintenance 06/09/2018 Details

Completed [09/09/2018 22:03]

Work completed around 2AM BST.

July 05 - Network Connectivity Issues Details

Resolved [05/07/2018 16:54]

This issue appears to be resolved now, although we are monitoring the situation carefully.

Reason For Outage - TalkTalk identified an issue with a third party peering provider, Iomart, who incorrectly advertised a subnet. This was a highly unusual event, and hopefully one we never see repeated.

Current Status