B2B ISP Fastnet staggers back to feet after VMware incident

Brighton-based ISP and hosting provider Fastnet has emerged from a trying week which involved battling VMware/Broadcom tech issues that have downed a number of its customers' websites.

The outage began on August 13, at which point Fastnet said its technical team was engaged "at the highest level with the vendor" and was working as quickly as possible to restore services.

Fastnet repurposed its homepage into a rolling live blog of progress updates, posting a new one roughly every two hours during workdays. The company didn't, however, reveal much in the way of technical details about the outage, only attributing the woes to its VMware hosting platform.

"Fastnet are currently experiencing a major, wide-scale service outage across our VMware hosting platform, which is affecting users with services that involve these products," its website read for part of last week. "This includes web/email/DNS hosting including cPanel services, Virtual Datacentre, and some other services that are dependent on the infrastructure. Connectivity, DSL, and Ethernet services are unaffected.

"Fastnet Broadband customers that currently have working internet connections are advised not to reboot their routers in order to troubleshoot issues with our other services as this may cause subsequent reconnections to fail and take your connection offline."

A source at one customer who wished to remain anonymous told El Reg they were disappointed with the company's communication of the incident:

"They offered to keep us informed by email but we would have been unlikely to receive them [given] the issues, so the only updates we have had have been the ones posted on the Fastnet website." The level of technical detail was described as "minimal".

The customer told us Fastnet is the registrar and DNS provider for its domain, and as its MX records weren't available and SPF records couldn't be checked, outbound email had been "hit and miss." Staff could access their email since it's provided through Office 365, but most organizations were not accepting any communications sent from their domain.

Fortunately, the customer's DNS zone came back online at 1100 on Thursday, so digital services were returning, but it still caused an unwelcome 36-hour outage for the company.

"I think it goes to show how important but overlooked DNS is in the underpinnings of the internet," the source told us.

• GitHub rolls back database change after breaking itself
• CrowdStrike president cheered after accepting 'Epic Fail' Pwnie award
• CrowdStrike hires outside security outfits to review troubled Falcon code
• Israeli hacktivist group brags it took down Iran's internet

There was no shortage of Fastnet customers affected by the outage. Professional bodies such as FICM and the Faculty of Pain Medicine (part of the Royal College of Anaesthetists, which was also downed) were affected.

Small businesses from wine distributors to seafood restaurants were also impacted, and even the Caravan and Motorhome Club resorted to telephone comms only, although many were back up and running by Friday.

Fastnet said on the evening of August 15 (UTC timezone) that the majority of its shared hosting environment was back and fully operational, but was still working to restore other customer-related services.

"​We are continuing work to bring further customer-facing services back online," it said on the morning of August 16. "We understand the importance of service restoration, and are proceeding as quickly as possible, in conjunction with our vendors. We will provide further updates as they become available."

El Reg got in touch with Fastnet and Broadcom asking for more details about the incident, and whether there were suspicions of foul play as our source feared, but neither responded.

Fastnet promised from the outset that a full explanation of the outage would be published after the incident was resolved, and consistently apologized to affected customers.

The most recent update came on Saturday afternoon, but didn't indicate any additional progress had been made. There remained some shared hosting platforms that were still being brought back online, but DNS, hosting, and email services were all fully restored by that point. ®