problem accessing site

Get help with technical problems and keep up to date with any changes to the forum.
Post Reply
Nash

problem accessing site

Post by Nash » Tue Jan 03, 2012 9:03 pm

Anyone else having problems getting on the site today or is it just me?

User avatar
Ros
Moderator
Moderator
Posts: 7955
Joined: Sun Dec 07, 2008 4:53 pm
antispam: no
Location: this hill-shadowed city/of razors and knives.
Contact:

Re: problem accessing site

Post by Ros » Tue Jan 03, 2012 9:06 pm

At a guess, Nicky might be upgrading the site - I keep seeing odd admin messages. So, no, it's not you, Nash, it's everyone.

Ros
Rosencrantz: What are you playing at? Guildenstern: Words. Words. They're all we have to go on.
___________________________
Antiphon - www.antiphon.org.uk

User avatar
Nicola
Site Admin
Site Admin
Posts: 1060
Joined: Sun May 09, 2004 5:35 pm

Re: problem accessing site

Post by Nicola » Tue Jan 03, 2012 9:22 pm

There has been an issue with the webhosts today - not sure how long that will last. I have also updated the site this evening - sorry I didn;t give any warning but it was a last minute decision and then took longer than expected.

Hopefully the update has been successful but I think there may be some more glitches from the webhosting. Tomorrow it should all be ok.

Nash

Re: problem accessing site

Post by Nash » Tue Jan 03, 2012 9:32 pm

Ahh, that's good then, I thought it was me.

Thanks both,
Nash.

brianedwards
Perspicacious Poster
Perspicacious Poster
Posts: 5375
Joined: Tue Jul 22, 2008 7:35 am
antispam: no
Location: Japan
Contact:

Re: problem accessing site

Post by brianedwards » Wed Jan 04, 2012 2:09 am

I was having trouble too, but all seems fine now.

B.

User avatar
Nicola
Site Admin
Site Admin
Posts: 1060
Joined: Sun May 09, 2004 5:35 pm

Re: problem accessing site

Post by Nicola » Wed Jan 04, 2012 5:07 pm

This was the message received from the webhosts.

"First and foremost we would like to thank you for your patience yesterday and hope that you will accept our sincerest apologies for the prolonged network outage. We fully appreciate the impact this issue will have had on your business/website/email and would therefore like to reiterate that we will be doing everything in our power to ensure something of this nature does not occur again.

Preliminary analysis indicates that the original cause of the incident was a large DDOS attack targeted against one or more tenants, such as us, but not us, hosted within our datacentre facility. The attack was of such magnitude that the external IP transit and peering links were saturated, causing instability. This was then magnified by the sheer volume of traffic being directed from the core routers towards their internal switching platform.

Despite the datacentre policy of over-provisioning all switch and network link capacity, this volume of traffic was sufficient to disrupt their control protocols that normally ensure proper operation of the switches. The network engineers worked quickly to null-route the DDOS traffic; however the induced instability within the switching network reintroduced problems and created others down-stream within their customer switching networks.

In order to contain the situation and restore normal operation, datacentre network engineers took the major step of shutting down segments of the core switching network and re-establishing each tenant connection (us being one) from scratch. This was ultimately successful, but entailed a large amount of reconfiguration and testing work which unfortunately took time and caused further intermittent packet loss and downtime for the large majority of our customers.

As of approximately 1:00 AM (GMT) datacentre engineers were able to restore full service. They have since decided that the quickest and most risk-free option for restoring full resiliency and adding additional network capacity is to bring forward planned maintenance to replace their existing core switches. Their network engineers will be working with J-TAC (Juniper Technical Assistance Centre) to complete staging of the new switches today, and will carry out emergency maintenance tonight (4th January) to perform the upgrade.

We anticipate this maintenance will commence around 2200 (10:00 PM GMT) and downtime should not exceed more than five minutes. We will issue an update once the network engineers confirm the time and an ALL-CLEAR update will be issued upon successful competition of this maintenance.

Once again, across the board we understand the negative impact that downtime can have on each and every customer. It is a nightmare for all parties involved and something that we take extremely seriously. And although what happened yesterday was extremely rare, we remain fully committed to doing everything possible to ensure that this will not happen again. performance improvements to all customers which will help mitigate similar issues in the future. We will announce these changes separate of this notification. "

Post Reply