RackCorp Managed Hosting
Dashboard View:   Timezone:   Datacenter:

Recent Maintenance / Outage Events

27/09/2017 01:07CLOSED
 Loss of network [New Zealand - Public Internet]
 27/09/2017 01:07 - We are investigating a loss of network that started a few minutes ago.
 27/09/2017 01:10 - It looks like an upstream provider in one of the NZ sites has lost all network connectivity. We'll await them to restore the network and will provide any updates as we hear them.
 27/09/2017 01:13 - Services are back up.
22/09/2017 04:36CLOSED
 vmh56 storage issue [AU (Sydney GlobalSwitch) - Virtual Hosts]
 22/09/2017 04:36 - We have had to restart a few VM's (All VMs that were >1TB storage and a couple of ones that had not been rebooted for a long time) that were attached to this storage node. Other ones were live migrated onto alternate infrastructure. We will be migrating these last few VMs over the next hour from smallest to largest. There should be no impact.
17/09/2017 15:20CLOSED
 Nz-sh-vmh2 restarted [New Zealand - Dedicated Servers]
 17/09/2017 15:20 - We have experienced an unexpected reboot of this host. Vms should be back online, please place a ticket if you are not back online
12/09/2017 12:45CLOSED
 Power outage [US (Fremont) - Public Internet]
 12/09/2017 12:45 - We have booted up all available devices after a brief complete power outage.

We are running on alternate routing avoiding this datacenter for any global transit or cdn services still at this time until we are confident of the cause.
 19/09/2017 04:13 - Routing has returned to normal
02/09/2017 20:31CLOSED
 Equinix POP Network Packetloss [AU (Sydney GlobalSwitch) - Public Internet]
 02/09/2017 20:31 - We have identified an issue starting at 20:11 AEST with our networking at this datacenter. We have implemented temporary routes to get services operating and have deactivated equinix peering until we get it back to normal.
23/08/2017 01:39CLOSED
 vmh55 storage [AU (Sydney GlobalSwitch) - Virtual Hosts]
 23/08/2017 01:39 - We're aware that storage performance is seriously degraded on this host. We are working on resolving it in a non intrusive manner.
 23/08/2017 05:44 - Performance is increasing as data is being shifted onto non-affected resources. We do not have an ETA at this stage other than to suggest observing your application performance to determine when your particular services will have been completely moved away from the affected resources.
 23/08/2017 09:36 - Migration has completed. please note that these hosts are running without SSD backing for speed, but theres 96 SATA drives so it should be somewhat okay performance.

We will contact each customer individually to arrange a good timeslot for migration to alternate storage as we'll retire vmh55 after this incident.
22/08/2017 18:16CLOSED
 Some prefixes being redirected overseas [AU (Sydney GlobalSwitch) - Public Internet]
 22/08/2017 18:16 - We have noticed some IP prefixes were being routed via long paths internationally. We have fixed this by way of a service restart and are investigating the incident further
15/07/2017 00:14CLOSED
 Network loss [AU (Sydney Equinix) - Public Internet]
 15/07/2017 00:14 - We're currently investigating the primary link connecting our GlobalSwitch SYD and Equinix SYD1 DCs being offline
 15/07/2017 00:25 - We've requested Equinix investigate the loss of link.
 15/07/2017 01:13 - Equinix has repaired the crossconnect and inter DC primary link operational again.
06/07/2017 10:56CLOSED
 Periodic Packetloss [AU (Sydney GlobalSwitch) - Public Internet]
 06/07/2017 10:56 - We are currently investigating some recurring periodic packetloss on the network
 06/07/2017 11:21 - We have isolated the issue to one of three devices and are troubleshooting further
 06/07/2017 12:13 - We've determined the failing device and are working on failing away from it without impact
 06/07/2017 12:52 - Device has been failed out of the routing path. Things should now be back to normal. If you're still noticing any issues, please raise a support ticket.
19/06/2017 23:21CLOSED
 Some services unreachable [Germany (Databurg) - Public Internet]
 19/06/2017 23:21 - We are currently investigating a handful of servers offline
 19/06/2017 23:51 - A tech is onsite and investigating
 20/06/2017 00:29 - Services have been restored after some faulty cabling was replaced
15/06/2017 01:47CLOSED
 Network Packetloss US [AU (Sydney GlobalSwitch) - Public Internet]
 15/06/2017 01:47 - We temporarily experienced some packetloss due to a large DDoS attack. We have applied some filters to reduce the attack profile. services should be back to normal
14/06/2017 08:36CLOSED
 Packetloss to some networks for 2 minutes [AU (Sydney GlobalSwitch) - Public Internet]
 14/06/2017 08:36 - We are aware of an outage to several networks between 8:23am and 8:25am. We are currently investigating
 14/06/2017 08:47 - We're still seeing high levels of international packetloss. We're working on routing around the upstream issue right now
 14/06/2017 10:19 - A large ddos attack has been blocked and issue resolved.
18/05/2017 13:31CLOSED
 Datacentee coolin failure [Germany (Databurg) - Public Internet]
 18/05/2017 13:31 - There has been a datacenter cooling failure in Interxion FRA10 which has progressively become worse. This has now reached a point where core routers have been forced to be shut down. We will advise as more information comes to hand.
 18/05/2017 13:41 - Datacenter engineers are onsite and are attempting to fix the problem.
 18/05/2017 14:30 - We understand this is a long outage. It is affecting a large number of services, not just RackCorp. We can only wait for the DC to restore cooling.
 18/05/2017 14:49 - One core router has been powered back up restoring a lot of connectivity.
 18/05/2017 16:25 - Cooling system is running again. As soon as temperature is back to an acceptable number, techs will turn on any equipment that was powered off
 18/05/2017 17:05 - Routers are now all back on. Any customer servers that were powered down are being powered back on now
18/04/2017 09:38CLOSED
 Network Packetloss [AU (Sydney GlobalSwitch) - Public Internet]
 18/04/2017 09:38 - We're aware of some packetloss to/from some networks a short time ago. We're currently investigating.
 18/04/2017 09:56 - Between 09:25 and 09:30 AEST today, one of our upstreams in sydney (Vocus) experienced an issue affecting several of their Sydney customers whereby our routes temporarily flapped up/down which caused inbound network issues in Sydney for many of our customers as various ISP network traffic reconverged.

Vocus is still investigating the cause. At this stage we have turned up preferencing on our other network providers in Sydney to limit any impact should it happen again.
13/02/2017 15:58CLOSED
 DNS resolution difficulties for some domains [AU (Sydney GlobalSwitch) - Public Internet]
 13/02/2017 15:58 - We are currently investigating this
 13/02/2017 16:24 - We understand we aren't the only network in Australia that had significant problems reaching nameservers for .au

At this time we believe the issue was related to servers running the .au level domain space and/or a networking issue with those that was well outside of our network
10/02/2017 09:22CLOSED
 Network unavailable from some locations [AU (Sydney Equinix) - Public Internet]
 10/02/2017 09:22 - We are currently investigating the network being unreachable for services in one of our racks in equinix sydney
 10/02/2017 13:03 - Issue is caused by a bad fibre that we're having replaced at this time.
 10/02/2017 13:13 - The existing cable is up and down, customers may notice services resume and then go back to a non-connectivity state while the new fibre is being laid.
 10/02/2017 14:45 - Services should now all be restored. If you're having any problems please raise a support ticket with us
09/02/2017 07:52CLOSED
 Packetloss to some limited networks [AU (Sydney GlobalSwitch) - Public Internet]
 09/02/2017 07:52 - We saw a few blips of ~50% packetloss between our Sydney POP and a major Sydney ISP as well as to a NZ ISP between 07:12 and 07:38 AEST. We're unsure exactly where the fault for this was but suspect it was with that major AU ISP, we're continuing to monitor things.
13/01/2017 17:14CLOSED
 Network Outage [AU (Sydney GlobalSwitch) - Public Internet]
 13/01/2017 17:14 - We're aware of a disconnection between two sets of our suites in GlobalSwitch causing suites on level 7 and one of the suites on level 6 to be disconnected. We have a tech approaching onsite now to investigate
 13/01/2017 17:55 - We have identified the network switch causing the problem and are working to isolate it entirely from the network. We hope to have this completed in the next 15 minutes
 13/01/2017 18:08 - We have restored access to level 7 suite a. We are working on restoring access to suite b.
 13/01/2017 18:10 - We believe at this stage that there may be some kind of power issue to blame in that suite.
 13/01/2017 18:25 - Primary power feed in our GlobalSwitch level 7 suite appeared to be cycling / surging quickly. We isolated equipment from this feed, and GlobalSwitch has now turned this power feed off entirely for the suite pending further investigation.

At this time we've replaced several network switches that failed to boot, but otherwise all VM's remain without issue.

A few customer routers/firewalls refuse to boot and we continue to work with those customers on a solution.
 13/01/2017 21:53 - We still have a network segment split between GlobalSwitch and Equinix. We've got some equipment coming in within the next hour which will resolve this. Only customers on private VLANs spanning GlobalSwitch and Equinix will be affected.
 15/01/2017 00:21 - Datacenter interconnect was restored last night. Private vlans between dcs should now be back to normal
28/12/2016 16:34CLOSED
 vmh70 rebooting [AU (Sydney GlobalSwitch) - Virtual Hosts]
 26/12/2016 17:25 - We've detected a freeze on vmh70 and have had to restart this host
23/12/2016 07:26CLOSED
 vmh52 storage slowdown [AU (Sydney GlobalSwitch) - Virtual Hosts]
 23/12/2016 07:26 - We are investigating a major slowdown in storage for this device causing issues across some customer VM's as well as some webhosting services
 23/12/2016 11:26 - Services have now been restored.
16/12/2016 12:29CLOSED
 Currently investigating some services offline at this pop [US (New York) - Public Internet]
 16/12/2016 12:29 - Tech is at datacenter already to investigate
 16/12/2016 14:46 - Looks like a hardware issue with one of the virtual hosts which is causing random reboots. We've got a tech addressing it now
 16/12/2016 16:08 - Tech is still working on hardware replacement
 16/12/2016 16:44 - Services are all now back online. If you are having any issues, please raise a support ticket
05/11/2016 17:29CLOSED
 Issue with vmh53 storage [AU (Perth DC1) - Public Internet]
 05/11/2016 17:29 - We are currently investigating
 05/11/2016 18:15 - Tech is on way to datacenter as it looks like a physical issue
 05/11/2016 19:20 - Services should be starting up again shortly
 05/11/2016 19:35 - Services are now all back online. If you are continuing to hae any problems please raise a support ticket. Issue was related to a failing drive causing problems with freezing up a raid controller.