RackCorp Managed Hosting
Dashboard View:   Timezone:   Datacenter:

Recent Maintenance / Outage Events

19/03/2018 10:03CLOSED
 Loss of Network [AU (Sydney GlobalSwitch) - Public Internet]
 19/03/2018 10:03 - We are investigating a loss of network incident
 19/03/2018 10:11 - Issue has been resolved.
03/03/2018 05:04CLOSED
 Packetloss issue Sydney [AU (Sydney GlobalSwitch) - Public Internet]
 03/03/2018 05:04 - We are investigating a packetloss issue
 03/03/2018 05:08 - We have disabled one upstream in sydney who are experiencing packetloss issues pending further information
 03/03/2018 05:30 - We've confirmed that provider is experiencing problems. We'll leave them turned off until things are restored
 03/03/2018 09:28 - Upstream provider has been re-added back into the mix
 03/03/2018 14:18 - Issue has been determined to be a ddos attack against one of their customers. Primarily only telstra traffic would have been affected.

We will work with them to better ensure these things do not affect us again.
26/02/2018 13:27CLOSED
 A segment of the network is experiencing large packetloss [AU (Sydney GlobalSwitch) - Public Internet]
 26/02/2018 13:27 - We are currently investigating
 27/02/2018 11:12 - We have found the culprit device which was a server with significant connectivity doing large volumes of localised traffic
22/02/2018 05:22CLOSED
 Routing change [US (Fremont) - Public Internet]
 22/02/2018 05:22 - We're noticing some routes are not available at this location. We are investigating and routing around the issue
 22/02/2018 05:24 - Issue is related to hurricane electric US Fremont going offline. We suspect they'll rectify this fairly soon.
 22/02/2018 07:29 - There is a power outage at Hurricane Electric's Fremont 1 Datacenter. We are awaiting for the DC to be brought back online.
19/02/2018 11:18CLOSED
 VM Host Offline [New Zealand - Dedicated Servers]
 19/02/2018 11:18 - Currently experiencing an issue with one of our New Zealand VM Hosts. This is affecting Servers situated on that VM Host.
 19/02/2018 11:24 - We are currently booting it back up. Looks to have been a temporary issue that we will investigate further once it is back up
 19/02/2018 11:34 - Services are all now back. If you're having any problems, please raise a support ticket. Apologies for the service disruption
 22/02/2018 07:28 -
15/02/2018 01:24CLOSED
 Loss of primary and secondary circuits [AU (Melbourne DC1) - Public Internet]
 15/02/2018 01:24 - We are currently investigating loss of all network access to this site.
 15/02/2018 01:43 - We believe an upstream vendor is doing maintenance that we do not believe we were notified about, that is affecting both our primary and secondary links. Their maintenance window is 30 minutes. We will work with the vendor to get things back running and sort out why both of our links are affected (they were not supposed to be)
 15/02/2018 01:49 - Services have just recovered. We are watching monitors clear now and will investigate any issues
13/02/2018 07:28CLOSED
 some outbound routes lost on legacy network [AU (Sydney GlobalSwitch) - Public Internet]
 13/02/2018 07:28 - We are currently investigating
 13/02/2018 07:57 - We have identified the issue to be a provider in a legacy part of our network. We are attempting to isolate them
 13/02/2018 07:59 - One of our upstreams has had a power outage at their datacenter which has caused several problems for them. We're still trying to isolate them from the network and have had good progress
 13/02/2018 08:39 - Issues are currently limited to our Sydney <-> Melbourne and Sydney <-> Perth traffic which is experiencing packetloss. We're attempting to migrate the legacy network traffic right now.
 13/02/2018 08:49 - We have turned off a significant amount of traffic between sites which should have resolved the majority of packetloss
 13/02/2018 08:56 - If you're still experiencing any issues, please raise a RackCorp support ticket.

We're planning to decommission the link that was affected so we'll be leaving it in it's current off status at this stage.
30/01/2018 12:07CLOSED
 loss of some network routes [AU (Sydney GlobalSwitch) - Public Internet]
 30/01/2018 12:07 - we are currently investigating
 30/01/2018 12:36 - We've currently got some routes in place to get around the majority of the issue. Looks like some international routes had the problems.
24/01/2018 19:15CLOSED
 Upstream Network Failover [AU (Sydney GlobalSwitch) - Public Internet]
 24/01/2018 19:15 - One of our upstreams has has a failure event. We have now switched them off. Some packetloss would have been noticed for 1-2 minutes.
 24/01/2018 20:09 - Cause for the outage has been identified and we do not expect it to occur again. Services have been put back to normal routing
11/01/2018 23:28CLOSED
 Packetloss to some destinations [AU (Sydney GlobalSwitch) - Public Internet]
 11/01/2018 23:28 - We are currently investigating packetloss in this location
 11/01/2018 23:37 - We have dropped an upstream provider who was delivering us erroneous routes and raised the incident with them. Network services should now be back to normal with complete reachability
03/01/2018 03:37CLOSED
 Packetloss issue Sydney [AU (Sydney GlobalSwitch) - Public Internet]
 03/01/2018 03:50 - Issue was related to several servers of a customer being hacked at the same time and used for DDoS attacks. They have now been disabled. Apologies for any service disruption.
14/11/2017 02:12CLOSED
 Packetloss [AU (Sydney Equinix) - Public Internet]
 14/11/2017 02:12 - We're currently investigating some packetloss over our Megaport link to Equinix SYD1
 14/11/2017 02:24 - We believe the link is once again stable
27/09/2017 01:07CLOSED
 Loss of network [New Zealand - Public Internet]
 27/09/2017 01:07 - We are investigating a loss of network that started a few minutes ago.
 27/09/2017 01:10 - It looks like an upstream provider in one of the NZ sites has lost all network connectivity. We'll await them to restore the network and will provide any updates as we hear them.
 27/09/2017 01:13 - Services are back up.
22/09/2017 04:36CLOSED
 vmh56 storage issue [AU (Sydney GlobalSwitch) - Virtual Hosts]
 22/09/2017 04:36 - We have had to restart a few VM's (All VMs that were >1TB storage and a couple of ones that had not been rebooted for a long time) that were attached to this storage node. Other ones were live migrated onto alternate infrastructure. We will be migrating these last few VMs over the next hour from smallest to largest. There should be no impact.
17/09/2017 15:20CLOSED
 Nz-sh-vmh2 restarted [New Zealand - Dedicated Servers]
 17/09/2017 15:20 - We have experienced an unexpected reboot of this host. Vms should be back online, please place a ticket if you are not back online
12/09/2017 12:45CLOSED
 Power outage [US (Fremont) - Public Internet]
 12/09/2017 12:45 - We have booted up all available devices after a brief complete power outage.

We are running on alternate routing avoiding this datacenter for any global transit or cdn services still at this time until we are confident of the cause.
 19/09/2017 04:13 - Routing has returned to normal
02/09/2017 20:31CLOSED
 Equinix POP Network Packetloss [AU (Sydney GlobalSwitch) - Public Internet]
 02/09/2017 20:31 - We have identified an issue starting at 20:11 AEST with our networking at this datacenter. We have implemented temporary routes to get services operating and have deactivated equinix peering until we get it back to normal.
23/08/2017 01:39CLOSED
 vmh55 storage [AU (Sydney GlobalSwitch) - Virtual Hosts]
 23/08/2017 01:39 - We're aware that storage performance is seriously degraded on this host. We are working on resolving it in a non intrusive manner.
 23/08/2017 05:44 - Performance is increasing as data is being shifted onto non-affected resources. We do not have an ETA at this stage other than to suggest observing your application performance to determine when your particular services will have been completely moved away from the affected resources.
 23/08/2017 09:36 - Migration has completed. please note that these hosts are running without SSD backing for speed, but theres 96 SATA drives so it should be somewhat okay performance.

We will contact each customer individually to arrange a good timeslot for migration to alternate storage as we'll retire vmh55 after this incident.
22/08/2017 18:16CLOSED
 Some prefixes being redirected overseas [AU (Sydney GlobalSwitch) - Public Internet]
 22/08/2017 18:16 - We have noticed some IP prefixes were being routed via long paths internationally. We have fixed this by way of a service restart and are investigating the incident further
15/07/2017 00:14CLOSED
 Network loss [AU (Sydney Equinix) - Public Internet]
 15/07/2017 00:14 - We're currently investigating the primary link connecting our GlobalSwitch SYD and Equinix SYD1 DCs being offline
 15/07/2017 00:25 - We've requested Equinix investigate the loss of link.
 15/07/2017 01:13 - Equinix has repaired the crossconnect and inter DC primary link operational again.
06/07/2017 10:56CLOSED
 Periodic Packetloss [AU (Sydney GlobalSwitch) - Public Internet]
 06/07/2017 10:56 - We are currently investigating some recurring periodic packetloss on the network
 06/07/2017 11:21 - We have isolated the issue to one of three devices and are troubleshooting further
 06/07/2017 12:13 - We've determined the failing device and are working on failing away from it without impact
 06/07/2017 12:52 - Device has been failed out of the routing path. Things should now be back to normal. If you're still noticing any issues, please raise a support ticket.
19/06/2017 23:21CLOSED
 Some services unreachable [Germany (Databurg) - Public Internet]
 19/06/2017 23:21 - We are currently investigating a handful of servers offline
 19/06/2017 23:51 - A tech is onsite and investigating
 20/06/2017 00:29 - Services have been restored after some faulty cabling was replaced
15/06/2017 01:47CLOSED
 Network Packetloss US [AU (Sydney GlobalSwitch) - Public Internet]
 15/06/2017 01:47 - We temporarily experienced some packetloss due to a large DDoS attack. We have applied some filters to reduce the attack profile. services should be back to normal
14/06/2017 08:36CLOSED
 Packetloss to some networks for 2 minutes [AU (Sydney GlobalSwitch) - Public Internet]
 14/06/2017 08:36 - We are aware of an outage to several networks between 8:23am and 8:25am. We are currently investigating
 14/06/2017 08:47 - We're still seeing high levels of international packetloss. We're working on routing around the upstream issue right now
 14/06/2017 10:19 - A large ddos attack has been blocked and issue resolved.
18/05/2017 13:31CLOSED
 Datacentee coolin failure [Germany (Databurg) - Public Internet]
 18/05/2017 13:31 - There has been a datacenter cooling failure in Interxion FRA10 which has progressively become worse. This has now reached a point where core routers have been forced to be shut down. We will advise as more information comes to hand.
 18/05/2017 13:41 - Datacenter engineers are onsite and are attempting to fix the problem.
 18/05/2017 14:30 - We understand this is a long outage. It is affecting a large number of services, not just RackCorp. We can only wait for the DC to restore cooling.
 18/05/2017 14:49 - One core router has been powered back up restoring a lot of connectivity.
 18/05/2017 16:25 - Cooling system is running again. As soon as temperature is back to an acceptable number, techs will turn on any equipment that was powered off
 18/05/2017 17:05 - Routers are now all back on. Any customer servers that were powered down are being powered back on now
18/04/2017 09:38CLOSED
 Network Packetloss [AU (Sydney GlobalSwitch) - Public Internet]
 18/04/2017 09:38 - We're aware of some packetloss to/from some networks a short time ago. We're currently investigating.
 18/04/2017 09:56 - Between 09:25 and 09:30 AEST today, one of our upstreams in sydney (Vocus) experienced an issue affecting several of their Sydney customers whereby our routes temporarily flapped up/down which caused inbound network issues in Sydney for many of our customers as various ISP network traffic reconverged.

Vocus is still investigating the cause. At this stage we have turned up preferencing on our other network providers in Sydney to limit any impact should it happen again.