18 Jun 2024
08:30	 We're doing part-1 of a major server cluster upgrade.
	 Various servers will experience a brief outage as they are
	 migrated to the new environment. Most will be around 2 minutes
	 downtime, but a few are going to be about an hour.
	 Completed by 12:30 with only a couple of minor glitches.
	 Part-2 planned for tomorrow.

06 Jun 2024
12:50	 Loss of multiple services on 203.27.67.0/24 subnet. This would
	 have only affected a small number of customers with specialty
	 services. Caused by a misconfiguration of a router upstream of
	 our servers and resulting ARP storm which required physical
	 corrective action at the datacentre.
	 Services all restored at 14:36.

23 Jan 2024
00:21	 Loss of all NBN connected services.
	 Upstream advises dark fiber failure. Services restored 01:48

06 Nov 2023
07:07	 A number of customers services all went offline at the same time.
	 Upstream can't see any planned maintenance or notice of fault.
	 Updated at 9:10 to say major outage at Albury POI. No ETR.
	 Services restored 10:03, waiting on PIR (Post Incident Report)

22-Dec-2022
07:03	 A SAN controller failure has seen several servers go offline. Engineers
	 got replacement parts quickly. Most services were back running at 8:45
	 but one server took forever to complete file system consistency checks
	 and didn't come back online until 9:37.
	 Auth, Phone and email all affected. Disruption to DNS will have affected
	 various unrelated services until restoration was complete.

20-Nov-2022
01:40	 Loss of connectivity to a small number of hosts (especially rural.net.au)
	 3rd-party router is showing multiple failures. Engineers working on it.
	 Router restored 02:30, however only internal traffic is flowing. More
	 analysis underway. 06:45 corrupted firewall rules rebuilt and all working.

01-Jul-2022
22:30	 BGP failure on core router saw pretty much all connectivity lost.
	 Reload of router by technicians at 23:00 restored service.
	 Investigations continuing.
	 
02-Jun-2022
19:00	 Network connections in WAIT_STATE inexplicably increasing exponentially.
	 No source identified, performance appears unaffected but not comfortable
	 leaving it without explanation. Subsystem being restarted after 374 days.
	 
06-Apr-2022
16:40	 Remarkably similar outage to 18-Feb. Sudden loss of all connectivity.
	 Problem identified as failed Redback. Engineers despatched to the
	 datacentre to install/configure the spare. Services restored 18:39
	 Waiting on post-incident report.

18-Feb-2022
11:25	 Sudden loss of all connectivity to all customers and all servers.
	 Upstream experienced a "power event" within the datancentre that
	 has destroyed both primary and secondary power supplies on their
	 aggregation switch, effectively isolating every part of the network.
	 Engineers on site installing backup router, and rebuilding configuration.
	 Seeing customers come back online at 14:12, restoring most of our core
	 infrastructure including authentication, mail and phones. Some servers
	 are still unreachable, we have no ETR but it is being worked on as a
	 priority.
	 
25-May-2021
02:00	 Over the last couple of hours, performance issues have been observed
	 with several hosts, ultimately being identified as a failure of a SAN.
	 Due to subsequent loss of ISCSI connections, multiple servers have
	 rebooted. Although most servers were restored soon after, phone, alarms,
	 monitoring, weather, graphing, some DNS and a few other services were
	 unable to be restored until 09:00, although all services have degraded
	 performance.
	 Nominal performance restored around 14:30.
	 
15-May-2021
09:00	 Have lost all our meteorological, albury cameras, lightning tracking,
	 amateur radio wormhole, IRLP, ADSB, websdr etc feeds due to being out
	 of town and suffering yet another UN-NOTIFIED NBN outage. At least it
	 was only 6 hours not the 12 they now suggest it may take! Restored just
	 before 15:00. How many more times, nbn???

25-Mar-2021
06:30	 We are advised there is a "Major incident impacting various POIs in NSW.
	 Services are experiencing no data flow. POIs include: Albury, Campbelltown,
	 Coffs Harbour, Gosford, Hamilton, Liverpool, Maitland, Nowra, Tamworth,
	 Wagga, Berkeley Vale. Engineers are investigating with highest priority.