Overview:
We have conducted our postmortem and root cause analysis and identified that the outage was caused by an incompatibility with a server update that occurred overnight. This caused a number of our services to have connectivity issues. As a consequence of this we have and are continuing to make a number of changes to our systems and processes to reduce the risk of such an outage occurring again.
Our reliability as a service is core to our commitment to our customers, and we are deeply sorry for the disruption. Thank you again for your patience.
Summary of Events:
6:30am AEST – Alert has been received that customers are having login issues
6:58am AEST – We have resolved the log in issue. Customers are able to log into Wisenet Portal and LRM.
08:01am AEST – Further support has been received that customers are not able to add logbook records, run reports or access Learner App
08:16am AEST – Identified that while LRM is functional, many supporting services are unavailable. We are investigating the issue.
10.30am AEST – Root cause has been identified. Applying an update to the test environment to validate that the fix will work.
12.50pm AEST – Testing has been successfully completed on the test environment. Scheduling full maintenance outage for Live.
01.15pm AEST – Maintenance Mode has commenced in order to apply fix on production
02.24pm AEST – Testing has started on production to verify all identified issues are fixed
02:45pm AEST – Maintenance Mode has been switched off. Almost all services are now functioning
03:13pm AEST – Error has been identified accessing the View Todays Exports page for some customers
03:42pm AEST – Customers are not able to receive LRM portal and Learner App based emails
04:42pm AEST – Export error has been resolved
06:11pm AEST – Incoming and outgoing emails from the portal has been resolved
Service Health