Resolved -
A necessary upgrade to the Dentally database performed on the weekend caused significant performance degradation. Despite comprehensive testing, this had not been observed prior to Monday. Isolating and resolving the issue took longer than expected, however full performance has been restored. No customer data was lost during this period. We are now analysing the root cause and the changes we can make to avoid such an incident occurring in future.
Nov 10, 11:37 GMT
Update -
Service was fully restored at 12.30 today. We will continue enhanced monitoring through until 10am tomorrow after the peak of users logging in during UK hours.
Audit data will be available by 9am tomorrow morning.
Nov 9, 14:41 GMT
Update -
We are continuing to monitor for any further issues.
Nov 9, 13:06 GMT
Monitoring -
Fixes have been applied. Calendar and Appointment functionality has been restored. Our monitoring shows that performance is back to normal.
We are monitoring the situation.
Audit data will be delayed and made available later today.
Nov 9, 12:26 GMT
Identified -
All users accounts that were locked out during the issue have been unlocked and users should now be able to access the application.
We have identified the cause of the issue and are looking into applying a fix so that the calendar can be enabled again. We will then look to begin restoring other services (Audit and Exports).
Nov 9, 11:48 GMT
Update -
We are seeing performance increases while the Calendar is turned off.
As part of our investigation we turned off other areas of the system that remain impacted: audit data will be delayed - no data will be lost, and exports will be delayed. We are looking into users who were unable to login and will look to unlock their accounts who were affected by this issue.
Nov 9, 11:29 GMT
Update -
We are temporarily disabling the calendar view from loading appointments. Users can view the appointments for a patient by searching for that patient and navigating to the “appointments” tab. Users will see errors when loading the calendar.
We continue to investigate the cause of slow application performance.
Nov 9, 11:16 GMT
Update -
We paused ability to view appointments, including the calendar, between 10:54 and 10:55. We see an improvement over that time.
We are working with our system hosting provider to understand performance issues we have with out database. We are also now narrowing our investigation to appointment handling.
Nov 9, 11:02 GMT
Update -
To investigate if we have an issue with Appointments we need to turn off the ability to view Appointments for 1 minute.
So far it has been possible to view the calendar. For this 1 minute window it will not be possible to view Appointments or the Calendar. Users will see an error.
Nov 9, 10:44 GMT
Update -
We are continuing to investigate this issue. We are turning off the audit log and will backfill the data later in the day. This is to reduce the load on the database. We have also paused processing of some background jobs, such as Recall processing.
Nov 9, 10:15 GMT
Update -
We continue to investigate the issue which is affecting the whole application (User Interface, APIs and Portal). Users will experience difficulty logging in and using any part of the application workflows.
Nov 9, 09:36 GMT
Update -
We are continuing to investigate this issue.
Nov 9, 09:32 GMT
Update -
We continue to investigate the root cause of the issue. At present we are increasing the system capacity to improve performance.
Nov 9, 09:05 GMT
Investigating -
Starting at 8am this moring users began experiencing application performance issues. User interface slow or not responding. We are investigating.
Nov 9, 08:38 GMT