Portal is not available
Incident Report for Lightfoot
Postmortem

An exceptional data query conflicted with a routine system update (index removal), causing a table lock that stopped users from being able to log into the Fleet Management Portal.

Killing the process allowed all the held off (dashboard) processes to run. This took a little while to pin it down unfortunately; analysing the client connections on the db for rogue processes, we could see that a lot of processes were held waiting for a metalock on that table.

A new automated process that monitors for exceptions queries has been introduced so this cannot reoccur.

Posted Sep 16, 2020 - 15:20 BST

Resolved
This incident has been resolved.
Posted Sep 16, 2020 - 15:12 BST
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Sep 16, 2020 - 15:09 BST
Update
Recovery is imminent.
Posted Sep 16, 2020 - 15:08 BST
Identified
The issue has been identified and a fix is being implemented.
Posted Sep 16, 2020 - 15:01 BST
Investigating
We are currently investigating this issue.
Posted Sep 16, 2020 - 14:54 BST
This incident affected: Fleet Management Portal (my.lightfoot.co.uk).