We are back at scaled capacity. User facing/high prio tasks are already back to normal, and the backlog of bulk computing tasks is being reduced at the moment.
Posted Jun 18, 2020 - 12:51 CEST
Update
Deploys are arriving successfully on new nodes, which will allow for maximum capacity again; while cleanup tasks are still ongoing.
Posted Jun 18, 2020 - 12:28 CEST
Monitoring
We are deploying the fix across all nodes in the infrastructure now, and will re-enable all scaling mechanism.
All queued up requests and generations will recover.
Posted Jun 18, 2020 - 10:58 CEST
Identified
We have deployed a preliminary fix and are monitoring the results.
Posted Jun 18, 2020 - 10:39 CEST
Investigating
Due to a infrastructure bug, our content production and data processing infrastructure is running behind in scaling.
This is potentially causing delay in content delivering and data updates. No data has been or will be lost.
Training editing and real time production via Direct API is not affected.
Posted Jun 18, 2020 - 09:58 CEST
This incident affected: NLG Cloud (Data API, Text Engine, NEXT (v3)).