Sep 13, 17:33 EDT
This incident has been resolved.
Sep 13, 17:15 EDT
The team has concluded the implementation of all temporary solutions on the platform.
This includes creating a timeout option on slow connecting sockets and also disabled Redis as a single connection.
The web socket chat will remain with limited chat support on the firebase instance until we add support for per instance local connection, which should happen next week.
The Redis cache team will be implementing a new permanent solution end of this week, Friday at the latest.
Sep 13, 13:07 EDT
We currently fixed an autoscaling Redis instance that was not able to redirect the load to a separate system. Redis is used by InEvent to quickly balance its write operations instead of relying only on a traditional SQL database.
The following components are still affected:
Firebase Web Sockets, which covers the live chat on the Virtual Lobby.
Sep 13, 11:45 EDT
We have identified the slowness from the Firebase product.
We are deploying an alternative with the Native websockets, under Company > Tools.
We are currently deploying a fix for UI improvements on the Firebase UI console while Firebase is not responsive.
Chat may be offline while the fix is being applied, but video and streaming should work normally.
Sep 13, 10:55 EDT
We are currently seeing a slow scalability response from the InEvent Firebase socket group. We are currently investigating the issue with Google Firebase team.
Sep 13, 09:45 EDT