Bots Playground is fully operational again. All scripts are now running smoothly.
The Playground was originally launched as a proof-of-concept to support a few dozen user scripts. Since then, it’s grown significantly and now runs several thousand scripts simultaneously. This scale has brought engineering challenges.
Issue details
During our recent platform migration, we upgraded the JavaScript sandboxes to run across multiple servers. However, a configuration oversight (specifically, a network socket limitation on the main Playground process) caused an unexpected cluster-wide crash.
(more…)
Some of our bots had serious problems today: 3 servers crashed after installing recent OS updates. Our tech guys together with provider’s support were trying to fix the problem all the day.