Elevated API Errors - LiveObjects

Ably·Transient load-balancer issue in us-west-1

Between 16:27-16:35 UTC, connection attempts to us-west-1 had an ~18% chance of being refused with an error due to a network load balancer issue. An Ably SDK will have been automatically retried to a fallback region, so an end user will have experienced no impact other than a slight increase in connection establishment latency.

Ably·Elevated connection errors in us-east-1

us-east-1 is reenabled and the service is operating normally. The root cause has been confirmed to be a defective Network Load Balancer that was causing a small fraction of connection attempts in the region to fail. We are awaiting a postmortem from AWS on the issue, and will publish our own postmortem here once we have that.

Ably·issue with rewind=1 and persist-last in some situations

Between about 14:00 UTC on the 20th and 17:00 on the 21st, on channels with the persist last message (https://ably.com/docs/channels#rules) channel rule enabled, attachments with `rewind=1` in situations where the most recent message was published over two minutes ago but still somewhat recently (and where the channel has been continuously active in a given region since the message was published) may not have had that rewound message correctly delivered to the attaching client. We apologise to anyone whose application was affected by this.

Ably·MongoDB ingress rule issues

Between 10:35 and 12:20 UTC today, MongoDB ingress rules may have experienced errors publishing changestream events. If you were affected by this, please contact support@ably.com.

Ably·Elevated API Errors

On October 1, a distributed denial-of-service \(DDoS\) attack caused elevated token-request errors across all Ably Realtime regions for approximately six minutes and intermittent unavailability of [ably.com](http://ably.com). We restored token-request service by increasing capacity and added targeted traffic-blocking and rate-limiting rules to mitigate the attack. We are strengthening endpoint protection and reviewing our firewall controls to reduce the risk of recurrence.