Isolated Outbound Calling Trouble

Resolved
{:closed=>"Closed", :complete=>"Complete", :false_alarm=>"False Alarm", :identified=>"Identified", :investigating=>"Investigating", :open=>"Open", :recovering=>"Recovering", :resolved=>"Resolved", :scheduled=>"Scheduled", :underway=>"Underway"}
After 5 days, 9 hours, and 29 minutes

On June 11th, 2015 we responded and resolved an issue affecting a few Hosted Phone Platform-based users that may have been prevented from making outbound calls. In brief, a small number of customers experienced increased errors with outbound calling for a period of time. Inbound calling, system features such as voicemail and menus, and the web portal for Platform were not impacted by the issue.

In brief, we began to receive reports about trouble making outbound late Wednesday afternoon. We began to investigate each of the reported issues. All other services like inbound calling and features were not affected. To investigate further, we issued a maintenance and preformed network diagnostics that evening. Our primary goal for these users was to quickly resolve the issue, but also not further impact the users from receiving calls while investigating.

We continued the maintenance throughout the night, and identified a problematic database server in our network stack as the culprit. The team worked meticulously to maintain all other services during that time (inbound calling and system features for example). By 430pm PST Thursday, all remaining users were restored.

On Friday June 12th, we personally contacted users who reported the trouble, including those who we detected may have been impacted but did not report trouble.

Since the event, our engineering and developers has been working hard to better understand the moments leading up to the issue, and are implementing a series of improvements to prevent future trouble. Since Wednesday night, our product developers and engineers have continued to work on these details with the project teams behind the database technology we use. This work has already led to improvements and enhancements.

If you have any question regarding this event, or if you are experiencing any trouble, please contact customer support anytime.

{:closed=>"Closed", :complete=>"Complete", :false_alarm=>"False Alarm", :identified=>"Identified", :investigating=>"Investigating", :open=>"Open", :recovering=>"Recovering", :resolved=>"Resolved", :scheduled=>"Scheduled", :underway=>"Underway"}
After 14 hours and 20 minutes

For the remaining customers still experiencing trouble with outbound calling, we've planned a special maintenance this evening to migrate their accounts away from the systems causing their earlier trouble. While a small fraction of our customers were effected, we'll continue monitoring and working with our development and engineering teams. Additional details regarding today's issue will be updated into a Incident Report/RFO and our accounting team will work with customers receiving SLA credits/refunds.

{:closed=>"Closed", :complete=>"Complete", :false_alarm=>"False Alarm", :identified=>"Identified", :investigating=>"Investigating", :open=>"Open", :recovering=>"Recovering", :resolved=>"Resolved", :scheduled=>"Scheduled", :underway=>"Underway"}
After 10 hours and 56 minutes

We are still monitoring a few customers who are still reporting isolated trouble with outbound calling. Our developer and engineering team is looking over a few final items before our next update shortly. We greatly apologize to those impacted by this issue, and we'll be updating our remaining affected customers on the cause soon.

{:closed=>"Closed", :complete=>"Complete", :false_alarm=>"False Alarm", :identified=>"Identified", :investigating=>"Investigating", :open=>"Open", :recovering=>"Recovering", :resolved=>"Resolved", :scheduled=>"Scheduled", :underway=>"Underway"}
After 7 hours and 52 minutes

Our team is still aware of a few customers impacted by this issue. We're continuing to work and monitor the trouble with our engineering and developers. The team is working towards a full resolution as quickly as possible, and will continue to update this here and with our customers.

{:closed=>"Closed", :complete=>"Complete", :false_alarm=>"False Alarm", :identified=>"Identified", :investigating=>"Investigating", :open=>"Open", :recovering=>"Recovering", :resolved=>"Resolved", :scheduled=>"Scheduled", :underway=>"Underway"}
After 3 hours and 45 minutes

Good morning! We've opened Customer Support at 530am PST this morning to assist and monitor customers reporting trouble. Our engineers are actively working to resolve the remaining customers experiencing trouble. We will continue to update here, and publish a full incident report upon resolution.

{:closed=>"Closed", :complete=>"Complete", :false_alarm=>"False Alarm", :identified=>"Identified", :investigating=>"Investigating", :open=>"Open", :recovering=>"Recovering", :resolved=>"Resolved", :scheduled=>"Scheduled", :underway=>"Underway"}

An issue we worked on late Tuesday affecting a small percentage of Hosted Phone customers appears to be trouble again this morning. If you experience trouble outbound calling such as calling a phone number or n extension, you may be affected. We've identified an issue with one our systems that's causing trouble for these users, and working to resolve it. Our developers and engineers are working around the clock to fully resolve this issue, and we greatly apologize for the inconvenience. Please contact Customer Support for additional assistance.

Began at: