Hi Folks:
My system was down for 8-10 hours... from 9pm Eastern (9/15/08) through 7:45am on 9/16/08. Flashing red on the hub. I reset the hub several times with no success. No one to call at OOMA. No System messages posted.
a. What was going on?
b. Is there a 24 hr help line for something like this?
John
system down
Re: system down
I am up here in Los Angeles county....never had ANY downtime with Ooma..
Did you try plugging the Oooma directly behind the modem?
Did you try plugging the Oooma directly behind the modem?
Re: system down
We did have an outage related to the ooma Lounge and Activation last night. We're currently taking a look to see if any ooma Hubs were affected last night. I'll update the service update page with the details.
-Bobby
-Bobby
Bobby B
Re: system down
Ah, my ooma hub must have been one of those affected then!Bobby B wrote:We did have an outage related to the ooma Lounge and Activation last night. We're currently taking a look to see if any ooma Hubs were affected last night. I'll update the service update page with the details.
-Bobby
Re: system down
SivRami wrote:Ah, my ooma hub must have been one of those affected then!Bobby B wrote:We did have an outage related to the ooma Lounge and Activation last night. We're currently taking a look to see if any ooma Hubs were affected last night. I'll update the service update page with the details.
-Bobby
Those that were down: are you guys up and running now?
Re: system down
I was back up this morning. All is well. Would like to know what the problem was and that it was addressed. More importantly, not posts on line to let me know what was going on and when ETA of repair was expected. No tech support available. I have been "up" without incident for months and months. Not having someone to call and report the problem and having no internet report on time for repair is not good!!!
John
John
Re: system down
Hi John,
I apologize for your downtime. It is inexcusable for an outage like that to occur. As Bobby has indicated in his status post, the problem affected Hubs that were trying to register to the ooma network. This includes new installations and Hubs which were power-cycled. Unfortunately the mode of failure was a new one for us, so our automated testing and monitoring systems did not alert us to the problem and we did not discover the problem until this morning.
We did what was considered a minor upgrade of our database last night which triggered a bug in the open source database that we use. Because the bug was in the database itself, not the code that we deployed, the failure modes were different than the validation tests we ran after the upgrade (which tested our code).
We're doing a few things to avoid a repeat of such a scenario. First, we are upgrading our database to avoid the bug that tripped us up last night. Second, we are expanding our post-upgrade validation tests to encompass all common usage scenarios, regardless of whether the upgrade touches those functions. Third, we will expand our automated testing and monitoring systems to look for more modes of failure so that we can respond immediately.
Thank you for your patience and understanding. Hopefully I will not have to ask for forgiveness in the future.
Dennis
I apologize for your downtime. It is inexcusable for an outage like that to occur. As Bobby has indicated in his status post, the problem affected Hubs that were trying to register to the ooma network. This includes new installations and Hubs which were power-cycled. Unfortunately the mode of failure was a new one for us, so our automated testing and monitoring systems did not alert us to the problem and we did not discover the problem until this morning.
We did what was considered a minor upgrade of our database last night which triggered a bug in the open source database that we use. Because the bug was in the database itself, not the code that we deployed, the failure modes were different than the validation tests we ran after the upgrade (which tested our code).
We're doing a few things to avoid a repeat of such a scenario. First, we are upgrading our database to avoid the bug that tripped us up last night. Second, we are expanding our post-upgrade validation tests to encompass all common usage scenarios, regardless of whether the upgrade touches those functions. Third, we will expand our automated testing and monitoring systems to look for more modes of failure so that we can respond immediately.
Thank you for your patience and understanding. Hopefully I will not have to ask for forgiveness in the future.
Dennis