Canuck Audio Mart Hifi and Audio ForumCanuck Audio Mart Hifi and Audio Forum
It is currently Mon Oct 15, 2018 9:36 pm

All times are UTC - 8 hours




Post new topic Reply to topic  [ 17 posts ]  Go to page 1, 2  Next
Author Message
PostPosted: Thu May 17, 2018 12:05 pm 
Offline
Site Admin
User avatar

Joined: Wed Apr 21, 2004 6:03 pm
Posts: 1349
Location: Vancouver, BC, CA
We just wanted you to know that last night at around 2:07AM PST (5:07AM EST) CAM and USAM went down and remained down for around 6hrs, until 8am PST/11am EST. This type of extended non-planned downtime is painful, but thankfully not too frequent. In this particular case, one of our systems that help with scam and abuse prevention went off the rails and took up more resources than it should have, while a series of checks and balances against that type of thing failed to alert us in time.

I believe we've brought the sites back up to full operation now, and we will monitor the situation further. In addition to this, we may schedule some additional maintenance tonight past midnight EST in order to clean up additional collateral consequences of the downtime.

I'm also auditing some of our processes and checks to try and minimize this type and length of downtime, as one of our priorities has always been a fast running and stable site.

Last night's downtime is particularly frustrating because Wednesday morning we had a brief 20minute outage due to routine maintenance gone awry. Please rest assured that we take the site's stability and availability seriously and invest significant time and resources towards keeping the site running smoothly and reliably, and we hope that after last night and after additional work is identified that we can return to the level of reliable service you've come to expect from CAM/USAM.

Thanks for your patience when we encounter these hiccups, and sorry for the inconvenience this has caused.


Top
 Profile  
Reply with quote  
PostPosted: Thu May 17, 2018 12:22 pm 
Offline

Joined: Mon Aug 08, 2016 5:46 pm
Posts: 923
Location: Markham, ON, CA
Thanks for all your hard work and dedication. We all appreciate there are small bumps in the technological road we are all on. Thanks for keeping us up to date as well.


Top
 Profile  
Reply with quote  
PostPosted: Thu May 17, 2018 12:54 pm 
Offline
Premium User
User avatar

Joined: Wed Aug 15, 2012 7:16 pm
Posts: 1411
Location: Kelowna, BC, CA
I work for one of the largest companies in the world. Trust me when I say the downtime was but a blip at our end. We appreciate the effort.

_________________
"Im...a rather simple person with a limited talent and perhaps a limited perspective" - Bill Evans

Ariston RD11 Superieur, RB250 & Tecnoweight, DV20X2H, LCR MkIII, Dacmagic100, Exposure 2010S2, AS Statement SEs, AS Testaments, PSB GT1s


Top
 Profile  
Reply with quote  
PostPosted: Thu May 17, 2018 4:00 pm 
Offline
User avatar

Joined: Tue May 21, 2013 12:54 pm
Posts: 6394
Location: Dodge City, NB, CA
I only became slightly unnerved. :wink:

_________________
BE KIND TO THE UNKIND...THEY NEED IT THE MOST.


Top
 Profile  
Reply with quote  
PostPosted: Thu May 17, 2018 4:51 pm 
Offline

Joined: Sat Oct 24, 2009 11:36 am
Posts: 185
Location: Aldergrove, BC, CA
missed my morning fix but I'm all right now.
Cam on.


Top
 Profile  
Reply with quote  
PostPosted: Thu May 17, 2018 5:30 pm 
Offline

Joined: Sun Apr 05, 2009 7:06 pm
Posts: 1258
Location: hawkesbury, ON, CA
cam is my homepage, and this morning when it tried to start internet it kept crashing.so I reboot my modem twice , to notice it was cam. who was down. DA !


Top
 Profile  
Reply with quote  
PostPosted: Thu May 17, 2018 5:53 pm 
Offline
User avatar

Joined: Wed Jan 26, 2011 1:08 pm
Posts: 57
Location: London, ON, CA
Good to have you back!


Top
 Profile  
Reply with quote  
PostPosted: Thu May 17, 2018 7:43 pm 
Offline

Joined: Mon Jan 25, 2010 6:43 am
Posts: 115
Location: Kelowna, BC, CA
admin wrote:
We just wanted you to know that last night at around 2:07AM PST (5:07AM EST) CAM and USAM went down and remained down for around 6hrs, until 8am PST/11am EST. This type of extended non-planned downtime is painful, but thankfully not too frequent. In this particular case, one of our systems that help with scam and abuse prevention went off the rails and took up more resources than it should have, while a series of checks and balances against that type of thing failed to alert us in time.

I believe we've brought the sites back up to full operation now, and we will monitor the situation further. In addition to this, we may schedule some additional maintenance tonight past midnight EST in order to clean up additional collateral consequences of the downtime.

I'm also auditing some of our processes and checks to try and minimize this type and length of downtime, as one of our priorities has always been a fast running and stable site.

Last night's downtime is particularly frustrating because Wednesday morning we had a brief 20minute outage due to routine maintenance gone awry. Please rest assured that we take the site's stability and availability seriously and invest significant time and resources towards keeping the site running smoothly and reliably, and we hope that after last night and after additional work is identified that we can return to the level of reliable service you've come to expect from CAM/USAM.

Thanks for your patience when we encounter these hiccups, and sorry for the inconvenience this has caused.


I worked 20+ years for a major software developer with offices throughout the US and Canada and over one thousand employees. Over the top high tech, I was on the senior technical operations management side of things. Our entire network of websites and several online products that are about 100 times more complex than the CAM site, were brought to their knees for 3 full days by a heavy DDOS attack. Nothing worked in that time, no customers could access their mission critical data that is essential to their daily business operations.

CAM has a great uptime record, I would not sweat it and just keep improving what you already do well.


Top
 Profile  
Reply with quote  
PostPosted: Fri May 18, 2018 4:32 am 
Offline

Joined: Sun Dec 13, 2015 9:36 am
Posts: 611
Location: scarborough, ON, CA
I was a little shaky, but managed to stay back from full blown panic.
I'm ok now... don't ever leave with out saying good bye.


Top
 Profile  
Reply with quote  
PostPosted: Fri May 18, 2018 4:40 am 
Offline
User avatar

Joined: Mon Jun 19, 2006 3:45 pm
Posts: 21065
Location: Panache Bay
KJT1 wrote:
I only became slightly unnerved. :wink:


...ich auch :lol:

_________________
It is certain, in any case, that ignorance, allied with power, is the most ferocious enemy justice can have. James A Baldwin


Top
 Profile  
Reply with quote  
PostPosted: Fri May 18, 2018 4:46 am 
Offline
User avatar

Joined: Fri Dec 02, 2011 8:49 am
Posts: 165
Location: Saint-Augustin-de-Desmaures, QC, CA
Admin,

Time to recognize your efforts and the great quality of service delivered:

Many thanks for your work that brings many of us a great deal of pleasure both online and in our listening rooms!

_________________
I like gear and good sound but in the end it’s all about music.


Top
 Profile  
Reply with quote  
PostPosted: Fri May 18, 2018 4:50 am 
Offline
User avatar

Joined: Mon Oct 17, 2011 6:55 pm
Posts: 3011
Location: Toronto, ON, CA
ppat2 wrote:
admin wrote:
We just wanted you to know that last night at around 2:07AM PST (5:07AM EST) CAM and USAM went down and remained down for around 6hrs, until 8am PST/11am EST. This type of extended non-planned downtime is painful, but thankfully not too frequent. In this particular case, one of our systems that help with scam and abuse prevention went off the rails and took up more resources than it should have, while a series of checks and balances against that type of thing failed to alert us in time.

I believe we've brought the sites back up to full operation now, and we will monitor the situation further. In addition to this, we may schedule some additional maintenance tonight past midnight EST in order to clean up additional collateral consequences of the downtime.

I'm also auditing some of our processes and checks to try and minimize this type and length of downtime, as one of our priorities has always been a fast running and stable site.

Last night's downtime is particularly frustrating because Wednesday morning we had a brief 20minute outage due to routine maintenance gone awry. Please rest assured that we take the site's stability and availability seriously and invest significant time and resources towards keeping the site running smoothly and reliably, and we hope that after last night and after additional work is identified that we can return to the level of reliable service you've come to expect from CAM/USAM.

Thanks for your patience when we encounter these hiccups, and sorry for the inconvenience this has caused.


I worked 20+ years for a major software developer with offices throughout the US and Canada and over one thousand employees. Over the top high tech, I was on the senior technical operations management side of things. Our entire network of websites and several online products that are about 100 times more complex than the CAM site, were brought to their knees for 3 full days by a heavy DDOS attack. Nothing worked in that time, no customers could access their mission critical data that is essential to their daily business operations.

CAM has a great uptime record, I would not sweat it and just keep improving what you already do well.


Fully agree. Over the course of my membership, I can’t recall any major outages. These things happen, even with large companies investing greatly into contingencies. But what is most appreciated is the time Admin took to inform its customers and detail what actually happened plus warn of further outages due to maintenance. A trait not common within the tech industry. Well done Admin!!


Top
 Profile  
Reply with quote  
PostPosted: Fri May 18, 2018 5:22 am 
Offline

Joined: Sun Jan 05, 2014 5:18 pm
Posts: 99
Location: etobicoke, ON, CA
Echoing the sentiment of previous posters, it's amazing that CAM is so consistently available and works so well. This was a tiny blip on the radar of a fantastic community that has given us so much.

I'll add also that with the site being down, I've come to appreciate it that much more!


Top
 Profile  
Reply with quote  
PostPosted: Fri May 18, 2018 7:31 am 
Offline
User avatar

Joined: Fri Mar 11, 2005 9:30 am
Posts: 1240
Location: Sainte-Marguerite-du-Lac-Masson, QC, CA
jazzbert wrote:
Admin,

Time to recognize your efforts and the great quality of service delivered:

Many thanks for your work that brings many of us a great deal of pleasure both online and in our listening rooms!


+1

_________________
I gave up everything for you, 'Cept the Blues.
Joe Bonamassa, 2014


Top
 Profile  
Reply with quote  
PostPosted: Fri May 18, 2018 10:11 am 
Offline

Joined: Sun Jun 18, 2017 7:17 am
Posts: 15
Location: Calgary, AB, CA
I used to run >5,000 computer infrastructure (+++ stuff). Not a lot of fun when it goes sideways.
I too nearly went for the scotch cabinet ... then recovered.
Through this site I've been able to meet some great people, and purchase some great gear.
Thanks for your hard work.


Top
 Profile  
Reply with quote  
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 17 posts ]  Go to page 1, 2  Next

All times are UTC - 8 hours


Who is online

Users browsing this forum: No registered users and 0 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB® Forum Software © phpBB Group