PDA

View Full Version : CF Database corrupted again on 27 January with 28 hour downtime, now fixed!


Erwin
27th January 2005, 07:17 PM
As members would know, CF had a database crash on 25 January caused by the wrong setting in a config file - I fixed it in about 12 hours with no data loss.

Well, I didn't learn my lesson and kept modifying the database config to try to optimize the database so I could put our "find member's posts" feature back in.
It worked but because of the way I recoded things, the database was cacheing a lot of disk writes into RAM, and this caused occasional writes not to be written - leading to another database corruption.

This database corruption affected the WHOLE database - I had to restore our backups 3 times but they ended up being corrupted, and in the end decided to try to fix up the corrupted database. Took about 12 ours again...

Praise God, I fixed it, with minimal or no data loss whatsoever.

So now CF is back. :)

I promise I won't try to be too smart now. I'll leave things alone!

AngelAmidala
27th January 2005, 07:20 PM
:bow: All hail you Erwin! :)

MKalashnikov
27th January 2005, 07:24 PM
Good Job Erwin.

Deamiter
27th January 2005, 07:24 PM
Horray for fixing a 24 hour problem! Just out of curiosity, IS the search feature back, or have you learned your lesson there too?

Marie
27th January 2005, 07:24 PM
Woo hoo! :clap:

Erwin
27th January 2005, 07:29 PM
Horray for fixing a 24 hour problem! Just out of curiosity, IS the search feature back, or have you learned your lesson there too?
I couldn't make it work without the database going crazy.

I'm now at work, so I'm going to have to concentrate. :) I woke up at 4 am this morning to work on CF... it's been a crazy CF week for me.

sinner/SAVED
27th January 2005, 07:30 PM
You are amazing, Erwin!

Plan 9
27th January 2005, 07:32 PM
Erwin, thanks for trying to restore the "find members' posts" feature; I really miss it. :sigh:

Iron Sun 254
27th January 2005, 07:35 PM
Thanks for all the hard work you do to keep this place running. I'm lucky that I never worry about being too smart.

tjboie2001
27th January 2005, 07:36 PM
Thank you Erwin for fixing CF

ukok
27th January 2005, 07:39 PM
Erwin, we already know you are smart, you don't need to play around doing fancy stuff to prove it ;)

Well done!

God Bless.

Revelations 22:20
27th January 2005, 07:42 PM
Go Erwin :clap: :clap: :clap: :clap: :clap:

brother daniel
27th January 2005, 07:43 PM
Thank you Erwin God is with you

CPman2004
27th January 2005, 07:46 PM
Thanks Erwin for all the hard work. I guess the mini vacation for staff is done now ;)

brinny
27th January 2005, 07:46 PM
Praise God!!! I was just praying before I logged in. God bless you Erwin!!! :clap:

TheBear
27th January 2005, 07:53 PM
As members would know, CF had a database crash on 25 January caused by the wrong setting in a config file - I fixed it in about 12 hours with no data loss.

Well, I didn't learn my lesson and kept modifying the database config to try to optimize the database so I could put our "find member's posts" feature back in.
It worked but because of the way I recoded things, the database was cacheing a lot of disk writes into RAM, and this caused occasional writes not to be written - leading to another database corruption.

This database corruption affected the WHOLE database - I had to restore our backups 3 times but they ended up being corrupted, and in the end decided to try to fix up the corrupted database. Took about 12 ours again...

Praise God, I fixed it, with minimal or no data loss whatsoever.

So now CF is back. :)

I promise I won't try to be too smart now. I'll leave things alone!

Thanks, Erwin! :wave:

rachewil15
27th January 2005, 08:02 PM
Muchas Gracias Erwin!!

Sephania
27th January 2005, 08:23 PM
I was praying for you Erwin as I am sure you were tearing your hair out trying to do this and have work and everything.

Just a quick question, I noticed when it first went haywire that while you could get into the opening page and chat but not the individual forums, you could access the PM and send and receive. Now I don't know a whole lot about databases, but this would seem to me to be on a different one? The PM's I mean. So if that is true, couldn't you somehow split the forums so as the "lighter" forums would be on a data base where you can't search for posts and the more "in-depth" forums would be on another that would have the search capability?

Just thinking, it really is a lose but I dont' think a necessity for the whole of CF.

Glad you can rest now, I don't know how you do it, the L-RD has blessed you mightely!

Palatka44
27th January 2005, 08:25 PM
Erwin,
I thought that I had caused CF to crash as I had just changed my Signature when every thing went down. Whew! I am not the cause, the burdens been lifted and now CF is back!
May you always have more smarts then me.:D

OrthodoxyUSA
27th January 2005, 08:29 PM
Erwin,

I cant sometimes can't imagine why you would want to keep up with this.....

God Bless You...

Forgive me..:liturgy:

bubblefish
27th January 2005, 08:30 PM
Well done Erwin :) You really do a good juob keeping the forums running...

Cin
27th January 2005, 08:31 PM
Thanks Erwin! Great job as usual. :thumbsup:

Alexis OCA
27th January 2005, 08:33 PM
Thank you for all you do Dr. Loh.

I_are_sceptical
27th January 2005, 08:43 PM
Thank you very much, Erwin. I was bored while CF was down. I spend more time here than on any other forum.

Did anyone see the Google link at the bottom of page 1 for "new and used Erwins"? I think we are doing just fine with the one we already have.

JunkYardFrog
27th January 2005, 09:26 PM
I'll leave things alone!

There's an old expression we old mechanics use. I'm sure you've heard it before:

"If it ain't broke, don't fix it!"

I know it doesn't exactly apply here, but I haven't had a chance to use it for a long time....

:D

Anyway, thank you for your tremendous work for our Lord. I think it's entirely possible that some people we will meet in heaven will be there as a result of this message board. Please make sure you take enough time to take of YOU also. I hope you have enough helpers here to lighten your load.

God bless you, my brother.

Good night,

Bruce

Zacharias
27th January 2005, 09:35 PM
As members would know, CF had a database crash on 25 January caused by the wrong setting in a config file - I fixed it in about 12 hours with no data loss.

Well, I didn't learn my lesson and kept modifying the database config to try to optimize the database so I could put our "find member's posts" feature back in.
It worked but because of the way I recoded things, the database was cacheing a lot of disk writes into RAM, and this caused occasional writes not to be written - leading to another database corruption.

This database corruption affected the WHOLE database - I had to restore our backups 3 times but they ended up being corrupted, and in the end decided to try to fix up the corrupted database. Took about 12 ours again...

Praise God, I fixed it, with minimal or no data loss whatsoever.

So now CF is back. :)

I promise I won't try to be too smart now. I'll leave things alone!

Erwin you've done a GREAT job with the CF. I've yet to find a forum that would come close to rivaling it.

Matrona
27th January 2005, 10:32 PM
Praise God, I fixed it, with minimal or no data loss whatsoever.

So now CF is back. :)

Good job, Erwin. :thumbsup:

I promise I won't try to be too smart now. I'll leave things alone!

But you are very smart!

I hope there is some way that feature can come back... not having it is driving me slightly nuts.

crossrunner
27th January 2005, 10:36 PM
Once again thanks for all your hard work and tireless dedication, Erwin. You are simply the best!

Glowworm
27th January 2005, 10:40 PM
Great job Erwin! Thanks for everything. :thumbsup:

Foofish
27th January 2005, 10:40 PM
You da man :thumbsup:

Nancaroo
27th January 2005, 11:02 PM
Thank you Erwin. I know I sure appreciate all the hard work you do. You are truly awesome and we thank the Lord for you

Erwin
28th January 2005, 12:37 AM
Thanks for all the kind words. :)

I was expecting members to be angry at me!

brinny
28th January 2005, 12:40 AM
Thanks for all the kind words. :)

I was expecting members to be angry at me!

Praying brother, praying :D :thumbsup:

Iron Sun 254
28th January 2005, 01:02 AM
Thanks for all the kind words. :)

I was expecting members to be angry at me!

After all you've done to create such a wonderful place? Not likely.

krstlros
28th January 2005, 01:50 AM
Thanks for all the kind words. :)

I was expecting members to be angry at me!
Nah!!!!

I figured there was a good reason for the shut down. There always is.

Thanks, Erwin. For all you do for us around here, you are exceedingly blessed.

Matrona
28th January 2005, 02:33 AM
Thanks for all the kind words. :)

I was expecting members to be angry at me!

Erwin is the coolest!

Most of us know how much you put into CF and we have benefited greatly from the sacrifices you've made. I wouldn't have met my sweetie Michael if it wasn't for CF. And I know Michael and I aren't the first couple to have met on CF! :clap: How could anyone be angry with you for doing the best you can? Even if you lost every single post in CF history I know I would not be angry with you for it, I would just be glad to get CF back! :crosseo:

MQTA
28th January 2005, 03:18 AM
Minimal data loss... and 28 hours of time. wow! Take a good break Erwin, you deserve it! :)

prodromos
28th January 2005, 03:57 AM
Even if you lost every single post in CF history I know I would not be angry with you for it, I would just be glad to get CF back! :crosseo:
Ditto from me too. All that we post here is transient anyway, and my poor words aren't worth much in terms of eternity. I think the occasional downtime helps us to appreciate how much is involved in keeping this ship afloat.

God bless you Erwin.

John

Robert43
28th January 2005, 03:57 AM
Thanks for all your work Erwin. :thumbsup:



Robert.

Ann M
28th January 2005, 04:40 AM
Thanks for all the kind words. :)

I was expecting members to be angry at me!

We're not mad at you Erwin.

We're more concerned about whether or not you're getting enough sleep and taking care of yourself. Please don't get sick.

Crazy Liz
28th January 2005, 04:40 AM
Erwin, thanks for trying to restore the "find members' posts" feature; I really miss it. :sigh:
Me, too. :sigh:

I can't find Plan 9 anywhere except CF Support threads!

Have you been considering my suggestion not to worry about drafts and appeals? I really could care less who sees mine, as long as everybody can't reply to them.

The Midge
28th January 2005, 05:05 AM
Once again a big Thank you for all the hard work that goes into maintaining and improving the forums.

:clap: Its good to have them back :clap:

Playerslight77
28th January 2005, 05:23 AM
p

ysl_75
28th January 2005, 06:01 AM
Thanks for all the kind words. :)

I was expecting members to be angry at me!
Actually I was so happy that CF is back again ... at first I was wondering who is Erwin this person, now only I know that you are the webmaster. I was telling myself that time how to say thank you to you because at least you informed us that is temporary offline. But just wondering yesterday the whole day I can't go into CF but I can see some other people can online to CF, just wondering why?I am not angry at all but feel quite empty when suddenly can't go into CF because this site is too great to be with. The best site I ever had. It help me a lot when I am bored. At least here got a lot of people to meet and a lot of things to learned as well.
Thanks to you Erwin and God bless you ya! http://smileys.smileycentral.com/cat/36/36_1_55.gif (http://www.smileycentral.com/?partner=ZSzeb001_ZS)

ysl_75
28th January 2005, 06:09 AM
Well, after this make sure that you have enough sleep ya ... take care! one more time thank you very much for your effort and your harworks! God bless you ... http://smileys.smileycentral.com/cat/36/36_1_39.gif (http://www.smileycentral.com/?partner=ZSzeb001_ZS)

Erwin
28th January 2005, 09:29 AM
Erwin is the coolest!

Most of us know how much you put into CF and we have benefited greatly from the sacrifices you've made. I wouldn't have met my sweetie Michael if it wasn't for CF. And I know Michael and I aren't the first couple to have met on CF! :clap: How could anyone be angry with you for doing the best you can? Even if you lost every single post in CF history I know I would not be angry with you for it, I would just be glad to get CF back! :crosseo:
Thanks for all the kind words people. :) It's quite touching. Maybe I should have CF offline more often... ;) Just kidding!!!

Actually, how many couples have met on CF? Just curious... I know of 2 only. :)

nvrbnunloved
28th January 2005, 09:37 AM
Thanks for everything you do, Erwin - you are truly appreciated ! :clap: :clap:

Rochir
28th January 2005, 09:37 AM
I promise I won't try to be too smart now. I'll leave things alone!

Thanks for all the work, Erwin:hug: , but ... does this mean you will drop the "search other members' posts" from your Things-to-do list now?:cry:

Matrona
28th January 2005, 12:16 PM
Thanks for all the kind words people. :) It's quite touching. Maybe I should have CF offline more often... ;) Just kidding!!!

Actually, how many couples have met on CF? Just curious... I know of 2 only. :)

I know a couple met on here and is married now... Michael and I will take a few more years but are headed in that direction :pray: :holy: :blush: Not sure if there are any others.

Paladin_Mark
28th January 2005, 10:16 PM
Erwin is cool . He does a great job . Ty Erwin !

aReformedPatriot
29th January 2005, 02:31 AM
I know a couple met on here and is married now... Michael and I will take a few more years but are headed in that direction :pray: :holy: :blush: Not sure if there are any others.

Glorianna I believe has as well.. not sure though... an official poll should be created on this topic. Im sure it would be a small number but interesting to see nonetheless.

BeautifulWorshipper
29th January 2005, 06:00 PM
yay!