MAJOR DOWNTIME for 36 hours...

[Erwin]

Yes, most of you would have noticed that CF was offline for the past 36 hours.

Let me just say that I've only slept 4 hours each night for the past 2 nights - this downtime has been especially stressful.

Basically, we initially thought we were hacked or were being DDOSed. It now appears that it is most likely due to our server software.

I have been taking turns with 2 US server technicians and working around the clock to get CF back online. We have been downgrading software, upgrading, then downgrading, then upgrading, backing up, restoring, again and again, and all of us have been banging our heads against the wall, because of the stress and frustration.

This is because our database is so large (optimized, it is now 15 Gb). So each time you backup and restore it, it would take a few hours.

Praise God that CF is now back online. No data appears to have been lost.

Everything is working, we are now on the latest server software (after hours of debugging and searching Google on trying to overcome problems).

I've taken today off work as I'm not feeling well at all. I'll keep an eye on CF and pray that things go well.

Thank you all for your patience. I apologise for the downtime, and hope that it never happens again.

[Erwin]

I'm going to go to bed soon. I'm exhausted!

[Redstiletto]

Erwin you are awesome!!!

Keep on rocking!!!

And go get some sleep!!

Thanks for keeping CF rocking!

Nic

[tapero]

Oh Erwin, Thank you so much for all your efforts. I'm so sorry you don't feel well. You'll be in our prayers. God bless.

[Randi]

Thank you SO much for your hard work! I can't imagine how hard it must have been, and I'm very grateful. :)

[angelwind]

We are all standing with you Erwin...thank you for working so hard for CF...we will pray too that the new changes will be stable...((((((hugs)))))).

[Col303]

Thank you so much for all of your hard work, it is very much appreciated! Now go get some sleep :)

[kelco]

Thanks for all the effort you put into CF. Go rest. God bless!

[bfly]

God Bless you, yes we have missed cf a bunch but we knew whatever the problem you were doing all that was humanly possible to fix the situation.

May God restore your health and you can get some rest. We complain, I know, but honestly we know you are doing what needs to be done. Thank you

[FriendsFellowship]

Thanks Erwin! My server went out a week ago. I can sort of understand what you're going through! Thanks for your dedication!

BTW: Profiles don't seem to load, but it can wait!

[MariaRegina]

Thanks, Erwin.

I am noticing that things are still running slow. And I was also getting database errors.

In Christ,
Elizabeth

[openup4christ]

You Rock Erwin

[RN4CHRIST]

THANK YOU SOOO MUCH!! I want you to know that we appreciate all your hard work and the dedication that you put into this site. It is truly a witness for Christ! God bless you for your sacrifice!

[Erwin]

Thanks, Erwin.

I am noticing that things are still running slow. And I was also getting database errors.

In Christ,
Elizabeth
Really? Where... please let me know.

[MN John]

I was praying for you the whole time, Erwin! :thumbsup:

Praise God! :clap:

Thank you, Erwin! :)

[Erwin]

A word of advice - don't ever try downgrading mysql 5 to mysql 4 - doesn't work. Can be done, but you lose info. We gave up after 28 hours of trying. :) Also, upgrade to php5 from 4 is not as easy as you think. Took me 6 hours straight.

[Angeldove97]

Congrads Erwin... thank you so much for your dedicating with getting the site back up. I'm sure many of us were worried about what was happening and praise the Lord everything got worked out.

God bless you and the team that helped you. ^_^

[Taylor43]

I pray you have a good long sleep you deserve it. Thankyou so much for Cf for me i missed so terribly. God bless you and thankyou so much for your hard work in keeping this site going.

Lots of Hugs
Taylor

[FriendsFellowship]

Speeds seem great here, no errors except for profiles.

I get a blank page for all user profiles.

[Dragonfly88]

Thank you, I really appreciate all the hard work you put into putting this site back up

[JDDCH]

Your efforts are well appreciated. Many Thanks.

--jd.

[Roadmap]

All I can say is that this is an awesome website, and we truly miss it when it isn't available. We appreciate what you have done here Erwin, and thank you for all your hard work. Especially over the last 36 hours. May you reap a ton of real blessings from the Lord for doing all you have done during this stressful period. Hope you can get rested and feeling better soon.

Thanks again for your efforts.

[Larry_Fout]

Thanks Erwin _ I have a message YOUR PM BOX IS FULL!
with a total of 86 messages and cannot seem to delete any more. I've never had that message before even with over 100 messages on inbox

[mesue]

I'm going to go to bed soon. I'm exhausted!
:hug: g'night Erwin. Sleep well. May God bless you richly with restful sleep and pleasant dreams.

[Alarum]

Thank you for your work! Good luck with these new servers, they seem a bit unruly. I'm sure you'll have them broken in soon enough ;)

[joyshirley]

Erwin.....bless you :hug:

Thank you for all the effort and hours and hours you put into this site. Thank you for caring enough to work round the clock on it. I pray for you, that God will bless you with a refreshing rest now and all that you need. I'm sorry it encroached on your work again.

Thank you!! :hug:

God bless you :prayer:

[praying]

God Bless you Erwin!

[Pentecostal Boy]

Thank ya Erwin :D ur 2 cool dude! I hope that ya get better man. Later,

-Pboy

[Erwin]

I'm still getting some weird server errors. :)

No sleep for me yet...

/libexec/ld-elf.so.1: /usr/local/lib/php/20050922/xml.so: Undefined symbol "XML_ParserCreate_MM"

What the...

[Larry_Fout]

Is there a place that we should report erros and bugs. Lost site supporter status

[Angeldove97]

Poor Erwin... more errors??? BLAH! I'm sure everything will get worked out eventually... you do amazing work! ^_^

[Greenthumb]

Erwin...thank you so much for all your hard work. I hope you are able to sleep well and wake refreshed. God bless you.

[FallingWaters]

Dear Erwin,

I've only been here for about two months and I had no idea how important CF had become to me until I had to go 2 days without it.

I have to tell you though, I think it's all my fault that your servers crashed... God was trying to get me away from my computer.(^_^)

There's nothing I love better than an interesting conversation, and there's always one here. Thank you for this site. May God quicken to your mind all that you need to think and remember to be successful in your present work at hand.

[Henaynei]

go get some sleep - get thyself off line!! :sleep:

[Angelwarrioress]

Thank you ! *Hugs!*

[Greg the byzantine]

go Erwin, Go Erwin Woot Woot. Thank you :)

[Godslilgurlalways]

Hi ewrin,

Just wanted to say thanks for getting the site working again I am sure that it wasn't easy did you ever think about having a team of assiant webmasters who can keep an eye for the site even some of the mods and when it's like down or something they could email you? Please don't worry about the site please go get some sleep, May God bless you indeed as I know he will and I pray that Jesus will watch over you while you sleep which I know he will:)

[dare2dream]

thanks 4 all the hard work u put into this site!!!its awesum!!!! now get sum rest....and about the downtime...no prob...don't stress over it i'm sure everyone understands


God Bless

[dianne050]

Erwin,I am praing you have a good nights sleep!You put so much in to this forum!We love this site and feel empty without our friends who we have learned to rely and have fellowship with!It is alot of reponsibility for you,must put a strain on your life and work!Thankyou from the bottom of my heart for your dedication and keeping us(cf) as a family!!!Hugs for you!!!

[Black_Ivy]

Thank you Erwin: the work that you put into this site is incredible, and all of us appreciate it. I hope you can get some rest soon. :)

[JTM3]

You rock Erwin!!

MUST FEED CF ADDICTION!!! :D

[Lenora56]

I'm still getting some weird server errors. :)

No sleep for me yet...

/libexec/ld-elf.so.1: /usr/local/lib/php/20050922/xml.so: Undefined symbol "XML_ParserCreate_MM"

What the...
Wow, I believe I'd be insane by now.

[kittystrawberry]

I appreciate everything you have done for us. Thank you Erwin! :thumbsup:

[beluverly]

Erwin:

Your hard work and dedication to this site is amazing. Get some sleep, and I hope you rest easy. :)

[Godslilgurlalways]

For some reason I was trying to view a friends profile but all that happens is the page goes blank it doesn't load even after refreshing anyone else have this problem?

[luvmyhubby]

Thanks Erwin for working so hard. Hope you feel better soon. I will pray for you and your family.

[J4Jesus]

Thanks for all your hard work Erwin.

There's always going to be some of the problems to fix and I need to report some myself, but since we are back online you need to get some rest first!

God bless you :)

[Renuka]

hats off to you
and praise the Lord for creating you for us ;)

[christianmomof3]

Thank you and bless you for all of your hard work. Rest in the Lord and awaken refreshed in Him.:prayer:

[laptoppop]

Been there done that, managing bi-coastal redundant servers. Praying for you (and your team?). LORD BLESS YOU for all you do around here. I'm a newbie, but I think I'm already hooked!
-lee-

[VioletAngel]

I have been sick most of this time it was down. I missed it for only about 12 hours of that 36. :D Sorry the thing was causing so much stress. I figured it was something to do with the servers, but was also afraid of an attack. I've been targeted on two of my Web sites, and I'm just about to them go down and stay down for a while. (some sick people out there!!) ... :o

[icedtea]

Erwin, thanks for your selfless help so those of us who need fellowship can get it here.
We prayed, you know.
Where shall we send the chocolates?

[NPH]

Thanks for all the hard work Erwin, I was so bored without CF to discuss things with people :P

Now go get yourself some sleep!

[Erwin]

test

[snoochface]

Sorry for all the headaches, and thank you for the many hours of hard work, Erwin. Your dedication is amazing.

Just to let you know, I am still getting very slow loading, and frequent time-outs (Cannot Load Page errors). Sorry...

[Ellesar]

thanks for all the hard work Erwin!

[Dr. Holly]

Thanks so much Erwin for the countless hours you put in to this site. Your work does not go unappreciated. You are amazing. I'm keeping you and this site in my prayers! :prayer: God bless you & yours!!

[Akathist]

Erwin, thanks for all your hard work.

I am having trouble getting pages to load. I have a short cut to my pm box and can get there but when I try to go to other areas, sometime it shows "server error".

I went to post in the staff area (Conference Room) and it did not post the reply instead it seemed to "time out" instead. On the other hand, I was able to unapprove a post (for SwS) and post in the report about it just fine.

Let's see if this post here loads....

[Akathist]

Erwin, suddenly my avatar and the clothes on my characer is not loading as well as the imiages for the edit quote, quick reply buttons..

[snoochface]

Images are down now... getting the lines and lines of "Soandso has a reputation beyond repute." :D

[icedtea]

Just thought I'd let you know I'm not getting any bugs at all.:thumbsup:

[stone]

shalom

*praying for cf forum to stay up and running*

:prayer:



*and for erwin to get a good few hours of sleep*

:angel:

[CaDan]

Whatever you did just worked, Erwin. I'm getting nice clean page loads and no errors connecting to the image server.

Now go to bed.

[Aquamarine81]

Thanks for fixing it, couldn't access the site and thought it was a problem on my end. Hope you get some sleep and that you feel better soon! :)

[JJay]

The site isn't really working for me. Lag time, not loading, timeouts. I'm able to get a few in but that's it.

It did it to me early as well and I counldn't post as well.

Hope all is well and you get sleep Erwin.

[aaronibus62]

For some reason a few profiles are coming up like they should while others are just blank screens. A pain in the neck if attempting to send a private message to someone (esp. if they're online) but unable to without the link on their profile.
Techology is wonderful but equally a pain. I'd guess Erwin knows that best. Sleep well, Erwin :) and God bless

[Nightson]

You're too nice to us Erwin, enjoy your sleep.

[InHisLight2006]

Thanks for the time and effort. I am beginning to be able to do more things on the site. I have been unable to access my pm's. I appreciate everything you are doing.

[Erwin]

The profile problem is finally fixed!!! :) Took me a while to trouble-shoot...

[LibertyChic]

Posting, loading, subscribed threads....all look good! :thumbsup:

Go to bed Erwin. The rest can wait. :)

[Erwin]

It's all looking good. :)

I had to delete all the php directories and extension on the servers and REINSTALL from scratch for like the 10th time, before I got rid of that error. :) It's ridiculous. It's a bit complicated as I'm running php as fast-cgi, for speed, with lighttpd. And also APC. All these stupid scripts have weird interactions. And php 5 is a bit different to php 4, I've come to realise.

[PaladinGirl]

Wow that sounds like a royal pain! I knew something major must have happened when it went down on Sunday evening and didn't come back up till tonight. Thank God it is fixed now! I was starting to suffer from withdrawal. :P Erwin, you are so awesome! :clap: :thumbsup:

[dignitized]

I've taken today off work as I'm not feeling well at all. I'll keep an eye on CF and pray that things go well. Prayers for your health

[icedtea]

Go to sleep! :hug:

[JPPT1974]

Yeah thanks my friend!

[dignitized]

are others having problems posting? It took 4 tries before my last post actually worked - kept getting a blank screen. . . .Just thought you should know erwin.

same problems making this post . . . .

[Nightson]

Yeah posting is buggy

[PaladinGirl]

I haven't been able to post at all except for my earlier post.

[ImHisServant]

I just want to echo all the thank you messages from previous posts on this thread. Can you ever hear it enough?? My husband is a programmer so I know what it is like when things crash - it's an on call 24/7 job. Thanks again and God Bless You for keeping this site going and upgrading it. There are so many people here who have deepened their relationship and furthered their walk with Christ through this site.

[dignitized]

MIND YOU - buggy posting is better than no posting at all!! quick reply is not working for me at all . . . .:sigh:

[Miracle Storm]

Thank you Erwin for everything, your efforts for CF and the reason for the existence of this place do not go unnoticed. God bless you and guide you dear one. Thank you for giving us all a way to share Christ and fellowship with our brothers and sisters in Him!! I will be praying for you to get some much needed rest and peace!!:hug:

[MQTA]

seems to take a while posting, too... I don't think all the bugs are gone yet

[Daughter of His]

Thank you for all you do for CF Erwin-God bless you !:wave:

[stillsmallvoice]

Hi all!

Erwin, as I've always said & thought, you da man!

On behalf of CF's orthodox Jewish members (all 3-4 of us :P ), we appreciate what you do :thumbsup: . Good luck & God bless!

Be well!

ssv :wave:

[GraceInHim]

to post as fast as possible for now until Erwin gets back.. I do this

go advanced, type words and hit submit and hit refresh.. then it works :).. after I did this a couple of times, I had to hit the refresh button many times until it finally goes through..

time going to post 2:40 am eastern time

Now I edited my post using the title button and it works just as I stated above, save changes and if not directed, hit refresh and retry if needed until it goes through..

time is 2:50 am

[CFfosterANGEL7]

Thank you so much for all your efforts.I hope you can sleep well

[Aggie]

What was it that actually caused the problem? Was it a bug that resulted from you upgrading to PHP 5? I’m not used to things like this just suddenly happening on their own.

By the way, I’m having the same problem trying to reply.

[Erwin]

I fixed the reply bug.

APC cache has a bug with PHP 5 and vBulletin. There is a workaround which I am using.

[fragmentsofdreams]

Thanks Erwin. :clap:

[joyshirley]

Thanks, Erwin.....:)

[She]

Thanks Erwin. I thought I had been banned! I haven't been able to post here since Monday 29 May.

Anyway, I've managed to do something very constructive with my time away from the computer. The future is looking very bright for myself and my family. So that was something positive to come out of this.

Thanks for all your hard work, anyway. It's good to be back posting again.

God bless.

[Erwin]

Thanks Erwin. I thought I had been banned! I haven't been able to post here since Monday 29 May.

Anyway, I've managed to do something very constructive with my time away from the computer. The future is looking very bright for myself and my family. So that was something positive to come out of this.

Thanks for all your hard work, anyway. It's good to be back posting again.

God bless.
Praise God! :)

Romans 8:28

[LivingRightForGod]

I would like to say a big big thank you to erwin.

ERWIN you are a awesome webmaster

Go get some rest
Keep up the good work
And hope you feel better soon

God bless

[snoochface]

I hope you were able to get some sleep, Erwin. As someone said earlier, buggy posting and slow loading is better than no CF at all. :)

8:48 a.m. EDT (U.S.) and I'm still having very slow loading pages, no visible images (reputation, avatars, etc. all replaced with text) and occasional time-outs. Just FYI, not complaining. I'm on CF, and that gives me a happy. :D

[dignitized]

I fixed the reply bug.

APC cache has a bug with PHP 5 and vBulletin. There is a workaround which I am using. Bravo :)

[Dragonfly88]

Hey is the site having problems again?
I noticed not everyting image wise is filling up, such as avatars, sigs and CF characters

[katylees]

Thank You Erwin SOO MUCHH! Sorry you've had no rest and been ill. Take care. :thumbsup: :amen:

[MQTA]

every time I think it's working again and try to post, it don't

[Erwin]

The image server died for some reason - I restarted it. Good think I check CF on my PDA in my bed. It's midnight here.

[angelwind]

I see Erwin is still online...((((((hugs)))))

[PorcelainHeart]

aww, :hug:

[Dimitree]

God Bless you Erwin !!! :)

[MQTA]

I hope you figure it all out soon!

[PaladinValer]

Are you sure it was a server error?

Perhaps you aren't aware of this Erwin, but I find it interesting that all this happened only weeks after someone vandalized the CF entry in Wikipedia with the notion of hacking CF.

Now perhaps this is a coincidence and more than likely, it is, but it would be wrong of me not to report this just in case CF's "fallout" was due to something more malicious.

[SumTinWong]

Thanks for all your efforts Erwin.

[Theresasjourney]

Take care and rest Erwin~~
TY for all your hard work!
God Bless!

[pineapplelesson]

Although I was initially 'ack - did CF disappear?' (especially on the second day *wink) I'm sure I speak for everybody when I say that I really apprecite the hard work that you've put into CF!!!

Now go and get some sleep Erwin!!

..Pineapple..

[J4Jesus]

Thanks Erwin
I know this has been really hard on you the last 3 days and nights !
We can post now. Please get some rest :)
God bless you

[Brian90]

Praise God! This is such a wonderful forum!

[GritsnGrace]

Glad all is working now (Erwin, you're the best!! :clap: ) I was going through withdrawal!!:( ^_^

[FreeinChrist]

Thanks Erwin!! Appreciate all your hard work!!

[MeekOne]

Yes, most of you would have noticed that CF was offline for the past 36 hours.

Let me just say that I've only slept 4 hours each night for the past 2 nights - this downtime has been especially stressful.

Basically, we initially thought we were hacked or were being DDOSed. It now appears that it is most likely due to our server software.

I have been taking turns with 2 US server technicians and working around the clock to get CF back online. We have been downgrading software, upgrading, then downgrading, then upgrading, backing up, restoring, again and again, and all of us have been banging our heads against the wall, because of the stress and frustration.

This is because our database is so large (optimized, it is now 15 Gb). So each time you backup and restore it, it would take a few hours.

Praise God that CF is now back online. No data appears to have been lost.

Everything is working, we are now on the latest server software (after hours of debugging and searching Google on trying to overcome problems).

I've taken today off work as I'm not feeling well at all. I'll keep an eye on CF and pray that things go well.

Thank you all for your patience. I apologise for the downtime, and hope that it never happens again.Wow, Erwin, I am so sorry that you had to go through all of that trouble! I just want to thank you for being so dedicated to this website and God's people. That is so totally awesome of you. Sending up a prayer now for you to feel better real soon. :prayer:

[PorcelainHeart]

I just wish there was a little more love on this forum. I actually cried when it came up, it made me sad. I been hurt alot on here and still get hurt, but for some reason I can't leave, I pray I will be able to leave one day or people would just leave me alone and stop trying to upset me.

[Called2Grace]

I was very distraught, I had other people check the site to see if it was just me :) I'm so obsessive! I love this site so much, I am so grate ful to you Erwin, and when I talk about this site to others I'm always like "and the createor and administrator lives in Australia!"

[cavell]

Appreciate your feed back. Was wondering why I could not connect......getting withdrawal symptoms myself

God be with You

[justanobserver]

As with all the others here, I greatly appreciate your time, efforts, sacrifices you make to keep this site up and runnin. I wish you plenty of rest and that you will feel better.

Erwin - you da man! :thumbsup:

[bill16652]

my prayers are with you andmay God richly bless all your effort

[WoundedImmortal]

I'd been wondering what was up! I run a message board as well (A small one) and we recently had server problems too...so we upgraded...these things happen :) When it first came back it was supper slow and I couldn't post-but it seems that problem has been fixed, all is working just fine-kudos to you to get it running again! No worries about the downtime :)

~Kat

[BLUEEYEANGEL]

http://www3.christianforums.com/images/icons/icon14.gif reps for you my brother ,God bless you in health and may you prosper in all areas of life:hug:

[Aggie]

Are you sure it was a server error?

Perhaps you aren't aware of this Erwin, but I find it interesting that all this happened only weeks after someone vandalized the CF entry in Wikipedia with the notion of hacking CF.

Now perhaps this is a coincidence and more than likely, it is, but it would be wrong of me not to report this just in case CF's "fallout" was due to something more malicious.
I know of a vulnerability in vBulletin 3.5.4 that could potentially be used against this forum, although it probably would have only corrupted the forum’s database rather than cause the entire webserver to go down. It’s described here: http://www.osvdb.org/24690

I’ve actually considered using this exploit against a vBulletin forum whose webmaster occasionally hacks into the computers of people who use it. But I like CF too much to want anyone using this here, so I think I should let you know about it just in case someone else has used it, or would consider using it.

[hiz_child]

Don't work yourself sick erwin...at the end of the day its NOT worth it. Cf is a great site and all...just not worth you health.