Server2 outage

Jeremy Howard · 30 Sep 2003, 12:36 AM

The new kernel on server2 is not performing as expected. Therefore we are rebooting the server to restore the previous kernel, until the performance problems are sorted out. There will be a 10 minute outage.

Jeremy Howard · 30 Sep 2003, 12:42 AM

The reboot is now underway...

leisuresuitlary · 30 Sep 2003, 12:43 AM

i wondered which server my account was kept on.....

now i know!!!!

Larry

Jeremy Howard · 30 Sep 2003, 12:49 AM

Reboot is now complete. Outage time was 5 minutes

elvey · 30 Sep 2003, 12:54 AM

Thanks for keeping us updated!!!
I was wondering why Mozilla was asking me for my password (which it normally stores) and this thread answered the question immediately.

Jeremy Howard · 30 Sep 2003, 01:05 AM

We've got an idea how to resolve the performance problem. We'll wait until it is night-time in the US and UK, and then we'll need another reboot to install our updated kernel. We'll schedule this so everyone will know when it happens.

Sorry about this short outage today. We tested the new kernel carefully on the beta server, however we tested under IO load (which is normally the constraint) not CPU load. The new kernel turned out to have CPU performance problems. We'll do some CPU performance testing on the beta server tomorrow with alternate kernels.

elvey · 30 Sep 2003, 03:49 AM

Cool.

There was an article I sent to Rob a while back on I/O.
Also comments on the very fast code produced by the Intel C++ compiler for Linux I/O. Rob mentioned some of the nice hardware you're working with. Have you considered/tried this compiler?

short article: Mix 2 parts Hyper-Threading with 1 part PCI-X and you have an I/O accelerant that delivers 3,400 IOPS and 266MB of data per second! (SCSI)

Read what happens when openBench Labs ignores hobgoblin thinking, to examine a new class of hardware as we test the mettle of the 1U Appro 12224Xi server to handle large (make that huge) amounts of I/O, using Adaptec's 39320 series Ultra320 SCSI controller and four Maxtor 10K IV drives. Check out a recipe that mixes 2 parts Hyper-Threading with 1 part PCI-X for an I/O accelerant of gargantuan proportions.

http://www.open-mag.com/35773583279.htm

Jeremy Howard · 30 Sep 2003, 05:12 AM

We use this to compile most stuff, but not the kernel - it needs too many patches to compile cleanly.

elvey · 5 Oct 2003, 05:21 AM

Quote:

Originally posted by Jeremy Howard
We use this to compile most stuff, but not the kernel - it needs too many patches to compile cleanly.

Thanks for the answer! Fair enough and interesting. I wonder which distributions come with kernels compiled with Intel's compiler. Of course it's in the kernel that the most impact would be felt.

Jeremy Howard · 5 Oct 2003, 07:10 AM

Quote:

Originally posted by elvey
Of course it's in the kernel that the most impact would be felt.

Not for us - most of our CPU time is in Cyrus and mod_perl. System time is not that high.

30 Sep 2003, 12:36 AM	#1
Jeremy Howard Ultimate Contributor Join Date: Sep 2001 Location: Australia Posts: 11,501	Server2 outage The new kernel on server2 is not performing as expected. Therefore we are rebooting the server to restore the previous kernel, until the performance problems are sorted out. There will be a 10 minute outage.

30 Sep 2003, 12:42 AM	#2
Jeremy Howard Ultimate Contributor Join Date: Sep 2001 Location: Australia Posts: 11,501	The reboot is now underway...

30 Sep 2003, 12:43 AM	#3
leisuresuitlary Senior Member Join Date: Mar 2003 Location: UK Posts: 168	i wondered which server my account was kept on..... now i know!!!! Larry

30 Sep 2003, 12:49 AM	#4
Jeremy Howard Ultimate Contributor Join Date: Sep 2001 Location: Australia Posts: 11,501	Reboot is now complete. Outage time was 5 minutes

30 Sep 2003, 12:54 AM	#5
elvey The "e" in e-mail Join Date: Jan 2002 Location: San Francisco Posts: 2,466	Thanks for keeping us updated!!! I was wondering why Mozilla was asking me for my password (which it normally stores) and this thread answered the question immediately.

30 Sep 2003, 01:05 AM	#6
Jeremy Howard Ultimate Contributor Join Date: Sep 2001 Location: Australia Posts: 11,501	We've got an idea how to resolve the performance problem. We'll wait until it is night-time in the US and UK, and then we'll need another reboot to install our updated kernel. We'll schedule this so everyone will know when it happens. Sorry about this short outage today. We tested the new kernel carefully on the beta server, however we tested under IO load (which is normally the constraint) not CPU load. The new kernel turned out to have CPU performance problems. We'll do some CPU performance testing on the beta server tomorrow with alternate kernels.

30 Sep 2003, 03:49 AM	#7
elvey The "e" in e-mail Join Date: Jan 2002 Location: San Francisco Posts: 2,466	Cool. There was an article I sent to Rob a while back on I/O. Also comments on the very fast code produced by the Intel C++ compiler for Linux I/O. Rob mentioned some of the nice hardware you're working with. Have you considered/tried this compiler? short article: Mix 2 parts Hyper-Threading with 1 part PCI-X and you have an I/O accelerant that delivers 3,400 IOPS and 266MB of data per second! (SCSI) Read what happens when openBench Labs ignores hobgoblin thinking, to examine a new class of hardware as we test the mettle of the 1U Appro 12224Xi server to handle large (make that huge) amounts of I/O, using Adaptec's 39320 series Ultra320 SCSI controller and four Maxtor 10K IV drives. Check out a recipe that mixes 2 parts Hyper-Threading with 1 part PCI-X for an I/O accelerant of gargantuan proportions. http://www.open-mag.com/35773583279.htm

30 Sep 2003, 05:12 AM	#8
Jeremy Howard Ultimate Contributor Join Date: Sep 2001 Location: Australia Posts: 11,501	We use this to compile most stuff, but not the kernel - it needs too many patches to compile cleanly.