Out with the old and in with the new.

by Mads
August 05, 2007 at 23:20 | categories: hardware, sun, solaris, asf, httpd

For quite some time, the infrastructure team at the ASF has been running our websites, mail-archives and wiki on a Sun Fire T2000 Server kindly donated by Sun. Along with the T2000 there's also a Dell SATA raid donated by ask.
Naturally, the machine is running Solaris 10 and that along with dtrace has already allowed us to find and correct pretty serious performance issue. Our load was hitting 500 and beyond and was close to knocking the machine over. Some digging around with DTRACE showd us an insane number of syscalls and almost all of them being reads.
More digging around with the following one-liner by Brendan Gregg:

# Read bytes by process,
dtrace -n 'sysinfo:::readch { @bytes[execname] = sum(arg0); }'

It gave a very clear picture that almost all reads were of 1k size and that allowed Joe Schaefer to create a patch for apr to Use buffered I/O with SDBM..
The current look of things is a lot better:

  httpd                                             
           value  ------------- Distribution ------------- count    
              -1 |                                         0        
               0 |                                         987      
               1 |                                         0        
               2 |                                         6        
               4 |                                         296      
               8 |                                         30       
              16 |                                         147      
              32 |                                         130      
              64 |                                         47       
             128 |                                         140      
             256 |                                         460      
             512 |                                         118      
            1024 |                                         19       
            2048 |                                         72       
            4096 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 336511   
            8192 |                                         11       
           16384 |                                         3        
           32768 |                                         0        
           65536 |                                         8        
          131072 |                                         0        

With the change, our load has dropped from over 500 to somewhere between 5 and 10.

For a long time we've also been wanting to add some redundancy by placing a similar setup at our European site. The board approved our request to go shopping and after lots of hassle trying to buy the machine from Sun (being a small customer at Sun is rarely much fun and I think we were even more unlucky than usual). Eventually we got there and Sander along with Colm got the machines racked.

The pictures are by Colm
before

after

The upper picture show the old Itanic and a broken X-serve. Below is the "after" picture, showing Aurora which is now the European mirror of Eos. The machine above Aurora is a Sun Fire X2200 M2 Server that will serve as a mail frontend.

and so ends the tale of how the rising Sun replaced the sinking Itanic :)

Linuxforum

by Mads
February 14, 2007 at 21:32 | categories: httpd, solaris, linux

This year I'll be speaking at LinuxForum (Don't let the name fool you, a more appropriate name would be Unix or Opensource Forum). My presentation will introduce some of the more interesting bits of OpenSolaris. The idea is to show how to use features like ZFS, zones, SMF, dtrace and least privileges while showing how they can be used with Apache httpd. This is a refinement on the talk that was rejected for ApacheCon.
Doing this talk is all part of a greater plan to generate some more interest in OpenSolaris and can hopefully be used to start up OpenSolaris User Group Denmark (see also Opensolaris user groups).
The video from my marathon session (3 hours in danish) is now available.

linuxforum

Linux to Solaris Administrators Guide

by Mads
January 26, 2007 at 12:18 | categories: sun, solaris

Well worth a read: Linux to Solaris Administrators Guide. The competitors have long had Solaris to linux guids, so it is nice to finally see a guide leading people back in the right direction.
Also, there's plenty of other sysadmin documentation in Solaris 10 System Administrator Collection, in the How-To Guides and on Bigadmin.
Or for the really basic instructions, try the OpenSolaris Screencasts.


No ApacheCon EU for me

by Mads
January 25, 2007 at 22:25 | categories: apache

Both my proposals (httpd on opensolaris and advanced httpd troubleshooting) were rejected, so I'm not speaking this time. I'm not too disappointed as this years edition is smaller than usual. I suppose I could also have chosen more popular proposals, but I chose to submit talks I'd have fun preparing.
Rumor also has it that the hackathon will be cut in half to only one day, so I'm not entirely sure I can justify going unless I find an employer before then to pay part of the bill.


Winter has arrived

by Mads
January 24, 2007 at 18:58 | categories: misc

Or rather, winter arrived 2 days ago and I just didn't get around to taking pictures till yesterday and posting till today. It took quite some time for it to get here.

snow

This entry is also testing some evil javascript to display the photos as a gallery. In theory, clicking on the image above should bring up the images in slightly larger form.
Note: you probably have to be on soulfood.dk for the images to load in the larger format.


« Previous Page -- Next Page »