Not going to ApacheCon

by Mads
March 25, 2008 at 23:27 | categories: apache, httpd, asf

Last week I got notified about some changes to the planned schedule that unfortunately put my talk at a time I would have a hard time making. So rather than stressing around trying to make ends meet in what was already to be a hit'n'run trip, my talk is off the schedule and I won't be going.
Instead I'll try to rewrite my talk into some blog entries as time permits.

ApacheCon US also looks very unlikely as I'm not very keen on going to New Orleans.


Out with the old and in with the new.

by Mads
August 05, 2007 at 23:20 | categories: hardware, sun, solaris, asf, httpd

For quite some time, the infrastructure team at the ASF has been running our websites, mail-archives and wiki on a Sun Fire T2000 Server kindly donated by Sun. Along with the T2000 there's also a Dell SATA raid donated by ask.
Naturally, the machine is running Solaris 10 and that along with dtrace has already allowed us to find and correct pretty serious performance issue. Our load was hitting 500 and beyond and was close to knocking the machine over. Some digging around with DTRACE showd us an insane number of syscalls and almost all of them being reads.
More digging around with the following one-liner by Brendan Gregg:

# Read bytes by process,
dtrace -n 'sysinfo:::readch { @bytes[execname] = sum(arg0); }'

It gave a very clear picture that almost all reads were of 1k size and that allowed Joe Schaefer to create a patch for apr to Use buffered I/O with SDBM..
The current look of things is a lot better:

  httpd                                             
           value  ------------- Distribution ------------- count    
              -1 |                                         0        
               0 |                                         987      
               1 |                                         0        
               2 |                                         6        
               4 |                                         296      
               8 |                                         30       
              16 |                                         147      
              32 |                                         130      
              64 |                                         47       
             128 |                                         140      
             256 |                                         460      
             512 |                                         118      
            1024 |                                         19       
            2048 |                                         72       
            4096 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 336511   
            8192 |                                         11       
           16384 |                                         3        
           32768 |                                         0        
           65536 |                                         8        
          131072 |                                         0        

With the change, our load has dropped from over 500 to somewhere between 5 and 10.

For a long time we've also been wanting to add some redundancy by placing a similar setup at our European site. The board approved our request to go shopping and after lots of hassle trying to buy the machine from Sun (being a small customer at Sun is rarely much fun and I think we were even more unlucky than usual). Eventually we got there and Sander along with Colm got the machines racked.

The pictures are by Colm
before

after

The upper picture show the old Itanic and a broken X-serve. Below is the "after" picture, showing Aurora which is now the European mirror of Eos. The machine above Aurora is a Sun Fire X2200 M2 Server that will serve as a mail frontend.

and so ends the tale of how the rising Sun replaced the sinking Itanic :)

ApacheCon 2006US

by Mads
October 09, 2006 at 22:40 | categories: apache, soulfood, asf

Right now ApacheCon is going full speed in Austin.
I'm not anywhere near Austin and will be missing ApacheCon for the first time since I started going in 2001. I did have a talk scheduled that I was really looking forward to (especially since it has been rejected the last two times I proposed it), but I "ran out of time and energy" and decided that it would be better to cancel.
I wish all the best to those lucky bastards who get to be there and promise to turn up at the next ApacheCon and hand over the beers I owe various people (anyone who thinks to collect interest on owed beer better let me know now so that I can get it opened and set aside in a glass ;).


Patching and Fair Share Scheduling

by Mads
April 15, 2006 at 00:22 | categories: sun, solaris, asf

I spent a very long time today getting the ASF zones server patched for the recent sendmail vulnerability. The Sun Alert and the Interim Relief patch were simple enough except for the dependency on 118844-30 which again depends on a grub bootloader update. One thing lead to another (or should I say: one patch led to another) and with me playing it safe (the machine is 9 timezones away) and the mess of having to sort out patches by hand it all ended up taking 4 or 5 hours. I'm used to having support contracts in order and TLP/EIS at hand at work, so this ended up taking a lot more time than I expected.

With the day being more or less gone anyway, I decided that I might as well go ahead and implement Fair Share Scheduling.
So far every zone got its own pool and each zone (except global) got the same number of shares. I doubt anyone will notice this change in their day to day running of things on the server, but in the case where something goes haywire and starts soaking resources, I hope the rest of the zones won't notice. I plan to let it run for a while and then see if there is a need for adjustment.


Dtrace support in non-global zones

by Mads
March 28, 2006 at 23:32 | categories: sun, solaris, asf

Dan Price has just announced support for dtrace in non-global zones. The mail went to dtrace discuss.

Well, I guess this is as good a time as any to announce that I've
integrated initial support for DTrace inside of Containers (a.k.a.
non-global zones) as of Friday, Mar 24, 2006.  This means that in
future Solaris Express and Community Express builds (those based on
Nevada B37 or higher), you can use a subset of DTrace functionality
inside of non-global zones.

Here's how to use this functionality:

        # zonecfg -z myzone
        zonecfg:myzone> set limitpriv=default,dtrace_proc,dtrace_user
        zonecfg:myzone> ^D

        # zoneadm -z myzone boot

        # zlogin myzone
        myzone# dtrace -l
        ...

        myzone# plockstat -Ap `pgrep startd`
        ...

Note that either or both of the dtrace_proc and dtrace_user privileges
may be granted to a zone, but dtrace_kernel may not be (zoneadm will
enforce this).  The lack of dtrace_kernel means that not every DTrace
script will work, since kernel state is not available to DTrace inside
of a zone; but we think this represents a good start.

Additional virtualization work has been done to ensure that data from
other zones is not visible inside the zone, and to ensure that the
interactions with other relevant privileges (proc_owner and proc_zone)
behave as expected.

        -dp

The whole thread is here.

I'm really looking forward to getting this in a release so that we can bring it onto the ASF zones server.


Next Page ยป