Skip to main content

Posts about linux (old posts, page 2)

Rebooting linux faster with kexec (and even faster with kexec-chooser!)

Somehow when reading through the Linux 2.6.17 changelog last week I came across a few articles discussing the kexec feature of recent Linux kernels. It's pretty neat, you can boot directly into another kernel image without having to go through a hardware / BIOS reboot. There's a Debian package called kexec-tools which gives you the ability to load these kernel images into memory and to boot into them. I found kexec a bit cumbersome to use, especially since all the kernels I care about booting into are the stock Debian kernels, and they all ship with ramdisk images that need to be used properly to boot. Using kexec by itself also requires that you have to manually bring the machine into a rebootable state first, or hack up some system scripts. You shouldn't just boot into a new kernel directly without shutting down devices, unmounting file systems, etc. So to scratch this itch, I wrote kexec-chooser. It's a small Python script that will allow you to easily warm-reboot into any of the stock Debian kernels installed on your system. It'll probably work with custom kernels as well, but I haven't tested that yet :) Downloads and more information can be found on the kexec-chooser page.

Quest for the Perfect Storage Solution!

I spent a little bit of time today looking into some of the storage solution choices out there.  I was mainly concerned with systems supported under Linux, but I didn't limit myself to just those.

My quest started this morning when I read a post on Gizmodo: Buffalo DriveStation: Serial ATA, Fanless Design. I had heard of Buffalo Technology before, I've often considered buying one of their products, or something like it.  What could be better than a 2 Terabyte box that you just plug into your network and configure?  Well...The Perfect Storage Solution of course!

A few simple use cases may describe what I'm looking for.  First, if a drive fails, I want to be able to replace it with no downtime, loss of data, and using any drive of sufficient size (at least as large as the one that failed) that I have on hand.  Second, if I'm running out of free space, I want to be able to add a brand new drive and start using it.  Third, if there is no more physical room for a new drive, I want to be able to migrate data off of the one of the drives (probably the smallest/oldest one), to make room for a newer, larger drive.  I suppose this is a direct consequence of satisfaction of the first requirement.

Out of these simple cases, I can distill a few must-have features:

  • Scalable! I should be able to add more space to this thing with a minimum of hassle.  I consider having to unmount a filesystem, grow the partition, then grow the filesystem a hassle.  I also consider having to get drives of exactly the same size as the drives already in there a hassle too.  I want to add more space, not replace the space I have!  LVM comes close to achieving this.
  • Fault tolerant: The Perfect Storage Solution should be able to handle at least one of the drives failing.  Better would be the ability to handle n drive failures.  RAID-5/RAID-6 work well here, but fail the scalability requirement.
  • Cheap!  I shouldn't need proprietary hardware/software for this.  I should just be able to add another drive to my enclosure and start using the space.  Or at the worst, buy another enclosure and add the new drive to the new enclosure :)
Some other nice features would be things like:
  • Snapshot support: great for backups, or when doing some kinds of admin work.
  • Very large filesystem support.  It's very easy to get more than a few terabytes of data (legitimately!) these days :)
And as long as I'm writing a wish list:
  • Transparent compression
  • Transparent encryption
  • Clustering (create one big pool of storage from drives scattered over a network)
So what's wrong with the devices like the one above?  Basically they don't scale well.  There's no nice way to integrate these things into one big pool of disk space.  You need to mess around with mount points, and put symlinks all over the place...unless you use LVM.

What's wrong with LVM?  It's not fault tolerant.  Sure, you can run LVM on top of a bunch of RAID devices.  But that means to add more storage in a fault tolerant way, you need to add a whole new RAID array since it's not really possible (as far as I know, somebody please correct me on this if I'm wrong!) to add a single drive into an existing RAID array.

Wikipedia's RAID page mentions the idea of a "write hole", and refers the reader to Jeff Bonwick's post on RAID-Z. The concept of the "write hole" does make some sense to me; basically if the drives lose power or crash while writing the parity data, then the data blocks and parity blocks may be inconsistent. It's not clear to me how RAID-Z solves this, and why you can't check those blocks when you restore power (especially when using some kind of journaled filesystem...although I suppose that the journal's parity data may have been corrupted as well!), but certainly data integrity is an issue that the Perfect Storage Solution must address!

I thought Sun's ZFS was promising for a little while.  But it seems that it doesn't handle drives of different sizes any better than RAID-5/6 does.  It's also unlikely that it will ever be available to Linux users because of licensing issues.

It certainly seems like LVM is the closest to what I'm looking for.  If only it managed parity data across physical blocks!  Then just polish up the ext3 online resizing functionality and life would be great!  Or maybe reiserfs/jfs/xfs would work better for a resizable filesystem?

Technorati Tags: , , ,

Kill zem all!

Wow! Every good linux user knows about the kill command. You use it all the time to kill off out of control or dead processes.

kill 1234
will kill off the process with pid 1234. But did you know that
kill -TERM -1234
will kill off all processes in 1234's process group? I didn't until just a few minutes ago! Super-handy!

Disowning your children (in bash/zsh)

You learn something new every day! Today I learned about the disown builtin in bash and zsh. When you disown a job, it will no longer receive a HUP signal when you exit your shell.

catlee@sherwood:~ [1015]% while true; do date >> date.log; sleep 10; done&

[1] 7380

So now you've got the current date being appended to date.log every 10 seconds. Try exiting your shell:

catlee@sherwood:~ [1016%1]% exit

zsh: you have running jobs.

But use the disown command:

catlee@sherwood:~ [1017%1]% disown %1

catlee@sherwood:~ [1018]% exit

and your shell exits without complaining. Meanwhile, your job keeps running in the background!

Ahhhhhh...Breathing room!

At least that's the way it feels! I just realized that my laptop supports 1600x1200 resolution! And I've been running it at 1024x768 all this time! So after fighting with the fglrx drivers for Debian, I can finally surf the web and read e-mails without having to scroll left and right all the time! Setting up fglrx was less than painless to put it mildly. For future reference, I had to do something like this:

aptitude install fglrx-kernel-src

cd /usr/src

tar jxf fglrx.tar.bz2

cd modules/fglrx

export KVERS=2.6.16-1-686

export KSRC=/usr/src/linux-headers-${KVERS}

debian/rules binary_modules

I would have hoped that m-a a-i fglrx would have worked, but alas... Oh, and other thing. ATI: it isn't cool to lock up the machine just because your driver doesn't support 16 bit colour. I had set my default colour depth to 16bpp previously in an effort to get higher resolution with the radeon driver, but when I started using the fglrx driver things would just lock up.

splittar 0.2

I've finally finished version 0.2 of splittar. In addition to some bug fixes, I've added a few new features:

  • Generate files with more consistent file sizes
  • Keep multiple files open so that smaller files can be added to archives with a bit of room left
  • Ability to tweak how splittar estimates how well a file will compress. This is a weighted average between the actual file size, and the file size multiplied by the current compression ratio