Saturday, August 28, 2010

Adventures in the Linux storage stack

I recently got my Gentoo box back from storage after three months, and promptly upgraded all the software on it and tried to install a shiny new SSD. Here are the problems I've run into so far:

  • Media drive failed to mount
  • My media drive is a btrfs RAID-0 between two 500GB drives that I've picked up over the years. One of them wasn't being detected. After ruling out the usual suspects (drive was still working, as confirmed by smartctl, and btrfs' drive detection tool (btrfsctl -a) had been run [but oh god why does it even need this]), I started looking at more exotic causes.

    Diagnosis: The problem turned out to be a regression in the md userspace tools (must be, since I didn't upgrade the kernel): if you have a device which used to belong to an md RAID array, but doesn't anymore, md will still pick it up as a RAID device and lock it. The recommended fix is to zero out the first part of the drive, which isn't an option for me because the drive ACTUALLY HAS DATA ON IT. :[

  • pvmove stalls at 100%
  • When using pvmove to transfer my OS drive to my shiny new SSD, pvmove stalled at 100% completed. No data was being written, which I know thanks to gkrellm. Then my entire system locked up, or at least every process that tried to write data locked up. After rebooting, I tried to resume the transfer (which pvmove supports), and the system locked up again the same way.

    Diagnosis: None. Chalk this one up to LVM being completely fucked. It is a master of walking the thin line between being so useful that it can't be ignored, and so broken that it can't be used. (UPDATE: might have been caused by creating the pv on a whole disk and not a partition. Not sure why that'd be the problem, but it's the only thing I changed and now it works.)

  • Boot environment doesn't see shiny new SSD
  • So I have a completely customized boot process. It's part of a 70% successful experiment from about 6 months back in making an operating system which could survive a hard drive failure. (Harder than you'd think!) Part of that is a heavily customized initrd - basically a micro-OS which is embedded into the kernel image itself, so that I can do funky stuff with the boot process itself. This is all well and good, until I find out that while my OS can see my new hard drive fine, my initrd can't. :(

    Diagnosis: This one was me getting sloppy. When I created the initrd, I deleted a few thousand miscellaneous device nodes from the static /dev directory (because udev is too complex, let's just embed dev nodes for every piece of hardware that could possibly be connected -_-), and it turns out this included the one used by the drive I just added. I only discovered this after a half hour of "oh shit oh shit did I just transfer my entire OS to a DOA drive", of course.

Monday, August 16, 2010

How [should?] we teach history

This blog post on Abd el-Kader came to me secondhand via my RSS reader today. It's pretty fascinating in and of itself, but it's also really long, so don't start on it unless you've got a good chunk of free time ahead of you.

The thing that depressed me about the post is that I have basically zero context for any of it. I never really took an interest in history, not enough to go beyond what they taught us in high school, so for me north Africa and the Middle East are basically blank areas of the map for large swaths of history. In Rumsfeld's terminology, it was an "unknown unknown" until about an hour ago - something I didn't even realize I didn't know. For somebody accustomed to being a know-it-all, this comes as kind of a shock.

The trouble with "all the stuff we didn't learn about in history class" is that there wouldn't be enough time to cover it all if we spent our entire lives studying history. We only spend a limited amount of time in school, and that's a good thing! It just means that we need to find ways to present more varied information in the time we do have. Standardized curricula have their advantages, but the biggest downside in my opinion is that anything which doesn't make the cut remains completely unknown to several consecutive crops of students.

Do I have a solution? Nope, not a chance. I just think it'd be neat if we'd had more structured opportunities for everybody to go out and learn about something different. We can't (and shouldn't have to) increase the amount of time each child spends learning about history, so if we don't want to end up in a situation where everybody knows about the same subset of history, we need to find some way to diversify the knowledge base that we're building in history classes.

After all, if everybody learns the same parts of history, then nobody has anything interesting to say to anyone else about history, right?

Monday, August 9, 2010

Piercing the Veil

So I've got a new hobby: when I look around, I try to see the story behind everything I see - everything it took to make it, all the different processes that were necessary for it to come into being, things like that. It's simultaneously exhilarating and humbling and occasionally enlightening [in the negative space sense of suddenly realizing something you don't know]. It can even be transformative, in that you may never look at the world the same way again.

An example: I am riding my bike home from work. The bike itself is a marvel of applied physics - the impossibly thin spokes on the wheels that actually support your weight by hanging the bike from the tops of the wheels; the clever use of gyroscopic forces to keep the bike stable while it's in motion; the adjustable gear system which you can reconfigure as you ride. You could teach an entire physics course just by looking at a bicycle in motion. A bus rolls past, inviting you to consider not only the mechanics of how it runs (the entire history of internal combustion could go here) or why it runs (the efficiencies of mass transit), but also the entire superstructure of government which makes public transportation possible. The sidewalks and roads I'm riding on are made of cement - a building material which is dirt-cheap, uniform, pervasive, and commoditized. Really, it's a perfect analogy for our mass-produced society. As for the roads themselves, they constitute a single, unbroken web which stretches across the entire continent; think about that for a minute. (There's an interested design principle embedded here: if you want a pervasive system of roads, you don't start by building roads. You start by building cars, and the roads will come.) There are buildings by the sides of the road, and every individual component and overall feature has a rich and varied history - glass, steel, central heating/cooling, architecture, fluorescent lights (and the electrical grid they imply), indoor plumbing, and don't forget the construction equipment used to put it all together. If you wanted to build a modern office building from scratch, it would take centuries of technological development.

There is a Veil of abstraction which we apply to everyday life, because our minds are simply not equipped to think like this all the time. We need to take things for granted if we want to be able to function. It can be fun to pierce the Veil once in a while, and take a sip from the geyser of extraneous information that fills our world, but it's not a sustainable state of mind. I won't bother with value judgments here. The Veil is neither good, nor evil, nor anything in between, it simply is.

And if we're bored enough and perhaps stubborn enough, we could even try to peel back the Veil from the Veil itself, from the very fact that we only see the world for what it is, and not everything underneath. There are Darwinian forces in there, enhanced but certainly not altered by capitalism, which drive us to produce and to be productive, and weed us out ruthlessly if we fall into the trap of looking too deeply at the world, and spend our lives wandering around in slack-jawed wonder. There's that fact that our mental processes are fundamentally and severely limited and finite, which lies somewhere at the awkward intersection of information theory and neuroscience.

So the Veil is a necessary part of our lives, and we couldn't pull it back if we wanted to, but I think it's fun to peek behind it every once in a while.