P. Static has got Opinions

Processing Wikipedia

2011-07-04T17:00:00.000-07:00

The other week I got it in my head that it might be fun to mess around with Wikipedia. I don't just mean the website, though - I mean their entire database of articles. I've been looking around for side projects lately, and this is a perfect opportunity/excuse to write some code that operates on Big-with-a-capital-B data, which should be exciting.

My day job is working on Microsoft Office. Now, don't get me wrong, it's interesting work... but the thing about being a junior dev on Office is that I spend most of my time writing extremely robust solutions for extremely small problems. I want a side project so that I can maintain a sense of perspective about programming, so that I don't get stuck in a rut. So, it's time for me to do something completely ridiculous. Let's datamine Wikipedia! :D

Getting started

Wikipedia posts dumps of their entire database online every once in a while. The latest one is from about two weeks ago, which is plenty fresh enough for my purposes. I'm using the pages-articles.xml.bz2 file, which is about 6.5 GB compressed, and 26 GB uncompressed. It contains just shy of nine and a half million articles. And it's a single XML file. Ouch. :(

There are probably text tools for Windows that are able to handle files that are larger than memory, but I'm not sure where they're hiding. So, I'm using Cygwin and UNIX tools for pretty much everything here; when in doubt, go old-school. "less" will get me a general sense of what the input data looks like pretty much instantly.

It looks like Wikipedia uses a consistent XML scheme for all their types of database dumps. I don't think it's documented anywhere, and the link in the XML to the schema is broken, but whatever. I can probably figure it out. >_> Anyway, it's not like their XML actually validates against any schema, as we'll see in a bit.

Splitting things up

A single XML file is not a terribly useful way to package up a database dump, but I'm sure they have their reasons. Before we can do anything with this data, we'll have to split it up into more manageable chunks. The structure I'm going for is one directory per article, with a single XML file in each directory. This makes it easy to find and read any article, and also leaves an easy way to add extra data to each article. It might be useful, for instance, to have the text of the article available without any extra formatting, or to have a list of links from the article, or things like that.

I wrote a python script to split the file into articles. It's available on the bitbucket page for this project. I decided not to use an XML parser to split up the file, and instead cobbled something together that just does substring searches to find article boundaries. This is generally a stupid thing to do, but streaming XML parsers are a pain, and this file is a hundred times bigger than I'd feel comfortable sticking in a DOM parser. (That said, I ended up using a real XML parser to get the article names out of the individual pages.)

Not using a real parser turned out to be a good move, because as it turns out, the database dump isn't actually valid XML all the way through. In the dump I happened to grab, there are six places where a page is truncated, including the closing tags. (Things may be better or worse in other dumps, but I really don't feel like grabbing another few-dozen-gigabyte dump file to find out.) If I were using a real XML parser, this is about where I'd give up, but with the script I wrote it was pretty straightforward to work around. We can figure out that we have a broken page when we see an unexpected opening page tag, dump what we have in the buffer to a special location to manually fix up later, and continue parsing from the next page.

Right now, the script is running at a bit over ten thousand articles per minute. I don't know if I can make that much faster, since it seems like the filesystem is the bottleneck at this point, but whatever. Once I've got a preprocessed copy of the database, the next steps will be to parse the articles and pull out a list of links, and to use that data to solve the "six degrees" problem. Stay tuned!

Thinking with Portals

2011-06-21T00:22:00.001-07:00

Portal is a fun little puzzle game, wherein you can create little wormholes that connect different places and walk through them. But it's also so much more! The puzzles by themselves would qualify Portal as a pretty neat game, but it's the writing and the story and the general atmosphere that make it one of the best (and most-quoted) games of the last decade.

I'm a couple years late to be singing Portal's praises, though; it came out in 2007. So why am I writing about it now?

[Spoilers follow. If you haven't played Portal, GO DO THAT RIGHT NOW OMG. No spoilers for Portal 2, so if you haven't played that one yet you're still good to keep reading.]

Actually, I've been thinking a lot about just what made Portal so awesome ever since I played through it a few weeks ago. (In general, I think it's worthwhile to examine awesome things!) One thing that's really struck me in retrospect is that the whole game is permeated by this incredible level of cognitive dissonance. I don't think it was an accident, or some happy quirk of the way they ended up writing it. It happens on so many levels, and in so many ways, that I think they must have planned it that way, or at least made a conscious decision about it at some point.

Example: turrets. There are automatic gun turrets sprinkled through the later levels, and they are adorable. They have little singsong voices, and they say things like "Are you still there?" and "Could you come over here?", and when you 'kill' them by knocking them over, they sometimes say "I don't hate you..." in a dejected little voice. You can't help but fall in love with them, and feel guilty about taking them down, even though they will basically kill you on sight. Relevant.

Example: GLaDOS. You start the game taking orders from a friendly-if-slightly-glitchy computer, and she gradually develops over the course of the game into the lying, omnicidal GLaDOS that we all know and love. GLaDOS herself is really a fine example of cognitive dissonance, on top of the unease she induces as she becomes more and more unhinged. She guides you through the levels with friendly-if-unsettling comments, and even goes as far as trying to be nice during The Escape, but it becomes extremely clear after a certain point that she wants nothing more than to kill you and be done with it. (The light at the end of that tunnel? It's not a cake, I'll tell you that.)

"Remember when the platform was sliding into the fire pit and I said 'Goodbye' and you were like 'No way!' And then I was all 'We pretended we were going to murder you?' That was great."

Example: your Weighted Companion Cube. Here, have a cube; this one is special! It has got hearts on it. Got a puzzle that needs solving? Your cube is part of the solution. Yep, it's just you and your cube, alone against the world! What's that, you've finished this level? Time to euthanize your only friend, then! Here, use this conveniently-placed incinerator. D: (Honestly, though? The very fact that they manage to make you form an emotional bond with a box with hearts on the sides is just another example of how masterfully they are messing with your head.)

Example: Still Alive, by Jonathan Coulton. This song plays during the end credits of the game, and is generally awesome. It's later than I planned to stay up writing this, so rather than explain why, I'll just refer you to its lyrics. <_<

So back to the title of this post, Thinking with Portals. Usually it's used in the context of "Now you're thinking with portals!", meaning that you're learning to use some of the cleverer tricks you can do with portals to solve puzzles. But can we interpret it another way? What if you actually could use portals to navigate your headspace; jumping between disparate and occasionally contradictory ideas, connecting things that would never normally be connected, being mentally in two places at once. Sound familiar?

Something to ponder.

I am done beating this horse. It is dead.

2011-02-10T20:50:00.000-08:00

You know what? I can handle it when mainstream media outlets call Anonymous a "hacking group". Everybody knows they don't exactly have a ready supply of technical expertise on hand; they probably couldn't tell an ethernet crimper from a can opener. We expect that from them. It's even sort of adorable sometimes, like seeing an elderly person trying to use a mouse like a foot pedal. But the point is, they don't know any better.

But when Ars Fucking Technica writes a long investigative piece about how dangerous it is to mess with Anonymous, because Anon is a Legitimate Group of Serious Hackers, that's just shameful. They, of all people, should know better. Nothing about Anonymous is secret except their names, and everybody who cares to ask knows that they're a bunch of angry teenagers with computers. But that doesn't make for a very good story, so now we have the tragic tale of Aaron Barr, who came too close to the leaders of Anonymous and got burned.

You know what I see when I read this story? A slightly deranged man, who happens to head a private security firm, decides to pick a fight with a group that's known mostly for their skill at harassment. His boss thinks he's going off the rails, his employees think he's completely deranged, but he presses on. Eventually a hacker shows up, exploits a SQL injection to get access to his company's internal network, and then passes out his passwords to Anonymous, who goes and has a field day, since that's what they do. However, his story happens to involve the major players of the Wikileaks scandal, so people pick it up as a useful proxy for real news.

So fine. I give up. If Ars Technica has decreed that Anon is a Serious Hacking Group, then that's it. Hacking is dead, nobody remembers what it means anymore. Long live Anonymous. :(

Chrome, H.264, and 2015

2011-01-30T17:29:00.000-08:00

Google ruffled a lot of feathers a few weeks ago when it announced that it would be dropping H.264 support from Chrome. In the short term, it seems like a ridiculous move - one guaranteed to kill HTML5 video, and keep Flash dominant for the foreseeable future - but come 2015, I think this move will make a lot of sense.

Some background on video codecs: A video codec is a method of compressing video. The measure of a codec is how well it compresses, and what sort of tradeoff you have to make between file size and image quality. Any codec can give you perfect video if you give it enough space to work with, but H.264 is one of the best when it comes to giving you high quality video at reasonable bitrates. Other prominent codecs for web streaming include Theora (which is open-source and royalty-free, but doesn't compress as well as H.264), Dirac (an experimental codec developed by the BBC, which has yet to gain any traction), and VP8 (Google's codec, bought from On2, and part of WebM). H.264 is dominant right now because it gives good results, is widely implemented, and because hardware accelerated decoders are available (crucial for mobile devices).

The HTML5 video tag doesn't specify any required video codecs, so sites are responsible for using one that all their users are able to use. Last year there was a huge shootout between proponents of H.264 and Theora. H.264 is technically the better solution, but it's owned by the MPEG-LA, and they intend to start charging royalties for it in 2015. Right now, though, H.264 seems to be winning - it's backed by Microsoft and Apple, and it's actually the only way to get video to play on iOS devices.

This is a problem for Google: YouTube would be hit pretty hard by the license fees, since every video on YouTube is streamed in H.264. Google doesn't plan to take this lying down, though. If Theora isn't up to the job, then Google will buy a codec that is competitive with H.264, and release it royalty-free for anybody to use. If it looks like H.264 is still winning, then Google is willing to drop H.264 from Google Chrome (just over 10% of the browser market) to try to kill its momentum, and replace it with Google's own codec.

MPEG-LA was originally set to start charging royalties for web streaming H.264 in 2010, but they moved it back to 2015 in order to allow H.264 to become more solidly entrenched. People say that Google is being shortsighted by promoting their own codec, since they'll never make any progress when IE9 and Safari (and iPhones!) support H.264 by default. Google doesn't need to win, though - they just need a solid alternative to H.264 to exist by 2015, so that MPEG-LA is in a weaker position when it comes time to work out what YouTube has to pay for using H.264. That's their real game here, and so far it seems like they're going to make it.

Using "make" as a download manager (or: how to parallelize EVERYTHING)

2011-01-09T13:33:00.000-08:00

I find myself in the somewhat tedious position of having to download about a hundred large files from S3. I need to only download certain files, so I can't even use S3's APIs and pull down the entire bucket. What a pain!

The first thirty or so, I did by copy/pasting the URLs onto a command line with wget, so that I could at least download them in batches. This worked, but was slower than necessary: S3 occasionally gives you a slow download, so that every once in a while I'd get a file that took ten times longer than the rest, holding everything up. You know what'd really be great? If I could download with wget, but parallelize it, so that several files were downloading at once.

There are a few different options here. I could get a download manager, which would do what I want but requires me to install some random program that I'll never use again. I could use the solution presented in this blog post (first hit for "wget parallel downloads"), but that's still too much code. Or, I could use this nifty UNIX trick for parallelizing anything. :3

The code

First, write a Makefile that looks like this:

%::
        wget -nv -c $@

Save that as "Makefile", of course. Then, in the same directory, run this:

make -j3 [URL1] [URL2] [URL3] ...

...and all the URLs you specify will be downloaded, with three downloads going at a time.

But how does it work?

Not everybody knows this, but make includes a really neat dependency-aware parallel work queue. If you give it a really long list of jobs, where some of them depend on others, make can do them in parallel, while keeping track of what depends on what and making sure that dependencies happen before things that depend on them. (In this case, though, we have no dependencies to express, so the Makefile is really simple.) The "-j" option controls the number of jobs that make will run in parallel.

Our Makefile consists of a single wildcard rule (%::), matching any input, and calls wget with whatever we pass in ($@ - clearly whoever wrote make thought Perl-style variables were a good idea :p). make works by taking its command-line parameters, finding a rule that matches each one of them, and executing that rule. So all we have to do is run make with a bunch of URLs as parameters; it will look for a file called Makefile (the default), find a rule in it that matches the URLs (the wildcard rule, in this case), and execute that rule (downloading the file).

Variant: what if I have a list of URLs

The coolest thing about the command line is the ease with which you can combine programs. In this case, let's combine the above hack with the "xargs" program, which translates streams of lines to command-line arguments. If you replace the command line above with this:

xargs make -j4 < list-of-urls.txt

...then you can download every URL in a list, in parallel. (As an aside, xargs has all the necessary logic to work around the maximum number of command line arguments you can use at a time, so this should work with an unlimited number of URLs.)

But how do I parallelize EVERYTHING?

Up to you! This technique works in a surprisingly wide array of situations, if you're using the command line. The last time I used it, it was batch-converting a lot of images - imagemagick doesn't use multiple CPUs, but using make let me run multiple instances of it at once, so that I finished twice as fast. (My downloads have all finished, though, so the necessary changes to the Makefile are left as an exercise to the reader. :3)

Adventures in UNIX: Pipe buffer edge cases, half-open sockets

2010-12-05T12:57:00.000-08:00

So yesterday I had a really neat idea: exposing pipe-based programs as network services. You could open a connection to a program, send it data, and the remote computer would put it through some predefined command and send it back. Then I realized that with IPv6, you could give each program its own IP address, and give them all names in the DNS, so that you'd have a perfectly usable system without using any higher-level protocols than DNS and TCP. Then I realized that this would be simple enough that you wouldn't even need to write any code to make this work - it's all doable with simple shell commands! (I was wrong about this last one, and that's the topic of this post.)

(By way of comparison: This is sort of an inversion of the Plan9 model of network computation. Instead of mounting a remote filesystem and piping it through a local program, you're piping local data through a remote program.)

Pipes are usually a one-way structure, but for this to work properly, I needed something a little more exotic. I need to be able to take output from a command, and pipe it back around to the beginning of the pipe, so that the pipe has a loop in it. If I could do that, I could combine netcat with any command, and that'd be a one-liner that implements a server. :D

So here's the first thing I tried.

Server:


mkfifo t
while true; do nc -l 127.0.0.1 9999 < t | tr a-z A-Z >> t; done

Client:


nc 127.0.0.1 9999 < testdata

In a perfect world, this would work! But here's where we get into the details of pipe buffers.

The first problem with this is in the server. When you use a pipe, data's actually buffered along the way, in the commands that are being piped. Normally, this is transparent, because the buffers are flushed out when the previous command exits. This doesn't work when you have a loop through a fifo, though! The data that's buffered in the tr command doesn't get flushed to the fifo until netcat exits, and so netcat never actually has the chance to send the tail end of the data. The only way I could think of to solve this was to write some code for the server - pretty disappointing, but probably necessary. (I'm not going to post that code here, because it's even messier than being a prototype should justify. >_>)

But that's not all - it turns out the client part is broken, for a completely different reason. When you give netcat an EOF (Ctrl-D), it doesn't know how to tell the remote side of the connection that there was an EOF. The server then doesn't have any way to know when to flush the buffers out and end the command, so the whole thing deadlocks waiting for more input that's never coming.

It turns out that TCP solves this problem; the bug is in netcat. With a TCP socket, you can close one direction of traffic, but keep using the other - for example, when you're done writing data to a socket, you can shut down the socket for writes, which signals to the remote side that you're done writing, and then read whatever the server sends back. This, unfortunately, required more code.

netpipe.py:


import sys
addr = sys.argv[1]

import select
def attempt_read(s, BUF_SIZE):
    if select.select([s], [], [], 0)[0]:
        return s.recv(BUF_SIZE)
    return ''

import socket
s = socket.create_connection((addr, 9999))

BUF_SIZE = 4096
buf = sys.stdin.read(BUF_SIZE)
while buf:
    s.sendall(buf)
    
    sys.stdout.write(attempt_read(s, BUF_SIZE))
    sys.stdout.flush()
    
    buf = sys.stdin.read(BUF_SIZE)

s.shutdown(socket.SHUT_WR)

buf = s.recv(BUF_SIZE)
while buf:
    sys.stdout.write(buf)
    sys.stdout.flush()
    buf = s.recv(BUF_SIZE)

(This is trivial enough that I'm planning to port it to C soon.)

Finally, some good news: this works perfectly! :D With this, you can open a connection and use it as a component in a pipe.

p2p DNS

2010-12-01T20:56:00.000-08:00

Now that the US is considering forcing pirate domain names out of the DNS, one of the founders of The Pirate Bay is floating the idea of a p2p DNS alternative.

Okay, wow. This is an incredibly terrible idea.

I'll start with the obvious objections:

The DNS is meant to be authoritative

NO.

The DNS is meant to be reliable
Performance
Secure decentralized systems are HARD

There are some people whose first reaction to any data management problem is to try to stick it in a magic DHT and forget about it. In many cases it works - see BitTorrent, for example. A DHT will work in any application where you don't especially need data to be reliable or trustworthy; it's a perfect fit for BitTorrent peer exchange, where reliability is optional because the DHT is only a backup for the real tracker, and trustworthiness doesn't matter because the peers aren't trusted in the first place. For the DNS, though, a DHT is exactly the wrong solution.

It may be possible, someday, to fully decentralize the DNS. To do it will take some fundamental advances in computer science, though, and Peter Sunde isn't going to be able to make that happen by rallying the pirates to his cause.

Julian Assange is a Terrorist (and I mean that in a good way)

2010-11-30T23:11:00.000-08:00

Politicians want to classify Julian Assange as a terrorist. Insane? Only at first glance.

I've been reading about something that Assange wrote a few years ago, which basically lays out his plans for Wikileaks. It's actually a pretty neat read. Summary: Assange sees today's American government as some kind of corporate conspiracy (can't argue there), and he wants to throw sand in the works of the conspiracy by increasing the cost of secret communication (without which any conspiracy dies). He intends to do this through random attacks on government secrecy, with the goal of forcing an expensive overreaction, which will end with governments being less secretive.

My first reaction: This dovetails perfectly with a blog post that I've been meaning to write (but will probably never get around to) about the tradeoff between trust and robustness in a networked system. It's actually a really cool tradeoff - trusting another entity in a decentralized system can be viewed as a dodgy optimization, which will usually work but occasionally crashes dramatically. (Bonus: the tradeoff even has a mathematical basis, in the FLP result!) Julian Assange is giving us a real-world demonstration of this principle, by poking at the relatively cosy relationships between governments and forcing them to shift into a less useful but more secure configuration.

My second reaction: You know how the .gov has been making a lot of noise about info-terrorists, even though they have no idea what that even means? DDoS kiddies are usually held up as an example of what to watch out for, but that stuff is so trivial that I'm surprised we waste our time talking about it. Julian Assange, on the other hand, is the real deal, and he's not even terribly sophisticated. He is using the power of the Internet, and the power of the (relatively) unrestricted flow of information, to do something radical to the state.

My third reaction: Oh, man. The government doesn't know how bad this could have been. If Wikileaks had wanted to publish this stuff anonymously, it wouldn't have been terribly difficult for them to do so. The technology already exists, and has for years; it's just a matter of using it effectively. They don't like their diplomatic cables being made public as it is; imagine how much it would suck for them to have a few thousand cables appearing every month, and to be completely unable to track where they were coming from. You know how I said that Assange wasn't terribly sophisticated? If he were, he'd be doing exactly what he's doing now - we'd just have no idea who he was.

Last reaction: I can't help but worry that Julian Assange is gearing up for a dramatic exit from this world. He is simultaneously making himself extremely visible, and making a lot of very powerful enemies. Wikileaks has already published an insurance file; that's not the sort of thing you do unless you expect to have a reason to use it. If Assange does end up assassinated, that may be all the proof we need that something like Wikileaks is desperately needed in today's world.

Rewind

2010-11-29T23:37:00.000-08:00

So I read the Void Trilogy by Peter Hamilton a few weeks ago, and one of the subplots went like this: in a world of psychics, one young man has exceptionally powerful abilities. Throughout the books, he learns of increasingly incredible things he can do, until he realizes that he can turn back time itself. Specifically, he can think about any moment in his past that he can remember clearly, and rewind the universe back to that moment (but with all his memories intact). This is where things get a little bit nuts.

For the rest of the book, he tries to make everything right with the world, because he's that sort of character. It takes a terrible toll on his mind at times, but in the end, he lives a life such that there's nothing he wants to go back and fix, and he has reached fulfillment. Happy ending, right? And then he goes and, on his deathbed, gives the secret of turning back time to everybody else in his city - and this is where my brain implodes in dismay.

If zero people know how to turn back time, then things make sense, and history proceeds in a boring linear fashion. If one person knows how to turn back time, then things are still simple enough to wrap your head around, because you can trace a single thread of narrative throughout whatever they do - by designating them the "main character" in the story, the story makes sense. But if two or more people know the secret, then things get Terribly Complicated.

Here's one trivial example of how screwed up the universe would become: imagine a game of Rock-Paper-Scissors between two especially competitive people that know how to turn back time (Rewinders?). The entire universe would be locked in a loop until one of them got bored.

There are weird issues surrounding seniority. If two Rewinders are going back and forth on something, the winner is going to be the one that can go the farthest back - back to before the other one existed, perhaps. If we follow this train of thought, then the winner in any conflict is going to be whoever is the oldest.

On the flip side, there are weird edge cases around death. If I sneak up on someone and kill them before they can react, then that's it for them, I've won, no second chances. This is the only way I can see to break out of a loop without first going through the infinite regression tango, and giving the victory to the older person. A world full of Rewinders would have a lot of immortals trying to kill each other, really - sort of like The Highlander but with more mindfuck.

I'm not really going anywhere with this post. Honestly, I just thought it'd be fun to actually think through some of the consequences of a world with Rewinders. :D

Diaspora!

2010-11-28T23:50:00.000-08:00

So at the beginning of this summer, a group of NYU CS students started on a project to build a decentralized social network, and then made waves when they raised over $200,000 on kickstarter, a crowdsourced funding website. They then proceeded to disappear into a cave for the entire summer, which killed the buzz around Diaspora pretty effectively. Then they put the project up on Github, and people immediately jumped all over them for security flaws. (Personally, I would expect to find security holes about that magnitude for a project this young. You fix them, and you move on.)

If I had to give my opinion of the project, it's somewhere around "cautious optimism". I'm not a Ruby or a Rails fan, but there are worse languages/frameworks they could have used. I think they're striking a reasonable balance between developing in secret and developing in public. On the one hand, they promised to make everything 100% open source, but on the other hand, the open source development model is pathologically incapable of making design decisions, and for the initial stage of a project you're making nothing but. I definitely like that they're piggybacking on existing protocols.

Apparently, they were inspired by Eben Moglen's idea of a "freedom box", which makes me sort of nervous, actually. Nervous, because the idea is good in principle, but completely unworkable and sort of silly in practice. Yes, it would be useful if we all had physical control of our own social media profile, but this has tremendous implications for the reliability of the network as a whole - if my Internet connection goes down, to what extent do I disappear from the web? And, of course, I'm glossing over all the real difficulties with hosting a website on a residential Internet connection. Quite simply, our infrastructure isn't up to the job, and I don't expect that to ever change. So, I kind of hope that the Diaspora devs aren't going to waste too much time on this particular use case.

There are also a ton of fundamentally hard problems that they are going to run into, and while I remain optimistic that they're thinking about them, we won't really know how they handle them until the software is in a more complete state. For example: how do you handle security updates in a worldwide distributed system? There are already a ton of insecure Diaspora instances running around in the wild, that people brought up as soon as the code landed on Github, and the problem is going to get worse unless they do something about it.

Overall, I have high hopes for Diaspora, but it's simply too early to make a call about the project. I'm expecting it to advance rapidly, though, and we may be looking at a 1.0 release within a year. Whether or not it's a "Facebook-killer", like people want it to be, it has a lot of potential to be a useful tool.

Wallet

2010-11-27T00:00:00.001-08:00

I seem to have lost my wallet! It is fucking with my head like you wouldn't believe.

I have looked everywhere. I have looked everywhere at least twice. I've crawled on the ground looking underneath maybe half the furniture in this house. I've torn my room apart - I don't think there's a square inch in there that I haven't looked at today, except for maybe spots that you have to disassemble furniture to get to. I have taken all the cushions off of all the couches, I think. I've called the last place I saw the wallet, and since I came home straightaway after that and haven't really gone anywhere since, there are no leads there. I am this close to trying to figure out a tactful way to call up everybody that was here for Thanksgiving and asking if they walked off with my wallet. Like I said, fucking with my head.

Why am I freaking out so much? This is my first time losing my wallet, and I suppose my first time finding out just how much of a pain it is. If I can't find it before my flight on Sunday, I'm going to have to cancel my credit card and debit card, replace my insurance card, replace my Social Security card, replace my ORCA card (free bus rides, one of many Microsoft perks) so I can ride the bus to work. (By a sheer stroke of luck, I still have my driver's license; don't even ask. XD)

And then, I have to figure out how to keep this from ever happening again. Because, see, I can't just leave a problem like this alone, and deal with it when it comes up. I'm a pathological overthinker. My reaction when a hard drive fails is to create increasingly elaborate system of redundant storage, culminating in what I built earlier this month. My reaction to almost losing my cell phone is to keep multiple backups of all the data on it, just in case. Now that I've lost my wallet once, I'm not sure that I'll be able to ignore the possibility of it happening again - my brain just doesn't work that way. I don't know how I'll solve the problem of randomly losing things that I need to carry around everywhere I go, but I know that I'll be kind of agitated and jittery until I do - it's just how I'm wired.

My mom, bless her heart, tried to help - not by helping me look, but by trying to make me feel better about losing my wallet. Frankly, it just made things worse. She asked me to imagine the worst-case scenario; thanks for the completely generic advice! I definitely feel inclined to sit and listen to you, when I know that you're taking this far less seriously than I am! Plus, I know on an almost subconscious level that she's just trying to make me feel better, and that's not what I want. I don't want to feel better about it, and I don't want to sit down and think about "how bad could it really be". I want to find my goddamn wallet. If all your help means to me is that I have to act calmer while I'm searching frantically, then please, just stop trying.

Anyway, I haven't given up yet. I still have another day to try and figure out where it went. I am convinced that it's still in this house somewhere, and that's the most incredibly frustrating thing - it's so close, but I may not have enough time to find it! Still, I've got until my flight on Sunday. Once I get on the plane, then I'll give up, and start figuring out what all I need to replace. Until then, there's still hope.

The Science of Code

2010-11-26T21:46:00.000-08:00

Found this blog post today via reddit. It has a really cool insight: The way people work with code is evolving into the same patterns that exist in the sciences today. At the one end you have the "physicists" - people that work with code on the lowest levels (either machine code, or algorithms, depending on your interpretation), and that can expect mathematical certainty. At the other end, you have code "biologists", that mostly work with whole organisms/programs, which are messy things, but which mostly work in mostly predictable ways.

There are a few neat consequences that you can pull out of the analogy. First, while wizards slinging machine code and novices putting scripts together are both "programmers", we probably need new designations for them, in the same way that you can't always lump physicists, chemists, and biologists together as scientists. Second, even though scripting is perceived as easier than low-level programming today, that could be because of the relative immaturity of the field, and not because it's inherently easier. See this comic, for example: physicists can look down on biologists, but biology is hard! Physics can be seen as the ultimate reductionism, and other sciences are simpler in terms of the physics they use, but harder precisely because they can't afford to reduce everything to that degree.

Higher-level programming languages, then, aren't just about simplification - they're also about specialization. (Maybe this is why domain specific languages (DSLs) are a big deal today? By creating a new language, you're jumping ahead of the existing languages in terms of specialization, which is akin to opening up a new field of study in our analogy.) By leaving some of the complexity of the lower levels behind, you're able to create new abstractions and concepts, which are interesting in and of themselves.

I think the analogy actually outstrips modern programming practices by a bit. If you want to write "organic" code, for instance, you need a specialized language like Erlang, since as far as I know it's the only language designed to handle failures of different parts of the program, and keep on running smoothly. Current languages mostly have the assumption that any fault is reason to terminate the program, because the whole thing should be 100% correct. From a physicists perspective, this is fine - if it's not 100% correct, you can't count on it doing anything right! I'm coming around to the "sloppy code" view the more I think about it, though.

The assumption that all code should be 100% correct is unreasonable in this day and age. It pains me to say it, because it goes against everything I've been taught (and quite a bit of what I've said in the past). All code is going to be a bit sloppy, simply because it's written by humans, and not by the faultless code-writing machines that those humans fancy themselves to be. What we need in the next generation of languages is more robust mechanisms for handling incorrect code; if we don't do that, we're not really designing languages to be used by human beings.

Layers of Fail

2010-11-25T23:01:00.000-08:00

So here's an annoying multi-layered fail, notable because it affects three adjacent layers of the network stack! As any programmer will tell you, the most interesting bugs to diagnose are the ones that result from the interaction of other bugs. This particular one results in me losing messages on AIM.

First fail: U-Verse and my Mac

I don't know if anybody else has seen this problem, but my Macbook Pro cannot keep up a reliable connection to any U-Verse modem. I've seen this problem with multiple U-Verse modems, and only my Mac, so the possibilities are (in order of likeliness):

Buggy AT&T software in the modem
Buggy firmware for my wireless card
Random hardware fault in my Mac (unlikely, because it only happens with U-Verse modems)

This is pretty annoying, because WiFi is supposed to be standardized! All implementations are supposed to be interoperable with all others. Either AT&T or Apple could have caught this (isn't it standard practice to test with other widely-used hardware?), so the fault could lie with either company. Luckily, I don't have U-Verse at home, so I don't have the need to diagnose this properly - it's only an issue when I'm visiting people that do, like my parents.

Second fail: Automatically dropping connections on interface down

This is such a widespread thing that I think it must be intentional, but I can't figure out any reason that it's not a terrible idea. On any OS that I've used, when a network interface goes down, all connections are severed automatically. The thing is, the IP protocol is explicitly designed to allow lost packets, and the TCP protocol on top of it is designed to handle it, so the dropping of connections is unnecessary. If operating systems just ignored the loss of the interface on the assumption that it'll come back up soon, everything will still work as designed, and a lot of situations involving intermittent connections will work much better!

In other words, effort went in to a feature which makes things worse, which definitely counts as a failure in my book.

Third fail: AIM protocol doesn't handle dropped connections cleanly

This one's pretty simple: if a connection drops, and a message is sent during the timeout before the server decides that the connection is dead, that message seems to be lost. Seems like a simple bug to fix on the server, but it's been going on for a while now, so apparently that's not going to happen. What's more irritating about this one is, AIM already seems to save messages that are sent to somebody that's offline - it just can't detect that you're offline during the timeout period.

The end result of these three (or two, if you don't want to count the middle one) bugs is that AIM is nearly unusable for me when I'm using the WiFi at my parents' house. (Yet another reason to switch to GTalk? :D No idea if it has the same problem, though.)

"Content" and "Consumers"

2010-11-24T00:01:00.000-08:00

So am I the only person that gets annoyed when people talk about "content"?

The phrase is aggressively bland and intolerably reductive. "Content" can be anything. "Content" is a book, an article, a song, a cool video, a funny joke, a clever program, a picture worth a thousand words - all glommed together into a single nondescript concept. Nobody who has created something worth creating, something that they're proud of, will refer to it as "content". "Content" is anonymous. "Content" is undifferentiated. "Content" is everything, and nothing in particular. "Content" is what you call it when you don't care what it is.

"Content" is what you feed to "consumers" - nothing more.

And there's another word that bugs the hell out of me: "consumers". A "consumer" isn't a person - it's a thing, a mindless black hole, which consumes whatever you put in front of it. People are complex and individual, but we know how to market to "consumers". A "consumer" isn't something you would ever have a conversation with. A "consumer" never produces anything of value. "Consumers" are all the same, and when they're not, they're amenable to market segmentation. "Consumers" exist because they're easier to deal with than people.

I think that the mindset embodied by these two words is a sickness. When you see the world in terms of "content", and "consumers" that want it, you're hiding away all the incredible richness of the world. "Content" is culture, viewed from the outside, with the intent to put it in boxes and sell it to people - sorry, "consumers" - and it honestly freaks me out a little that there are people out there that are willing to think in those terms.

So please. Can we all stop using these words? I think we'll all be happier in the long run.

The Shallows

2010-11-23T22:39:00.000-08:00

So like I mentioned yesterday, I've just finished reading The Shallows, by Nicholas Carr. It's about the effect that the Internet is having on our thought processes, on a fundamental level, and it's a pretty amazing book.

We see the Internet as an incredibly powerful tool for organizing and retrieving information, and we're right - as a multiplier for our own abilities, it's an unprecedented achievement. What some people are realizing, however, is that it is not without its downsides. Ask yourself this: how many books do you read for fun these days, and how many did you read before you discovered the Internet? Carr contends that this is not just because our reading has shifted online; instead, our brains have been rewired by the constant cheap stimulus of the web, so that we find it more difficult to read extended prose than we used to.

Historical precedent

Believe it or not, this isn't the first time that a new technology has fundamentally changed how we think. Carr looks all the way back to the invention of written language, which represented the beginning of the shift from an oral culture to a written one. There's some interesting stuff here that I didn't know about. For example, written languages were originally written using only letters, with no punctuation, or even spacing between words. See, written language was originally just a transcription of what people said out loud, and only intended to be read out loud. It wasn't until hundreds of years later that these things were added to language, to make reading easier.

The revolution brought on by written language reached a fever pitch with Gutenberg's invention in the mid-1600s. All of a sudden, books (previously reserved for the wealthy) were cheap enough that everybody could have them. Simultaneously, the automation of printing meant that the cost of introducing a new book decreased dramatically, so that authors were free to experiment with radical new styles of writing. (Critics of the time decried the new styles, a product of new technology, as a massive dumbing down of literature - sound familiar?)

The introduction of reading to the population as a whole caused a fundamental change in the way we thought. Carr starts the book by just asserting this, but when he really gets into the meat of his argument, he backs it up with a lot of neurological evidence - mostly the dramatic changes that occur in our brains when we learn to read, as seen through an MRI.

Shallows

So how is the Internet changing things? Carr cites study after study that all point in the same direction - the Internet diminishes our ability to read deeply, by encouraging skimming and by allowing us to skip around to other pages easily. Furthermore, it does so on a fundamental neurological level. This is the core argument of the book, and Carr backs it up well, going all the way down to what we know about the mechanisms of human memory.

There are lots of counterarguments to be made here, of course. Even if the Internet makes it harder for us to absorb a book of information, doesn't it make up for it by giving us easy access to anything we might want to know? Doesn't the fact that looking up information is now instantaneous make it more efficient for us to know a wide variety of subjects shallowly, rather than learning a few subjects more deeply? (Personally, I'd say that it depends - some questions are harder to answer than others using a tool like Google, and it's an effect that's difficult to take into account when deciding what to learn.)

What should we do about this? It's hard to say. Really, it's hard to even say what we could do - people aren't even convinced that this is a problem today, and that would be a necessary prerequisite before we could even talk about steering the tech industry based on the effects that technology would have on us.

The technocrat in me thinks that we should just let technology happen, and deal with the consequences after the fact. Technology can solve technology's problems, and everything will work itself out, right? I used to believe that, anyway, but now I'm not so sure. (Maybe I don't have as much faith in humanity as I used to?)

In any case, though, everybody should read this book! Even if you don't agree with this author's premise, you'll probably find it an interesting read.

Nook!

2010-11-22T23:58:00.000-08:00

I've so far neglected to mention it here, but I got a Nook!

Compared to the Sony Reader that I've had for a few years now, this is a huge improvement. I'll start with the most visible difference - while the Reader has buttons numbered 1-10 plus a few more for navigation, the Nook has a color touchscreen. They use it with the main E-Ink screen fairly effectively, too - the interface is generally divided between them sanely, with stuff being displayed on the big screen, and all the controls on the touchscreen.

The E-Ink screen on the Nook is noticeably better than the one on the Sony Reader. The contrast is much higher, and the resolution seems a bit better too (though that could also be the font, I guess). The Nook is also much smarter about updating the screen; where the Reader would have to refresh the whole screen with an annoying invert-everything action, the Nook can update individual pixels without disturbing the rest of the display.

The other most important difference is the inclusion of a wireless connection, like the Kindle. I'm pretty impressed by how well this feature works - I can browse the bookstore, buy a book, and download it to the device within a span of a few minutes. (Browsing over a cellular internet connection is a bit sluggish, though. If you can, it's much easier to buy the books through the B&N website.) There's also a WiFi-only version, but I think that's not as much fun. WiFi can be hard to find, and it's nice to be able to get new books from a moving vehicle. As far as ebook pricing goes, I don't have any complaints about the B&N store.

Battery life is a different story. While reading, the battery lasts pretty much forever (and by forever, I mean it should last through any one book, at least that I've found), as expected of an E-Ink device. When you put the device to sleep, though, it only lasts for a few days before needing to be charged. This is pretty disappointing, honestly, and I hope they fix it in a future software update.

Today on my flight to Houston, I brought the Nook and finished The Shallows, which is about how technology is changing our reading habits. (Wooo, irony!) I think I'll be talking more about that in tomorrow's post, but as far as the reading experience goes, it was at least as comfortable as reading a real book. Probably more so, in fact, because the Nook is easier to fit in my bag than the paperback (The Prefect, by Alistair Reynolds; somebody lent it to me and now I need to read it!) I brought along as a backup.

Summary: The Nook is kind of awesome, and I need to finish this post within the next minute or I'm not going to finish in time.

The Man-in-the-Middle Defense

2010-11-21T22:26:00.000-08:00

I'm not sure why this is, and it's probably a really bad idea, but for some reason as BitTorrent gets harder to use (with all the major trackers being targeted), I become more and more motivated to come up with a p2p protocol that doesn't have BitTorrent's weaknesses. So in that vein, here's something I've been thinking about.

One of BitTorrent's biggest problems is that every user knows about all the other users, and so sending nasty letters to everybody downloading a given file is as simple as grabbing a list of IP addresses and contacting a bunch of ISPs. What we need, then, is a way to anonymize connections, and transfer data between two computers without either of them knowing the identity of the other. There are already a few protocols that get us partway there: using a traditional proxy, you can mask the address of one party, but not both; using something like Tor, you can prevent any one proxy server from knowing both the client and server address, but the client still has to know the server address. So what can we do instead?

Repeaters

The golden rule of system design: When in doubt, go for the simplest thing that could possibly work!

So in this hypothetical protocol, two peers (Alice and Bob, why not) have been communicating pseudonymously. At some point, Alice wants to send Bob a file which is too big for the channel they've been using, so they both agree on an anonymous repeater. At some predetermined time (in most protocols, the time would just be "now"), they both connect to a server, and the server just repeats everything it sees on one connection to the other.

(Once you've got the simplest thing, you may need to fix up a bunch of other problems.)

First, we need a way for a single server to distinguish between a lot of people connecting to it at once - which of them actually want to talk to each other? One solution is to have Alice and Bob agree on a shared secret ahead of time. When they both contact the repeater, they send the shared secret, and the repeater knows to establish a connection between them.

That's weak against eavesdroppers, though, and in today's world of relatively widespread DPI gear, you just can't trust the network. This can be solved with SSL, but only if Alice and Bob can agree on the repeater's public key ahead of time. (If not, then SSL certs are easy enough to fake - it's a little extra effort to do so, but if we were only defending against unmotivated eavesdroppers, then this would be pretty easy!) Solving this problem properly will take a little more thought than I'm willing to put into this problem tonight, unfortunately. (Maybe passing the SSL layer through the proxy?)

How do we keep the repeaters from learning about Alice and Bob's identities? For that, we can use proxies - we only need to keep their identities safe from the repeater, not the other way around, so existing proxy mechanisms will work for this.

How will we handle the performance cost of going through so many layers? Alice and Bob can agree on some protocol ahead of time for dividing the data across several paths, or something like that. I'm explicitly not designing that layer here - proxies and repeaters are general enough mechanisms that more complex protocols can be implemented on top of them pretty easily. (That's one of the biggest advantages of using the simplest thing!)

As a completely unintended bonus, if anonymous repeaters like I've described here become widespread, that could be a solution to the problem of establishing a connection between two computers that are both stuck behind NAT.

Digital Dead Drops

A completely different solution to the original problem would be establishing "dead drops" - locations where you can drop a file for a certain amount of time, and somebody else can pick it up later. (I've already seen pastebin used like this, come to think of it!) If both parties use proxy chains, and the data is encrypted, then this is even more secure than using repeaters because you avoid the simultaneous connection problem - an eavesdropper has a hint about who you're talking to because you're connected (however indirectly) to the other user.

The Next Problem

The other major problem with p2p networks is that search is public - by making a file available for others to download, I'm also announcing to the world that I have that file, and some people might be upset about that.

I have some ideas about how to solve that, but I've probably spent too much time blogging about this topic already. Instead, I think it's time for me to start coding these things up, and see what works. Should be fun!

Chaos Theory

2010-11-20T22:21:00.000-08:00

The point is, ladies and gentleman, that chaos, for lack of a better word, is good. Chaos is right, chaos works. Chaos clarifies, cuts through, and captures the essence of the evolutionary spirit. Chaos, in all of its forms; chaos in life, in money, in love, knowledge has marked the upward surge of mankind. And chaos, you mark my words, will not only save Android, but that other malfunctioning entity called the US cellular industry.

This completely unnecessary riff on Gordon Gekko was inspired by this article, which asserts that Windows Phone will have an advantage over Android because it's less "chaotic". How horrid! That kind of platform lockdown moves Windows Phone onto the iPhone's turf,

The iPhone model (one OS running on one device, all controlled by one company) works for Apple, but I personally think it's a fluke. By launching the first smartphone platform in the multitouch paradigm, Apple snatched up enough of the market to become entrenched, and by the time competitors showed up, Apple was on the second or third revision of the iPhone, giving them a lead that could take a decade to wear off. I would argue that they succeed today in spite of (and certainly not because of!) their locked-down ecosystem. The iPhone hasn't changed significantly since the first revision, and at the rate things are going right now, it might not be premature to label the iPhone a legacy platform.

Android, on the other hand, is chaos. Google all but throws the software out there, yells "Come and get it!", and lets the carriers and manufacturers do whatever they want with it. It's sometimes inconvenient, but it also leads to some really cool stuff. Can you imagine something like the Nook (which secretly runs Android) being built on top of the iPhone OS?

I sometimes joke about Windows Phone copying the iPhone to an astonishing degree, but it still worries me. The iPhone model is something that will work once, for the first company to come up with a revolutionary new paradigm, but after that a single entity can't keep up with the rest of the industry for very long. Microsoft does seem to be doing a lot of things better than Apple (C# doesn't make me want to kill myself the same way Objective-C does, for instance), so Windows Phone is still a compelling platform, but if Microsoft wants long-term viability it's going to have to look beyond what Apple is succeeding at today.

We must remember that Windows - and, hell, the entire concept of an operating system - came into being to manage chaos, not eliminate it. Back in the Bad Old Days, hardware platforms were fairly standardized, and introducing any changes was likely to break existing software. This is where turbo buttons originally came from, and it's not a model we should be aspiring to. Having a number of different hardware profiles may make Android harder for developers to test their apps on, but it also means that the platform is much more flexible, and by forcing developers to account for different hardware profiles, Google is creating a better platform for the long term.

Any Windows program will (with very few exceptions) run on any Windows computer. Microsoft's past success with that model wasn't a fluke - it was a testament to the fundamental power of a model that allows for a wide variety of future hardware improvements. It's interesting to see how closely the development of mobile phones is mimicking the development of PCs 25 years ago; some of the players (like Apple) are even moving into the same positions. Microsoft has already managed to invent the PC once, and I hope that some people at the company remember enough to be able to invent it again.

In the end, it comes down to how mature you think the smartphone market is. If you think most of the innovation has already happened, then bet on the vertically integrated platform (like the iPhone). If you think most of the innovation has yet to happen, then bet on the open, flexible platform (like Android). It's still early enough in Windows Phone's development for Microsoft to make a choice, and I really hope they choose the latter.

Ken Robinson on Education

2010-11-19T20:02:00.000-08:00

Today: lazy. Videos!

New phone!

2010-11-18T23:35:00.000-08:00

So I've started thinking about getting a new phone!

Windows Phone

On the one hand, there's Windows Phone. I haven't had great experiences with Windows Mobile in the past, but Windows Phone is a totally new platform, and it actually seems pretty nice. I get one for "free", as a Microsoft employee - free with a two-year contract, though, so really it's closer to half price. Everybody else around the office is pretty stoked about them, and the enthusiasm is a bit infectious. The development tools look pretty awesome, too.

On the other hand, it's a totally new and somewhat unproven platform, with a radical new user interface to boot, and hardly any apps written for it yet. There's also the Microsoft factor, as much as I hate to admit it - I just don't have a ton of confidence in Microsoft's ability to execute in this space. (Here's an example: I use Microsoft My Phone to sync my phone to the web right now. It's a really useful service, but Microsoft is killing it in favor of Windows Live, and I have yet to see any official solution for migrating my data to Windows Phone. No other company in the world could get away with this kind of lack of focus.)

Pros: Employee price, easy to code on, supporting my employer, shiny.
Cons: New; unproven; Microsoft

Android

In the other corner, we have Android. It's proven to be a solid competitor in the mobile space, and is usually mentioned in the same breath as iOS these days. It's also Linux-based and relatively hackable (in the good way), and I have to confess to a certain amount of nerdy glee at the thought of being able to ssh in to my cell phone and poke around. >_> The open source factor is also a plus, even if Android isn't going along with the spirit of open source at all.

There are also problems with Android, foremost among them being that manufacturers usually don't release updates to phones that aren't ridiculously popular. The Nexus One gets updates, the Droid family gets updates, and so do other phones of similar notoriety, but most aren't so lucky. I'd also be the only guy at Microsoft without a Windows Phone, which could get awkward.

Pros: Solid platform, open source(-ish), hackable, has momentum
Cons: Carriers have too much control, Google is slightly evil
Bonus: Microsoft's strategy against Android is pretty much the sketchiest thing possible (future blog topic), and actually inclines me to support Google. <_<

iPhone

The thought of carrying millions of lines of Objective-C code in my pocket just makes me shudder. No iPhone for me.

MeeGo

MeeGo is kind of the dark horse in this race. It's a joint thing between Intel and Nokia (and a few others?) to put together a proper open source Linux-based phone OS. (None of this Android-style "code drop" crap.) This would be a really compelling option for me if they'd gotten around to releasing any phones yet, but... well.

There is the Nokia N900, which you can run MeeGo on, but I don't get the impression that it's especially well supported, and the N900 is already a year old. Overall, it seems like MeeGo is at too early a stage in its development to consider, but if Nokia or Intel announced a phone tomorrow that had high-end specs and ran MeeGo natively, well, that'd certainly throw a wrench into my decision.

So. Anybody got opinions? :D

The Decline and Fall of the Facebook Empire (part II)

2010-11-17T23:28:00.000-08:00

(click here for part I)

Yesterday, I blogged about a lesson that Facebook is soon going to learn. That lesson is this, and comes in two parts: when you integrate your services, you're actually competing with the Internet. And, when you compete with the Internet, you lose.

The Second Thing

So what does it mean to compete with the Internet, and how can you avoid it?

The Internet is really just a consensus built around a set of protocols; it only holds together as well as it does because everybody agrees to either work within the framework of all the protocols that exist, or to build new protocols and try to build consensus around those. When you build a new service that follows existing protocols, you're working with the Internet, and strengthening the whole network, because it makes it easier for people to benefit from your innovations. So, people innovate within the framework of the Internet because the users are there, and users are there because innovation happens there - it's a virtuous cycle, in other words.

(If you've ever heard people talk about "open standards" in reverent, near-religious terms, it's because they've realized just how powerful this virtuous cycle is.)

That's why competing against the Internet is a losing proposition. Its creators built a system where anybody can improve the whole by adding their bit, and by now it's built such a tremendous momentum that you have to be as large as Facebook to make any headway against the flow.

Facebook has sort of a mixed history when it comes to open protocols. The website (their main service) is pretty locked-down; things like RSS feeds exist, but they sure don't want you to find them! On the other hand, Facebook chat is built on XMPP, and as a result it was trivial for most instant messaging clients to add support for it - an example of the power of using open protocols.

It's possible that they'll somehow manage to map their new messaging service onto an existing protocol (IMAP, if we're lucky), but I have my doubts. If they were doing that, it seems like the sort of major feature that they'd want to mention! Far more likely that they'll expect you to use it through their web interface, and maybe have some limited integration with other services.

On the other hand, some companies actually get it. I may have misclassified Google the other day, for instance - they expose a lot of their services through their GData APIs, and tend toward using open standards without restrictions when possible. (For instance, they were one of the first free webmail services to give people access to IMAP and POP for free.)

The Decline and Fall of the Facebook Empire (part I)

2010-11-16T23:04:00.000-08:00

There is a hard lesson that Internet titans learn eventually. AOL was the first to learn it, and it led to their long, drawn-out demise. Yahoo learned it, and is now a shadow of its former self. Microsoft and Google have both come up against it, but been reasonably successful in avoiding it; the former by shoveling money at the problem, and the latter through sheer heroic engineering effort. And, with yesterday's announcement about a unified messaging system, Facebook is about to learn it - and doesn't have the resources to power through anyway, like some others.

That lesson is this, and comes in two parts: when you integrate your services, you're actually competing with the Internet. And, when you compete with the Internet, you lose.

The First Thing

Why does integration equate to competing with the Internet?

Let's say I think this new messaging platform is awesome. IM? Great, all my friends use Facebook chat anyway! SMS? Sounds pretty cool! Email? ...Wait, hold on a second. You mean I have to give up GMail to get all the benefits of this integration? :(

In other words, this kind of integration is a form of soft vendor lock-in: they make it more convenient to use all of their services, rather than using the best of what the 'Net has to offer. AOL is probably the strongest example of this - back in the day, they combined an ISP, web browser, and email service into one package. And, they quickly started hemorrhaging customers, because they couldn't compete with the best ISPs, browsers, and email services all at the same time.

See, savvy Internet users know how to mix and match services. Back in the '90s, AOL wasn't just competing against all the other companies in the same space - it was competing against every possible permutation of an ISP, a web browser, and an email service that people could come up with. And today, Facebook is trying something similar. By combining a bunch of similar services under the umbrella of "messaging", Facebook is competing with every possible permutation of messaging services available online.

And here's the kicker: They're not just competing with every permutation of services that do the things they do. They're also competing with every new service that lets people communicate. They don't have an answer for things like Skype, for instance. And the thing about new services is, there's never a right answer for how to deal with them. The only right answer is not to put yourself in a situation where you're pressured to mimic every new technology that comes along.

I mentioned earlier that the savvy Internet users can often find better combinations of services than any one company can provide. What about the non-savvy users? They stick around, sometimes for years after a service has lost its vitality, and you end up with digital ghettoes like Hotmail and AOL. That's how I see yesterday's announcement - it's Facebook's first step toward becoming yet another digital ghetto.

(continued in part II, tomorrow!)

Procrastination

2010-11-15T23:12:00.000-08:00

So here's the thing about procrastination: I think I actually enjoy it. For some reason, having something that I need to do, and procrastinating by doing something else, is way more fun than just doing the something else. This worries me.

In that sense, of course, blogging for a month is a good thing, because it's something not to do while I do other stuff, but it does lead to situations like right now, where I have an hour to write something an absolutely nothing coming to mind. >_>

headache

2010-11-14T22:42:00.003-08:00

headache

cannot blog

sleep now

:(

EDIT: Okay rather than skipping a day, I will retroactively make this a post about what I spent most of yesterday on: Harry Potter fanfiction. >_> So here are two fics that I've read lately, that are worth linking!

Harry Potter and the Methods of Rationality: Most of you have already seen this, because I've been recommending it to everyone. The story branches off of canon with Petunia realizing that she can't stand Vernon Dursley, and marrying a college professor instead. Harry grows up immersed in the scientific method, and applies it to the magical world, with sort of awesome results. As a bonus, this story also has Professor Quirrell as a real character (and one of the more interesting ones, at that), instead of just a cardboard cutout. As a further bonus, Harry is slightly evil! As a third and completely gratuitous bonus, it's full of references to pretty much everything else - the latest chapter, for instance, has (after a few chapters worth of buildup) the greatest Army of Darkness homage possible, I think.

Anyway, I enjoy it trememendously (that's present tense; it's an ongoing story) but apparently not everybody does? Definitely not my problem, though. :D

Harry Potter and the Wastelands of Time: Found this one because it was recommended by the author of HP:MoR. It's... not like any Harry Potter fic you've read before, I think. It's incredibly dark, for one thing.

The premise is that Voldemort won the war (the second time around), by discovering Atlantis and using the knowledge that he found there. Harry can't even come close to stopping him, so when he finds himself alone in the world, he makes a deal with a demon for a second chance. This puts him in a timeloop (think Groundhog Day), which begins shortly after OotP and loops back whenever he dies. It takes a lot of tries for him to accumulate enough skills across his lifetimes, and figure out a plan - thousands of tries, you're left to infer, because the story never specifies. Again, that's where the story starts, with a jaded and tired Harry Potter in a sixteen-year-old body getting ready for another go.

It's an interesting take on first person omniscient - Harry mostly knows what's going to happen next, but he doesn't let on much to the reader, and he's still occasionally surprised by things. It's also a good way to keep the action going - because Harry's been refining his plan for so long, he always knows what to do next, and the action never really slows down. Be careful before you start reading this fic, because you might have a hard time stopping.

Anyway. On to a post for today. XD

2010-11-13T23:00:00.000-08:00

Sometimes, in the middle of a conversation, I'll spin off into a tangent which is somewhat factual sounding, and completely untrue. The example that comes to mind at the moment is convincing a friend of mine that there's no such thing as narwhals ("Sea unicorns? Really?"), while we were working on a project for some class. (Apparently it took a while for other people to convince him otherwise; I am vaguely proud of this, if not exactly proud of being proud of this, if that makes any sense.) This is a thing I do all the time.

Why? Part of it is because it's hilarious, sure. Another part is that I'm terrified that if I didn't tell blatant lies from time to time, people would just accept everything I said without questioning it. It sounds like a silly concern, but it actually happens more than I'd like, and it bothers me whenever it does. I've tried being that guy that always knows what he's talking about and never leads you astray, and it is no fun. :(

It's also a lot of fun to tell elaborate fictions, of course! You get to re-cast a piece of the world under an alternate system of rules, which isn't quite internally consistent, and try to run ahead of the other person to paper over the inconsistencies before they find them. If they catch up and figure out a hole in my story that I can't work around, then I lose; it's sort of like a mental game of tag.

On the other hand, there's also the possibility that I just inherited it, since my dad does the same thing sometimes. >_> There's a story my family loves to tell, about how one of my aunts asked my dad how he kept the lawn so nicely mown. He told her all about the Rent-a-Sheep service, where they'll bring a few sheep out to your house every once in a while to munch on your grass, and he told it so convincingly that everybody bought it. My dad: occasionally pretty awesome. :D

The Cloud Scalability Distraction

2010-11-12T21:48:00.000-08:00

I have this sneaking suspicion that Amazon has done something extremely clever with the cloud. When they launched their cloud services a few years ago, they made a big deal about how well they scaled up - S3 can store unlimited data! You can start up as many EC2 instances as you want! They even drive the point home with their pricing - S3 pricing is tiered, and there's a tier for people storing more than 5 petabytes of data, "proving" that S3 will easily scale to that amount. (Having that tier listed on the website is a bit unnecessary - if you're going to be spending millions of dollars per year on S3, you're probably going to get to talk to a salesperson face-to-face, just saying.)

Why is this clever? Because all of Amazon's competitors in the cloud space followed suit, and talked about how well their services scaled up, too. Meanwhile, Amazon's been advancing in precisely the other direction - the ability to scale down. For example, they recently launched "micro instances", which are cloud servers that rival the cheapest VPS providers in price.

Why does this matter? Because scaled-down cloud services are going to be the next revolution in computing. Right now, "cloud" is more of a marketing victory than anything else; there's nothing there that a few competent sysadmins and devs couldn't put together in a week or two, for a few times the price. The biggest advantage is the pricing (and it's telling that Amazon is on the forefront of it; the expertise they had that contributed most to the cloud wasn't technology, it was payment processing on a massive scale). Cloud services are still pretty coarse-grained, compared to what they could be.

Right now, the technology that would make the cloud "revolutionary" hasn't been invented yet. (I may have said this before.) I think that that technology is going to be something that enables partitioning up cloud services a few orders of magnitude more efficiently than we can today. Imagine a system where the resources can be sliced finely (and cheaply) enough that it makes sense to integrate them into desktop applications, and have the user pay for it - that's when you'll really have something revolutionary on your hands.

(aside: the title of this post makes me think of the big bang theory T___T)

Ice, Ice, Baby

2010-11-11T23:19:00.000-08:00

The weather the past few days has reminded me that I just moved two thousand miles north. In the evenings it's getting down into the 40s, and there's a permanent light drizzle which just maker things unpleasant. I've also been reminded that I really, really hate the cold. Whoops!

On the other hand, I'm definitely doing better with the cold than I would have in the past. Somehow, I feel like I've begun to adapt - temperatures in the 50 to 60 range feel like 60-70 degrees used to feel, and I can usually get away with just wearing an extra layer of shirts, or maybe a sweater. Actually, a lot of the time, the weather here is downright pleasant. The only problem is when it gets dark and cools off, and that's only a problem because it gets dark by 5 now. :(

That's one thing that totally caught me off guard about moving north. I was here over the summer, and didn't catch on that the long daylight hours would reverse themselves later in the year. It's darker at 5 these days than it was by 9'o'clock most days during the summer, and I expect it's going to keep getting worse until the solstice. It's actually pretty surreal - when I see that it's completely dark outside, I know that I still need to stick around at work for another hour. :/

People are saying that this winter is going to be hellish; in the figurative sense, of course, because they actually mean a lot of snow and ice. I don't know how much snow is considered normal here, but "a lot more" will have to be a decent amount. It'll be fun for the first day or two, at least! I'm pretty glad that I can take public transit to work, because if I tried to drive to work in the snow, I would probably die.

shortened URLs = sadface

2010-11-10T21:39:00.000-08:00

So on Twitter, I can understand URL shorteners. People pretend to like the 140-character limit, but are somehow totally willing to circumvent it when they really need to. I get that. URL shorteners are completely valid (if still annoying) in that context.

What I cannot for the life of me understand is why somebody would use shortened URLs in a blog. I was reading a post the other day (seem to have lost the link) which used shortened URLs exclusively. If you control the markup (as you do on a blog post) then there is literally no possible reason to use short URLs, ever, unless you are trying to annoy your readers for some reason. Remember how, five years ago, everybody knew to make your URLs as expressive as possible for SEO purposes? And how it ended up being really convenient, because a user could hover over a link, see the URL, and have a sense of what you were linking to? All that's out the window now, thanks to fucking Twitter! :D

I'm not even going to get into the fact that shortened URLs are a horrible idea technically, and occasionally place you at the mercy of the Libyan government. Everybody knows that they're terrible. Twitter is the only reason that we tolerate their proliferation.

(side rant here about how 140 characters is far too short. I am feeling lazy tonight so please imagine an appropriate rant. >_>)

Actually, short URLs are one thing that Identica gets completely right. When it detects a shortened URL, it sets the

title

attribute on the link to the real URL, so that you can see where the link goes by hovering over it. It is impossible to appreciate how useful that is, until you use another mblogging site like Twitter and become sad because you actually have to click the stupid links to see where they go, and then write a disjointed blog post on the topic.

Tech Support Counter-scripts

2010-11-09T00:00:00.000-08:00

So when you call in to tech support, they are reading off a script. The script walks you through the most common resolutions to people's problems ("Have you tried turning it off and then on again?"), and is entirely useless if you're at all technically competent. If you're like me, your support calls usually end up being exercises in absurdity - can I go along with the script long enough for them to become convinced that they should put me through to someone competent?

Randall Munroe had the right idea. He just didn't take it far enough. It would be really great if everybody implemented backdoors for techies, but really. If we want backdoors, we're going to have to make them ourselves.

What we should do is make a repository of tech support counter-scripts. Let's say (as in the linked xkcd) that you know exactly what the problem with your internet connection is, and you want to minimize the time it takes to convince the person on the other end of the line to just fix your problem. We could crowdsource it, and have everybody who tries the script try tweaking it on actual calls, and iterate toward the fastest way to resolve a given situation.

So here's the really cool bit. We also include some distinctive phrases in the scripts (backdoors, if you will), so that after a while the support people will catch on to what we're doing. At that point, they have a choice: they can either look up the backdoor phrase in the counter-script archive, and find out what specific problem we're trying to get them to solve; or, they could adjust their scripts to counter the counter-script.

Few people are stupid enough to choose a fight against a crowd (just ask /b/), so eventually they will add the backdoor phrases to their scripts, with the instructions that we wanted them to have in the first place. In effect, we'd be social engineering the backdoors into the support system ourselves. Pretty cool, huh? :D

The Terrorists are Running the Asylum

2010-11-08T21:05:00.000-08:00

Actually, I sort of feel sorry for the TSA. Their job is to screen millions of airline passengers, with a 100% success rate. Since that's obviously a fool's errand, their real job is to keep us mollified that they're doing enough every time there's a terrorist attack, and meanwhile try to convince us that there is a such thing as going too far in the name of security.

They started out with only the first one, I think. After Richard Reid, they had everyone remove their shoes before getting on flights. It was a perfectly logical course of action for them - a reaction to exactly the attack that had occurred, so we could be confident that that would never happen again, and just enough inconvenience at security checkpoints that we felt safer. Their reasonableness, in fact, was their only mistake, even though they didn't realize that for several more years.

The analogy of a frog in a boiling pot of water is ridiculously overused, but also fits this situation. By taking a series of reasonable steps to individual threats, the TSA can slowly turn up the temperature of airport security, and we'll all go along with it and accept it as necessary. In fact, that's not in their best interests - in the long term, they need us to realize that there are things we're just not willing to do for more security, or they're going to be in an awkward position in the long run. Looking at it from this point of view, the best course of action for the TSA would have been a dramatic overreaction, which would convince us that they were going too far, so that we could all take a breather and then tone things back down.

Banning liquids on flights was a good move, crazy and arbitrary enough (and in response to a narrow enough threat) that we should have all woken up to the ridiculousness of it all. I mean, you can still take several smaller bottles onto a flight; it's a huge inconvenience to you and me, but won't stop a motivated attacker. Who could possibly accept that? But nope, it wasn't far enough, and now we're all used to it. The bar has been raised, and they're going to have to really go over the top if they want a backlash that will let them finally restore sanity to airports.

So fast forward to a year ago. In response to the Underwear Bomber (a suicide bomber who failed to even kill himself. Terrifying!), they've begun installing new X-ray machines which can literally see through our clothes. They claim there are safeguards in place to keep airport employees from looking at our naughty bits, but it would seem that the safeguards aren't too hard to disable, and there's already been one case of security personnel saving nude pictures of people for their own use.

This is the TSA's Hail Mary pass. The government is literally paying people to look at pictures of naked children. If they have any sense, their next move is going to be to wait a few more months, and then uncover a pedophile ring that's been saving pictures from the new X-ray machines. With any luck, that will be a big enough jolt for people to realize that security is about tradeoffs, not absolutes, and that we'll never be 100% effective in preventing terrorist attacks. If we can get past that, we might even be able to have a meaningful national conversation about a reasonable level of security.

In the meantime, though, I should reiterate: The government is paying people to look at naked children. Courtesy of terrorists, even. I don't know when we're all going to get hit with a jolt of sanity, but it can't happen soon enough.

Sherlock!

2010-11-07T21:04:00.001-08:00

"I am the closest thing to a friend that Sherlock Holmes is capable of having."

"And what's that?"

"An enemy."

So I found out the other day that there's a new Sherlock Holmes miniseries being made - a modernized one, no less. Normally that would be the sort of thing that I would avoid like the plague, but I found out about this through a blog post that described it pretty favorably, so what the hell, right? Anyway, long story short, I'm halfway through the first episode and getting the next few and at this rate (and with 1.5 hour episodes) I probably won't have time to think about anything else to blog tonight. XD

(Mild spoiler alert for the rest of the episode. No major plot details, but some things that you wouldn't have expected.)

You know what surprised me about this show? Not the writing; it was entertaining in that dry fast-paced British way, but I was expecting that. Not the way they modernized (though I was pleasantly surprised by how well they did some things - instead of bolting technology onto a classic plot, they basically wrote a new story with things like cell phones playing a central role). Not the story: it was about on par with one of the better episodes of something like CSI, which actually makes it pretty decent, but then again you expect that these days.

What surprised me was the characters. They are far darker than those in any other Sherlock Holmes adaptation I've seen. Watson doesn't just have a vague background as a military surgeon - he has flashbacks, sees a therapist, and discovers halfway through the episode that he misses the danger and excitement of being in a war zone. Holmes, meanwhile, is practically amoral - in his own words, a "high-functioning sociopath" - and cares about nothing except for solving crimes to avoid boredom. He's even occasionally a jerk about being the smartest guy in the room (admittedly, the original Holmes did this from time to time). The character is reflected in the other characters, too. The police don't like him (which was also present in the original stories, I suppose, but not to this extent), with one of them referring to Holmes as "the freak" and even speculating that he might end up on the other side one day.

Inevitable comparison: How does this compare to the Sherlock Holmes movie that came out last year? I enjoyed the film, but more as a Robert Downey Jr. movie than as a Sherlock Holmes movie. RDJ drew comparisons to Hugh Laurie (who stars in another Sherlock Holmes spinoff), and it was a fun movie to see, but that's just it: it was more "fun" than "interesting".

The next episode is about to finish downloading, so I'll leave off on this note: I think this adaptation has the potential to be Interesting.

My Desktop

2010-11-06T12:55:00.000-07:00

So a few days ago I was trying to think of a topic to blog about, and checked the NaBloPoMo homepage, since they post a prompt every day. That day, it was to write about a piece of jewelry that I own, which was a little bit less than helpful. >_> But then, Kiriska suggested that I should write about a piece of hardware that I own. I don't think she was serious, but you know what? That'll totally work. XD

I've owned my desktop computer for about eight years now. None of the parts are original, but the spirit of it lives on!

Origins

So back in the eighth grade (around 2001 or 2002), I had heard about this newfangled "Linux" thing, and I wanted to try it out. With little or no thought as to the consequences, I downloaded an ISO (Mandrake Linux, if you're wondering) and installed it on our family computer. (This actually took several months - we take broadband for granted now, but have you ever tried to download a Linux ISO over dial-up?) Immediately, problems: I set up the dual boot properly, but nobody else in my family was able to figure out the GRUB boot menu, and as a result nobody was able to use the computer unless I was around to boot it up. (I didn't think it was too hard - you go down to the item labeled "Windows", and hit enter - but whatever.)

I'm not sure, but I think this episode contributed a lot to my parents agreeing to let me build my own computer a few months later. I went online, picked out parts from a ton of different websites (either Newegg didn't exist at the time, or I didn't know about it yet), and ended up spending about $600 on my first computer, if I recall correctly. Back then, that got you an Athlon XP 1500 processor (actual speed 1333 MHz - this was the beginning of AMD's labeling shenanigans), 256 MB of RAM, a 40 GB hard drive, and various other bits.

Ever since then, it's been my primary machine, and also a testbed for every crazy idea that I've ever wanted to try out. (When you're me, you have a lot of crazy ideas, apparently.) To be honest, I think that most of my current skill with Linux comes from trying out something a little bit too crazy, and then fixing everything that breaks as a result.

Operating systems

So I mentioned that I started off on Mandrake. After a while, I switched to Red Hat; Mandrake was never especially great, and Red Hat seemed pretty solid. (An aside: Red Hat 8 or 9 was when BitTorrent actually hit the mainstream - every official download mirror was crawling at dial-up speeds, but the BitTorrent download was working just fine. This was when a lot of people sat up and noticed it.) Red Hat (and later Fedora, a spinoff of Red Hat) was nice for a while, but around the middle of high school I decided to take it to the next level. I switched to Gentoo.

For those of you unfamiliar with Gentoo: it is hardcore. Graphical installers? Feh! Installing Gentoo involves downloading and extracting an extremely minimal base system (about 100 MB), and then building the rest of the OS as you see fit from the command line. Prebuilt packages? Decadent! On Gentoo, you build every single piece of software you install from source code. (Sometimes it's overkill, but it's become the defining characteristic of Gentoo.)

Gentoo has this nice property called "rolling releases" - basically, they don't release major versions every once in a while like other OSes. All updates to the system are distributed incrementally through the existing update mechanism. This is useful - it means you never have to reinstall your OS, unless you really want to (or, in my case, really screw it up). End result: After using Gentoo for something like six or seven years, I've only actually had to completely reinstall it once.

Current Incarnation

Today, my desktop has a dual-core 2.5GHz Athlon 64, 2 GB of RAM, several hard drives (the exact count depends on your definition of hard drive), and I think it's about due for another major upgrade. It's on its third CPU, its fourth motherboard, its third case, and in fact I don't think there are many parts that have only been replaced once. Yet somehow, it's still the same computer it was when I first built it all those years ago.

Pirate TV

2010-11-05T00:32:00.000-07:00

BitTorrent is a pretty amazing way to get "completely legitimate" TV recordings. Using it, you can distribute bits around a network so efficiently that a several-hundred-megabyte file can be distributed to tens of thousands of people within an hour or two. It's not perfect, but it's so close to optimal that nobody's been able to improve upon it significantly in the past decade. That's pretty amazing, given the speed at which technology advances - ten years is an eternity.

So. What would it take to significantly improve upon BitTorrent in this space?

I have this crazy idea for a protocol that should allow for secure p2p-style live streaming video. Basically, you'd have a single video source, streaming through a tree of nodes that act as stream multipliers to get it widely distributed. Repeaters within the network would have the option of restricting client access based on whatever they wanted - invite-only access, or an add-supported stream, or a micropayment scheme (let's pretend I didn't just open up a can of worms), or something else. All connections would be SSL-encrypted (using both server and client certs would be used, since clients need to be authenticated as well), and some sort of forward error correction encoding would be useful too, I guess.

Inevitably, The Man is going to try to shut this down, so we need to think about defenses. One of the weaknesses of BitTorrent in this area is the tracker, which works by giving out users' IP addresses to entire swarm to facilitate downloads. With connections managed by the user, the data on connections remains a lot more private, so users can't be easily tracked.

The video source needs extra protection, because it represents a single point of vulnerability for a network. The easiest way to track it down would be traffic analysis. This can be mitigated by having a network of a few dozen fast machines (the video source among them) all start sending data to each other at once; most of the data being passed around will be garbage data, but an eavesdropper can't know that because all connections are SSL-enabled. Thus, a dedicated attacker could narrow the source down to one of several nodes, but its location within that group of nodes can be effectively hidden.

I'm not arrogant enough to think this protocol is any good, to be honest. (The incentives are incredibly sloppy, for one thing - it's an area that I hadn't started thinking about until yesterday.) The point I really want to make here is that better p2p protocols that BitTorrent exist; they just aren't being deployed because BitTorrent is widely-used and good enough. It's something that anybody who works against piracy needs to keep in mind: the successor to BitTorrent is going to be a lot harder to shut down, and it's going to appear out of the woodwork as soon as BitTorrent is no longer an attractive option for pirates.

An Increasingly Frantic Search for Novelty (or: Philosophical Segfaults)

2010-11-04T00:14:00.007-07:00

...or at least that's how life feels some days.

I am, in general, a purpose-driven individual; as long as I have something to be working on I'm pretty contented. The flip side is that, when I actually don't have anything to be doing, I handle the listlessness pretty poorly.

About a year and a half ago, I went through a pretty dark spot when I started wondering about my overall purpose in life; what was I actually aiming for? Up until that point, I had sort of powered through school on a weird mixture of arrogance and obligation. That had started to crack as far back as high school, though, and so I decided: I would figure out my purpose in life!

As it turns out, this is a terrible idea. Because, after not finding an answer for a while, I started to wonder about my life so far. Maybe everything I was doing was actually meaningless, maybe everything I had done so far in life was meaningless, maybe there wasn't a point to doing anything anymore, because I couldn't find a reason for any of it... It was the philosophical equivalent of dereferencing a null pointer.

Eventually, I grabbed at a lifeline. Forget about the long term, I decided, because I could just do whatever seemed interesting in the short term, and keep going like that until I found a better answer. It was a total cop-out, but it also got me out of the pit I had dug for myself, and keeps me out when I wander too close to it.

This story has no resolution; I'm still basically operating in magpie-mode, chasing after whatever looks shiny when I find myself with nothing else to do. The alternative is to seriously think about the possibility that there is no purpose to what I'm doing, and that I'm just an empty puppet acting out a script that was written over the past billion years of evolution. (I'd rather not.) It's not exactly fulfilling, but then again, fulfillment isn't a thing I expect anymore.

Honestly, this blog post is about as close as I feel like coming to the topic, and even writing this much sent me on about an hour's worth of meandering melancholy mental tangents. I can't say that I've given up on the problem, that I'm satisfied with not having an answer, without setting off the intellectually honest part of my brain... but if I end up living my whole life without actually finding a reason for having lived it, well, I think I've made my peace with that.

oh god did I just write a blog post about the meaning of life, dammit I think I did

An obsession with unpopularity that is, frankly, a bit creepy

2010-11-03T00:00:00.000-07:00

A few months ago, I picked up a copy of 2600. (Yes, an actual physical copy. Ironic, right?) I got it at Book People, if I recall correctly, and if you're ever in the Austin area looking for a cool bookstore, Book People is the place you want to be, I think. I could digress for paragraphs about why it is a cool place, but I think I will get to my point instead.

Anyway, it was a pretty neat issue, especially the article about hacking Bluetooth. There was another article, though, wherein the author just talked about how he was picked on in high school, and how now that he was a hacker, he was going to get back at all the people that had wronged him. And you know what? That article creeped me the hell out. Partly because the guy sounded like a total psychopath, but mostly because it crystallized my general unease with how easily geek culture has absorbed the Disneyfied high school narrative of geeks getting picked on. Hold up, that deserves some explanation.

An Aside on my Longstanding Objections to the Generic Disney Channel High School

Have you ever noticed how far divorced the Disney channel is from reality, when it comes to high school? I don't just mean the liberties that they've had to take to fit various shows into a TV format; I mean the fact that what they portray as high school is completely and tragically divorced from reality. I'm mostly referring to the concept of well-defined and insular cliques, and the social structure that implies - the rest of how they portray high school is equally messed-up, but tangential to my actual point. They've created this parallel universe that's sort of like high school, and they apply it so consistently to all their shows that I actually wonder sometimes if there's another country where school actually is like that, and all their writers just happen to be from that country. Then I realize that it's much more likely that all their writers grew up watching movies like Animal House and Revenge of the Nerds, and thought they were hilarious, and are unconsciously trying to emulate a parody. BUT WHATEVER.

So back on topic:

What is it with geeks and this particular kind of martyrdom?

Another example: various slashdot comments which have creeped me out in the same way. (Entirely too lazy to find links.) You will, with alarming frequency, find comments on "geeky" news sites where people assert that geeks are driven toward technology because they're picked on and unpopular with their peers. There are also plenty of creepy misogynistic comments about women only wanting "bad boys", at the expense of geeks, who are the "nice guys"; another narrative that falls more in line with the Disneyfied high school than any real one. I don't know where all this is coming from, but it worries me.

Another example: The second episode of the third season of Leverage, a show that I otherwise admire entirely too much. The plot: the team has to con a software executive, and the only way to get inside his head is through high school, because he's totally obsessed with high school! Because he was picked on in high school, and now fantasizes about showing off to everybody how successful he is now! Because he's a Geek!

(The worst part is, it was otherwise a good episode, but I couldn't enjoy it because of my all-consuming rage at the blatant stereotyping going on >_>)

So, stereotypes I can understand. They happen all the time, and it's not exactly mysterious why they happen. What I can't understand, though, is why geek culture has adopted this particular stereotype.

Attention, geeks everywhere! The "high school martyr" trope is a bad thing, and kind of creepy when you pair it with "but I'll show them all!" later on! It might even be worse than the "geeks = autism" trope, so let's stop perpetuating it, k?

Building a storage cluster (on the cheap)

2010-11-02T00:00:00.010-07:00

So RAID is one thing, but (as I found out the hard way, recently) you're still screwed if the computer you put the RAID in starts going flaky. Now that I have a bit of cash, I decided it was time to take things to the next level. :O

My initial plan was to build a super-awesome NAS - basically an external hard drive that you plug into the network, so you can use it from multiple computers at once. After a lot of thought, I abandoned this plan. I don't trust commercial NAS systems (though the Drobo looks pretty sweet, I must admit), and building my own would have been too expensive. It'd also be kind of a pain to add storage - take it offline, install a new hard drive, mess around with cables (possibly forgetting to plug some in, which actually happens more often than you'd think >_>), and when you fill up the box, you either start throwing drives out (wasteful!) or upgrade the entire thing. Plus, a NAS is still a single point of failure, which is lame! I can totally do better than that.

So the new plan is to build a Ceph storage cluster. Ceph is a relatively new distributed filesystem that one of the Dreamhost guys did for his PhD thesis. It's Linux-based (and even included in recent kernels), actively maintained, well-designed (aside: I'm still pretty grateful to the distributed systems class I took at UT; without it, I probably wouldn't even know what well-designed meant in this context), and generally meets most of my standards for a distributed filesystem. At the time, it seemed like a pretty good option, and it still does! Now, if only I could find some ridiculously cheap computers to use as servers...

It was at about this point in my thought process that I discovered that Jetway makes some ludicrously cheap minimal servers. You know that plug computer that Marvell came out with the other year? Jetway's thing has much higher specs, runs x86, has a full complement of ports and room for a hard drive or two, and is well-reviewed on Newegg. The only thing Marvell's plug computers really have over it is that they're ridiculously tiny, and who really needs that?

So, long story short: I pick up two of those plus some RAM, add my desktop machine, and I have a three-way storage cluster that's infinitely expandable, open source, has no single points of failure (except for the network parts, maybe? but those things last forever), and ended up costing less than even the cheapest Drobo.

Belatedly, justification

Somebody is inevitably going to comment that I'm trying way too hard, and that backups to an external are cheaper. The trouble with doing your own backups is that you actually have to pay attention to them. My goal here is to get a system going where I don't actually have to think about it day-to-day - everything should just work.

Somebody else will probably say that I should just store everything in the cloud - then I wouldn't have to worry about anything, because they'll certainly take better care of my data than I could. The trouble with the cloud is that it's too expensive (about an order of magnitude more than I'm paying to buy my own storage cluster, which is a lot more than I'm willing to pay). Plus, there aren't any cloud storage providers that I actually trust not to snoop around in my emails, or whatever.

Dogs and Cats Living Together

2010-11-01T00:24:00.000-07:00

It's November again! Time for another round of NaBloPoMo. So I recently found out that NaBloPoMo is no longer just a November thing, but you know what? I sort of don't really care. It is a Thing I Do In November, and I don't see any reason to change. :D Plus, there's also the fact that it coincides with NaNoWriMo. By blogging, I sort of feel like a casual jogger tagging along with people who are running a marathon - I know that there are people (hi, Kiriska! :3) that are doing a lot more work than I am this month, but I'm going at a pace that I find comfortable.

In past years, I've written a post or two ahead, so that I'd have a buffer in case something unexpected came up and I didn't have time to blog. I don't think I'm doing that this year. First, it sort of feels like cheating, and second, now that I'm actually working I have a much more predictable schedule.

Anyway, to kick off this month, I'll write about an idea that's been knocking around in my head for a while now.

Everybody loves puppies and/or kittens! This is an Inarguable and Incontrovertible Fact. However, a lot of people aren't in a position to own a pet (particularly those living in college dorms), and others don't really have the time to take care of one properly. What I'm proposing here is really, when you get right down to it, just a way to let college students play with cute animals; therefore, it cannot possibly fail, and might even end up being profitable.

I'm envisioning a big, lounge-type room, with comfortable chairs everywhere, and lots of dogs and cats. (Thus, the name: "Dogs and Cats Living Together".) The animals would all be adopted from local shelters, naturally. There would be chewy things for the dogs to play with, and random structures for cats to climb around on, and all the furniture would be the cheap stuff because there's no way it'll last very long. People would be able to come in and play with the animals whenever they wanted, without any of the worries that come along with taking care of them, or cleaning up after them.

(So who would take care of all the animals? That's the beauty of it - I can't imagine that you'd have any trouble finding applicants for the job on a college campus, when the job description is basically "taking care of dogs and cats".)

As far as actually making money, there are a lot of ways. You could charge a few dollars at the door. You could sell snacks (for both humans and pets!) once people are inside. You could let people adopt the animals, if they really wanted to. There are merchandising opportunities - cat calendars are a dime a dozen, but a cat calendar where you can actually go and play with the cats in the pictures might be enough to set it apart, I dunno. And, as a last resort, you could train the animals to rob banks - wait, pretend I didn't say that one. >_>

Anyway, yeah! I think that'd be pretty neat.

2010-10-15T09:07:00.000-07:00

The hardest thing is to go to sleep at night, when there are so many urgent things needing to be done. A huge gap exists between what we know is possible with today's machines and what we have so far been able to finish.

--Donald Knuth

That said, having programming as a hobby is sort of troublesome when you also program 9-to-5. So I think I need a new hobby, preferably one that doesn't involve computers in any way, and ideally one that involves me getting some exercise. >_>

Anybody got suggestions?

Adventures in the Linux storage stack

2010-08-28T23:36:00.001-07:00

I recently got my Gentoo box back from storage after three months, and promptly upgraded all the software on it and tried to install a shiny new SSD. Here are the problems I've run into so far:

Media drive failed to mount

My media drive is a btrfs RAID-0 between two 500GB drives that I've picked up over the years. One of them wasn't being detected. After ruling out the usual suspects (drive was still working, as confirmed by smartctl, and btrfs' drive detection tool (btrfsctl -a) had been run [but oh god why does it even need this]), I started looking at more exotic causes.

Diagnosis: The problem turned out to be a regression in the md userspace tools (must be, since I didn't upgrade the kernel): if you have a device which used to belong to an md RAID array, but doesn't anymore, md will still pick it up as a RAID device and lock it. The recommended fix is to zero out the first part of the drive, which isn't an option for me because the drive ACTUALLY HAS DATA ON IT. :[

pvmove stalls at 100%

When using pvmove to transfer my OS drive to my shiny new SSD, pvmove stalled at 100% completed. No data was being written, which I know thanks to gkrellm. Then my entire system locked up, or at least every process that tried to write data locked up. After rebooting, I tried to resume the transfer (which pvmove supports), and the system locked up again the same way.

Diagnosis: None. Chalk this one up to LVM being completely fucked. It is a master of walking the thin line between being so useful that it can't be ignored, and so broken that it can't be used. (UPDATE: might have been caused by creating the pv on a whole disk and not a partition. Not sure why that'd be the problem, but it's the only thing I changed and now it works.)

Boot environment doesn't see shiny new SSD

So I have a completely customized boot process. It's part of a 70% successful experiment from about 6 months back in making an operating system which could survive a hard drive failure. (Harder than you'd think!) Part of that is a heavily customized initrd - basically a micro-OS which is embedded into the kernel image itself, so that I can do funky stuff with the boot process itself. This is all well and good, until I find out that while my OS can see my new hard drive fine, my initrd can't. :(

Diagnosis: This one was me getting sloppy. When I created the initrd, I deleted a few thousand miscellaneous device nodes from the static /dev directory (because udev is too complex, let's just embed dev nodes for every piece of hardware that could possibly be connected -_-), and it turns out this included the one used by the drive I just added. I only discovered this after a half hour of "oh shit oh shit did I just transfer my entire OS to a DOA drive", of course.

How [should?] we teach history

2010-08-16T00:38:00.000-07:00

This blog post on Abd el-Kader came to me secondhand via my RSS reader today. It's pretty fascinating in and of itself, but it's also really long, so don't start on it unless you've got a good chunk of free time ahead of you.

The thing that depressed me about the post is that I have basically zero context for any of it. I never really took an interest in history, not enough to go beyond what they taught us in high school, so for me north Africa and the Middle East are basically blank areas of the map for large swaths of history. In Rumsfeld's terminology, it was an "unknown unknown" until about an hour ago - something I didn't even realize I didn't know. For somebody accustomed to being a know-it-all, this comes as kind of a shock.

The trouble with "all the stuff we didn't learn about in history class" is that there wouldn't be enough time to cover it all if we spent our entire lives studying history. We only spend a limited amount of time in school, and that's a good thing! It just means that we need to find ways to present more varied information in the time we do have. Standardized curricula have their advantages, but the biggest downside in my opinion is that anything which doesn't make the cut remains completely unknown to several consecutive crops of students.

Do I have a solution? Nope, not a chance. I just think it'd be neat if we'd had more structured opportunities for everybody to go out and learn about something different. We can't (and shouldn't have to) increase the amount of time each child spends learning about history, so if we don't want to end up in a situation where everybody knows about the same subset of history, we need to find some way to diversify the knowledge base that we're building in history classes.

After all, if everybody learns the same parts of history, then nobody has anything interesting to say to anyone else about history, right?

Piercing the Veil

2010-08-09T00:51:00.000-07:00

So I've got a new hobby: when I look around, I try to see the story behind everything I see - everything it took to make it, all the different processes that were necessary for it to come into being, things like that. It's simultaneously exhilarating and humbling and occasionally enlightening [in the negative space sense of suddenly realizing something you don't know]. It can even be transformative, in that you may never look at the world the same way again.

An example: I am riding my bike home from work. The bike itself is a marvel of applied physics - the impossibly thin spokes on the wheels that actually support your weight by hanging the bike from the tops of the wheels; the clever use of gyroscopic forces to keep the bike stable while it's in motion; the adjustable gear system which you can reconfigure as you ride. You could teach an entire physics course just by looking at a bicycle in motion. A bus rolls past, inviting you to consider not only the mechanics of how it runs (the entire history of internal combustion could go here) or why it runs (the efficiencies of mass transit), but also the entire superstructure of government which makes public transportation possible. The sidewalks and roads I'm riding on are made of cement - a building material which is dirt-cheap, uniform, pervasive, and commoditized. Really, it's a perfect analogy for our mass-produced society. As for the roads themselves, they constitute a single, unbroken web which stretches across the entire continent; think about that for a minute. (There's an interested design principle embedded here: if you want a pervasive system of roads, you don't start by building roads. You start by building cars, and the roads will come.) There are buildings by the sides of the road, and every individual component and overall feature has a rich and varied history - glass, steel, central heating/cooling, architecture, fluorescent lights (and the electrical grid they imply), indoor plumbing, and don't forget the construction equipment used to put it all together. If you wanted to build a modern office building from scratch, it would take centuries of technological development.

There is a Veil of abstraction which we apply to everyday life, because our minds are simply not equipped to think like this all the time. We need to take things for granted if we want to be able to function. It can be fun to pierce the Veil once in a while, and take a sip from the geyser of extraneous information that fills our world, but it's not a sustainable state of mind. I won't bother with value judgments here. The Veil is neither good, nor evil, nor anything in between, it simply is.

And if we're bored enough and perhaps stubborn enough, we could even try to peel back the Veil from the Veil itself, from the very fact that we only see the world for what it is, and not everything underneath. There are Darwinian forces in there, enhanced but certainly not altered by capitalism, which drive us to produce and to be productive, and weed us out ruthlessly if we fall into the trap of looking too deeply at the world, and spend our lives wandering around in slack-jawed wonder. There's that fact that our mental processes are fundamentally and severely limited and finite, which lies somewhere at the awkward intersection of information theory and neuroscience.

So the Veil is a necessary part of our lives, and we couldn't pull it back if we wanted to, but I think it's fun to peek behind it every once in a while.

Tahoe project ideas

2010-07-27T21:08:00.000-07:00

Tahoe-LAFS is a pretty cool project (which I have blogged about before), but it has some weaknesses. Coincidentally, I've been looking around for a project, so I think I'll try helping out with it.

The current state of Tahoe

Don't get me wrong - as far as cloud storage goes, Tahoe might be the only one I've seen that has some potential. (Maybe I'm setting the bar too high, but maybe everybody else is setting it too low!) If I had to sum up their architecture, though, it looks something like this:

1) A filesystem access layer, which isn't terribly interesting as long as it works
2) A completely badass data management layer, which handles all the tricky bits surrounding security, privacy, and reliability
3) A half-baked distributed storage layer

It's that last layer that I'm interested in, for a lot of reasons. First, there's been a lot of good work in that area over the past 20 years (particularly, a lot of really neat stuff with distributed hash tables 8-10 years ago). Second, it's an area that I've had an interest in for a while, and I think it'd be fun to work on. Third, I think that a more scalable (at a minimum, handling millions of nodes) backend is what's needed to take Tahoe from being a neat and useful program, to being a capital-D Disruptive one. A secure, reliable, fully distributed filesystem running at Internet scale? Oh, hell yes. :D

So with that in mind, here's what I'm thinking as far as possible avenues of work.

Project ideas

Tahoe bug #999

DHT-based storage

DHT-based storage shim

Storage availability zones

Amazon S3-compatible interface

and

Unbreakable static web hosting

those

Favorite thing about working at Microsoft

2010-07-17T20:45:00.000-07:00

Well, I can't really pick just one thing, so I'm just going to list a lot of awesome things.

Free beer at company events

could

Working with extremely geeky people

Completely awesome company events

Challenging work environment

Real-world impact

Fully furnished apartment

Novel work environment

Intern perks

Flexible work hours

The weather

oh no

2010-07-10T00:44:00.001-07:00

I have forgotten how to blog D:

Index page for Amazon S3

2010-06-25T00:37:00.000-07:00

So I've been looking for a way to get index pages on Amazon S3. Turns out Amazon doesn't provide them, and nobody else really does either. Well, we can fix that!

I just put together http://s3index.p-static.net, which generates index pages for any S3 bucket. If your bucket is publicly listable (go to bucket permissions, and enable "List" permission for "Everybody"), then it should work. The URL format is http://s3index.p-static.net/BUCKETNAME.

Bonus: it's stylable! The script automatically includes a CSS file named "s3index.css" in the bucket, and all the elements have classes declared.

Source code coming later, it's still extremely messy right now. >_>

Everything you need to know about Tahoe-LAFS

2010-06-06T12:56:00.000-07:00

There are not a lot of pieces of software that I would describe as revolutionary, but Tahoe-LAFS is one of them. Unfortunately, as is all too common with open source software, the potential of the software far outstrips the documentation, with the predictable result that nobody understands what Tahoe is capable of. This post is an attempt to explain the important bits of Tahoe as succinctly as possible.

High-level goal

Tahoe is an almost-completely-decentralized, secure file store. You can think of it as a reliable black box which you can upload files and directories to. The data is highly resilient (up to 70% of the distributed data can be lost, and the file will still be recoverable, and that's just with the default settings) and secure (RSA is built right into the URLs you use to access data, so nobody who doesn't have access to the data can read it). By giving somebody a secret URL, you can give them read-only or read-write access to any file or directory. Tahoe guarantees that, if somebody does not have this secret URL, they cannot access the file data in any way.

The advantage of Tahoe over something like Amazon S3 is that you don't have to trust your cloud storage provider. S3 provides very similar capabilities, but it's a given that Amazon employees will be able to read your data, which makes it unsuitable for some applications, and which makes people that care about their privacy nervous.

When you upload a file to Tahoe, it's passed through an erasure code, and then distributed across multiple computers participating in a cluster, none of which can read the actual data. When you access data in Tahoe, a form of authentication is built into the URL you use, so you can keep data private simply by keeping the URL used to access it a secret.

Erasure coding

Wikipedia explains the theory far more faithfully than I would be able to, so I'll just try to give a more concrete example of how it works. Let's say you've got a 300 kB file. A 3-of-10 erasure code (what Tahoe defaults to) would take that file, and give you ten 100 kB chunks of data, or 1 MB total. You'd then spread this data around as widely as possible, and the erasure code guarantees that given any three chunks out of the ten, you can reconstruct the original data. This means that if you gave data chunks to ten different computers, and seven of them crashed, you'd still be able to access your data just fine.

Erasure coding by itself isn't secure, so Tahoe encrypts the data before the erasure coding step. This makes it possible to give a certain limited capability (a "verify cap", explained later) to an untrusted machine with a lot of bandwidth, which can then download some of the encrypted data, rerun the erasure code to fill in any missing data, and re-upload the complete coded file - all without being able to read the actual file data.

Introducers

The introducer is the one part of a Tahoe cluster that isn't decentralized. A network is built around an introducer, which does what it sounds like; it introduces Tahoe nodes in a cluster to one another. If you're familiar with BitTorrent, the introducer is very similar to a BitTorrent tracker.

There's been talk of replacing introducers with a DHT network (similar to what BitTorrent does these days) but as far as I know, there's been no action on this.

Capabilities

A "capability" for a piece of data is just a very long URL that you can use to access that data in a certain way. It's called a capability because it determines what you can do with the data - a "write capability", for instance, contains enough information for you to overwrite the file with a new version, while a "read capability" will only let you read the file, but not modify it. There's also something called a "verify capability" (mentioned earlier), which gives you access to the encrypted contents of the file, and can really only be used for verifying the data. Encryption keys for the data are part of the capability itself (which is part of the reason that they're very long), so there's no need to store those separately.

Data security in Tahoe is accomplished by keeping these capabilities - "caps" - secret. If you're the only one who has the write cap to a file, Tahoe guarantees that nobody else can modify the file, and if you're the only one who has a read cap, Tahoe guarantees that nobody else can view the file. Capabilities are the only method of access control. This isn't how we're used to thinking about data security - where are the passwords? - but once you wrap your head around it, it makes sense.

Capabilities can be demoted - given a write cap, you can generate the read cap automatically, and given a read cap, you can generate the verify cap. Thus, having a certain level capability for a file also means you can grant other people the same or lower level capability. I mentioned earlier that untrusted servers can refresh your data without reading it - this is what verify caps are used for. You can make the verify cap for your data public, and anybody will be able to make sure it stays in the cloud, but nobody will be able to read it.

Directories are treated the same way as files, so you can also have read/write/verify caps for directories. I'm not entirely sure how this part works, but the read-only-ness of a directory read capability carries through - if you give somebody a read cap for a directory, all the files they can see in the directory will be read-only for them as well.

Final thoughts

So far, I've been unable to find any information about how Tahoe would scale in very large configurations (thousands to millions of storage nodes), but it doesn't seem like it would. The documentation that I've found specifies that at startup, a node connects to every running storage node that the introducer gives it, which seems like a huge scalability bottleneck. I kind of want to ask the developers if this is really as bad as it sounds, but I haven't had a chance yet.

Assuming it can (or will eventually be able to) scale, I really hope Tahoe becomes more widely used. If it was, it would be a whole new way to think about data management - like a crowdsourced, secure version of Amazon S3. As far as I know, though, there aren't any large Tahoe clusters running - their public test cluster only has two or three nodes in it, as far as I can see. (If you want to start one with me, leave a comment - I can provide an introducer and some storage to start with, but there's no point if it'll be just me. XD)

portable functions braindump

2010-05-31T01:24:00.000-07:00

I have just had an incredibly cool idea, and I'm afraid that it'll leave me once the drinks wear off, so I'm going to go ahead and write it down now.

First-class functions are pretty cool, so let's take them to the next level. Imagine for a moment that we could represent a function as a file in a reasonable clean way. (This doesn't take a lot of imagination, since it turns out we can, and the only disagreement is over how to do it best, but bear with me for a second.) What could we do with this? As a start, we could expose functions over the network, and call them dynamically from programs. Instead of having to have a copy of all relevant code on all computers that want to use it, we could store functions online, and dynamically pull them into a cache when we actually want to call them, and abstract away software updates entirely. But wait, there's more.

Cloud computing services are drifting in fits and starts toward a clean model where you submit jobs to the cloud and they happen. Right now, Amazon EC2 is considered state of the art, which is just ridiculous - they make you virtualize a whole OS instance! What the hell is that?! It maximizes flexibility at the cost of incredible inefficiencies. The major bottleneck to a better cloud compute service, though, is a common format for executables. If you only want to submit a single program to a compute service, you and the cloud compute service provider need to agree on a format for the program, and that's basically impossible today. There's simply no consensus at all, not even the beginning of one. (Services like PiCloud are a nice first step, but they do it by standardizing on one language and ignoring the rest of the universe. Useful, but not especially interesting.)

So here's the utopia I'm imagining:

* You put a chunk of code online
* You give your choice of cloud compute service the URL for the code
* They fetch it and execute it
* Everybody is happy and nobody has to fiddle around with what availability zone their EC2 instances are in, and whether or not it matches their data

We'd use HTTP, naturally; HTTP is the new TCP, and everybody uses it as a starting point for their protocols these days. As long as we're exposing code over HTTP, we could expose documentation and other metadata in the same way. http://foo.com/somefunction would give and executable, /somefunction/doc would be rendered documentation, /somefunction/sig would be a signed hash of the code, which is important if people are ever going to let it run on their machines. I'm nodding off right now, but pretend I am describing a good security model instead.

(This would be even cooler when combined with Tahoe-LAFS, which I'm going to blog about soon, but not tonight.)

I vaguely remember somebody having worked on this before in one of those languages nobody uses - Haskell? Erlang? Clojure? I forget - but it was a language-specific solution and therefore boring.

Anyway. My real goal in this is to abstract away the languages that people code in. It's 2010, and we should have papered over all of this decades ago. Up until a few years ago, the command line was the closest thing we had to my ideal. These days, web services might be closer - they support structured data through JSON or XML, generally, which is a hell of a lot more than you can say about the command line. They're also a lot harder to get at, but that's just because of a lack of decent tools.

Actually, though, now that I think about it, what I'm describing here has a lot in common with the #! functionality of the command line. Actually, it's almost exactly the same. Maybe building this will be a lot easier than I thought. :D

It's been too long since I blogged. I am forgetting how to write. :( Gonna hit publish before I sober up totally.

Stupid Python tricks, volume 1

2010-04-30T09:52:00.003-07:00

So my thesis is due in less than a week, but that doesn't mean I can't write random nifty things in my free time. >_>

Like I've said in previous posts, UNIX pipes are pretty neat. So let's say I want to replicate them in Python. First: is it even possible? Turns out yeah; Python has operator overloading. In general I am opposed to operator overloading, since it makes it possible to write really horrible code. Really horrible. On the other hand, it also enables Stupid Language Tricks, which might just balance out. :D

I'll do this in two parts, 'cos the full trick is really a combination of two stupid tricks.

Trick 1

First: we want a pipe-like syntax for doing stuff. We can do this by overloading the | operator in Python, using the __or__ and __ror__ functions. Initially, I tried to write classes that implemented the __or__ function to sort of stick together, and strung instances of these classes into a pipeline. Let's say I want to take the absolute value of an input, convert it to a string, and reverse the digits. (No reason - it's just an operation that takes several functions to do.) It'd look like this:

pipe_wrap(abs) | pipe_wrap(str) | pipe_wrap(lambda s: s[::-1])

This turned out to be really messy. I want to write a pipe using regular functions, and having to wrap the functions like that just looks messy.

The approach I ended up with is having "joiner" objects in between. First, I defined a joiner class which implements both __or__ and __ror__, so it can handle |s on both the left and right sides. When it's received things on both sides, it magically turns into the result of calling a provided function on the two things. Here's the class:

class incomplete_join(object):
  def __init__(self, compose, l=None, r=None):
    self.compose = compose
    self.l = l
    self.r = r
  
  def __call__(self, *args, **kwargs):
    raise Exception("Incomplete join: l=%s r=%s" % (repr(self.l), repr(self.r)))
  
  def _do_join(self, l=None, r=None):
    if l is None:
      l = self.l
    if r is None:
      r = self.r
  
    if l is not None and r is not None:
      return self.compose(l, r)
   
    return incomplete_join(self.compose, l, r)
 
  def __or__(self, right):
    return self._do_join(r=right)
  
  def __ror__(self, left):
    return self._do_join(l=left)

def compose(f1, f2):
  return lambda x: f2(f1(x))

# Print function, because Python doesn't yet let you treat "print" as a function
def prnt(x):
  print x

j = incomplete_join(compose)

This is all we need to start composing functions together using pipes. compose implements standard functional composition, and incomplete_join handles the pipe-like syntax. To use our previous example, the following two things are equivalent:

temp = abs |j| str |j| (lambda s: s[::-1]) |j| prnt
temp2 = compose(compose(compose(abs, str), (lambda s: s[::-1])), prnt)

Pretty neat! However, we started out wanting to implement pipes. We've got pipe syntax, sort of, but not pipe behavior. Ideally, we'd like to have a set of objects which can run in parallel in separate processes, which have user-defined behaviors, and which can be combined into a pipeline. So:

Trick 2

In the first example, compose was just a simple thing that composed functions together. What if we used something fancier in its place? What if, for instance, we had a function that took two objects (which stored their results in multiprocessing queues), and called the second one in a new process, and took care of shuffling data from the output queue of the first into the second?

import multiprocessing

class queue_pipe(object):
  def __init__(self, func):
    self.func = func
    self.output_queue = multiprocessing.Queue(1)
  
  def __call__(self, item):
    if item is None:
      self.output_queue.put(None)
    else:
      self.output_queue.put(self.func(item))
  
  def next(self):
    item = self.output_queue.get()
    if item is None:
      raise StopIteration()
    return item

class queue_sink(object):
  def __init__(self, func):
    self.func = func
  
  def __call__(self, item):
    if item is not None:
      self.func(item)

def queue_compose(q1, q2):
  def transfer(q1, q2):
    try:
      item = q1.next()
      while True:
        q2(item)
        item = q1.next()
    except StopIteration:
      q2(None)
  
  proc = multiprocessing.Process(target=transfer, args=(q1, q2))
  proc.start()
  return q2
 
q = incomplete_join(queue_compose)

So what does this do? Using this, you can create a pipeline out of callable iterators. queue_pipe implements both the callable and iterator interfaces, and queue_sink is a (sort of unnecessary) callable which filters out the terminal None value in the pipe. transfer() runs in a new process, and bridges the gap between two pipe components.

An example would be a lot more descriptive here.

@queue_pipe
def double(x):
        return x * 2

qprnt = queue_sink(prnt)

iter(range(50)) |q| double |q| qprnt

This example demonstrates two different ways you can use the wrapper classes, just for the heck of it. The single line of code at the end constructs a pipeline across three processes, and executes each function for each item in the input, in a separate process. (Just like a UNIX pipe!) As an added bonus, the ends of the pipe are standard Python objects, so you can use any iterable at the front and any callable at the back. It even cleans up the processes afterward, since they exit when the end of the input is reached.

So, yeah. This is the kind of stuff I work on when I'm procrastinating on my thesis.

Disclaimer: I do not recommend using this trick in any critical code. It is a hack, and an incomplete one at that. I take no responsibility for bad things that may happen if you actually use this in production code. I cannot guarantee that this code will not fork-bomb your systems, give you the flu, or kidnap your pets.

JSON pipes

2010-04-08T18:31:00.000-07:00

So I mentioned this in my pre-vacation post, but I'm actually running with this now - I've got a sort of prototype going and everything. It's written in Python, but I'm designing it to be trivial to use from any language with a good JSON library, which I think is pretty much every language these days.

Rationale

The UNIX command line really has two problems. First, it's too user-friendly. ("Whaa?" Keep reading.) Any tools which deal with structured data have to present that data to the user, we assume, so programs commonly do things like wrapping or truncating long lines, rounding numbers to be more readable, lining up columns, things like that. However, one of the founding principles of UNIX is that tools should be designed so that their output can be passed to other programs as input. What's good for humans is bad for programs; too much cleaning up the output and it's basically unusable in a pipeline. (The curses library is an extreme example of where that can lead.) There's a constant tension between user-friendly and parsable, so most programs have landed at a weird compromise, and aren't very good at either one.

The second problem is delimiting and escaping. UNIX handles them in a completely ad-hoc and disorganized manner, when it addresses them at all. The worst example of this in my mind is "find -print0". find passes filenames around as newline-separated raw data, and this works fine as long as the filenames don't contain blanks or newlines. Since it turns out that some filenames actually do, they added a special case to find where it would delimit its output with null characters, and then they went and had xargs accept null-delimited data. Problem solved! As long as every program you want to pipe into has implemented the same hack, of course.

And really, find is one of the better examples out there. Most programs ignore delimited and escaping completely, and just do whatever comes to mind. As an example: I'm pretty sure it's impossible to correctly parse the output of "df" if any of your mounted devices contains spaces.

Solution

Programs should use a more structured data format on their stdin and stdout streams, when it makes sense. I'm using JSON for this, because it has everything I need, nothing I don't, and it's dead easy to parse. In a nutshell, every line going through the pipe is a valid JSON value, usually either an object or a string. This buys us a lot - records are trivial to parse (and even named, because you can use dicts for them - hallelujah!), strings are properly escaped, and if we want to then convert to nice human-readable output, the program to do it only needs to be a few lines long. And, it turns out, writing programs to use JSON streams isn't any harder than writing programs to use raw text streams.

(I should emphasize this point: I'm using exactly one line per JSON document. It's entirely possible to parse JSON streams without this restriction - I have a parser that does it, actually - but it's more trouble than necessary. Every language can parse lines out of a stream, and just about every language has a JSON parser. No point requiring a more advanced parser when adding this restriction lets us parse a stream using only a few lines of code.)

Here is a simple example, using the tools I've written so far. It finds the three largest files in a directory:

jls | jsort size | head -n 3

(Note that it works seamlessly with existing line-oriented UNIX tools.)

For comparison, here's the best standard UNIX equivalent I can write:

ls -l | sort -rk 5 | head -n 3

It's about the same length, but it took longer to write, and I had to consult the sort man page and try it out a few times before it worked. (And, really, it would have taken much longer if I hadn't happened to know that sort could sort by arbitrary fields in whitespace-delimited data.) In this example, the main thing JSON buys us is named fields for the sort.

"But hey," you might say, "that's a pretty basic example! Show me something that's really difficult to do with standard command line tools."

RSS feeds are a great example of the sort of structured data that's difficult to fit in a pipe. Let's say you want to write a generic, reusable UNIX program for fetching an RSS feed. Your options for output are either dumping the raw XML (ewww, gross! plus, XML streams are poorly defined and hard to parse), or making up an ad-hoc format ("okay, so every line will be one piece of data, in Key: Value format, and a double newline will be the separator between RSS entries, and the entry text will be base64 encoded so we don't have to worry about double newlines in it, and...."). Or, you could dump the data as a JSON stream! Not only is this trivial (my example of this is only 35 lines long), it's also really easy to work with. For example:

jrss $feed_url | jget link | jxargs -n 1 firefox

I dare you to solve this problem this nicely using standard UNIX practices. (No, you can't just write a program that reads an RSS feed and only spits out the links. The whole point is to get stuff done by combining very general pieces. Yes, I know that jrss has the feedparser library doing most of the work. The point isn't really the implementation; it's the fact that jrss is an extremely general-purpose command line tool.)

Conclusion

I've always thought that pipes were far more elegant and useful than the tools we use with them. I feel like I've finally managed to justify that, by coming up with a better way to pass data through them. Even if you're not convinced that we should start putting JSON everywhere, I hope I've at least convinced some people to start thinking about the shortcomings of traditional UNIX tools. There has been entirely too little of that lately.

For the curious: the code I've been playing around with is online at bitbucket. Suggestions and discussion are, naturally, welcome.

Vacation miscellany

2010-04-01T20:33:00.000-07:00

* For some reason I've been sitting on this post for a week. Wanted to make sure I didn't miss anything, I guess. Plus I've been really busy. :(

* ISPs are by nature jerks about SMTP servers, but on this vacation for the first time, their stupid policies have actually cost me a meal. :[

* Everybody in Guyana apparently has a nickname, or "call name". People get them when they're kids, usually, and they just sort of stick. My mom's nickname is Sister, because one of my aunts used to call her that all the time when they were kids. My Uncle Val's nickname is Valmiki, after the poet. One of my great-uncles is Doctor, because he wanted to be a doctor when he was a kid, and would go around giving people pretend injections. (This, despite the fact that he's been a practicing lawyer for several decades now.)

* The next time you're vacationing in the Caribbean, here's an Authentic Caribbean Experience: Find a coconut vendor; they'll be the guys with the carts full of coconuts and the machetes. They chop the tops right off the coconuts, stick a straw in them, and give them to you to drink. When you're done, you can go back to them with the shell and they'll chop it up for you, and cut off a piece of the shell so you can scrape the jelly out from the inside. If you like coconut, it's kind of awesome. If you don't like coconut, it's possible that you've never had a good coconut D:

* Somehow, I feel like I was able to think more clearly after being Internet-deprived for a few days. It was the weirdest thing, like the steady stream of distractions I get online have more than the obvious effect of distracting me. This seems like something worth looking into later on.

* Dear Blogger: If I am going through the trouble of writing my own HTML, feel free to take that as a hint that I don't want you attempting to rewrite my URLs and breaking them in the process every time I edit a post. Wordpress has its faults, but I'm pretty sure it can handle a damn anchor link.

Waterfalls and Rivers

2010-03-20T21:14:00.002-07:00

The day started at 5:30 AM. Boo, hiss.

We drove out to the tour office, and my sister started throwing up - luckily, it was temporary, probably from waking up so damn early; she was fine for the important parts of the day. After we signed a bunch of forms, the tour people drove us out to the airport, where we got in the smallest plane I have ever flown in. It was an 8-person plane, so my youngest sister actually got to sit in the co-pilot's seat on the way there. I was kind of jealous. D:

After a flight of about an hour, we landed at this little airstrip out in the middle of nowhere. (Why did we fly? There are no roads anywhere near there.) We grabbed icy water bottles, took a quick nature hike through the rainforest, and after admiring a few random plants and miscellaneous buildings (and also some rainforest fauna [1]), we started to hear a roar off in the distance and we knew we were getting close. Then the source of the roar was right in front of us (that amount of vegetation muffles sounds like crazy) and we were standing on a tiny lip of rock jutting out over Kaieteur Falls [2].

I have seen Niagara Falls up close, so trust me when I say that Kaieteur Falls is freaking unbelievably huge.

Surveyors agree with me; at 741 feet straight down, it's the worlds tallest waterfall. The water is the color of iced tea, which is an effect that I'll touch on more later. The bottom half of the fall is completely covered by mist half the time, but the other half of the time, it's just completely breathtaking. One thing that you can't help but notice is the complete lack of guardrails or other safety features - this isn't some coddled first-world natural wonder, where you go out on a little boat wearing a custom-made poncho, and stop by the gift shop on the way back. [3] The only thing to stop you from crawling right up to the edge and sticking your head over is the tour guide, who became increasingly agitated with my mom when she did exactly that. It was kind of scary, actually - he was trying to tell her that the rocks were slippery, and that she shouldn't do that, and she was completely oblivious that he was saying anything, or that the rest of us were trying to get her attention. "Mom! The tour guide has been yelling at you for the past two minutes to come back from there, and so have the rest of us." "Whaa?"

On the way back I actually managed to stay awake for most of the plane ride (Dramamine knocks me right out, apparently), and got some nice videos. One thing most people don't realize about the rainforest: there are a hell of a lot of trees in there. I seriously would estimate that I saw a few million trees today. Horizon to horizon, nothing to see but the tops of trees.

The next leg of the trip was to a smallish riverfront resort, which was a nice change from hiking through the woods. We got there via a small boat, because - surprise, surprise - there are no roads going there. The first thing that strikes you is the water. In the Demerara river, the water is the color of chocolate milk, because of all the silt it picks up. In the tributary we spent most of our time in, on the other hand, the water was the color of iced tea, because of all the dead leaves that are in the water. (Probably the same reason that Kaieteur Falls is that color.) [4] (Looking back at the analogies we came up with, it's possible that we were a little bit thirsty after sitting on a boat for two hours during the hottest part of the day.)

The resort was really really nice - beach volleyball (that we lost horribly at, but still) and kayaking. They also had rice, dal, and assorted other foods. (I don't think that a day has gone by on this trip that we haven't eaten rice and dal. It's kind of the quintessential Guyanese food.) They even had hammocks, which are amazing under pretty much any circumstances, but especially so in the tropics. Not a lot to say about the resort, really. It was very nice, but not a lot happened.

Te tributary we used to get to it has an interesting property - the flow changes direction based on the tide, every six hours. I'm not completely certain, but I'm pretty sure that we went upstream both ways, going and coming. We got back just before it got completely dark out, which was kind of a relief. We were going down the river without any lights, and it was kind of dicey toward the end. Still, we made it. (I could swear I saw a completely black butterfly on the way back, but I may have imagined it. Still, the two people that read my blog and have seen Bleach may appreciate the reference.)

The day ended with a lot of Chinese food. Best day of the vacation so far? I think so.

[1] Rainforest ants = holy fucking insanely huge. I was just walking along when I saw an ant carrying something, the damn thing must have been an inch long.

[2] "Kaieteur Falls" is actually a misnomer, according to our tour guide. "Kaieteur" means "Kai Falls" in whichever language it came from, so the name everybody uses is redundant. And, for the curious, it's pronounced "kai-choor."

[3] On a somewhat more sobering note, there was a suicide there last year. Both our tour guide and our driver remembered it; the latter mentioned that he knew because there was one fewer seat filled in his van on the next leg of the trip. :(

[4] I know it's ridiculous, but given Guyana's history, I can't help but imagine a mad Englishman at the head of the river, dancing around a machine he's built which can prepare enough tea to fill a river with it.

The One-Dimensional City

2010-03-18T20:36:00.001-07:00

Blogging live from Guyana! Because if I don't write this stuff down now, I'm going to forget it by the time I get back and have reliable Internet access.

So the first thing I noticed about driving in Guyana is that they drive on the left side of the road here. I guess that makes sense; it was originally a British colony after all, but it's still unsettling to see. With the caveat that my experience is all secondhand, from the passenger seats in taxis, driving in Guyana doesn't seem as awful as someplace like India. There are still pedestrians walking in the street randomly, as well as feral dogs [1], and cows and donkeys in the rural areas. But it's certainly not as crowded as the city streets in New Delhi, and the roads are actually pretty good here. As good as the roads in Houston, at least, which definitely surprised me.

The last taxi ride we took (from a Chinese restaurant back to the hotel) cost us $300, but that's not so bad when you consider that the exchange rate from Guyanese dollars to US dollars is 200:1. It turns out that Chinese food is big in Guyana - probably even more so than Indian food. Considering the number of Indians that live here, this is actually pretty surprising.

But anyway, the reason I'm talking about roads is the Big Road. I'm calling it that, because I can't find an official name for it. Between Georgetown and Skeldon, there's one long highway that runs across half of Guyana's coastline. It's just your basic two lane highway, except that there are houses and offices and basically all sorts of buildings running along it uninterrupted the whole way down. Aside from a few empty lots, or a few acres of farmland here and there, it's one long unbroken urban area. In most places, the development only goes back a few lots, so you end up with an urban area that's several dozen miles long and a few hundred feet wide. It's like an entire city was stretched out along a single road. Somewhat arbitrarily, it's divided into villages; I think the divisions might even be on mile markers. If I recall correctly, the villages are numbered one to sixty-something, going from West to East.

All of this was necessary for the next bit to make sense: we visited the house where my mom grew up, in 43. Really, the official name is Village #43, but everybody just calls them by their numbers. (We also visited the school she used to go to - that was neat, if a little embarrassing. She even ended up making a speech. D:) It's kind of amusing/amazing how many people know our family. We joke about how everybody in Guyana knows everybody else, but only because there's a grain of truth to it. Example: at the hotel we stayed at the other night, the owner heard that my uncle's last name was S_____. He asked, "Not the S_____'s from 43?" Similar scenes have happened several times, believe it or not. It's really cool and at the same time a bit disturbing.

Anyway, speaking of my uncle, this is his laptop, so this is all I can write for now. The next post is going to be about waterfalls, and if I'm late enough writing it, it may contain videos.

[1] Feral puppies: just as adorable as tame puppies. (From a distance, of course.)

Pre-vacation idea dump

2010-03-12T10:16:00.000-08:00

I'm going to be out of the country for a week and a half! Gotta braindump this stuff or I will forget it by the time I get back.

Mobile filesystem

My search for a network filesystem that supports offline writes can best be described as a dismal failure. AFS is the closest thing that I could find, and it was designed in the '80s and is kind of archaic, and looks to be a pain to set up. (Plus I doubt my home stuff will ever scale up to tens of thousands of computers. :p) DRBD+NFS was the next best candidate, but I dunno. I'm sick of all the interesting stuff being implemented at the block layer, while filesystem designers are still dicking around with consistency models (see for instance the drama last year with ext4 and fsync()). Plus it won't do me any good for disconnected operation, whoops. Rsync is the third best candidate, which is seriously just sad. (Honorable mention goes to XtreemFS, which (in addition to having a truly awful name) only supports read-only replication. Way to only solve the easy problems, guys!)

So, fine. If it doesn't exist, I'll just build the damn thing myself.

My design is based on BTRFS writable snapshots, for a lot of reasons. First because they're freaking cool, second because it means this design degrades in the worst case to a completely usable filesystem. Third, because they're really freaking cool. :D

So, computer A wants to sync with computer B. A creates a writable snapshot, and then uses rsync to pull in the updates from B. Now A has a copy of the current state of B's filesystem, without using up excessive space because it was copied over a snapshot, and we're assuming that the differences are relatively small. (Directory renames mess this up, and are a weakness of this approach, but not a deal-breaker.) This can all happen in the background, so far. Now, the user at A can apply a merge strategy using a hypothetical merge program, and bring their main copy of the data into sync with B's copy. They can do this at their convenience, because the data's all stored locally, so there's no need for B to even be connected. If there are conflicting changes, we can look back at the original data (because we have a snapshot) and the two divergent copies; a three-way merge is pretty much the best we can hope for here. Advantages of using writable snapshots: saves disk space, lets this strategy work with any number of other computers, freaking cool.

The merge program is completely hypothetical, because whoops, it turns out that merging sets of files automatically is a fundamentally hard problem. Still, there are all sorts of useful heuristics we can use, before we kick the problem up to the user. If a copy of the filesystem isn't being used, it could be set in a "receiver" mode, where it just takes all the updates it sees from other computers. Files that are only modified on one side or the other of the merge could also be copied safely, probably. Er, maybe not. Gotta think about that more carefully. N-way merges could be handled as a series of 2-way merges, I think.

Vector clocks are basically magic, and this scheme will have to use them to keep track of merge results (so that they can be propagated), and for garbage collection (because disk space is cheap, but it's not that cheap). All the accounting information can be stored as a special file in the filesystem, which is doubly convenient because it means that every system has access to the state of every other system. Ooooh, we could store the vector clock across all of those - it needs to store a logical clock per-replica, and we conveniently have per-replica storage already because of the nature of the system. Slick. :D

Structured (JSON) pipes

The philosophy of UNIX involves creating programs as commands that can receive input from other programs, and pass output to other programs. In that sense, UNIX has failed us. Text streams can be easy to read, and they can be easy for a program to parse, but they are rarely both. This is really a topic for a proper rant, some other time.

But what to do about it? My solution (which I will implement when I get some free time) is structured pipes. Pipes should take a stream of JSON objects (or some other format, but JSON is nice for this sort of thing) on standard input, and write a stream of JSON objects to standard output. Then, instead of writing fragile and error prone nonsense to parse text using cut and awk and sed and miscellaneous tools, we could have a standard JSON query command that pulls out a given field of an object. Instead of trying to use the same output stream for humans and programs, we could have a standard program to translate the JSON stream to something human-readable. (Hell, a terminal could do this automatically - then we wouldn't even have to think about the JSON layer underneath. Everything would just work.) Programs could output a format specifier as part of the stream - then we could get output at least as nice looking as the crap we have now, and the potential for it to be a lot better. Plus, programs could change their output format without breaking every damn script that uses them written in the past three decades. (Example: df. It's hopelessly out of date, and there's no good way to shoehorn information about RAID, or compression, or modern disk stuff, into its output. But nobody can change the output format, because scripts depend on it.)

UNIX also sucks at delimiting fields, which you'd think would be a basic thing when your entire operating system philosophy is based on processing text streams. It uses a random mishmash of spaces, tabs, and null characters for cases when whitespace won't do. Why not set a standard delimiter? Like, I don't know, one of the ASCII characters explicitly set aside for use as delimiters? The problem is, UNIX text streams have no concept of escaping, so any character could potentially appear in the stream. UNIX gets around this, when it bothers to address the problem at all, by letting you choose between a few output delimiters. For instance, if you think your output will have newlines, you can tell the find command to use nulls instead. This complicates the program reading the output of find, of course, so the ones that are likely to be hooked up to find (xargs, for example) also have special cases for handling null-terminated input.

This is kind of turning into a rant, so I will summarize: The whole thing is an ugly mess. And using JSON as an intermediate format would also get us standard delimiters for free.

When I write this, I suppose I'll have to also write a set of core utilities that operates on JSON streams. It shouldn't be too much work, I don't think.

Donut math

2010-03-09T02:33:00.000-08:00

There is a little donut shop down the street from me, called Ken's Donuts. The donuts are good (better than any chain I've ever eaten at, at least), cheap, and best of all, they're open 24 hours a day. They have maple-glazed donuts, which are kind of seriously amazing. And to top it all off, they have samosas!

So, because I am trying to save money, I've started quantifying impulse buys in terms of Ken's donuts. As in, the money I want to spend on this could buy me X donuts. Will it make me happier than X donuts? (Yes, there's the obvious fallacy with diminishing returns. Obviously I don't apply this logic to stuff like groceries, jeez.) Mediocre fast-food value meals don't look quite so good next to a dozen donuts. Or, on the flip side, a slice of cheese at that Death Metal Pizza place on 6th Street will cost me about five donuts, but it's worth it because I can say I've eaten at a Death Metal pizza place.

But then, I was thinking as I was trying to fall asleep. My rent is $1300 per month, split with my sister, so $650 per month. Over the course of the academic year, this comes out to about $6000 that I'm spending on rent. A bit of quick mental arithmetic reveals that instead of living here, I could buy about ten thousand donuts.

Ten.

Thousand.

Donuts.

I would be the world's happiest hobo.

Linux advocate hate

2010-02-18T20:44:00.000-08:00

I like Linux, I really do. It's a reasonably flexible foundation for doing things with computers that you'd normally not be able to do. As a rule, I don't like the way operating systems are done right now, but Linux at least has the potential to be a good system at some point in the future. That is why I use Linux, and have been using it for most of this decade.

The above disclaimer is necessary because I'm going to spend the rest of this post taking down some of the more vapid and/or blatantly wrong arguments people have made (and still make!) in favor of Linux. Just today, somebody on identi.ca sent this whylinuxisbetter.net site to the linux group. At first, it just annoyed me, but then I saw an amazing opportunity: here, condensed into one convenient page, are most of the outright idiotic claims people make about Linux. I've been wanting to write a post like this for a long time, and here I've found my chance.

I'm just going to go through and address the sections of the site one at a time:

Forget about viruses.

The good news: nobody writes viruses for Linux! The bad news: nobody really writes applications for Linux, either, for some of the same reasons. "Linux" isn't really an operating system, something that people never seem to be aware of - it's a bunch of pieces of an OS that distributions combine to make operating systems. If you actually want to write a program that runs on all Linux machines, you're going to have a hell of a time of it, especially with some of the machines running on non-x86 architectures. Oh well, you win some, you lose some.

Granted, there are aspects of Linux that actually do mean something when it comes to security. Users don't, by default, have access to the entire operating system to do what they want with it, as they did for the entire Windows 9.x series. But come on, that's like bragging that you're potty trained. It's great when you're a kid, but once you're 19 years old (Linux: started in 1991), it's well past the time to make a big deal of it.

Is your system unstable?

Look, if basic security measures are like potty training, then this is like being able to walk. People write operating systems for college classes that can manage this much. (People other than me, unfortunately - when I took this class, we had a sneaky bug in the syscall interface that we never tracked down. :( ) If you have to brag about Linux being able to sit and do nothing for long periods of time, I seriously have to wonder about the other things it has to offer. This is an especially useless point now that other operating systems have basically caught up on this front - back in the day, system crashes were normal for desktops, and Linux (having its roots in server operating systems) was better than the competition. Things have changed a lot in the past 10 years, though, and Linux advocacy rhetoric hasn't kept up.

Linux protects your computer

Hey, isn't this the same as the first point? >_> This focuses more on the "more eyes = less bugs" idea, though, so I'll run with that. The fact is, even though there are potentially "hundreds of thousands" of people looking at the source code to a given Linux app, it's far more likely that there are hundreds, or thousands if it's a really big one. That's on par with the number of people Microsoft has working on their major applications. Given the security track record of Linux distributions, it's difficult to say that more eyes actually means fewer bugs - they do all right, but they're not exactly OpenBSD.

Don't pay $300 for your operating system

Yeah, yeah, Linux is free. For a lot of people, that would be great news, except they have one or two Windows- or Mac-only applications that they just can't live without. Linux saves these people $0. (And just for grins: the absolute basic package of RHEL costs more than $300. Annually.)

Freedom!

Software freedom sounds great, and actually, it basically is. I've benefited from having source code too many times to really say anything about this. >_> Unfortunately, it has a dark side, too.

There's a slightly disturbing segment of the Free Software movement that considers non-Free software to be antisocial, immoral, or even pure evil. And there's a contingent that actively seeks out free software heretics, in true old-fashioned witchhunt style. BoycottNovell is a prime example - Novell makes the SuSE Linux distribution, and thanks to a deal they made with Microsoft, they're now evil incarnate. (Never mind that they've probably contributed more to Linux than everybody at BoycottNovell put together.)

When the system has installed, why would you still need to install stuff?

I always love how Linux zealots will swear up and down that Windows is bloated, that it's filled with stuff nobody wants, and that Microsoft did a horrible thing by including Internet Explorer by default all those years ago - and then turn around and brag that Linux is better because it comes with a bunch of software preinstalled. Um, hypocritical much?

There are bonus fail points here, actually, because they're claiming that every Linux install comes with X, Y, and Z software preinstalled. Um, no, it's completely distro-dependent. Gentoo, for instance, comes with a root shell and a package manager - everything you need, in other words, to build your own system. This is a pet peeve of mine, when people claim that "Linux" can do some neat thing. In most cases, they're flat-out wrong; they're actually talking about some variant of some Linux distribution. It is an important distinction, dammit.

Forget about drivers

Because there aren't any! *rimshot* Seriously, this one practically wrote itself. :/

In all seriousness, though, the reason Linux doesn't need a lot of drivers is because it has drivers for just about every piece of hardware it supports built into the kernel itself. This is a design decision that I've never been comfortable with, but the people that actually write the kernel seem fine with it, and I'm willing to accept that they know a bit more about the situation than I do. XD

If the driver isn't in the kernel, though, good luck. The kernel devs also maintain a policy that they can break the internal kernel APIs whenever, which is great for them, but in my opinion also discourages companies from writing third-party drivers. (Some do, of course; nVidia is probably the most high-profile example.) They want to encourage companies to contribute open-source drivers, which is laudable, but I'm more of a pragmatist; I'd be happy to see drivers no matter what form they took.

Update all your software with a single click.

...after carefully reading the update list to make sure that nothing will break. I actually had a bad experience with this back in the day, using Red Hat 8 (not Fedora 8, this was pre-Fedora ;). Naively, I thought you could just install updates like on Windows, so I went ahead and upgraded glibc, and stuff broke. People were aghast: "you should have been more careful updating that! glibc is important!" I get the impression as I'm writing this, though, that this anecdote may be a little dated. >_>

No, the bigger problem I have with this section is the idea of a package manager. (I've already blogged about that here.) Linux advocates like to think that the use of a package repository is a completely uncontroversial choice, but there are tradeoffs there that invariably get glossed over, such as the relative difficulty of installing software from outside the repository. On Gentoo, for example, you have to either find an overlay that contains the software and add that, or find an ebuild, add it to your own personal overlay (creating one if you don't already have one), generate a digest for the ebuild, and then install it.

(I hate to only say negative things about Linux software installation, so I'll also point to Zero Install, which is a really well designed system. Fully decentralized systems are automatically awesome.)

Why copy software illegally if you can get it for free?

Hey, kids! Copyright infringement is bad! This page is a little disjointed, though, because it also lists free alternatives to programs that are already free. (Is Kazaa really pirated all that often?) I agree with the gist of what they're saying, but come on, that's just sloppy.

Need new software? Don't bother searching the web, Linux gets it for you.

This actually is a genuine advantage of package managers, and training users not to download and install random executables from the Internet is certainly a plus when it comes to security. Again, though, you run into problems when you want to install software that's not in the repositories. (I also think the page way oversells the ease of searching for packages - it just doesn't work that way in practice. Looking for stuff online isn't much better, though, so I can't criticize too much there.)

Jump into the next generation of desktops.

No amateur Linux advocacy page would be complete without a screenful of Compiz videos. When it comes to convincing people that Linux is awesome, it helps to have some flashy demos that speak for themselves; it's hard to get somebody excited about a flexible base system, or software freedom, or whyever you use Linux. Compiz has always been a bit hackish and temperamental, though, and the software and configuration requirements can be tricky unless your distro comes with it preinstalled and preconfigured.

Compiz also has a real problem with configurability; there's too much of it. Look, I don't really want to have to set up keybindings for all the actions (a necessity, because many of them use the Windows key, which my keyboard predates), and I don't want to have to go through and decide which effects I want. A cohesive user experience is not too much to ask. The only really widely used parts of Compiz are the desktop cube effect and the wobbly windows, and even those are optional and overly-configurable. (Okay, setting the spring constant for wobbly windows is pretty cool. Can't deny that.)

Plus, when you get right down to it, Compiz just isn't that useful. I've used it on and off ever since it came out, and while it's nice to look at and good for impressing the opposite sex, I always get bored of it and go back to a boring old 2D desktop.

Does your digital life seem fragmented?

Teh rage. Hey, person who wrote this page: come a little closer, so I can beat you to death with my fragmented root drive. You claim that's impossible, so it won't hurt, right? :D

First off, this page hasn't been accurate since NTFS became the default filesystem on Windows. I admire the dedication it takes to keep talking about FATs poor allocation strategy for something like a decade past its relevance, but seriously, it's time to give it a rest and move on. (Although, it might have something to do with Linux not having full NTFS support until 2006.) So there's half the page that's just flat-out wrong.

The other half is even more wrong. Oh, where to begin.

* It claims that Linux filesystems have a certain strategy. Linux has a lot of filesystems. Do you mean to say that all of them use the same block allocation strategy that you describe, that all of them have separate magical fragmentation avoidance subsystems that work like this, or that you don't know what the hell you're talking about and you're just making stuff up?

* It claims that Linux filesystems don't get fragmented. Um, bullshit. It's common knowledge that filling a filesystem past 90% will lead to serious fragmentation, if you leave it that way for too long. It's common knowledge that Gentoo's portage tree causes severe fragmentation on whatever filesystem you put it on, to the point that a lot of people recommend putting it on its own isolated filesystem. Ext4 and BTRFS (two next-generation Linux filesystems) both support online defragmentation. Does that sound like a feature that filesystem devs would put in just for fun? They know that filesystems become fragmented. Everybody who has any idea what they're talking about knows that Linux filesystems become fragmented.

Choose what your desktop looks like

"Wait, what? Can't every OS do that?" I'll save you some confusion: this page is actually about Linux having multiple desktop environments. Of course, the way they describe it, there's nothing there that you can't do with themes. Philosophical differences? Competing toolkits? No, the real difference between different desktops is that they use different colors. Way to miss the point.

Why does your Windows get slower day after day?

Frankly, I'm still not convinced that this is even an inherent Windows thing. I have friends who are only too happy to tell me that Windows isn't that bad, and that their Windows installs are still running fine after X amount of time. How? It probably has something to do with them keeping track of what's installed on their systems, and what's running in the background - slightly advanced computer maintenance, in other words. Know what else requires slightly advanced maintenance? Linux. :p

Do something for the environment.

I've gotta say, I am impressed with this one. That is a really clever way to spin the fact that neither Linux nor Linux-compatible software is sold in stores. (Except it sometimes is, which negates this entire point. But hey, let's not even get into that.)

There's one thing on this page that still bugs me, though - it's the same fallacy again where they conflate "Linux" and an unspecified subset of Linux distros and configurations. Some (but not many) distributions will run great on your old 486, some won't run at all. Hell, you can run Linux on your router, or your cell phone, if you want to. Should we start claiming that desktop Linux runs great on these too? Or do we draw the line at "only a little misleading"? Because that's what it is when you tell somebody that "Linux" runs great on X.

It's especially entertaining to see people actually believe this sort of argument, and try to run Linux on old hardware, and then come online asking for help. Invariably, you will have a few people who just insist that it'd be easier for them to just buy a new system, proving once again that just because you can, doesn't mean that anybody actually thinks it's a good idea.

No backdoors in your software.

Sigh. There are two reasons that this is wrong; one practical, and the other more theoretical and much more worrying. First, it turns out that some Windows users actually do keep an eye on what their systems are sending over the network, and they would definitely notice any attempt by the OS to "phone home". If there actually was some sort of secret user monitoring software hidden in Windows, the backlash against Microsoft would be pretty severe once it got out. Microsoft isn't as stupid a company as some seem to think, and I don't see them trying this anytime soon.

Second is a paper by Ken Thompson titled Reflections on Trusting Trust. It basically details how, using a compromised compiler, you can compromise a system where every component, even the compiler, is open source, without any indication whatsoever in the source. It's actually a pretty cool concept, and if you're at all interested in computer security then this paper is a must-read.

Enjoy free and unlimited support

This is one of the things about the Linux advocacy movement that pisses me off the most. Every time somebody says this (and they say it a lot) they're implicitly enlisting every Linux enthusiast in the world to act as unpaid tech support. You know what? Not all of us are "happy to see more and more people switch to Linux"; not all of us buy into your weird world domination fantasies. Some of us would rather just use Linux, and not have to hold the hand of every poor schmuck that you installed Ubuntu for and then dumped on our collective doorstep. Some of us are Linux Snobs, and that's our prerogative. :p

(Another fun counterexample: ESR has written a fairly extensive guide on how to ask questions about Linux without being flamed or told to RTFM. Yep, Linux users sure are friendly and helpful, as long as you follow all these rules.)

And then, of course, this entire point is predicated on the completely ridiculous implicit assumption that other OSes don't have online communities that you can ask for help.They do, of course. The idea with "support" is that you provide more than the ad-hoc online communities do; otherwise, you may as well go home. When you claim that Linux has free, unlimited tech support from online communities, you're effectively claiming nothing at all.

Too many windows? Use workspaces.

Oh, man, this is such an awesome idea! I wonder why other OSes don't do the same thing. I didn't wonder for very long, though, because unlike the author of this page, I can use a search engine. OS X has had multiple workspaces for a few years now, and there are a couple different Windows programs that will get you the same thing.

Reporting bugs

Wait, you mean you can't send in bug reports for OSes other than Linux? Should I go and retract the bug reports I've made to Apple, or what? Now I'm very confused. :(

As for not having to wait years for fixes, this is my favorite Firefox bug. It turns 10 this year! But don't worry, it's not a security bug unless you count denial of service as a security problem.

Are you tired of restarting your computer all the time?

As near as I can tell, the entire point of this section is that Windows will nag you to reboot when it needs to reboot, while Linux (by virtue of being a server OS?) never needs to reboot. I'm deliberately leaving aside the issue of conflating different Linux distributions (desktop and server distros are exactly the same, apparently!) because there's another issue in here.

Linux doesn't need to be rebooted very often, but that's because of a specific behavior: if you upgrade a library out from under a program that's using it, it'll just keep using the old version until it exits. If the upgrade is a security upgrade, Linux systems will happily keep using the old, insecure version of the library for an arbitrary amount of time. There are some cases where you really and truly do want to reboot after installing an upgrade, and unless your distribution is exceptionally on top of things, Linux isn't going to tell you when that is.

Let your old computer have a second life

Disappointingly, this section isn't about Second Life. Instead, it rehashes something I addressed earlier in the section about the environment.

Play hundreds of games for free.

This reminds me of those "hundreds of games in one" mini-consoles that nobody ever buys. The harsh truth is, most users don't care about how many games are available, they care about how many games are available that they want to play. All the free games in the world won't help a WoW addict.

Help other countries, and your own

"Custom solutions, support, and consulting? You're right, all of those things are completely impossible with anything but Linux!" I'm not entirely sure what this section is saying, nor that it applies to me at all, seeing as I live in the US.

Use MSN, AIM, ICQ, Jabber, with a single program
Get a great music player
Keep an eye on the weather

For some reason, the last three things listed on the page are available on any major platform. I'm not really sure what they have to do with Linux, but whatever. XD

So what do I think we should do as far as Linux advocacy goes? Hell, I don't care. Like I said above, I'm not really interested in world domination or anything like that. I wouldn't care at all what Linux advocates said one way or the other, were it not for the fact that it gets on my nerves when people are wrong on the Internet.

What I (will) have lived for

2010-02-16T22:00:00.000-08:00

Or: A portrait of the blogger as an old man

Or: The assignment said to write what you'd like to be able to write when you're 90 years old, so this is what I wrote.

I have never intended to make the world a better place.

That's not to say that there's nothing in the world that could be improved. On the contrary, there have been any number of things about the world, about society, about technology, and especially about myself, which I have felt should be fixed. I have done what I could when I've had the chance, but most of these things have been out of my reach for my entire life, and show no signs of changing now. I have lived in this imperfect world, and I have overlooked its flaws, but I have never accepted that this is the way the world must be.

From time to time, though, I have had the opportunity to solve some odd problem, or repair some minor flaw. In doing so, I have made the world a less irritating place to live, for myself, and hopefully for others. It is this idea - acting on the issues that bother me, instead of leaving them as they are - that has been my guiding principle over the years. This sort of principle will probably not solve all the problems of the world, but I sincerely believe that if everybody acted according to it, it could at least solve all the problems we care about.

Some problems, on the other hand, are too enduring to even begin to solve in a human lifetime. Boredom, pain, sadness, and regrets are never far away, if we let them approach, and this is simply a part of being human. Thus far, I have managed to live an interesting, comfortable, happy, and fulfilling life, and I have tried not to stop others from having the same. To the extent that I have succeeded, I am glad to have improved the lives of those around me, or at least glad to have not made things worse.

I have lived my life fighting the good fight, and if the world has become any less a terrible place because of it, so much the better.

Convergent evolution

2010-02-12T13:21:00.000-08:00

There were two announcements from web giants this week that represent, to me, an interesting trend. First, Google announced Google Buzz, which is basically their take on Twitter-style social networking. (I think they've completely missed the point, but that's really a topic for another post.) Second, Facebook has extended their Facebook Chat system to use XMPP, making it a fully-fledged IM system that you can use through many chat clients. (One nice thing that most people don't know about XMPP: you can log in from several locations simultaneously, and messages [even your own replies] go to all locations. None of this AIM-style nonsense with prompting you to log off, or needing multiple accounts.) They've also announced that they're extending Facebook messages into a full email service, with POP and SMTP access.

So why's this interesting? Two of the largest players on the web today have announced services that cover ground the other already does pretty well - Google is basically copying the way Facebook's wall works (and also Twitter, to some extent), and Facebook is copying Google Talk and GMail. In some sense, GMail's interface and Facebook's home page are both evolving into the same kind of social networking site, from completely different directions.

It's no secret that Facebook wants to be the next Google, so that part isn't really surprising. By itself, the fact that they're extending their chat and messaging services is news, but not a trend. What caught me off guard was Google's move, and it got me thinking. I'd always dismissed Facebook's ambitions - we already have one Google, and this Internet ain't big enough for another one. But Facebook isn't looking to replace Google's core business; they're looking to have the same kind of dominance in the social networking space that Google has in the search space. Google understands this, and isn't going to take it lying down.

(Aside: Twitter isn't part of this trend, but that's not really surprising. In my mind, Twitter is the Sarah Palin of social networks - became famous by a fluke, inexplicably popular despite a total lack of substance and many embarrassing failures, considers simplicity a virtue even when it's a liability, unable to cope with a sudden rise to prominence, receives more than its/her fair share of media attention.)

Project Blog: Collatz

2010-01-27T23:14:00.000-08:00

(Note: this post is an extra-credit assignment for my software engineering class.)

The first project is a simple numerical problem based around the Collatz Conjecture. The assignment involves finding the maximum "cycle length" (the number of steps it takes to reach 1) for any number in a given range. Because this is a problem which would involve a lot of repeated computation if it was solved naively, we implemented it using a memoizing cache. Our cache implementation is pretty simple, and uses a HashMap in Java and a dict in Python. This implementation seems to perform well.

Of course, the focus of this assignment isn't on the actual problem, but on the infrastructure we build around it. In particular, we were encouraged to use an issue tracker, plan out the design on a project wiki, use a version control system, and create a comprehensive test suite. We were to use Google Code for the first two. As usual, Google's online code hosting tools are excellent, and even though we didn't actually store any code on it, the wiki and issue tracker were very easy to get started using once we created the project.

My partner and I used Mercurial for version control. Instead of setting up some sort of hosted version control thing, we simply created world readable (chmod 755) clones of a repository on our CS accounts. This isn't exactly secure, but it was the simplest way to get started, and we're reasonably sure that other students won't try to poke around and guess the directory name we used. In my opinion, Mercurial is the simplest of the three suggested VCSes, once you wrap your head around the separation between the repository and the files.

A helpful Mercurial tip: If you add a .hgignore file (note the . at the front of the name - it's a hidden file) to the top level of your repository, Mercurial will ignore files matching certain patterns. This can keep you from accidentally committing a bunch of compiled class files or other generated files, and it also makes the output of "hg status" a lot more usable.

Unit testing isn't something I do often, but I think it's something I should get used to doing on all my projects. For any codebase that I intend to keep around for a while, the value of a test framework that I can quickly try out new changes in would definitely outweigh the cost of writing the tests in the first place. It's pretty cool to be able to get a sanity check that all the code actually does what you think it does. One thing I didn't initially realize is that you can do more than just test the provided functions; I now have a test case which checks the contents of the cache after a test run.

Implementation stuff

It should be possible to write the divide-by-two step as a right shift, and the times-three-plus-one step as a left shift and two adds. This might improve performance, but I haven't done it because I strongly believe in leaving that sort of thing to the compiler. The way I see it, if I'm not using a language that has a good optimizing compiler, I probably don't care about performance all that much to begin with.

(There's another possible optimization that was mentioned in passing during class - combining iterations when possible. Since it's guaranteed that the result of a (3n+1) operation is even, you can implement ((3n+1)/2) as (n + (n >> 1) + 1), and get two iterations done at once.)

There's a tradeoff between using an array for the cache (faster, more memory usage) versus using a mapping type (slower, less wasted memory). It would be interesting to evaluate both of these against a hybrid which uses a fixed-size array for values below a certain threshold, and a mapping type for values above the threshold, since it seems like this would get the best aspects of both types of cache. We haven't implemented this, however, because it would complicate the project unnecessarily.

Problems

We haven't yet submitted our project to the UVa online thing, despite a few unsuccessful tries. Now that the class has discovered the class name requirement, and the fact that the inputs aren't necessarily given in order, we intend to try submitting again. It would certainly be nice if UVa would give a more descriptive error message than "runtime error", though.

Despite what UVa says about the problem, there are a few values (113383, for one) which actually will overflow a 32-bit signed integer. They probably designed the problem with C/C++ in mind, where you would have access to unsigned data types. We worked around this by using long integers.

The assignment seems to imply that it's a good idea to check user input using assertions, but as Professor Downing mentioned in class, that's not right. Because they're usually disabled, assertions should only be used as a sanity check for "impossible" situations. Anything that could actually happen (like the user entering bad input) should be checked separately from the assertions. I think the bit in the assignment about using assertions to check argument validity should be clarified, so people don't get the wrong idea.

Final remarks

Overall, this seems like a very worthwhile introductory project. The actual program is pretty simple, but the real goal is to get the whole class some experience with version control and unit testing, both of which are necessary tools, and both of which are frequently neglected. It always amazes me how far you can get in a computer science program without using version control. :(

Gentoo customizations

2010-01-19T21:27:00.000-08:00

I've had this Gentoo install running for several (three? four? I forget) years now, because reinstalling your OS is for chumps. Unfortunately, that means that I've forgotten a lot of the tweaks I've made to my system, so that when (not if!) my main hard drive gives out, I'll be out of luck. :(

First, a note: my system hard drive is actually an 8 GB CF disk, because I'm way too cheap to get an SSD. Programs load really quickly, but writing to it can be really slow, so I try to keep write-mostly stuff off of it in general.

SquashFS portage tree

The portage package tree is made up of about a squillion tiny text files, and it's mostly read-only - it's basically the ideal use case for SquashFS. I've combined it with unionfs to make a fully read-write portage tree that only takes up 32 MB, instead of hundreds.

SquashFS is included in the Linux kernel now (gzip compression only, though - not a huge deal), but unionfs isn't. You'll have to grab the patch from here. Apply it with "gunzip unionfs-whatever_for_whatever.diff.gz | patch -p1", if I recall correctly. Then enable SquashFS, unionfs, and tmpfs - I'm assuming you know how to build a kernel, not gonna get into that. You'll also need sys-fs/squashfs-tools installed.

I use a few incredibly trivial scripts to manage this; you can get them here. To get started, run make-portage-snapshot.sh, which clones /usr/portage into a new SquashFS image, called portage-snapshot-new.sfs. Then, create /mnt/portage-ro and /mnt/portage-rw. My setup has a read-only mount of the SquashFS image in -ro, and a tmpfs mount in -rw. Next, rename portage-snapshot-new.sfs to portage-snapshot.sfs, and run mount-portage.sh. If all of this went smoothly, you have a fully-functional copy of the portage tree at /usr/portage. If so, run umount-portage.sh, wipe the portage tree, and mount the unioned portage tree again. Congratulations, you have a few hundred MB extra disk space! To update the SquashFS image, just run make-portage-snapshot.sh again, unmount portage, and then replace the old snapshot with the new snapshot and remount.

For bonus points, you can drop the ChangeLog files from the tree, since portage doesn't actually use them - they're just there because they're sometimes useful. Add the following line to your make.conf:

PORTAGE_RSYNC_EXTRA_OPTS="--exclude ChangeLog --delete-excluded"

and then sync your portage tree again.

Moving stuff to /tmp

One thing that kind of sucks about Gentoo is having to keep an eye on /usr/portage/distfiles and /var/tmp/portage. I would guess that a lot of users don't even know that you have to, which ends up leading to a lot of wasted disk space. I solved this by moving both of them to /tmp, which (if you have WIPE_TMP enabled in /etc/conf.d/bootmisc) is wiped automatically at every reboot. The following two lines from my make.conf are the only configuration necessary:

PORTAGE_TMPDIR="/tmp"

DISTDIR="/tmp/portage-distfiles"

Another benefit of this: I'm one step closer to being able to mount /usr read-only! Which I'll get around to doing someday, I swear. >_>

Mounting by UUID

Know what sucks? Getting a new hard drive, or a new version of udev, and suddenly having all your disk devices under different names. None of your filesystems will mount! You have to go through them and fix all the entries in /etc/fstab! Oh noes!

Luckily there's a solution: you can list your filesystem UUIDs in fstab instead of your device names. Let's start with this:

/dev/hdb1 /home ext4 defaults,noatime,data=ordered 0 0

You can find the filesystem UUID like so:

$ ls -l /dev/disk/by-uuid

total 0

lrwxrwxrwx 1 root root 10 Jan 17 15:24 0f36c009-feff-4271-866d-a8796f5c92a7 -> ../../hdc2

lrwxrwxrwx 1 root root 10 Jan 17 15:24 1cb3076f-94ee-4a02-ab01-6186aaeb429a -> ../../sda1

lrwxrwxrwx 1 root root 21 Jan 17 15:24 43dd79e6-af80-4791-938b-0a53f4d6d645 -> ../../hdb1

lrwxrwxrwx 1 root root 20 Jan 17 15:24 902c0d0b-b184-4571-90c3-246f6664e7cc -> ../../hdb2

lrwxrwxrwx 1 root root 10 Jan 17 15:24 cbdcbf38-c8f3-4b32-82f1-26ebb7245273 -> ../../sdb1

We're looking for hdb1, so the UUID for the filesystem is 43dd79e6-af80-4791-938b-0a53f4d6d645. Then we can slap that into the fstab:

UUID=43dd79e6-af80-4791-938b-0a53f4d6d645 /home ext4 defaults,noatime,data=ordered 0 0

...and the system will find that filesystem no matter what the drive it's on is called.

Alternate keyboard layout

I use the Colemak keyboard layout, and switching was kind of a pain. :( You have to configure the console and X separately.

Console: Download this tarball, grab colemak-1.0/linux_console/colemak.iso15.kmap from it, gzip it, and drop it in /usr/share/keymaps/i386/colemak/ (which you'll have to create). Then, set:

keymap="colemak"

in /etc/conf.d/keymaps.

X: It used to be enough to set "XkbLayout" "us(colemak)" in the input section of your xorg.conf, but then they went and rewrote the configuration for Xorg completely. Awesome! For Xorg with HAL, save this file as /etc/hal/fdi/policy/10-x11-input.fdi, and you should be good.

Apparently, they're rewriting the configuration system from scratch again for the next version of Xorg. I can't wait to see what I have to do to be able to type with Xorg+udev. :-/

Default applications

There are two complementary approaches I'm experimenting with right now for default applications, because I don't especially trust gnome or KDE to get it right.

First: I have a symlink in /usr/local/bin that points to my current browser, and an eselect script which manages that symlink. I can then point all my applications at "browser" as the default browser, and control it pretty easily. Next best thing to the OS actually having some way to set a default browser built-in. This is pretty similar to debian's /etc/alternatives thing... actually, hmm, it's basically identical. I should look into this. >_>

Second: I've written a tool which can automatically find a program that supports a given file, and open it. I'm still not terribly fond of the configuration system for it, but it does work, and if I set it as the handler for a given file type in all my programs, I can change the program used to open that type in a single place, instead of doing it everywhere.

Automatically mounting external drives

Last I checked, there's no good way to do this that doesn't involve using gnome or KDE. halevt is a mostly acceptable solution, but it has some annoying quirks, like absolutely refusing to mount drives that are listed in fstab. Not only that, it's doomed in the long run, thanks to the inexplicable and premature deprecation of HAL. (2006: "Everybody, use HAL! It will make everything awesome!" 2009: "Guys, it turns out HAL is terrible! Stop using it immediately! Also, we've got this new thing called udev - it'll make everything awesome!")

Actually, I'm not even sure why I'm mentioning this, except to be bitter. Thanks, Gnome and KDE, for your tremendous contributions to ~~Linux~~ ~~UNIX~~ ~~free software~~ Gnome and KDE!

How Not to be Evil

2010-01-11T13:49:00.000-08:00

"Don't be evil." Words to live by, especially if you're a giant corporation like Google, which has more potential to do evil than perhaps any other organization in history. They have become pervasive over the past decade - at the moment, I am blogging on a Google service, with another one (GMail) open in another tab; my browser (like most browsers today) has a Google search box built right in at the top, and if I wanted to I could switch to Chrome, a Google-branded browser, in a few seconds. And that's just what I see right now in front of me. They're also pervasive in the search and advertising spaces - the vast majority of Internet users use Google first and foremost for search, and I believe the majority of ads shown on the web are from Google as well.

Once upon a time, Google made a reasonable effort not to abuse their position of influence too much. Sure, they show you ads when you search, but that's understood to be an implicit transaction - the ads pay for the search service. Sure, GMail reads your email to target ads at you, but users understood that that was the whole reason Google was running a mail service in the first place when they chose to use GMail. Whenever Google did something like that, there was a reasonable enough explanation. Notably, they wouldn't give themselves free advertising, and abuse their position as the owner of this massive ad network.

Recently, though, that's changed. It may have started with Chrome, it may have started earlier - I don't really know - but these days Google advertises their own stuff on the search page. When Chrome was first released, there were big ads on the Google home page urging you to install Chrome if you hadn't already. This has a tremendous effect on the market - just recently, Chrome overtook both Safari and Opera, which have been around for far longer. It's now the third most-used browser, behind only Firefox and Internet Explorer. It turns out that, in a shocking turn of events, if Google tells everybody that uses Google to install something, a lot of them will do it.

Today, I'm looking at an ad for the Nexus One, the long-awaited Googlephone, or "gPhone" to those that have a lingering iPhone fixation. It's a significant device, in that it's Google's attempt to tweak the collective nose of the cell phone industry in the US, and introduce people to the benefits of buying unlocked phones. That's still no excuse for giving it ad space - for free - that any company in the world would sell their souls for. Used to be that the most change you would see on the Google homepage was an artistic logo for special occasions, but apparently, that's changed.

Google is usually a benevolent corporate overlord, as corporate overlords go, but this sort of advertising raises all kinds of ethical questions. They are in a unique position as a company which controls online advertising, and also sells many other services, and they should realize that they've crossed some sort of line.

The Mind-takers

2009-12-26T22:35:00.000-08:00

If I were to tell you that there was a group, funded by billions of dollars anually, supported by every major corporation in the world, whose sole purpose was to convince you to think a certain way, you'd be a little bit worried, right? What if I then told you that, instead of eliminating this activity, governments of the world merely seek to regulate it, so that beyond outright lying they can use any means necessary to bring you around to their point of view? They have wormed their way onto nearly every page on the Internet, sometimes even altering the text of the page itself, in order to bend you to their will. They not only appear in every form of mass media; in a very real sense they control all forms of mass media. They aggressively target children of any age. They use whatever means they can to stake out some territory inside your mind, and leave their message there, so that it appears again when they want it to, like a posthypnotic suggestion.

At some point while you were reading the preceding paragraph, you probably realized that I was talking about the ad industry, and a switch flipped in your head. "Oh," you thought, "he's talking about the ad industry, they're actually harmless! I will read the rest of this paragraph with that in mind, and appreciate the joke instead of being worried." Frankly, it frightens me a little, how willing everybody is to trust their impression of an industry that spends billions of dollars every single year with the goal of manipulating our impressions.

The Attention Economy

It seems pretty clear that people's attention has value - it's the premise that the entire ad industry is founded on, really. If attention has value, we should be able to treat it as property; it belongs to each of us. It can be thought of as similar to intellectual property, since it's an inherent product of our minds. When you have people's attention, you can put ideas in their heads, and this is immensely valuable if you have a cause (or a product) you'd like to spread around.

(Why don't we spend every waking moment taking in messages from others, giving away our attention to all who seek it? Because attention has value to us individually. To actually do things, and not just passively absorb things that others have done, we need to reserve some of our attention for ourselves.)

Attention can be sold. Every time you turn on the television, you're engaging in an economic transaction: In exchange for letting me watch Hugh Laurie cure people's ills with sarcasm, I agree to give up X number of minutes of my attention to watch these advertisements which are conveniently interspersed within the show. This commercial arrangement (pun unintended) was forced on us, so most of us don't really take it seriously - we ignore ads most of the time, or switch channels, or edit them out entirely if we have a good DVR. The idea is still there, though, even when we're not holding up our end of the bargain.

Attention can also be stolen. Take billboards, for example - you're not getting a damn thing in exchange for having to look at billboards during your morning commute. They've taken your property without your freely-given consent. Why do they get away with this? People don't generally consider their own attention to be their property, so forcing them to sit through commercial messages against their will is usually seen to be at most an annoyance.

Without realizing it, we've developed ways to get discounts on the attention we are charged. When commercials come on TV, we change the channel, and escape without paying attention. (Broadcasters see this as shoplifting, and would love to prevent you from changing the channel during commercials like DVDs already do - but they know how well that would be received. Pun unintended.) On the Web, our eyes slide right past increasingly distracting ads designed to hold our attention hostage - we've learned how not to pay them any attention. When we receive spam, we go several steps further, and have complex technological systems designed with the sole purpose of classifying and deleting spam before we even have to look at it. Marginal attention cost: zero.

Do Marketers Dream of Hypnotic Sheep?

(Yeah, that's not how hypnotic is used, but I really wanted it to rhyme :( )

Targeted advertising is incredibly primitive today. Marketers target incredibly broad categories, and while that level of targeting produces some results for them, it's laughable compared to what they could be doing. We know that they have access to increasingly comprehensive and diverse data about each and every one of us, and it really is worrisome. Today, though, given the sorry state of the art in targeted advertising, we don't have anything to worry about yet.

Yet.

There are really three pieces here. They need raw data to start with, but they already have that coming out their ears, with the advent of electronic point-of-sale systems, and tracking cookies on the Internet, and pervasive CCTV cameras, and RFID tags, and any number of other new technologies. They need computer scientists (specifically, data mining specialists) to sift through that data, tease out correlations and useful facts, and find out everything there is to know about each of their consumers. They're making progress on this one. Finally, they need psychologists, to optimize their marketing so as to maximize their chance of influencing their targets. I don't know what the state of the art is here, but I sincerely hope it's nowhere interesting.

Because the psychological side of this could get very interesting indeed. There's already significant research that's been done on decision making, and on obedience, and on hypnosis, and on other tricks you can play with the human mind that I don't even know about. What if some enterprising young psychologist combined all that into a predictive model of the mind, where you could try out different inputs, and figure out how a person would respond to them, based on an existing profile of the sort we already have? It would be the next best thing to mind control, albeit for an individual.

Let's call this the manufactured meme. It's not a matter of if, but when - someday, people will be able to design viral ideas, and deploy them into an unsuspecting society. Viral advertising and catchy commercial jingles are the beginnings of this. Advertising is trying to evolve into meme manufacturing at this very moment, and the only thing holding it back is the limitation placed on it by our current understanding of the human mind.

The fear I have about advertising - the fear that makes me think we should dismantle the entire industry, while there's still time - is that they will connect the dots, and that they will pull in the specialists they need to sift through the mountains of data that they're collecting, and the psychologists to direct the use of that data and figure out what makes people obey, and they will create the most perfect form of population control that has ever been conceived of by man. Only a fool could think that, after creating a system that can influence populations with a high degree of accuracy, those in power would use it only for good.

History has repeatedly shown that, once a tool exists, it will eventually find its way into the worst possible hands. In this case, the worst hands I can imagine are those of a totalitarian government, or a corporate fiefdom. 1984 tried to predict exactly this, and Brave New World came somewhat closer to what's possible, but both these books depict some fairly coarse-grained population control. The possibilities that manufactured memes and individually-targeted advertising present are far more subtle and terrifying. Imagine an ad for a political candidate that changes what it says to be as convincing as possible to each person that looks at it. The candidate's actual views are irrelevant - as we've learned repeatedly, it doesn't matter a whole lot what they promise during an election. Imagine a government that enforced conformity by turning people against their neighbors when those neighbors didn't toe the party line. Imagine untraceable targeted assassinations through advertising - remember the killer Pokemon episode, that induced seizures? An ad platform that can show everybody a different message could do that to a select group for next to no cost. (Okay, that last one is a little farfetch'd. Pun intended this time.)

This is not science fiction. It may sound farfetched now, but I don't believe we're all that many breakthroughs away from this becoming reality. And it's not a reality I want any part of.

Late-night braindump!

2009-12-14T00:54:00.000-08:00

Guys! We've totally missed the real potential with ChromeOS!

It's got potential as a netbook OS, sure. But, beyond that, appliance computing - embedding the OS in the hardware is a huge deal! Especially if it's stateless. Put two copies of the OS on separate flash drives, glue them in, reduce the moving parts (how many do you really need, anyway? OLPC got it down to 0, and something like 3 internal plugs), and you've got a 100% self-contained system that never needs maintenance! As long as we're borrowing tricks from OLPC, you could put a battery on it and have it portable too! Add a hand-crank and you've got truly mobile computing. Everything else along this line of thinking is just implementation detail, but yeah!

An interface (monitor, input devices, whatev) would be nice, but even nicer would be a system that's implicitly based on clustering. You'd take two or three or four of these, plug them into a switch, plug a monitor and a hard drive or two into that same switch, and you've got a computer that's resistant against hardware faults and completely network-based. You could upgrade it just by plugging new components in and... that would be it, yeah! Probably wouldn't even need a restart. We already have the foundations for zero-configuration networking, so this would be the easy part.

Oh hey! as long as it's network based, you could add multiple user terminals and have several people using a shared cluster with full access to all resources. I think most of the problems here are solved; the interesting one I see is networked storage - you'd need encryption, because we have to assume that anybody can see any hard drive, and hmm. That might actually be it. Obviously you'd need a way to provide a private key - maybe adding a USB port to the input devices and plugging in a flash drive. Idea! The input device could be a thin client with a custom interface, which you could carry around and plug in to any cluster. That would take a lot of the burden off the OS on the cluster nodes (which would have to be pretty stripped-down and generic to begin with) and oooh. That'd be neat.

Problem: untrusted clusters. Crypto-computation might be impossible (I have seen some laughably bad research attempting to solve it), so we might have to assume that any given node can potentially snoop on computations and alter them. It might be possible to finagle something with a signed system on the cluster nodes, but they'd be too much of a high-value target to trust that some other kind of manipulation wasn't going on. Trust is a fundamentally hard problem, let's ignore it for now and work around it. Either we implicitly trust the cluster (boring, stupid) or we figure out a way to run computations on untrusted nodes so that - dammit, back at crypto-computation. :[

If Light Peak lives up to its potential, it would be kickass for this. Everything would only need one plug. Problem: fiber is a lot less resilient than CAT-5. Ah, well, not like the layer-1 matters too much.

Appliances require maintenance sometimes! What we really want here is a computer that you can embed in the wall and forget about, and have as a seamless part of the network doing useful stuff, and be reasonably confident that it'll outlast the building. I can't think of a name for this right now, but it'd be pretty cool. Transparent computing? Ubiquitous computing? Ooooh! As long as we're sticking these in the walls, we could create a wireless mesh network with the neighbors and if this actually becomes widespread it'll be usable for stuff. Routing in mesh networks is hard and really outside the scope of this braindump but the potential payoff is too cool for me to ignore.

Anyway, back to transparent networked storage! I am imagining an interface where you can just dump data onto a network, and "pull" all data owned by you back onto your own system when you leave the network. This would be at least a billion times nicer than our current home network storage metaphor, which is complete crap. Network storage shows up as hard drives, which sometimes vanish, and may or may not be accessible based on remote configurations, which can change at any time, and concurrent access is a bad joke, and seriously who came up with this ridiculousness? But I digress. We want all storage to be accessed in a unified way, and always writable would be nice - a reciprocal protocol would guard against people just hogging all the storage, but dunno how that'd work yet. Future braindump topic! Maybe require people to provide all the storage they actually use, but distribute the data around the network so they get the benefit of insurance against disk failure.

(For any of this to be feasible, we have to be able to build 100% secure operating systems. Attention, programmers everywhere! Stop sucking. :( )

If we could set this up in such a way that you could charge people for using resources on your nodes, we'd have a completely distributed clone of EC2. Cool! (Oh, hey, that'd work for storage too. Two birds with one distributed ripoff! :D)

Back to the software side: to share compute resources, we'd need a virtual machine and a stable OS API so that different computers can use the cluster. There are a few good choices right now (JVM and CLR mainly, and LLVM is looking good). Randomly: it'd be nice to have a VM that supported distributed execution of a single program. Might get messy with failures -- no, wait, the VM should handle failures! And the program should just see a single (highly parallel) machine that never fails! Gosh, that'd be neat.

Years of utterly worthless computer security have left us in a state where we can't even pass a file between two computers without bypassing six different "security features" designed to keep computers from ever communicating, just in case one side or the other is infected with a virus. This is pathetic! Hardware is improving at an incredible rate, but that hardly matters when software is completely and pathologically unable to keep up. The true cost isn't that some things are harder than they should be; sending files at all is a parlor trick. The true cost of poor security is all the cool stuff we could be doing if we didn't have to treat every damn network service as an attack vector.

Technology is pretty neat and all, but sometimes I wonder that we don't fully appreciate how primitive this all really is.

Goodbye Twitter

2009-12-12T22:53:00.000-08:00

It's been about eight months since I decided I'd start microblogging. How has it turned out? Actually, you can probably guess from the title that I'm ditching Twitter and just sticking with Identica.

So what's the difference? I started with both of them basically cold, knowing very few people on each. Twitter has the advantage of being a fad right now, so all sorts of people are using it. It's nearly achieved ubiquity; the API is simple enough that you can integrate Twitter support into just about anything. Identica, on the other hand, has some really, really nice features:

* Groups: Twitter has hashtags, where you can tag a notice as being about a particular topic. Groups take that to the next level, and let you subscribe to a topic. Twitterers: imagine being able to follow a hashtag and you'll understand. This makes it tremendously easier to get started with Identica. Instead of having to find interesting people to follow, you can subscribe to a group that interests you, and find the people you'd like to follow as they post interesting stuff to the group. Or, another way they're useful: on Twitter, if you want to ask a question, you have to either address it to somebody you know is knowledgeable (and hope they're around), or ask the world in general, and hope that somebody following you knows the answer. With Identica, you can address the question to a group (or even multiple groups), and get a good answer much more easily.

* XMPP: In all fairness, Identica's XMPP support doesn't always work (it tends to lag during upgrades, for instance), but when it's working, it's a thing of beauty. People sometimes describe Twitter as realtime, but I'm pretty sure that's just because they've lowered their standards; the latency is still measured in minutes even under ideal conditions. Using XMPP on Identica, I can use the site through my existing IM client (pidgin, yaaay), and the latency is measured in seconds. That's getting down to the speed of instant messaging - you can have actual realtime conversations with people on Identica through a chat client! Yeah, it's pretty badass.

* Context: Twitter includes a link on tweets to the tweet you're replying to. That's cute, Twitter! Identica has a similar link on updates, only instead of showing you the update it's replying to, it shows you the entire threaded conversation. Seriously, head over to my profile, and try one of the "in context" links. It's not perfect, since it's not always possible to infer which conversation an update is part of, but it works something like 90% of the time.

If I had to try to put my finger on the biggest difference between Identica and Twitter, it'd be this: Twitter seems to be focused on people, while Identica is focused on conversations. Twitter's social dynamics are interesting in an abstract, six-degrees-case-study sort of way ("how far can we get using just the social graph?"), but it seems to be remarkably resistant to the formation of communities. Conventions like retweets and Follow Fridays help here, but they don't exactly constitute a solution. Instead, they just serve to outline Twitter's real problems.

So farewell, Twitter. It's been fun, but Identica is just more interesting.

Against Sleep

2009-11-30T01:35:00.000-08:00

I don't dislike sleep itself.

Right now I should be going to sleep, but I'm not because for some reason, some nights, the apex of my alertness for the entire day hits as soon as my head hits the pillow. Tonight already feels like one of those nights where I lie awake for hours waiting for my brain to shut itself off so I can stop staring at my clock and watching the hours tick by. And every ten minutes I remember something else I could be doing instead of waiting for sleep (like writing this blog post) and I either do it, or it bounces around inside my skull for the next half hour until I remember something more important that I've forgotten to do.

What's worse than the nights, though, is the mornings. I always feel incredibly groggy in the mornings, sometimes (rarely, luckily) to the point that I can't even get myself out of bed - and that's when I've had enough sleep. When I don't get enough sleep, it feels like microscopic ninjas have snuck under my skin while I was sleeping and sprinkled a layer of fine grit right under my epidermis. If I could switch the way I feel in the morning with the way I feel in the middle of the night, I might not be a happier person, but I'd probably at least be a lot more balanced.

And in between these two wonderfully inverted polar opposites, I dream. I'm really glad I don't remember the vast majority of my dreams, because based on the ones I do remember, I apparently dream in low-budget sci-fi horror flicks. Let's look at some recent dreams that I remember. In the most recent one, I had the power to travel back in time and redo parts of my life, but I was hunted and repeatedly killed by somebody with a similar power. In another recent dream, I die in some sort of zombie outbreak caused by a mind-controlling fungus. I wake up decades later after it revives me for some reason; it's taken over the world and nobody realizes it (because the fungus altered everybody's memories, natch). It's never bright and happy dreams, it's always stuff that freaks me the hell out.

So anyway, I've got nothing against sleep, really. It's everything associated with it that I can't stand.

Decentralization VII: Definitions

2009-11-29T21:59:00.013-08:00

I started this series of posts to try and get a better handle on what decentralization is, what it's good for, and what general situations require it. The level of centralization is an important characteristic of systems, but people don't always understand the tradeoffs that exist there. I think, at this point, that it all comes down to the division of control.

The definition I think I'm going to use: A system is decentralized in a certain aspect if control over that aspect is divided up over the components of the system. It's too vague to say that a system is a decentralized system without specifying what aspect you're talking about. Systems can be centralized in some aspects and decentralized in others, and in fact I think nearly all systems are a bit of both.

Mini case study: Blogspot has centralized ownership (Google) and code, along with centralized authentication (Mostly Google accounts, I think?) and centralized execution (runs on Google's servers). On the other hand, it has decentralized authorship (contrast with a newspaper or something like that), and participates in several decentralized protocols (linkbacks, RSS, etc). Contrast with Wordpress, which optionally has decentralized authentication and execution (you can run it on your own servers), and somewhat decentralized (open-source) code. Because of this, we can say that, broadly speaking, Wordpress is less centralized than Blogspot.

Systems can vary in the degree that they're decentralized. Federated systems, which I've mentioned before, have a set of independent servers which are decentralized, but clients connecting to the system don't have to worry about this and can just pretend they're connecting to a centralized system. (Contrast with p2p networks like Gnutella, where every user on the network is simultaneously a client and a server.) Once we understand decentralization as a division of control, it becomes clearer what's going on here: control is being divided among only the participants in the system that want to run servers, and thus reflects the reality that only a few users will actually want the responsibility that comes with that control.

Can we divide up control in more complex ways? Yes, but there's not a whole lot of benefit to it, and it increases the complexity of the system significantly. Simplicity is inherently valuable when it comes to code; complexity is usually a problem to be eliminated. I can't think of any computer systems right now that have more complex control hierarchies than federated systems - they almost certainly exist, but they're not common.

So what are the tradeoffs? Control is important in systems for both technical and business reasons, which aren't always the same. Twitter would definitely be more reliable if they made it more decentralized (more on robustness in a bit), but it would also impact their (nebulous) ability to turn a profit if they gave up control.

Why are decentralized systems more robust? There are a lot of factors that come into play here, really, so I don't have a clean answer yet. Let's look at some failure modes of systems. System overload is less of a problem in decentralized systems, because they're necessarily designed in more scalable ways, which makes it easier to throw hardware at the problem until it goes away. Hardware failure is also less of a problem, because systems that execute in a decentralized manner can be made immune to individual pieces of hardware failing more easily, Software bugs are less harmful in systems with decentralized codebases, because different implementations of the same protocols rarely have the same bugs.

Maybe this is the answer: centralization implies control, and control implies an opportunity for the one in control to make a mistake. If we define robustness as the number of mistakes or hardware failures that would have to occur to take the whole system down, or better yet the probability of a sufficient number of mistakes happening, it's easy to see why decentralized systems are robust. Thought experiment: if we had a system that was decentralized in such a way that any failure would break the system, we would have a system where decentralization made it far more fragile, rather than robust. Luckily, you'd have to be crazy to design something like that.

This is the last post in this series, I think - I've found what I was looking for.

Climategate

2009-11-28T21:15:00.001-08:00

Is it just me, or is it kind of ridiculous the way every scandal in American politics is eventually referred to as something-gate?

For those of you that don't keep up with every little thing to happen in climate politics, somebody hacked into the servers of a climate research group the other week, and released an archive containing selected portions of the past 10 years of their email correspondence, and probably some other stuff too. Naturally, people were able to find all sorts of embarrassing stuff in there. The most talked-about email is one referring to a "trick" somebody used to combine two data sets, back in 1999 - this obviously proves that the entire field of climate research is a hoax, to hear people talk about it.

I won't be talking about any of the specifics of the leaked data, though, because I can't look at the source data in good conscience. Breaking into somebody's servers and putting over a decade of correspondence online for somebody's rivals to pore over is beyond reprehensible. It's especially bad in this case, when the rivals in question are playing politics and have very low standards for proof.

One particular issue that keeps cropping up is the removal of context. People will go and cherry-pick quotes (here is a recent and out-of-mainstream example), and somebody will eventually point out that the code in question only applied to one graph that was only used for "cover art". It's easy, when you're looking at correspondence over a span of many years, to find quotes that look bad out of context - but what does that actually prove?

Let's be miserly to the researchers here. Let's assume that some of these quotes really are as bad as they look out-of-context, that there really is a massive conspiracy to rewrite the data, which is somehow only evident in a few snippets. Even assuming the worst, what does this prove? This is only correspondence for a single institution, which is competing with several other groups - and yet the results all still point in the same direction. Should we posit that every group working on climate science is in cahoots, and engaged in a massive conspiracy? What scares me about this is that many people would say that we should.

Maybe I'm wrong here, but here's the impression I get watching the climate change debate. First, there are serious researchers, who (generally) agree that climate change is probably happening, and do actual work to figure out the extent of it. Second, there are the "green" cheerleaders on the political left, who say mostly inane, common-sensical things about global warming, and push for specific political/technical solutions (looking at you, Al Gore). Their hearts are generally in the right places, except for the ones that are only on this team because they've found a way to make a profit (I'm watching you, Al Gore).

Then there's the anti-global-warming crowd, made up of both right-wing political types and people reacting against the second group, who seem content to take random potshots at the first two groups indiscriminately, cry about conspiracies, and (like all conspiracy theorists) don't really care about having a consistent response to the people doing actual science. It's telling that there are almost no climate scientists in this group. (Members of this group will respond: "because of the conspiracy!", and thus prove my point.) This group is especially irritating to me, because they tend to be very anti-scientist.

Seriously, though - hacking people maliciously is bad enough, but calling it Whatevergate is just awful.

My room is not my room

2009-11-27T21:58:00.005-08:00

In the tried and true tradition of people trying to come up with something under a deadline, I'm going to look around and write about the first thing that comes to mind.

I'm staying at home this weekend 'cos of Thanksgiving, and I'm realizing that my room no longer really feels like my room. Everything in the room, with the exception of a floppy red hat that I picked up this summer, is exactly as it was when I left a little over three years ago. Thanks to the ruthless forces of natural selection, everything I really care about has moved with me to my apartment, and everything else (a lot of plastic dinosaurs, for example) is stuff I simply don't need. I suppose you could say that my stuff has evolved.

After a series of ruthless throwings-away, just about everything is gone. Just like with archaeology, the only bits that are left from the 12-year or so span are the ones that happened to land or get stuck in favorable spots. Here's a jigsaw puzzle that's a National Geographic cover; we thought it looked neat so we framed it. I used to do huge jigsaw puzzles all the time with my mom, I wonder when that tradition died out. I kind of miss that.

Next to it is a set of collapsible shelves. I was never content with just having ordinary shelves, so I put them together in some weird way, with varying heights. The whole thing is kind of rickety since it was never supposed to work this way, but I kind of like it. I think it looks better this way, really - it's a pretty nondescript piece of white plastic furniture. There's a brown plastic Tyrannosaurus on the shelf. I've had it for as long as I can remember.

I should mention that all the posters on the wall are tilted by about 10-15 degrees. During some nonconformist phase (back in middle school, I think?) I went and made everything off-center, because I liked the way it looked. I probably would have done the furniture that way too, if it'd been feasible. They're all basically as they were when I left; the only poster I really cared to bring with me to college was a six- or seven-foot-tall picture of a Shuttle launch (it is pretty awesome).

Something that I'm realizing as I look around. The summer after eighth grade is when I got my own computer, and it's also when I really stopped caring what my room looked like, since I was spending a lot more time online and a lot less time in my room. There's almost nothing in here that reflects me after I started high school. Maybe that's part of the reason the room feels so alien to me now - it's largely still my room from eighth grade. That's kind of a depressing thing to realize. :/

At some point (probably next May) I'm going to have to move out of this room properly. I'll take down all the posters, clean out all the drawers, throw out whatever I don't want to bring with me, and officially return the room to its status as a guest bedroom. After that, I'll come back and visit this house, but I'm never going to live here again. That's another depressing thing to realize, but one that I'm going to try to come to terms with before it becomes painfully relevant in about six months. I've already started to refer to my apartment as home; I think that's a good first step.

Taking Back Chrome OS

2009-11-25T22:00:00.000-08:00

Chrome OS is a pretty interesting platform, all the more so before it's open source. It's pretty strongly tied to Google right now, but there's nothing stopping me or anybody else from taking out all the Google-specific bits, and building up something new in their place. That's the interesting thing about open source - rather than being controlled by whoever wrote the software, it can go in directions that the creators may not have anticipated, and may not want.

The obvious first change would be to switch out the default browser - Firefox OS? Opera OS? This would depend on putting the X stack back into Chrome OS, since if I recall correctly Google dropped that. It's definitely possible, though. I think the harder part would be making the necessary modifications to the browser to allow it to be used as the sole interface to the system. Chrome can apparently view folders, and supports pinning web apps to a menu; I don't know if other browsers would want to do things exactly the same way, but they'd have to do something similar.

We could also switch in other non-browser applications. The command-line addict in me would love to have an OS that boots straight to a command line, and can connect to the internet so I can ssh. Having that on a netbook that I can carry around would be insanely useful. Application vendors could also create single-application systems as demos (and if they do, people will be trying out their apps running on Linux!).

Some people are going to ask what the point is. We already have much more capable Linux distros that can do the same thing, so what's the point? The interesting thing that Chrome OS adds is completely safe, transparent upgrades, and a read-only root. This has two humongous implications for the end-user: they don't have to worry about software upgrades ever, because the system auto-updates, and they can always reboot the system to get it back to a known-good state. Yes, this also means that you lose a lot of flexibility with the system; it's a tradeoff, and in many cases, the benefits of this outweigh the costs.

The core interesting idea of Chrome OS that we can use is the idea of "appliance computing". Looking at it a certain way, the OS really doesn't matter at all - what the user cares about is the application. Chrome OS, minus Chrome, gives us the necessary foundation to build completely stripped-down single-application systems. Obviously, this isn't going to be useful for everybody in all cases, but it's a pretty interesting idea to play around with.

I think the first step should be factoring out the "OS" part of Chrome OS. It'd be useful to have a core OS part, which automatically updates in the background, and an application part, as an image which is overlaid on the OS and contains the application. This way, there could be a single shared source for OS upgrades for all the different single-app systems. It would also make it pretty easy to make your own computing appliance, since you'd just have to package up the necessary files. There would need to be a dependency system so that version mismatches between the two parts don't cause problems, but this is a problem that Linux distros have explored pretty thoroughly.

This is kind of drifting into some ideas I've had about new ways that distros could be structured, but that's a whole other topic for a whole other post. ;)

Free electronic stuff!

2009-11-25T15:12:00.003-08:00

This is going to be kind of awesome.

It's a really interesting idea for a promotion, actually - everybody loves free stuff, so the news that this is happening is going to be all over the place by January 7th. I, for one, am going to start planning a shopping list for that day well in advance, just in case I actually get the free stuff (they're capping the giveaway at $100k, which will probably dry up really quickly). If I don't get it, I might actually still buy the stuff - it depends on how I feel about whatever project I come up with.

I really want to build some kind of flying robot, dirigible-style. No idea how that's going to work, yet - ideally it'd be able to float mostly unpowered, so I'd either have to buy a lot of helium, or learn how electrolysis works (hydrogen rocks! :D). That's my tentative plan, anyway. We'll see how long it lasts. No matter what, I'm going to have some kind of plan come January 7th.

This sort of promotion ("hey, guys! free stuff!") would be really neat if more retailers started doing it, actually. Imagine if Newegg or Amazon just announced that 1% of orders on the site would be totally free. People really love lotteries, as history has shown, so this could be really effective. If I'm choosing between a deal offered on two sites, and one of them has a chance of giving me the order for free, I know who I'm buying from.

Anyway, short post today, because Thanksgiving is tomorrow! I've made cornbread and cranberry sauce so far. :D

Terrorists we need to watch out for

2009-11-24T21:29:00.006-08:00

With the terror alert level pegged at Yellow (or Orange, if you're flying), and thousands of American troops in the Middle East fighting "terrorists", I think it's important that we know who the enemy is. Here are some prominent terrorists of the past few years:

* Richard Reid, the Shoe Bomber - Despite the stringent security measures in place at airports, Reid did the unthinkable, and managed to smuggle a bomb aboard an airplane. After being admonished once by a flight attendant for lighting a match, he was later found struggling with a fuse, another match, and one of his shoes. Next time you have to take your shoes off at the airport, remember this: every air traveler has had to undergo this inconvenience for several years now, because of a terrorist that couldn't detonate his own bomb.

* Gary McKinnon, Super-hacker - The US government is currently trying to extradite this Extremely Dangerous man from the UK. McKinnon broke into nearly a hundred military computer systems during 2001 and 2002, using advanced hacker techniques such as guessing default or blank passwords. He used his access to top-secret computer systems to delete some files, and search for information on alien life, which he believe NASA and the US government were hiding. Gary McKinnon is one of the most credible cyber-terrorists discovered to date.

* Iyman Faris - Suspected of plotting to destroy the Brooklyn Bridge. This would have been a tremendous disaster, on several levels. Luckily, the plot involved cutting the bridge's support cables with an acetylene torch, which would have taken some pretty serious time and effort, since the cables are necessarily pretty sturdy, and consist of thousands of individual wires.

* Najibullah Zazi - One of the most recent terrorists uncovered in the US, Zazi received terrorist training directly from al-Qaeda, including instructions on bomb making. Had he managed to construct a bomb, he could have done some serious damage. Unfortunately for him, he failed to build a bomb despite nearly a year of repeated attempts. Nevertheless, this is hailed by some as the most serious terrorist plot uncovered in the US since September 11th.

* The Fort Dix attackers - Their entire plan was to show up at a military base and kill as many people as possible. This could have been a really serious plot, if they'd managed to acquire the weapons the plot required. However, they made a few fatal blunders: they videotaped themselves firing automatic weapons and talking about the plot, and then sent the footage to a Circuit City to get it made into a DVD. They also attempted to buy their weapons from an FBI informant.

* Star Simpson - While picking up a friend at the airport while wearing a homemade nametag with blinking lights, Simpson was arrested by officers carrying submachine guns. For a few days, the media was in a frenzy, gleefully reporting it as a threat with a "fake bomb". They stopped talking about the situation, naturally without any kind of retraction, once it became clear that she wasn't actually any kind of threat. I feel safer already!

So remember, folks: terrorists are people you need to be afraid of. The next time you see a major terrorist plot being reported on the news, pay attention to the details - given the media's track record in this area, it's far more likely that they alleged terrorist is incompetent, insane, or just plain innocent.

Cop-out post again

2009-11-23T21:45:00.000-08:00

No time to write a proper post today :( Instead, I'll just point you to MLIA - hopefully it is enough of a distraction from this fail post >_>

Monad documentation hate

2009-11-22T19:29:00.004-08:00

Monads are a useful construct in Haskell. Even though it's a pure-functional language, monads allow you to "cheat" a little, or at least pretend to, and carry state through computations (or do other stuff). This is pretty important, since a truly pure functional language wouldn't be able to do useful things like generating output.

Unfortunately, just looking at the documentation that exists online about monads, I wouldn't be entirely convinced that they're not just an elaborate joke cooked up by functional programming partisans - the equivalent of a linguistic snipe hunt. Let's look at some examples.

First, the Haskell wiki article on monads, since that comes up pretty high in search results. This is pretty close to coming straight from the source; I bet they have some great, readable documentation, right?

This is the introductory image they use. The caption reads: "Representation of the Pythagorean monad". Thanks, Haskell wiki! Since you haven't yet explained what a monad is, or why there would be a Pythagorean one, this diagram means nothing at all to me. Actually, it's worse than that - having put some serious effort into learning what monads are, and actually having a basic understanding of what they do, this diagram still makes no sense to me. Awesome!

Or, let's look at this tutorial called "A Gentle Introduction to Haskell". It's the first result when I search for "haskell using monads", so it must be pretty useful, right? Unfortunately, it begins by diving straight into the "mathematical laws" that "govern" monads, whatever that means. Actually, I'll just quote it:

Mathematically, monads are governed by set of laws that should hold for the monadic operations. This idea of laws is not unique to monads: Haskell includes other operations that are governed, at least informally, by laws. For example, x /= y and not (x == y) ought to be the same for any type of values being compared.

Which is great and all, but doesn't explain a whole lot to somebody who doesn't yet know what monads are. This is not how you write introductory documentation, guys! To borrow an example from the quote, you don't introduce the == operator by saying that it follows a mathematical law. You say what it actually does first!

So what is a monad, actually? As far as my (admittedly extremely rough) understanding goes, it's the functional equivalent of a wrapper class in OO languages. You can use it to wrap up some boilerplate code that would otherwise be really irritating, in a more or less standard way. Haskell also provides some syntactic sugar in the form of "do" blocks, which you can use for some monads to make your code look like imperative code.

See, now that wasn't so hard, was it?

Building ChromeOS on Gentoo

2009-11-21T14:00:00.001-08:00

Google released the source code to Chrome OS, and there are already some disk images floating around, but we Gentoo users know that if you want it done right, you have to compile it yourself. ;) Here's how I built a VMware image on Gentoo.

The build instructions say you need Linux, but Google, as usual, seems to assume that you're running Ubuntu when you're trying to build Chrome OS. Here are the steps I took to build it, following the documentation that starts here:

(If you don't really care about how I built it, and just want the image, click here.)

1. emerge debootstrap

(debootstrap is necessary to actually build the OS, but you can leave this running in the background while you do the next few steps.)

2. wget http://src.chromium.org/svn/trunk/tools/depot_tools.tar.gz

3. tar xzvf depot_tools.tar.gz

4. export PATH=$PATH:$PWD/depot_tools

5. gclient config http://src.chromium.org/git/chromiumos.git

6. gclient sync

(wait for a really long time while it downloads the source)

7. cd chromiumos.git/src/scripts

(at this point, you'll need debootstrap installed, it'll be in /usr/sbin so let's put that in PATH)

8. export PATH=$PATH:/usr/sbin:/sbin

(if this fails, and you have to rerun it, you might need to delete the repo/ directory between runs. also, a few of these scripts will sudo, mainly to set up the mounts in the chroot; read through the script if you don't trust it, it's pretty short)

9. ./make_local_repo.sh

10. ./make_chroot.sh

11. cd ../..

(now, we download the browser. the link on the build instructions is broken, but this one should work:)

12. mkdir -p src/build/x86/local_assets

13. wget http://build.chromium.org/buildbot/continuous/linux/LATEST/chrome-linux.zip -O src/build/x86/local_assets/chrome-chromeos.zip

(now, we enter the chroot and continue with the build)

14. cd src/scripts

15. ./enter_chroot.sh

(at this point you should be in the chroot)

(grab a snack, this'll take a while)

16. ./build_platform_packages.sh

17. ./build_kernel.sh

(I ran into a conflict here: HAL on the host system somehow blocks ACPI from being installed in the chroot. Stopping hald on the host system worked around it successfully.)

18. ./build_image.sh

If you made it this far, you have an image built! Awesome.

To build a VMware image, use the image_to_vmware.sh script:

19. ./image_to_vmware.sh --from=../../src/build/images/999.999.32509.204730-a1

Note that this script requires qemu-img. You can edit the script and replace qemu-img with kvm-img if (like me) you have kvm installed but not qemu.

I haven't tried building a USB image, but it should work something like this:

20. ./image_to_usb.sh --from=../../src/build/images/999.999.32509.204730-a1 --to=/dev/sdb

So, as of right now, I'm running Chrome OS in VirtualBox. It's pretty slow, being in a VM and all; I'm going to try to get it on my Eee later, when I have more time.

First impressions:

The boot time is pretty freaking ridiculous, especially since it's running in a VM. Something like ten seconds to a login screen. Can't wait to see how it does on real hardware.

There's some kind of bug where I get certificate errors for Google sites - but only the pinned ones. The pinned Gmail tab errors, for instance, but logging into Gmail in a new tab looks fine. Other people have reported similar problems for other builds, it looks like, so I'm expecting that it'll get fixed eventually. It might have something to do with how it tries to automatically log in based on your login credentials; that's complete speculation on my part, though.

The Chrome menu that was in the demo is missing in my build. Not really sure why. :( Could be because I used a different Chrome binary than the one they listed in the install docs. Will have to try that again once they fix the link. >_>

Decentralization VI: Package Management

2009-11-20T21:30:00.000-08:00

I have issues with software installation on Linux distributions. As opposed to Windows and Mac OS X, where (for better or worse) you can download a package from a website and install it, Linux packages come through centralized repositories. There are advantages and disadvantages to this method, but Linux distros don't do it because of the advantages; they do it out of necessity. I'll explain why in a bit, but I want to get the pros and cons out of the way first.

Centralizing packaging has certain benefits from a QA standpoint. It allows distros to ensure compatibility between all the various components of the system, and enforce various standards across all the packages. It also allows them to smooth over incompatibilities between distros - if Ubuntu does things a certain way, but the person that wrote the software had Red Hat in mind when they wrote it, central packaging allows Ubuntu devs to sort that out before the software lands on people's machines. Distros also prefer to have control over packaging because there are a lot of different package formats used on Linux, and it would be kind of ridiculous if every software author out there had to support all of them. There aren't hard and fast rules about which parts of installation are distro responsibilities, but there are conventions, at least.

Distros also use centralized distribution, for the most part: when you install an Ubuntu package, you download it from an Ubuntu server, using an Ubuntu package manager. This simplifies finding and installing software, obviously. You don't have to look very far to find any given program, and you're assured that what you're installing is actually the program, and not some random virus. The organization behind the distro also has to provide servers, of course, but this isn't too much of a problem. Bandwidth is cheap these days, and for a distro of any significant size, there are plenty of people willing to donate a spare server or two.

As for the disadvantages, centralized software creates a barrier to entry. Anybody can write a program for Linux, but actually getting it into the distro repositories takes a certain amount of notoriety, which is more difficult to gain without being in the repos in the first place. The result is that there's a lot of software out there that doesn't exist in any repository. Users generally don't like to install software outside of the distro package managers, because when you do, you don't get any of the nice features (such as, oh, being able to uninstall the software) that the package manager provides.

Distributions also get saddled with the unenviable job of cataloguing every useful piece of software for Linux that people have written. This takes a huge amount of developer effort; Gentoo, for instance, has hundreds of people (the vast majority of people working on it!) dedicated to just maintaining the programs you can install. We can really take a more general lesson from this: When you try to centralize something which is, in its natural state, decentralized, it's an expensive and ongoing job.

But pros and cons aside, I said earlier that Linux distributions do this out of necessity, not just because they want to. If you write a piece of software for Windows, Microsoft has a lot of people working pretty hard to ensure that it'll work on future versions of Windows. Backwards compatibility is intricate, uninteresting work. Since Linux is written mostly by volunteers, it's exactly the sort of work that never gets done, because nobody really wants to do it. The result is that backwards compatibility on Linux is a bad joke. Developers break compatibility, often gratuitously, often without warning, and so Linux software needs to be constantly maintained or it simply ceases to function.

In an environment like that, you absolutely need somebody doing the hard work and making sure a piece of software still works every once in a while. That job falls to the distros, because the original authors of the software don't always care that it works outside of the configuration that matters to them. Look at it from the distros' perspective; if you're trying to make a coherent system, but the components of the system are prone to randomly break when you upgrade them, you need to maintain centralized control if you want to have any hope of keeping things stable. In other words, the lack of backwards compatibility on Linux forces distros to centralize software distribution, and do a lot more work than they would otherwise.

These posts are supposed to be case studies in decentralization, so I'll summarize. The difference between Linux and commercial platforms is the degree of compatibility in the system. The degree of compatibility determines the amount of control you need to make sure the system works as a coherent whole. With Linux, the need for control is much higher, so distributions are pushed towards a centralized software distribution model.

Chrome OS

2009-11-19T13:30:00.000-08:00

I just watched the live webcast announcing Google Chrome OS. I was expecting a lot from Google with this, but they've gone even beyond that; this announcement is serious business. They're talking about fundamentally changing the way people use computers.

First, the basics: Chrome OS is exactly what it sounds like. It's an operating system that boots directly into Chrome. The OS is a stripped down Debian install, but that doesn't really matter, as we'll see in a bit. Everything happens through the browser window - there's a file browser built into the browser, for instance. The start menu equivalent (of course there's one) is a Chrome logo in the top left corner of the browser. There's no desktop, no My Computer, nothing else - just Chrome.

This brings us to the first major difference between Chrome OS and other OSes. There are no applications to install; everything you could conceivably want as an application is a web application. They make this a bit easier by pinning some shortened tabs ("application tabs", they call them) at the front of the tab list, so that you have one-click access to your Gmail, for instance. Obviously, this is a pretty radical design choice. The emphasis is definitely on shifting to online services for everything, rather than using desktop applications - and, not at all coincidentally, Google has spent years polishing their online versions of desktop applications. (They showed the online version of Office during the demo, but it looked terrible. Half the screen was taken up by the controls. I can't see how it's a serious competitor to Google Docs, in its current incarnation.)

Not only does this moot one of the usual objections to Linux ("I can't run my apps on it"), it dramatically simplifies securing the system. My understanding of it is, Chrome OS has a small number of signed and/or whitelisted programs that it runs, and the system can assume that anything outside that set is a virus. This is such a fundamentally powerful idea that I'm surprised it took this long for somebody to try it out in a consumer OS. Chrome OS then takes it to the next level by signing the kernel and OS, so that there's (probably) no way at all for malicious code to get in. Their goal is for you to be able to trust that if it boots Chrome OS, it's safe to use. As for updates, it automatically updates itself in the background - this is a difficult thing to get right, but Google is more likely than most to pull it off.

Because there are no local applications, they can get away with having no local storage. This bears repeating, because again, this is pretty radical: you can't save files to the hard drive. You can use flash drives and stuff like that, which makes sense, but the OS drive itself is locked down and mounted read-only. This will be one of the more controversial decisions, I'm sure. It forces you to store all your files and settings in the "cloud", which makes migration easier, but is probably going to be kind of a pain.

I'm not totally clear on how it handles user accounts, but my impression is that you'll be able to sign in to any Chrome OS machine using any Google account, and have it work the same. This is an incredibly powerful idea! It essentially means that you're completely independent of the hardware you're using. If your computer explodes, or you spill apple juice on it, or it's stolen by pirates, or whatever - no problem, it's easy to replace. This ties in with the no-local-files thing - if all of your files are already in the cloud, then it makes sense that you'd be able to log into any Chrome OS machine and have it work the same as your own.

A word on the cloud: This is going to be a sticking point for some people. When I say the "cloud", what I really mean is Google's servers - I doubt they'll allow you to switch to other providers, if that's even feasible. There are legitimate user interface reasons for doing it this way, but it still has the potential to be a privacy nightmare, not to mention the power Google is going to have once they hold everybody's user data. Again, this is one of those things that Google can get right if anybody can; the question is whether or not anybody actually can.

Another word on the cloud: When your internet connection is down, or when you're on an airplane, or when Google's servers go down (it's happened before, it'll happen again), your Chrome OS computer is going to be basically useless. People asked about this in the Q&A, and the answers boiled down to "HTML 5 lets you use web apps offline" and "Google has better uptime than most personal computers, so nyah!" It's not really a satisfactory answer, since HTML5 offline storage is only useful for web apps that have been specifically designed to work offline. In the end, Chrome OS won't be that useful if you're someplace without free and easy Internet access.

Random thought: It's accepted among security experts that local access = root access, for a sufficiently determined attacker, for lots of reasons. Google has taken some steps to prevent that (signed OS is a big one), but it remains to be seen whether it's enough. There are a lot of really devious attacks that you can use if you have unfettered local access.

So what's the big idea?

The paradigm that Google is aiming for with this is something called "appliance computing" - treating the computer like any other appliance. You have a basic expectation that a refrigerator, for example, will just work, without requiring constant maintenance. Appliance computing is when computers are that simple to use. This is something that people have wanted for a long time, but the existing model of computation made really difficult. Designing an OS that doesn't require administration is hard, but based on the info I've read so far, it seems like Google might have pulled it off. Designing a truly secure OS is really hard, but while I'm never willing to bet against hackers, I think Google has at least done better than anybody else who's tackled this problem.

Their goal with Chrome OS is netbooks and things like that, which makes sense. While Chrome OS looks nice, I wouldn't ever use it as my primary operating system. There's a certain level of control over my systems that I prefer to have, and one of the key goals of Chrome OS is shifting most of that control (and the associated responsibility) to Google. As a spare system, on the other hand, Chrome OS will be really useful, and that's how most netbooks are used these days anyway.

Death of Advertising, round 2: personal RFP

2009-11-18T19:45:00.000-08:00

I've blogged on this topic once before, but I didn't have as much to say.

Advertising is annoying, isn't it? It sometimes feels like an overenthusiastic robot salesman that always follows you around, interjecting every few minutes about so-and-so product that people who fit your profile might enjoy. Not only is it annoying, it's remarkably inefficient - I don't care about 99.5% of the ads I see on a daily basis. This really sucks!

So, why not kill the ad industry?

I've wanted to say that for a long time, but until recently I couldn't back it up. If we want to get rid of advertising, we need to replace it with something better, otherwise nobody is going to listen. This is setting a remarkably low bar, actually: "we need to improve on a universally-reviled system." It's actually pretty surprising that nobody's managed to do this so far.

Enter Project VRM. You may have heard of CRM; VRM is basically that in reverse. They've floated the idea of a "personal RFP", the idea being that instead of being advertised to, you put something up on a website when you need something, detailing your requirements, and vendors come to you. Think of it as a reverse eBay.

This is a pretty disruptive (and therefore awesome?) idea. It could completely change the dynamic between buyers and sellers, for the better on both sides. If I'm a buyer, I don't have to spend as much time on comparison shopping - options are brought to me, instead. If I'm a seller, I don't have to waste time and money on broadcast advertising - I can just search for people who want what I'm selling, and contact them directly. I can also customize my offer to each individual, something which companies would love to do right now, but can't because broadcast advertising is such a limiting medium.

One problem on the way to adoption could be the chicken-and-egg problem. There's not really any reason for buyers to use something like this until there are sellers, and vice versa. We could get out of this by crowdsourcing comparison shopping, at least initially - having people look at requests, and finding the best deal online that matches, in exchange for a cut of the sale. This would probably even happen spontaneously, as long as the service doesn't explicitly discourage it.

Would this actually kill advertising? Maybe not, but it would certainly change the nature of it. Right now, advertising has two major functions: convincing you that you need a product, and directing you to someplace you can get it. The former goal wouldn't really be affected at all; only the latter would change. All isn't lost, though. In a perfect world, people would see the advertisements, consider them, decide they want the product... and instead of clicking on them, go to a personal RFP site, and get it that way. If this takes off in any meaningful way, we could see advertising take a huge hit.

Decentralization V: Regulating thepiratebay

2009-11-17T19:15:00.006-08:00

So I heard today that thepiratebay is shutting down their tracker for good, and shifting to using DHT exclusively. Hooray, this blog deals with current events for once!

There are plenty of arguments either way between centralized trackers versus trackerless DHT. Centralized trackers are much simpler for everybody, they allow you to keep statistics on your torrents, and they let you control who can access a torrent. DHT is more robust, since it cannot be taken down except by extraordinary measures, and cheaper, since that's one less server you need to keep running. More importantly, in this case, it also lets you spread around the legal liability, diffusely enough that the network can't be taken down by lawsuits.

Let's be honest, that's the real reason TPB is doing this. For all their talk of this being the right technical decision, and not motivated by the current lawsuits, the fact is that they're potentially facing a lot of liability because they run a tracker in addition to indexing torrents. Switching to DHT exclusively means that they're free from that, and their defense that they're equivalent to a search engine has a better chance of working.

So here's the interesting point. If you want to shut down a website, it's relatively easy to do - there are several centralization points, such as the domain name, or the server it's hosted on, that are owned and paid for by people, and people are subject to the law. A fully decentralized service, on the other hand, is much more tricky. Take Freenet, for example. The system is specifically designed to be completely anonymous, secure, and deniable - if you're careful, it's next to impossible to prove that you downloaded or uploaded something from or to Freenet. I'll be blunt here - child pornography is traded occasionally on Freenet, and while there are a lot of people that would like to shut it down, for good reasons, it is basically impossible. If you want to shut down BitTorrent DHT, or the Freenet network, you're basically out of luck.

Here's the relevant question, then. How do you regulate a completely decentralized system? Is it even possible? I would argue that, with the Internet's current architecture, it's not. This is a huge deal - right now, it is possible to build completely secure and untraceable communication networks, at next to zero cost to yourself, which cannot be taken down by anything less than the scale of a massive military operation. It doesn't even have to have a lot of people using it. Take, as another example, any of the various mega-botnets running around these days. These are networks composed of millions of computers, next to impossible to shut down, where almost all of the participants in the network are there involuntarily.

What does it mean for society, now that communication has the potential to be completely unregulable? How do we shut down a terrorist cell, when they talk to each other over encrypted VoIP instead of cell networks? (Actually, that's not much of a problem. Despite what the government tells you, most terrorists would be lucky to set off a firecracker, much less talk about setting one off over VoIP.) Do laws about libel still mean anything, if speech can be truly, untraceably anonymous? What about copyright law? Now I'm drifting into old questions, though.

There are interesting parallels between legal regulation and hardware failure, actually. Decentralization, in its ability to protect against the latter, actually seems to prevent the former very effectively.

Things I nearly forgot today

2009-11-16T20:10:00.000-08:00

* To get out of bed.

* To wake up, once I got out of bed. It took a few extra hours.

* Today is Monday. Crap.

* To finish writing my statement of purpose for grad school apps. I've had that 95% complete for way too long.

* To do other stuff with my grad school apps.

* What I'm doing here.

* To avoid tvtropes.org like the plague.

* What we're doing for our stats project. Luckily, one of my group members remembered! (The other found out for the first time today, I am pretty sure.)

* That I needed to pick up my sister after I got done with the stats group meeting.

* That ice cream doesn't actually count as dinner, no matter how much I want it to.

* To eat dinner. (Fried rice from home, plus Sriracha, eventually!)

* To do laundry. No, wait, put that under things that I actually did forget.

* To update this blog.

Decentralization IV: Nonresident societies

2009-11-15T12:40:00.003-08:00

Wouldn't it be neat if we could be citizens of the Internet? And not just because it'd vindicate this xkcd. More and more, we can live our lives almost completely online (Second Life being the classic-if-somewhat-irrelevant example). Taking this to the extreme, what if we could become citizens of the Internet? What would the consequences be? Would it even make sense?

(There are obvious problems with treating the Internet as a place, of course, and I blame sci-fi for a lot of them, going back to Neuromancer, and probably well before that. Sci-fi has stuck us with the gigantic lie that is "cyberspace", and convinced a lot of people that would otherwise know better that the Internet is a place, filled with things, which are separated by distance, and separated by [fire]walls. All of this is complete nonsense.)

What I'm talking about here is decentralized governance, maybe, until I find a better word for it. All governments and countries today are centralized in a really important way: centralized jurisdiction. The government has control, imperfect though it may be, over your presence in the country at any given time. They can force you to stay or to leave, and it's difficult to stay or leave without the government's tacit approval. There's a rough balance between the government's jurisdiction over you, and your ability to travel off the grid. This is enforced by the laws of physics, on some level: if we could teleport, for instance, or become invisible, the balance here would be very different.

Whenever there's a limitation imposed by the laws of physics, we can expect technology to take a crack at it. High speed transportation is changing the balance: with the advent of air travel, for example, you can pass through a country without ever being in it in any meaningful sense. This doesn't really change the balance in any meaningful way, though, because governments can simply step up control of airports, and pull the balance back in the other direction. Fundamentally, it's still the same situation.

Jurisdiction is important. It forms the basis for both laws enforced by governments, and services provided by governments. A military, for instance, protects a territory. It wouldn't make sense for countries to have militaries, unless there was a clear-cut equivalence between a government and the territory it owns. With an Internet-based government, though, you would lose that equivalence. When every citizen of a country can leave at will, simply by disconnecting, there are many services that it's impossible to provide.

You should be objecting at this point that this is all moot, since any Internet citizen would also be a citizen of whatever country they happened to be living in offline. While this is true now, it's not a necessary condition. We could imagine a region, with minimal (or without any) governance, designed specifically for people from various Internet nations to coexist. This could become arbitrarily ludicrous; imagine being a police officer in such a place, and not knowing whether or not any given person you saw was within your jurisdiction. Or, imagine trying to collect taxes from citizens that suddenly switch their allegiance for a few weeks every April.

Since I'm running short on time now, I'll just go ahead and assert that an Internet nation can only regulate what happens on its own servers and systems. This is not entirely useless, and a lot of services (especially identity-related services, which every government in the world is currently sucking at) could be provided this way. Even so, at this point we're talking about such a watered-down and neutered conception of citizenship that an Internet nation ceases to have any meaning at all.

Exclusive Internet citizenship is pretty much a bust. All is not lost, though. We could imagine a form of Internet dual-citizenship, where an online "virtual country" provides some additional services, and works with governments to provide them across national borders. (Actually, I really hope the phrase "virtual country" doesn't ever catch on. I'm starting to hate it already. >_>) This would represent a hybrid decentralization of government - doing what can be done in a decentralized way, but falling back to the existing central government for everything else.

College Puzzle Challenge non-post!

2009-11-14T00:25:00.002-08:00

Today is the College Puzzle Challenge! Non-stop puzzle solving from 11 AM to 11 PM, it'll be awesome. But, if you're expecting a real post from me, you're crazy. :p

Instead, I have a fortuitous link or two!

http://www.ribbonfarm.com/2009/10/07/the-gervais-principle-or-the-office-according-to-the-office/

http://www.ribbonfarm.com/2009/11/11/the-gervais-principle-ii-posturetalk-powertalk-babytalk-and-gametalk/

These two blog posts look at corporate politics via The Office. Even if you don't watch The Office, they are filled with seriously awesome stuff! But you should maybe consider watching The Office anyway. <_<

Review: Sony Reader PRS-500

2009-11-13T13:05:00.001-08:00

Somebody asked me to do a review of this, since I occasionally talk about how awesome it is. (My theory is that, because it's a relatively niche product, Sony forgot to have their crack team of anti-engineers work on it and cripple it in some way.)

So what's good about it? Mostly, it rides on the strengths of the e-ink screen. The battery life is simply ridiculous - I only have to charge it every few weeks, or every few days if I'm reading non-stop. In terms of books, the battery lasted long enough to get through Anathem (nearly 1000 pages) on a single charge. E-ink only uses energy when it updates the screen, so it's perfect for this kind of device. For people that say that an iPhone is all the ebook reader they'll ever need: yeah, let's see your iPhone last this long.

Physically, it's comparable in size to a really thin paperback - it's a bit larger, thinner, and heavier. It'll even fit in your pocket, if you have big pockets. It comes with a fairly sturdy cover, which is pretty nice - as long as you don't manage to snap it in half or something, it's almost as sturdy as a real book. There are multiple buttons you can use to flip pages, which is convenient if you want to hold it in different ways. It connects to a computer using a normal mini-usb plug.

The contrast on the screen could be higher, but it's still perfectly readable in most lighting. Bright light definitely helps, though - imagine a book printed on medium gray paper. That's about how reading the screen feels. You can adjust the font size, but I usually keep it on the smallest setting so I can see more text at a time. You can't change the font that it uses, which would be nice, but isn't really essential. The screen has 170 DPI, which I sometimes wish was higher, because the edges of the letters look jagged if you look really closely. It's not bad enough to be distracting when you're reading, though. Overall, the reading experience is pretty good - it's much easier on the eyes than an LCD screen.

It has about 100MB of usable storage built in, which is enough for several dozen ebooks - I haven't even come close to using up the space yet, since I usually delete books from it when I finish reading them. It also comes with an SD card slot, though, so if you wanted to you could put thousands of books on it at a time, and swap out SD cards for even more space. Basically, for all intents and purposes, you can treat it as having unlimited storage.

One thing I wish it had is a way to jump to a specific page. It's usually pretty good about remembering your page, but if you lose it somehow it's a huge pain to get back to it. It'd also be nice if you could do more from the interface: you can't delete books, for instance. For that, you have to use Sony's provided software, which is a gripe in its own right, because it is kind of shittastic. It took me forever to figure out how to even copy a book onto the device.

This is largely mitigated, though, because there's an pretty good open source replacement for it, called Calibre. Not only does it actually work (on Windows, Mac, and Linux, no less), it handles conversions between different ebook formats pretty smoothly. The only thing you might conceivably need Sony's software for is firmware updates, and since I'm pretty sure they've stopped supporting the PRS-500, that's not a huge concern anymore. >_>

One more thing: Sony is launching a bookstore based on EPUB, but for some inexplicable reason, they've decided that the PRS-500 isn't important enough to update with EPUB support. I applaud their zeal in trying to retroactively screw over this device, but since EPUB DRM has already been broken, I'm not anticipating too much trouble converting to an older format.

Edit: Wow, I should complain about stuff more often! I just saw via mobileread that Sony will offer a free upgrade to PRS-500 owners. XD

Decentralization III: B.Y.O.I.

2009-11-12T12:18:00.001-08:00

There's one question you can ask of any new Internet technology that can usually predict whether or not it has any chance of succeeding as infrastructure: "Can I just run my own, and have it work the same way?"

This sort of decentralization is common to nearly all successful Internet-scale technologies. I can run my own Web server, and anybody can access it just fine - I don't have to get permission from the Web Corporation, or run my website on hardware provided by them, or anything like that. Same with e-mail - anybody can make their own mail server, and send messages to anybody (unless the ISP blocks it, which many do these days). Same with XMPP - anybody can run their own XMPP server, and use it to chat with anybody else in the world using XMPP. Same with Google Wave - Google knows what I'm talking about! They designed it to run in a federated model, so that people can set up their own Wave servers, and communicate with people using other ones. Same with dozens or hundreds of other protocols.

Of course, there are some notable exceptions, in every direction. Google is arguably Internet-scale, even though they're a centralized service. Internally, though, they've got decentralized infrastructure all over the world. Basically, they've gotten some of the benefits of decentralized infrastructure, by paying for all of it themselves. IRC, on the other hand, is sort of a counterexample because even though it has decentralized infrastructure, it scales pretty poorly - if you're on one IRC server, you can't talk to people on other servers. It's a tradeoff, really. IRC servers have enough problems to deal with even without being able to send messages between servers.

Twitter's a counter-counterexample, because I love to hate on Twitter: They fancy themselves to be an Internet-scale service, but I don't believe they can scale to match the demands that come with truly being Internet-scale. (They're still getting taken down by DDoS attacks every once in a while. Can you imagine the kind of DDoS it would take to knock Google offline?!) Clever engineering may save them yet, but I wouldn't count on it.

As a counter-counter-counterexample (can he do that? :O), we have Identi.ca and the OpenMicroBlogging network. This is a twitter-like service that actually can scale, because they've designed it to be usable across multiple servers. For example, if I have an account on Identica and somebody else has an account on Brainbird, I can subscribe to them and everything just sort of works. I guess that instead of being a counter^3 example, this is just an example: anybody can run their own instance and have it work with everybody else's.

The BYOI (bring your own infrastructure) approach works. But what are the tradeoffs? Decentralization in these cases brings scalability and robustness, both of which are really important for systems that are trying to gain traction. On the other hand, you lose centralized control. This makes it more difficult (next to impossible, in some cases) to update the protocol, and it means that some unwanted uses of the system (such as spam) are impossible to control.

It also means you have to subdivide your namespace, and this can be a tricky problem. With Twitter or AIM, you can talk to people using just a username. With email or XMPP, you have to specify what server they're on as well (user@server.com), because people can use the same username across different servers. People have tried to design decentralized systems that use a centralized namespace before, but it's a fundamentally hard problem to solve without compromising your design. I would argue that DNS (the system that maps domain names to addresses) has managed it - whenever you type in a website address, you're pretty confident that anybody else in the world will see the same website, because it's a centralized namespace on a decentralized system. On the other hand, they only managed this by charging money for domain names, which isn't something that'd work well in other places.

Finally, for a system based on a decentralized protocol, you get some level of additional security over time. There are two classes of security holes: holes in individual applications, and holes in the protocol itself (the latter being much rarer). With a centralized system, these have basically the same impact, since there's really only one instance and one implementation. With a decentralized system, there are usually a few major implementations, and a lot of minor ones around the edges. If tomorrow a major bug was found in BIND (the most common DNS server) and all BIND servers had to be taken down until it was fixed, the Internet would mostly continue to function. You can't get that with a centralized service.

PiCoWriMo

2009-11-10T22:30:00.001-08:00

As some of you are intimately aware, November is National Novel Writing Month! One of these years, I'm actually going to do it, but right now is really a bad time I think. Maybe next year. Several of my friends are doing it, though, so this is a tribute to them!

A NaNoWriMo is a bit much for me right now, but thanks to the magic of the metric system, I can at least manage a PiCoWriMo. So here is a fifty word story:

The suburban lights sprawled beneath them, in jarringly uniform rows, like government-issue constellations. As they fell, they tumbled, and glimpsed for a moment their makeshift dirigible, scudding silently against the Moon's halo. They landed hand in hand, mad grins on their faces, as noiselessly as two trees in the forest.

Default applications on Linux

2009-11-09T22:00:00.001-08:00

Guys, I'm interviewing with Microsoft today, wish me luck! To commemorate this, I'm going to do a tribute to the anonymous Linux Hater's excellent blog.

How to set default applications

On Windows: Right-click on a file of a given type, go to "Open With", and set the application that you want it to open with.

On a Mac: Pretty similar, except you use "Get Info" instead of "Open With".

On Linux: It depends on what desktop environment you're using. Gnome, KDE, XFCE, and every other desktop, does it in a completely different, completely uncoordinated way. Also, if you switch desktops, you lose all your settings.

Q: But doesn't that make people's lives more difficult?

A: Linux is about choice! Specifically, the choice we made to do whatever the hell we wanted.

Q: But what if I want my program to set the default application for a certain filetype?

A: Oh, you should never need to do that. We won't tell you what you should do instead, but we're very sure that you'll never want to do that.

Q: Wait, I thought Linux was about choice! What about my choice as an application developer?

A: Did we say that? Actually, it's just an excuse we use to avoid any of the hard work that might be involved in actually standardizing things. See, because when we don't, users have to make choices they otherwise wouldn't need to worry about, but this way they feel better about it. Choice, baby!

But wait, never fear! There's the Portland Project, which had the lofty goal of unifying Gnome, KDE, and all the other desktops out there! Surely, an effort this important to the success of the Linux desktop would receive the huge amount of developer attention it needs to become successful!

...no, wait, my bad, I'm thinking of a less dysfunctional operating system. Actually, xdg-utils (the result of the Portland project, which was supposed to unify all this) is buggy, mostly unmaintained, and hasn't actually seen a release in years. On top of that, instead of being an awesome unifying solution for the completely separate default application systems that exist, it's just a few shell scripts that support Gnome, KDE3, and sometimes XFCE. There's not even a generic fallback mechanism in there! It's completely useless for users of anything other than Gnome and KDE (and really, XFCE too).

Wouldn't it be nice if there was a tool that would let you just go to a command line and type "open whateverfile.whatever"? All the desktops could standardize on it, users of other desktops wouldn't be left out, and everything would be simpler. I have been thinking about this for a while, and I even took a few shots at writing such a tool. So you can imagine how dismayed I was when I discovered that this tool already exists on Macs and it works perfectly. You'd think the Linux community would have been all over that; ripping off Apple is what they do best!

Decentralization II: currency

2009-11-08T22:00:00.000-08:00

There are two major systems of currency in this country right now. (There are probably others that I'm overlooking, but we'll simplify.) I'm referring to cash, and credit cards. Ostensibly, these two are a single currency, since they're readily interchangeable and tied together, but they're clearly two separate systems.

Cash is a decentralized system. Any given bill holds its value more or less independent from its surroundings, assuming of course that the government is still around. Credit and debit cards, on the other hand, form a centralized system - your credit card may have a fancy picture on it, but unless whoever you're trying to pay has a dedicated communication channel with the credit card company, it's just a very pretty piece of plastic. This distinction is made clear in several ways.

First, authentication of value: Cash uses a relatively weak distributed authentication scheme - the bill or coin only has to look valid, which can be made difficult, but never impossible. This only works because breaking the authentication (counterfeiting) on a large scale can be expensive, and each individual break only nets you a relatively small payoff (the value of the coin or bill). Credit and debit cards, on the other hand, use relatively strong authentication for value - you contact a centralized server, which can keep track of all currency in the system. This is far more difficult than the decentralized cash model, but it's also next to impossible to break, assuming the server is written securely. (Note that authentication of value isn't the same as authenticating the owner of the money - both cash and credit cards are pretty bad at this, in different ways.)

There's also the issue of robustness. It's not unheard of (even if it's not exactly common) for credit card processing to "break" at a given venue, forcing them to process all transactions with cash, or by other means. This can happen because credit cards simply don't work without being able to contact a central location to get the balance on the card, and subtract from it. Cash, on the other hand, is available for use under a much wider range of circumstances. Because it holds its value in a fully decentralized manner, it's not subject to any kind of communication breakdown, and so it works as long as people are willing to accept that it has value. (Note, though, that the authentication is weaker because of this - thus, counterfeiting is possible for cash, while a fake credit card wouldn't get you very far.)

Cash also has the property that, since no centralized communication is required for its use, it can be used without being tracked. In other words, cash can be completely anonymous. With credit cards, there's a single point through which all transaction data flows, and this in turn means that data about when the card is used can be collected very efficiently. This is another property that applies in general to centralized systems, but it can be applied to decentralized systems too - it's just harder. We could imagine, for instance, a system in which people were required to scan all currency that they handled into some sort of centrally controlled device, to prevent counterfeiting. This could be circumvented, since the scanning step isn't required for the transaction, but it's a way to graft centralized notions of control onto an otherwise distributed system.

This is an important point, actually. In a lot of cases, centralized systems can have decentralized aspects added, or vice versa. With money, for instance, it's all printed in a few centralized locations, and this contributes to it having value: if anybody could print their own money, it'd be worthless. When it comes to distributed systems, introducing centralization can be a powerful technique, if the tradeoff is worth it.

In other news, I haven't figured out everything I want to do for the rest of these, so I just might take requests. XD

Decentralization I

2009-11-07T22:00:00.000-08:00

Which is better, centralization or decentralization? I've been giving this question a lot of thought lately, in the context of distributed systems, and I've come to realize a few things.

First, the question applies to more things than you'd expect. For once, I'm not going to spend the entire time talking about tech - command and market economies, for instance, can be understood as centralized and decentralized systems, respectively, and analyzed as such.

Second, there are fundamental limitations in both directions. For example, a fully centralized system will always undergo some sort of failure eventually, thanks to Murphy's law, while a completely decentralized system has to deal with malicious individuals, asynchrony, and other such issues that keep people that work on distributed systems awake at night.

Third, it's not an all-or-nothing question; it may not even be a smooth gradient. In addition to the tradeoff between centralization and decentralization, we need to consider hybrid decentralized systems, which have proven to be a good compromise. Popular examples include e-mail, and XMPP (aka Jabber, aka Google Talk).

Over the course of this month, I'm going to be writing a series of posts about decentralization. Many of them won't even be about computers, but about other systems that I'm looking at from this perspective. I think that there are a few core principles that account for the difference between centralized and decentralized systems, and I'm trying to tease out what those are.

The OOM Killer

2009-11-06T22:00:00.000-08:00

There is a small but vocal contingent of Linux "advocates" that are only too happy to tell you that Linux is super-awesome and will solve all your problems. "Linux will do everything that wind0ze will do, only better, and it'll do it for free!" and on and on like that.

I'm not writing this post specifically to annoy people like that, but it certainly wouldn't hurt. :3

So what's this "OOM killer"? It's something that Linux fanboys generally don't like to talk about, assuming they even know it exists. Let's get some background first, on memory allocation.

When a process is created on Linux, the operating system needs to copy all the memory from the parent process into the new process. It does this because, following the traditional process abstraction, the child process needs to inherit all the data from the parent, but also needs to be able to modify it without messing up the parent. Linux uses a technique called "copy on write", or COW, to do this really quickly. The trick with COW is, instead of actually copying the data, you just mark it read-only, and point both the parent and the child to the same copy. Then, if either of them tries to write to it, you copy it secretly and pretend that it was a writable copy all along. This works really, really well, since the vast majority of memory ends up never being written to for various reasons.

Normally, Linux will handle when it runs out of memory by returning an error to the process that tried to request the memory. This works more or less well. There's an unfortunate tendency among programmers to ignore the result of malloc, though, which means that some programs will start to randomly crash when you get close to running out of memory. The point is, though, that at least there's a way to detect the situation - carefully written programs can avoid crashing by checking the return value of malloc and reacting properly if the system is out of memory.

But there's another problem here. Notice that with COW, the actual memory allocation happens at some random time, when you try to write to some random variable. If the system is out of memory, and has to make a copy, then you've got a problem. (Ideally, you'd make sure that there's enough free memory when you create the process, but then you're wasting a lot of memory - can't have that!) You can't just tell the program that you couldn't allocate memory, because the program didn't try to allocate memory in the first place! You have an error that can't be properly handled. So, Linux handles this situation with... the OOM killer.

The OOM (out of memory) killer does exactly what it sounds like: when you run out of memory, it kills a random process so that the system can keep going. They've developed some rather elaborate heuristics for how it selects the process to kill, so that it's less likely to be a process that you really care about, but as described in this awesome analogy, that's somewhat akin to an airline trying to decide which passengers to toss out the airplane if they're low on fuel. No matter what you do, the fact remains that you've gotten yourself into a bad situation.

I've seen swap provided as a solution to this, but that's basically just saying "don't run out of memory in the first place" - it's not terribly helpful. The fact is, no matter how carefully you write your program, and how meticulous you are about checking for errors, there's still a chance that your program will crash randomly, for no reason at all. Yay, Linux!

An Egg-and-Chicken Situation

2009-11-05T22:11:00.002-08:00

I was reminded today that, while it's perfectly obvious to me that the egg came first, there are a lot of people that still aren't convinced. So, without further ado...

Initially, we can trivially say that all chickens were preceded by dinosaur eggs. This is no fun though, so I'll go ahead and strengthen the "paradox": which came first, the chicken or the chicken egg?

We define a creature as a "chicken" based on its DNA being sufficiently close to the modern species of chicken.

First, we take it as a given that any creature sufficiently chicken-like to be called one will have hatched from an egg. (Indeed, if a chicken-looking creature was born by some other means, we would probably not accept it as a chicken - and this is a proof in and of itself, albeit a less interesting one.) Thus, every chicken is preceded by at least one egg. However, this does not preclude an infinite cycle, which is the source of the paradox.

We next note that, because a chicken does not have the same DNA as either of its parents, and we're defining chicken-ness based on DNA, its possible that a chicken could be born where one or both of its parents are not-quite-chickens. Furthermore, I assert that, since there has been a point in time at which chickens did not exist, this must have happened at least once. If we consider that first chicken, it had the same chicken DNA when it was an egg, so it was in fact preceded by an egg. QED.

An Unnecessary Evil

2009-11-04T22:00:00.000-08:00

What do antivirus software and URL shorteners have in common? They're both elaborate solutions to fixable problems that should never have existed in the first place. They should have both been implemented by the one responsible for the problem, but both were instead solved by third parties. Also, they're both annoying. >_>

Start with antivirus software. The problem, obviously, is the rampant insecurity of Windows as a platform, combined with (let's be fair to Microsoft here, after all) a user base that's been trained to believe that running programs (installers) you've downloaded from random places on the Internet is okay, or even normal. The combination of these two factors has led to a malware industry that's actually pretty impressive, in terms of scope.

These days, Windows security is a lot better than it used to be, and Microsoft is even providing their own antivirus product. Some complain that it's anticompetitive, and that Microsoft is in a position to wipe out the rest of the antivirus market. I will be blunt: I would be perfectly happy to see that entire market shrivel up and die. It should never have existed in the first place, and if Microsoft can secure their OS to the point that we don't need antivirus, we'll all be better off for it. They haven't solved the problem yet, but they're trying, at least.

As for URL shorteners, let's look at why they exist at all. Twitter has a hard 140-byte limit on updates, a limit which is actually imposed by the 160-byte limit of text messages. People who receive updates through texts are the only ones for whom the limit is relevant; for the rest of us, it can easily be glossed over in the interface.

So, why doesn't twitter just run their own URL shortener for SMS users, keep the messages with shortened URLs within their own system, and automatically expand them when displaying them to users that aren't using SMS? Then users wouldn't have to worry about shortened URLs at all when reading tweets; Twitter clients could automatically shorten URLs to an appropriate length when posting updates; Twitter could even charge for use of really short URLs and finally have an actual revenue stream. (Or not, on that last one, I dunno if there's enough scarcity there to even support micropayments.) Ideally, the shortener would guarantee some minimum length of URL that it could provide, but then actually use the longest possible URL that would fit in the message, to preserve the URL space.

As with antivirus software, a market has sprung up to address this unnecessary flaw in Twitter's service. There was a rash of URL shorteners popping up with the rise in Twitter's popularity, though I think that's cooled off quite a bit with Twitter's use of bit.ly as the default, and is.gd going out of business (or not? I forget now). Given that a lot of people aren't using Twitter via SMS, URL shorteners are an unnecessary evil in most cases. Twitter should follow Microsoft's example, step up to the plate, and give us back our URLs.

Idea post: low-latency caching proxies for mobile internet

2009-11-03T22:00:00.001-08:00

This is just an idea post, since I seriously don't have time to implement this right now.

A problem that has been getting worse with mobile internet (on cell phones, etc) is latency. Rather, I should say, the bandwidth on mobile devices has been getting better and better, and is starting to rival slow DSL connections for throughput, but the latency is still atrociously bad. This could be addressed by mobile network operators, but since I'd like to see this problem solved before I am dead, I think it's fair to look into alternate solutions.

Quick explanation first: Bandwidth is more complicated than people usually think. There are two main components to it: throughput, which is the number of bytes per second, and latency, which is the amount of time those bytes take to arrive. Internet connections are usually sold exclusively in terms of throughput, but latency can be really important, especially for applications that have to go back and forth between you and a server a lot. Wireless networks (cell networks and wi-fi both) generally have way higher latency than wired connections of any type. For a good wired connection, 30-150 milliseconds latency is normal, while for a wireless connection, it ranges from a few hundred to a few thousand milliseconds. Congratulations! You now know more about internet connections than a Level 2 AT&T tech support person.

idea 0: low-latency caching proxies

This is actually a pretty trivial thing - anybody can set up a proxy server, and point their mobile web browser at it. (Well, theoretically.) This doesn't gain us that much, though. It'll mainly reduce latency caused by the website, not by the mobile network, so it doesn't really address the actual problem. There are two reasons for even mentioning this: it gives us compression, since that's a relatively straightforward thing that proxies can add, and it makes the rest of the ideas possible.

idea 1: aggressive HTTP pipelining

HTTP allows for an optimization called "pipelining". Basically, the client makes a bunch of requests at the same time, on the same connection, and the server responds to them in order. This can do wonders for reducing latency. The usual sequence of actions goes something like this: send request, wait, get response, send request, wait, get response, send request... Pipelining eliminates most of the waiting. Clients will usually limit the amount of pipelining they do to be kind to web servers, but if we're getting everything from our own proxy server from step 0, we can send as many requests as we want.

It's not perfect, though. On the web, documents will usually pull in other files, like images, scripts, etc, and those can themselves pull in other files. So while we'd like to make all the requests up front, we usually don't know which files we'll need before the responses start coming back. Pipelining helps, but we still end up having to wait more than we'd like.

idea 2: inlining images and scripts and stylesheets

What if instead of depending on the client to fetch everything, we had the server help a bit? A lot of files on the web can actually be either linked to as external documents, or rendered inline inside a page. Javascript and CSS can be inlined pretty easily, and every web browser supports that. It turns out it's possible to also inline images this way, using something called a data URL.

The proxy server could fetch the web page, fetch all the other parts of the page that the client would normally have to request individually, and package them up into a giant page, before sending that to the client. If this works perfectly, then the client only has to make one request, and wait for one response.

This has some pretty serious disadvantages, though. For one thing, it bypasses caching on the client device, so anything included using this method would have to be fetched each time the page loads. This would be fine for small files, but there would have to be some kind of heuristic on the server for what to inline, and what to leave out. I'm also not entirely sure that javascript behaves exactly the same way when it's linked versus inline, but given how well mobile devices support javascript to begin with, that may not be such a big issue.

idea 3: image prescaling

Jumping back to decreasing bandwidth here. It seems kind of a waste to download a full-size image to a mobile device when it's just going to scale it down before it's displayed. Why not scale the image down to the display size before it leaves the proxy? We'd need some way to control this from the client end, since otherwise the server won't know what size to scale to. A custom request header would work, probably. This would also save a lot of CPU time on the mobile device, which translates to faster rendering and less battery usage.

This may not work so well for zoomable browsers, though, such as the iPhone. In this case, it would be interesting for the server to make the image a progressive scan image, so that the client could receive a scaled version first, and then the rest of it later. We could even imagine something fancy, with progressive scan images and HTTP Range headers, where the client first downloads the first progressive part of each image, enough to display a scaled version, and then goes back and fetches the rest of each image file.

(And while we're at it, we can convert GIFs to PNGs. :p)

idea 4: speculative server-initiated prefetch (needs a better name) blah blah

We've been trying to work around the fact that clients have to fetch everything that's on a webpage to load it. What if we bypass that entirely? (this one is more me thinking out loud, actually)

What I'm proposing here is a way (that I haven't thought through properly) to have the server push files to the client, rather than waiting for the client to request them. This would allow the client to cache page elements properly, but still have the same performance as inlining page elements. The problem we'd run into then is that the server has no way of knowing what's in the client's cache (and really, shouldn't), so there will still be a lot of wasted bandwidth here. Maybe the server could send a list of page elements and -- nope, then we're back to client fetch anyway. Hmm. This one might be a lost cause. >_>

Existing stuff

Naturally, I'm not the first one to think of this. There are a few implementations of parts of this; the one that comes to my mind first is Opera Turbo. What I'm proposing here, though, is an open-source server that anybody can run, rather than an add-on controlled by one company. After all, the Internet has proven time and time again that the usefulness of a technology is proportional to how easy it is for an enterprising geek to roll their own.

Lesser-known racist fallacies

2009-11-02T22:00:00.000-08:00

"My friend, who is a member of (ethnic group), was not offended by this joke; therefore, it's not racist." This is really a generalization of the inexplicable tendency of people to assume that any member of another race speaks for all members of their race, which is roughly the intellectual equivalent of "You all look the same to me." I, for example, am a pretty nonrepresentative sample of Indians. If you want to know what all Indians think, give me a few days so I can go around and ask all 1.1 billion of them.

"I didn't mean for it to be taken as racist." a.k.a., "The road to racism is detoured by good intentions." The intent was never the problem to begin with; racism is as problematic as it is because it offends people, not because you wanted to. (okay, so this is debatable.)

"Minorities, when speaking about racism, are above reproach." If anything, minorities are frequently just as racist or more racist than others, just because nobody thinks to call them out on it. (Asians, for example, can be incredible nationalistic snobs. Just try asking a first- or second-generation immigrant parent about all the amazing things that were done first in their country.) This is still a bad thing!

"Traditions and culture are especially important to minorities." In my (unnecessarily inflammatory) opinion, traditions are mainly important to people who don't have much else. I am more than what I've been handed down by thousands of years of handing-down, or at least, I aspire to be.

"Meat is delicious, you should try it sometime. :o" Nah, see, I might be convinced by this reasoning, except that it kind of misses the entire point. >_> I will not be swayed by your delicious bacon!

Barriers to entry for open-source contributions

2009-11-01T22:00:00.000-08:00

Contributing to open-source projects is relatively easy, but it could definitely be easier. There have been times that I've found a problem, decided on a solution, even written a patch once or twice, and then lost interest because of some random restriction along the way. This is wasted potential!

Moreover, contributing to open source is easy for me, but I'm already an open source contributor. For new contributors, the process can be really daunting, for a variety of reasons. I'm going to go through a list of things that projects can do to encourage contributions, roughly in order of increasing difficulty.

Ask!

It's really surprising how few open-source projects even get this far. Every project needs to have a page somewhere that says something along the lines of, "If you want to contribute, we could use a hand with X, Y, and Z. Contact so-and-so for details." If you don't say something like this, most people - especially those that aren't familiar with open source - will just assume that you don't really want outside contributions.

Keep some easy bugs around for new contributors

This is pretty easy for any project that has a reasonable volume of bug reports coming in. Invariably, some of them will be for really easy stuff - spelling fixes in documentation, or other really trivial fixes. Instead of just fixing these, give them a special tag or something on your bug tracker, or make a list of them on your website, and advertise this list to your users. The ones that are interested in contributing will have something quick and easy to get started on, and you'll have that many fewer bugs to fix - it's a win-win! Ideally, once you have a system like this in place, people will feel more comfortable about filing trivial bugs too, and you'll end up with higher-quality software overall.

Don't require registration to submit a bug report

Trac is kind of bad about this since everybody runs their own instances, though if I recall correctly recent versions support OpenID, which is a big step forward. If I find a bug, but I have to go through the whole registration dance just to report it, I'm just not going to bother reporting it unless it's a really severe bug. You may say that I'm not a serious contributor if I let a registration page stop me, but that's kind of missing the point. Casual contributors can be just as valuable as really serious ones, and there are a lot more of them out there.

Allow editing through a web interface

This one doesn't actually exist yet, but it could. The normal process of submitting a patch involves checking out the source code, making modifications, generating a patch, and submitting that to the maintainers of the program. 3/4 of those steps could be automated on a code hosting site such as SourceForge or github or bitbucket. The code could be cloned on the server; the user could be presented with a simple text editor in the browser (or something more fancy, like Bespin), the user could save the code with a commit message when they're done, and the commit could automatically be submitted as a pull request on the server. I wouldn't want to use this to make serious changes to the code, but this isn't designed for people that are already making serious changes. For somebody who's just making a small cosmetic change, this would be a huge timesaver, and that in turn increases the number of contributions you get.

NaBloPoMoTwoOh

2009-10-31T22:00:00.000-07:00

To be honest, I'm not even sure that this counts.

The date today is October 14th, but I'm scheduling this post to appear on November 1st - thus starting into National Blog Posting Month 2009. I'm planning to write at least a week's worth of posts ahead of time, so that I'll have a chance of sticking to a schedule, because from back here it looks like November is shaping up to be a pretty rough month. It feels kind of like cheating, though - does it really count as NBPM if I'm not writing all the posts during the month?

On the one hand, it does kind of defeat the purpose of NBPM. The idea is to write one post per day, not just have one post appear every day. From that perspective, it looks very much like I'm cheating. >_>

But I've said this before, and it bears repeating: I'm not blogging so that people will read what I write, or for an annual event. I blog for purely selfish reasons. I need to keep in practice with writing, or I'm afraid that I'm going to forget one day, and lose a valuable skill. I've also found that a blog format is a great way to get my thoughts on a subject in order, and simultaneously put them on the Internet for future reference. Finally, as with my post from May about my research project, putting something on this blog occasionally saves me a lot of breath explaining things to people.

In that case, why go through all this rigamarole? Why bother with NBPM, when I could spend my time on other things that I desperately need to finish? Honestly, I just think it'll be fun. What other reason do I need?

(random parting thought: scheduled blog posts could be a really neat twist to a murder mystery :o)

hiatus

2009-10-01T22:00:00.000-07:00

Okay, so. As much fun as this blog is to write, I need to turn in a draft of my honors thesis (average length: 20-30 pages) in just over a month, and I've so far written not a word of it. The next month is going to be kind of a pain, I think, so I'm probably better off not updating for a while.

I'm still doing NaBloPoMo this year, though. :3 See you again in November!

Fun with find

2009-09-24T22:00:00.003-07:00

The find command is incredibly useful, if a bit arcane, so it's a shame that more Linux users aren't aware of it. Basically, if what you're trying to do sounds like "search for files with these characteristics, and do something to them", then find is probably the tool you're looking for.

Let's start with the very basics. If you run find with no arguments, it'll start listing all the files it can find. This isn't especially useful behavior, but let's take a closer look at it. find takes a list of paths, and looks at all files contained in each of them - if you don't give it any, it'll assume the current directory. We can introduce a bit of bash-fu to start doing something that looks like it might be useful:

$ find ${PATH//:/ }

This will list all the programs that are available on your path. (The weird-looking variable reference just replaces colons with spaces - look here if you're curious.)

Just listing files is no fun, though. You can do that already, with ls, which also has the advantage of being a few less letters to type! So let's start getting into the real power of find, with expressions.

After the paths to search on, you can specify any number of expressions - basically, filters that look at the list of files and only select the ones matching some criteria. One simple one is -name:

$ find /usr/portage -name ChangeLog | wc -l

(If the bar thing looks funny to you, you need to read up on pipes. If you don't know pipes, you can't really say you know how to use the command line, they're that important.) This is a command I used just a few hours ago, to find out how many ChangeLog files there are in Gentoo's portage tree. Without the find command, this would have been kind of a pain. find also has a -iname filter, that does a case-insensitive match - useful if you're looking for files that have inconsistent capitalization.

There are a lot of other possible filters, too many to list here, so you'll have to read the find man page to see them all. Here are just a few examples:

$ find ${PATH//:/ } -name "mkfs.*"
This is the earlier example, but with a twist - this prints out the full path to programs matching a given pattern. (If you only want one program, the which command is easier, though.)

$ find ~ -empty
This lists all empty (zero-length) files in your home directory.

$ find / -user root
This will list all files owned by root. (You probably have to be root for this to actually list all of them, for obvious reasons.)

$ find / -size +500M
This finds all files on your system larger than 500 megabytes, and requires some explanation. Filters that take numerical arguments can usually also take a + or - modifier, to mean "greater than this" or "less than this". If you leave it out, then you can search for files that have some exact size.

$ find ~ -mmin -30
List all the files in your home directory that were modified in the past 30 minutes. (No more wondering about where you saved that important file!)

$ find /usr/bin -not -executable
There shouldn't be any non-executable files there, but I found one on my system - probably a bug in the package that installed that file. (Want more logical operators? You can stick a -or between two filters and find will return the file if it matches either of them.)

"But wait," you might be thinking. "You said find would look for files and let me do stuff to them, but listing them isn't terribly interesting!" Don't worry, the fun is just beginning. :D

The simplest way to get find to do stuff with files is not to use find at all: pipe the output to xargs instead. For most simple tasks, this is way easier than using find's execution capabilities. The following three commands do basically the same thing:

$ find ~ -size 0 | xargs rm
$ find ~ -size 0 -exec rm "{}" +
$ find ~ -size 0 -delete

The first one just pipes the list of files to xargs, which is a nifty little utility that runs the command it's given on each filename it gets through the pipe. In this case, it runs rm and deletes all the files it's passed, but you could use any command there.

The second one uses find's -exec option, which gives you more control over how the command is constructed. After the -exec, you find rm, which is pretty self explanatory - it's the command you're executing. The "{}" thing is find's weird way of saying "the file that was found" - this is where the filename gets substituted into the command. The + ends the command, but there's a twist here. If you end the command with a semicolon instead, find runs the command once for each file. (NB: you have to put the semicolon in quotes or bash messes with it. This took me forever to figure out :( ) If you use +, on the other hand, it has the same effect as far as terminating the command, but it also tells find to jam as many filenames as it can in there, subject to whatever limitations the OS imposes. For large file lists, this can be the difference between your command running thousands of times or just a few times, so using + wherever possible is a good habit to get into.

The third is mainly for completeness - find has a builtin function for deleting files, making this example a bit pointless. :)

Here are a few more practical examples.

$ find /usr/portage -name ChangeLog -exec du -c "{}" + | grep total
I used this to find the total disk space on my system taken up by ChangeLog files in the portage tree. "du -c" will print out the total disk space used by all the files you give it, and the grep filters the output down to just those totals.

$ find -type d -exec chmod 755 "{}" +
Somehow I had a pile of directories on my NFS share that had no execute permissions for all users, so other users couldn't even enter those directories. This fixed all that in a single command.

$ find ${PATH//:/ } -perm -4111 -user root
Shows you all binaries available on your path that are suid root. These can be serious security risks if the programs are written insecurely.

$ find -mtime +365 -exec mv "{}" archive/ +
Moves all files that haven't been modified in more than a year to another directory.

$ find -nouser -exec chown root "{}" + , -nogroup -exec chgrp root "{}" +
Find all files that are owned by a nonexistent user or group, and change that ownership to root. Note the comma in there; it splits up the expression so that you can operate on multiple sets of files in a single find command, and only have to actually scan the directory tree once. If you want to do something like this in the absolute fastest way, find is your friend.

That's about the limit of my knowledge, but the find man page has loads more information, as well as some more examples.

no post today

2009-09-18T11:30:00.000-07:00

too lazy