Piercing the Veil

So I've got a new hobby: when I look around, I try to see the story behind everything I see - everything it took to make it, all the different processes that were necessary for it to come into being, things like that. It's simultaneously exhilarating and humbling and occasionally enlightening [in the negative space sense of suddenly realizing something you don't know]. It can even be transformative, in that you may never look at the world the same way again.

An example: I am riding my bike home from work. The bike itself is a marvel of applied physics - the impossibly thin spokes on the wheels that actually support your weight by hanging the bike from the tops of the wheels; the clever use of gyroscopic forces to keep the bike stable while it's in motion; the adjustable gear system which you can reconfigure as you ride. You could teach an entire physics course just by looking at a bicycle in motion. A bus rolls past, inviting you to consider not only the mechanics of how it runs (the entire history of internal combustion could go here) or why it runs (the efficiencies of mass transit), but also the entire superstructure of government which makes public transportation possible. The sidewalks and roads I'm riding on are made of cement - a building material which is dirt-cheap, uniform, pervasive, and commoditized. Really, it's a perfect analogy for our mass-produced society. As for the roads themselves, they constitute a single, unbroken web which stretches across the entire continent; think about that for a minute. (There's an interested design principle embedded here: if you want a pervasive system of roads, you don't start by building roads. You start by building cars, and the roads will come.) There are buildings by the sides of the road, and every individual component and overall feature has a rich and varied history - glass, steel, central heating/cooling, architecture, fluorescent lights (and the electrical grid they imply), indoor plumbing, and don't forget the construction equipment used to put it all together. If you wanted to build a modern office building from scratch, it would take centuries of technological development.

There is a Veil of abstraction which we apply to everyday life, because our minds are simply not equipped to think like this all the time. We need to take things for granted if we want to be able to function. It can be fun to pierce the Veil once in a while, and take a sip from the geyser of extraneous information that fills our world, but it's not a sustainable state of mind. I won't bother with value judgments here. The Veil is neither good, nor evil, nor anything in between, it simply is.

And if we're bored enough and perhaps stubborn enough, we could even try to peel back the Veil from the Veil itself, from the very fact that we only see the world for what it is, and not everything underneath. There are Darwinian forces in there, enhanced but certainly not altered by capitalism, which drive us to produce and to be productive, and weed us out ruthlessly if we fall into the trap of looking too deeply at the world, and spend our lives wandering around in slack-jawed wonder. There's that fact that our mental processes are fundamentally and severely limited and finite, which lies somewhere at the awkward intersection of information theory and neuroscience.

So the Veil is a necessary part of our lives, and we couldn't pull it back if we wanted to, but I think it's fun to peek behind it every once in a while.

Tahoe project ideas

Tahoe-LAFS is a pretty cool project (which I have blogged about before), but it has some weaknesses. Coincidentally, I've been looking around for a project, so I think I'll try helping out with it.

The current state of Tahoe

Don't get me wrong - as far as cloud storage goes, Tahoe might be the only one I've seen that has some potential. (Maybe I'm setting the bar too high, but maybe everybody else is setting it too low!) If I had to sum up their architecture, though, it looks something like this:

1) A filesystem access layer, which isn't terribly interesting as long as it works
2) A completely badass data management layer, which handles all the tricky bits surrounding security, privacy, and reliability
3) A half-baked distributed storage layer

It's that last layer that I'm interested in, for a lot of reasons. First, there's been a lot of good work in that area over the past 20 years (particularly, a lot of really neat stuff with distributed hash tables 8-10 years ago). Second, it's an area that I've had an interest in for a while, and I think it'd be fun to work on. Third, I think that a more scalable (at a minimum, handling millions of nodes) backend is what's needed to take Tahoe from being a neat and useful program, to being a capital-D Disruptive one. A secure, reliable, fully distributed filesystem running at Internet scale? Oh, hell yes. :D

So with that in mind, here's what I'm thinking as far as possible avenues of work.

Project ideas

• Tahoe bug #999
Multiple storage backends are kind of orthogonal to what I wish Tahoe would do in the long term, but it'd still be really neat.


• DHT-based storage
It would be really cool if Tahoe stored data in a big DHT; several implementations of DHTs exist, but I think that actually integrating one into Tahoe would be a pretty invasive job. File this one under "overambitious" for now.


• DHT-based storage shim
Less ambitious: set up a DHT storage network, and run Tahoe-compatible storage servers on a few of them so that existing clients can use DHT-based storage transparently. This would get us to Storage Nirvana much more quickly, and mainline Tahoe could transition to using a DHT at some later time.


• Storage availability zones
Tahoe falls into the same trap as pretty much every other distributed storage system, in that it assumes that failures are somewhat randomly distributed. In fact, this is almost never the case, thanks to networking: if I'm using Tahoe, then when my flaky home router dies, I will lose access to all my files until I can get reconnected. The goal for this idea is being able to subdivide the available storage servers into availability zones, and distribute shares in a zone-aware way.

Right now Tahoe lets you specify a replication level as a ratio A/B, and uses an erasure code to generate B shares of data such that the original data can be rebuilt from any A of them. What I'm proposing for availability zones is specifying a separate replication level for each zone (the default, global zone should have A/B, the zone for my home network should have C/D, the zone for some paid cloud storage provider should have E/F, etc etc), generating shares using a (A+C+E+...)/(B+D+F+...) erasure code, and distributing those appropriately.


• Amazon S3-compatible interface
Something is nagging at the back of my mind, saying that this already exists. Wait, no! I think that was Ceph (another cool distributed filesystem, but with totally different goals). S3 seems to be becoming the de facto standard for cloud storage migration, so it'd probably be useful.

Actually, taken to the extreme, there's no reason there shouldn't be frontends compatible with all storage providers (OpenStack is another interesting one). What's more, if this was implemented in addition to #999, we could solve cloud storage portability and privacy in one fell swoop.


• Unbreakable static web hosting
You could upload a static website to a geographically-diverse Tahoe cluster, and set up multiple frontends (with caching reverse proxies for performance), and put up multiple DNS round-robins over those, and have reasonably fast static web hosting that's completely resilient to hardware failures, natural disasters, and (depending on the geographic distribution) legal threats.

Favorite thing about working at Microsoft

Well, I can't really pick just one thing, so I'm just going to list a lot of awesome things.

• Free beer at company events
If I seriously wanted to, I could get drunk at work every other week on Microsoft's dime. Nobody actually does, of course, because that'd be idiotic, but there's certainly enough beer that you could.


• Working with extremely geeky people
Today: Me: "Yeah, waiting doesn't really count as work." Coworker: "Unless you're a spinlock! :D" If you don't get it, it would take way too long to explain. XD


• Completely awesome company events
The big intern event for the summer was taking us all to go see a private showing of Cirque du Soleil. (Oh, and while we were there? They gave us all free Zune HDs.) Microsoft is trying pretty hard to make a good impression on us while we're interns, and you know what? It's working.


• Challenging work environment
This is the first job I've ever had where I feel like I need to work extra hours just to keep up. It's a humbling experience, and that's something I kind of needed. I feel like I'd kind of reached a plateau with my skills, and this internship is really helping me to push up to that next level.


• Real-world impact
There's a good chance that some of the code I'm writing this summer will ship in the next version of Office. That's a pretty cool feeling. :D Internships here are structured the same way as projects that full-time employees do, only shorter.


• Fully furnished apartment
Yeah, I could have gotten by without this, but it sure is convenient to just show up the weekend before you start work and actually have a place to sleep. Related to this: I'm living about a ten minute bike ride away from my office.


• Novel work environment
This is the first time I've really done Windows-style development, and since I left my Linux box at home, it's a complete immersion sort of thing. (I know it sounds geeky, but this is the coder equivalent of going to live in a foreign country for a while.) There are a lot of things I'm learning about that I never would have gotten a chance to learn about otherwise (.NET, in particular, is freaking cool).


• Intern perks
There are a lot of these, but the one that springs to mind is a free year of MSDN access. If I wanted to go and actually buy this, it'd cost something comparable to what they're paying me for the entire summer. o_o


• Flexible work hours
Instead of 9-5, I work 9+X to 5+X, where X is a function of when I wake up in the morning. It is incredibly convenient. :D


• The weather
I seriously cannot remember any time before coming to Washington where I could go outside during the summer and find that it's a completely pleasant temperature. Surprisingly, there aren't as many rainy days as people say to expect - I guess that's more a winter thing.

oh no

I have forgotten how to blog D:

Index page for Amazon S3

So I've been looking for a way to get index pages on Amazon S3. Turns out Amazon doesn't provide them, and nobody else really does either. Well, we can fix that!

I just put together http://s3index.p-static.net, which generates index pages for any S3 bucket. If your bucket is publicly listable (go to bucket permissions, and enable "List" permission for "Everybody"), then it should work. The URL format is http://s3index.p-static.net/BUCKETNAME.

Bonus: it's stylable! The script automatically includes a CSS file named "s3index.css" in the bucket, and all the elements have classes declared.

Source code coming later, it's still extremely messy right now. >_>

Everything you need to know about Tahoe-LAFS

There are not a lot of pieces of software that I would describe as revolutionary, but Tahoe-LAFS is one of them. Unfortunately, as is all too common with open source software, the potential of the software far outstrips the documentation, with the predictable result that nobody understands what Tahoe is capable of. This post is an attempt to explain the important bits of Tahoe as succinctly as possible.

High-level goal

Tahoe is an almost-completely-decentralized, secure file store. You can think of it as a reliable black box which you can upload files and directories to. The data is highly resilient (up to 70% of the distributed data can be lost, and the file will still be recoverable, and that's just with the default settings) and secure (RSA is built right into the URLs you use to access data, so nobody who doesn't have access to the data can read it). By giving somebody a secret URL, you can give them read-only or read-write access to any file or directory. Tahoe guarantees that, if somebody does not have this secret URL, they cannot access the file data in any way.

The advantage of Tahoe over something like Amazon S3 is that you don't have to trust your cloud storage provider. S3 provides very similar capabilities, but it's a given that Amazon employees will be able to read your data, which makes it unsuitable for some applications, and which makes people that care about their privacy nervous.

When you upload a file to Tahoe, it's passed through an erasure code, and then distributed across multiple computers participating in a cluster, none of which can read the actual data. When you access data in Tahoe, a form of authentication is built into the URL you use, so you can keep data private simply by keeping the URL used to access it a secret.

Erasure coding

Wikipedia explains the theory far more faithfully than I would be able to, so I'll just try to give a more concrete example of how it works. Let's say you've got a 300 kB file. A 3-of-10 erasure code (what Tahoe defaults to) would take that file, and give you ten 100 kB chunks of data, or 1 MB total. You'd then spread this data around as widely as possible, and the erasure code guarantees that given any three chunks out of the ten, you can reconstruct the original data. This means that if you gave data chunks to ten different computers, and seven of them crashed, you'd still be able to access your data just fine.

Erasure coding by itself isn't secure, so Tahoe encrypts the data before the erasure coding step. This makes it possible to give a certain limited capability (a "verify cap", explained later) to an untrusted machine with a lot of bandwidth, which can then download some of the encrypted data, rerun the erasure code to fill in any missing data, and re-upload the complete coded file - all without being able to read the actual file data.

Introducers

The introducer is the one part of a Tahoe cluster that isn't decentralized. A network is built around an introducer, which does what it sounds like; it introduces Tahoe nodes in a cluster to one another. If you're familiar with BitTorrent, the introducer is very similar to a BitTorrent tracker.

There's been talk of replacing introducers with a DHT network (similar to what BitTorrent does these days) but as far as I know, there's been no action on this.

Capabilities

A "capability" for a piece of data is just a very long URL that you can use to access that data in a certain way. It's called a capability because it determines what you can do with the data - a "write capability", for instance, contains enough information for you to overwrite the file with a new version, while a "read capability" will only let you read the file, but not modify it. There's also something called a "verify capability" (mentioned earlier), which gives you access to the encrypted contents of the file, and can really only be used for verifying the data. Encryption keys for the data are part of the capability itself (which is part of the reason that they're very long), so there's no need to store those separately.

Data security in Tahoe is accomplished by keeping these capabilities - "caps" - secret. If you're the only one who has the write cap to a file, Tahoe guarantees that nobody else can modify the file, and if you're the only one who has a read cap, Tahoe guarantees that nobody else can view the file. Capabilities are the only method of access control. This isn't how we're used to thinking about data security - where are the passwords? - but once you wrap your head around it, it makes sense.

Capabilities can be demoted - given a write cap, you can generate the read cap automatically, and given a read cap, you can generate the verify cap. Thus, having a certain level capability for a file also means you can grant other people the same or lower level capability. I mentioned earlier that untrusted servers can refresh your data without reading it - this is what verify caps are used for. You can make the verify cap for your data public, and anybody will be able to make sure it stays in the cloud, but nobody will be able to read it.

Directories are treated the same way as files, so you can also have read/write/verify caps for directories. I'm not entirely sure how this part works, but the read-only-ness of a directory read capability carries through - if you give somebody a read cap for a directory, all the files they can see in the directory will be read-only for them as well.

Final thoughts

So far, I've been unable to find any information about how Tahoe would scale in very large configurations (thousands to millions of storage nodes), but it doesn't seem like it would. The documentation that I've found specifies that at startup, a node connects to every running storage node that the introducer gives it, which seems like a huge scalability bottleneck. I kind of want to ask the developers if this is really as bad as it sounds, but I haven't had a chance yet.

Assuming it can (or will eventually be able to) scale, I really hope Tahoe becomes more widely used. If it was, it would be a whole new way to think about data management - like a crowdsourced, secure version of Amazon S3. As far as I know, though, there aren't any large Tahoe clusters running - their public test cluster only has two or three nodes in it, as far as I can see. (If you want to start one with me, leave a comment - I can provide an introducer and some storage to start with, but there's no point if it'll be just me. XD)

portable functions braindump

I have just had an incredibly cool idea, and I'm afraid that it'll leave me once the drinks wear off, so I'm going to go ahead and write it down now.

First-class functions are pretty cool, so let's take them to the next level. Imagine for a moment that we could represent a function as a file in a reasonable clean way. (This doesn't take a lot of imagination, since it turns out we can, and the only disagreement is over how to do it best, but bear with me for a second.) What could we do with this? As a start, we could expose functions over the network, and call them dynamically from programs. Instead of having to have a copy of all relevant code on all computers that want to use it, we could store functions online, and dynamically pull them into a cache when we actually want to call them, and abstract away software updates entirely. But wait, there's more.

Cloud computing services are drifting in fits and starts toward a clean model where you submit jobs to the cloud and they happen. Right now, Amazon EC2 is considered state of the art, which is just ridiculous - they make you virtualize a whole OS instance! What the hell is that?! It maximizes flexibility at the cost of incredible inefficiencies. The major bottleneck to a better cloud compute service, though, is a common format for executables. If you only want to submit a single program to a compute service, you and the cloud compute service provider need to agree on a format for the program, and that's basically impossible today. There's simply no consensus at all, not even the beginning of one. (Services like PiCloud are a nice first step, but they do it by standardizing on one language and ignoring the rest of the universe. Useful, but not especially interesting.)

So here's the utopia I'm imagining:

* You put a chunk of code online
* You give your choice of cloud compute service the URL for the code
* They fetch it and execute it
* Everybody is happy and nobody has to fiddle around with what availability zone their EC2 instances are in, and whether or not it matches their data

We'd use HTTP, naturally; HTTP is the new TCP, and everybody uses it as a starting point for their protocols these days. As long as we're exposing code over HTTP, we could expose documentation and other metadata in the same way. http://foo.com/somefunction would give and executable, /somefunction/doc would be rendered documentation, /somefunction/sig would be a signed hash of the code, which is important if people are ever going to let it run on their machines. I'm nodding off right now, but pretend I am describing a good security model instead.

(This would be even cooler when combined with Tahoe-LAFS, which I'm going to blog about soon, but not tonight.)

I vaguely remember somebody having worked on this before in one of those languages nobody uses - Haskell? Erlang? Clojure? I forget - but it was a language-specific solution and therefore boring.

Anyway. My real goal in this is to abstract away the languages that people code in. It's 2010, and we should have papered over all of this decades ago. Up until a few years ago, the command line was the closest thing we had to my ideal. These days, web services might be closer - they support structured data through JSON or XML, generally, which is a hell of a lot more than you can say about the command line. They're also a lot harder to get at, but that's just because of a lack of decent tools.

Actually, though, now that I think about it, what I'm describing here has a lot in common with the #! functionality of the command line. Actually, it's almost exactly the same. Maybe building this will be a lot easier than I thought. :D

It's been too long since I blogged. I am forgetting how to write. :( Gonna hit publish before I sober up totally.