An Unnecessary Evil

What do antivirus software and URL shorteners have in common? They're both elaborate solutions to fixable problems that should never have existed in the first place. They should have both been implemented by the one responsible for the problem, but both were instead solved by third parties. Also, they're both annoying. >_>

Start with antivirus software. The problem, obviously, is the rampant insecurity of Windows as a platform, combined with (let's be fair to Microsoft here, after all) a user base that's been trained to believe that running programs (installers) you've downloaded from random places on the Internet is okay, or even normal. The combination of these two factors has led to a malware industry that's actually pretty impressive, in terms of scope.

These days, Windows security is a lot better than it used to be, and Microsoft is even providing their own antivirus product. Some complain that it's anticompetitive, and that Microsoft is in a position to wipe out the rest of the antivirus market. I will be blunt: I would be perfectly happy to see that entire market shrivel up and die. It should never have existed in the first place, and if Microsoft can secure their OS to the point that we don't need antivirus, we'll all be better off for it. They haven't solved the problem yet, but they're trying, at least.

As for URL shorteners, let's look at why they exist at all. Twitter has a hard 140-byte limit on updates, a limit which is actually imposed by the 160-byte limit of text messages. People who receive updates through texts are the only ones for whom the limit is relevant; for the rest of us, it can easily be glossed over in the interface.

So, why doesn't twitter just run their own URL shortener for SMS users, keep the messages with shortened URLs within their own system, and automatically expand them when displaying them to users that aren't using SMS? Then users wouldn't have to worry about shortened URLs at all when reading tweets; Twitter clients could automatically shorten URLs to an appropriate length when posting updates; Twitter could even charge for use of really short URLs and finally have an actual revenue stream. (Or not, on that last one, I dunno if there's enough scarcity there to even support micropayments.) Ideally, the shortener would guarantee some minimum length of URL that it could provide, but then actually use the longest possible URL that would fit in the message, to preserve the URL space.

As with antivirus software, a market has sprung up to address this unnecessary flaw in Twitter's service. There was a rash of URL shorteners popping up with the rise in Twitter's popularity, though I think that's cooled off quite a bit with Twitter's use of bit.ly as the default, and is.gd going out of business (or not? I forget now). Given that a lot of people aren't using Twitter via SMS, URL shorteners are an unnecessary evil in most cases. Twitter should follow Microsoft's example, step up to the plate, and give us back our URLs.

Idea post: low-latency caching proxies for mobile internet

This is just an idea post, since I seriously don't have time to implement this right now.

A problem that has been getting worse with mobile internet (on cell phones, etc) is latency. Rather, I should say, the bandwidth on mobile devices has been getting better and better, and is starting to rival slow DSL connections for throughput, but the latency is still atrociously bad. This could be addressed by mobile network operators, but since I'd like to see this problem solved before I am dead, I think it's fair to look into alternate solutions.

Quick explanation first: Bandwidth is more complicated than people usually think. There are two main components to it: throughput, which is the number of bytes per second, and latency, which is the amount of time those bytes take to arrive. Internet connections are usually sold exclusively in terms of throughput, but latency can be really important, especially for applications that have to go back and forth between you and a server a lot. Wireless networks (cell networks and wi-fi both) generally have way higher latency than wired connections of any type. For a good wired connection, 30-150 milliseconds latency is normal, while for a wireless connection, it ranges from a few hundred to a few thousand milliseconds. Congratulations! You now know more about internet connections than a Level 2 AT&T tech support person.

idea 0: low-latency caching proxies

This is actually a pretty trivial thing - anybody can set up a proxy server, and point their mobile web browser at it. (Well, theoretically.) This doesn't gain us that much, though. It'll mainly reduce latency caused by the website, not by the mobile network, so it doesn't really address the actual problem. There are two reasons for even mentioning this: it gives us compression, since that's a relatively straightforward thing that proxies can add, and it makes the rest of the ideas possible.

idea 1: aggressive HTTP pipelining

HTTP allows for an optimization called "pipelining". Basically, the client makes a bunch of requests at the same time, on the same connection, and the server responds to them in order. This can do wonders for reducing latency. The usual sequence of actions goes something like this: send request, wait, get response, send request, wait, get response, send request... Pipelining eliminates most of the waiting. Clients will usually limit the amount of pipelining they do to be kind to web servers, but if we're getting everything from our own proxy server from step 0, we can send as many requests as we want.

It's not perfect, though. On the web, documents will usually pull in other files, like images, scripts, etc, and those can themselves pull in other files. So while we'd like to make all the requests up front, we usually don't know which files we'll need before the responses start coming back. Pipelining helps, but we still end up having to wait more than we'd like.

idea 2: inlining images and scripts and stylesheets

What if instead of depending on the client to fetch everything, we had the server help a bit? A lot of files on the web can actually be either linked to as external documents, or rendered inline inside a page. Javascript and CSS can be inlined pretty easily, and every web browser supports that. It turns out it's possible to also inline images this way, using something called a data URL.

The proxy server could fetch the web page, fetch all the other parts of the page that the client would normally have to request individually, and package them up into a giant page, before sending that to the client. If this works perfectly, then the client only has to make one request, and wait for one response.

This has some pretty serious disadvantages, though. For one thing, it bypasses caching on the client device, so anything included using this method would have to be fetched each time the page loads. This would be fine for small files, but there would have to be some kind of heuristic on the server for what to inline, and what to leave out. I'm also not entirely sure that javascript behaves exactly the same way when it's linked versus inline, but given how well mobile devices support javascript to begin with, that may not be such a big issue.

idea 3: image prescaling

Jumping back to decreasing bandwidth here. It seems kind of a waste to download a full-size image to a mobile device when it's just going to scale it down before it's displayed. Why not scale the image down to the display size before it leaves the proxy? We'd need some way to control this from the client end, since otherwise the server won't know what size to scale to. A custom request header would work, probably. This would also save a lot of CPU time on the mobile device, which translates to faster rendering and less battery usage.

This may not work so well for zoomable browsers, though, such as the iPhone. In this case, it would be interesting for the server to make the image a progressive scan image, so that the client could receive a scaled version first, and then the rest of it later. We could even imagine something fancy, with progressive scan images and HTTP Range headers, where the client first downloads the first progressive part of each image, enough to display a scaled version, and then goes back and fetches the rest of each image file.

(And while we're at it, we can convert GIFs to PNGs. :p)

idea 4: speculative server-initiated prefetch (needs a better name) blah blah

We've been trying to work around the fact that clients have to fetch everything that's on a webpage to load it. What if we bypass that entirely? (this one is more me thinking out loud, actually)

What I'm proposing here is a way (that I haven't thought through properly) to have the server push files to the client, rather than waiting for the client to request them. This would allow the client to cache page elements properly, but still have the same performance as inlining page elements. The problem we'd run into then is that the server has no way of knowing what's in the client's cache (and really, shouldn't), so there will still be a lot of wasted bandwidth here. Maybe the server could send a list of page elements and -- nope, then we're back to client fetch anyway. Hmm. This one might be a lost cause. >_>

Existing stuff

Naturally, I'm not the first one to think of this. There are a few implementations of parts of this; the one that comes to my mind first is Opera Turbo. What I'm proposing here, though, is an open-source server that anybody can run, rather than an add-on controlled by one company. After all, the Internet has proven time and time again that the usefulness of a technology is proportional to how easy it is for an enterprising geek to roll their own.

Lesser-known racist fallacies

"My friend, who is a member of (ethnic group), was not offended by this joke; therefore, it's not racist." This is really a generalization of the inexplicable tendency of people to assume that any member of another race speaks for all members of their race, which is roughly the intellectual equivalent of "You all look the same to me." I, for example, am a pretty nonrepresentative sample of Indians. If you want to know what all Indians think, give me a few days so I can go around and ask all 1.1 billion of them.

"I didn't mean for it to be taken as racist." a.k.a., "The road to racism is detoured by good intentions." The intent was never the problem to begin with; racism is as problematic as it is because it offends people, not because you wanted to. (okay, so this is debatable.)

"Minorities, when speaking about racism, are above reproach." If anything, minorities are frequently just as racist or more racist than others, just because nobody thinks to call them out on it. (Asians, for example, can be incredible nationalistic snobs. Just try asking a first- or second-generation immigrant parent about all the amazing things that were done first in their country.) This is still a bad thing!

"Traditions and culture are especially important to minorities." In my (unnecessarily inflammatory) opinion, traditions are mainly important to people who don't have much else. I am more than what I've been handed down by thousands of years of handing-down, or at least, I aspire to be.

"Meat is delicious, you should try it sometime. :o" Nah, see, I might be convinced by this reasoning, except that it kind of misses the entire point. >_> I will not be swayed by your delicious bacon!

Barriers to entry for open-source contributions

Contributing to open-source projects is relatively easy, but it could definitely be easier. There have been times that I've found a problem, decided on a solution, even written a patch once or twice, and then lost interest because of some random restriction along the way. This is wasted potential!

Moreover, contributing to open source is easy for me, but I'm already an open source contributor. For new contributors, the process can be really daunting, for a variety of reasons. I'm going to go through a list of things that projects can do to encourage contributions, roughly in order of increasing difficulty.

Ask!

It's really surprising how few open-source projects even get this far. Every project needs to have a page somewhere that says something along the lines of, "If you want to contribute, we could use a hand with X, Y, and Z. Contact so-and-so for details." If you don't say something like this, most people - especially those that aren't familiar with open source - will just assume that you don't really want outside contributions.

Keep some easy bugs around for new contributors

This is pretty easy for any project that has a reasonable volume of bug reports coming in. Invariably, some of them will be for really easy stuff - spelling fixes in documentation, or other really trivial fixes. Instead of just fixing these, give them a special tag or something on your bug tracker, or make a list of them on your website, and advertise this list to your users. The ones that are interested in contributing will have something quick and easy to get started on, and you'll have that many fewer bugs to fix - it's a win-win! Ideally, once you have a system like this in place, people will feel more comfortable about filing trivial bugs too, and you'll end up with higher-quality software overall.

Don't require registration to submit a bug report

Trac is kind of bad about this since everybody runs their own instances, though if I recall correctly recent versions support OpenID, which is a big step forward. If I find a bug, but I have to go through the whole registration dance just to report it, I'm just not going to bother reporting it unless it's a really severe bug. You may say that I'm not a serious contributor if I let a registration page stop me, but that's kind of missing the point. Casual contributors can be just as valuable as really serious ones, and there are a lot more of them out there.

Allow editing through a web interface

This one doesn't actually exist yet, but it could. The normal process of submitting a patch involves checking out the source code, making modifications, generating a patch, and submitting that to the maintainers of the program. 3/4 of those steps could be automated on a code hosting site such as SourceForge or github or bitbucket. The code could be cloned on the server; the user could be presented with a simple text editor in the browser (or something more fancy, like Bespin), the user could save the code with a commit message when they're done, and the commit could automatically be submitted as a pull request on the server. I wouldn't want to use this to make serious changes to the code, but this isn't designed for people that are already making serious changes. For somebody who's just making a small cosmetic change, this would be a huge timesaver, and that in turn increases the number of contributions you get.

NaBloPoMoTwoOh

To be honest, I'm not even sure that this counts.

The date today is October 14th, but I'm scheduling this post to appear on November 1st - thus starting into National Blog Posting Month 2009. I'm planning to write at least a week's worth of posts ahead of time, so that I'll have a chance of sticking to a schedule, because from back here it looks like November is shaping up to be a pretty rough month. It feels kind of like cheating, though - does it really count as NBPM if I'm not writing all the posts during the month?

On the one hand, it does kind of defeat the purpose of NBPM. The idea is to write one post per day, not just have one post appear every day. From that perspective, it looks very much like I'm cheating. >_>

But I've said this before, and it bears repeating: I'm not blogging so that people will read what I write, or for an annual event. I blog for purely selfish reasons. I need to keep in practice with writing, or I'm afraid that I'm going to forget one day, and lose a valuable skill. I've also found that a blog format is a great way to get my thoughts on a subject in order, and simultaneously put them on the Internet for future reference. Finally, as with my post from May about my research project, putting something on this blog occasionally saves me a lot of breath explaining things to people.

In that case, why go through all this rigamarole? Why bother with NBPM, when I could spend my time on other things that I desperately need to finish? Honestly, I just think it'll be fun. What other reason do I need?

(random parting thought: scheduled blog posts could be a really neat twist to a murder mystery :o)

hiatus

Okay, so. As much fun as this blog is to write, I need to turn in a draft of my honors thesis (average length: 20-30 pages) in just over a month, and I've so far written not a word of it. The next month is going to be kind of a pain, I think, so I'm probably better off not updating for a while.

I'm still doing NaBloPoMo this year, though. :3 See you again in November!

Fun with find

The find command is incredibly useful, if a bit arcane, so it's a shame that more Linux users aren't aware of it. Basically, if what you're trying to do sounds like "search for files with these characteristics, and do something to them", then find is probably the tool you're looking for.

Let's start with the very basics. If you run find with no arguments, it'll start listing all the files it can find. This isn't especially useful behavior, but let's take a closer look at it. find takes a list of paths, and looks at all files contained in each of them - if you don't give it any, it'll assume the current directory. We can introduce a bit of bash-fu to start doing something that looks like it might be useful:

$ find ${PATH//:/ }

This will list all the programs that are available on your path. (The weird-looking variable reference just replaces colons with spaces - look here if you're curious.)

Just listing files is no fun, though. You can do that already, with ls, which also has the advantage of being a few less letters to type! So let's start getting into the real power of find, with expressions.

After the paths to search on, you can specify any number of expressions - basically, filters that look at the list of files and only select the ones matching some criteria. One simple one is -name:

$ find /usr/portage -name ChangeLog | wc -l

(If the bar thing looks funny to you, you need to read up on pipes. If you don't know pipes, you can't really say you know how to use the command line, they're that important.) This is a command I used just a few hours ago, to find out how many ChangeLog files there are in Gentoo's portage tree. Without the find command, this would have been kind of a pain. find also has a -iname filter, that does a case-insensitive match - useful if you're looking for files that have inconsistent capitalization.

There are a lot of other possible filters, too many to list here, so you'll have to read the find man page to see them all. Here are just a few examples:

$ find ${PATH//:/ } -name "mkfs.*"
This is the earlier example, but with a twist - this prints out the full path to programs matching a given pattern. (If you only want one program, the which command is easier, though.)

$ find ~ -empty
This lists all empty (zero-length) files in your home directory.

$ find / -user root
This will list all files owned by root. (You probably have to be root for this to actually list all of them, for obvious reasons.)

$ find / -size +500M
This finds all files on your system larger than 500 megabytes, and requires some explanation. Filters that take numerical arguments can usually also take a + or - modifier, to mean "greater than this" or "less than this". If you leave it out, then you can search for files that have some exact size.

$ find ~ -mmin -30
List all the files in your home directory that were modified in the past 30 minutes. (No more wondering about where you saved that important file!)

$ find /usr/bin -not -executable
There shouldn't be any non-executable files there, but I found one on my system - probably a bug in the package that installed that file. (Want more logical operators? You can stick a -or between two filters and find will return the file if it matches either of them.)

"But wait," you might be thinking. "You said find would look for files and let me do stuff to them, but listing them isn't terribly interesting!" Don't worry, the fun is just beginning. :D

The simplest way to get find to do stuff with files is not to use find at all: pipe the output to xargs instead. For most simple tasks, this is way easier than using find's execution capabilities. The following three commands do basically the same thing:

$ find ~ -size 0 | xargs rm
$ find ~ -size 0 -exec rm "{}" +
$ find ~ -size 0 -delete

The first one just pipes the list of files to xargs, which is a nifty little utility that runs the command it's given on each filename it gets through the pipe. In this case, it runs rm and deletes all the files it's passed, but you could use any command there.

The second one uses find's -exec option, which gives you more control over how the command is constructed. After the -exec, you find rm, which is pretty self explanatory - it's the command you're executing. The "{}" thing is find's weird way of saying "the file that was found" - this is where the filename gets substituted into the command. The + ends the command, but there's a twist here. If you end the command with a semicolon instead, find runs the command once for each file. (NB: you have to put the semicolon in quotes or bash messes with it. This took me forever to figure out :( ) If you use +, on the other hand, it has the same effect as far as terminating the command, but it also tells find to jam as many filenames as it can in there, subject to whatever limitations the OS imposes. For large file lists, this can be the difference between your command running thousands of times or just a few times, so using + wherever possible is a good habit to get into.

The third is mainly for completeness - find has a builtin function for deleting files, making this example a bit pointless. :)

Here are a few more practical examples.

$ find /usr/portage -name ChangeLog -exec du -c "{}" + | grep total
I used this to find the total disk space on my system taken up by ChangeLog files in the portage tree. "du -c" will print out the total disk space used by all the files you give it, and the grep filters the output down to just those totals.

$ find -type d -exec chmod 755 "{}" +
Somehow I had a pile of directories on my NFS share that had no execute permissions for all users, so other users couldn't even enter those directories. This fixed all that in a single command.

$ find ${PATH//:/ } -perm -4111 -user root
Shows you all binaries available on your path that are suid root. These can be serious security risks if the programs are written insecurely.

$ find -mtime +365 -exec mv "{}" archive/ +
Moves all files that haven't been modified in more than a year to another directory.

$ find -nouser -exec chown root "{}" + , -nogroup -exec chgrp root "{}" +
Find all files that are owned by a nonexistent user or group, and change that ownership to root. Note the comma in there; it splits up the expression so that you can operate on multiple sets of files in a single find command, and only have to actually scan the directory tree once. If you want to do something like this in the absolute fastest way, find is your friend.

That's about the limit of my knowledge, but the find man page has loads more information, as well as some more examples.