The Right Language For The Job

The right language for the job is ... all of them. All of them at once. I want a garbage collected, bounds checked language that lets me drop down to C-level memory fiddling, runs at over 90% of the speed of plain-old-C and interprets code I type into a prompt. I want a language that checks types at compile time, except when I'm just tossing off some small quick thing. I want a functional programming language ... for about 20% of what I write, object orientation for 30% of my code, and miscellaneous glue of no particular dogma for the rest. I want a language that is terse but explicit and unambiguous. And finally I want a large standard library that helps me get done most of the common tasks I do these days, with a vast community supported set of libraries that cover even more (and most of those libraries have no dependency bloat and depend on nothing more than the standard library).
Did I forget anything?

Developing Portably, For the Future

If I were to start a new project today, what language and environment would I write it in? (Assuming I'm not targetting an environment that makes the choice for me (Android, iOS, browser-side javascript).)
So, let's say I'm doing some data processing or some web serving. I'm writing a command line tool or daemon. I think I'd like it to be portable between my Mac and Linux machines that I own. I'd like to not regret my choice of environment if I'm still using this thing 1-5 years from now.
What's in?
C/C++, Python. I am almost ready to concede adding JavaScript to this list, but I don't actually like the language, though I'm fine with using it when I need to for getting things done in browser UIs. These languages I believe will be solidly supported into the future and remain free of deathspiralling suck. The Python 3 transition is going slowly, but I think it could yet turn out okay. I wish Python had static type checking. C++ has lots of stuff added in the last 10 years that I don't want to use.
What's out?
Java - Oracle seems to be killing it. Damn shame, a few years ago I called it my favorite language.
C# - Microsoft actually made a decent language, I've been using it at work lately, and they did some decent things about pushing it into an open-ish standard. But I don't see any compelling reason to use it and I haven't heard that I can rely on the Mono framework to be reliable now and into the future.
PHP, Ruby - I hear nothing but grief about using these. I dislike PHP as a language and an environment. It's amazing that things like mediawiki and drupal are built on it, but I'm disinclined to hack on them because of PHP.
perl. Been there done that. I use Python now.
Any functional language. They make hard what is easy in other languages.
What's left?
Go - Google has a cute little language there. It has my favorite features checked off like: compiled, bounds-checked, and garbage-collected. It's BSD licensed, so no one company can kill it the way Oracle is doing to Java, but I feel like it's not ready yet. Also it's a kinda weird little language and there are a few major language features that I feel if go had them I would be much more productive when using it. Another major language revision, and some growth in the community library support, and hopefully it'll be better a year from now.

Archive Random Read Performance in Python

I was curious about the performance of random read access of archive files in Python. I knew ZipFile was okay, and I was curious how it would perform compared to an sqlite3 database, and I threw in Tarfile (gzipped) for extra curiosity. The motivation for this was the pytz package and its nearly 600 small read-only data files. In one system in the past it was annoying to install and manage this menagerie, and in a fit of premature optimization I figure it ought to be better to read from one archive file anyhow, so I scratched this curiosity itch and found out some things.

In short:
Zip archives are pretty good.
Gzipped tar files are smaller than zip archives, but much slower to read (at least in the Python implementation). Sqlite3 databases are slightly faster to read than zip files, and a little bit faster still if you don't compress the data.
Sqlite3 databases are much faster to read the first record. Repeated reading from an already open zip archive or sqlite3 database are relatively close in performance.

Here's some data:

The test file generated 1000 files with names between 5 and 100 characters long and data between 50 and 5000 bytes long (uniform distributions, random filler data).
I cut off each test at 10 seconds, so the slower methods wound up with fewer reads done. The test data was made up of random data which was pretty uncompressable. So, everything was trying to compress the data, and decompress it on read, but the resulting archive file sizes were all pretty consistent. On real data (the pytz files) tar.gz can be half the size of zipfile or sqlite3 (which are similar though sqlite is slightly larger). The reopen=TRUE trials re-open the archive file for read before each random read. So the time reported is to open the file and read one random record. Otherwise the archive is opened once and read many times for all the random reads. This makes a huge difference in performance, but sqlite3 has the best archive startup time.

I have a mercurial archive of the sources I wrote to test all this here:
http://bolson.org/~bolson/sqlitearchive
There is also a utility to convert a directory of files into a sqlite3 database that maps paths to BLOBs for each file.

Git is backwards, sometimes

hg diff -r from -r to
svn diff -r from:to
git diff to..from
Let's say "from" is 1969 and "to" is 2010. I think normally I'd write the earlier thing on the left and the later thing on the right. It's a culture bias of our writing system that I assume.
If I run:
git diff 41af332c2c071d941a0aa90b963e4e499e6b16ed 730c38d1a5c7a35b23245c7f4acb03c292ea0c03
and 41… is earlier than 73… then pretty much what I expect happens. I get a forward diff.
But if I run:
git diff master mybranch
I get a reverse diff, which if applied would undo all the changes in mybranch. I have to run it backwards, `git diff mybranch master` in order to get a forward patch I could use to apply my changes. Why?
I can `svn diff ${repo}/trunk ${repo}/branches/mybranch` and get what I want.
I can `hg diff -r otherbranch -r mybranch` and get what I want.
git is backwards, but worse, it's backwards in this one seemingly random case. WTF.
I see git gaining popularity, and I see git sucking, and I hate that. I hate lousy technology winning. I hated that when it was Windows, or 386, or anything. Git sucks. Use Mercurial or Subversion.

Google will never do small projects, and doesn't see why you should either

Google AppEngine announced a new pricing model and the killer is that as soon as an 'app'* costs anything it costs $9 per month. The implication to me is clearly this: never do anything small. This probably won't be a barrier to people building a business on AppEngine, they're probably planning on moving many more dollars per month than $9. But I think it will be a barrier to tinkering and creativity.

In some sense 'never do anything small' could be good advice, but in another sense it is a path to missed opportunity. I have criticized Google's current corporate culture as being unable to do anything small. If they can't service 10000 users on opening day and a million within 6 months they're not interested (and really I think they'd be much more interested in 10 million). I think this means they'll never do anything truly new that would have to go through a phase of being small and experimental and unfinished and creative. Reddit or Twitter or Craigslist won't start there. And Google won't try to. They might acquire the next thing like that, but they won't start it.

I now see this culture polluting AppEngine. Under this new regime, I would not start a new AppEngine product speculatively just to see if something worked. I would not experiment there. Making a business decision, it is still viable, but it's not a place for creativity. Now, I'd almost certainly rather go with an Amazon Web Services EC2 micro instance or mini instance. I have one. It's great. It will host any number of 'apps' for the flat rate per month I'm already paying. Great. I can run any software I want and play there and try things.

(* What is an 'app'? It's a DNS name. reset.appspot.com or www.comicchopshop.com are a couple examples. Now, I can kinda cheat this and build any amount of functionality under that at /foo and /bar, but there are limits to how far it is a good idea and good design to go with that.)

Firmware Update on GoogleIO Samsung Galaxy Tab 10.1"

I went to GoogleIO and they gave me a Samsung Galaxy Tab 10.1"
It's a pretty nice piece of hardware.
Too bad about some of the software.
I have an Android phone to compare to, and in some ways it is better than the tablet. The phone browser has a better bookmark manager.
Today I discovered the lameness of Samsung's firmware update procedure.
1. Go to Settings: About Tablet: Software Update
2. Manually click the button for it to check for updates
3. Make an account with samsung (okay, sure have my email and password I don't care about)
4. click through a few other things to make it actually download the update
5. click through a couple more things to actually do the update while it warns you that you will not be able to make phone calls (not a phone, wifi only) and that the 'phone' will reboot.
6. and here's the part that failed for me twice: the tablet should not be plugged into USB. It will simply fail to try to apply the firmware update with USB plugged into my computer (maybe logicless power would work). So, with 80% charge I called it good enough and ran the update unplugged. It took a couple minutes for the very slow progress bar to pass the firmware update. Actually I wrote this whole blog post while waiting for it to do that. Hmpf.

Election Reform, IRV and Politics

I don't like Instant Runoff Voting, but I'm a little sad that Burlington VT repealed IRV election of their mayor. Sure, their second IRV run election was a flop, where three different counting methods could find three different winners, demonstrating that all of the anti-IRV FUD, dismissed as vaguely possible mathematical oddities, could actually happen in the real world. Still, I'm a little sad.

I want to see 'election reform', even if it's IRV, go forward and spread. I'll yell and scream at every stage I can that if we're going to do it we should do it right and use something better than IRV, but if that's the compromise I get I'll take it. And I'm afraid that because Burlington had a bad experience with 'election reform' (really all IRV's fault, IMFO), all such efforts will be tarnished. Every establishment politician who wants to raise Fear, Uncertainty and Doubt will be able to point at Burlington and make spooky noises about what terrible things could happen.

I've been focusing on redistricting lately, because programming that has been the more interesting puzzle and because the 2010 Census is making it timely, but I still think that getting people voting on rankings and ratings ballots could be the biggest thing to happen to Democracy since the US Constitution. Making that change may have just gotten a little harder because some people did it badly. (Which does not bode well for the current pretty-bad-compromise Health Care bill. :-/ )