I'm hacking on some tools that use the Search::Xapian module to build up search indexes. It's an excellent Perl interface to Xapian, but unfortunately it seemed to be too slow for our purposes. Tracing our code showed that much of the slowness was in passing data back and forth between Perl and the C++ library for every call.
I decided to write my own XS module to speed things up. Instead of using Search::Xapian, I'd bundle everything up into a Perl datastructure, and pass it down to libxapian through my own module, once, and do all the indexing work in C++. This worked great -- until I started trying to do some exception handling.
I'd much rather use Git over Subversion, but I still have one project (leaguerunner) using Subversion on Google Code. Migrating the code itself to git is fairly simple, but being unable to migrate the issues still needs some tool development.
You may have heard that WIND Mobile is going to be offering service in Ottawa shortly. I got curious as to what their coverage might be and started looking for coverage maps. I didn't find any, but I ended up graphing their wireless backbone instead.
A while back, I posted about detecting virtualization on Linux from a shell, using various tricks. Around that time, I also implemented those tricks as a perl script for internal use at work with the intent of eventually cleaning it up for public consumption. I finally got around to doing it, and the result is Sys::Detect::Virtualization availabe from CPAN, or from GitHub.
Currently, it can only handle Linux, and is only tested on a small number of virtualization hosts. Patches are welcome.
Until today, this blog ran on vimblog, a hand-rolled minimal script for displaying blog entries. In the last year or so, though, I've become a convert to git, and so what I really want to do is edit my posts on any system, commit them to a git repository, push to a remote and have them end up as blog entries on my server. Fixing vimblog to do this would have been more work than I want to deal with, but thankfully ikiwiki exists, and can do most of what I need.
So, as of now, this blog is in ikiwiki. To get there, I followed (more or less, since I'm documenting after-the-fact) these steps:
I wanted a command-line-editable blog, so in yet another fit of wheel reinvention, I rewrote bart's PHP blog in more maintainable Perl.
It still needs some features, like: - rss feeds - other formatting types (ie: wiki formatting, markdown)
but it works.
I'm not sure why exactly, but I keep wanting to add more features to this silly blog script. It started out as just a quick, featureless flat-file based blog tool, written to a) let me maintain a blog with vim, and b) prove to myself that I can still write small, maintainable Perl hacks.
It's still small (under 500 lines, including all the HTML and CSS templates), but it has oh so many features:
- support for multiple content-types in posting (currently plain text, HTML, and Markdown)
- articles stored in plain text for easy editing
- per-post tagging, and sidebar tag cloud
- full-content RSS feed
- permalinks to individual articles
- configurable per-article link naming
- view articles by year, month, or day
- view articles by tag
I'm hoping I run out of stuff to add soon. One of my main goals is to keep this as a small one-file Perl script, and there's only so far you can go with that sort of limitation. If it gets any bigger, I may as well just give up and use Catalyst.
So, I went to YAPC::NA 2006 Sunday through Wednesday this week. As I get time to transcribe my notes I'll post more details here. There were several interesting talks, but I didn't learn anything groundbreaking. However, the Perl Email Project seems to be getting off the ground again, so I'm happy about that. I also met the author of Rubric, which has inspired me to clean up my pile of Rubric hacks (feed aggregator, IRC integration, etc) for submission to CPAN.
More to come.
One of the RSS feeds I read regularly is of newly uploaded CPAN modules. Today, I saw one worthy of The Daily WTF: File::ReadSimple.
rjbs has been trying to improve Email::Simple's memory consumption, so I figured I'd give his test a try on MIME::Parser, just for the heck of it.
It's not a very good test for MIME::Parser, because Email::Simple doesn't itself do most of the stuff MIME::Parser does, but it's something.
Given the numbers he posted, for Email::Simple, I expected MIME::Parser to be much worse, but instead, for an 8MB message with 50 headers I get:
$ perl ./readmail-mimetools big.msg just started : 1556 3384 after require File::Slurp : 2564 4180 after slurping : 18204 19812 after require MIME::Parser : 21928 23460 after construction : 37684 39236
So... the code over head for MIME::Parser is about 3.7MB, and we take just under 2x the message size for internal representation.
Trying again, with MIME::Parser's tmp_to_core and output_to_core turned off (this makes it litter your filesystem with temp files named "msg-22195-1.txt" for each MIME body) it's a bit better:
$ perl ./readmail-mimetools-noncore big.msg just started : 1560 3384 after require File::Slurp : 2564 4180 after slurping : 18204 19812 after require MIME::Parser : 21928 23460 after construction : 29832 31412
So, what does this prove? Well, MIME-tools doesn't suck as bad as I thought. That's about it.