Anyway, I first thought, “I could just make copies of that for each account.” The I could call, “twitter-swelljoe” for my personal account and “twitter-virtualmin” for my business account. But that seemed wrong, somehow.

So, I wrote a version that accepts a command line argument of the username, and then sends the rest as a tweet.

#!/bin/bash
USER=$1
shift
TWIT=$@
PASS=password
curl -u $USER:$PASS -d status="$TWIT" http://twitter.com/statuses/update.xml>/dev/null

Nothing really fancy going on here, but the “shift” is saying, “drop the first token from the arguments list”, and then $@ says “give me the rest”. So, no quotes required for the tweet. I saved the filed as tw, and use it like so:

tw swelljoe Tweeting from the command line!
tw virtualmin Business-y kinds of tweets from the command line.

This assumes one password for all Twitter accounts, which is exactly what I do anyway (I use SuperGenPass, which always generates the same password when given the same URL and passphrase). You could instead store the passwords in an associative array (gotta be using Bash 4.0+ for that), something like this (I can’t test this, as I don’t have bash 4.0!):

#!/bin/bash
USER=$1
shift
TWIT=$@
 
declare -A PASS
PASS[swelljoe]="passwd"
PASS[virtualmin]="notapasswd"
 
curl -u $USER:${PASS[$USER]} -d status="$TWIT" http://twitter.com/statuses/update.xml>/dev/null

There are ways to fake associative arrays in older versions of bash, or use other tools, but since I don’t need multiple passwords, I’m going to just hand-wave that away.

I find it amusing that the first version of this twitter client, at 137 characters, is a valid tweet. It makes me want to go golfing and see how small I can make an even more functional version…at the very least, I can’t resist:

tw swelljoe `cat tw`

And then I have to go change my password after realizing I tweeted the real version rather than the demonstration version. Good thing nobody follows my personal tweets.

Someone over on Hacker News asked me what I thought about the new G1 (aka “Google phone”), as compared to the iPhone (which Jamie wrote about here).  I figured those comments would be generally useful to folks thinking about picking between the two coolest phones available right now.  I’ve now had a full day to play with it, and I’ve spent several days tinkering with iPhones, so here’s my off-the-cuff “review” of the new G1.

The hardware is mildly disappointing. Not as nice “feeling” as the iPhone (though the 3G iPhone also now has a plastic back and doesn’t feel as nice as the first-gen iPhone, so I guess things are tough all over), or even as solid feeling as my old Sidekick. When taking the back off to put in the battery and SIM card, I felt like it was going to break. It didn’t, but it felt like it was. Likewise for the little covers for the data/charge port and the SD memory slot…they’re plastic and tiny and feel fragile. The one exception is the keypad, which feels very nice to me.  But, keeping in mind that the G1 is dramatically cheaper than the iPhone 3G ($179 up front and cheaper per month for the plan–though I haven’t actually looked at different plans, I just kept the one I’ve been on, which is about $10 cheaper than iPhone–so $240 over 2 years), I think it’s a great buy, and a whole lot of hardware for a very small price.  I think I probably would have been happy to pay $20 more for slightly sturdier construction, though, particularly on the port cover…I have a bad feeling that it’s going to get broken long before I’m ready to upgrade to a new phone.

Price wasn’t my primary deciding factor (openness was), but it certainly didn’t hurt that I wouldn’t have to pay more per month to upgrade to the G1, while I’d pay out $240 more over 2 years for the iPhone 3G for the same service. I’m going to see if I can drop down to a smaller voice plan–the Sidekick I had before had minimum plan requirements, but I didn’t see any such requirements when signing up for the G1, so I might even be able to save $5 or $10 per month with the G1 over the Sidekick, since I rarely use the phone. I use maybe 100 minutes per month of a 1000 minute plan.

Software-wise, it’s plain awesome. The lack of two-finger gestures, as found on the iPhone, is somewhat disappointing, but it’s no slower to use the popup magnifier buttons, once you’re accustomed to it. Dragging and such is smooth and accurate (seeimngly more accurate than the iPhone, for me, but maybe it just feels that way because the keypad means that I don’t have to use it for typing fiddly stuff, as on the iPhone), so I guess the touch screen is pretty good quality. Since the Android developers are some of the same folks that developed the Sidekick, everything feels very intuitive to me (where, as usual, “intuitive” means, “what I’m used to”). It also has Linux underneath everything, so that may also be a “comfort” factor for me, I’m not sure.

Basic phone features work well and are easy to use, it sounds good and clear, and having Google mail, contacts, calendar, etc. is sweet (my old phone couldn’t handle more than about 500 messages via IMAP, and I get more messages than that in a week, and obviously GMail just works great with practically infinite mail). Web pages look great, and browsing is fast. WiFi was quick and easy to setup. YouTube videos work great, both on 3G and on WiFi. It’s my understanding that T-Mobile’s 3G network is still somewhat small…so if you don’t live in the valley, your mileage may vary, but it works fine for me here in Mountain View.

I installed an Open Source ssh client off of the web called ConnectBot; no jail breaking required, which was a big issue for me with the iPhone. I don’t want to have to have permission to install arbitrary apps that I’ve written or someone else has written. I also installed Compare Anywhere, and a bubble game from the app store, and the quality of everything is really slick. Really impressive for a launch day catalog, especially since everything is free right now. I haven’t spent a lot of time with the apps on the iPhone, so I don’t have much to compare to. But, I’m excited to play with more stuff from the catalog, and I think I’m going to try my hand at writing an app or two for the platform.

Also, it worked right away when I plugged it into my Linux desktop. No futzing around with weird stuff to get music onto the device. The iPhone/iPod is a bitch in that regard. That one thing made me ecstatic in ways I haven’t felt over a device in a long time. Coming off of years of messing around with iPods and an iPhone, and it never quite working right, having a drag and drop music experience is miraculous. Take this with a grain of salt, as I may be strange. I find iTunes incredibly confusing and difficult to use, so on those occasions when I’ve given up on getting Linux to work and rebooted into Windows, or borrowed a friends Macbook, and run iTunes for the purpose of putting music onto the device, I’ve ended up spending a long time futzing around anyway, because I never could figure out what all the syncing options meant or how to use them…so, every time I would fiddle until something happened, and occasionally it would just end up deleting all of my songs either on the PC or the device, and I would give up in disgust. I also have trouble using several other Apple software products, and find them hard to use, so I could just be too stupid to be trusted with a computer without some adult supervision.

I think it will be interesting to see how this battle plays out in the market, and I’m certain that the G1 is just the opening salvo.  Even if the G1 “loses” the battle against iPhone (if selling out all available units even before the launch date can really be considered a “loss” in anyone’s book), and fails to pickup significant market share, there will be another battle in a few months, and the battles will come faster and more furiously as other manufacturers adopt Android.  And each one will wear Apple down, and will open a new front on which Apple will either have to engage or ignore.  For example, very low end phones will come into existence next year that provide smart phone capabilities, as will higher end devices with more capabilities or special purpose capabilities to answer niche markets.  Apple can’t fight on all fronts, and every niche it loses strengthens the value of the Android platform to developers, and thus to end users.  If Android sucked, like Windows Mobile, this wouldn’t be a cause for concern for the folks at Apple.  But Android doesn’t suck.  It’s really nice to use.  As nice as iPhone?  Maybe not…but getting better rapidly (if you tried the last developer release a few months ago, you’ll know that there have been a lot of improvements since then).

As with PC vs. Mac two decades ago, it’s a battle of two different ideologies.  On one side, you have openness: the ability for hardware makers to produce widely varying hardware while still providing the same basic user interface; and on the other, you have a single source eco-system: Apple designs every aspect of its products and can control every element of the user experience, down to the very applications that run on the platform.  Except, this time around, there are two additional elements: the telcos, and Open Source.  While the telcos are going to fight to keep all cell phones locked down pretty tightly, and try to insure that they can extract money for just about everything novel you want to do with it, the Open Source nature of Android is going to allow people to do things with mobile devices that have been impossible to date (even on very powerful, but locked down, devices like the iPhone).

To me, it looks like Apple is going to make the same mistakes they made with the Mac years ago: Treating their application developers poorly by competing with them unfairly or simply locking them out of the market, disrespecting users that want to use their devices in ways not imagined by Jobs and treating them not as customers but enemies, and finally, denial that price is a major factor in peoples purchasing decisions.  If it plays out as it did in the PC wars, Apple will find that they have fewer applications for their devices, and a steadily declining market share (even as actual sales continue to increase, since the smart phone market is growing rapidly, and all ships rise in a growing market).  Since it isn’t Apple vs. Microsoft, this time, but instead Apple vs. Open Source (and openness in general), I know which side I’m on.  I’ve had a tendency to pull for the scrappy underdog in the past, and Apple has very frequently been the scrappy underdog…but unfortunately it’s an underdog with a Napoleon complex, so it’s not exactly a good choice for replacing the old tyrant.  But, luckily, this time around, we have a wide open alternative…and it’s actually really good and really easy to use.  Maybe the “Open Source on the desktop” movement, to date, has all been preparation for this moment…the moment when Open Source can finally be a great option for regular, non-technical, consumers.  I think that’s pretty exciting.  And we’ll just have to wait and see if Apple winds up on the wrong side of history, again.

A great article about Webmin showed up today at Free Software Magazine, by Gary Richmond, that is a “bird’s eye view” of the subject, just touching on the high points.  In it, he gets to the heart of some of the things that make Webmin great, like it’s ability to edit configuration files directly and safely, rather than generating them from templates, as most similar configuration tools do.  All in all, a nice little “boss-friendly, while still being accurate” introduction to Webmin.

And Monday, the awesome blog The Next Web covered Virtualmin, the business and the project, kicking off their new series on profitable web businesses (rather than those that are running on dreams and VC cash) with this article: Companies who make money: Virtualmin. Lots of great stuff over there.

And, of course, no discussion of the blogosphere would be complete without mentioning the new browser from Google. I haven’t tried it yet, as it’s only available for Windows and I’m a Linux user when I’m working, but Jamie tells me that Webmin works just fine (expected) and there was only one quirky bit in the Virtualmin theme (surprising, since we test against Konqueror and Safari, both WebKit browsers, but easily fixed). So, if you’re running Windows, give it a go: Chrome. If you’re an IE user, please give it a go.

Around the start of the year, I created a theme for Webmin designed to make it easier to access from mobile devices like smartphones and cellphones with simple built-in browsers. At the time I had a Treo 650, which had a very basic browser – certainly not powerful enough to render the standard Virtualmin or Webmin framed themes.

By using Webmin’s theming features, I was able to create a UI that used multiple levels of menus to access global and domain-level settings, instead of frames. The theme also changed the layouts of forms to be more vertical than horizontal, use fewer tables, and remove all use of Javascript and CSS for hiding sections.

This was released in the virtual-server-mobile theme package version 1.6, and all was good in the world. Anyone using it could now access all the features of Virtualmin from a very basic browser, and read mail in Usermin without having to rely on the awful IMAP implementations in most smartphones.

This shows what Virtualmin looked like on a Treo :

Then I bought an iPhone.

It has a much more capable browser, technically the equal of any desktop browser like Firefox or IE. The regular Webmin themes actually worked fine, although a fair bit of zooming is needed to use them. The mobile theme looked like crap, as it didn’t use any of the browser features like CSS and Javascript that the iPhone supports. Plus the layout rendered poorly due to the use of long lines of text that didn’t get wrapped at the browser’s screen width.

On the iPhone, the Create Alias page in mobile theme looked like this :

And in the regular Virtualmin theme, the Create Alias page looked like :

I mentioned this to Joe, and he pointed me at iUI, an awesome library of CSS and Javascript that allows developers to create websites that mimic the look of native iPhone applications. After trying out the demos and looking at their source code, it was clear that iUI would be perfect for creating an iPhone-specific theme.

It wasn’t quite as simple as I first thought, but after some hacking on both the theme code and iUI itself I was able to come up with a pretty good layout, as you can see in this screenshot of the Create Alias page :

Menu Implementation

Actually getting IUI to play nicely with the Webmin theming system was slightly more complex than I originally expected though. For example, an iPhone-style multi-level menu that slides to the left is implemented in IUI with HTML like :

<ul id='main' title='My Menu' selected='true'>
<li><a href='#menu1'>Submenu One</a></li>
<li><a href='#menu2'>Submenu Two</a></li>
</ul>
<ul id='menu1' title='Submenu One'>
<li><a href='foo.cgi'>Foo</a></li>
<li><a href='bar.cgi'>Bar</a></li>
</ul>
<ul id='menu2' title='Submenu Two'>
<li><a href='quux.cgi'>Quux</a></li>
<li><a href='#page'>Some page</a></li>
</ul>
<div id='page' class='panel' title='Some page'>
Any HTML can go here.
</div>

As you might guess, CSS and Javascript are used to show only one menu or div at a time, even though they are all in the same HTML file. This is quite different to the way menus are usually created in Webmin.

To get this kind of HTML from the theme, I created an index.cgi that generates a large set of <ul> lists and <div> blocks containing all the Virtualmin domains, global settings, Webmin categories and modules. This is loaded by the iPhone when a user logs in, and allows quick navigation to without any additional page loads. For example, these screenshots show the path down to the Users and Groups module. Only the last requires an extra page load :

The index.cgi script is able to fetch all Webmin modules and categories with the functions get_visible_module_infos and list_categories, which are part of the core API. It also fetches Virtualmin domains with virtual_server::list_domains and global actions with virtual_server::get_all_global_links.

For example, the code that generates the menus of modules and categories looks roughly like :

my @modules = &get_visible_module_infos();
my %cats = &list_categories(\@modules);
print "<ul id='modules' title='Webmin Modules'>\n";
foreach my $c (sort { $b cmp $a } (keys %cats)) {
    print "<li><a href='#cat_$c'>$cats{$c}</a></li>\n";
    }
foreach my $c (sort { $b cmp $a } (keys %cats)) {
    my @incat = grep { $_->{'category'} eq $c } @modules;
    print "<ul id='cat_$c' title='$cats{$c}'>\n";
    foreach my $m (sort { lc($a->{'desc'}) cmp lc($b->{'desc'}) } @incat) {
        print "<li><a href='$m->{'dir'}/' target=_self>$m->{'desc'}</a></li>\n";
        }
    print "</ul>\n";
    }
print "</ul>\n";

The actual IUI styling and menu navigation comes from CSS and Javascript files which are referenced on every page in the <head> section, generated by the theme’s theme_header function which overrides the Webmin header call.

Other Pages

Other pages within Webmin are generated using the regular CGI scripts, but with their HTML modified by the theme. This is done by overriding many of the ui_ family of functions, in particular those that generate forms with labels and input fields. Because the iPhone screen is relatively narrow, it is more suited to a layout in which all labels and inputs are arranged vertically, rather than the Webmin default that uses multiple columns.

For example, the theme_ui_table_row override function contains code like :

if ($label =~ /\S/) {
    $rv .= "<div class='webminTableName'>$label</div>\n";
    }
$rv .= "<div class='webminTableValue'>$value</div>\n";

The label and value variables are the field label and input HTML respectively. The actual styling is done using CSS classes that were added to IUI for the theme. The same thing is done in functions that render multi-column tabs, tabs and other input elements generated with ui_ family functions.

The only downside to this approach is that not all Webmin modules have yet been converted to use the functions in ui-lib.pl, and so do not get the iPhone-style theming. However, I am working on a long-term project to convert all modules from manually generated HTML to the using the UI library functions.

Headers and Footers

In most Webmin themes, there are links at the bottom of each page back to previous pages in the heirarchy – for example, when editing a Unix group there is a link back to the list of all groups.

However, IUI puts the back link at the top of the page next to the title, as in native iPhone applications. Fortunately, CSS absolute positioning allows the theme to place this link at the top, even though it is only generated at the end of the HTML. The generated HTML for this looks like :

<div class='toolbar'>
<h1 id='pageTitle'></h1>
<a class='button indexButton' href='/useradmin/index.cgi?mode=groups' target=_self>Back</a>
<a class='button' href='/help.cgi/useradmin/edit_group'>Help</a>
</div>

The toolbar CSS class contains the magic attributes needed to position it at the top of the page, even though the theme outputs it last.

Refactoring for Clarity and Brevity

Despite the changes, so far, the code is pretty much as it was when we started. A little more verbose due to the changes for strict compliance, so that’s a negative, and the addition of a main function and the oschooser wrapper just adds even more lines of code. The main code block was already a little bit long for comfort at several pages worth of 80 column text, assuming a 50 row editor window. Jamie seems capable of holding a lot of code in his head at once…me, I’m kinda slow, so I like small digestible chunks. So, let’s start digesting.

Looking through the code, this bit jumped right out. It’s a little bit unwieldy:

    if ($auto == 1) {
      # Failed .. give up
      print "Failed to detect operating system\n";
      exit 1;
      }
    elsif ($auto == 3) {
      # Do we have a tty?
      local $rv = system("tty >/dev/null 2>&1");
      if ($?) {
        print "Failed to detect operating system\n";
        exit 1;
        }
      else {
        $auto = 0;
        }
      }
    else {
      # Ask the user
      $auto = 0;
      }

It seemed like this could be shortened a little bit by making a have_tty function, and using && in the elsif. Not a huge difference, but if we then flip the tests over (there’s only four possible values of $auto and they only result in two possible outcomes) and add an || we can lose a few more lines, one conditional, and cut it down to:

    if (($auto == 3 && have_tty()) || $auto == 2) {
      $auto = 0;
      }
    else {
      # Failed .. give up
      print "Failed to detect operating system\n";
      exit 1;
      }

That’s a lot less code to read. I also think it makes more sense to have a single failure block and a single block setting $auto. I’m still trying to figure out the purpose of auto=2, since it seems like it would only be possible to fall back to asking a question, if there’s actually a TTY. But Jamie knows the quirks of various systems far better than I do, so we’ll leave it alone, for now, and keep the same behavior. I bet it’s to accommodate something funny on Windows!

I’ve also made a few tweaks, during this process, adding a package OsChooser; statement to the beginning of the file. I also discovered that $uname actually is being used in this code! It’s hidden inside the os_list.txt definitions. There are a couple of eval statements being used to execute arbitrary code on each line found in the OS list. Jamie must have been a Lisp hacker in a former life, with all this willy nilly mixing code and data. This is a pretty clever bit of code, but it took me a little while to grok, but now that I know what’s going on, we can solve some of the problems we had with testing detection of systems other than the one we’re running on.

In the meantime, just know that $uname has returned as an our variable, so that it can be “global” without ticking off strict.

More Tests

So, automated testing of an OS detection program isn’t a whole lot of good, if I can only test detection of one operating system (the one it happens to be running on right now). So, we need to introduce a bit more flexibility in where the OS-related data comes from. This is trickier than it sounds. UNIX and Linux has never standardized on one single location for identifying the OS. Many systems identify themselves in /etc/issue, while those from Red Hat use /etc/redhat-release (they also have a reasonable issue file, but I’m guessing it’s not reliably present or reliably consistent in its contents, as Jamie has chosen to use the release file, instead), and Debian has a /etc/debian_version file. Sun Solaris and the BSD-based systems seem to all use uname, and that’s just the popular ones. Webmin also supports a few dozen more branches of the UNIX tree, plus most modern Windows versions!

So, looking at oschooser.pl you’re probably wondering where the heck all of that extra stuff happens, because it doesn’t really have any detection code of its own. The answer is in os_list.txt, which is a file with lines like the following:

Fedora Linux     "Fedora $1" fedora  $1    `cat /etc/fedora-release 2>/dev/null` =~ 
/Fedora.*\s([0-9\.]+)\s/i || `cat /etc/fedora-release 2>/dev/null` =~ /Fedora.*\sFC(\S+)\s/i

This is a tab-delimited file. Why tabs? I have no idea, and it’s been a source of errors for me several times…even Jamie isn’t sure why he chose tabs as the delimiter, but that’s the way it is. It is plain text, plus numbered match variables, plus an optional snippet of Perl that will be executed via eval if it exists. This makes for an extremely flexible and powerful tool, if a wee bit intimidating on first glimpse.

So, that last field is the tricky bit. The thing I’m going to have to contend with if I want to be able to test every OS that Webmin supports, rather than just the one that happens to be sitting under the code while the tests are running. I’ll need a new argument to our oschooser function for starters, called $issue, which will generically contain whatever it is that os_list.txt uses to recognize a particular OS. On my Fedora 7 desktop system, that’s /etc/redhat-release, which contains:

Fedora release 7 (Moonshine)

So, oschooser now contains:

sub oschooser {
my ($oslist, $out, $auto, $issue) = @_;
...
}

Next, we need to make sure we keep the provided $issue if we got it, so we change this:

  # Try to guess the OS name and version
  if (-r "/etc/.issue") {
    $etc_issue = `cat /etc/.issue`;
    }
  elsif (-r "/etc/issue") {
    $etc_issue = `cat /etc/issue`;
    }
  $uname = `uname -a`;

Into:

# Try to guess the OS name and version
my $etc_issue;
if ($issue) {
  $etc_issue = `cat $issue`;
  $uname = $etc_issue; # Strangely, I think this will work fine.
  }
elsif (-r "/etc/.issue") {
  $etc_issue = `cat /etc/.issue`;
  }
elsif (-r "/etc/issue") {
  $etc_issue = `cat /etc/issue`;
  }

Note that $uname is defined earlier in the code now…and merely gets over-written if we’ve set the $issue variable in our function call.

And then we have to do something about the contents of the last field in os_list.txt before it gets evaluated. This is where it gets a little hairy. In the foreach that iterates through each line in the file testing whether we have a match or not, I’ve added a new first condition, so it now looks like:

foreach my $o (@list) {
  if ($issue && $o->[4]) {
    $o->[4] =~ s#cat [/a-zA-Z\-]*#cat $issue#g;
    } # Testable, but this regex substitution is dumb.XXX
  if ($o->[4] && eval "$o->[4]") {
    # Got a match! Resolve the versions
    $ver_ref = $o;
    if ($ver_ref->[1] =~ /\$/) {
      $ver_ref->[1] = eval "($o->[4]); $ver_ref->[1]";
      }
    if ($ver_ref->[3] =~ /\$/) {
      $ver_ref->[3] = eval "($o->[4]); $ver_ref->[3]";
      }
    last;
    }
  if ($@) {
    print STDERR "Error parsing $o->[4]\n";
    }
  }
  return $ver_ref;
}

Which performs a substitution on the last field, if it contains a cat command. It replaces it with the issue file that we’ve provided in the $issue variable. Thus, we can now pass in t/fedora-7.issue and put a copy of the /etc/redhat-release file mentioned above, and we’ll be able to test detection of Fedora 7, no matter what operating system the test is actually running on. I suspect we may run into trouble when we expand our os_list.txt to the full Webmin list, since I’m working with just the limited subset of systems the Virtualmin installer supports (or that I might support in the next year or so). I’ve made a comment in the code with XXX (merely a convention used in the Webmin codebase, though any odd sequence of characters that you’ll remember works fine…many folks use FIXME) to remind myself of this suspicion later if I do run into problems that this is the first place I ought to look.

After these changes, it’s possible to get serious about testing. So, I’ve added tests for a couple dozen systems, which was more Googling than coding due to the data-driven nature of my tests, and confirmed the new code is behaving identically to the old. Which means it’s time for…

More Refactoring

If you’ve been following along, you know that oschooser is still awfully long. A good tactic in such situations is to look for bits of functionality that can be pushed down into their own subroutines. One good choice is the parsing of patterns file at the very beginning of the function:

my @list;
my @names;
my %donename;
open(OS, $oslist) || die "failed to open $oslist : $!";
while(<OS>) {
  chop;
  if (/^([^\t]+)\t+([^\t]+)\t+([^\t]+)\t+([^\t]+)\t*(.*)$/) {
    push(@list, [ $1, $2, $3, $4, $5 ]);
    push(@names, $1) if (!$donename{$1}++);
    $names_to_real{$1} ||= $3;
    }
  }
close(OS);

This is a good place to start, because it only depends on one variable from outside the work, $oslist, which is the name of the OS definitions file. And, of course, file access is always a good candidate for abstraction…what if, some day, we want to pull these definitions from a database or a __DATA__ section? Having it all in one obvious location might be a win. For now, I just want that bloody long oschooser function to be a little bit shorter, so we’ll create this parse_patterns function:

sub parse_patterns() {
my ($oslist) = @_;
my @list;
my @names;
my %donename;
# Parse the patterns file
open(OS, $oslist) || die "failed to open $oslist : $!";
while(<OS>) {
  chop;
  if (/^([^\t]+)\t+([^\t]+)\t+([^\t]+)\t+([^\t]+)\t*(.*)$/) {
    push(@list, [ $1, $2, $3, $4, $5 ]);
    push(@names, $1) if (!$donename{$1}++);
    $NAMES_TO_REAL{$1} ||= $3;
    }
  }
close(OS);
return (\@list, \@names);
}

That’s not too bad, and it shaves about 13 lines off of oschooser at a cost of 3 or 4 more lines of function baggage in the whole file. The biggest irritant might be that I’m now passing around two array refs (one of which is already an array reference, so now we’ve got a reference to an array of references). I get confused when I use too many references, because I’m addle-brained that way, but these are only a little bit nested and so not too complicated, so I think future readers of the code should be fine. At least, no worse than they were before I got ahold of this script.

I’ve also converted %names_to_real to %NAMES_TO_REAL as it has become a package scoped global variable, and it’s considered good form to warn folks when they’ve come upon a global by shouting at them. Of course, I have another global, $uname, which I haven’t renamed to all caps, as one of my mandates for myself on this project is to require no changes to the Webmin os_list.txt. As I write this, I’m beginning to have second thoughts about $uname needing to be a global…so we’ll come back to that later.

Capturing the results of parse_patterns and dumping them out into @name and @list lets us run our tests again.

And More Refactoring Still

Things have improved a little in oschooser. It almost fits into two screenfuls on my 20″ monitor. But I think we can do better. I’m aiming for one page or less per function, in this exercise, so we’ve gotta keep moving. The next distinct piece of functionality I see is the automatic OS detection code, so I’ll add a new auto_detect function, something like this:

sub auto_detect() {
my ($oslist, $issue, $list_ref) = @_;
my $ver_ref;
my @list = @$list_ref;
my $uname = `uname -a`;
 
# Try to guess the OS name and version
my $etc_issue;
 
if ($issue) {
  $etc_issue = `cat $issue`;
  $uname = $etc_issue; # Strangely, I think this will work fine.
  }
elsif (-r "/etc/.issue") {
  $etc_issue = `cat /etc/.issue`;
  }
elsif (-r "/etc/issue") {
  $etc_issue = `cat /etc/issue`;
  }
 
foreach my $o (@list) {
  if ($issue && $o->[4]) {
    $o->[4] =~ s#cat [/a-zA-Z\-]*#cat $issue#g;
    } # Testable, but this regex substitution is dumb.XXX
  if ($o->[4] && eval "$o->[4]") {
    # Got a match! Resolve the versions
    $ver_ref = $o;
    if ($ver_ref->[1] =~ /\$/) {
      $ver_ref->[1] = eval "($o->[4]); $ver_ref->[1]";
      }
    if ($ver_ref->[3] =~ /\$/) {
      $ver_ref->[3] = eval "($o->[4]); $ver_ref->[3]";
      }
    last;
    }
  if ($@) {
    print STDERR "Error parsing $o->[4]\n";
    }
  }
  return $ver_ref;
}

You may note that I did rethink the globalization of $uname and found that it fit comfortably into this block, so I’ve killed a global introduced earlier in this process. Now I’ve not even sure why I thought I needed it somewhere else, which is a nice thing about refactoring: You realize how little you understood what was going on when you first looked at the code. Here’s also where we make use of the @list built during parse_patterns, and I dereference it before using it, though that’s probably more verbosity than needed. I could also access it directly within the ref with, @{$list_ref}.

Finally, I’m returning $ver_ref, which contains a reference to an array of fields that describes the operating system detected. Now that I’ve split this out, I realize that this OS version array could quite easily be mapped into a hash and turned into an object rather trivially, but that’s an exercise for another day. For now, I just want to feel confident that I’ve made a functionally identical clone of oschooser.pl that I can use and extend painlessly and without fear of breakage. So, let’s keep going.

A Few More New Functions, and Killing Unused Code Softly

As with auto_detect there is a big chunk of code that is used specifically for asking the user to choose the operating system and version from a list of options. This is triggered in the following cases: $auto is set to 0 or any other false value, $auto is not false but auto-detection failed and one of the non-exit auto options is chosen and viable. So, we can easily break out this whole bunch of functionality into its own function, called ask_user. Like auto_detect, it requires the $list_ref array reference, and it also needs the $names_ref, since it will be interacting with the end user and they’ll be more comfortable seeing the “real names” of the available operating systems. Also like auto_detect, it returns a $ver_ref which points to the array containing the full description of the OS.

When I got to this function, I noticed a huge block of unused code, which provides support for the dialog command on systems that support it (mostly just Red Hat based Linux distributions). dialog is a simple tool for adding attractive ncurses interfaces to shell scripts. I’m not sure why the code is being skipped with an if (0) statement, but I have only two choices for what to do about it, if my goal is to simplify this script and make it more robust: Enable it and fix whatever problems it has, possibly making it into its own reusable and independently testable function; or, simply remove the code altogether. Webmin and the installer libraries for Virtualmin are both in SVN. If I decide to remove the code, it won’t be lost forever…I could pull it back in the future. I could even tag the current version with “pre-dialog-removal” before stripping it out. After consulting with Jamie the last option is the one I’ve chosen. So, we can kill not just those pieces of code, we can also remove the has_command function, since it is only used in that part of the code. Big win!

So, I’ll make a tagged copy before ripping stuff out:

svn cp lib tags/lib/pre-dialog-removal

Now I know I can always go back and refer to that code if I want to. It’s not really particularly precious, but it’s a good practice to get into, since copies in Subversion are cheap and fast (likewise for git, and most other modern distributed revision control systems), and I never know when I might want to go back and see how something was done before. I’ll do the same in the Webmin tree before I make the changes needed to merge the new OsChooser.pm in place of the old oschooser.pl.

So, after killing the dialog pieces of the code, and converting the user interaction to its own function, we have:

# ask for the operating system name ourselves
sub ask_user {
my ($names_ref, $list_ref) = @_;
my @names = @$names_ref;
my @list = @$list_ref;
my $vnum;
my $osnum;
my $dashes = "-" x 75;
print <<EOF;
For Webmin to work properly, it needs to know which operating system
type and version you are running. Please select your system type by
entering the number next to it from the list below
$dashes
EOF
{
my $i;
for($i=0; $i<@names; $i++) {
  printf " %2d) %-20.20s ", $i+1, $names[$i];
  print "\n" if ($i%3 == 2);
  }
print "\n" if ($i%3);
}
print $dashes,"\n";
print "Operating system: ";
chop($osnum = <STDIN>);
if ($osnum !~ /^\d+$/) {
  print "ERROR: You must enter the number next to your operating\n";
  print "system, not its name or version number.\n\n";
  exit 9;
  }
if ($osnum < 1 || $osnum > @names) {
  print "ERROR: $osnum is not a valid operating system number.\n\n";
  exit 10;
  }
print "\n";
 
# Ask for the operating system version
my $name = $names[$osnum-1];
print <<EOF;
Please enter the version of $name you are running
EOF
print "Version: ";
chop($vnum = <STDIN>);
if ($vnum !~ /^\S+$/) {
  print "ERROR: An operating system number cannot contain\n\n";
  print "spaces. It must be like 2.1 or ES4.0.\n";
  exit 10;
  }
print "\n";
return [ $name, $vnum,
    $NAMES_TO_REAL{$name}, $vnum ];
}

Not too bad. It just fits into one 50 row editor window without scrolling, so that’s a small enough bite for me. We make use of the %NAMES_TO_REAL global in this function, to convert from the short names to the longer human-friendly names, and I’m beginning to get a vague feeling something could be done to encapsulate that functionality, even without making this an Object Oriented library (which seems like overkill for such a simple program), so I’ll probably be coming back to that global in a later post (and I thought I would have a hard time getting two full posts worth out of this exercise!).

Wrapping Up

I’m feeling pretty good about the code now. I think it’s more readable than before I started messing with it, it’s certainly shorter due to bits of refactoring and some removal of dead or redundant code, and it’s got quite a few tests. All of its variables are reasonably scoped to the areas where they are used, except for %NAMES_TO_REAL, which is a package scoped my variable (turns out eval gets the scope of the containing block, so it doesn’t need to be an our variable as I’d first assumed).

The various utility functions aren’t very useful to outsiders and may change…the only function I really want to be public is oschooser, so I can see several opportunities for further enhancements, like encapsulating the rest into private methods within an OsChooser object. But that’ll be a project for another day. You can see the current code, plus an example os_list.txt. Next time, I’ll begin work on wrapping this up for CPAN, and releasing a large OS definition list (Webmin’s list is incredibly long and detects hundreds of systems and versions, but needs a bit of massaging to be generically useful, due to its own internal version requirements).

Who knew one simple script could present so much interesting work? It’s been so much fun, I think I’ll start a Perl Neighborhood Watch and do this to every little Perl script I come across. Who’s with me!? (Or maybe I should just focus on our own code for a little while longer, since we’ve got quite a few nooks and crannies that haven’t seen any attention in years. Perl makes us lazy with its peskily perfect backward compatibility.)

So, when I found myself needing to make a modification to oschooser.pl, a small program for detecting the operating system on which Webmin is running (sounds trivial, but when you realize that Webmin runs on hundreds of operating systems and versions, it turns out to be a rather complex problem), I decided to take the opportunity to put into practice some of the niceties of modern Perl. This article is a little different than what I usually write for In the Box, in that it covers a lot of ground fast, and most of it is probably pretty mundane stuff for folks already writing modern Perl. But, I think there’s enough old Perl code running around out there, running the Internet and such, that it’s worth talking about modernization work.

So, let’s go spelunking!

Introduction to oschooser.pl

The code we’ll be picking apart, and putting back together, is probably one of the more heavily used pieces of Perl code, and certainly one of the oldest, in the wild. It’s the OS detection code that Webmin and Usermin use to figure out what system they’re running on during installation. With Webmin having 12 million (give or take several million) downloads over its ten year history, this equals a lot of operating systems successfully detected. Perhaps I should have picked something a little less important for my first stab at modernization, but I’ve rarely been accused of being smart about making sweeping changes! (Jamie will reel me in, before I break actual Webmin code. I manage to break Virtualmin every now and then…but he’s more suspicious when I check code into Webmin, since it happens quite rarely.)

The oschooser.pl program actually loads up a rather complex definitions file called os_list.txt (by default, though it’s configurable, and we use different lists for Virtualmin and Webmin, since they have different requirements for version identification). The definitions file can contain snippets of Perl code, which will be executed via eval, when appropriate. Most of the updates to OS detection over the years have happened in os_list.txt, so oschooser.pl hasn’t seen a lot of grooming over the years, which makes it a prime candidate for modernization. Assuming, of course, that it works identically when I’m done with it.

Where to start?

My end goal with this project is to make oschooser.pl usable as a library from Perl programs, since our new product installer is written in Perl rather than POSIX shell. I also figured it’d be nice to make it testable, since I’ve made several mistakes in the detection code (in os_list.txt, specifically) over the past few years that led to our product being uninstallable on some systems until the bug was tracked down. But, first things first. Almost nothing in Webmin is strict compatible, and even warnings can cause some complaints, so that seems like a good starting point.

The code we’re starting with can be found here, so you can follow along at home.

Enabling warnings reveals the following (don’t worry about the arguments for now):

$ perl -w oschooser.pl os_list.txt outfile 1
Name "main::uname" used only once: possible typo at oschooser.pl line 31.
Name "main::donename" used only once: possible typo at oschooser.pl line 17.

Not too bad, actually. Just a couple of variables that are only seen once, easy enough to fix by giving them a my declaration. Though, in this case, it looks like enabling warnings turns up some unused code. While donename is actually keeping track of what names we’ve seen, so far, and it’s one of several idiomatic ways to build an array of unique values, the uname variable seems to have no purpose. So I’m going to kill that whole line rather than declare it.

Next up in our “low-hanging fruit” exercise is enabling use strict. Turns out this is quite a lot more intimidating:

$ perl -c oschooser.pl
Global symbol "$oslist" requires explicit package name at oschooser.pl line 15.
Global symbol "$out" requires explicit package name at oschooser.pl line 15.
Global symbol "$auto" requires explicit package name at oschooser.pl line 15.
Global symbol "$oslist" requires explicit package name at oschooser.pl line 16.
Global symbol "$oslist" requires explicit package name at oschooser.pl line 16.
Global symbol "@list" requires explicit package name at oschooser.pl line 20.
Global symbol "@names" requires explicit package name at oschooser.pl line 21.
Global symbol "%names_to_real" requires explicit package name at oschooser.pl line 22.
Global symbol "$auto" requires explicit package name at oschooser.pl line 27.
Global symbol "$etc_issue" requires explicit package name at oschooser.pl line 30.
Global symbol "$etc_issue" requires explicit package name at oschooser.pl line 33.
Global symbol "$o" requires explicit package name at oschooser.pl line 36.
Global symbol "@list" requires explicit package name at oschooser.pl line 36.
Global symbol "$o" requires explicit package name at oschooser.pl line 37.
Global symbol "$o" requires explicit package name at oschooser.pl line 37.
Global symbol "$ver" requires explicit package name at oschooser.pl line 39.
Global symbol "$o" requires explicit package name at oschooser.pl line 39.
Global symbol "$ver" requires explicit package name at oschooser.pl line 40.
Global symbol "$ver" requires explicit package name at oschooser.pl line 41.
Global symbol "$o" requires explicit package name at oschooser.pl line 41.
Global symbol "$ver" requires explicit package name at oschooser.pl line 41.
Global symbol "$ver" requires explicit package name at oschooser.pl line 43.
Global symbol "$ver" requires explicit package name at oschooser.pl line 44.
Global symbol "$o" requires explicit package name at oschooser.pl line 44.
Global symbol "$ver" requires explicit package name at oschooser.pl line 44.
Global symbol "$o" requires explicit package name at oschooser.pl line 49.
Global symbol "$ver" requires explicit package name at oschooser.pl line 53.
Global symbol "$auto" requires explicit package name at oschooser.pl line 54.
Global symbol "$auto" requires explicit package name at oschooser.pl line 59.
Global symbol "$rv" requires explicit package name at oschooser.pl line 61.
Global symbol "$auto" requires explicit package name at oschooser.pl line 67.
Global symbol "$auto" requires explicit package name at oschooser.pl line 72.
Global symbol "$auto" requires explicit package name at oschooser.pl line 77.
Global symbol "$cmd" requires explicit package name at oschooser.pl line 80.
Global symbol "$i" requires explicit package name at oschooser.pl line 81.
Global symbol "$i" requires explicit package name at oschooser.pl line 81.
Global symbol "@names" requires explicit package name at oschooser.pl line 81.
Global symbol "$i" requires explicit package name at oschooser.pl line 81.
Global symbol "$cmd" requires explicit package name at oschooser.pl line 82.
Global symbol "$i" requires explicit package name at oschooser.pl line 82.
Global symbol "@names" requires explicit package name at oschooser.pl line 82.
Global symbol "$i" requires explicit package name at oschooser.pl line 82.
Global symbol "$tmp_base" requires explicit package name at oschooser.pl line 84.
Global symbol "$temp" requires explicit package name at oschooser.pl line 85.
Global symbol "$tmp_base" requires explicit package name at oschooser.pl line 85.
Global symbol "$cmd" requires explicit package name at oschooser.pl line 86.
Global symbol "$temp" requires explicit package name at oschooser.pl line 86.
Global symbol "$osnum" requires explicit package name at oschooser.pl line 87.
Global symbol "$temp" requires explicit package name at oschooser.pl line 87.
Global symbol "$osnum" requires explicit package name at oschooser.pl line 88.
Global symbol "$osnum" requires explicit package name at oschooser.pl line 88.
Global symbol "$osnum" requires explicit package name at oschooser.pl line 89.
Global symbol "$name" requires explicit package name at oschooser.pl line 96.
Global symbol "@names" requires explicit package name at oschooser.pl line 96.
Global symbol "$osnum" requires explicit package name at oschooser.pl line 96.
Global symbol "@vers" requires explicit package name at oschooser.pl line 97.
Global symbol "$name" requires explicit package name at oschooser.pl line 97.
Global symbol "@list" requires explicit package name at oschooser.pl line 97.
Global symbol "$cmd" requires explicit package name at oschooser.pl line 98.
Global symbol "$i" requires explicit package name at oschooser.pl line 99.
Global symbol "$i" requires explicit package name at oschooser.pl line 99.
Global symbol "@vers" requires explicit package name at oschooser.pl line 99.
Global symbol "$i" requires explicit package name at oschooser.pl line 99.
Global symbol "$cmd" requires explicit package name at oschooser.pl line 100.
Global symbol "$i" requires explicit package name at oschooser.pl line 100.
Global symbol "$name" requires explicit package name at oschooser.pl line 100.
Global symbol "@vers" requires explicit package name at oschooser.pl line 100.
Global symbol "$i" requires explicit package name at oschooser.pl line 100.
Global symbol "$cmd" requires explicit package name at oschooser.pl line 102.
Global symbol "$temp" requires explicit package name at oschooser.pl line 102.
Global symbol "$vnum" requires explicit package name at oschooser.pl line 103.
Global symbol "$temp" requires explicit package name at oschooser.pl line 103.
Global symbol "$vnum" requires explicit package name at oschooser.pl line 104.
Global symbol "$vnum" requires explicit package name at oschooser.pl line 104.
Global symbol "$temp" requires explicit package name at oschooser.pl line 105.
Global symbol "$vnum" requires explicit package name at oschooser.pl line 106.
Global symbol "$ver" requires explicit package name at oschooser.pl line 110.
Global symbol "@vers" requires explicit package name at oschooser.pl line 110.
Global symbol "$vnum" requires explicit package name at oschooser.pl line 110.
Global symbol "$dashes" requires explicit package name at oschooser.pl line 114.
Global symbol "$dashes" requires explicit package name at oschooser.pl line 115.
Global symbol "$i" requires explicit package name at oschooser.pl line 121.
Global symbol "$i" requires explicit package name at oschooser.pl line 121.
Global symbol "@names" requires explicit package name at oschooser.pl line 121.
Global symbol "$i" requires explicit package name at oschooser.pl line 121.
Global symbol "$i" requires explicit package name at oschooser.pl line 122.
Global symbol "@names" requires explicit package name at oschooser.pl line 122.
Global symbol "$i" requires explicit package name at oschooser.pl line 122.
Global symbol "$i" requires explicit package name at oschooser.pl line 123.
Global symbol "$i" requires explicit package name at oschooser.pl line 125.
Global symbol "$dashes" requires explicit package name at oschooser.pl line 126.
Global symbol "$osnum" requires explicit package name at oschooser.pl line 128.
Global symbol "$osnum" requires explicit package name at oschooser.pl line 129.
Global symbol "$osnum" requires explicit package name at oschooser.pl line 134.
Global symbol "$osnum" requires explicit package name at oschooser.pl line 134.
Global symbol "@names" requires explicit package name at oschooser.pl line 134.
Global symbol "$osnum" requires explicit package name at oschooser.pl line 135.
Global symbol "$name" requires explicit package name at oschooser.pl line 141.
Global symbol "@names" requires explicit package name at oschooser.pl line 141.
Global symbol "$osnum" requires explicit package name at oschooser.pl line 141.
Global symbol "$name" requires explicit package name at oschooser.pl line 142.
Global symbol "$vnum" requires explicit package name at oschooser.pl line 146.
Global symbol "$vnum" requires explicit package name at oschooser.pl line 147.
Global symbol "$ver" requires explicit package name at oschooser.pl line 153.
Global symbol "$name" requires explicit package name at oschooser.pl line 153.
Global symbol "$vnum" requires explicit package name at oschooser.pl line 153.
Global symbol "%names_to_real" requires explicit package name at oschooser.pl line 154.
Global symbol "$name" requires explicit package name at oschooser.pl line 154.
Global symbol "$vnum" requires explicit package name at oschooser.pl line 154.
Global symbol "$out" requires explicit package name at oschooser.pl line 159.
Global symbol "$ver" requires explicit package name at oschooser.pl line 160.
Global symbol "$ver" requires explicit package name at oschooser.pl line 161.
Global symbol "$ver" requires explicit package name at oschooser.pl line 162.
Global symbol "$ver" requires explicit package name at oschooser.pl line 163.
Global symbol "$d" requires explicit package name at oschooser.pl line 170.
Global symbol "$rv" requires explicit package name at oschooser.pl line 172.
Global symbol "$rv" requires explicit package name at oschooser.pl line 174.
Global symbol "$d" requires explicit package name at oschooser.pl line 177.
Global symbol "$d" requires explicit package name at oschooser.pl line 178.
Global symbol "$rv" requires explicit package name at oschooser.pl line 178.
Global symbol "$d" requires explicit package name at oschooser.pl line 178.
Global symbol "$rv" requires explicit package name at oschooser.pl line 181.
oschooser.pl had compilation errors.

Wow! I think that might be more lines than the program itself. Luckily, it’s almost entirely unscoped variables. A quick pass over the code, adding my declarations to the obvious candidates, gets things looking a little better. One tricky bit is the $i loop variables used in for loops. We don’t want those to be declared several times in the code, and we don’t want them to leak out into the scope of the rest of the program. In modern Perl, this is no problem, as you can use the following:

    for(my $i=0; $i<@names; $i++) {
      $cmd .= " ".($i+1)." '$names[$i]'";
      }

And $i will be local to the for loop. I momentarily feared that I’d need to use an outer block to accomplish this, as Webmin needs to be compatible with quite old Perl versions (5.005, for core Webmin, unless Unicode support is needed, in which case 5.8.1 is required), but after downloading and installing Perl 5.005_4, I found that was an unnecessary precaution. The foreach loops can also make use of this convenient feature. If you do happen to be stuck with an even more ancient version that 5.005 (but still higher than 4)–though I can’t imagine how you could, as 5.005 is over nine years old–you can use the following:

  {
  my $i;
    for($i=0; $i<@names; $i++) {
      $cmd .= " ".($i+1)." '$names[$i]'";
      }
  }

Which provides similar private scope for the $i variable, at the cost of three extra lines.

Making it Testable

So far, I haven’t made any changes that are likely to break the code. It’s merely been cleanup and syntax tweaks. But, to accomplish everything I’d like in this exercise, we’ll be doing some refactoring and refining the code. To do that with confidence, it’d be nice to have some tests to insure the code works the same before and after any changes.

Since this is not historically a library, it’s not particularly easy to test. One could write a custom test harness, or use Test::Command, and test its behavior as a whole, but since it’s written in Perl and one of my goals is to make it useful as a library from Perl scripts, I decided instead to make it loadable as a module and use Test::More. A trick that’s very common in the Python world, but doesn’t seem as well-known amongst Perlmongers is a main function which is called if the script is executed independently rather than via use or require. The main function then calls whatever the script would normally do, optionally setting up variables or parsing command line arguments.

So, I added the following near the beginning of the file:

# main
sub main() {
if ($#ARGV < 1) { die "Usage: $0 os_list.txt outfile [0|1|2|3]\n"; }
my ($oslist, $out, $auto) = @ARGV;
oschooser($oslist, $out, $auto);
}
main() unless caller();  # make it testable and usable as a library

I also took this opportunity to add a simple usage message if the command is executed with fewer than two arguments (@ARGV, like all Perl arrays starts counting at 0). I also needed to wrap the main function of the script in a sub block, so that the script doesn’t do anything immediately if loaded as a library.

Make it a Module

Since I want to use this code as a library, I face a choice. The use statement is functionally equivalent to:

BEGIN { require Module; Module->import( LIST ); }

Which means, I suppose, I could keep the name oschooser.pl and use:

require 'oschooser.pl';

We don’t need BEGIN level assurance, since we have no prototypes in this library and only use simple subroutines. But, I find this a bit unsatisfying, since it’s no longer in common use amongst Perl developers, and use provides the ability to export functions explicitly. Test::More has both a use_ok and a require_ok function, so it’s irrelevant from a testing perspective. It’ll probably remain oschooser.pl in Webmin proper, and OsChooser.pm in my Virtualmin installer library, at least for the foreseeable future. Not really a lot of difference between the two.

Some Tests

So, now that we can call the library roughly the way we want, using use, it’s time to write a few tests to be sure things actually work after we begin making more sweeping changes.

We can start with simple compile tests (I usually call these types of tests t/return.t, as they just check to be sure the module returns without error on load and the functions within return the data type that is expected):

#!/usr/bin/perl -w
# These tests just check to be sure all functions return something
# It doesn't care what it is returned...so garbage can still pass,
# as long as the garbage is the right data type.
 
use strict;
use Test::More qw(no_plan);
 
use_ok( 'OsChooser' );
 
isa_ok(\OsChooser::have_tty(), 'SCALAR');
isa_ok(\OsChooser::has_command("cp"), 'SCALAR');

Hmm…OK, so we don’t actually have a lot to test yet, just a couple of utility functions (and I’ve even cheated a little and looked ahead to where I introduced a have_tty function, or this would be an even shorter set of tests). The most important function, oschooser, doesn’t know how to return anything very useful yet. It can only write out its findings to a file. But, since we’re always going to be creating that file, regardless of how nice the module usage becomes, we need to figure out how to test it anyway.

Unsurprisingly, there is already a full-featured module on CPAN for testing the contents of files, called, unlikely though it may seem, Test::Files. So, we’ll just grab that:

$ sudo perl -MCPAN -e shell
 
cpan shell -- CPAN exploration and modules installation (v1.7602)
ReadLine support enabled
 
cpan> install Test::Files
...

And then create as many Operating System definition files as we want in the t directory. We’ll just name them for the OS they represent. This is the kind of testing I love, because the actual test file will be extremely simple, no matter how many operating systems I want to test on:

#!/usr/bin/perl -w
use strict;
use OsChooser;
 
# Get a list of the example OS definition files
opendir(DIR, "t/") || die "can't opendir t/ $!";
my @files = grep { /\.os/ } readdir(DIR);
closedir DIR;
use Test::More qw(no_plan);
use Test::Files;
 
foreach my $file (@files) {
  $file =~ /(.*)\.os$/;
  my $osname = $1;
  my $outfile = "t/outfile";
  OsChooser::oschooser("os_list.txt", $outfile, 1);
  compare_ok("t/$file", $outfile, $osname);
 
  # Cleanup
  unlink $outfile;
}

I love data-driven software, and this is a fun little example of it. We can run as many tests as we want, merely by adding more OS data files–one with the “os” suffix to provide what should be output by oschooser and one to contain the file that oschooser would normally use to identify the OS (/etc/issue, among others), which isn’t yet supported, but I’ll talk about it in the next post. Speaking of being data-driven, I think it’d also be pretty nifty to get the test count from the @files array, rather than using no_plan, but because modules loaded with use are loaded early during compile time (in a BEGIN block, effectively) we don’t actually have anything in @files yet.

However, as mentioned, the oschooser function doesn’t yet allow one to specify the issue file to look at, so no matter how many definitions I provide, it’ll never be able to test anything but the OS the test is running on. Oh, well, for now we’ll just create one OS definition file that matches my current OS, and make it a priority to make the function more testable somehow, possibly via an optional parameter to oschooser.

Alright, so now that we have some rudimentary tests in place, we can break stuff with confidence! We’ll come back to testing again in the near future, since we’re leaving so much untested right now.

Plain Old Documentation

I’m going to take a quick detour now that we’ve got some basic tests in place. Testing is one practice that most developers agree makes for great code, and the other practice that most folks can agree on is documentation.

Since this is such a simple piece of code, and was intended exclusively for use during installation of Webmin and Usermin, Jamie never really documented it. Now that I’m forcing it to be useful in other locations, and having some fun giving it a modern Perl face lift, it’s as good a time as any to add some documentation. POD isn’t the only documentation format usable within Perl code, but it is, by far, the most popular, and it has lots of great tools for processing and testing coverage, so that’s what Jamie recently chose for use in documenting the Virtualmin API. It’s also easy to learn, and results in text that is pretty readable even before processing.

I’m not sure of the recommended practices for documenting scripts that work on both the command line and as a module, but here’s what I came up with:

=head1 OsChooser.pm
 
Attempt to detect operating system and version, or ask the user to select
from a list.  Works from the command line, for usage from shell scripts,
or as a library for use within Perl scripts.
 
=head2 COMMAND LINE USE
 
OsChooser.pm os_list.txt outfile [auto]
 
Where "auto" can be the following values:
 
=over 4
 
=item 0
 
always ask user
 
=item 1
 
automatic, give up if fails
 
=item 2
 
automatic, ask user if fails
 
=item 3
 
automatic, ask user if fails and if a TTY
 
=back
 
=head2 SYNOPSIS
 
    use OsChooser;
    my ($os_type, $version, $real_os_type, $real_os_version) =
       OsChooser->oschooser("os_list.txt", "outfile", $auto, [$issue]);
 
=cut

Pretty simple, but covers the basics.

Next Time

Unfortunately, the code is now longer and probably a little less readable than before! It’s probably more robust to changes, since it now has reasonably scoped variables. And it’s more friendly to others who might want to use it, due to the new documentation and the ability to use it as a library in Perl or as a command in shell scripts.

Next time we’ll start in on the refactoring, and we’ll also write some more tests. This is turning into a real challenge, due to the data-driven nature of the script, and the fact that it’s somewhat hardcoded to look for OS data in very specific locations. Since, a big part of what I want to test is in the os_list.txt file, we don’t have the luxury of just saying, “It’s configuration…we’ll just make a special version for testing purposes.” We’ll have to get far more clever.

Virtualmin, Inc. is sponsoring this years YAPC::NA, and Jamie and I will both be in attendance throughout the event. If you’re attending, drop us a line. We’ll have T-shirts (Virtualmin and VM2 logos in all sizes and a multitude of sweet colors!) and a bunch of Virtualmin products to give away.

The reason this is interesting, to me, is that Y Combinator likes community-based companies, and my impression (admittedly supported by merely anecdotal and circumstantial evidence) is that they prefer them to value-based businesses, though this preference may be changing over time.  So, the question it raises is whether having a preference (even a mild one) for community-based businesses results in better outcomes for Y Combinator.  I suspect that it has a negative impact, but I don’t know that for sure.

We can break it down by exit, though:

Community-based YC exits

• Reddit – Rumored 12-ish million dollar acquisition by Condé Nast

Value-based YC exits

• Zenter – Undisclosed acquisition by Google
• Parakey – Undisclosed acquisition by Facebook
• Auctomatic – 5 million dollar cash+stock acquisition by Live Communicate
• TextPayMe – Undisclosed acquisition by Amazon
• Anywhere.fm* – Undisclosed acquisition by Imeem

I’ve put an asterisk by Anywhere.fm above, as I’m certain some folks would consider it a community-based business…but it provides most of its value even with only one user, so it fits my requirements for a value-based business, and I think it’s fair to say they were not acquired based predominantly on the size of their community or traffic growth but on the strength of the application they’d built.

Of course, since all but one of those acquisitions in the “value-based” category are for an undisclosed sum, and I’m basing the reddit acquisition price on rumors (though rumors from disparate sources, I’ve never heard the numbers from any of the people actually involved), it’s hard to say with any confidence that the five exits in the value-based group are worth more to YC than the one community-based exit.  But, one can safely say that value-based businesses have made a lot more young YC-backed entrepreneurs rich than community-based businesses.

This isn’t the whole picture, of course.  Some of the biggest potential YC success stories have yet to have an exit.   It’s hard to argue that Loopt isn’t going to  be a big success story for YC, and it’s pretty much purely community-based, though they have the added hurdle of having to sell to mobile providers to attain their customer base.  Scribd has seen tremendous traffic growth and it’s a community-based site with a small amount of value independent of the community.  Obviously, “going viral” is a fantastic marketing opportunity that is only generally available to community-based businesses, but historically it seems that building a useful application is more likely to result in a successful exit than building a product that relies on a large community to have value.

I found this result surprising when I began listing off the exits and thinking about what kind of business they were.  I’ve always thought of Virtualmin as an outlier among YC companies, since we build a traditional installable web-based application, but when broken down into value vs. community, we’re not so far out.

Anyway, I don’t think I’ve stumbled onto anything earth-shaking in this little exercise, of course, but I do think it would be interesting to break down all YC companies (and perhaps even everything in the TechCrunch Crunchbase) into this categorization and see how things shake out with regard to who is still around and growing.  Maybe later…right now I’ve gotta go build some value.