Old School to New School: Refactoring Perl (part 2)

When I left off, I’d made an old chunk of Perl code (that mostly pre-dates widespread availability of Perl 5) warnings and strict compliant, converted it to be easily usable as both a module and a command line script, added some POD documentation, and built a couple of rudimentary tests to make it possible to change the code without fearing breakage. Now we can get rough with it.

Refactoring for Clarity and Brevity

Despite the changes, so far, the code is pretty much as it was when we started. A little more verbose due to the changes for strict compliance, so that’s a negative, and the addition of a main function and the oschooser wrapper just adds even more lines of code. The main code block was already a little bit long for comfort at several pages worth of 80 column text, assuming a 50 row editor window. Jamie seems capable of holding a lot of code in his head at once…me, I’m kinda slow, so I like small digestible chunks. So, let’s start digesting.

Looking through the code, this bit jumped right out. It’s a little bit unwieldy:

    if ($auto == 1) {
      # Failed .. give up
      print "Failed to detect operating system\n";
      exit 1;
      }
    elsif ($auto == 3) {
      # Do we have a tty?
      local $rv = system("tty >/dev/null 2>&1");
      if ($?) {
        print "Failed to detect operating system\n";
        exit 1;
        }
      else {
        $auto = 0;
        }
      }
    else {
      # Ask the user
      $auto = 0;
      }

It seemed like this could be shortened a little bit by making a have_tty function, and using && in the elsif. Not a huge difference, but if we then flip the tests over (there’s only four possible values of $auto and they only result in two possible outcomes) and add an || we can lose a few more lines, one conditional, and cut it down to:

    if (($auto == 3 && have_tty()) || $auto == 2) {
      $auto = 0;
      }
    else {
      # Failed .. give up
      print "Failed to detect operating system\n";
      exit 1;
      }

That’s a lot less code to read. I also think it makes more sense to have a single failure block and a single block setting $auto. I’m still trying to figure out the purpose of auto=2, since it seems like it would only be possible to fall back to asking a question, if there’s actually a TTY. But Jamie knows the quirks of various systems far better than I do, so we’ll leave it alone, for now, and keep the same behavior. I bet it’s to accommodate something funny on Windows!

I’ve also made a few tweaks, during this process, adding a package OsChooser; statement to the beginning of the file. I also discovered that $uname actually is being used in this code! It’s hidden inside the os_list.txt definitions. There are a couple of eval statements being used to execute arbitrary code on each line found in the OS list. Jamie must have been a Lisp hacker in a former life, with all this willy nilly mixing code and data. This is a pretty clever bit of code, but it took me a little while to grok, but now that I know what’s going on, we can solve some of the problems we had with testing detection of systems other than the one we’re running on.

In the meantime, just know that $uname has returned as an our variable, so that it can be “global” without ticking off strict.

More Tests

So, automated testing of an OS detection program isn’t a whole lot of good, if I can only test detection of one operating system (the one it happens to be running on right now). So, we need to introduce a bit more flexibility in where the OS-related data comes from. This is trickier than it sounds. UNIX and Linux has never standardized on one single location for identifying the OS. Many systems identify themselves in /etc/issue, while those from Red Hat use /etc/redhat-release (they also have a reasonable issue file, but I’m guessing it’s not reliably present or reliably consistent in its contents, as Jamie has chosen to use the release file, instead), and Debian has a /etc/debian_version file. Sun Solaris and the BSD-based systems seem to all use uname, and that’s just the popular ones. Webmin also supports a few dozen more branches of the UNIX tree, plus most modern Windows versions!

So, looking at oschooser.pl you’re probably wondering where the heck all of that extra stuff happens, because it doesn’t really have any detection code of its own. The answer is in os_list.txt, which is a file with lines like the following:

Fedora Linux     "Fedora $1" fedora  $1    `cat /etc/fedora-release 2>/dev/null` =~ 
/Fedora.*\s([0-9\.]+)\s/i || `cat /etc/fedora-release 2>/dev/null` =~ /Fedora.*\sFC(\S+)\s/i

This is a tab-delimited file. Why tabs? I have no idea, and it’s been a source of errors for me several times…even Jamie isn’t sure why he chose tabs as the delimiter, but that’s the way it is. It is plain text, plus numbered match variables, plus an optional snippet of Perl that will be executed via eval if it exists. This makes for an extremely flexible and powerful tool, if a wee bit intimidating on first glimpse.

So, that last field is the tricky bit. The thing I’m going to have to contend with if I want to be able to test every OS that Webmin supports, rather than just the one that happens to be sitting under the code while the tests are running. I’ll need a new argument to our oschooser function for starters, called $issue, which will generically contain whatever it is that os_list.txt uses to recognize a particular OS. On my Fedora 7 desktop system, that’s /etc/redhat-release, which contains:

Fedora release 7 (Moonshine)

So, oschooser now contains:

sub oschooser {
my ($oslist, $out, $auto, $issue) = @_;
...
}

Next, we need to make sure we keep the provided $issue if we got it, so we change this:

  # Try to guess the OS name and version
  if (-r "/etc/.issue") {
    $etc_issue = `cat /etc/.issue`;
    }
  elsif (-r "/etc/issue") {
    $etc_issue = `cat /etc/issue`;
    }
  $uname = `uname -a`;

Into:

# Try to guess the OS name and version
my $etc_issue;
if ($issue) {
  $etc_issue = `cat $issue`;
  $uname = $etc_issue; # Strangely, I think this will work fine.
  }
elsif (-r "/etc/.issue") {
  $etc_issue = `cat /etc/.issue`;
  }
elsif (-r "/etc/issue") {
  $etc_issue = `cat /etc/issue`;
  }

Note that $uname is defined earlier in the code now…and merely gets over-written if we’ve set the $issue variable in our function call.

And then we have to do something about the contents of the last field in os_list.txt before it gets evaluated. This is where it gets a little hairy. In the foreach that iterates through each line in the file testing whether we have a match or not, I’ve added a new first condition, so it now looks like:

foreach my $o (@list) {
  if ($issue && $o->[4]) {
    $o->[4] =~ s#cat [/a-zA-Z\-]*#cat $issue#g;
    } # Testable, but this regex substitution is dumb.XXX
  if ($o->[4] && eval "$o->[4]") {
    # Got a match! Resolve the versions
    $ver_ref = $o;
    if ($ver_ref->[1] =~ /\$/) {
      $ver_ref->[1] = eval "($o->[4]); $ver_ref->[1]";
      }
    if ($ver_ref->[3] =~ /\$/) {
      $ver_ref->[3] = eval "($o->[4]); $ver_ref->[3]";
      }
    last;
    }
  if ($@) {
    print STDERR "Error parsing $o->[4]\n";
    }
  }
  return $ver_ref;
}

Which performs a substitution on the last field, if it contains a cat command. It replaces it with the issue file that we’ve provided in the $issue variable. Thus, we can now pass in t/fedora-7.issue and put a copy of the /etc/redhat-release file mentioned above, and we’ll be able to test detection of Fedora 7, no matter what operating system the test is actually running on. I suspect we may run into trouble when we expand our os_list.txt to the full Webmin list, since I’m working with just the limited subset of systems the Virtualmin installer supports (or that I might support in the next year or so). I’ve made a comment in the code with XXX (merely a convention used in the Webmin codebase, though any odd sequence of characters that you’ll remember works fine…many folks use FIXME) to remind myself of this suspicion later if I do run into problems that this is the first place I ought to look.

After these changes, it’s possible to get serious about testing. So, I’ve added tests for a couple dozen systems, which was more Googling than coding due to the data-driven nature of my tests, and confirmed the new code is behaving identically to the old. Which means it’s time for…

More Refactoring

If you’ve been following along, you know that oschooser is still awfully long. A good tactic in such situations is to look for bits of functionality that can be pushed down into their own subroutines. One good choice is the parsing of patterns file at the very beginning of the function:

my @list;
my @names;
my %donename;
open(OS, $oslist) || die "failed to open $oslist : $!";
while(<OS>) {
  chop;
  if (/^([^\t]+)\t+([^\t]+)\t+([^\t]+)\t+([^\t]+)\t*(.*)$/) {
    push(@list, [ $1, $2, $3, $4, $5 ]);
    push(@names, $1) if (!$donename{$1}++);
    $names_to_real{$1} ||= $3;
    }
  }
close(OS);

This is a good place to start, because it only depends on one variable from outside the work, $oslist, which is the name of the OS definitions file. And, of course, file access is always a good candidate for abstraction…what if, some day, we want to pull these definitions from a database or a __DATA__ section? Having it all in one obvious location might be a win. For now, I just want that bloody long oschooser function to be a little bit shorter, so we’ll create this parse_patterns function:

sub parse_patterns() {
my ($oslist) = @_;
my @list;
my @names;
my %donename;
# Parse the patterns file
open(OS, $oslist) || die "failed to open $oslist : $!";
while(<OS>) {
  chop;
  if (/^([^\t]+)\t+([^\t]+)\t+([^\t]+)\t+([^\t]+)\t*(.*)$/) {
    push(@list, [ $1, $2, $3, $4, $5 ]);
    push(@names, $1) if (!$donename{$1}++);
    $NAMES_TO_REAL{$1} ||= $3;
    }
  }
close(OS);
return (\@list, \@names);
}

That’s not too bad, and it shaves about 13 lines off of oschooser at a cost of 3 or 4 more lines of function baggage in the whole file. The biggest irritant might be that I’m now passing around two array refs (one of which is already an array reference, so now we’ve got a reference to an array of references). I get confused when I use too many references, because I’m addle-brained that way, but these are only a little bit nested and so not too complicated, so I think future readers of the code should be fine. At least, no worse than they were before I got ahold of this script.

I’ve also converted %names_to_real to %NAMES_TO_REAL as it has become a package scoped global variable, and it’s considered good form to warn folks when they’ve come upon a global by shouting at them. Of course, I have another global, $uname, which I haven’t renamed to all caps, as one of my mandates for myself on this project is to require no changes to the Webmin os_list.txt. As I write this, I’m beginning to have second thoughts about $uname needing to be a global…so we’ll come back to that later.

Capturing the results of parse_patterns and dumping them out into @name and @list lets us run our tests again.

And More Refactoring Still

Things have improved a little in oschooser. It almost fits into two screenfuls on my 20″ monitor. But I think we can do better. I’m aiming for one page or less per function, in this exercise, so we’ve gotta keep moving. The next distinct piece of functionality I see is the automatic OS detection code, so I’ll add a new auto_detect function, something like this:

sub auto_detect() {
my ($oslist, $issue, $list_ref) = @_;
my $ver_ref;
my @list = @$list_ref;
my $uname = `uname -a`;
 
# Try to guess the OS name and version
my $etc_issue;
 
if ($issue) {
  $etc_issue = `cat $issue`;
  $uname = $etc_issue; # Strangely, I think this will work fine.
  }
elsif (-r "/etc/.issue") {
  $etc_issue = `cat /etc/.issue`;
  }
elsif (-r "/etc/issue") {
  $etc_issue = `cat /etc/issue`;
  }
 
foreach my $o (@list) {
  if ($issue && $o->[4]) {
    $o->[4] =~ s#cat [/a-zA-Z\-]*#cat $issue#g;
    } # Testable, but this regex substitution is dumb.XXX
  if ($o->[4] && eval "$o->[4]") {
    # Got a match! Resolve the versions
    $ver_ref = $o;
    if ($ver_ref->[1] =~ /\$/) {
      $ver_ref->[1] = eval "($o->[4]); $ver_ref->[1]";
      }
    if ($ver_ref->[3] =~ /\$/) {
      $ver_ref->[3] = eval "($o->[4]); $ver_ref->[3]";
      }
    last;
    }
  if ($@) {
    print STDERR "Error parsing $o->[4]\n";
    }
  }
  return $ver_ref;
}

You may note that I did rethink the globalization of $uname and found that it fit comfortably into this block, so I’ve killed a global introduced earlier in this process. Now I’ve not even sure why I thought I needed it somewhere else, which is a nice thing about refactoring: You realize how little you understood what was going on when you first looked at the code. Here’s also where we make use of the @list built during parse_patterns, and I dereference it before using it, though that’s probably more verbosity than needed. I could also access it directly within the ref with, @{$list_ref}.

Finally, I’m returning $ver_ref, which contains a reference to an array of fields that describes the operating system detected. Now that I’ve split this out, I realize that this OS version array could quite easily be mapped into a hash and turned into an object rather trivially, but that’s an exercise for another day. For now, I just want to feel confident that I’ve made a functionally identical clone of oschooser.pl that I can use and extend painlessly and without fear of breakage. So, let’s keep going.

A Few More New Functions, and Killing Unused Code Softly

As with auto_detect there is a big chunk of code that is used specifically for asking the user to choose the operating system and version from a list of options. This is triggered in the following cases: $auto is set to 0 or any other false value, $auto is not false but auto-detection failed and one of the non-exit auto options is chosen and viable. So, we can easily break out this whole bunch of functionality into its own function, called ask_user. Like auto_detect, it requires the $list_ref array reference, and it also needs the $names_ref, since it will be interacting with the end user and they’ll be more comfortable seeing the “real names” of the available operating systems. Also like auto_detect, it returns a $ver_ref which points to the array containing the full description of the OS.

When I got to this function, I noticed a huge block of unused code, which provides support for the dialog command on systems that support it (mostly just Red Hat based Linux distributions). dialog is a simple tool for adding attractive ncurses interfaces to shell scripts. I’m not sure why the code is being skipped with an if (0) statement, but I have only two choices for what to do about it, if my goal is to simplify this script and make it more robust: Enable it and fix whatever problems it has, possibly making it into its own reusable and independently testable function; or, simply remove the code altogether. Webmin and the installer libraries for Virtualmin are both in SVN. If I decide to remove the code, it won’t be lost forever…I could pull it back in the future. I could even tag the current version with “pre-dialog-removal” before stripping it out. After consulting with Jamie the last option is the one I’ve chosen. So, we can kill not just those pieces of code, we can also remove the has_command function, since it is only used in that part of the code. Big win!

So, I’ll make a tagged copy before ripping stuff out:

svn cp lib tags/lib/pre-dialog-removal

Now I know I can always go back and refer to that code if I want to. It’s not really particularly precious, but it’s a good practice to get into, since copies in Subversion are cheap and fast (likewise for git, and most other modern distributed revision control systems), and I never know when I might want to go back and see how something was done before. I’ll do the same in the Webmin tree before I make the changes needed to merge the new OsChooser.pm in place of the old oschooser.pl.

So, after killing the dialog pieces of the code, and converting the user interaction to its own function, we have:

# ask for the operating system name ourselves
sub ask_user {
my ($names_ref, $list_ref) = @_;
my @names = @$names_ref;
my @list = @$list_ref;
my $vnum;
my $osnum;
my $dashes = "-" x 75;
print <<EOF;
For Webmin to work properly, it needs to know which operating system
type and version you are running. Please select your system type by
entering the number next to it from the list below
$dashes
EOF
{
my $i;
for($i=0; $i<@names; $i++) {
  printf " %2d) %-20.20s ", $i+1, $names[$i];
  print "\n" if ($i%3 == 2);
  }
print "\n" if ($i%3);
}
print $dashes,"\n";
print "Operating system: ";
chop($osnum = <STDIN>);
if ($osnum !~ /^\d+$/) {
  print "ERROR: You must enter the number next to your operating\n";
  print "system, not its name or version number.\n\n";
  exit 9;
  }
if ($osnum < 1 || $osnum > @names) {
  print "ERROR: $osnum is not a valid operating system number.\n\n";
  exit 10;
  }
print "\n";
 
# Ask for the operating system version
my $name = $names[$osnum-1];
print <<EOF;
Please enter the version of $name you are running
EOF
print "Version: ";
chop($vnum = <STDIN>);
if ($vnum !~ /^\S+$/) {
  print "ERROR: An operating system number cannot contain\n\n";
  print "spaces. It must be like 2.1 or ES4.0.\n";
  exit 10;
  }
print "\n";
return [ $name, $vnum,
    $NAMES_TO_REAL{$name}, $vnum ];
}

Not too bad. It just fits into one 50 row editor window without scrolling, so that’s a small enough bite for me. We make use of the %NAMES_TO_REAL global in this function, to convert from the short names to the longer human-friendly names, and I’m beginning to get a vague feeling something could be done to encapsulate that functionality, even without making this an Object Oriented library (which seems like overkill for such a simple program), so I’ll probably be coming back to that global in a later post (and I thought I would have a hard time getting two full posts worth out of this exercise!).

Wrapping Up

I’m feeling pretty good about the code now. I think it’s more readable than before I started messing with it, it’s certainly shorter due to bits of refactoring and some removal of dead or redundant code, and it’s got quite a few tests. All of its variables are reasonably scoped to the areas where they are used, except for %NAMES_TO_REAL, which is a package scoped my variable (turns out eval gets the scope of the containing block, so it doesn’t need to be an our variable as I’d first assumed).

The various utility functions aren’t very useful to outsiders and may change…the only function I really want to be public is oschooser, so I can see several opportunities for further enhancements, like encapsulating the rest into private methods within an OsChooser object. But that’ll be a project for another day. You can see the current code, plus an example os_list.txt. Next time, I’ll begin work on wrapping this up for CPAN, and releasing a large OS definition list (Webmin’s list is incredibly long and detects hundreds of systems and versions, but needs a bit of massaging to be generically useful, due to its own internal version requirements).

Who knew one simple script could present so much interesting work? It’s been so much fun, I think I’ll start a Perl Neighborhood Watch and do this to every little Perl script I come across. Who’s with me!? (Or maybe I should just focus on our own code for a little while longer, since we’ve got quite a few nooks and crannies that haven’t seen any attention in years. Perl makes us lazy with its peskily perfect backward compatibility.)