my $blog;

Every once in a while you want your Perl program to call an external program of some kind. Let’s say you want to see how much virtual memory the current process using, using ps.
Here are some very convenient ways to do this.

#!/usr/bin/perl -Tw

use strict;
use warnings;
use English qw( -no_match_headers $PID );

local $ENV{PATH} = '/bin';

# the dreaded back-ticks
my $vm_size_a = int `ps --no-headers -o vsz --pid $PID`;

print "vm size: $vm_size_a\n";

# the more attractive qx() list
my $vm_size_b = int qx( ps --no-headers -o vsz --pid $PID );

print "vm size: $vm_size_b\n";

Run this and you see something like this.

dylan@doxey.org$: ./vm_check.pl
vm size: 23004
vm size: 23004

There, mission accomplished.

From a security standpoint this particular program is fine. But as a general practice it’s somewhat not awesome to put potentially tainted strings in system calls this way. This technique also lacks distinct, and clearly defined, references to stdout, stderr, and the return status code. Of course there are ways of merging stdout and stderr with techniques like 2>&1 and directing these streams to files which can be slurped after the fact. Oh, but what about when the external program prompts for some input to stdin? With all the creativity and the compromises it just starts to seem very kludgy.

I like IPC::Run. There, I said it.

#!/usr/bin/perl -Tw

use strict;
use warnings;
use English qw( -no_match_headers $PID );
use IPC::Run qw( run );

my ( $in, $out, $err, $code ) = ("") x 4;
{
    local $ENV{PATH} = '/bin';
    
    $code = run(
        [qw( ps --no-headers -o vsz --pid ), $PID ],
        \$in, \$out, \$err
    );  
}

print "vm size: $out";

For this simple example it’s overkill. But for the general case it’s approaching just right. (Supposing you want to monitor the progress of the program and respond to prompt messages on stdout with appropriate input to stdin, then you’d want to use the start()/pump() technique.)

The beauty of IPC::Run is that you can pass it a code reference as well. Take the following.

#!/usr/bin/perl -Tw

use strict;
use warnings;
use IPC::Run qw( run );

my $code_rc = sub {
    my ($n) = @_;
    print "stdout: $n\n";
    warn "stderr: $n\n";
    return 1;
};

my ( $status, $in, $out, $err ) = ("") x 4;
{   
    print "\n--running--\n";
    
    $status = run( sub { $code_rc->( 42 ) }, \$in, \$out, \$err );
    
    print "\n--done--\n";
}

print "stdout: [$out]\n";
print "stderr: [$err]\n";
print "status: [$status]\n";

When this runs you get:

dylan@doxey.org$: ./runner.pl 

--running--

--done--
stdout: [stdout: 42
]
stderr: [stderr: 42
]
status: [1]

So precious.
But now check this out. What if that code ref had a fatal error, such as this?

my $code_rc = sub {
    my ($n) = @_;
    print "stdout: $n\n";
    die "stderr: $n\n";
    return 1;
};

dylan@doxey.org$: ./runner.pl 

--running--

--done--
stdout: [stdout: 42
]
stderr: [stderr: 42
]
status: []

OMG.

I became acquainted with ZeroMQ on a pretty involved networking messaging project last year. I’ve since started thinking of using it for smaller applications also.

Take this for example.

#!/usr/bin/perl -Tw
# file: sender.pl

use strict;
use warnings;
use ZeroMQ qw( ZMQ_XREQ );

my $ctx = ZeroMQ::Context->new();
my $skt = $ctx->socket( ZMQ_XREQ );

$skt->bind("tcp://*:70000");

while (1) {

    $skt->send("hello world");
    
    sleep 1;
}

#!/usr/bin/perl -Tw
# file: receiver.pl

use strict;
use warnings;
use ZeroMQ qw( ZMQ_XREP );

my $ctx = ZeroMQ::Context->new();
my $skt = $ctx->socket( ZMQ_XREP );

$skt->connect("tcp://*:70000");

while (1) {

    my $sender_id  = $skt->recv();
    my $msg        = $skt->recv();
    
    print "msg: ", $msg->data(), "\n";
}

Running this example we see what we expect:

dylan@doxey.org$: ./receiver.pl 
msg: hello world
msg: hello world
msg: hello world
msg: hello world
msg: hello world
msg: hello world
...

But there are some properties of this ZMQ connection which are not evident here.
• The call to send(…) blocks until the receiver as made its call to recv().
• When there are multiple receivers the messages will be round-robin distributed among the receiver instances.

To help illustrate these points I’m going to add some extra clutter to make each message distinct.

#!/usr/bin/perl -Tw
# file: sender.pl

use strict;
use warnings;
use ZeroMQ qw( ZMQ_XREQ );

my $ctx = ZeroMQ::Context->new();
my $skt = $ctx->socket( ZMQ_XREQ );

$skt->bind("tcp://*:70000");

my $n = 0;

while (1) {

    $skt->send( time . ':' . $n++ . ': hello world' );

    sleep 1;
}

I’ll start sender.pl, wait a few seconds and then start two instances of receiver.pl.

dylan@doxey.org$: ./receiver.pl 
msg: 1363457771:0: hello world
msg: 1363457777:1: hello world
msg: 1363457778:2: hello world
msg: 1363457779:3: hello world
msg: 1363457781:5: hello world
msg: 1363457783:7: hello world
...

dylan@doxey.org$: ./receiver.pl 
msg: 1363457780:4: hello world
msg: 1363457782:6: hello world
msg: 1363457784:8: hello world
...

Here you can see that sender.pl sent message 0 at 1363457771 and then blocked, waiting for the receiver.pl to come online and invoke recv() at 1363457776. Then the one second interval is apparent until the second instance of receive.pl comes online which then exhibits the round robin message distribution starting at message 4.

I would like to eliminate the blocking. I prefer that sender.pl just does a fire & forget.

#!/usr/bin/perl -Tw
# file: sender.pl

use strict;
use warnings;
use ZeroMQ qw( ZMQ_XREQ ZMQ_NOBLOCK );

my $ctx = ZeroMQ::Context->new();
my $skt = $ctx->socket( ZMQ_XREQ );

$skt->bind("tcp://*:70000");

my $n = 0;

while (1) {

    $skt->send( time . ':' . $n++ . ': hello world', ZMQ_NOBLOCK );

    sleep 1;
}

Again repeating the test …

dylan@doxey.org$: ./receiver.pl 
msg: 1363466086:10: hello world
msg: 1363466087:11: hello world
msg: 1363466088:12: hello world
msg: 1363466090:14: hello world

dylan@doxey.org$: ./receiver.pl 
msg: 1363466089:13: hello world
msg: 1363466091:15: hello world
msg: 1363466093:17: hello world

In this case we can see that messages 0 through 9 were lost before I started the first instance of receive.pl. Upon starting the second instance of receive.pl we can see the round robin message distribution resume.

Next I’d like to make a tweak which eliminates the round robin distribution of messages and instead sends the messages simultaneously to all instances of receive.pl equally — in a true pub-sub model. The examples in Perl are a little spotty. Here’s a minimal ZMQ pubsub in Perl.

#!/usr/bin/perl -Tw
# publish.pl

use strict;
use warnings;
use ZeroMQ qw( ZMQ_PUB );

my $ctx = ZeroMQ::Context->new();
my $skt = $ctx->socket( ZMQ_PUB );

$skt->bind("tcp://*:70000");

my $n = 0;

while (1) {

    $skt->send( time . ':' . $n++ . ': hello world' );

    sleep 1;
}

#!/usr/bin/perl -Tw
# subscribe.pl

use strict;
use warnings;
use ZeroMQ qw( ZMQ_SUB ZMQ_SUBSCRIBE );

my $ctx = ZeroMQ::Context->new();
my $skt = $ctx->socket( ZMQ_SUB );
$skt->setsockopt( ZMQ_SUBSCRIBE, "" );

$skt->connect("tcp://*:70000");

while (1) {
    
    my $msg = $skt->recv();
    
    print "msg: ", $msg->data(), "\n";
    
    sleep 2;
}

The obvious change here is the use of the socket types ZMQ_PUB & ZMQ_SUB. The one that didn’t come easily was the use of the ZMQ_SUBSCRIBE socket option for subscribe.pl. (The empty string value for the ZMQ_SUBSCRIBE tells ZMQ that I’m interested in subscribing all messages published for that socket. I could put in a non-empty string value and that would instruct ZMQ to filter any messages which don’t begin with that prefix.) If you omit the ZMQ_SUBSCRIBE option then ZMQ has no criterion to use for filtering the messages and that socket will receive nothing. Nice.

Running this example gives use the results we’d expect.

dylan@doxey.org$: ./subscribe.pl 
msg: 1363470603:57: hello world
msg: 1363470604:58: hello world
msg: 1363470605:59: hello world
msg: 1363470606:60: hello world
msg: 1363470607:61: hello world
...

dylan@doxey.org$: ./subscribe.pl 
msg: 1363470605:59: hello world
msg: 1363470606:60: hello world
msg: 1363470607:61: hello world
msg: 1363470608:62: hello world
msg: 1363470609:63: hello world
...

Consider this scenario …

my $failed = system "program.pl &";

die 'failed to start program.pl'
    if $failed;

Given the trailing & this program will be running on its own and you have no access to it’s STDIN, STDOUT or exit status. All we know is that the system call either succeeded or failed to initiate the process.

Let’s try putting this to use.

Imagine a scenario where you have a program which wants to dispatch jobs as separate processes. After issuing the jobs the dispatcher would like to get status reports from the job processes so that it can determine when all the jobs have completed successfully. (This is a fan-in pattern, where many senders are writing to a single receiver. This is the reverse of the pubsub fan-out pattern.)

Here’s an example of a job handler which does a little something.

#!/usr/bin/perl -Tw

use strict;
use warnings;
use ZeroMQ qw( ZMQ_XREQ ZMQ_IDENTITY );

my ($id) = @ARGV;

my $ctx = ZeroMQ::Context->new();
my $skt = $ctx->socket(ZMQ_XREQ);
$skt->setsockopt( ZMQ_IDENTITY, $id );
$skt->connect('tcp://*:70000');

for my $i ( ord 0 .. ord $id, 'done' ) {

    $skt->send("status: $i");
}

$skt->close();

This program demonstrates how we use the ZMQ_IDENTITY socket option. The value used as the socket identity is passed as the first section of the multipart message that an ZMQ_XREQ socket sends.

Here’s a job dispatcher.

#!/usr/bin/perl -Tw

use strict;
use warnings;
use ZeroMQ qw( ZMQ_XREP );

my $job_count = 0;
#
# Start Jobs
#
for my $id ( 'A' .. 'Z' ) {

    my $command = "perl -Tw job.pl $id &";
    {
        local $ENV{PATH} = '/usr/bin';
        system $command
            and die "failed: $command";
    }

    $job_count++;
}

my $ctx = ZeroMQ::Context->new();
my $skt = $ctx->socket(ZMQ_XREP);
$skt->bind('tcp://*:70000');

my %is_done;
#
# Monitor Jobs
#
while ( keys %is_done < $job_count ) {

    my $sender_id = $skt->recv()->data();
    my $message   = $skt->recv()->data();

    print "$sender_id: $message\n";

    if ( $message =~ m{: \s done \z}xms ) {

        $is_done{$sender_id} = 1;
    }
}

print "All jobs are done!\n";

This program issues the job processes as system calls. Then it creates an ZMQ_XREP socket to listen for status updates. The messages are queued and delivered in the order which they were sent. There is no blocking. But the final ‘done’ status message might be at the back of the queue giving the false impression that the job is still running if the dispatcher program is not dequeuing messages as fast as their queuing up.

Further reading:
http://api.zeromq.org/2-1:zmq-socket
http://api.zeromq.org/2-1:zmq-setsockopt

Unfortunately, ZeroMQ (http://search.cpan.org/~dmaki/ZeroMQ-0.23/lib/ZeroMQ.pm) is noted as deprecated. This is what apt-get installs on Ubuntu 12.04. But I suppose it’s time to move on to ZMQ::LibZMQ (http://search.cpan.org/~dmaki/ZMQ-LibZMQ3-1.10/).

I like to have a module with my most common constants.

I’m particularly fond of $INTERACTIVE which generally indicates the difference between a program being called by a person via the command line vs. via cron (or via another program, or from a pipe).

Another is $MT_STR which makes empty strings stand out a bit. The alternative is usually ” or “” which are fine. But $MT_STR is visually unambiguous without question. It’s also recommended in Perl Best Practices.

I’m also using $EXIT_OK and $EXIT_ERR for values which Perl returns to the system. These are nice because your Perl habit is to return 1 for an okay status and 0,undef,$MT_STR for non-okay status. But UNIX is the opposite. Just returning $EXIT_OK preempts that conversation.

package Dox::Const;

use strict;
use warnings;
use base qw( Exporter );
use Readonly;

@Dox::Const::EXPORT_OK = qw(
    $INTERACTIVE
    $MT_STR
    $NEWLINE
    $YEAR_IN_SECONDS
    $DAY_IN_SECONDS
    $HOUR_IN_SECONDS
    $EXIT_OK
    $EXIT_ERR
);

Readonly our $INTERACTIVE => -t \*STDIN && -t \*STDOUT;

Readonly our $MT_STR  => $MT;
Readonly our $NEWLINE => $NL;

Readonly our $YEAR_IN_SECONDS => 31_556_926;
Readonly our $DAY_IN_SECONDS  => 86_400;
Readonly our $HOUR_IN_SECONDS => 3_600;

Readonly our $EXIT_OK  => 0;
Readonly our $EXIT_ERR => 1;

1;

It’s been a few years since I first saw the Graffle demo (http://raphaeljs.com/graffle.html) on Raphaël.js (http://raphaeljs.com/). It was the first time I’d seen dynamic SVG on a web page.

There are so many things that I like to imagine as a node diagram of some kind. Be it a directed graph, a mind map, or a relational diagram of some sort, the only way to do this in an HTML document would be to painstakingly create the graphic in something like the Gimp, PaintShop Pro, or Photoshop. There are flowcharting programs which are more ideally suited. But in either case the process of generating the graphic and then getting it to the right scale and format seems to be an unwanted distraction.

The interactive SVG on an HTML document has a sense of instant gratification that I’d like to capture. This just screams out to me as a user interface for manipulating real resources, such as records in a relational database.

I’ve also been interested in seeing what I could do to generate graffle like SVG documents and then export them for use in other contexts. In this respect, I tend to use the Perl Graph::Easy module. This offers a simple interface for defining a node diagram and emitting ASCII art, BoxArt, and even SVG.

#!/usr/bin/perl -Tw

use strict;
use warnings;
use Graph::Easy;

my $g = Graph::Easy->new();

$g->add_edge( 'a' => 'b' );
$g->add_edge( 'a' => 'c' );
$g->add_edge( 'a' => 'd' );
$g->add_edge( 'a' => 'e' );
$g->add_edge( 'e' => 'f' );

print $g->as_boxart();

This gives you a nice little boxart chart like:

          ┌───┐
          │ d │
          └───┘
            ∧
            │
            │
┌───┐     ┌───┐     ┌───┐     ┌───┐
│ c │ <── │ a │ ──> │ e │ ──> │ f │
└───┘     └───┘     └───┘     └───┘
            │
            │
            ∨
          ┌───┐
          │ b │
          └───┘

However, I’m still not satisfied with Graph::Easy’s layout algorithm. I want more control without the overhead of a complicated programming interface. It would be nice if I could have the simple programming interface combined with the ability to manually manipulate the graph like the Graffle SVG demo.

With a little Graffle demo hacking and some back-end programming, here’s my working prototype.

http://d.doxey.org/o/charts/

TODO

• directional edges
• node properties (fonts, colors)
• variable image size

Happy computing.

I was looking over a couple of my old JavaScript projects recently. Both my life resources widget and my machine learning widgets are based on grids of blinking lights. I think it’s time I created the obvious grid of blinking lights project — Conway’s Game of Life.

Conway’s Game of Life

Conway’s Game of Life (also known simply as Life) is a zero player game which requires an initial state as input. The subsequent pixel states are determined by a some simple rules based on the state of the surrounding pixels.

From wikipedia.org:

The universe of the Game of Life is an infinite two-dimensional orthogonal grid of square cells, each of which is in one of two possible states, alive or dead. Every cell interacts with its eight neighbours, which are the cells that are horizontally, vertically, or diagonally adjacent. At each step in time, the following transitions occur:

1. Any live cell with fewer than two live neighbours dies, as if caused by under-population.
2. Any live cell with two or three live neighbours lives on to the next generation.
3. Any live cell with more than three live neighbours dies, as if by overcrowding.
4. Any dead cell with exactly three live neighbours becomes a live cell, as if by reproduction.

I remember doing one of these as one of my very early projects in Kube’s CSE 12. It seemed so much more challenging back then. Please enjoy my HTML version.

I’ve fiddled around with different ways of getting my code to format well on a web page. There’s perltidy and :TOhtml in vim. But the dynamic CSS/JavaScript solution on http://search.cpan.org/ really rocks.

It only took a little bit of googling to find this: http://shjs.sourceforge.net/

Getting this to work on my new WP site wasn’t too tough.

First, download the files.

wget http://shjs.sourceforge.net/css/sh_acid.min.css
wget http://shjs.sourceforge.net/sh_main.min.js
wget http://shjs.sourceforge.net/lang/sh_perl.js
wget http://shjs.sourceforge.net/lang/sh_sh.js
wget http://shjs.sourceforge.net/lang/sh_html.js

Then add them to the header.php.

<link rel="stylesheet" type="text/css" media="all" href="/css/sh_acid.min.css" />
<script type='text/javascript' src='/js/sh_main.min.js'></script>
<script type='text/javascript' src='/js/sh_perl.js'></script>
<script type='text/javascript' src='/js/sh_sh.js'></script>
<script type='text/javascript' src='/js/sh_html.js'></script>

Then add onload=”sh_highlightDocument();” to the <body> tag.

Finally, the code content needs to be in a <pre> tag with a class name that indicates the language type.

<pre class=”sh_sh”>#!/bin/bash
echo Where the class name is like: sh_[language type]
exit</pre>

It’s not so unusual that I want to create a quick Perl program.

My usual habit is:

touch program.pl
chmod +x program.pl
vim program.pl

And then there is the standard header.

#!/bin/bash

name="$1"

if [[ "$name" == "" ]]
then
    echo "no program name specified"
    exit 0
fi

extension=`echo $name | awk -F. '{print tolower($NF)}'`

if [[ "$extension" == "" ]]
then
    echo "no file extension given"
    exit 0
fi

if [[ ! -e $name ]]
then

    touch $name
    chmod +x $name

    if [[ "$extension" == 'pl' ]]
    then

        echo "#!/usr/bin/perl -Tw" >> $name
        echo ""                    >> $name
        echo "use strict;"         >> $name
        echo "use warnings;"       >> $name
        echo ""                    >> $name
        echo ""                    >> $name

    elif [[ "$extension" == 'pm' ]]
    then

        chmod -x $name

        package=$(echo $name | sed 's/\\/::/g' | sed 's/\.pm$//')

        echo "package $package;" >> $name
        echo ""                  >> $name
        echo "use strict;"       >> $name
        echo "use warnings;"     >> $name
        echo ""                  >> $name
        echo ""                  >> $name
        echo "1;"                >> $name

    elif [[ "$extension" == "php" ]]
    then

        echo "#!/usr/bin/php" >> $name
        echo "<?php"          >> $name
        echo ""               >> $name
        echo ""               >> $name
        echo "?>"             >> $name

    elif [[ "$extension" == "sh" ]]
    then

        echo "#!/bin/bash" >> $name
        echo ""            >> $name
        echo ""            >> $name
    fi
fi

vim +6 $name

I saved this to: ~/bin/create If it’s not already, you can add export PATH=”~/bin:$PATH” to your .bashrc (assuming you’re a bash user).

With that in place creating a new program goes like this:

$: create hello_world.pl