Asynchronous Distributed Job Processing with Helios

By Andrew Johnson from Phoenix.pm
Date: Thursday, 14 August 2008 13:30
Duration: 30 minutes
Language:
Tags: distributed helios processing

You can find more information on the speaker's site:

Talk: http://www.transformedplanet.com/media/ye2008/AsynchronousDistributedJobProcessingwithHelios.pdf

Helios is a system for building asynchronous distributed job processing applications. Applications that need to process millions of small units of work in parallel can use the Helios system to scale this work across the multiple processes and servers that form a Helios collective. Helios may also be used to improve the user experience on websites. By utilizing the framework's asynchronous job processing services, potential timeout issues can be eliminated and response times decreased for larger tasks invoked in response to user input. The web server application can "fire and forget" in the background, immediately returning control to the user. Using Helios, simple Perl applications can be written to distribute workloads throughout the Helios collective while still retaining centralized management.

The Helios package is freely available on CPAN under the same licensing terms as Perl itself. It is built on a free/open source software foundation, including Perl, TheSchwartz, and MySQL. The Helios framework builds on the excellent TheSchwartz job queue library to form a system designed to handle generalized processing needs. Helios's API contains functions that provide logging, centralized configuration, and process control. Because of its generic underpinnings, it should run on most *nix-like operating systems to which Perl has been ported. To date it has been tested successfully on several GNU/Linux distributions, including Debian Etch and Ubuntu, and even Apple Mac OS X.

With the Helios framework, developers define worker classes to accomplish specialized tasks. These are written in Perl and implement the functionality of a particular job type. The Helios API provides functions to perform logging, to parse XML-formatted job arguments, to access configuration parameters, and to manage processes. This API also defines a set of basic exception classes to signal processing failures and warnings. There are several run modes available to enable the caching of data, database connections, and other resources. This enables their use across multiple jobs. Once a worker class has been created, it can be loaded by Helios daemons on as many systems as necessary. Helios daemons manage available processes for a particular worker class, launching new worker processes as necessary to handle jobs submitted to the system. Jobs may be submitted via an HTTP POST request or from the command line.

Helios also includes a web interface that provides a central point of control for all of the workers and jobs in the system. This web interface can be used to track jobs through the system and manage workloads on a per worker class and per host basis. Available workers may be increased or decreased as necessary, tuned to match available resources. Worker classes may be brought online, or taken offline, independently. Similarly, hosts may be added or taken out of the collective without affecting other Helios hosts.

Helios is a flexible solution for distributed job processing. It provides a framework where applications may be deployed across multiple servers, running multiple jobs in parallel. Because of its flexibility, users can create Helios applications easily and deploy them to a server farm or small "cloud" of machines, while at the same time centrally manage their workload processing via its web interface.