Batch Processing With Job Control
The Inq distribution includes an application called Job Control. It runs in an Inq server to schedule batch jobs, for example to run an end-of-day process. Job Control supports the following features:
- A task is a job that performs some processing. It returns an exit status where zero indicates success and non-zero an error.
- A box contains tasks or other boxes. When a box is scheduled it runs its child jobs. If a child returns error status the first such job to do so establishes the exit status for the box.
- Box Continuation
- If a child job returns an error then a box can be set to either terminate at that point or continue with subsequent children.
- Job Activation
- If a box or task is disabled it will not be executed, however all jobs can be run manually either via the admin GUI client or by calling runJobNow(any Job) in the server.
- Job Output and Logging
- A task is given an output stream to which it can write (or pass as standard output to system commands) to capture any output it generates. The Job Control application itself (residing in package inq.jobcontrol) defines its own logger (see etc/server.log.properties in the distribution).
- Timers and Scheduling
- Any job can define a timer. If several timers are defined beneath a top-level box then the earliest one becomes that box's start time. Within a job hierarchy a box can either run its children consecutively or respect any start time they define.
- The Administrative GUI
- Job Control includes an administrative GUI that can be used to create jobs and job hierarchies, edit timers and task actions, configure box behaviour and run job subtrees manually.
First we discuss setting up Job Control and establishing some test jobs. These are used to experiment and show where a job's output goes.
After that we deconstruct Job Control to examine the Inq features used in its implementation. These include
- Using conditional monitors to coordinate cooperating Inq processes
- Storing blobs in the database
- A Tree-table GUI
- Parsing Inq script at runtime
- Processing the events generated in a transaction as a batch, rather than individually
- Managing streams and working with files
- Creating non-persisted (in-memory) typedefs
- Working with timers
- Sorting node structures
Creating The Database
Job Control stores the job tree in a single database table. In preparation for future applications bundled into the Inq distribution, the database inqwell is created. A MySQL example is provided in the file app/inq/db/mysql/inq-mysql-schema.sql. Run the following commands:
cd $INQHOME/app/inq/db/mysql mysql -uroot -pyour-password < inq-mysql-schema.sql
This script also creates a table Inq uses to allocate ID sequence numbers.
Starting The Server
This is the same as starting the server for petstore. If you have a server running already then move on to loading Job Control.
Loading the Job Control Script
Again commands are similar to the petstore example. There are two boot files, one for the bundled services in the inq.boot and inq.util packages (to implement ID sequences) and one for Job Control itself. Issue commands similar to these:
inq -load -u admin -p inqwell -server speakinq://localhost -url file:/home/tom/apps/inq-1.1.1-SNAPSHOT/app/inq/inqBoot.inq inq -load -u admin -p inqwell -server speakinq://localhost -url file:/home/tom/apps/inq-1.1.1-SNAPSHOT/app/jobcontrol/jobBoot.inq
We don't discuss the inq package files - if you have investigated petstore and the other examples by now these are straightforward enough.
Any jobs present that are active and define a timer will be scheduled at this point.
Starting The Client
Unlike petstore, Job Control does not define a login service so it is not possible to log in to the inq.jobcontrol package. Instead its GUI can be launched from the (early release) server administration client. Launch the Inq client environment, for example:
inq -client -lookandfeel none
and fill out the screen like this, the default administrator password is inqwell:
The Job Control server is implemented in app/jobcontrol/jobControl.inq. At the end of this file is the statement:
// Declare ourself to the package meta data call inq.meta:createPackage(Package = "\p", ClientURL = absurl("gui/jcAdmin.inq", "\i"), InitStmt = "call jobMain();", LongName = "Job Control");
This call declares the specified package (in this case expanded from the string constant escape\p meaning the current package) to the server's meta data. The arguments are:
|Package||The package being declared|
|ClientURL||The absolute URL for the client. The absurl function returns the absolute url from the relative first argument based on the absolute second argument.|
|InitStmt||If present, text that will be appended to the above script when sent to the client. In this example jobMain() is called, which kicks of Job Control's GUI.|
|LongName||A name for the package|
Having declared the package, it appears in the list and its client can be run:
Selecting the package and clicking the Run Client button brings up the Job Control client. In fact, when developing a client this is a quick way of restarting it as the code is modified.
If we look at the Meta tab then each package's types can be seen:
In fact, meta data can be used by applications and is employed by the Attribute Editor, demonstrated in petstore.
Loading Some Test Jobs
When the inqwell database is empty the client shows no jobs of course. Running the following script sets up some test jobs:
inq -load -u admin -p inqwell -server speakinq://localhost -url file:/home/tom/inqwell/inq/src/main/app/jobcontrol/testSetup.inq
The jobs this script creates will popup in the client, so it should look something like this:
These jobs are used to experiment with Job Control.