Farmer Batch Process Server, or Farmer BPS is database server side solution for realization of batch processing in relational database environment. Farmer BPS is dedicated for Oracle database applications with significant batch processing component.

Farmer solution is founded on the following paradigm: 
All batch processes are “almost” the same.

What does it mean? What is that “almost”? And what is the remainder? 
From business logic point of view batch processing is doing something with elements of some set of data. If so, theoretically to realize batch process we need only to implement set of data and a subroutine that represents operation we perform on dataset element. But experience shows that realization of such processes in database environment has to meet number of common requirements and constraints emerging on various stages of realization: design, implementation, testing, operation, maintenance.

 

The problem

One of main functionality in systems with significant database component is batch processing objects represented in database. Experience shows that realization of such processes has to meet number of common requirements and constraints emerging on various stages of realization: design, implementation, testing, operation, maintenance. Therefore implementation of process has to consider following issues:

  • database transaction management policy
  • progress control mechanism including stopping and resuming of executing process,
  • logging, diagnostics
  • optimization, debugging
  • absolute processing performance and scalability
  • exception handling policy including error resistance
It implies that batch processes have many common features to meet these constraints and requirements.

 

The solution

Farmer – Batch Process Server – is solution that enables you to separate implementation of business logic from common batch process features. Therefore implementation of batch process is reduced to two objects only:

  • view defining processed set of objects
  • procedure processing one object from set
All other requirements and constraints – mostly common for batch processes - are provided by Farmer BPS. In extreme, but not rare cases implementation of batch process does not include loop instruction, commit/rollback statement, exception handling section.

 

The benefits

Using Farmer BPS results many benefits on various stage of application life cycle.

Stage Benefits
Design - clear and universal design pattern
- easy prototyping
- classification types of batch processes into few subtypes
Coding - highly reduced code size
- isolation of business logic
- automatic code generation 
Testing - separation business functionality for unit tests
- transaction transparent test units
- easy test data set reconfiguration
Operation - highly increased absolute performance and scalability due of concurrent processing
- increased error resistance
- easy stopping and resuming of process execution
Maintenance - many predefined logs from both server and business implementation side at one place
- many diagnostic information of process performance, wait events, hardware utilization, load balancing over concurrent threads
- easy diagnosing and debugging based on error stack tracing

 

High level architecture concept

Above illustration shows concept of high-level architecture for systems with significant component of batch processing. It can be considered as subsequent tiers in system architecture



Illustration 1: High level architecture model

  1. Data tier makes base for such system. This tier includes data as well as its physical organization in tables, indexes, basic consistency validation in constraints, clusters etc.
  2. Data Business Logic tier mainly consist of procedural interfaced access to data which generally means encapsulation, specific business validation, business consistency checking, data versioning. It is very profitable when, entities are organized in meta-data dictionaries – it allows to make significant part of data business logic code auto-generated. In Farmer BPS model it may be great profit due, if business logic code provides mass mode in DML instructions.
  3. Process Business Logic tier cover functionality specific for particular process. In Farmer model this is process view, package and some supplemental objects.
  4. Process Logic is covered by Farmer BPS. It contains functionality common for all processes (like logging, transaction management) as well as individually configured options (like concurrency, incrementality etc. ).

Scott Tiger S.A. © 2011