Message Queue solution for Visual FoxPro applications

Download from git repo here: https://bitbucket.org/mattslay/foxpro_message_queue

Job Queues, Message Queues, Message Broker, Asynchronous Processing…  There are many names for this very helpful architecture where you take a process that holds up a user’s workstation for several seconds, or even minutes, and move the work to another machine (i.e. server or other workstation) to be processed offline, “in the background”, aka “asynchronously”,  either immediately or at some later time.

Using some helpful classes from the West Wind Web Connection framework (also available in the West Wind Client Tools package) we can implement a robust Message Queue solution in Visual FoxPro.  All it takes is two simple steps:

Step 1 – Submit a “Message”:  Create and submit a simple Message record to the QueueMessageItems database table which has the name of the Process or Action that you want to execute, along with any required parameters, values, settings, user selections, other other criteria required to complete the task.  Boom… in a blink the local user workflow is finished and they think your system is the fastest thing in the world, because now their computer is free in a few milliseconds and they can resume other tasks.

Step 2 – Process the “Message”: Down the hall, you are running a Message Queue Processor on a server or other workstation that can access the database, network drives, printers, internet, make API calls, etc needed to complete the task.  Once the Message (i.e. task) is picked up by the Processor it is marked as Started, and when all the processing is finished the record is marked as Complete. Along the way you can send other other messages, emails, SMS text messages, reject messages, etc.

Message Queue Monitor – UI Form

A simple UI form allows you to observer all the Message processing as it happens.

message_queue_monitor_form

Other great benefits of this pattern and architecture:

  • You wind up with a complete log of who submitted what, and when, how long it took, and if there were any errors processing the various messages.
  • You can direct certain types of Message to different “Queues” so that you can halt processing of certain Queues if you wanted to.
  • You can re-process any requests that failed, in case there were errors other reasons that the task could not be completed at that time.
  • You can launch multiple instance of the Queue Processor to handle Messages in one or more Queues if you have heavy workload or wanted to focus horsepower on a certain Message type.

“Multi-threaded” processing

Another thing this allows is that on the “server” (i.e. whatever machine is running the Message Processor app) , you can actually launch MULTIPLE instances of the Message Processor, and that lets it run essentially achieve the performance of a multi-threaded app because you are processing several Messages simultaneously.  FoxPro’s single-threaded architecture can be circumvented with this approach. No need to wait for sequential processing of each Message.  When running on Sql Server, it calls a Stored Proc on GetNextMessage() and it also flags the record as in-process so that other running instances will not pick up the Message.

Example uses

  • Generating and printing reports
  • Sending emails to clients or customers
  • Batch processing that run long queries, Insert, or Update statements
  • Calling external APIs that may have slow response times, or be totally offline that the time the task is initiated.

Specific case

I recently used Message Queue to solve this nagging problem… When printing a particular report from a form in our FoxPro app, the form (and user) is held up for about 3-4 seconds while the report renders and spools. That doesn’t sound like much, but I *hate* waiting that amount of time every time we print a new Shop Order to sent out to production. It happens dozens of times per day for several users.

So… Using the West Wind Message Queue, it now submits a Print Request Message to the Message Queue, which takes about 30-50 milliseconds, and the user form is never held up at all waiting for the report to print.  The Queue Message Manager (which is a VFP app running on the server), picks up the Message and prints the report.

Conveniently, it prints to the default printer defined on the server, which is the main printer central to all users, which is the same one that is the default printer on everyone’s desktop.  If you needed to send the report to a specific printer, you’d just have to add that printer name using loAsync.SetProperty(), then do Set Printer in the Queue Manager code to control where the report comes out.

You can do it too

Consider this asynchronous architecture if you any such hold-up, or long-running-processes that your users wastefully wait for each day on their job.

References

You can read Rick Strahl’s extensive whitepaper on his wwAsync class. (Note: the whitepaper is kind of old, and the focus there is on how his Web Connection framework uses the wwAsync class to handle back end processing in web apps, but using this on a LAN or desktop/Client Applicaiton works exactly the same way.) Documentation of each method and property can be found here.

.Net Version

There’s also a more robust .Net version of this same framework. It has even more features for managing multiple threads, multiple Queue, and much better live GUI that runs in in a web browser.  It’s an open source project on GitHub :  https://github.com/RickStrahl/Westwind.QueueMessageManager

FoxPro Code samples

Submitting a Message:

message_queue_code_sample

Message Processor – handling the Message request

message_queue_processor_sample

XML data to pass values

The required parameters, values, user choices, etc, are stored on the Message record in an XML field using the SetProperty() method when creating the Message, and can be retrieved using the GetProperty() method when processing a record.

XML example of data stored on a Queue Message:

message_xml_sample

Here’s what the message look like in the QueueMessageItems table:

messages

Message item data structure

Here is the data structure of the fields used on the QueueMessageItems table. Some of the fields are managed by the Message Manager class, several are available for your use in constructing and processing the message.

queue_message_item_data_structure

Leave a Reply

Your email address will not be published. Required fields are marked *