Batch processing will also benefit end users

Perhaps every ERP implementation brings with it the need to plan and execute operations in batch mode. An example is the regular booking of goods or invoicing of sales made. However, batch processing is also used by end users to run one-off operations that are demanding on computing power - for example, creating a report or running a closing at the end of fiscal periods.

Microsoft Dynamics AX has supported batch processing since its early versions. With each new release always comes a number of important changes - while Axapta 3.0 processed batch jobs using a dedicated client, AX 2009 introduced the concept of server-side processing. AX 2012 then brought another significant performance boost by moving batch processing into the .NET framework world.

At Blue Dynamic, we have extensive experience running batch jobs on different environments for different customers. In the following text we would like to share some tips in the form of five valuable knowledge and experiencethat we've gained in the process. In the following, we will refer to the Blue Dynamic Batch package with the application modifications we have developed to solve the problems associated with batch operation.

Tip 0: A little theory won't kill anyone?

Each batch job consists of a header (Batch Job) and one or more rows.

Or this is not a tip yet. However, to make it easier to write, you need to be a little clear about the terminology. Each batch job consists of a header (Batch job) and one or more rows (Batch job). The Czech translation here is not entirely happy, the meaning of the words task and task in common Czech is practically identical.
The English texts are more concise in this respect - headers are called Batch jobs and rows are called Batch tasks - and that is why I will stick to English terminology in the following text, linguistic conservatives will forgive me.

It is important to know what Batch jobs and Batch tasks are for and how to access them Microsoft Dynamics AX. Batch jobs define the behaviour of the batch as a whole - current status, processing schedule including repeats, date and time of the next run. Batch tasks, on the other hand, represent specific steps in batch processing - define what actions are to be performed and with what parameters (e.g. input filters are defined here). Dependencies can then be defined between the individual steps to determine the correct order of processing or to enable parallel processing.

Tip 1: Good naming as a basis

If we operate in Dynamics AX solution five batches, then their names don't matter too much, we can easily keep an overview of what batches are running in the system. Over time, however, the situation usually gets complicated. Some batches are to run only during the day, others only at night. Some batches we want to run every few minutes in "process changes only" mode and then once a day in "process all" mode for safety (a good example is e.g. transfers of retail master data from the headquarters to individual stores). Some batches are so time-consuming that we decide to relieve the system somewhat and process only a section of data each time (e.g. when looking for duplicates in a list of millions of customers, start with those whose last name starts with the letter A). And so sooner or later, we will find that the list of active batches will yield several screens.

It is therefore sensible to introduce a system into the confusion right from the start. We have found it useful to use names with different prefixes to name batches. These can have several levels. As an example, we give two-level names accordingly:

Whether it runs during the day (D) or at night (N)
What area (module) is involved - e.g. logistics (LOG), finance (FIN), administration (ADM), etc.

The batch can then be called e.g. N-FIN-Update balances for dimension set Ucet.

The advantages of this system are obvious - when sorting by name, we get thematically related batches together. In addition, easily distinguish batches that are properly scheduled and set up from batches that are ad-hoc triggered by a normal user. Last but not least, the name of the batch provides information about its content and schedule.

Tip 2: Administrator on holiday

Let's say we have a Batch job with thirty Batch tasks, each of which has some complex filter set. In addition, complex dependencies are defined between each Batch job. After some time, a request comes to add one more Batch task. The administrator Peter, who defined the original Batch job, is on vacation, so Pavel is in charge of the solution. He gradually discovers that he must be logged in as administrator to complete the task and that the header state must be Withold (the English translation of Collision is completely out of reality). Eventually, however, he finds that he is unable to complete the task anyway. - when you create a new record in Batch tasks, you get a pretty unremarkable error message Cannot create a record in Batch transactions (Batch). The corresponding AOS validation failed. With the same, he would have, or rather not, if he only wanted to change dependencies between existing Batch tasks. And now what?

the owner of the benefit, i.e. the person who created the original definition of the benefit
system administrator
neither of the above

The first and third cases represent two extremes. The owner of the benefit may dispose of his benefit at will. On the other hand, an ordinary user without administrative privileges cannot do anything with someone else's batch. The second case is interesting - an administrator who is not the author of the batch. He or she can change data related to the execution of the batch itself - e.g. suspend it or schedule it according to a new schedule. However, he or she cannot change the batch in terms of its substance - he or she cannot create new Batch tasks, change parameters (e.g. input filters) of existing Batch task records, or change the action itself (i.e. the class name). The reason for these restrictions is to allow auditing of changes - Paul cannot be allowed to make changes to the batch settings leading to undesirable behaviour, all on behalf of Peter. So Pavel now has two options - either to get Peter or to create his own definition of a Batch job from scratch.

In order not to disturb the principles of the audit and at the same time to allow the administrator Peter to go on vacation, we have developed in the package Blue Dynamic Batch new functionality. Instead of the "Created by" field value, we use a new "Batch Job Owner" field for auditing purposes. Its value is the same as the value in the Created field after the batch is created, but it can be changed using the "Take ownership" functionality.

After taking ownership, Pavel can then make the required changes, of course in his own name. The "Batch job owner" value is also written into the batch processing history, so we can easily find out when the change of ownership occurred.

Tip 3: What benefit groups are and are not for

Let's start with what batch groups are not good for - they are not good for organizing work with individual Batch jobs. The reason is simple - a batch group is a flag on a Batch task record, not a Batch job. If the need arises to better organize Batch jobs, you need to bring order to the naming of batches, see Tip 1.

So what are benefit groups good for? The main purpose is to be able to process different Batch tasks on different AOS servers. In installations where only one AOS server processes batch jobs, even a single batch group is sufficient (for this purpose, an unnamed group is automatically created after installing AX). However, there are situations where splitting jobs into individual AOSs can be useful. Examples might be batches that require computing power or batches that require some special resource that is only available on some AOS servers.

At Blue Dynamic, we've been thinking about how batch groups could serve even better. One of the problems we often encounter when running batches is the negative interaction between two or more batches running simultaneously. For example, sometimes the retail batches "Post Inventory" and "Post Statement" meet. The "Post Inventory" batch is scheduled to run every 10 minutes, while the "Post Statement" batch is scheduled to run once a day after the cash registers close. However, the "Post stock" batch runs "empty" after the cash registers are closed - logically it is clear that no new transactions will come in. So if we could set the "Post stock" batch to run only until a certain hour and then again the next morning, we could easily avoid such a collision and save ourselves the arduous task of figuring out how the two batches affect each other and how to override them to account for the parallel run.

The above problem can only be solved in standard AX by using multiple application servers. Create two batch groups - DAY and NIGHT. Put the "Post Inventory" batch into the DEN group and the "Post Statement" batch into the NOC group. Set the first AOS to run from 8:00 to 22:00 and to process only the DAY batch group. The second AOS is then set to run from 22:15 to 7:45 (leaving a 15 minute margin for longer running batches) and process only the NOC batch group.

But what to do if using another AOS server is not possible? We solve the problem by adding the option to specify the schedule at the batch group level.

Batch Group in Dynamics AX 2012 Blue Dynamic

After this modification, which is part of the Blue Dynamic Batch, we can get by with a single AOS server and still not have a conflict.

Tip 4: When batches start to "self-propagate" and a "nameless" batch group

In this tip, we will return to our "Post a statement" batch from the previous tip. The customer operates a network of stores and uses the AX Retail module. He sells during the day and records sales in the POS. In the evening, the POS closes and the data is transferred to AX. A report is then created in AX and needs to be posted. Since we want to run everything automatically, the job to post the statement is scheduled as a night job. So just after scheduling, we have a Batch job called "Post Statement", which has one Batch job with the same name and with the batch group NIGHT. However, after this batch starts processing, suddenly new Batch jobs appear that were not there before. What are they?

Some batch tasks are written in such a way that they divide the overall work into smaller units and these units are then processed in parallel in multiple Batch tasks. Dependencies can also be defined between these dynamic Batch tasks - for example, a final Batch task can be included at the end to take care of cleanup activities that can only take place when all Batch tasks have finished.
There is one tricky thing associated with this nice functionality. Dynamically created Batch tasks may not have the same batch group as the original Batch task, the exact behavior is determined by the programmer during batch development. What this can lead to is obvious - if dynamically created Batch tasks have a group assigned to them, whose execution is suspended, then they will never be started and the batch as a whole will remain "hanging" in the Processing state.

This situation is illustrated in the following screen - the last three Batch tasks have an unnamed batch group assigned to them. If the processing of the unnamed batch group is suspended, the posting of the report is never completed.

Batch jobs and their display in Dynamics AX 2012

This is actually a typical case - dynamically created Batch tasks are likely to have the same batch group as the main Batch task or an empty one. Since the internal logic of creating dynamic Batch tasks cannot always be influenced, it seems like a good idea not to restrict the processing of a batch group without a name. This is also important for another reason - one-off batches run by ad-hoc end users will very often just have a batch group without a name, because these less experienced users will often forget to specify the batch group.

Tip 5: How to get a restful night's sleep

Batches often run during the night hours, the successful completion of which depends on the proper functioning of the company the next day - For example, it can be a master plan generating order proposals and store replenishment or a batch preparing work for picking from the central warehouse.

In general, these are relatively complicated tasks, with many obstacles lurking, which may not always lead to a good finish. In addition, these are batches that take a long time to complete, so there may not be enough time to fix them the next morning. Therefore, it is very important to react quickly to the problems that are associated with these batches.

These are generally relatively complicated tasks that may not run out of time. It is important to respond quickly to batch-related problems.

Standard AX offers the use of alerts in this respect. It is possible to have AX create an alert for each batch if one of the events occurs:

the batch is duly completed
the batch ends with an error
the benefit is cancelled

The administrator can view the alerts in the AX application or have them emailed to a predetermined address.

Alert functionality is undoubtedly useful, but it is not sufficient to ensure a restful sleep. The problem occurs when the batch does not start processing at all, or takes an unusually long time to process. A common cause of extended duration may be negative interaction with another computationally complex task.

Often, AX is not the only system in the company that contributes to the success of the next day. Companies often run separate CRM, e-shop, POS systems, etc. In such cases, the complexity of monitoring the entire ecosystem increases and it is advisable to start monitoring the infrastructure from one central location. Then aletry seems to be a non-system solution and it is necessary to connect AX batches to central monitoring instead.

Blue Dynamic has successfully used Check MK in conjunction with dose monitoring. We have developed scripts that allow critical batches to be monitored for their progress - whether processing has started on time, what the progress is and whether they will finish within the prescribed time - and to respond to critical situations by sending an SMS message to the support phone. We are currently working on generalization and the ability to enter monitoring related parameters directly in the AX environment.