File Handling

The goal of the Data Mirroring system is to allow file updates to be replicated on different systems, on different files, and/or a SQL database. This is accomplished by recording all file system updates on the main system to the system journals, transferring these journals to a target system, then re-applying the updates as required. Keyed files (VLR and EFF), as well as Indexed files, can be replicated.

Important Note:
A prerequisite to using Data Mirroring Configuration is that File System Journalization must already be enabled.

Data Mirroring cannot be currently used on files that have auto-increment enabled, as the generated auto-increment field is inserted after the journalization. This could result in different values on the files on the server and the system journals.

Prior to PxPlus 2023 Update 1:  Files with extended records cannot be journalized and therefore cannot be used with Data Mirroring.

PxPlus 2023 Update 1 or later:  Files with extended records can be journalized and therefore can be used with Data Mirroring. There is a size limit of 8MB for extended records. Records larger than that cannot be journalized, and any attempt to write the record will result in an Error #108: Record size error.

(Support for the journalization of files with extended records was added in PxPlus 2023 Update 1.)

Data Mirroring can be used for a variety of reasons. For instance, you can use it to create a reserve copy of your existing data in the event of a system failure. You can use it to create a central repository for branch operations or to load an external database for data analysis and reporting.

The Data Mirroring replication system consists of three processes, each handled by a different background program:

These programs are all designed to be run in the background, allowing the existing PxPlus application to continue to function regardless of the state of the connection to the target system. See Background Processes and Configuration Overview.

In addition, by enabling File System Journalization, there is minimal impact to the performance on the primary application. Should the connection to the target system be lost or is temporarily interrupted, these programs will retain pending and new updates in the journal files and forward them to the target system once communication is restored. (Data Mirroring using system journaling was added in PxPlus 2014.)

Note:
Setting up and maintaining a Data Mirroring system is easier with the addition of the Data Mirroring Configuration interface. (as of PxPlus 2017)

This interface allows you to configure the Send, Receive and Apply processes, monitor the activity for each process at a glance and view a detailed activity log.

Background Processes and Configuration Overview

Each background program (*plus/jrnl/send, *plus/jrnl/recv and *plus/jrnl/apply) is associated with its own configuration file (send.conf, recv.conf and apply.conf) that will define the input and output directories, target system and rules to apply for each process. These files are clear text files that contain the configuration parameters for each program. They need to reside in the directory from which each of the background programs is launched.

Standard configuration files are provided in the *plus/jrnl sub-directory and can be tailored to meet your needs.

Important Note:
It is strongly advised that you do not edit the files in this directory directly but instead copy them to another directory and run the processes from there to avoid software updates from potentially overwriting any customization you may do.

The default values for most of the values used by the system have been placed in these files as comments starting with #!!. Each line in the configuration file can be a comment or variable definition used to set a run-time option/parameter.

A variable definition contains the name of the variable, an = (equals sign) and the value. If the value is numeric, then the variable name will be considered a numeric variable. If not numeric, then a string variable will be created. If, due to the nature of the value, you need to force the assignment of a string value, simply surround the value with " (quotes).

Note:
The text values of YES or TRUE will be considered numeric 1, while NO or FALSE 0.

Example:

COMPRESS_DATA=YES is the same as COMPRESS_DATA=1.

Blank lines or lines starting with #, ! or ; are considered comment lines and can be interspersed anywhere in the configuration file.

For information on the configuration file parameters associated with the Send, Receive and Apply processes, see Send Configuration, Receive Configuration and Apply Configuration.

Controlling and Monitoring the Background Processes

The programs for the processes run in the background and can be managed by the existence of related files:

Where:

xxxxx is the name of the process (send, recv or apply).

Example:

Pauses if the file recv.pause (or send.pause/apply.pause) is found.
Terminates if the file recv.stop (or send.stop/apply.stop) is found.

By creating/removing these files, external utilities can easily control the execution of the background processes.

One typical use of this facility is to temporarily suspend the updating of a remote system while a backup is taking place. Prior to running a backup on the target system, you simply need to create a .pause file, wait for the process to pause, and then you may proceed with the backup without worrying about inconsistencies in the file contents, as all the updates will be deferred until the .pause file is deleted.

In a production system, if you wanted to take the mirrored files off-line to do a backup, you would need to pause the apply process. You could pause any of the processes (send, recv or apply); however, as the send and receive can be used to make off-site copies, it is generally better to pause the apply process if doing a backup.

The current state of each of the background tasks can be determined through the presence of the following files:

• The file recv.is.active (or send.is.active/apply.is.active) while active.
• The file recv.is.paused (or send.is.paused/apply.is.paused) while paused due to .pause file presence.
• The file recv.is.connecting (or send.is.connecting) while trying to establish communications.

Journal Files Transfer Process

The send and receive background processes are used to transfer the journal files to the target site:

• The send process (*plus/jrnl/send) monitors the journal files for updates (on the system with the application data) and sends them to the target system. For the "sender" to connect to the "receiver", the receiving machine must be accessible either by IP address or by name. Both will wait for the other to connect, providing the "receiver" has an open connection port.
• The receive process (*plus/jrnl/recv) receives the journal files and creates equivalent journals on the target (mirroring) system.

See Also

File System Journalization
Data Mirroring Configuration