Data Mirroring

Data Mirroring Configuration

Data Mirroring Configuration simplifies the setup and maintenance of the Data Mirroring system by providing an interface for configuring the Send, Receive and Apply processes, monitoring the activity for each process at a glance and viewing a detailed activity log.

(Data Mirroring Configuration was added in PxPlus 2017.)

Important Note:
A prerequisite to using Data Mirroring Configuration is that File System Journalization must already be enabled.

Prior to PxPlus 2023 Update 1:  Files with extended records cannot be journalized and therefore cannot be used with Data Mirroring.

PxPlus 2023 Update 1 or later:  Files with extended records can be journalized and therefore can be used with Data Mirroring. There is a size limit of 8MB for extended records. Records larger than that cannot be journalized, and any attempt to write the record will result in an Error #108: Record size error.

(Support for the journalization of files with extended records was added in PxPlus 2023 Update 1.)

To invoke Data Mirroring Configuration, use one of the following methods:

Location

Method

From the PxPlus IDE Main Launcher

Expand the Data Management category and select Data Mirroring Configuration.

From the PxPlus Command line

Enter: DM

(The Command line method of invoking Data Mirroring Configuration was added in PxPlus 2018.)

The screen shot on the left is displayed initially when no processes have been set up. The screen shot on the right is an example showing the Send, Receive and Apply processes already set up and active.

 



Data Mirroring Configuration (No Processes Set Up)



Data Mirroring Configuration (Processes Set Up and Active)

This interface consists of the following:

(Status Bitmap)

Indicates the status of the process. Possible statuses are:

Connecting

The process is currently waiting to connect.

Active

The process is currently running.

Stopped

The process is currently not running.

Paused

The process is currently paused.

Left or right clicking on the bitmap displays a popup menu with selections for Start Process, Pause Process, Resume Process and Halt Process, which are the same as selecting the Start, Pause, Resume and Halt buttons beside the grid.

(Left clicking to display popup menu was added in PxPlus 2017 Update 0002.)

Description

Displays four lines of information about the process:

 

1st Line:

Server: followed by the process Mirror ID, Process Type, Description

 

2nd Line:

Process status (Connecting, Active, Stopped, Paused)

 

3rd Line:

Home directory pathname

 

4th Line:

Configuration file name

Double click on the description to launch the Data Mirroring Configuration - Edit Process interface for the selected process.

Note:
If the configuration file for a process has not yet been defined, the Description for that process displays in red text with the words Not Defined after the Configuration File Name.

 

The buttons beside the grid (explained below) are used to change the status of a selected process. They are enabled/disabled depending on the current status of the selected process.

Example:

If the selected process is Stopped, only the Start and View Log buttons are enabled.

Note:
If the configuration file for a process has not been defined, the Start, Pause, Resume, Halt and View Log buttons are disabled.

Start

Starts the process as a minimized, spawned PxPlus task. This button will be enabled if an existing process with a Stopped status is selected.

Pause

Pauses the process. This button will be enabled if an existing process with an Active status is selected.

Resume

Continues the process. This button will be enabled if an existing process with a Paused status is selected.

Halt

Stops the process and removes the minimized, spawned PxPlus task. This button will be enabled if an existing process with either a Connecting or Active status is selected.

View Log

Displays a separate panel with the log file pathname and contents. This button will be enabled if a log file exists for the selected process.

Note:
These log files are text files that could potentially become quite large over time. Once a log file reaches its capacity, it will no longer be able to display all of the logged entries. Therefore, to prevent the file from becoming too large, it is strongly recommended that you reset the log file periodically by either renaming it or deleting it.

 

New

Launches the Data Mirroring Configuration - New Process interface for creating a new process.

Edit

Launches the Data Mirroring Configuration - Edit Process interface for editing the selected process.

Delete

Launches the Confirm Deletion dialog for confirming the deletion of the selected process.

Select the Include Configuration/Log files check box on this dialog to delete the associated Configuration and Log files for the selected process. By default, this check box is not selected.

Exit

Closes the Data Mirroring Configuration interface.

Right clicking on a process in the grid displays a popup menu with selections for Start Process, Pause Process, Resume Process and Halt Process. These are the same as selecting the Start, Pause, Resume and Halt buttons beside the grid or using the popup menu when you left or right click on the Status Bitmap.

Note:
Any change made to the status of a process is reflected in the grid when the change is actually applied. This may take several seconds after the change was requested, depending on configuration settings.

Data Mirroring Configuration - New Process

The Data Mirroring Configuration - New Process interface is used to create a new process, It is invoked by selecting the New button on the main Data Mirroring Configuration interface.

This interface consists of the following:

Mirror ID

An ID used to identify the process. Maximum length is 20 characters.

Helpful Tip:
Assigning the same Mirror ID to associated processes makes it easier to identify which processes are interconnected.

Example:

A branch office is configured to send sales data to the head office. The branch office defines the Send process with a Mirror ID of SALES. The head office defines the Receive process with a Mirror ID of SALES and the Apply process with a Mirror ID of SALES.

Process Type

(Available when Mirror ID is entered)

Type of process. Drop box selections are Send, Receive, Apply.

Description

(Available when Mirror ID and Process Type are entered)

An optional description that is used to identify the process. Maximum length is 30 characters. After the new process is written, this description can be modified, if needed.

Home Directory

(Available when Mirror ID and Process Type are entered)

Working directory for launching the process. Enter the full pathname or click the Query button to specify the directory. When defining a new process, this defaults to the current working directory.

Configuration File Name

(Available when Mirror ID and Process Type are entered)

Name of the text configuration file to be associated with the process. Defaults to send.conf, recv.conf or apply.conf, depending on the Process Type selected. This field cannot be left blank.

When you enter a file name and click the Write button or the Edit Configuration button (described below), a message asks if you want to copy from the standard configuration file (for the selected Process Type).

Responding Yes creates the new configuration file in the Home Directory using a copy of the standard configuration file and launches a separate interface for specifying the configuration file parameters.

Responding No does not create the new configuration file. You are returned to the main Data Mirroring Configuration interface with the description for the new process displayed in red text and the words Not Defined appearing after the Configuration File Name in the description. When you are ready to define this file, select this process for editing and click the Edit Configuration button. See Edit Configuration below.

Note:
You cannot have duplicate configuration file names in the same Home Directory.

(Duplicate configuration file name checking was added in PxPlus 2018.)

Edit Configuration

This button next to Configuration File Name is used to edit the configuration file parameters for the process. See Edit Configuration below.

View Log

Displays a separate panel with the log file pathname and contents. This button is enabled if a log file exists for the selected process.

Write

Writes the process record. If a non-existent Configuration File Name has been entered, a message asks if you want to copy from the standard configuration file (for the selected Process Type). See Configuration File Name above.

Clear

Clears the process record.

Exit

Closes the interface without saving the process record.

Data Mirroring Configuration - Edit Process

The Data Mirroring Configuration – Edit Process interface is used to edit an existing process and is invoked by clicking the Edit button on the main Data Mirroring Configuration interface.

Note:
An existing process can be edited only when its current status is Stopped; otherwise, the settings on this interface are for viewing only.

The Data Mirroring Configuration – Edit Process interface is displayed. This interface is similar to the Data Mirroring Configuration - New Process interface with a few exceptions: Mirror ID, Process Type and Clear are disabled, and a View Log button, similar to the one found on the main Data Mirroring Configuration interface, is displayed. The current status (i.e. Stopped) is indicated in red text next to the Process Type.

Edit Configuration

Click the Edit Configuration button next to Configuration File Name to invoke the Edit Configuration interface that is used to view and edit the configuration file parameters associated with a process. These parameters are stored in a text configuration file (send.conf, recv.conf or apply.conf), depending on the Process Type.

Below is the Edit Configuration interface for a Send process. The Receive and Apply processes have their own Edit Configuration interfaces. The configuration file parameters associated with the Send, Receive and Apply processes are explained in the tables below.

Send Configuration

This file (default is send.conf) contains the configuration parameters for the journal transmission program *plus/jrnl/send. This program monitors journal files for updates and sends a series of journal files to another system.

Parameter

Variable

Description

General Information

 

 

Directory to Hold Journal Files

DIR

The DIR value can be defined to override the directory in which the journal files to send can be found. This directory will also be used to hold the log and other related files.

If not specified, the current directory will be used.

Log File Name for Error Messages

LOGFILE

The LOGFILE value defines the pathname for the log file that will receive any error messages. The default value is "send.log".

If not set, the system will use the FILEID+".log" as the name of the file in the journal directory. Should an error occur in the configuration file prior to the definition of the log file, the default log file of "send.log" will be used.

ID to Uniquely Identify Control Files

FILEID

The FILEID value can be used to uniquely identify control files (error log, etc.) used by this sending process. The default value is "send", thus the default error log will be "send.log".

The FILEID can be changed if running multiple transmission programs from within the same directory. Using different values allows you to properly track and monitor the different transmission programs.

If present and not "send", all log files will be prefixed by the FILEID.

Compress Data Being Sent

COMPRESS_DATA

The COMPRESS_DATA value can be used to indicate that the data being sent is to be compressed. It must be set the same at both ends (sender and receiver). By default, this check box is set to YES.

Connection Information

IP Address / Name of Target System

TCPADDR

The TCPADDR value defines the IP address and/or name to the target system.

If not entered, "localhost" is assumed.

See the server parameter in [TCP] Transmission Control Protocol.

Port

TCPPORT

The Port to use is defined by the TCPPORT value. The default value is 4913.

See the socket parameter in [TCP] Transmission Control Protocol.

Note:
Both a sender and receiver must use the same Port, which must be unique for each corresponding sender and receiver.

TCP Options

TCPOPTIONS

Any additional PxPlus TCP options can be provided in TCPOPTIONS.

See the tcp_opts parameter in [TCP] Transmission Control Protocol.

Password

PASSWD

The PASSWD value is used to secure the handshake. Both a sender and receiver must have the same PASSWD value. Using the PASSWD helps eliminate spoofing of the connection.

If not specified, the default value (defined in the text file) is used.

Seconds Before Checking for New Data

IDLE_WAIT

The IDLE_WAIT defines how long (in seconds) that the system will sleep between checking the journal files to see if more data is ready to be sent. The default value is 5.

Seconds Before Re-connect Attempt

RECONNECT_TIME

The RECONNECT_TIME defines the time (in seconds) that the system will wait before attempting to reconnect to the target system should an error occur and the connection gets dropped. The default value is 30.

Seconds Before Checking Connection is Active

CHECK_TIME

The CHECK_TIME defines how often (in seconds) that the system will forcibly send a message to the target system to check that the connection is still alive. The default value is 30. The check request will only be sent out if no other activity occurs.

If the connection is not active, the port and journal will be closed, and a reconnect will be attempted after a wait of RECONNECT_TIME.

Seconds Before Considering Response Lost

RECV_TIME

The RECV_TIME defines how long (in seconds) that the system will wait for a response from the target system before considering the message lost. The default value is 10.

If no response is received within the RECV_TIME, the port and journal will be closed, and a reconnect will be attempted after a wait of RECONNECT_TIME.

Notification Information

Email Address for Error Reporting

MAILTO

Whenever a serious error occurs, the system can automatically send an email to the email address specified in MAILTO.

Note:
If MAILTO and SMTP are not set, no mail is generated.

Examples of Send Errors:

** Password rejected by remote -- stopping

** Target is in use -- stopping

** Invalid response: [response] received from [tcp addr] – Resetting line

** Journal [journal number] send error – Receiving system file is larger than host - stopping

** Missing journal.[journal number] on server (target system asked for file higher than server) – stopping

SMTP Server

SMTP

The error email message uses the SMTP server defined in SMTP.

Note:
If MAILTO and SMTP are not set, no mail is generated.

Email From Address

MAILFROM

The From address for the error email message can be set using the MAILFROM parameter.

 

View File

Button that is used to view the configuration parameters for the associated process.

Note:
While you can modify the parameters directly within this file, it is strongly recommended to use the Edit Configuration interface to make any changes.

Save

Saves changes and closes the Edit Configuration interface.

Cancel

Closes the Edit Configuration interface without saving any changes.

Receive Configuration

This file (default = recv.conf) contains the configuration parameters for the journal transmission program *plus/jrnl/recv. This program receives the journal files sent by *plus/jrnl/send and replicates the journals on a new system.

Parameter

Variable

Description

General Information

 

 

Directory to Hold Journal Files

DIR

The DIR value can be defined to override the directory in which the journal files will be created. This directory will also be used to hold the log and other related files.

If not specified, the current directory will be used.

Log File Name for Error Messages

LOGFILE

The LOGFILE value defines the pathname for the log file that will receive any error messages. The default value is "recv.log".

Compress Data Being Sent

COMPRESS_DATA

The COMPRESS_DATA value can be used to indicate that the data being received has been compressed. It must be set the same at both ends (sender and receiver). By default, this check box is set to YES.

Connection Information

Port

TCPPORT

The Port to use is defined by the TCPPORT value. The default value is 4913.

See the socket parameter in [TCP] Transmission Control Protocol.

Note:
Both a sender and receiver must use the same Port, which must be unique for each corresponding sender and receiver.

TCP Options

TCPOPTIONS

Any additional PxPlus TCP options can be provided in TCPOPTIONS.

See the tcp_opts parameter in [TCP] Transmission Control Protocol.

Password

PASSWD

The PASSWD value is used to secure the handshake. Both a sender and receiver must have the same PASSWD value. Using the PASSWD helps eliminate spoofing of the connection.

If not specified, the default value (defined in the text file) is used.

Seconds Before Checking for New Data

IDLE_WAIT

The IDLE_WAIT defines how long (in seconds) that the system will sleep between checking the .stop / .pause files and awaiting connection/data. The default value is 5.

Seconds Before Considering Response Lost

RECV_TIME

The RECV_TIME defines how long (in seconds) that the system will wait for a response from the target system before considering the message lost. The default value is 10.

If no response is received within the RECV_TIME, the port and journal will be closed, and a reconnect will be attempted after a wait of RECONNECT_TIME.

Notification Information

Email Address for Error Reporting

MAILTO

Whenever a serious error occurs, the system can automatically send an email to the email address specified in MAILTO.

Note:
If MAILTO and SMTP are not set, no mail is generated.

Examples of Receive Errors:

** Cannot open server port [tcp path] – shutting down

** Expected Minimum journal file number from sender

** Improper password received from [key of tcp channel]

** Cannot create file journal.[journal number] on receiver: [message]

** Received end of journal.[journal number] yet no data received – stopping

SMTP Server

SMTP

The error email message uses the SMTP server defined in SMTP.

Note:
If MAILTO and SMTP are not set, no mail is generated.

Email From Address

MAILFROM

The From address for the error email message can be set using the MAILFROM parameter.

 

View File

Button that is used to view the configuration parameters for the associated process.

Note:
While you can modify the parameters directly within this file, it is strongly recommended to use the Edit Configuration interface to make any changes.

Save

Saves changes and closes the Edit Configuration interface.

Cancel

Closes the Edit Configuration interface without saving any changes.

Apply Configuration

This file (default = apply.conf) contains the configuration parameters for the journal application program *plus/jrnl/apply. This program applies the changes recorded in a series of journal files onto the files on the target system/directory.

Note:
The *plus/jrnl/apply program will check for the apply.stop file every 100 transactions to allow the Apply process to be stopped before completing the entire series of journal files.

(The check for the apply.stop file was added in PxPlus 2021.)

 

Parameter

Variable

Description

General Information

 

 

Directory to Hold Journal Files

DIR

The DIR value can be defined to override the directory in which the journal files are found. This directory will also be used to hold the log and other related files.

If not specified, the current directory will be used.

Log File Name for Error Messages

LOGFILE

The LOGFILE value defines the pathname for the log file that will receive any error messages. The default value is "apply.log".

If not set, the system will use "apply.log" as the name of the file in the journal directory. Should an error occur in the configuration file prior to the definition of the log file, the default log file of "apply.log" will be used.

Connection Information

Seconds Before Checking for New Data

IDLE_WAIT

The IDLE_WAIT defines how long (in seconds) that the system will sleep between checking the journal files to see if more data is ready to be applied. The default value is 5.

Target Output Directory

OUTPUT_DIR

The OUTPUT_DIR value defines the target output directory. If not specified, the current working directory will be used.

Prefix

PREFIX

The PREFIX value can be used to define a standard PxPlus prefix value to be applied to the output.

PxPlus Prefix File

PREFIX_FILE

The PREFIX_FILE value can be used to define a PxPlus prefix file to be used when looking up the actual pathnames for the files.

Program for User Journal Messages

USERJRNL

The USERJRNL directive defines the name of the program to call to process any user defined journal messages. This program will be passed the contents of the string from a SYSTEM_JRNL WRITE <string> directive.

Redirect Options

REDIRECT

The REDIRECT option (there may be multiple) allows you to define a mask and replacement target file name. The mask and replacement name must be separated by a comma. If the replacement name is - (dash), then the output is discarded.

In addition, in the file name, %0, %1, %2, ... will be changed with the full mask or segments.

Examples:

REDIRECT=AR(.*),AcctRec%1
REDIRECT=GL9,-

Program Option (mask,file)

PROGRAM

The PROGRAM option provides a file name mask, which, if matched, will cause the system to send the data to a program. This program will be passed the following values:

Journal Record Type

1 character (string)

Journal Data

Record contents (string)

Journal Key Info

External key size for Keyed file, Index for Indexed (numeric)

File Name

Adjusted file name (string)

File Pathname

Original file pathname (string)

Notification Information

Email Address for Error Reporting

MAILTO

Whenever a serious error occurs, the system can automatically send an email to the email address specified in MAILTO.

Note:
If MAILTO and SMTP are not set, no mail is generated.

Examples of Apply Errors:

** Error X on read of journal file: [path to journal file]

** Program [program name] has exited with an error [error number] processing [file name]

SMTP Server

SMTP

The error email message uses the SMTP server defined in SMTP.

Note:
If MAILTO and SMTP are not set, no mail is generated.

Email From Address

MAILFROM

The From address for the error email message can be set using the MAILFROM parameter.

 

View File

Button that is used to view the configuration parameters for the associated process.

Note:
While you can modify the parameters directly within this file, it is strongly recommended to use the Edit Configuration interface to make any changes.

Save

Saves changes and closes the Edit Configuration interface.

Cancel

Closes the Edit Configuration interface without saving any changes.