10 Data Interchange Improvements

10.1 pdrBatcher/AEMO Gateway Software

The latest version of the pdrBatcher Software responsible for transferring files to and from AEMO's participant file server to the participant's local Data Interchange folders. It can be installed in any environment supporting Java. is now referenced as the AEMO Australian Energy Market Operator Gateway Software v8.0.0 release, which is an upgrade from the pdrBatcher v7.6.0 installation.

10.1.1 Functional improvements

The functional improvements in AEMO Gateway Software v8.0.0 release include:

  • Encrypting variables in the properties file using ${encrypted:XXXX} syntax.
    batcher_{DS}_password_encrypted is discontinued. Replaced by ${encrypted:xxx} on batcher_{DS}_password.

  • Support for OAuth2 authentication.
    Change to data source configuration for Google cloud to reference OAuth2 framework.

  • Renaming application package from au.com.nemmco.* to au.com.aemo.*.

  • Each thread now using its own subfolder within the holding directory.

  • SYNCHRONISE mode now supporting delete operations at destination using configuration.
    New configuration properties:

  • batcher_thread_{N}_mode_param1

  • Property renaming batcher_thread_{N}_source_ack_translator to batcher_thread_{N}_ack_translator.

  • Adding extensibility documentation and examples.

  • Extensibility methods always execute in a daemon thread to reduce risk of custom code impacting overall performance.

  • Deprecated property:

  • batcher_thread_{N}_max_size_kB
    replaced with
  • batcher_thread_{N}_max_size_bytes

  • File sizes can now be entered in human readable format.

  • Azure blob supporting OAuth2 client credentials against EntraId.

  • pdrConnectionTest having command line parameters --configuration and --sources allowing the specific data source to test.

  • Adding SFTP and FTPS as data source providers to make these service types easier to configure. The system continues to support the legacy configuration of SFTP/FTPS under FTP.

  • Adding DEBUG log capability for transform configuration. New property batcher_translator_debug_dir and new --debug parameter for transform test CLI tool.

  • An example for configuring structured logging for JSON output.

  • Adding a feature for archiving transactions. New properties:

  • batcher_transaction_archive_data_source
  • batcher_transaction_archive_root_dir
  • batcher_transaction_archive_data_dir
  • batcher_thread_{N}_transaction_archive
  • batcher_thread_{N}_transaction_archive_data_dir

  • Adding a stop file status API for monitor reporting.

  • Adding parameters to control thread check and reporting interval:

  • batcher_thread_check_interval
  • batcher_thread_report_interval

  • Adding a command line tool for testing a sample file for validation against registered schemas.

  • Setting content-type based on filename when uploading to cloud blob.

  • Windows service wrapper migrating from WinRun4J (legacy) to Apache Procrun.

  • Various new file translation functionalities added:

  • XML_VARIABLE: Creates a variable based on an XSLT path specification
  • ALIAS_VARIABLE: Creates a variable based on an expression
  • JSON_TO_XML: Converts JSON to XML
  • XML_TO_JSON: Converts XML to JSON
  • SIGNATURE: Creates or verifies a digital signature

10.1.2 Bug fixes

The bug fixes in AGS includes:

  • File translators are now loaded in a predictable order, ensuring it applies the correct filemask.

  • Source and destination connections are now tested if there is an error in processing a file.

10.1.3 Security upgrades

Security upgrades to numerous third-party libraries.

10.1.4 Upgrading AEMO Gateway

Refer to the pdrBatcher/AEMO Gateway Software implementation instructions for details on how to perform an upgrade install.

10.2 pdrMonitor

The pdrMonitor is an optional monitoring and support utility used to observe and troubleshoot the operation of the AEMO Gateway Software and related components.

The pdrMonitor v1.4.0 release is an upgrade from any existing v1.3.0 installation.

10.2.1 Functional improvements

The functional improvements in the pdrMonitor v1.4.0 includes:

  • Encrypting variables in the configuration file using ${encrypted:XXXX} syntax.
    Property DbPasswordEncrypted is discontinued and is replaced by ${encrypted:xxx} on DbPassword.

  • Renaming application package from au.com.nemmco.* to au.com.aemo.*    

  • An example for configuring structured logging for JSON output.

  • Exposing B2B performance data in application transaction reporting.

  • Adding flow control visualisation for batcher.

  • Adding user event auditing and reporting.

  • Adding new dashboard widgets for B2B flow control and stopfile reporting.

  • Adding context sensitive help links.

  • Adding visualisation and description for batcher and loader settings tab.

  • Adding support for http2 protocol. New configuration:

  • /WebServer/Provider

  • Upgrade GUI installer extracting database connection parameters from existing XML configuration.

  • Adding new dashboard widgets for application and security events.

  • Windows service wrapper migrating from WinRun4J (legacy) to Apache Procrun.

  • Configuration to control thread check and reporting status into the log file.

10.2.2 Bug fixes

Corrects the defect that did not allow an IP address to be typed into monitor browser for adding a new connection/source.

10.2.3 Security upgrades

The security upgrades in pdrMonitor v1.4.0 include:

  • Security upgrades to numerous third-party libraries.
  • Add session timeout warning to browser interface.

10.2.4 Upgrading pdrMonitor

Refer to the pdrMonitor implementation instructions for details on how to perform an upgrade install.

10.3 pdrLoader

The pdrLoader Software responsible for loading files to and from participant's local Data Interchange folders to the participant's DBMS. It can be installed in any environment supporting Java. is an optional, participant‑hosted component that works alongside the AEMO Gateway Software to load data received from AEMO into a participant’s local relational database.

The pdrLoader v7.7.0 release is an upgrade from the v7.6.0 installation. 

10.3.1 Functional improvements

The functional improvements in pdrLoader v7.7.0 release include:

  • Encrypting variables in the properties file using ${encrypted:XXXX} syntax.
    Property db_password_encrypted is discontinued and replaced by ${encrypted:xxx} on db_password.

  • Renaming application package from au.com.nemmco.* to au.com.aemo.*

  • pdrFileEventInt changing from abstract class to interface.

  • A null option for db_user and db_password to allow for externally identified database access.

  • Adding extensibility documentation and examples.

  • Extensibility methods always execute in a daemon thread to reduce risk of custom code impacting overall performance.

  • Improving efficiency of SPARSE data row filter. This change to the row filter interface may impact any custom row filter implementations.

  • pdrConnectionTest having command line parameters --configuration and --sources allowing the specific data source to test.

  • An example for configuring structured logging for JSON output.

  • Adding parameters to control thread check and reporting interval:

  • loader_thread_check_interval
  • loader_thread_report_interval    

  • Upgrade GUI installer extracting database connection parameters from existing properties configuration.

  • Adding installation options to disable manifest and automate configuration processes.

  • Windows service wrapper migrating from WinRun4J (legacy) to Apache Procrun.

10.3.2 Bug fixes

The bug fixes in the pdrLoader includes:

  • Defect fix - validate any update to reconciliation point in time is historic.

  • Archive re-request only increments in table PDR_MANIFEST_LOG where the rejection message is other than Maximum number of files in trickle directory reached.

10.3.3 Security upgrades

Security upgrades to numerous third-party libraries.

10.3.4 Upgrading pdrLoader

Refer to the pdrLoader implementation instructions for details on how to perform an upgrade install.