6 Participant Data Replication Loader

The Participant Data Replication Loader (pdrLoader Software responsible for loading files to and from participant's local Data Interchange folders to the participant's DBMS. It can be installed in any environment supporting Java.) upgrades from v7.5.0 to v7.6.1.

6.1 Functional enhancements

The functional enhancements for pdrLoader are:

  • File purge configuration now supports additional mask_mode logic.
  • New data source internal can be defined to use an alternate persistent data store (for example, cloud blob storage). The default is a local filesystem
  • Performance and logging files for pdrLoader are now prefixed with pdrLoader to ensure uniqueness
  • Now automatically restarts worker processing threads if they are detected as dead
  • Adds functions to DATA_FILTER row filter matches, exists, !exists
  • New optional property to customise PDR_* configuration in memory cache timeout in seconds.
  • New optional property to customise activity timeout on reconciliation in minutes
  • New optional property to allow a delay to be introduced to the reconciliation process.
  • Supports loading of flat CSV files without column headings, the column headings are implied by the database table
  • Adds support for managing configuration of non data model files configuration
  • Adds sparse data row filter
  • LASTCHANGED row filter now supports operator in PDR_REPORT_RECORDS.ROW_FILTER_PARAM2. The default is value is <
  • Supports file handler lookup based on filename as well as extension, for example, .gas.csv Comma Separated Values. A file format for data using commas as delimiters. is equivalent to .gascsv
  • Adds native support for Market JSON JavaScript Object Notification (IDX). Filename mask for this processor is *.market.json
  • Enhances processing of plain JSON file to handle either nest object or array under data element
  • Adds EXTRACTED_AT row filter for supporting load of public WEM data
  • Adds generic XML eXtensible Mark-up Language. file processor loading capabilities
  • Adds enhancements to file process order to support PRIORITY and sort chaining definitions
  • Adds new column transform DATABASE_SEQUENCE, DEFAULT_VALUE, DEFAULT_VALUE_SQL
  • Populates PDR_REPORT_AUDIT.FILE_PROCESS_START column when a report starts processing
  • Supports inactivity timeout on loader internal data source
  • Adds functionality to gracefully handle primary key violations on concurrent threads
  • Adds row filters COLUMNS_MATCH and COLUMNS_NO_NOT_MATCH

6.2 Bug fixes

The bugs fixes for pdrLoader are:

  • Fixes defect with process_priority and file bump operations - ensure resumption of file load starts from where file load was previously paused
  • Fixes defect for purging manifest request rows from PDR_REQUEST_LOG for mySQL
  • Fixes defect for DELETE transactions
  • Fixes defect with error handling with loss of database connection after file process but before audit table update
  • Fixes issue with reconciliation which could result in unnecessary report re-requests
  • Adds PDR_DATA_FILTER_CONFIG to support dump file

6.3 Security upgrades

The security upgrades for pdrLoader are: