Participant Impact

For more information, see Project.

DO NOT apply the scripts to pre-production or production environment.

Participant development support environment

Participants need to set up a separate instance of the environment to deploy the draft scripts for Data Model The definition of the interface to participants of data published by AEMO for gas or electricity. A database conforming to the Data Model can contain a local copy of all current participant-specific data recorded in the main database. The Data Model includes database tables, indexes, and primary keys. 5.3. For more information on the participant development support environment, see IESS Participant Development Support Environment. For transition steps in the PDSE environment, see the PDSE- MMS Data Model Release Notes in Releases\MMS Data Model\PDSE\v5.3_BETA in the Participant File Share server.

Pre-production refresh

For more information on the pre-production refresh timeline, see Technical Specification Portal.

EMMS data model v5.3

Participants wanting to receive the new and updated Data Model information in their Data Interchange A set of cooperating applications used to replicate data between AEMO's energy market systems and a participant's DBMS conforming to the MMS Data Model. environments must upgrade to the latest version of the Data Model v5.3.

NEXT_DAY bids

The following section provides more information on NEXT_DAY_* tables:

During the pre-release of Data Model 5.3 in PDSE, AEMO Australian Energy Market Operator added the DIRECTION field to the following reports as a primary key.

File_id

Added report column

Target tables

Report version

Latest/legacy

Data subscription

NEM_BIDS

DIRECTION

BIDDAYOFFER
BIDOFFERPERIOD

1

Keep same report version

Not required

BID_MOVE_COMPLETE

DIRECTION

BIDDAYOFFER_D
BIDPEROFFER_D

2

Keep same report version

Not required

NEXT_DAY_OFFER_ENERGY_LEGACY

DIRECTION

BIDDAYOFFER
BIDOFFERPERIOD

1

Keep same report version

Not required

NEXT_DAY_OFFER_FCAS_LEGACY
NEXT_DAY_OFFER_FCAS_2_LEGACY

DIRECTION

BIDDAYOFFER
BIDOFFERPERIOD

1

Keep same report version

Not required

NEXT_DAY_OFFER_ENERGY_SPARSE

DIRECTION

BIDDAYOFFER
BIDOFFERPERIOD

1

Keep same report version

Not required

NEXT_DAY_OFFER_FCAS_SPARSE

DIRECTION

BIDDAYOFFER
BIDOFFERPERIOD

1

Keep same report version

Not required

With the release of Data Model 5.3, the current version of the reports additionally includes the PERIODIDTO field. This field ensures participants who apply the prerelease scripts, have the PERIODIDTO field available and populated to transition to Sparse model.

File_id

Target tables

Report version

Latest/legacy

Data subscription

NEM_BIDS

BIDDAYOFFER - DIRECTION
BIDOFFERPERIOD- DIRECTION, PERIODIDTO

1

Keep same report version

Not required

BID_MOVE_COMPLTE

BIDDAYOFFER_D- DIRECTION
BIDPEROFFER_D- DIRECTION

2

Keep same report version

Not required

NEXT_DAY_OFFER_ENERGY_LEGACY

BIDDAYOFFER- DIRECTION
BIDOFFERPERIOD- DIRECTION, PERIODIDTO

1

Keep same report version

Not required

NEXT_DAY_OFFER_FCAS_LEGACY
NEXT_DAY_OFFER_FCAS_2_LEGACY

BIDDAYOFFER- DIRECTION
BIDOFFERPERIOD- DIRECTION, PERIODIDTO

1

Keep same report version

Not required

NEXT_DAY_OFFER_ENERGY_SPARSE

BIDDAYOFFER- DIRECTION
BIDOFFERPERIOD- DIRECTION, PERIODIDTO

1

Keep same report version

Not required

NEXT_DAY_OFFER_FCAS_SPARSE

BIDDAYOFFER- DIRECTION
BIDOFFERPERIOD- DIRECTION, PERIODIDTO

1

Keep same report version

Not required

The Data Model 5.3 release adds new version of the following reports. Participants subscribed to the current version are moved to LEGACY.

File_id

Target tables

Report version

Data subscription

NEM_BIDS_LEGACY

 

BIDOFFERPERIOD- DIRECTION, PERIODIDTO

1

Participants move to LEGACY version of these reports and MUST subscribe to latest version to receive the ENERGY_LIMIT field after they have upgraded to the latest Data Model 5.3.

NEM_BIDS

BIDOFFERPERIOD- DIRECTION, PERIODIDTO, ENERGY_LIMIT

2

BID_MOVE_COMPLTE_LEGACY

BIDPEROFFER_D- DIRECTION

2

BID_MOVE_COMPLETE

BIDPEROFFER_D- DIRECTION, ENERGY_LIMIT

3

PD7DAY changes

The PD7DAY tables have the following changes:

  • The Data Model 5.3 release adds a new PD7DAY package along with new Data Model tables and reports.
  • The Data Model 5.3 upgrade renames the existing PD7DAY tables to *_PRE53 and adds new structure for these tables.
  • After applying the Data Model 5.3 upgrade, participants need to manually subscribe to the new PD7DAY reports.
  • To migrate historical data from any custom loading solutions, participants must develop appropriate scripting for the backfilling activity.
  • Participants currently subscribed to PD7DAY_GPG are moved to the PD7DAY_LEGACY report with Data Model 5.3 release. With this, participants must perform below additional steps:
    • Run the ensure below query to PD7DAY latest config is ingested.

      select * from pdr_report_records t where t.REPORT_NAME like '%PD7DAY%';

    • Participants who have applied the custom loader mapping configuration for loading PD7DAY_GPG data need to ensure that their destination table matches with the PD7DAY mapping configuration as shown below.

    • Participants who have active report configuration as a part of custom loader mapping will need to manually activate MARKET_SUMMARY,1 configuration and leave MARKET_SUMMARY,2 configuration as 0 till Data Model is upgraded to v5.3. 
    • Participants fetching PD7DAY_GPG data from nemweb, need to remove that config from pdrBatcher.properties file and instead subscribe to PD7DAY_LEGACY file from AEMO portal.
  • With these changes and observing existing practices to only publish Energy Rules Terms latest reports within nemweb, AEMO will stop publishing PD7DAY_GPG and PD7DAY_LEGACY files in nemweb from 17th May.

Existing PD7DAY_GPG non data model reports

File_id

Report_id

PD7DAY_GPG

GPG_CASESOLUTION

PD7DAY_GPG

GPG_MARKET_SUMMARY_1

PD7DAY_GPG

GPG_CONSTRAINTSOLUTION_1

PD7DAY_GPG

GPG_INTERCONNECTORSOLUTION_1

PD7DAY_GPG

GPG_PRICES_2

New PD7DAY tables in Data Mode 5.3

File_id

Data model table

Changes to the existing structure

PD7DAY

PD7DAY_CASESOLUTION

No changes

PD7DAY

PD7DAY_MARKET_SUMMARY

DATETIME field has changed to INTERVAL_DATETIME.

This is to cater to some database platforms in which the DATETIME field can be a reserved word.

PD7DAY

PD7DAY_CONSTRAINTSOLUTION

No changes

PD7DAY

PD7DAY_INTERCONNECTORSOLUTION

No changes

PD7DAY

PD7DAY_PRICESOLUTION

No changes

DISPATCHLOAD table

In the DISPATCHLOAD table, the ENABLEMENTMIN and ENABLEMENTMAX columns can be negative values for BDUs.

Bidding data model table updates

AEMO introduces a new Primary Key (PK) of DIRECTION field to Bid See Relevant Rules or Procedures Package tables:

  1. The 10-band bid model is preserved with inclusion of a new DIRECTION field in the primary key (PK) of the main bidding tables of BIDDAYOFFER, BIDOFFERPERIOD, BIDDAYOFFER_D, BIDPEROFFER_D.

  2. For a BDU, adding PK implies there are 2 records with same bid data fields with DIRECTION informing which 10 bands are associated to GEN (generation) and LOAD side, that is, separate 10 band model for GEN and LOAD side of the BDU.

  3. AEMO back populates the DIRECTION field for all retrospective bid reports (LEGACY/LATEST) in PROD/PREPROD for all unit types and provide participants with an executable script as part of the Data Model 5.3 bid table pre-release (early/mid February) to back populate same in their own databases (as required).

  4. The DIRECTION field is populated in bid reports for all unit types regardless of if not provided by participant in submitted bid, that is, AEMO derives if not provided in bid (DIRECTION is mandatory for BDU, optional for all other unit types).

  5. Participants remaining on Data Model version 5.2 or less (and associated PDRloader):

    • not consume DIRECTION field present in reports (LEGACY/LATEST) but not supported by PDRloader version to consume.

    • have limitation for BDU only consume one side of bid bands provided (first 10 bid bands) and not second set of 10 bands.

    • for participant to be in line with AEMO data, they must upgrade to the latest version of Data Model v5.3.

Settlements transition plan

Previously, AEMO had communicated the Settlements See Relevant Rules or Procedures transition would complete with the next Data Model release. However, given there are no Settlements or Billing related changes in the Data Model 5.4 scheduled in October 2024, the IESS Integrating Energy Storage Systems Settlements transition period will now continue to the subsequent Data Model release tentatively scheduled for April/May 2025.

Accessing the Settlements tables during transition

Participants must subscribe to the SETTLEMENTS_LEGACY report to continue receiving the SETCPDATA, SETGENDATA, SETSMALLGENDATA after 2 June 2024.

Participants upgrading to Data Model 5.3, must also subscribe to SETTLEMENTS report to get updates for the new tables.

Settlements data

Settlement dates prior to 2 June 2024

Settlement dates on or after 2 June 2024

Existing tables (SETCPDATA, SETGENDATA, SETSMALLGENDATA)

Yes, provided through SETTLEMENTS and SETTLEMENTS_LEGACY reports

Yes, provided through SETTLEMENTS_LEGACY report only until the next Settlements related Data Model release

New tables (e.g. SET_ENERGY_TRANSACTIONS)

Not provided

Yes, provided through SETTLEMENTS report

Data subscription

  • Participants are automatically subscribed to any newly introduced reports.

  • Participants currently subscribed to the existing reports modified in this release move to the legacy version of these reports on the day of the Data Model release.

  • Participants should only unsubscribe from legacy version after subscribing to the current version of the report.

  • Data Model releases contains updates for various projects. When the project goes live, the updated versions of the reports are made available.

  • Participants are only able to subscribe to the new versions of these reports when the project goes live. AEMO notifies participants when the reports are available for subscription.

For help, see:
Unsubscribe from files
Subscribing to new EMMS Data Model files

What happens if I do not upgrade to Data Model 5.3?

If the participants do not upgrade to EMMS Energy Market Management System (formerly MMS); software, hardware, network and related processes. Data Model 5.3, there will be impacts to both Settlements and Billing tables.

Settlements

The following tables are populated for participants staying with Data Model 5.2 as part of the IESS Transitional Reporting.

  • SETCPDATA

  • SETGENDATA

  • SETSMALLGENDATA

If you do not upgrade to Data Model 5.3, the EXPENERGY and EXPORTENERGY columns in the SETGETDATA and SETSMALLGENDATA tables are UFEA adjusted. You will not be able to reconcile the UFEA amount.

After the release of the Data Model 5.3, the Export MWh displayed in these tables are UFE Adjusted MWh Values.

The details of columns modified:

SETCPDATA

IENERGY

NUMBER(16,6)

Import Gross energy into the pool – MWh

XGENERGY

NUMBER(16,6)

Export Gross energy from the pool – MWh

IENERGY

NUMBER(16,6)

Import energy into the pool – MWh. After IESS INEnergy is not adjusted by UFEA

XNENERGY

NUMBER(16,6)

Export energy from the pool – MWh, Plus UFE Allocated MWh. UFEA is always adjusted with Export Energy post IESS. Negative UFEA increases the XNEnergy and Positive UFEA decreases the XNEnergy

SETGENDATA

GENERGY

NUMBER(16,6)

Generated Gross Energy -MWh

NETENERGY

NUMBER(16,6)

Generated  Gross Energy – MWh

EXPENERGY

NUMBER(16,6)

Export Energy (Generator Purchases) (MWh) – This is adjusted by UFEA

EXPENERGYCOST

NUMBER(16,6)

Export Energy Cost ($) ($ value corresponding to the Exp Energy which is adjusted by UFEA

SETSMALLGENDATA

IMPORTENERGY

NUMBER(16,6)

The import direction value for the meter read (MWh)

EXPORTENERGY

NUMBER(16,6)

The export direction value for the meter read (MWh) adjusted by UFEA

IMPENERGYCOST

NUMBER(16,6)

Import Energy Cost ($)

EXPENERGYCOST

NUMBER(16,6)

Export Energy Cost ($) (($ value corresponding to the Export Energy which is adjusted by UFEA)

Billing

If participants do not upgrade to Data Model 5.3 and are subscribed to bidding and NEXT_DAY files, the tables start failing since there are primary key changes to the Bidding tables.