Sparse Data Model FAQs

Is the Sparse Model applicable to BIDDAYOFFER_D and BIDPEROFFER_D files containing 5-minute data?

We looked at it but decided no because the data in those tables is a composite offer across an entire market day, representing the 5-minute quantities for a particular DUIDDispatchable Unit Identifier and service so it doesn’t suffer from the same data volume problems.

Is it possible to add an additional column which is the last period the record applies to?

Participant suggestion: We can add that column to the database so the loader can automatically populate and verify by the start and end dates that we have 288 records for a bid. AEMOAustralian Energy Market Operator think this suggestion has merit and have implemented it.

If I opt-in for the Sparse Model, will AEMO provide a view, allowing us to continue to see a fully qualified offer?

AEMOAustralian Energy Market Operator are conscious of the significant change for participants and are considering options.

Is it possible to have a _SPARSE file?

AEMOAustralian Energy Market Operator are conscious of the significant change for participants and have implemented a _SPARSE file.

Where do I find the NEXT_DAY_* SPARSE files?

Data SubscriptionA web application in the Markets Portal used to subscribe to Data Interchange files and to obtain the latest MMS Data Model. These files are typically csv files containing public data and participant’s private data. AEMO recommends participant use of the Data Interchange suite of products, supplied by AEMO, to automate receiving and storage of the files to their systems. AEMO provides working sample software for FTP access (Participant Batcher) to MarketNet and working sample software for Data Interchange. > Add subscriptions:

Is AEMO going to provide a script to back populate the Sparse Data Model with historical bid offer period data?

No, but there are two approaches AEMOAustralian Energy Market Operator can consider:

  • Set the PERIODITTO field equal to the PERIODID field for all the historical data ensuring participants have a consistent representation, although this is inefficient storage.

  • A more comprehensive approach is collapsing the historic data into a sparse model to realise the storage benefits.

Please provide Feedback.

Is AEMO planning to provide Sparse versions of bidding tables?

Currently, we have no formal plans, but if participants are interested and see the value, we'd appreciate Feedback so we can consider it as a future service.

Since BIDS report - sparse is not available yet, can participants choose one of two bid report versions to avoid mixed format data in the table?

AEMOAustralian Energy Market Operator can gauge participant interest and see if it there is value as a future change.