7 Configurations Examples

7.1 Oauth2 configuration

The AEMO Australian Energy Market Operator gateway provides the ability to configure Oauth2 token endpoints for use as an authentication mechanism for “data sources”. It provides services such as caching for tokens, including those defined with a limited scope. AEMO recommends that the client ID and client secret are maintained in a secrets management solution and referenced in the GW configuration.

oauth_config_list=aemoidam
oauth_token_eviction_cycle_secs=60 oauth_aemoidam_client_id=xxxx
oauth_aemoidam_client_secret=xxxx oauth_aemoidam_request_url=https://apis.nem.sit.aemo.com.au/oauth/token oauth_aemoidam_provider=DEFAULT
oauth_aemoidam_media_type=application/x-www-form-urlencoded oauth_aemoidam_refresh_before_expiry_secs=60

7.2 JSON schema configuration

The AEMO gateway provides the ability to configure JSON JavaScript Object Notification schemas which are used to validate IDX Industry Data Exchange payloads. The JSON schemas are provided by AEMO. Multiple configurations are supported. The configuration names need to be structured to ensure that the business function and schema version can be identified.

json_schema_configurations=idx json_schema_idx_base_cache=${batcher_install_dir}/schemas/IDX json_schema_idx_pqd_1_schema=pqd-schema-v1-0-0.json

7.3 Data source configuration

The base AEMO GW installation will be defined with the following data sources:

  • aemoapi – interacts with the AEMO IDX platform API channel

  • aemolfs – interacts with the AEMO IDX platform large file share (sFTP) channel

  • Local – the enterprise integration point Note the references to the Oauth configuration and JSON schema configuration

batcher_aemoapi_mode=IDX
batcher_aemoapi_url=https://apis.nem.sit.aemo.com.au batcher_aemoapi_auth_provider=OAUTH2
batcher_aemoapi_oauth_config=aemoidam batcher_aemoapi_api_business_response_codes=200,201 batcher_aemoapi_default_ssl_context=false batcher_aemoapi_flow_control_url=flowcontrol/v1/participants batcher_aemoapi_flow_control_header_values=x initiatingParticipantId=${participant_identifier} batcher_aemoapi_flow_control_refresh_secs=30 batcher_aemoapi_bf_fire_and_forget=pqd
batcher_aemoapi_json_schema_config=idx

7.4 Thread source configuration

To describe thread concepts, we take a simplified example of an API Application Programming Interface; a set of clearly defined methods of communication between various software components. send thread, excluding WebSocket eventing channel, and break the thread configuration into the following components:


General Mode is the message exchange pattern. The IDX_SEND supports both synchronous, asynchronous, and fire and forget patterns. We have configured this example thread for API polling at 30 second intervals:

batcher_thread_2_description=Generic API send thread batcher_thread_2_mode=IDX_SEND batcher_thread_2_timeout=30 batcher_thread_2_polling_interval=30 batcher_thread_2_activity_monitor=Y batcher_thread_2_schedule_mode=POLL

For this send thread, the source is [local] picking up files from the Inbox folder:

batcher_thread_2_source=local batcher_thread_2_source_dir=${data_staging_dir}/Inbox

7.5 Thread source - translation - configuration

For this send thread, the translator performs the following actions:

  1. Extracts data from the JSON message header and adds this data into eventing variables

  1. Derives a MessageContextId based on the IDX specification 3. Signs the payload with a digital signature

7.6 Thread source - destination - configuration

For this send thread, the destination is [aemoapi]. A number of eventing variables derived in the translation stage are used to deliver the payload to the destination:

batcher_thread_2_dest=aemoapi batcher_thread_2_dest_dir={lowercase:businessFunctionId}/v1/{businessFunctionResourceId}
batcher_thread_2_dest_api_header_values=x-market:{market}\ ,x-initiatingParticipantId:{initiatingParticipantId}\ ,x-messageContextId:{messageContextId}\ ,x-signature:{payloadSignature}

7.7 Azure Blob Storage configuration

Instructions: Replace {parameters} with appropriate values.

batcher_azure_account={account name}
batcher_azure_url=https://{account name}.blob.core.windows.net
batcher_azure_API_key={Azure API key - use this if accessing as "owner"}
batcher_azure_API_SAS_token={Azure SAS token - use this if accessing as permissioned user, not owner}
batcher_azure_timeout_connection=10
batcher_azure_timeout_transfer=120
batcher_azure_proxy_host={proxy host name if proxy in place, otherwise omit property}
batcher_azure_proxy_port={proxy host port if proxy in place, otherwise omit property}
batcher_azure_mode=azure
batcher_azure_dir={main container/folder}

7.8 AWS S3 configuration

Instructions: Replace {parameters} with appropriate values.

batcher_aws_region={AWS region code e.g. ap-southeast-2}
batcher_aws_API_access_key={AWS access key}
batcher_aws_API_secret_key={AWS secret key}
batcher_aws_timeout_connection=10
batcher_aws_timeout_transfer=120
batcher_aws_proxy_host={proxy host name if proxy in place, otherwise omit property}
batcher_aws_proxy_port={proxy host port if proxy in place, otherwise omit property}
batcher_aws_mode=aws
batcher_aws_dir={main container/folder}

7.9 Google Storage configuration

Instructions: Replace {parameters} with appropriate values. Use either the batcher_google_service_account_key_file or batcher_google_service_account_key_contents sourced from Secrets Manager.

batcher_google_url=https://storage.googleapis.com/
batcher_google_service_account_key_file={the full path to your service acocunt JSON key file}
batcher_google_service_account_key_contents={the text contents of your service account JSON key file}
batcher_google_oauth_token_expiry_mins=1440
batcher_google_timeout_connection=10
batcher_google_timeout_transfer=120
batcher_google_proxy_host={proxy host name if proxy in place, otherwise omit property}
batcher_google_proxy_port={proxy host port if proxy in place, otherwise omit property}
batcher_google_dir={main storage bucket}

7.10 Credentials-Referencing data vault variables

7.10.1 Docker properties

data_vault_configurations=docker
data_vault_docker_provider=PROPERTIES_FILE
data_vault_docker_reference=/run/secrets/AEMO.properties

7.10.2 Azure Key Vault

data_vault_configurations=azure
data_vault_azure_provider=AZURE_KEY_VAULT
data_vault_azure_reference={fully qualified URL to vault e.g. https://{vault_name}.vault.azure.net/}
data_vault_azure_auth_method=VMMI

7.10.3 AWS Secrets Manager

data_vault_configurations=aws
data_vault_aws_provider=AWS_SECRETS_MANAGER
data_vault_aws_reference={secret_name - as defined in the Secrets Manager, must be hosted in same region as which app runs}
data_vault_aws_auth_method=VMMI
data_vault_aws_service_context={role_name - under which app runs. Defined in Identity and Access Management roles with appropriate permissions}

7.10.4 GCP Secrets manager

data_vault_configurations=gcp
data_vault_gcp_provider=GOOGLE_SECRETS_MANAGER
data_vault_gcp_reference={fully qualified URL to the secret manager e.g. https://secretmanager.googleapis.com/v1/projects/{your_context}/secrets/}
data_vault_gcp_auth_method=VMMI