Definitions¶
This document describes general concepts about the payload format which didn’t fit in the asyncapi.yaml itself.
API Definition as a Contract¶
The IE Common Databus Payload Async API spec is a contract which allows many variations. As long as a connector is fulfilling the contract, it is considered to be compliant, even when it changes its behavior from one version to another one.
For example, numeric values are no longer quoted in the SIMATIC S7 Connector V1.2. The API spec allows both, the quoted and the unquoted variant, but the quoted one is deprecated. This is considered a non-breaking change because both variations are allowed and all clients should be prepared to handle both variants.
This means clients need to:
- Implement against the API (and not against example messages)
- Don’t make any assumption about the content of the “id” field (don’t assume you get the tag name there)
- Be prepared that numbers might be quoted as strings or might be actual numbers (the quoted variant is deprecated, and all connectors shall switch to the non-quoted numbers)
- Expect that the OPC UA Connector uses real arrays instead of using pipe-separated values in strings
- 64-bit integer values are special because they cannot be exactly represented by the JavaScript 'Number' data type and many libraries for other languages face the same problem (that's why we use strings to represent such numbers)
- Test against example messages
Implementing Subsets¶
Most connectors only support the “bulk” publish type. Some connectors, for example the PROFINET IO Connector, only support timeseries
or binarytimeseries
.
Not all connectors support all data types. That means, only a subset of the possible features is implemented by a specific connector.
Removing a feature (e.g. removing the support of the bulk
publish type and only supporting timeseries
format instead) is also considered a non-allowed breaking change, even when both versions comply with the API spec.
Each connector needs to document which subsets are implemented and which API version is implemented. This is important for clients if they did not implement all the features, for example the timeseries
publish type.
Compatibility Rules¶
When the minor version of the payload schema changes, only optional fields (example: qx) are added to an object and only new alternative objects are added.
Many fields are not formally required, because they have been added after V1.1.0, but newer versions of the connectors shall support them (e.g. pubTopic, hashVersion, qx).
Some parts of the API spec are marked as deprecated (e.g. encoding numbers as strings). New connectors shall not use any deprecated fields and existing ones shall consider to remove them in newer releases.
The “v1” as major version in the topic name means we could add a new version “v2” in parallel but the effort of doing this is too high, so we do everything to avoid that. The performance of notifying all values twice would also be very bad.
Meaning of the API Version¶
Semantic versioning is being used, that means:
- In case of a breaking change, the major version will be changed
- Changes in the minor version are optional additions (new fields in existing messages or new message types)
The version of the Common Payload API is in principle independent of the IE Platform and IE App versions. That means, when there are no changes for a new Industrial Edge release, there won’t be a new API version. This also means that a connector might still implement according to an older version of the API.
Metadata Message Handling¶
Metadata messages need to set the retain=true
flag for the MQTT message. This is needed to make the metadata available to clients which connect at a later time.
When to publish this message?
- When the connector starts
- When the MetaData has changed
- When the connector has lost the connection to the Databus and reconnects. This is needed because the Databus might has been restarted and lost the value.
Refer also to the
mdHashVer
andhashVersion
fields of the asyncapi definition for more information.
Value Message Handling¶
Value messages should set the retain=false
flag for the MQTT message. This improves the performance and avoids that clients work with outdated values.
There might be some specific connectors for which the retain=true
flag is reasonable but they should be aware that the retain values are not preserved when the Databus restarts and needs to be refreshed from time to time.
QoS¶
In case of the local MQTT message bus, the QoS setting shall be always "0" because TCP guarantees that no messages are lost between the publisher and the broker. The additional guarantees of higher QoS values are not existent when no persistent buffering is implemented (which we don’t have with Mosquitto).
QoS=1
and QoS=2
add a lot of additional overhead and reduce the throughput. In high load scenarios, instead of guaranteeing message delivery, this instead leads to an earlier collapse of the system because of the higher load and is therefore counterproductive.
In case a proxy is between the publisher and the Databus or when they are running on a different machine and the network or machine might fail, the situation might become different.
Connector Specific Messages¶
Some connectors (e.g. the PROFINET IO Connector) implement additional specific messages which are not part of the common API spec. Those are defined in an extra asyncapi.yaml (e.g. asyncapi_profinet_io_conn.yaml
).
Some connectors are implementing features which would be considered common but are not part of the API definition (e.g. connection status of the SIMATIC S7 Connector). Those are considered experimental and can change in the future once they have been standardized by the common API.
Default Values¶
Values which are identical to the default values can be omitted, the client just uses the default value.
Unexpected Behaviour¶
Publish Mode bulk
¶
In some cases (e.g. when the CPU load is high), the 'bulk' format might contain the same datapoint multiple times from a different point in time.
Case Sensitive / Case Insensitive¶
Keys of JSON messages are always case sensitive. When it is not explicitly described, all other values are case sensitive. This applies especially for datapoints and data type names as well as for enum values.
Quality Information¶
There can be 2 types of quality information for each datapoint value:
- qc: only 4 major states
- qx: available from API V1.2 (currently only implemented in the SIMATIC S7 Connector V1.2).
qx contains additional information compared with qc, for example sub-states for the qualities and some single bit flags.
qc¶
This is a small subset of qx (actually the bits 7+6 of qx).
qc can have 1 of the following numeric values:
- 0 : BAD - The dp value is not useful
- 1 : UNCERTAIN - The quality of the dp value is less than normal but the value may still be useful
- 2 : GOOD (non-cascade) - The quality of the dp value is good
- 3 : GOOD (cascade) - The quality of the dp value is good and may be used in control
qx¶
qx is an extension of qc which is 15 bits long instead of 2 bits.
In the following, the overview of the qx structure:
- High Byte: Industrial Edge Specific Information
- Bit 15: Reserved
- Bit 14...12: Flags
- Bit 11...8: Extended sub-status
- Low Byte: Quality code according to PROFIBUS PA / OPC DA
- Bit 7...6: Quality
- Bit 5...2: Sub-status
- Bit 1...0: Limits
Evaluation of qc/qx in a consumer App¶
qx will be only added to a message, if it contains additional information to qc. That means, when any of the bits 14 to 8 or 5 to 0 is non-zero. As a consumer, if you would like to use the additional information of qx, you should do the following in your code to simplify the evaluation and to take care if the qx field is not there but just the qc field as fallback:
uint16_t qx;
if(val.hasField("qx") {
qx = val.getFieldValue("qx");
} else {
// qx doesn’t exist, use qc as fallback and convert it to same structure as qx
qx = val.getFieldValue("qc") << 6;
}
// work just with value of qx
if(qx & 0x80) { // qx is either 'Good' or 'Good (cascade)'
...
}
Quality Code¶
The tables below reflect the quality, sub-status codes, and limit description of the quality code which is encoded in the low-byte of Industrial Edge quality attributes.
Quality¶
Value of bits 7...6 (decimal) | Quality | Description |
---|---|---|
0 | BAD | The value is not useful for reasons indicated by the sub-status. |
1 | UNCERTAIN | The quality of the value is less than normal but the value may still be useful. The reason is indicated by the sub-status. |
2 | GOOD (non-cascade) | The quality of the value is good. |
3 | GOOD (cascade) | The quality of the value is good and may be used in control. |
Only sub-status codes are listed that are assigned by the IE Connector. Sub-status codes assigned by the PLC and/or periphery only are not listed but reserved to provide peripheral status codes.
Sub-status for Quality "BAD"¶
Value of bits 5...2 (decimal) | "BAD" sub-status | Description |
---|---|---|
0 | Non-specific | There is no specific reason why the value is BAD. |
1 | Configuration error | The value is not useful because of some inconsistency regarding the configuration. |
2 | Not connected | The value is not reliable because the connection to the provider has been disconnected at consumer-side (e.g. a communication driver actively disconnects from a PLC on user request or by design). |
4 | Sensor failure | The value is not useful because it cannot be converted, i.e. a value from the device (PLC) cannot be converted to the corresponding HMI tag. |
5 | No communication, with last usable value | The value is not useful because the communication to the data source failed, however a last known value is available. |
6 | No communication, no usable value | The value is not useful because the communication to the data source failed or has never been established since it was last out of service and a last known value is not available. |
7 | Out of service | The value is not reliable because the provider side has been disabled or shut-down (e.g. a PLC is in stop mode, or a tag is disabled for maintenance purposes). |
For further "BAD" sub-status codes assigned by PLC/periphery only, refer to PROFIBUS-PA.
Sub-status for Quality "UNCERTAIN"¶
Value of bits 5...2 (decimal) | "UNCERTAIN" sub-status | Description |
---|---|---|
0 | Non-specific | There is no specific reason why the value is UNCERTAIN. |
1 | Last usable value | The connection to the data source is still established, however the data source stopped updating the value for some reason. |
2 | Substitute value | A predefined value is used in case of an invalid value due to communication issues with the data source or a range violation. The reason for providing substitute values is configurable. |
3 | Initial value | A predefined value intended for the startup of the HMI system (or a sub-ordinate device) is used while not being able to provide values from the data source. |
5 | Range violation | The value lies outside the range defined by the min. and max. value. The limits define which direction (min. or max.) has been exceeded. |
6 | Sub-normal | A value derived from multiple values has less than the required number of good sources. This includes data aggregation by means of data compression algorithms. |
For further “UNCERTAIN” sub-status codes assigned by PLC/periphery only, refer to PROFIBUS-PA.
Sub-status for Quality "GOOD (cascade)"¶
Value of bits 5...2 (decimal) | "GOOD (cascade)" sub-status | Description |
---|---|---|
0 | Non-specific | No error or special condition is associated with this value. |
6 | Local override | The value has been overridden by the user or some logic in to continue operation. Typically, the input has been disconnected and a manually entered value has been 'forced', or a value has been corrected. |
For further "GOOD" (non-cascade and cascade) sub-status codes assigned by PLC/periphery only, refer to PROFIBUS-PA.
Limits¶
Value of bits 1...0 (decimal) | "Limit" | Description |
---|---|---|
0 | Ok | Data quality unrelated to limits. |
1 | Low limit violation | The value has exceeded its low limit. |
2 | High limit violation | The value has exceeded its high limit. |
3 | Constant | The value cannot move, no matter what the process is doing. |
Industrial Edge specific Quality Information¶
The tables below reflect the extended sub-status and flag description of the quality code which is Industrial Edge specific and encoded in the high-byte of Industrial Edge quality attributes. The intention of Industrial Edge specific extended sub-status is to describe the data quality more distinct than the sub-status according to PROFIBUS-PA. Nevertheless, to support 3rd party collaboration based on standards only, a best-effort sub-status is always set, even if a more distinct extended sub-status is present.
Extended sub-status "BAD"¶
Value of bits 11...8 (decimal) | Extended sub-status "BAD" | Description |
---|---|---|
0 | Non-specific | No Industrial Edge specific extended bad sub-status is associated with this value. |
1 | Aggregated value | The value has been calculated out of multiple values with less than the required number of good sources. This includes data aggregation by means of data compression algorithms. The corresponding sub-status is set to ‘non-specific’. |
3 | Unusable value | A (logged) value has been identified to be incorrect but a respective correction value is not available. The corresponding sub-status is set to ‘non-specific’. |
7 | Disabled | The provider of the value (e.g. logging tag for logged value) has been disabled and the previous value was BAD. The corresponding sub-status is taken from the last (previous) sub-status. |
Extended sub-status "UNCERTAIN"¶
Value of bits 11...8 (decimal) | Extended sub-status "UNCERTAIN" | Description |
---|---|---|
0 | Non-specific | No Industrial Edge specific extended uncertain sub-status is associated with this value. |
1 | Aggregated value | The value has been calculated out of multiple values with less than the required number of good sources to be GOOD as well as less than required number of bad sources to be BAD. This includes data aggregation by means of data compression algorithms. The corresponding sub-status is set to ‘sub-normal’. |
7 | Disabled | The provider of the value (e.g. logging tag for logged value) has been disabled and the previous value was GOOD or UNCERTAIN. In case of GOOD, the corresponding sub- status is set to ‘last usable value’. In case of UNCERTAIN, the corresponding sub-status is taken from the last (previous) sub-status. |
Extended sub-status "GOOD (cascade)"¶
Value of bits 11...8 (decimal) | Extended sub-status "GOOD (cascade)" | Description |
---|---|---|
0 | Non-specific | No Industrial Edge specific extended good sub-status is associated with this value. |
1 | Aggregated value | The value has been calculated out of multiple (good) values. This includes data aggregation by means of data compression algorithms. The corresponding sub-status is set to ‘non-specific’. |
2 | Manual input | A (logged) value has been created manually. The corresponding sub-status is set to ‘non-specific’. |
3 | Corrected value | A (logged) value has been corrected. The corresponding sub-status is set to ‘non-specific’. |
6 | Initial value | The local data source has been initialized with the configured initial value. The corresponding sub-status is set to ‘non-specific’. |
Extended sub-status "GOOD (non-cascade)"¶
Please note that Industrial Edge doesn’t define extended sub-status for good (non-cascade).
Flags¶
Bit Number | Extended sub-status "GOOD (cascade)" | Description |
---|---|---|
Bit 12 | Source quality | The data quality has been determined and assigned by an external data source. |
Bit 13 | Source time | The data timestamp has been produced and assigned by an external data source. |
Bit 14 | Time corrected | The data timestamp applied by an external data source has been corrected by the system. Thus, Bit 13 (“Source time”) is not set. Time correction happens, if the external timestamp is older than the timestamp of the last known value. |
Bit 15 | Reserved | Ignore when reading, set to 0 when creating, preserve when copying. |
Overview of Message Formats¶
The following tables should give an overview about the different message types.
Defined in Common Databus Payload API (asyncapi.yaml)¶
These definitions are common for all connectors.
Data Point Payload Format for read/subscribe¶
publishType | Message Type | Message Example in principle | Used by connector |
---|---|---|---|
single | Data Point Payload (deprecated since V1.2, no longer supported for new installations) | { seq, val, ts, qc } | SIMATIC S7 V1.1, Modbus TCP V1.0, Ethernet IP V1.0 |
bulk | Data Points Payload (subDpValueSimaticV1Payload) | { seq, (ts), vals[ { id, val, ts, qc, (qx) } ] } | SIMATIC S7 V1.1/V1.2, PROFINET IO V1.0.2 (no oversampling), Modbus TCP V1.0, Ethernet IP V1.0 |
timeseries | Data Points Time Series Payload (subDpValueSimaticV11TimeSeriesPayload) | { seq, records[ { ts, rseq, vals[ { id, val, qc } ] } ] } | PROFINET IO V1.0.2 (with oversampling) |
binarytimeseries | Data Points Binary Payload (subDpValueSimaticBinaryMsgV1). Binary raw format with almost same content as “timeseries” format. | { seq, records[ { ts, rseq, vals[ { id, val, qc } ] } ] } Currently only defined in the user documentation of the PROFINET IO Connector, not yet in asyncapi. | PROFINET IO V1.0.2 (with oversampling) |
Data Point Payload Format for write/publish¶
publishType | Message Type | Message Example in principle | Used by connector |
---|---|---|---|
single | Data Point Payload (deprecated since V1.2, no longer supported for new installations) | { seq, vals{ id, val, ts, qc } } | SIMATIC S7 V1.1, Modbus TCP V1.0, Ethernet IP V1.0 |
bulk | Data Points Payload (pubDpValueSimaticV1Payload) | { seq, vals[ { id, val, ts, qc } ] } | SIMATIC S7 V1.1/V1.2, PROFINET IO 1.1, Modbus TCP V1.0, Ethernet IP V1.0 |
PROFINET IO specific Payload Formats¶
These messages are part of the Common Payload API but are defined in the separate file asyncapi_profinet_io_conn.yaml
. They are specific for PROFINET and are only supported by the PROFINET IO Connector. Even in future, probably no other connector will implement them.
Message Type | Message Example in principle | Used by connector |
---|---|---|
Read PROFINET Records request | { (seq), reqs[ { laddr, index} ] } | PROFINET IO 1.1 |
Read PROFINET Records response | { (seq), rsps[ { (laddr), (index), (val), status, (errorDescription), ts} ] } | PROFINET IO 1.1 |
Write PROFINET Records request | { (seq), reqs[ { laddr, index, val} ] } | PROFINET IO 1.1 |
Write PROFINET Records response | { (seq), rsps[ { (laddr), (index), status, (errorDescription), ts} ] } | PROFINET IO 1.1 |
Private Messages not defined in Common Databus Payload API¶
These messages are local extensions of the connectors and are not part of the Common Databus API definition. That means, they have not been standardized in the Common Databus Payload API. The main reason is that the maturity level is not high enough to ensure that the definition will not change. In future versions of the API, they might be included but there may be some changes in the content or even the topic name. If a connector implements such private extensions, it probably must change the implementation when the message gets part of the Common Payload API with a changed content and also must define a transition strategy for clients which use the private extension. Client Apps should be aware that those private extensions have a higher chance of being changed than those of the Common Payload API messages.
Other Payload Formats for read/subscribe¶
Message Type | Message Example in principle | Used by connector |
---|---|---|
Connection and Disconnection Status (not yet part of common payload) | { id, msg } | SIMATIC S7 V1.1/V1.2 |
Alarms | { seq, evs[ { area, clsName, evTxt[ ], evTxtExt[ ], id, … } ] } | SIMATIC S7 V1.1/V1.2 |
Lifetime of the IDs¶
When the metadata is updated, the IDs can change. The metadata is only published on change (with retain=true
). That means, when you subscribe to it and receive an update, you need to update your internal name to ID map.
There is a field called “mdHashVer” which is currently only implemented by the PROFINET IO Connector. It should be also implemented by other connectors in future versions. The "mdHashVer" field of the "dpValue" payload must match the "hashVersion" number in the metadata definition.
This is a synchronized way to know exactly which metadata version was used for which data message because the update of the metadata and data of MQTT topics is asynchronously and the notification order between Metadata message and Value message might be reversed (see also the sequence diagram below).
Sequence Diagrams¶
App Startup and Configuration Update¶
sequenceDiagram
participant EdgeRuntime
participant Connector
participant Databus
participant App
EdgeRuntime ->> Databus : Start
EdgeRuntime ->> Connector : Start
Connector ->> Connector : Read configuration (hashVer=0x4567)
Connector ->> Databus : Connect and Login
Connector ->> Databus : Publish Metadata (retain=true, hashVer=0x4567)
Connector ->> Databus : Publish Values(retain=false, seq=1, hashVer=0x4567)
EdgeRuntime ->> App : Start
App ->> Databus : Subscribe for Metadata and Values
Databus ->> App : Notify Metadata (retain=true, hashVer=0x4567)
App ->> App: Build Name <=> Id Map
Connector ->> Databus : Publish values(retain=false, seq=2, hashVer=0x4567)
Databus ->> App : Notify Values(retain=false, seq=2, hashVer=0x4567)
Connector ->> Databus : Publish values(retain=false, seq=3, hashVer=0x4567)
Databus ->> App : Notify Values(retain=false, seq=3, hashVer=0x4567)
Connector ->> Connector : Detected Configuration Change
Connector ->> Connector : Restart
Connector ->> Connector : Read changed Configuration (hashVer=0x7825)
alt normal case: Metadata notification arrives before first Value notification
Connector ->> Databus : Publish Metadata (retain=true, hashVer=0x7825) (new hash)
Databus ->> App : Notify Metadata (retain=true, hashVer=0x7825)
App ->> App: Build Name <=> Id Map
Connector ->> Databus : Publish Values(retain=false, seq=1, hashVer=07825) (seq restarts)
Databus ->> App : Notify Values(retain=false, seq=1, hashVer=0x7825)
else inversion of message notification order: Value before new Metadata
Connector ->> Databus : Publish Metadata (retain=true, hashVer=0x7825) (new hash)
Connector ->> Databus : Publish Values(retain=false, seq=1, hashVer=07825) (seq restarts)
Databus ->> App : Notify Values(retain=false, seq=1, hashVer=0x7825) (requires updated metadata: ignore or put on hold)
Databus ->> App : Notify Metadata (retain=true, hashVer=0x7825)
App ->> App: Build Name <=> Id Map
end
Conformance Testing / Validation¶
Not-Conforming Connectors¶
OPC UA Connector¶
The OPC UA Connector uses arrays packed into a string where the values are separated by pipe characters, e.g. “10 | 21 | 33”. This fails the validation because this is not an integer value (not even a quoted one).
Ethernet IP and Modbus Connector¶
This connector uses boolean values (“TRUE” and “FALSE”) instead of 1 and 0.
WinCC Unified MQTT Adapter¶
The WinCC Unified MQTT Adapter uses IDs longer than the allowed 8 characters (e.g. “1.0.1000.2.0.0”).
Using the MQTT Message Payload Validator¶
The validator script currently takes the following input files:
- The asyncapi.yaml specification file
- MQTT recorded file of metadata (is needed to validate IDs and data types)
- MQTT recorded value data
It responds if each input message is according to the specification. It can optionally create 1 pretty printed JSON file per input message (when the output path is set). Unfortunately, at the moment this only works when the validation was successful.