Ecosystem Testing Phase¶

Linking Info

If you want to permanently link to one of the test cases, please use the this schema, since the title of the test case might change in the future.

https://docs.eu1.edge.siemens.cloud/develop_an_application/app_onboarding/ecosystem_review/testing-phase.html#ecosystemreview.test.id. ¶

The testing phase is separated into four different categories:

Functional Qualities
Non Functional Qualities
Ecosystem Control Point
Ecosystem Scalability

Functional Qualities¶

Functional qualities, also known as functional requirements, describe the specific features, capabilities, and functions that the system or software must perform to satisfy the user's needs and requirements. They define the "what" of the system, focusing on its behavior and functionality from the user's perspective. Examples of functional qualities include user authentication, data validation, report generation, search functionality, and transaction processing. Functional qualities are typically specified in terms of specific features, use cases, and user interactions that the system should support.

Application runs on at least 2 Industrial Edge device versions¶

Test Title	App runs on at least 2 Industrial Edge device versions
Test ID	100
Test Category	Release blocking check
Description	App runs on the latest (n) and previous (n-1) Industrial Edge Device firmware version of at least one supported Industrial Edge Device. The hardware requirements documentation provided by the app developer specifies the device type. If the previous (n-1) Device firmware version does not provide all of the required functionality for the app to run without restrictions, the app must run on the latest (n) version of the Industrial Edge device.this must be specified in the hardware requirements documentation.
Failed condition	The app does not work as expected on one of these Industrial Edge device versions.

Procedure

Steps	Action	Expected Result
1.	Install the app on a supported device with the latest Industrial Edge device version.	The app runs with at least a minimal use case.
2.	Install the app on a supported device with a previous (n-1) Industrial Edge device version.	The app runs with at least a minimal use case.

Application can be updated from the latest released application version¶

Test Title	Application can be updated from the latest version provided
Test ID	101
Test Category	Release blocking check
Description	If the app does not have a new MAJOR version number, which implies a breaking change of the app, it must continue to run after an upgrade from a previous released version without loss of data or user input.
Warning Condition	During the update, some data is lost.
Failed condition	After the update, the app fails to run or all of its configuration/data is lost.

Procedure

Steps	Action	Expected Result
1.	Install the (n - 1) version of the app on your device.
2.	Set up a minimal use case,for example via configs/data.
3.	Update the app to the latest (currently tested) version via the Industrial Edge Management.
4.	Verify that the app is still running with the previous configuration and that there has been no loss of data.	All stored data is available and the app is still running. If necessary, stored data will be converted to a new format.

Restart Policy - Disposability¶

Test Title	Restart Policy - Disposability
Test ID	103
Test Category	Release blocking check
Description	To ensure a scalable and resilient ecosystem, we've implemented the 12-factor methodology, which includes robust processes against failures and a policy to review exceptions where developers justify the need for specific app behaviors. There are containers that are not restricted to these rules, if they are defined init-container (see this specification)
Failed condition	Any services, not specified as init-container, have a restart policy set to "NO". This means that the app will not restart after a device reboot or after a failure, and therefore does not comply with the "12-Factor" strategy.

Procedure

Steps	Action	Expected Result
1.	Open the app's docker-compose.yml.
2.	Check the restart policy for each container.	The restart policy of each container should be "unless stopped", "on failure" or "always".
3.	Install the app on your device with a minimal setup. Reboot the device.	After rebooting the device, the app should start automatically. The app should restart without any loss of data.

Non functional Qualities¶

Non-functional qualities, also known as non-functional requirements or quality attributes, describe the overall characteristics, properties, and behaviors of the system or software that affect its performance, reliability, usability, security, and other aspects. They define the "how" of the system, focusing on its qualities and attributes rather than its specific features or functions. Examples of non-functional qualities include performance, reliability, usability, security, scalability, maintainability, portability, and interoperability. Non-functional qualities are typically specified in terms of constraints, thresholds, and metrics that the system must meet to be considered acceptable from a quality perspective.

No Config provided¶

Test Title	No Config provided
Test ID	104
Test Category	Release blocking check
Description	The app should be able to run without a configuration. Therefore, the app is not allowed to die in this scenario, but instead should notify the user to provide a configuration. This must be done via the app's Web UI. If the app does not provide a Web UI, the application must provide this information instead through its logs instead.
Warning Condition	The app requests a configuration but does not provide a config template.
Failed condition	- The app's Web UI does not notify the customer that a configuration file must be provided. - If the app does not provide a Web UI and the logs do not include information about the missing configuration.

Procedure

Steps	Action	Expected Result
1.	Install the app on a device without any configuration chosen.	The app is running and not shutting down.
2.	Click on the app icon and check the application's web UI. If the app does not provide a UI, check the logs. You should be informed here that providing a configuration is necessary.

In place configuration¶

Test Title	In place configuration
Test ID	105
Test Category	Release blocking check
Description	Administration and configuration is not done by an extra process or tool, it should be part of the normal IE environment. All tools, for example configurator, are released and bundled with the app. An exception applies to Engineering tools.
Warning Condition	The app can be configured by configuration files, for which it is necessary to start additional software (not hosted on IE platform).
Failed condition	To change the configuration of the app, you must start additional software that is not hosted on the IE platform; the configuration cannot be changed via the IEM directly.

Procedure

Steps	Action	Expected Result
1.	Deploy the app on a device.
2.	Set up a minimal use case.	You do not need to leave the IE environment to configure the minimal use case..

Logging¶

Test Title	Logging
Test ID	106
Test Category	Release blocking check
Description	The app needs to align with the logging mechanism that is defined in the Ecosystem Framework.
Warning Condition	The logs provided by the app are incorrectly formatted, have minimal readability, or do not provide important information.
Failed condition	No logging of: - start - stop - restart

Procedure

Steps	Action	Expected Result
1.	Install the app on a device without a configuration chosen.
2.	Restart the app on the device.
3.	Download the log files of the app via the Device UI.	By reading the logs, you can recap what happened.

Accessible Web Interface¶

Test Title	Accessible Web Interface
Test ID	107
Test Category	Release blocking check
Description	If an app provides a web interface through one of its containers, the Redirect URL must point to this service. This ensures that when the user clicks the app icon on the device, they are redirected to the web interface.
Failed condition	The app has a web UI service, but the Redirect URL either points to the wrong container or is not set.

Procedure

Steps	Action	Expected Result
1.	Install the app on a device.
2.	Navigate to the "Apps" view on the device.
3.	Click the icon of the app.	You will get redirected to the app's web UI.

Available via IEM Remote Access¶

Test Title	Available via IEM Remote Access
Test ID	108
Test Category	Release blocking check
Description	Our IEM has a feature called "Remote Access" that allows you to access the IED through your connection to the IEM. It works like a VPN tunnel through the IEM and the device establishes the connection to the IEM (southbound to northbound). Many customers only access their device using this functionality, so to give the customer the full experience, if an app provides a web interface, the app must be able to support this.
Failed condition	The app UI can not be accessed via the "Remote Access" of the IEM.

Procedure

Steps	Action	Expected Result
1.	If your IEM does not has a "Relay Server" configured, create it for your IEM in the "Admin Management".
2.	Enable the remote access for the Edge device\n	The Remote Access can be enabled.
3.	Install the app to the device where the remote access has been enabled.
4.	Left click on the device in the "Edge Devices" view of the IEM and click on "Connect".	You will get redirected to the device UI
5.	Log on to the device and navigate to the "Apps" view on the device.
6.	Click the icon of the app.	You will get redirected to the app's web UI.

Backup & Restore¶

Test Title	Backup & Restore
Test ID	109
Test Category	Release blocking check
Description	The app can be restored using the Backup & Restore feature of Industrial Edge. Note: When using the backup and restore functionality in the IEM, all apps on the device are stopped. For more information regarding the backup of volumes, see also: How to backup/restore a Device
Failed condition	The app does not start after a restore, with persistent configurations or user data.

Procedure

Steps	Action	Expected Result
1.	Install the app on a device.
2.	Go to IEM > Edge Devices.
3.	Execute a backup of the device via the system command button. Include the backup of app volumes via checkbox.
4.	Uninstall the app from the device.	The app is removed from the device.
5.	Execute a restore of the device in the "Backups" tab in your IEM.	The app is deployed to the device and starts up with the previous configuration and data storage without any failures.

Comfortable Mass Deployment¶

Test Title	Comfortable Mass Deployment
Test ID	110
Test Category	Release blocking check
Description	To scale the deployment of the app, we also evaluate the user experience for mass installation. We aim to deploy the app on multiple devices (at least 5) within 5 clicks, ensuring it runs on each device without failure. Depending on the App classification, the requirements slightly vary. Data Integration and Configurator apps must be mass deployed and fully configured within 5 clicks. Connectors, Suppliers, Consumable, and Monolithic apps must simply be mass deployable and running within 5 clicks.
Warning Condition	The app is able to be mass deployed, but the installation takes more than 5 clicks.
Failed condition	No mass deployment is possible as the configuration to run the app, needs to be adjusted per device.

Procedure

Steps	Action	Expected Result
1.	Log in to an IEM with more than one onboarded device.
2.	Navigate to the "Catalog" and choose the app to install.
3.	Check, if the app installation with configurations can be executed with maximum 5 clicks.

Industrial Edge Publisher warnings/errors¶

Test Title	Industrial Edge Publisher warnings/errors
Test ID	111
Test Category	Not Release blocking check
Description	We also rely on the Industrial Edge Publisher to provide some additional checks.
Warning Condition	Industrial Edge Publisher displays at least one warning.
Failed condition	Industrial Edge Publisher displays at least on error.

Procedure

Steps	Action	Expected Result
1.	Prepare a environment with IECTL and a docker engine enabled.
2.	Initializes the workspace and connect the IECTL to the docker engine.
3.	Run the command: iectl publisher standalone-app validate --path $1 --skip "validate_docker_images,validate_signature" --verbose	The command should return no error.

Also see iectl publisher standalone-app validate

Ecosystem Control Point¶

An ecosystem control point typically refers to a specific juncture or component within a complex system or ecosystem where control or management actions can be applied to influence the behavior, performance, or outcome of the system as a whole. This concept is often used in the context of environmental management, ecological systems, or complex technological ecosystems. Overall, the concept of an ecosystem control point emphasizes the strategic identification and targeting of key leverage points within complex systems or ecosystems to achieve desired outcomes, whether it be environmental conservation, system optimization, or risk management.

Release Versioning¶

Test Title	Release Versioning
Test ID	112
Test Category	Release blocking check
Description	Only released versions are offered on the SDEX market, so we only accept release candidates for testing. Our system does not work with downgrading apps. Therefore, we need to ensure that the new version of the app deployed is higher than the last version deployed. Also, the app must be approved by our Publisher and Industrial Edge Hub according to the policy defined at https://semver.org/. So called Pre-Release version (e.g. 1.0.0-rc2) are not accepted for the Ecosystem Review or the onboarding to the marketplace.

Procedure

Steps	Action	Expected Result
1.	You can validate your version against this regular expression: https://regex101.com/r/WzajZg/1	Your version string should be valid.

Service Name¶

Test Title	Service Name
Test ID	113
Test Category	Release blocking check
Description	Containers can address each other by using their service name. This could cause some problems when addressing a specific container. The service name must be unique. We have defined a syntax to ensure this: "application name" "-" "service name"

PassedException conditionWarning conditionFailed condition

Here is an example that would pass the test.

docker-compose.yml

version: '2.4'
services:
  mytestapplication1-webui:
    image: 'hello-world:linux'
    ports:
      - '8080:80'
    mem_limit: 50mb
  mytestapplication1-database:
    image: 'mysql:9.0.0'
    ports:
      - '5000'
    mem_limit: 100mb

The exception condition is met if the app has defined a network and the network type is set to INTERNAL. This is an exception because the app will be on a network that cannot be accessed or addressed by other containers. Therefore the naming is irrelevant. ALL CONTAINERS MUST CONNECT TO THE INTERNAL NETWORK

docker-compose.yml

version: '2.4'
services:
  web-ui:
    image: 'hello-world:linux'
    ports:
      - '8080:80'
    mem_limit: 50mb
    networks:
      - mynetwork
  database:
    image: 'mysql:9.0.0'
    ports:
      - '5000'
    mem_limit: 100mb
    networks:
      - mynetwork
networks:
  mynetwork:
    internal: true

If the app service names do not match, but are unique enough that they are unlikely to exist in different apps, then the warning condition is met.

docker-compose.yml

version: '2.4'
services:
  myfirstweb-ui:
    image: 'hello-world:linux'
    ports:
      - '8080:80'
    mem_limit: 50mb
  myowndatabase:
    image: 'mysql:9.0.0'
    ports:
      - '5000'
    mem_limit: 100mb

The test fails if at least one app service has a default name, for example database, web-ui, proxy, nginx, mysql, frontend, backend.

docker-compose.yml

version: '2.4'
services:
  web-ui:
    image: 'hello-world:linux'
    ports:
      - '8080:80'
    mem_limit: 50mb
  database:
    image: 'mysql:9.0.0'
    ports:
      - '5000'
    mem_limit: 100mb

Procedure

Steps	Action	Expected Result
1.	Check the Service name of all the services described in the docker-compose.yml	These Services must have unique names

App specific URL subdirectory path¶

Test Title	App specific URL subdirectory path
Test ID	114
Test Category	Release blocking check
Description	App specific URL subdirectory path (aka. Revers Proxy name) are set by the app builder per service. These naming need to be unique upon one device, otherwise the app installation would fail. Best practice is to use your service name as the App specific URL subdirectory path

PassedException conditionWarning conditionFailed condition

Here is an example that would pass the test.

The Application name is: My very special application Service 1 Name: My-very-special-application-Web-UI"*

nginx.yml

{"myapplication-webui":[{"name":"My-very-special-application-Web-UI","protocol":"HTTPS","port":"80","headers":"","rewriteTarget":"/"}]}

The exception condition is met if the app provide supplies twice the exact same nginx configuration name, in the scope of preventing these apps to run on the same device at the same time.

If the app nginx configuration names do not match, but are unique enough that they are unlikely to exist in different applications, then the warning condition is met.

nginx.yml

{"myapplication-webui":[{"name":"My-Web-UI","protocol":"HTTPS","port":"80","headers":"","rewriteTarget":"/"}]}

The test fails if at least one nginx configuration has a default name, for example database, web-ui, proxy, nginx, mysql, frontend, backend.

nginx.yml

{"myapplication-webui":[{"name":"Web-UI","protocol":"HTTPS","port":"80","headers":"","rewriteTarget":"/"}]}

Procedure

Steps	Action	Expected Result
1.	Check the nginx configuration name of all the services described in the nginx.yaml
2.	These nginx configuration name must have be unique

Unique App Repository Name¶

Test Title	Unique App Repository Name
Test ID	115
Test Category	Release blocking check
Description	The repository name of your app must be unique upon our system, this is to ensure the uniqueness of an image. The Industrial Edge Hub does not check the uniqueness of the repository name by default, so we need to manually check that it is not already in use. To do this, we should find the application's repo name in the digests.json file and match it against our database. Once this is done, we need to add the repo name to the list.
Failed condition	The repository name is already in use, or has a name that could occur multiple times. The application developer should be notified of this problem so that the repo name can be changed.

Procedure

Steps	Action	Expected Result
1.	Download the .app file.
2.	Open the file with 7zip.
3.	Open the digests.json file.
4.	Locate the part of the file that describes the docker-compose.yml file.
5.	The name in front of "/docker-compose.yml" is the name of the repository.
6.	Check the database (Only possible by Ecosystem Team of Siemens AG). See if the name already exists.
7.	If it does not exist, enter the repository name as a new entry.

Unique ID for Service Registry¶

Test Title	Unique ID for Service Registry
Test ID	116
Test Category	Release blocking check
Description	Applications that wish to use the registry service must ensure that their appId is unique when making themselves public to the service. To guarantee this uniqueness, it is best practice to use the repository name of the application as the appId for the service registry.
Failed Condition	The appId is already in use or has a name that could occur multiple times.

Port usage / standard network ports¶

Test Title	Port usage / standard network ports
Test ID	117
Test Category	Release blocking check
Description	Port numbers are assigned in various ways, based on three ranges: System Ports (0-1023), User Ports (1024-49151), and the Dynamic and/or Private Ports (49152-65535); the different uses of these ranges are described in RFC6335. System Ports are prohibited to be used by applications as these are reserved for the Industrial Edge system and underlying OS-System.
Warning Condition	Standard protocol does not use its standard port.
Failed condition	The app is using a System Port (0-1023).

Procedure

Steps	Action	Expected Result
1.	Check docker-compose.yml for exposed ports.
2.	Compare it to the default ports list.

Monetization¶

Test Title	Monetization
Test ID	118
Test Category	Release blocking check
Description	Using payment mechanisms for software other than one offered by the Marketplace isn’t permitted. Any deviation to this must be aligned and approved with the Siemens Industrial Edge EOM Team via a contract.
Failed condition	It is possible through other channels than the Siemens offered SDEX-Marketplace, to buy feature/functions/packages for the application.

Versioning of Docker images¶

Test Title	Versioning of Docker images
Test ID	119
Test Category	Not Release blocking check
Description	The state of the art is to define an image version in docker-compose.yml files to ensure a reproducible combination of containers as an app version.
Warning Condition	The images are not addressed via Semantic Versioning 2.
Failed condition	At least one image is referenced by a non-unique term (e.g. latest) or no version is specified.

Procedure

Steps	Action	Expected Result
1.	Check the docker-compose.yml file.
2.	Verify that all images have their version references.	All parts of an app are referred to by their version number.

Ecosystem Scalability¶

Ecosystem scalability pertains to the ability of a digital ecosystem to grow, adapt, and handle increased demands without compromising its functionality or performance. It involves expanding the ecosystem while maintaining its efficiency, responsiveness, and value to customers. Scalability ensures that the ecosystem can accommodate more users, services, and interconnected products as it evolves.

Inter-app communication¶

Test Title	Inter-app communication
Test ID	121
Test Category	Release blocking check
Description	Industrial Edge devices communicate using network protocols. Therefore, all services must make data transparent to other applications over IP-based networks. Services are addressed by a combination of device service name/IP address and port. All services should be accessible through a URL.
Failed Condition	Other constructs such as sockets, files, etc. are used for inter-application communication.

Procedure

Steps	Action	Expected Result
1.	Install the application on a device.
2.	Set up the minimal use case of the application.
3.	Install a network diagnostics tool.
4.	Monitor traffic of the application.
5.	Check the reachability of its subcomponents, for example via telnet.	All the services communicate via ports and are bound to them.

Self-contained¶

Test Title	Self-contained
Test ID	122
Test Category	Release blocking check
Description	The app provides all necessary components for their container and is not depended on other app within it's App Classification.
Warning Condition	It is designed to work with a subset of functions that could be expanded by combining it with other apps.
Failed condition	Without the installation of additional apps within the same App Classification, the main functionality of the app will not work.

Procedure

Steps	Action	Expected Result
1.	Install all apps necessary for fulfilling the End-to-End use case.	There is no other app deployed, that fits the same App Classification as the one that is in scope right now.

Adapted device requirement of the use case¶

Test Title	Adapted device requirement of the use case
Test ID	124
Test Category	Release blocking check
Description	The recommended device must suit the use case description based on the app's resource allocation. The device you recommend must have at least 25% of its resources unallocated when the entire use case is installed on the device.
Failed condition	The entire end-to-end use case takes up more than 75% of the available resources.

Integration in the Data-Domains¶

Test Title	Integration in the Data-Domains
Test ID	125
Test Category	Release blocking check
Description	An app must be part of at least one data domain as specified by its classification. All domain rules must be adhered to.
Failed Condition	The app is part of an incorrect data domain or violates one or more domain rules.

Payload Formats¶

Test Title	Payload Formats
Test ID	126
Test Category	Release blocking check
Description	For every supplier app, a payload has been defined. If no payload has been defined yet, the App Provider has to suggest and align a payload with the Industrial Edge Ecosystem Team. All apps have to use this payload format.
Failed condition	The app is communicating with a supplier app, without using the defined payload formats.

Application size¶

Test Title	Application size
Test ID	129
Test Category	Not Release blocking check
Description	The larger the .app file, the longer the user has to wait for the app to download to the IEM/IED. App providers should aim to reduce the size of their images as much as possible to improve the user experience.
Warning Condition	App-file is bigger than 1 GB.
Failed condition	App-file is bigger than 10 GB.

Procedure

Steps	Action	Expected Result
1.	Check the size of the .app-file.