Design and Development

Design and development must follow design guides and best practices

See Best Practices and Design Guidelines

Persistence through eHealth Infrastructure Services

Telemedicine solutions do not get their own persistence but persistence must be handled through the eHealth Infrastructure services (including key-value store).

Deployment and application definition

Applications must be based on predefined Docker Images

1. Telemedicine solution must be built using predefined eHealth docker base images (Docker Base Images)

2. The applications in the container must run as non-root  (Docker Base Images Security)

3. The docker image that is pushed to the central docker image repository must be signed with an approved private key. (Image Signing)

Applications must be based on predefined Helm charts

1. The application must be deployed using one of the official eHealth helm charts available here: https://registry.admin.ehealth.sundhed.dk/harbor/projects/5/helm-charts. See Helm Charts

2. The application must be deployed using the most recent version of the helm chart.

Application Requirements

1. The application containers must not have any state that cannot be recreated

   1. A service may keep an in-memory cache, as long as all instances of the same service can also handle requests for objects in the cache. And the service has a sane strategy to refresh/clean up the cache.

   2. A service cannot have any state that cannot be recreated if the service is moved to another worker node, or if the service is called

   3. A service cannot hold sessions locally

   4. A service cannot run its database

2. The application containers must accept restart and reset at any time.

   1. The container may at any time receive a SIGTERM signal to the main process (PID 1).

   2. At restart the application filesystem is reset to the one from the docker image.

   3. If the application fails to exit gracefully before a timeout of 10 seconds the application will be killed by the platform.

3. The application containers should be able to scale to multiple instances.

   1. It is recommended that all services run at least in two instances in different data centres to have high availability.

   2. If a service is busy it may be scaled to more than two instances, to handle extra requests.

   3. For this reason, a service cannot have any state that cannot be recreated if the service is moved to another worker node, or if the service is called

      1. A service cannot hold sessions locally

      2. A service cannot run its database

      3. A service may keep an in-memory cache, as long as all instances of the same service can also handle requests for objects in the cache. And the service has a sane strategy to refresh/clean up the cache.

Health Checks

1. Each application container must expose two endpoints for readiness and liveness status. See also this article: Kubernetes liveness and readiness probes difference

2. Liveness Probe

   1. The liveness probe shall check the container's health.

   2. If for some reason, the liveness probe fails (x times), the eHealth Infrastructure restarts the container.

   3. Suppose that a Pod is running our application inside a container, but due to some reason let’s say memory leak, CPU usage, application deadlock etc the application is not responding to our requests, and is stuck in an error state.

3. Readiness Probe - such as a /healthz endpoint

   1. The endpoint must respond with status code 200 when the application is ready to receive requests

   2. In some cases, we would like our application to be alive, but not serve traffic unless some conditions are met e.g., populating a dataset, waiting for some other service to be alive etc. In such cases, we use a readiness probe. If the condition inside the readiness probe passes, only then the application can serve traffic.

Deployment

1. Every release must follow a strict release plan where all eHealth Environments (see Environments ) are visited in the show order.

   1. external test → pre-production → production

2. Rollback must always be possible. A new release must never have contradictory demands with the previous version.

Monitoring and Operations

Logging requirements

1. Follow the specification for the application log (Logging model)

2. Errors and essential incidents must be found in the application log

Request Headers

1. Headers used for authentication and authorization must be set

2. B3 header propagation

   1. Tracing headers must be propagated as described in https://istio.io/docs/tasks/telemetry/distributed-tracing/.

   2. This can be handled by libraries like https://github.com/jaegertracing/jaeger-client-java

3. Applications accessing the infrastructure are encouraged to expose their application identity by using the HTTP Header User Agent, eg: User agent : HAPI-FHIR/4.1.0 (FHIR Client; FHIR 3.0.2/DSTU3; apache) or User agent : CGI-CC360-COPD/1.0.3. This information can then be used to provide proper redirects.

Documentation

1. Documentation of the application purpose, service requirements and resource usage