Logging model

This page describes the logging model for developing eHealth services or Telemedicine solutions. The intended audience is architects and developers.
There are a several types of logging;

  1. Audit logging (or “Anvendelseslog”) - to record the occurrence of an event, the time at which it occurred, the responsible user or service, and the impacted resource. This is done automatically by the eHealth Infrastructure.

  2. System logging (or “driftlog”) - to enable problem-solving. The application must record the events that indicate the service process and errors.

  3. Metrics logging

  4. Request Response logging

The audit log is the source for events sent to MinLog2.

Table of Contents

Logging Overview

This section provides an overview of logging in the eHealth infrastructure. When e.g. Telemedicine solutions access the Infrastructure services several logs are created:

  • Audit logging - capturing information on who made the call, what action was performed, on behalf of which organization, who was the subject/patient etc. This forms the basis for the registration in MinLog2.

  • Request Response logging - documents the entire payload of the request and responds to an s3 bucket. Primary used for debugging or problem-solving.

  • System log - to be able to trace events for problem-solving. These are logged to Splunk.

  • Application metrics are measurements used to evaluate and assess a system’s efficiency, responsiveness, and overall health. These are logged to Splunk.

 

 

Archimate - logging.png
The internals of the logging mechanisms in the eHealth Infrastructure. Telemedicine Solutions and eHealth services can log system logs and metrics to Splunk. For eHealth services, additional Audit Logs and Request Responses are logged.

The audit logs are pseudonymized and only log the eHealth assigned Patient ID, Organisations ID, and Practitioners ID are stored. These must be correlated with e.g. information in the eHealth database to get e.g. name or CPR number.

Note, the audit log in the database is only temporary until the information is registered in NSP Min Log 2. To audit log in Splunk should be used to look up who did want and when.

System log (application log)

The purpose of the system log is to be able to trace events for problem-solving. This is useful for a single component and across the entire system.

Every component must:

  1. Log Errors and essential events

  2. System logs are sent out on stdout. (the eHealth Infrastructure will pick these up and forward them to Splunk)

  3. System logs must not contain sensitive data ex. CPR.

  4. System logs must be in valid JSON format.

  5. The System log must contain the following elements (part of CIM: https://docs.splunk.com/Documentation/CIM/4.12.0/User/Overview)

Field name

Data type

Description

Field name

Data type

Description

time

time

The time of the incident in format yyyy-mm-dd hh:mm:ss:nnnnnnz ("time"="2019-03-01T08:58:26.986123Z") Zulu time zone

app

string

The application involved in the event, such as win:app:trendmicrovmwarenagios.

body

string

The body of a message.

id

string

The unique identifier of a message. TraceId as explained in Call Tracing

severity

string

The severity of a message. 

Allowed values (see also Alerts - Splunk Documentation):

  • critical

  • high

  • medium

  • low

  • informational

If using frameworks like log4j this is a proposal for translating log4j logging levels to CIM:

  • TRACE > informational

  • DEBUG > informational

  • INFO > low

  • WARN > medium

  • ERROR > high

  • FATAL > critical

subject

string

The message subject. Could be Java class name or anything that defines the context.

type

string

The message type. 

Allowed values (see also Alerts - Splunk Documentation):

  • alarm

  • alert

  • event

  • task

This field is used for addressing the log;

  • alarm - an unexpected event occurred. The event calls for manual action.

  • alert - a plausible error occurred. Ex a server has received invalid data.

  • event - a normal event occurred for which a log entry is desired.

  • task - undecided

If the application wishes to log other things than the ones in the table above, they must match fields in the relevant CIM model.

The infrastructure handles the collection of resource usage and trace data - Istio (https://istio.io/docs/reference/config/policy-and-telemetry/metrics/)

Hint: In Java use MDC to easily add required log entities. 

Audit log and Request Response log

The Telemedicine Infrastructure ensures all requests to the Infrastructure services are audit logged and the request and responses are also logged.

Request-Response-log

The Request-Response-log documents the entire payload of the request and response.

The Request-Response log and Audit-log are only performed on calls to the Infrastructure Services. Calls to the components in the Telemedicine Solutions (e.g. BFFs), are not audited and request are logged.

So, if an Application caches data locally e.g. in the BFF, the access to this data cannot be audit-logged by the Infrastructure and thereby trace who accessed the information.

Audit log

The Audit-log documents who made the call, what action was performed, on behalf of which organization, who was the subject/patient, if the call failed, etc...

Audit-log data is sent via ActiveMQ to a central service. This service is responsible for persisting the audit log.

When a call enters the cluster a TraceId is generated. See Call Tracing

Audit logging is based on the FHIR AuditEvent resource (http://hl7.org/fhir/r4/auditevent.html). JSON-formatted AuditEvent entities are sent to the topic "virtual.ehealth-auditevent".

The "who", "when", and "what" that must be provided to supply a meaningful AuditEvent are the following:

  • who: Practitioner/Patient reference as "agent.userId.value"

  • when: the value of "recorded", eg. "2019-12-04T11:59:28.646+00:00"

  • what: as entries in the "entities" list (typically Patient ID)

Link: https://drive.google.com/file/d/17lkCVScT2nyweapw6CtL20QRKCUoCCuj/view

Integration to NSP MinLog 2

The eHealth Infrastructure registers practitioners' access to citizens' data to the MinLog 2 log. This is technically done by the eHealth Infrastructure forwarding certain audit events to the MinLog 2.

With the registration in MinLog2, the citizens can access MinLog2 through eg. sundhed.dk or fmk-online.dk and see which employees (typically practitioners) have accessed information about the citizen.

Audit event eligibility for registering in the MinLog 2 log

Audit events eligible for registering in the MinLog2 log are:

  1. Practitioners read and update citizens' data.

  2. Practitioners read and update multiple citizens' data. This is handled as several MinLog2 log registrations.

The following audit events are not eligible for registering in the MinLog2 log:

  1. Citizens read/update their data.

  2. AuditEvent with AuditEvent.purposeOfEvent set to Coding with system = “http://ehealth.sundhed.dk/fhir/PurposeOfUse" and code = “INTERNAL_AUDIT_ONLY".

Access being registered in MinLog 2

The eHealth Infrastructure registers practitioners read and updating of citizen data in MinLog 2. The read/update of the following resource types do cause MinLog 2 registration:

Logical Resource Type

FHIR Resource Types (described in AuditEvent.outcomeDesc)

Logical Resource Type

FHIR Resource Types (described in AuditEvent.outcomeDesc)

telemedicinsk aktivitetsplan

EpisodeOfCare, Condition, CarePlan, ServiceRequest, Provenance

Observation, QuestionnaireResponse, Media

Appointment, AppointmentResponse

DeviceUseStatement

DocumentReference, PlanDefinition, Goal

telemedicinske stamdata

Consent, Patient, RelatedPerson

telemedicinsk aktivitetsopfølgning

ClinicalImpression, GuidanceResponse, Task

telemedicinsk kommunikation

Communication

telemedicinsk kommunikationsopsætning

CommunicationRequest

telemedicinske data

Bundle, Library, and any other resource types not mentioned above

Practitioners read and update the following eHealth Infrastructure resource types do not cause any MinLog 2 registration:

  • Administrative resource types

    • PlanDefinition

    • ActivityDefinition

    • DocumentReference

    • Library

    • Basic (ehealth-actionguidance/ehealth-view)

    • Questionnaire

    • StructureDefinition (ehealth-definedquestion)

    • Organization

    • CareTeam

    • Practitioner

    • PractitionerRole

  • Non-citizen-specific

    • DeviceMetric

    • Device

  • Terminology

    • CodeSystem, ValueSet, ConceptMap, NamingSystem

Organization Information sent to MinLog 2

The MinLog API enables the inclusion of the Practioners organization in the registration.

The eHealth Infrastructure derive the organization from the user's selected 'context,' i.e., the organization chosen by the user in the solution.

The registration of the organisation is done through the "OrganisationId" and "OrganisationId attribut source fields, respectively. See https://www.nspop.dk/display/public/web/MinLog2+-+Min+Log+Registrering+-+Guide+til+anvendere#MinLog2MinLogRegistreringGuidetilanvendere-Request.

The eHealth Infrastructure uses “SOR” as default as the “OrganisationId attribut source".

If the organisation's SOR identifier is not available in the eHealth Infrastructure, the integration will use the CVR number in the MinLog registration.

The OrganisationName is optional and is not set by eHealth Infrastructure.

See Importing and updating Organization information from SOR and FK-Organisation into eHealth Infrastructure for how the SOR identifier becomes available in the eHealth Infrastructure.

Practioner Information sent to MinLog 2

The MinLog API enables the inclusion of both Practioners CPR and user names through the "UserPersonIdentifier" and "UserPersonName" fields, respectively.

See https://www.nspop.dk/display/public/web/MinLog2+-+Min+Log+Registrering+-+Guide+til+anvendere#MinLog2MinLogRegistreringGuidetilanvendere-Request.

The eHealth Infrastructure only sets the UserPersonIdentifier. The UserPersonName is optional and is not set by eHealth Infrastructure.

In the MinLog2 API, it is not possible to specify whether the Practioner should be anonymous.

What and how Practioner information is displayed, is up to entities such as sundhed.dk and other clients, such as fmk-online.dk.

Aggregation of registering in MinLog 2

The eHealth Infrastructure aggregates identical actions to fewer registrations in the MinLog 2 to avoid data "flooding" in the citizen's log.

The aggregation of messages to MinLog2 takes place via a 1-hour "sliding window", such that two or more identical loggings to MinLog2 only become one logging in practice if they happen within the same hour.

Two AuditEvents are considered equal if they share the same combination of:

  • AuditEvent.action, for instance, a Coding with system = http://hl7.org/fhir/audit-event-action and code = C for Create.

  • Logical Resource Type, see above table, mapping of AuditEvent.outcomeDesc.

The internals of audit logging

What is audit-logged

The eHealth Infrastructure sends AuditEvents to ActiveMQ for each request processed, regardless of whether any entities were modified or returned. 
Non-authorized requests or those failing due to a system error do not need to be audit-logged (they are logged in the request/response log). However, if they are logged audit-logged, it is essential to configure the actionOutcome accordingly (4 for HTTP status 4xx responses and 8 for HTTP status 5xx responses).
HEAD requests do not need to be audit-logged (they are logged in the request/response log).
Requests where the JWT user_type=SYSTEM do not need to be audit-logged (they are logged in the request/response log).

Sending AuditEvents to ActiveMQ

Additional requirements to the AuditEvent sent to ActiveMQ, besides the one listed on http://hl7.org/fhir/r4/auditevent.html:

  • action (what): The action taken (C=create, R=read/search/history, U=update/patch, D=delete, E=custom operations).

    • If action=E then the name of the custom FHIR operation must be provided in the "subtype.code" element. If the action is not E, the proper value from http://hl7.org/fhir/r4/valueset-audit-event-sub-type.html (the "RestOperationTypeEnum") must be present as the "subtype.code", eg. "search-type".

  • outcomeDesc (what)

    • Must be the name of the primary FHIR resource type on which the event relates to

  • agent (who)

    • Must have exactly 1 requestor that has a userId value

    • When an organization is supplied in the JWT context, it must be supplied as an extension on the requestor with the URL "http://ehealth.sundhed.dk/fhir/StructureDefinition/ehealth-responsibleOrganization".

    • purposeOfUse (optional) may be supplied to indicate the agent’s underlying purpose

  • source (where)

  • entity (what): Accessed data entities must be listed in the entity list (prefer fully qualified FHIR references for FHIR entities).

    • resource:

      • role.code=4

      • lifecycle is used to show what dao operation was performed on the resource

    • patient: 

      • Patient entities must have role.code=1

      • Even If a patient is not the entity accessed, but is somehow related to the entity accessed, then that patient should also be added as an entity, with entity.role.code=1

        • This is important for the MinLog2 integration (otherwise no entry is made in MinLog2 regarding the event)

        • This also makes the FHIR ID of the patient go into a special "patient" JSON property in the audit log, making it easily queryable in Splunk

        • This also applies to search operations: For each entity returned in the search, if the entity is related to a patient, that patient must also be added as an entity

      • If data of multiple patients is accessed then 1 AuditEvent should be generated per patient with data being accessed. Each such AuditEvent should be identical except for the included entities. There should be 1 "patient" entity (role.code=1) and any number of other entities that belong to that patient

        • This means there is not a 1:1 correspondence between 1 HTTP request generates 1 audit log entry, but trace-ids can link the audit log entries together if required. 

    • trace-id:

      • value of "x-b3-traceid" header must be provided as an entity with type.code=2 and role.code=21 and identifier.system="http://ehealth.sundhed.dk"

    • search: AuditEvents for search operations and "get page" requests must include any search parameters in entity.query and any search result bundle id in a separate AuditEvent.entity with role.code=24

      • query:

        • The query parameters must be serialized to a UTF-8 encoded string (preferably as JSON) before they are encoded as base64 encoded bytes, as required per the data type of the query property

        • Both search parameters in HTTP query strings and parameters in the body of POST-style searches must be included in the AuditEvent

        • Note: CPR numbers must never be present anywhere in AuditEvents. If a search parameter contains a CPR number then it should be masked (in an obvious way, e.g. replaced by 'xxxxxxxxxx').
          (Privileged administrators can find the actual CPR number used in the search operation in the request/response log, should it prove necessary for an investigation)

        • Example: An HTTP POST call to https://example.com/Patient/_search with body identifier=urn:oid:1.2.208.176.1.2|2603200001 could result in an AuditEvent.entity.query={"identifier": "urn:oid:1.2.208.176.1.2|xxxxxxxxxx"}

      • bundle: The bundle ID should be added to the value of the identifier of the entity.

      • Example entity: {"identifier":{"value":"ce6d8410-c67f-42d5-8de3-9ebb2a1aef65"},"type":{"system":"http://hl7.org/fhir/security-source-type","code":"4","display":"Application Server"},"role":{"system":"http://hl7.org/fhir/object-role","code":"24","display":"Query"},"description":"search entity"}

  • purposeOfEvent (optional)

    • Any purposeOfEvent codings present on the AuditEvent will be added to the audit log entry

    • A special coding can be added to prevent the AuditEvent from being sent to external audit services (e.g. MinLog2)

      •  system="http://ehealth.sundhed.dk/fhir/PurposeOfUse"

      • code="INTERNAL_AUDIT_ONLY"

An example is displayed below.

{ "resourceType":"AuditEvent", "type":{ "system":"http://hl7.org/fhir/audit-event-type", "code":"rest", "display":"RESTful Operation" }, "subtype":[ { "system":"http://hl7.org/fhir/restful-interaction", "code":"$createPatient" } ], "action":"E", "recorded":"2019-12-04T11:59:28.646+00:00", "outcome":"0", "outcomeDesc":"Patient", "agent":[ { "extension":[ { "url":"http://ehealth.sundhed.dk/fhir/StructureDefinition/ehealth-responsibleOrganization", "valueReference":{ "reference":"https://organization.inttest.ehealth.sundhed.dk/fhir/Organization/10357" } } ], "userId":{ "system":"http://ehealth.sundhed.dk", "value":"https://organization.inttest.ehealth.sundhed.dk/fhir/Practitioner/143473" }, "requestor":true }, { "purposeOfUse": [ { "coding": [ { "system": "agent1 system 1", "code": "agent1 code 1" } ], "text": "a1-c1-text" } ] } ], "source":{ "identifier":{ "system":"http://ehealth.sundhed.dk" "value":"https://patient.exttest.ehealth.sundhed.dk" }, "type":[ { "system":"http://hl7.org/fhir/security-source-type", "code":"4" } ] }, "entity":[ { "reference":{ "reference":"https://patient.inttest.ehealth.sundhed.dk/fhir/Patient/852" }, "role":{ "system":"http://hl7.org/fhir/object-role", "code":"1" } }, { "identifier":{ "system":"http://ehealth.sundhed.dk", "value":"6b507ee2d716780372c255df69ece653" }, "type":{ "system":"http://hl7.org/fhir/security-source-type", "code":"2", "display":"Data Interface" }, "role":{ "system":"http://hl7.org/fhir/object-role", "code":"21", "display":"Job Stream" } } ], "purposeOfEvent": [ { "coding": [ { "system": "http://ehealth.sundhed.dk/fhir/PurposeOfUse", "code": "INTERNAL_AUDIT_ONLY" } ] } ], "headers":[ ] }

Splunk processing

A central service handles the AuditEvent entities and writes them to Splunk to allow for search, statistics, report generation etc.

https://docs.ehealth.sundhed.dk/latest-released/ig/StructureDefinition-ehealth-auditevent.html#splunk-processing describes the AuditEvent elements mapping to attributes in logging. E.g. FHIR AuditEvent.outcomeDesc is mapped to log attribute actionResource.

However, only a simplified version of the AuditEvent is logged, containing the following attributes:

Splunk Event Attribute

FHIR AuditEvent source

Splunk Event Attribute

FHIR AuditEvent source

actionOutcome

outcome

actionResource

outcomeDesc

actionType

action

entities

entity.reference (where role.code<>21)

issuerId

agent.userId.value

organizationId

agent.extension(ehealth-responsibleOrganization).valueReference.reference

patientId

entity.reference (where role.code=1)

subtype

subtype.code

time

recorded

traceId

entity.identifier.value (where role.code=21 and type.code=2)

queryParameters

entity.query (where role.code=24)

bundleId

entity.identifier.value (where role.code=24)

source

source.identifier.value

purposeOfEvent

purposeOfEvent

agents

agent with purposeOfUse assigned

An example:

{ "traceId":"nPPwpZQqDdCIHCcNXjKB", "issuerId":"http://organization.fut.trifork.com/fhir/Practitioner/35205", "organizationId":"http://organization.fut.trifork.com/fhir/Organization/10357", "patientId":"http://patient.fut.trifork.com/fhir/Patient/286", "time":"2019-12-04 11:01:59.601", "actionType":"R", "actionResource":"Patient", "actionOutcome":"0", "subtype":"_search", "entities":[ "http://patient.fut.trifork.com/fhir/Patient/286" ], "purposeOfEvent": [ "http://ehealth.sundhed.dk/fhir/PurposeOfUse|INTERNAL_AUDIT_ONLY" ], "agents": [ { "purposeOfUse": [ "agent1 system 1|agent1 code 1" ], "purposeOfUseText": [ "a1-c1-text" ] } ], "type": "audit" }

Once indexed in Splunk, entities can be queried wrt. generating reports, statistics etc. For example, to view the number of audit entries for the different practitioners on for example the Internal Test Environment (INTTEST), use this Splunk query  (notice: only available if additional Splunk access is provided due to the sensitive nature of audit data):

index="inttest_k8s_ehealth-audit_audit" kubernetes_deployment_name="auditlog-consumer" sourcetype=kubernetes_logs kubernetes_container_name="auditlog-consumer" | top issuerId

Usage log

It is possible with the right permissions to Splunk indexes to make reports covering the usage of the system. These reports will be made in Splunk. A report could show the number of patients accessed in a period ordered by organizationid (see image below).


The organizationId could be resolved separately or by giving Splunk a translation map between the number and a more user-friendly organization name. This is only one way to process the data - in Splunk, it is possible to create many views and statistics based on these data.

When a template of a fulfilling report has been generated an automatic report generation could be scheduled for instance monthly. After creating the report it could be sent by email.