Table of Contents

Logging Overview

This section provides an overview of logging in the eHealth infrastructure. When e.g. Telemedicine solutions access the Infrastructure services several logs are created:

• Audit logging - capturing information on who made the call, what action was performed, on behalf of which organization, who was the subject/patient etc. This forms the basis for the registration in MinLog2.

• Request Response logging - documents the entire payload of the request and responds to an s3 bucket. Primary used for debugging or problem-solving.

• System log - to be able to trace events for problem-solving. These are logged to Splunk.

• Application metrics are measurements used to evaluate and assess a system’s efficiency, responsiveness, and overall health. These are logged to Splunk.

Open

The internals of the logging mechanisms in the eHealth Infrastructure. Telemedicine Solutions and eHealth services can log system logs and metrics to Splunk. For eHealth services, additional Audit Logs and Request Responses are logged.

System log (application log)

The purpose of the system log is to be able to trace events for problem-solving. This is useful for a single component and across the entire system.

Every component must:

1. Log Errors and essential events

2. System logs are sent out on stdout. (the eHealth Infrastructure will pick these up and forward them to Splunk)

3. System logs must not contain sensitive data ex. CPR.

4. System logs must be in valid JSON format.

5. The System log must contain the following elements (part of CIM: https://docs.splunk.com/Documentation/CIM/4.12.0/User/Overview)

If the application wishes to log other things than the ones in the table above, they must match fields in the relevant CIM model.

The infrastructure handles the collection of resource usage and trace data - Istio (https://istio.io/docs/reference/config/policy-and-telemetry/metrics/)

Hint: In Java use MDC to easily add required log entities. 

Audit log and Request Response log

The Telemedicine Infrastructure ensures all requests to the Infrastructure services are audit logged and the request and responses are also logged.

Request-Response-log

The Request-Response-log documents the entire payload of the request and response.

Audit log

The Audit-log documents who made the call, what action was performed, on behalf of which organization, who was the subject/patient, if the call failed, etc...

Audit-log data is sent via ActiveMQ to a central service. This service is responsible for persisting the audit log.

When a call enters the cluster a TraceId is generated. See Call Tracing

Audit logging is based on the FHIR AuditEvent resource (http://hl7.org/fhir/r4/auditevent.html). JSON-formatted AuditEvent entities are sent to the topic "virtual.ehealth-auditevent".

The "who", "when", and "what" that must be provided to supply a meaningful AuditEvent are the following:

• who: Practitioner/Patient reference as "agent.userId.value"

• when: the value of "recorded", eg. "2019-12-04T11:59:28.646+00:00"

• what: as entries in the "entities" list (typically Patient ID)

Open

_{Link: https://drive.google.com/file/d/17lkCVScT2nyweapw6CtL20QRKCUoCCuj/view}

Integration to NSP MinLog 2

The eHealth Infrastructure registers practitioners' access to citizens' data to the MinLog 2 log. This is technically done by the eHealth Infrastructure forwarding certain audit events to the MinLog 2.

With the registration in MinLog2, the citizens can access MinLog2 through eg. sundhed.dk or fmk-online.dk and see which employees (typically practitioners) have accessed information about the citizen.

Audit event eligibility for registering in the MinLog 2 log

Audit events eligible for registering in the MinLog2 log are:

1. Practitioners read and update citizens' data.

2. Practitioners read and update multiple citizens' data. This is handled as several MinLog2 log registrations.

The following audit events are not eligible for registering in the MinLog2 log:

1. Citizens read/update their data.

2. AuditEvent with AuditEvent.purposeOfEvent set to Coding with system = “http://ehealth.sundhed.dk/fhir/PurposeOfUse" and code = “INTERNAL_AUDIT_ONLY".

Access being registered in MinLog 2

The eHealth Infrastructure registers practitioners read and updating of citizen data in MinLog 2. The read/update of the following resource types do cause MinLog 2 registration:

Practitioners read and update the following eHealth Infrastructure resource types do not cause any MinLog 2 registration:

• Administrative resource types

  • PlanDefinition

  • ActivityDefinition

  • DocumentReference

  • Library

  • Basic (ehealth-actionguidance/ehealth-view)

  • Questionnaire

  • StructureDefinition (ehealth-definedquestion)

  • Organization

  • CareTeam

  • Practitioner

  • PractitionerRole

• Non-citizen-specific

  • DeviceMetric

  • Device

• Terminology

  • CodeSystem, ValueSet, ConceptMap, NamingSystem

Organization Information sent to MinLog 2

The MinLog API enables the inclusion of the Practioners organization in the registration.

The eHealth Infrastructure derive the organization from the user's selected 'context,' i.e., the organization chosen by the user in the solution.

The registration of the organisation is done through the "OrganisationId" and "OrganisationId attribut source fields, respectively. See https://www.nspop.dk/display/public/web/MinLog2+-+Min+Log+Registrering+-+Guide+til+anvendere#MinLog2MinLogRegistreringGuidetilanvendere-Request.

The eHealth Infrastructure uses “SOR” as default as the “OrganisationId attribut source".

If the organisation's SOR identifier is not available in the eHealth Infrastructure, the integration will use the CVR number in the MinLog registration.

The OrganisationName is optional and is not set by eHealth Infrastructure.

See Importing and updating Organization information from SOR and FK-Organisation into eHealth Infrastructure for how the SOR identifier becomes available in the eHealth Infrastructure.

Practioner Information sent to MinLog 2

The MinLog API enables the inclusion of both Practioners CPR and user names through the "UserPersonIdentifier" and "UserPersonName" fields, respectively.

See https://www.nspop.dk/display/public/web/MinLog2+-+Min+Log+Registrering+-+Guide+til+anvendere#MinLog2MinLogRegistreringGuidetilanvendere-Request.

The eHealth Infrastructure only sets the UserPersonIdentifier. The UserPersonName is optional and is not set by eHealth Infrastructure.

In the MinLog2 API, it is not possible to specify whether the Practioner should be anonymous.

What and how Practioner information is displayed, is up to entities such as sundhed.dk and other clients, such as fmk-online.dk.

Aggregation of registering in MinLog 2

The eHealth Infrastructure aggregates identical actions to fewer registrations in the MinLog 2 to avoid data "flooding" in the citizen's log.

The aggregation of messages to MinLog2 takes place via a 1-hour "sliding window", such that two or more identical loggings to MinLog2 only become one logging in practice if they happen within the same hour.

Two AuditEvents are considered equal if they share the same combination of:

• AuditEvent.action, for instance, a Coding with system = http://hl7.org/fhir/audit-event-action and code = C for Create.

• Logical Resource Type, see above table, mapping of AuditEvent.outcomeDesc.

The internals of audit logging

What is audit-logged

The eHealth Infrastructure sends AuditEvents to ActiveMQ for each request processed, regardless of whether any entities were modified or returned. 
Non-authorized requests or those failing due to a system error do not need to be audit-logged (they are logged in the request/response log). However, if they are logged audit-logged, it is essential to configure the actionOutcome accordingly (4 for HTTP status 4xx responses and 8 for HTTP status 5xx responses).
HEAD requests do not need to be audit-logged (they are logged in the request/response log).
Requests where the JWT user_type=SYSTEM do not need to be audit-logged (they are logged in the request/response log).

Sending AuditEvents to ActiveMQ

Additional requirements to the AuditEvent sent to ActiveMQ, besides the one listed on http://hl7.org/fhir/r4/auditevent.html:

• action (what): The action taken (C=create, R=read/search/history, U=update/patch, D=delete, E=custom operations).

  • If action=E then the name of the custom FHIR operation must be provided in the "subtype.code" element. If the action is not E, the proper value from http://hl7.org/fhir/r4/valueset-audit-event-sub-type.html (the "RestOperationTypeEnum") must be present as the "subtype.code", eg. "search-type".

• outcomeDesc (what)

  • Must be the name of the primary FHIR resource type on which the event relates to

• agent (who)

  • Must have exactly 1 requestor that has a userId value

  • When an organization is supplied in the JWT context, it must be supplied as an extension on the requestor with the URL "http://ehealth.sundhed.dk/fhir/StructureDefinition/ehealth-responsibleOrganization".

  • purposeOfUse (optional) may be supplied to indicate the agent’s underlying purpose

• source (where)

  • Should have an identifier with

    • system: http://ehealth.sundhed.dk

    • value: external URL of the resource (e.g. https://patient.exttest.ehealth.sundhed.dk)

• entity (what): Accessed data entities must be listed in the entity list (prefer fully qualified FHIR references for FHIR entities).

  • resource:

    • role.code=4

    • lifecycle is used to show what dao operation was performed on the resource

      • system: HL7.TERMINOLOGY\ - FHIR v4.0.1

      • update: "Amendment", delete: "Logical deletion", create: "Creation", read/search: "Access"

  • patient: 

    • Patient entities must have role.code=1

    • Even If a patient is not the entity accessed, but is somehow related to the entity accessed, then that patient should also be added as an entity, with entity.role.code=1

      • This is important for the MinLog2 integration (otherwise no entry is made in MinLog2 regarding the event)

      • This also makes the FHIR ID of the patient go into a special "patient" JSON property in the audit log, making it easily queryable in Splunk

      • This also applies to search operations: For each entity returned in the search, if the entity is related to a patient, that patient must also be added as an entity

    • If data of multiple patients is accessed then 1 AuditEvent should be generated per patient with data being accessed. Each such AuditEvent should be identical except for the included entities. There should be 1 "patient" entity (role.code=1) and any number of other entities that belong to that patient

      • This means there is not a 1:1 correspondence between 1 HTTP request generates 1 audit log entry, but trace-ids can link the audit log entries together if required. 

  • trace-id:

    • value of "x-b3-traceid" header must be provided as an entity with type.code=2 and role.code=21 and identifier.system="http://ehealth.sundhed.dk"

  • search: AuditEvents for search operations and "get page" requests must include any search parameters in entity.query and any search result bundle id in a separate AuditEvent.entity with role.code=24

    • query:

      • The query parameters must be serialized to a UTF-8 encoded string (preferably as JSON) before they are encoded as base64 encoded bytes, as required per the data type of the query property

      • Both search parameters in HTTP query strings and parameters in the body of POST-style searches must be included in the AuditEvent

      • Note: CPR numbers must never be present anywhere in AuditEvents. If a search parameter contains a CPR number then it should be masked (in an obvious way, e.g. replaced by 'xxxxxxxxxx').
        (Privileged administrators can find the actual CPR number used in the search operation in the request/response log, should it prove necessary for an investigation)

      • Example: An HTTP POST call to https://example.com/Patient/_search with body identifier=urn:oid:1.2.208.176.1.2|2603200001 could result in an AuditEvent.entity.query={"identifier": "urn:oid:1.2.208.176.1.2|xxxxxxxxxx"}

    • bundle: The bundle ID should be added to the value of the identifier of the entity.

    • Example entity: {"identifier":{"value":"ce6d8410-c67f-42d5-8de3-9ebb2a1aef65"},"type":{"system":"http://hl7.org/fhir/security-source-type","code":"4","display":"Application Server"},"role":{"system":"http://hl7.org/fhir/object-role","code":"24","display":"Query"},"description":"search entity"}

• purposeOfEvent (optional)

  • Any purposeOfEvent codings present on the AuditEvent will be added to the audit log entry

  • A special coding can be added to prevent the AuditEvent from being sent to external audit services (e.g. MinLog2)

    •  system="http://ehealth.sundhed.dk/fhir/PurposeOfUse"

    • code="INTERNAL_AUDIT_ONLY"

An example is displayed below.

Splunk processing

A central service handles the AuditEvent entities and writes them to Splunk to allow for search, statistics, report generation etc.

https://docs.ehealth.sundhed.dk/latest-released/ig/StructureDefinition-ehealth-auditevent.html#splunk-processing describes the AuditEvent elements mapping to attributes in logging. E.g. FHIR AuditEvent.outcomeDesc is mapped to log attribute actionResource.

However, only a simplified version of the AuditEvent is logged, containing the following attributes:

An example:

Once indexed in Splunk, entities can be queried wrt. generating reports, statistics etc. For example, to view the number of audit entries for the different practitioners on for example the Internal Test Environment (INTTEST), use this Splunk query  (notice: only available if additional Splunk access is provided due to the sensitive nature of audit data):

Usage log

It is possible with the right permissions to Splunk indexes to make reports covering the usage of the system. These reports will be made in Splunk. A report could show the number of patients accessed in a period ordered by organizationid (see image below).

The organizationId could be resolved separately or by giving Splunk a translation map between the number and a more user-friendly organization name. This is only one way to process the data - in Splunk, it is possible to create many views and statistics based on these data.

When a template of a fulfilling report has been generated an automatic report generation could be scheduled for instance monthly. After creating the report it could be sent by email.