Logging model
This page describes the logging model for developing eHealth services or Telemedicine solutions. The intended audience is architects and developers.
There are a several types of logging;
Audit logging (or “Anvendelseslog”) - to record the occurrence of an event, the time at which it occurred, the responsible user or service, and the impacted resource. This is done automatically by the eHealth Infrastructure.
System logging (or “driftlog”) - to enable problem-solving. The application must record the events that indicate the service process and errors.
Metrics logging
Request Response logging
The audit log is the source for events sent to MinLog2.
Table of Contents
Logging Overview
This section provides an overview of logging in the eHealth infrastructure. When e.g. Telemedicine solutions access the Infrastructure services several logs are created:
Audit logging - capturing information on who made the call, what action was performed, on behalf of which organization, who was the subject/patient etc. This forms the basis for the registration in MinLog2.
Request Response logging - documents the entire payload of the request and responds to an s3 bucket. Primary used for debugging or problem-solving.
System log - to be able to trace events for problem-solving. These are logged to Splunk.
Application metrics are measurements used to evaluate and assess a system’s efficiency, responsiveness, and overall health. These are logged to Splunk.
The audit logs are pseudonymized and only log the eHealth assigned Patient ID, Organisations ID, and Practitioners ID are stored. These must be correlated with e.g. information in the eHealth database to get e.g. name or CPR number.
Note, the audit log in the database is only temporary until the information is registered in NSP Min Log 2. To audit log in Splunk should be used to look up who did want and when.
System log (application log)
The purpose of the system log is to be able to trace events for problem-solving. This is useful for a single component and across the entire system.
Every component must:
Log Errors and essential events
System logs are sent out on
stdout
. (the eHealth Infrastructure will pick these up and forward them to Splunk)System logs must not contain sensitive data ex. CPR.
System logs must be in valid JSON format.
The System log must contain the following elements (part of CIM: https://docs.splunk.com/Documentation/CIM/4.12.0/User/Overview)
Field name | Data type | Description |
---|---|---|
time | time | The time of the incident in format yyyy-mm-dd hh:mm:ss:nnnnnnz ("time"="2019-03-01T08:58:26.986123Z") Zulu time zone |
| string | The application involved in the event, such as |
| string | The body of a message. |
| string | The unique identifier of a message. TraceId as explained in Call Tracing |
| string | The severity of a message. Allowed values (see also Alerts - Splunk Documentation):
If using frameworks like log4j this is a proposal for translating log4j logging levels to CIM:
|
| string | The message subject. Could be Java class name or anything that defines the context. |
| string | The message type. Allowed values (see also Alerts - Splunk Documentation):
This field is used for addressing the log;
|
If the application wishes to log other things than the ones in the table above, they must match fields in the relevant CIM model.
The infrastructure handles the collection of resource usage and trace data - Istio (https://istio.io/docs/reference/config/policy-and-telemetry/metrics/)
Hint: In Java use MDC to easily add required log entities.
Audit log and Request Response log
The Telemedicine Infrastructure ensures all requests to the Infrastructure services are audit logged and the request and responses are also logged.
Request-Response-log
The Request-Response-log documents the entire payload of the request and response.
The Request-Response log and Audit-log are only performed on calls to the Infrastructure Services. Calls to the components in the Telemedicine Solutions (e.g. BFFs), are not audited and request are logged.
So, if an Application caches data locally e.g. in the BFF, the access to this data cannot be audit-logged by the Infrastructure and thereby trace who accessed the information.
Audit log
The Audit-log documents who made the call, what action was performed, on behalf of which organization, who was the subject/patient, if the call failed, etc...
Audit-log data is sent via ActiveMQ to a central service. This service is responsible for persisting the audit log.
When a call enters the cluster a TraceId
is generated. See Call Tracing
Audit logging is based on the FHIR AuditEvent resource (http://hl7.org/fhir/r4/auditevent.html). JSON-formatted AuditEvent entities are sent to the topic "virtual.ehealth-auditevent
".
The "who", "when", and "what" that must be provided to supply a meaningful AuditEvent are the following:
who: Practitioner/Patient reference as "agent.userId.value"
when: the value of "recorded", eg. "2019-12-04T11:59:28.646+00:00"
what: as entries in the "entities" list (typically Patient ID)
Link: https://drive.google.com/file/d/17lkCVScT2nyweapw6CtL20QRKCUoCCuj/view
Integration to NSP MinLog 2
The eHealth Infrastructure registers practitioners' access to citizens' data to the MinLog 2 log. This is technically done by the eHealth Infrastructure forwarding certain audit events to the MinLog 2.
With the registration in MinLog2, the citizens can access MinLog2 through eg. sundhed.dk or fmk-online.dk and see which employees (typically practitioners) have accessed information about the citizen.
Audit event eligibility for registering in the MinLog 2 log
Audit events eligible for registering in the MinLog2 log are:
Practitioners read and update citizens' data.
Practitioners read and update multiple citizens' data. This is handled as several MinLog2 log registrations.
The following audit events are not eligible for registering in the MinLog2 log:
Citizens read/update their data.
AuditEvent with
AuditEvent.purposeOfEvent
set to Coding withsystem
= “http://ehealth.sundhed.dk/fhir/PurposeOfUse
" andcode
= “INTERNAL_AUDIT_ONLY
".
Access being registered in MinLog 2
The eHealth Infrastructure registers practitioners read and updating of citizen data in MinLog 2. The read/update of the following resource types do cause MinLog 2 registration:
Logical Resource Type | FHIR Resource Types (described in |
---|---|
| EpisodeOfCare, Condition, CarePlan, ServiceRequest, Provenance Observation, QuestionnaireResponse, Media Appointment, AppointmentResponse DeviceUseStatement DocumentReference, PlanDefinition, Goal |
| Consent, Patient, RelatedPerson |
| ClinicalImpression, GuidanceResponse, Task |
| Communication |
| CommunicationRequest |
| Bundle, Library, and any other resource types not mentioned above |
Practitioners read and update the following eHealth Infrastructure resource types do not cause any MinLog 2 registration:
Administrative resource types
PlanDefinition
ActivityDefinition
DocumentReference
Library
Basic (ehealth-actionguidance/ehealth-view)
Questionnaire
StructureDefinition (ehealth-definedquestion)
Organization
CareTeam
Practitioner
PractitionerRole
Non-citizen-specific
DeviceMetric
Device
Terminology
CodeSystem, ValueSet, ConceptMap, NamingSystem
Organization Information sent to MinLog 2
The MinLog API enables the inclusion of the Practioners organization in the registration.
The eHealth Infrastructure derive the organization from the user's selected 'context,' i.e., the organization chosen by the user in the solution.
The registration of the organisation is done through the "OrganisationId
" and "OrganisationId attribut source
fields, respectively. See https://www.nspop.dk/display/public/web/MinLog2+-+Min+Log+Registrering+-+Guide+til+anvendere#MinLog2MinLogRegistreringGuidetilanvendere-Request.
The eHealth Infrastructure uses “SOR” as default as the “OrganisationId attribut source
".
If the organisation's SOR identifier is not available in the eHealth Infrastructure, the integration will use the CVR number in the MinLog registration.
The OrganisationName
is optional and is not set by eHealth Infrastructure.
See Importing and updating Organization information from SOR and FK-Organisation into eHealth Infrastructure for how the SOR identifier becomes available in the eHealth Infrastructure.
Practioner Information sent to MinLog 2
The MinLog API enables the inclusion of both Practioners CPR and user names through the "UserPersonIdentifier
" and "UserPersonName
" fields, respectively.
The eHealth Infrastructure only sets the UserPersonIdentifier
. The UserPersonName
is optional and is not set by eHealth Infrastructure.
In the MinLog2 API, it is not possible to specify whether the Practioner should be anonymous.
What and how Practioner information is displayed, is up to entities such as sundhed.dk and other clients, such as fmk-online.dk.
Aggregation of registering in MinLog 2
The eHealth Infrastructure aggregates identical actions to fewer registrations in the MinLog 2 to avoid data "flooding" in the citizen's log.
The aggregation of messages to MinLog2 takes place via a 1-hour "sliding window", such that two or more identical loggings to MinLog2 only become one logging in practice if they happen within the same hour.
Two AuditEvents are considered equal if they share the same combination of:
AuditEvent.action
, for instance, a Coding withsystem
=http://hl7.org/fhir/audit-event-action
andcode
=C
for Create.Logical Resource Type, see above table, mapping of
AuditEvent.outcomeDesc
.
The internals of audit logging
What is audit-logged
The eHealth Infrastructure sends AuditEvents to ActiveMQ for each request processed, regardless of whether any entities were modified or returned.
Non-authorized requests or those failing due to a system error do not need to be audit-logged (they are logged in the request/response log). However, if they are logged audit-logged, it is essential to configure the actionOutcome accordingly (4 for HTTP status 4xx responses and 8 for HTTP status 5xx responses).HEAD
requests do not need to be audit-logged (they are logged in the request/response log).
Requests where the JWT user_type=SYSTEM
do not need to be audit-logged (they are logged in the request/response log).
Sending AuditEvents to ActiveMQ
Additional requirements to the AuditEvent sent to ActiveMQ, besides the one listed on http://hl7.org/fhir/r4/auditevent.html:
action (what): The action taken (C=create, R=read/search/history, U=update/patch, D=delete, E=custom operations).
If action=E then the name of the custom FHIR operation must be provided in the "
subtype.code
" element. If the action is not E, the proper value from http://hl7.org/fhir/r4/valueset-audit-event-sub-type.html (the "RestOperationTypeEnum") must be present as the "subtype.code
", eg. "search-type".
outcomeDesc (what)
Must be the name of the primary FHIR resource type on which the event relates to
agent (who)
Must have exactly 1
requestor
that has auserId
valueWhen an organization is supplied in the JWT context, it must be supplied as an extension on the requestor with the URL "
http://ehealth.sundhed.dk/fhir/StructureDefinition/ehealth-responsibleOrganization
".purposeOfUse (optional) may be supplied to indicate the agent’s underlying purpose
source (where)
Should have an identifier with
system: http://ehealth.sundhed.dk
value: external URL of the resource (e.g. https://patient.exttest.ehealth.sundhed.dk)
entity (what): Accessed data entities must be listed in the
entity
list (prefer fully qualified FHIR references for FHIR entities).resource:
role.code=4
lifecycle is used to show what dao operation was performed on the resource
system: HL7.TERMINOLOGY\ - FHIR v4.0.1
update: "Amendment", delete: "Logical deletion", create: "Creation", read/search: "Access"
patient:
Patient entities must have
role.code=1
Even If a patient is not the entity accessed, but is somehow related to the entity accessed, then that patient should also be added as an entity, with
entity.role.code=1
This is important for the MinLog2 integration (otherwise no entry is made in MinLog2 regarding the event)
This also makes the FHIR ID of the patient go into a special "patient" JSON property in the audit log, making it easily queryable in Splunk
This also applies to search operations: For each entity returned in the search, if the entity is related to a patient, that patient must also be added as an entity
If data of multiple patients is accessed then 1 AuditEvent should be generated per patient with data being accessed. Each such AuditEvent should be identical except for the included entities. There should be 1 "patient" entity (role.code=1) and any number of other entities that belong to that patient
This means there is not a 1:1 correspondence between 1 HTTP request generates 1 audit log entry, but trace-ids can link the audit log entries together if required.
trace-id:
value of "x-b3-traceid" header must be provided as an entity with
type.code=2
and role.code=21
andidentifier.system="
http://ehealth.sundhed.dk"
search: AuditEvents for search operations and "get page" requests must include any search parameters in entity.query
AuditEvent.entity with role.code=24
query:
The query parameters must be serialized to a UTF-8 encoded string (preferably as JSON) before they are encoded as base64 encoded bytes, as required per the data type of the query
property
Both search parameters in HTTP query strings and parameters in the body of POST-style searches must be included in the AuditEvent
Note: CPR numbers must never be present anywhere in AuditEvents. If a search parameter contains a CPR number then it should be masked (in an obvious way, e.g. replaced by 'xxxxxxxxxx').
(Privileged administrators can find the actual CPR number used in the search operation in the request/response log, should it prove necessary for an investigation)Example: An HTTP
POST
call to https://example.com/Patient/_search with bodyidentifier=urn:oid:1.2.208.176.1.2|2603200001
could result in an AuditEvent.entity.query={"identifier": "urn:oid:1.2.208.176.1.2|xxxxxxxxxx"}
bundle: The bundle ID should be added to the value of the identifier of the entity.
Example entity: {"identifier":{"value":"ce6d8410-c67f-42d5-8de3-9ebb2a1aef65"},"type":{"system":"http://hl7.org/fhir/security-source-type","code":"4","display":"Application Server"},"role":{"system":"http://hl7.org/fhir/object-role","code":"24","display":"Query"},"description":"search entity"}
purposeOfEvent (optional)
Any purposeOfEvent codings present on the AuditEvent will be added to the audit log entry
A special coding can be added to prevent the AuditEvent from being sent to external audit services (e.g. MinLog2)
system="http://ehealth.sundhed.dk/fhir/PurposeOfUse"
code="INTERNAL_AUDIT_ONLY"
An example is displayed below.
{
"resourceType":"AuditEvent",
"type":{
"system":"http://hl7.org/fhir/audit-event-type",
"code":"rest",
"display":"RESTful Operation"
},
"subtype":[
{
"system":"http://hl7.org/fhir/restful-interaction",
"code":"$createPatient"
}
],
"action":"E",
"recorded":"2019-12-04T11:59:28.646+00:00",
"outcome":"0",
"outcomeDesc":"Patient",
"agent":[
{
"extension":[
{
"url":"http://ehealth.sundhed.dk/fhir/StructureDefinition/ehealth-responsibleOrganization",
"valueReference":{
"reference":"https://organization.inttest.ehealth.sundhed.dk/fhir/Organization/10357"
}
}
],
"userId":{
"system":"http://ehealth.sundhed.dk",
"value":"https://organization.inttest.ehealth.sundhed.dk/fhir/Practitioner/143473"
},
"requestor":true
},
{
"purposeOfUse": [
{
"coding": [
{
"system": "agent1 system 1",
"code": "agent1 code 1"
}
],
"text": "a1-c1-text"
}
]
}
],
"source":{
"identifier":{
"system":"http://ehealth.sundhed.dk"
"value":"https://patient.exttest.ehealth.sundhed.dk"
},
"type":[
{
"system":"http://hl7.org/fhir/security-source-type",
"code":"4"
}
]
},
"entity":[
{
"reference":{
"reference":"https://patient.inttest.ehealth.sundhed.dk/fhir/Patient/852"
},
"role":{
"system":"http://hl7.org/fhir/object-role",
"code":"1"
}
},
{
"identifier":{
"system":"http://ehealth.sundhed.dk",
"value":"6b507ee2d716780372c255df69ece653"
},
"type":{
"system":"http://hl7.org/fhir/security-source-type",
"code":"2",
"display":"Data Interface"
},
"role":{
"system":"http://hl7.org/fhir/object-role",
"code":"21",
"display":"Job Stream"
}
}
],
"purposeOfEvent": [
{
"coding": [
{
"system": "http://ehealth.sundhed.dk/fhir/PurposeOfUse",
"code": "INTERNAL_AUDIT_ONLY"
}
]
}
],
"headers":[
]
}
Splunk processing
A central service handles the AuditEvent entities and writes them to Splunk to allow for search, statistics, report generation etc.
https://docs.ehealth.sundhed.dk/latest-released/ig/StructureDefinition-ehealth-auditevent.html#splunk-processing describes the AuditEvent elements mapping to attributes in logging. E.g. FHIR AuditEvent.outcomeDesc
is mapped to log attribute actionResource
.
However, only a simplified version of the AuditEvent is logged, containing the following attributes:
Splunk Event Attribute | FHIR AuditEvent source |
---|---|
actionOutcome | outcome |
actionResource | outcomeDesc |
actionType | action |
entities | entity.reference (where role.code<>21) |
issuerId | agent.userId.value |
organizationId | agent.extension(ehealth-responsibleOrganization).valueReference.reference |
patientId | entity.reference (where role.code=1) |
subtype | subtype.code |
time | recorded |
traceId | entity.identifier.value (where role.code=21 and type.code=2) |
queryParameters | entity.query (where role.code=24) |
bundleId | entity.identifier.value (where role.code=24) |
source | source.identifier.value |
purposeOfEvent | purposeOfEvent |
agents | agent with purposeOfUse assigned |
An example:
{
"traceId":"nPPwpZQqDdCIHCcNXjKB",
"issuerId":"http://organization.fut.trifork.com/fhir/Practitioner/35205",
"organizationId":"http://organization.fut.trifork.com/fhir/Organization/10357",
"patientId":"http://patient.fut.trifork.com/fhir/Patient/286",
"time":"2019-12-04 11:01:59.601",
"actionType":"R",
"actionResource":"Patient",
"actionOutcome":"0",
"subtype":"_search",
"entities":[
"http://patient.fut.trifork.com/fhir/Patient/286"
],
"purposeOfEvent": [
"http://ehealth.sundhed.dk/fhir/PurposeOfUse|INTERNAL_AUDIT_ONLY"
],
"agents": [
{
"purposeOfUse": [
"agent1 system 1|agent1 code 1"
],
"purposeOfUseText": [
"a1-c1-text"
]
}
],
"type": "audit"
}
Once indexed in Splunk, entities can be queried wrt. generating reports, statistics etc. For example, to view the number of audit entries for the different practitioners on for example the Internal Test Environment (INTTEST), use this Splunk query (notice: only available if additional Splunk access is provided due to the sensitive nature of audit data):
index="inttest_k8s_ehealth-audit_audit" kubernetes_deployment_name="auditlog-consumer" sourcetype=kubernetes_logs kubernetes_container_name="auditlog-consumer"
| top issuerId
Usage log
It is possible with the right permissions to Splunk indexes to make reports covering the usage of the system. These reports will be made in Splunk. A report could show the number of patients accessed in a period ordered by organizationid (see image below).
The organizationId could be resolved separately or by giving Splunk a translation map between the number and a more user-friendly organization name. This is only one way to process the data - in Splunk, it is possible to create many views and statistics based on these data.
When a template of a fulfilling report has been generated an automatic report generation could be scheduled for instance monthly. After creating the report it could be sent by email.