How to set up alert notifications about integration issues for both QA and Production environments

Introduction

Notifications in ONEiO are used to inform the customer about issues in their integration. Either temporary or permanent issues can be the cause and can be set independently from each other. Furthermore, the configuration settings of the environment and notifications can be selected and adjusted accordingly.

Setup

The notification settings can be found in the subscription settings and can be accessed by the subscription owner only.

Step-by-Step

1. Add a new notification recipient by pressing the "Add Notification" button on the right side of the notification screen.
2. Fill in the recipient email address for the mailbox you wish to receive your notifications.
3. Choose the notification settings for the environment in question.
   1. Production Environment
   2. QA Environment
4. Choose the type of Notification you wish to receive per environment
   1. Warnings
   2. Errors
   3. Queues
5. Save your selection by pressing the "Confirm" button

After saving your selection, you are presented with the following UI:

The indicator (pop-up) behind the email address shows the current status of the verification process.

• Green: The email address has been verified.
• Yellow: The email address is pending verification. This will expire in 7 days.
• Red: The email verification has expired. You can resend the verification link in the UI be clicking the edit icon  on the right side and pressing "Resend"

Notification Types

ONEiO has two types of notifications to inform customers about integration issues - warnings and errors.

Warnings

• These are defined as temporary issues that can be solved by user intervention, also known as Recoverable Errors.
   
• These include, for example:
  • Endpoint timeouts
  • Authentication errors
  • General connection problems.

A Warning is also triggered if a message has been in "Pending" state for longer the 5 mins. Only one alert is sent for this warning, even if the message is being retried for longer periods.

Errors

These are defined as problems that can't be resolved by user intervention.

These include, but are not limited to:

• Unrecoverable HTTP error codes, defined in the endpoint configuration.
• Invalid response from the target system, defined by the HTTP success codes in the endpoint configuration.
• Invalid response from the target system, defined by the Response success value. These are only used when the target system cannot produce an HTTP success code.

Here is an example from ServiceNow for Unrecoverable HTTP error codes.

And an example for HTTP Success codes, also from ServiceNow.

Queues

Message queuing is used to improve reliability and prevent message loss when temporary issues occur during message processing or delivery.

• Inbound queue: An inbound queue forms when the platform receives messages faster than they can be processed. This typically happens during traffic spikes, large batch imports, or temporary slowdowns in downstream processing systems. Messages are stored in the queue and processed in order as they arrive.
   
• Outbound queue: An outbound queue forms when the platform is unable to successfully deliver a message to an external server or destination. The platform retries the failed message until delivery succeeds or the target system responds with an unrecoverable error. 
  • To preserve message order and consistency, subsequent outbound messages may remain queued until the failing message is resolved.
  • An example is when a server fails to authenticate the integration user. If the error returned (i.e., 401) is not part of the endpoint's unrecoverable errors, we will keep retrying it indefinitely until the target system responds to us with a valid status code.

1.  Trigger when a queue has existed longer than a defined threshold period and
2. Re-send notifications at a specified interval for as long as the queue remains active.

This allows administrators to detect persistent queue issues and receive repeated reminders until the condition is resolved.

In the example below, I configured the queue alert to trigger when messages start to queue for 30 min, and if the queue is not cleared by itself, I'll keep receiving notifications about the queue every 60 min. 

In this case, I configured it for inbound or outbound queues, but you can set different values for inbound and outbound queue alerts.

Alerts

All Alert notifications will be sent from the same email address and follow the same structure. This makes it possible to consume alert notifications and process them further if needed.