Exactly Once in SAP Hana Cloud Integration

The Issue


In one of our first proof of concepts with SAP Hana Cloud Integration (HCI) we noticed that, when a message failed in HCI, you couldn’t restart the message in HCI and it had to be restarted/resent at the source system.


When we went one step further and added a second receiver to the scenario with the multicast (in parallel mode) operation, we found out that, when the message to the first receiver went successful and the message to the second receiver failed, the message could not be restarted in HCI or the source system. It was not possible to guarantee the delivery to the second receiver.


We have discussed this with SAP and they confirmed that HCI (version 2.14.6) has no Quality of Service (QoS) Exactly Once (EO) out of the box, but it will be part of the HCI roadmap.


The Solution


Together with SAP we developed an EO pattern in HCI with the help of the HCI Data Store. The iFlow consists of four parts:
The Main Integration Process for the normal processing of the incoming message.
A local Integration Process to send the message to the receiver, including error handling in case of an error.
A second Integration Process to retrieve the message from the data store after the occurrence of an error.
Second local Integration Progress to send the message to the receiver and delete the message on success from the Date Store

Part 1: The Main Integration Process

The main integration process has the normal processing operations, like Content Modifiers and Mappings. The last Content Modifier operation will set the dynamic file name (header variable “CamelFileName”) and will save the payload of the message in a property. The last operation is a Process Call to a local integration process. In this case it is a single receiver; in case of multiple receivers you will need a Multicast operation (in serial mode) with each a call to separate local integration process.


Part 2: The first local Integration Process


The first local Integration Process contains a send operation (“Service Call” operation in “Send” mode) and an Exception SubProcess. The Exception Subprocess will catch a potential error during the send step. The Content Modifier will make sure the message that has failed, is put back in the body of the message using the property set by the last Content Modifier operation in the main process. The Write operation (Data Store Operations in “Write” mode) will persist the body of the message to the HCI Data Store and the file name (as it is dynamic we don’t want to lose it) as Entry ID.


Part 3: The Retry Sending failed message


The third part is the “Retry sending failed messages” Integration Process within the same iFlow. This is a timed event that will trigger every 5 minutes a Select operation (Data Store Operations in “Select” mode).


This operation is configured to retrieve 100 message at the time and the checkbox “Delete After Fetch” is unchecked. The Select Operation will add a small envelop to the message:


/messages/message/

The next step, a Router operation, will check if there are any messages retrieved from the data store with in our case the following Xpath expression:

count(/messages/message/OutputXMLReplicationRequest/DocumentType) > 0

If you don’t do this, you can process an empty message.

As we can retrieve a potential 100 messages at the same time from the data store, we need a Splitter operation to split up the messages and process them one by one.

The Content Modifier sets header variable “CamelFileName” for the dynamic file name. Since we used the file name as the Entry ID in the Data Store we can retrieve it with the following Xpath expression:

/messages/message/@id

The last step before sending the message to the receiver via the local Integration Process is a Filter operation to remove the envelop from the message with in our case the following Xpath expression:

//OutputXMLReplicationRequest


Part 4: The second local Integration Process

The message will be resend from a second local Integration Process. If the message has been sent successfully, the message will be deleted from the data store by the Delete operation (Data Store Operations in “Delete” mode). If the message fails again, The Delete operation will not be executed, the message will stay in the Data Store and will be processed again in the next occurrence of the Retry Integration Process.

Finally
We hope this Exactly Once Pattern will help you to build more robust interfaces in SAP Hana Cloud integration.