You are here: Running solutions in production > The Monitor > Large messages

Large Messages - Introduction

The MD Link Monitor has some special behaviour in order to efficiently handle large messages. MD Link considers a message large if it either 1) is larger than 100,000 characters, or 2) contains base64 data larger than 1,000 characters.

In the messages typical in health care integration - whether they are HL7, HTTP / JSON, or otherwise - base64 is the de facto standard for including binary data in messages. This binary data could be an image, a sound file, or any other file. An HL7 message without such data is typically around 1 kilobyte in size. The same message with image data added could easily be 5 megabytes in size. This represents a 5000-fold increase.

The behaviour described on this page pertains to the Monitor only - that is, the Monitor's presentation of the messages to you. The message flow in the MD Link service is unaffected.

Viewing

"Viewing" refers to the act of selecting a message in the Monitor so that you can see the contents of that message. This could be in the Output Messages tab or the Input Queue tab. The ways in which MD Link ensures high-performance message viewing are:

If the message you selected is larger than 100,000 characters, then the Monitor will only show you the first 100,000 characters of it, along with a warning message telling you that you aren't seeing the entire message.
If a task outputs multiple messages per execution, and there are more than 1000 of them, then the Monitor will only show you the first 1000.

In either case, if you want to see the entire message or list of messages, then right-click on it and choose "Download", then open the resulting file in a text editor.

Searching

Searching is different from viewing in that viewing involves only a single message, while searching involves many messages. With viewing, the message in question is the one that you selected. With searching, the messages in question are all of the matching messages in the database.

The ways in which MD Link ensures high-performance searching are:

MD Link will only search the first 100,000 characters of each message.
MD Link will not search any part of the message that is base64 data and is over 1000 characters long.
- This does not mean that you can search the first 1000 characters of a base64 string in a message. It means that if the base64 string in the message is 999 characters long, then you can search all of it, and if it's 1000 characters long, then you can search none of it.
- This minimum length of 1000 characters ensures that regular strings of letters and numbers, which might otherwise appear to be base64, will be unaffected, so that you can search them as usual.
- If a single base64-encoded file is split across multiple HL7 segments, then MD Link examines each of these segments separately for the 1000-character limit.
  - This applies to both pipe-delimited HL7 and HL7 that is parsed into XML.
  - This has performance ramifications. MD Link does not know that all of the segments are meant to be interpreted as a single base64 string. For example, if a typical message of yours contains a base64 string that is 1,000,000 characters (i.e. 1 megabyte) long, and it's split across HL7 segments that are each 500 characters long, this will result in bad search performance. Furthermore, your search might delay message flow in other interfaces for several minutes while the service responds to your search. Therefore if your messages contain base64 data split across multiple segments, the number of characters per segment has significant consequences.
- If a single base64-encoded file is split across multiple lines, is not part of an HL7 message, and takes up all of the XML element that it's in, then MD Link will examine all of these lines together for the 1000-character limit.
  - In this case, MD Link knows that all of the lines are meant to be interpreted as a single base64 string. For example:
    <responseBody>neCtsfjLvJyctJuFIGmiqctKdpqLaaJoplaRfAjqPHVerN+NSMfqhTBJBnikizHtMscYFQPGEWAI
    CFbczdTXHGxEpnu/IJTPbjnyWqGLBGBjJAHAQQzkXnof+kTlwdfaUoWuIrWMOdUeEibtZpsjzMUW
    krndYTqSzTCLWZaioYUDiOHBh/sGKwwZFrwIiGoVkopkjhYTYdQefEPPybXynzvdhedqBKVq+hqr
    ...
    </responseBody>
    where the "..." section contains enough characters that the whole base64 string exceeds the 1000-character minimum.

Assumptions

MD Link's behaviour regarding large messages was built on these assumptions:

The most interesting human-readable data will be in the first 100,000 characters of a message. Users will only rarely want to see past this point.
No user will need to search data after the first 100,000 characters of a message. The user might need to view it , but they will never need to search it.
All large data is machine-readable, not human-readable.
All large data - base64 or otherwise - will be at the end of the message. There will be no human-readable data after it.
No user will need to search any base64 data that is more than 1000 characters long.
- This applies both to data that is base64, and data that only resembles base64.

If your messages don't meet these assumptions, contact MDI Solutions for information on how to customize your installation accordingly.