Processing large payloads with the esb script mediator iteratively

Overview

WSO2 ESB uses Rhino engine to execute JavaScripts. Rhino engine converts the script to a method inside a Java class. Therefore, when processing large JSON data volumes, the code length must be less than 65536 characters, since the Script mediator converts the payload into a Java object. However, you can use the following alternative options to process large JSON data volumes.

The script mediator which is used in ESB is powered by the Rhino engine. Therefore, when processing large JSON data volumes, the code length must be less than 65536 characters which is a limitation in the script mediator being used in the esb versions less than 5.0.0. In ESB 5.0.0 there is a higher capability to process larger payloads using script mediator. In order to process such large payloads we can follow the below two approaches.

1. Replace the javascript tranformation logic using java code by writing a custom mediator. [1]
2. Break down the large payload and execute them as sections using the script mediator.

This blog post will be demonstrating how to carry out the use case #2 using wso2 esb.

Let's get started

The approach we would be using is to iterate over the large response and pass each section through the script mediator. After which we will be aggregating the tranformed sections to display the modified and transformed response.

1) Let's create a mock service which returns the response mentioned in this text file. I have used the online tool http://www.mocky.io/ to generate this response. The invocation URL for my mock response is http://www.mocky.io/v2/57a881a3110000b6171d4546

2) Then we need to create the In sequence. In the ESB in the Service bus section , select the Sequences option and add a new sequence. This is the source view of the in sequence we will be using which simply has a endpoint invocation. This is the in sequence configuration and you can also download it from HERE

    
<sequence name="sendSequence" xmlns="http://ws.apache.org/ns/synapse">
    <log level="custom">
        <property name="test" value="Execution Started"/>
    </log>
    <send>
        <endpoint>
            <http method="GET" uri-template="http://www.mocky.io/v2/57a881a3110000b6171d4546"/>
        </endpoint>
    </send>
</sequence>

3) Next we need to create the out sequence which does the needed transformation. I have listed what happens within the sequence to get a better understanding.

• I have used the response generated from the above invocation here which has an array of json objects. Since the objects inside the array are repititive we can split it using the iterate mediator. (In this sequence the splitting is done by the ARR_ITEM expression)

• After splitting it using the iterate mediator we can manipulate each JSON object by passing it through the script mediator.

• At the end we are placing an aggregate mediator which will aggregate all the modified JSON elements which went through the iterate mediator and display the response.

• We will be storing the aggregated response inside the rootElement property as json objects. Follow is the sequence which executes this logic. This is the outsequence of our API.

Simillar to above you need to add another sequence which we would be using as the out sequence. This is the sequence source view corresponding to the out sequence described above. You can also download the sequence HERE

       
<sequence xmlns="http://ws.apache.org/ns/synapse" name="iterOutTest">
   <property name="rootElement" scope="default">
      <jsonObject xmlns=""/>
   </property>
   <iterate xmlns:ns="http://org.apache.synapse/xsd"
            continueParent="true"
            expression="//ARR_ITEM">
      <target>
         <sequence>
            <log level="custom">
               <property name="iterator" value="in the iterate"/>
            </log>
            <payloadFactory media-type="json">
               <format>
                $1
            </format>
               <args>
                  <arg evaluator="json" expression="$."/>
               </args>
            </payloadFactory>
            <script language="js">
               mc.setProperty('CONTENT_TYPE', 'application/json');
               var payLoad = mc.getPayloadJSON();
               var catalogItem = payLoad.ARR_ITEM;
               catalogItem.CT.CT_ITEM.ED = "456"; 
               catalogItem.DISCRIPTIONS = "This is the modified description";                
               mc.setPayloadJSON(payLoad);</script>
            <log level="full"/>
            <aggregate>
      <completeCondition>
         <messageCount min="-1" max="-1"/>
      </completeCondition>
      <onComplete xmlns:ns="http://org.apache.synapse/xsd"
                  expression="//jsonObject/ARR_ITEM"
                  enclosingElementProperty="rootElement">
         <log level="full">
            <property name="MSG" value="Aggregated Msg"/>
         </log>
         <respond/>
      </onComplete>
   </aggregate>
         </sequence>
      </target>
   </iterate>
</sequence>

4) After we have created the sequences we need to then add it to the API. Refer the below screenshot. We need to add the above created in and out sequences to the API respectively

Here you can find the xml configuration corresponding to the above API. Also you can download the API from HERE.

<api xmlns="http://ws.apache.org/ns/synapse" name="IterateAggregateAPI" context="/iterate">
   <resource methods="POST" uri-template="/*" inSequence="sendSequence" outSequence="iterateOUT"/>
</api>

5) Using the soap UI or any other rest client you can invoke the API and observe how the response payload is iterated and processed to display the aggregated and tranformed response.

I hope that following this tutorial you were able to understand how large payloads could be split using common elements to process using the script mediator.

References:

[1] https://docs.wso2.com/display/ESB490/Creating+Custom+Mediators
[2] https://docs.wso2.com/display/ESB490/Iterate+Mediator
[3] https://docs.wso2.com/display/ESB490/Aggregate+Mediator
[4] https://docs.wso2.com/display/ESB490/Creating+APIs
[5] https://docs.wso2.com/display/ESB490/Mediation+Sequences