Impact of Gateway Hub on cache server cluster performance

Hi

We are observing a performance issue when we implement Gateway Hub for async-write-behind purpose on cache server cluster.

The test which takes around 7 - 8 seconds with Gateway Hub implemented, takes 600 - 700 milliseconds after removing Hub from the cluster.

Below is the description of our test environment.

Gemfire cluster consist of two cache servers, running on separate VMs, and is managed by a locator process. Gateway hub is implemented to send events to MySQL database asynchronously.

Gemfire clients, initialized in a web application, connect to same cluster using locator.

These clients, two in number, are configured to run at same time and put some data in same region. These clients run continuously for 10 minutes and invoke multiple threads which put data/update data in the region simultaneously.

What we have observed is that when Hub is implemented in the cluster, one of the cache server in cluster doesn't work efficiently and continuously logs a warning message like ' DistributedCacheOperation$CacheOperationReplyProcessor: waiting for 1 replies from <other cache server> whose membership list is ......' and

'Wait for reply completed’. This particular server is the one which have secondary hub. CPU usage of this cache server lies to 5-10% while usage of other cache server goes to 80-90%.

After removing Hub from cluster, both cache servers share the work equally and both use about 80-90 % of CPU.

Please suggest if there is something wrong in the implementation.

If this is the general behavior of Gateway Hub then is there any other alternative for backing the data in Database without affecting performance?

Note: The timings shown above represent the results of load testing of web applications by Jmeter. Each application user connects to cache server cluster and puts some data in the region. These are the average time taken by a user when 500 concurrent users are using the application.

Thanks

Mahesh

-------------------------------------------------------------------------------------------------------

Deployments: Gemfire662, Java6-64bit, Windows Server2008 64bit

Hub Settings on both cache servers (cache.xml):

<gateway-hub id="TimeBookingsDBWriter" port="-1"

                                startup-policy="primary">

                                <gateway id="TimeBookingsDBWriter">

                                                <gateway-listener>

                                                                <class-name>listeners.TB_GatewayListener_Hib</class-name>

                                                </gateway-listener>

                                                <gateway-queue batch-size="100" batch-time-interval="1000" batch-conflation="true" />

                                </gateway>

</gateway-hub>

<region name="ProjletTask">

                                <region-attributes scope="distributed-ack"

                                                data-policy="replicate"

                                                enable-subscription-conflation="true" hub-id="TimeBookingsDBWriter" enable-gateway="true">

                                </region-attributes>

                                <index name="projletIDIndex">

                                                <functional from-clause="/ProjletTask pt" expression="pt.projletID" />

                                </index>

                                <index name="quickTaskIDIndex">

                                                <functional from-clause="/ProjletTask pt" expression="pt.quickTaskID" />

                                </index>

</region>

Client Side settings (client-cache.xml):

<pool name="rende-pool"  min-connections="250" max-connections="500" read-timeout="600000" socket-buffer-size="64000">

    <locatorr host="107.21.164.4" port="10335"/>

</pool>

<region name="ProjletTask" refid="PROXY">

</region>

Data Insert Method:

        Region.putAll()