Request Time Out / Sessions Stalling through IPTABLE (DNAT)

tooreply

Scenario:

Customer recently Migrated Clustered HANA DB Servers to Azure Cloud Platform but these are Physical Servers on Azure (Offering: Azure HLI). Usually these HLIs (HANA DB Servers) in Azure cannot be accessible directly, even not from Azure VNETs. In other Words, HLIs cannot access Servers hosted on other VNET or on Customer On-Prem Datacentres and vice-versa. So, customer has provisioned express route between HLIs and one of the Prod VNET. So, any Servers or VM's can access these HLIs. But other VNETs (like DEV, TEST) or from Customer Locations, these HLIs are not accessible.
 
So, to overcome the above problem, we have provisioned SLES 12 VM on Prod VNET and made it as IPTABLE Server for NAT Translations to solve two type of issues
 
a) When HLI Servers needs to access APPs/DBs (here, Hadoop Server).
b) When On-Prem Datacentre Servers needs to access HLIs.
 
So, in this case, we have assigned Natted IPs for On-Prem Servers and also HLIs as follows. Refer the Diagram attached
 
Original Subnet – 10.10.x.x

HADNA001 - 192.168.1.2
HADNA002 - 192.168.1.3
HADNA003 - 192.168.1.4
 
Original Subnet – 172.168.x.x

HANANA001 – 192.168.1.5
HANANA002 – 192.168.1.6
HANANA003 – 192.168.1.7
 
Problem:
**
**Case 1: When SAP Support team of customer, started pulling data from Hadoop spark controller, they are seeing time out requests. As per them, they are seeing this only in case of High loads. For normal loads this is not happening.
Stages of Pulling Data:

1) Using HANA Studio which is installed on users Laptop.
2) Connect to any one of the HLI Server (In this case, HANANA001)
3) Configure loads and trigger loads from Hadoop Controller (HADNA001) via HLI (HANANA001)

And the request timeout is dynamic, it may happen 1 – 10 times in 4 hours of loads.
 
Case 2: It is observed that when we try to copy data using rsync with multiple parallel sessions from on-prem servers to HLI via iptable, after sometime sessions are getting stalled. This is not happening in case of rsync with 5 Sessions. If it is more than 5 Sessions, sessions are getting stalled.
The data we are trying to copy is 2 TB having 5 GB files each. This is the reason we are trying with multiple parallel sessions, assumption is with one go more 5 GB files will be copied.

We have tried capturing tcpdump and analysed the logs, we have seens only [F] and [R] flags which is not giving much information. And also there are other networking devices in between on-prem and HLI which are not shown in attached diagram and also we have no clue on those.

This is issue is not there with other iptable servers on different environment or VNET. Till, now customer has reported us few errors they can see in HANA Studio

1) SQL Error, 403 Request Time out
2) Unable to Create Virtual Table, Request Time out.