Greenplum SSH connection issue due to MaxStartups in /etc/ssh/sshd_config is too small

Greenplum is MPP architecture, and sometimes it utilizes ssh sessions to complete some tasks.
For large clusters, MaxStartups in /etc/ssh/sshd_config needs to be increased to large enough.

What is MaxStartups?

Specifies the maximum number of concurrent unauthenticated connections to the sshd daemon. 
Additional connections will be dropped until authentication succeeds or the LoginGraceTime expires for a connection.
The default is 10.

Under which situations, Greenplum may use ssh sessions?

• Utility tools, for example, gpexpand, gpinitstandby, gpinitsystem, gpstart, etc.
• Web external table.
• Could be more which I do not remember...

Symptoms:

1. Utility tools may error out:

ssh_exchange_identification: Connection closed by remote host

2. Below writable external table may error out:

create writable external web table test ( col1  int ) 
execute 'ssh mdw "mkdir -p ''/tmp/test''; cat > /tmp/test/test_$GP_SEGMENT_ID"' 
format 'TEXT' ( delimiter as '|'  null as E'\\N'  escape as E'\\') ; 

insert into test select generate_series(1,3); 
ERROR:  external table test command ended with SHELL TERMINATED by signal UNRECOGNIZED (127)  
(seg26 xxx.xxx.xxx.xxx:40002 pid=23091) 
DETAIL:  Command: execute:ssh mdw "mkdir -p '/tmp/test'; cat > /tmp/test/test_$GP_SEGMENT_ID"

Solution:

One best practice is to increase the MaxStartups to a value which is larger than the total segments count(primary+mirror).

1. Increase MaxStartups in /etc/ssh/sshd_config on all servers.

2. Restart sshd on all servers.

/etc/init.d/sshd restart