Default max concurrent search req. numNodes * 5 (#31171)

We moved to 1 shard by default which caused some issues in how many
concurrent shard requests we allow by default. For instance searching
a 5 shard index on a single node will now be executed serially per shard
while we want these cases to have a good concurrency out of the box. This
change moves to `numNodes * 5` which corresponds to the default we used to 
have in the previous version.

Relates to #30783
Closes #30994

Author: s1monw

server/src/main/java/org/elasticsearch/action/search/TransportSearchAction.java

@@ -346,7 +346,8 @@ private void executeSearch(SearchTask task, SearchTimeProvider timeProvider, Sea
              * it sane. A single search request that fans out to lots of shards should not hit a cluster too hard while 256 is already a
              * lot.
              */
-            searchRequest.setMaxConcurrentShardRequests(Math.min(256, nodeCount));
+            // we use nodeCount * 5 as we used to default this to the default number of shard which used to be 5.
+            searchRequest.setMaxConcurrentShardRequests(Math.min(256, nodeCount * 5));
         }
         boolean preFilterSearchShards = shouldPreFilterSearchShards(searchRequest, shardIterators);
         searchAsyncAction(task, searchRequest, shardIterators, timeProvider, connectionLookup, clusterState.version(),