This can occur when spark.executor.port is set explicitly in the spark job configuration and spark.port.maxRetries is set too low or the default is too low for the number of executors or other port consuming processes running on a node.
When a Spark executor starts running on a Node Manager it will try to bind to a port to enable communication with the Spark driver. To find an unused port number, and executor will first try to bind to the port defined by spark.executor.port. If this fails (due to ports already being in use), it will try spark.executor.port + 1, then spark.executor.port + 2, etc. The executor will keep trying to bind to ports up to spark.executor.port + spark.port.maxRetries. If these attempts all fail the executor will exit.
If too many (spark.yarn.max.executor.failures) executors fail to start due to this, the application attempt will be terminated.