Advice please on replacement brushless.
There are fittings for this size drive shaft freely available from most model suppliers.
Your ESC is designed to run faster in one direction as per the specs so you need to adjust your Tx to give the max speed on your chosen stick setting , then adjust the motor leads to give correct rotation for the prop.
The motor cut out may be due to thermal rather than battery problem. This ESC does need water cooling to operate at the specified voltage and current. ideally you should measure the current draw with the prop in the water, it will be higher than when the boat is moving but will give a good indication of the worst (most current) draw. if your prop is too big or too coarse a pitch the motor could be drawing too much current and exceeding the wattage (volts x amps) rating of the motor and/or the ESC. Suitable meters are available cheaply from Giant Cod.
Others appear to have struggled with the ESC settings and lack of info. There is a black box tool for setting the parameters which may help.
How long were you running the boat on the water before it cut out? I usually time my runs and make sure I am near the shore when the usual run time is expiring.
Looks like you are nearly there and I'm sure you are enjoying the performance with the brushless setup.
Dave
Dave