In a gear reduction system, the input gear typically has a smaller number of teeth than the output gear, which has a larger number of teeth. This means that for each revolution of the input gear, the output gear will rotate fewer times, resulting in a slower speed but with increased torque.
Without gear reduction, the motor takes the load. To avoid overheat you cannot use a very large prop. Also, the boat can hardly pick up speed initially. AND, you cannot use a really strong battery. Something under 30C would be good. Anything stronger than that can cook the motor in minutes.
The good thing is that you have a simpler structure to work with.
This boat uses a RS550S 14448, which works quite well under 3S.
This vid shows a 65cm boat running on a direct drive 550 motor and 3S.
How do you tell if the motor is almost overloaded? Check the crimp bullet connectors of the motor. If the protective plastic around it is melting, chance is that the motor is failing soon.

