I'd like to offer a few suggestions on the subject of this thread as an FCC licensed professional with 35 years of experience in the business...
It has been mentioned earlier in the thread- a good antenna properly installed makes a huge difference. Gain is also important, but remember that in an antenna, as the gain is increased in one portion of the sphere of radiation, signals are decreased in others. High-gain marine antennas radiate in an omni-directional pattern with a very narrow vertical lobe close to the horizon. This means that if your boat pitches and rolls significantly, that area of radiation (and reception) will be moved away from the horizon causing signal flutter and fading. Lower gain antennas with broader radiation (and reception) lobes will minimize this phenomonon. They key is to use the highest gain antenna that your platform will support, and get it as high as is practical.
I can't agree strongly enough with the discussion regarding connectors and connector installation. In my years of checking, most of the connectors installed by users just weren't installed properly. Installation of these connectors looks simple enough, but it isn't. There are a lot of things that can be gotchas to someone inexperienced with the procedure. If you don't do it right, you won't have a reliable radio system. If you are going to try to do the connector job yourself, find an expert to show you the proper way to install the connector. If you don't feel comfortable installing it yourself, have it installed by a professional. Whatever you do, avoid crimp-on or screw-on connectors like the plague. They are unreliable junk.
Once that connector has been installed, it must be verified. A good thru-line power meter can be a help here. Good ones are very expensive. The accuracy of a power meter is rated at the range's full scale value. That means that trying to make accurate readings at the bottom of a scale is problematic. A decent wattmeter will have removeable slugs that allow the range to be changed for taking forward and reflected power readings. Once that connector is installed, verify that the forward and reflected power is within acceptable limits. Most radio shops use high-quality power meters to make these checks. I prefer an additional check with a time-domain reflectometer. This instrument uses traveling pulses to analyze the entire connector/feedline/antenna system. Defects in connector installation, coaxial cable anomalies, and antenna issues can be readily discovered. They are very expensive, but the wealth of information made available is priceless.
And to re-iterate a previous post, there are not separate antenna elements for reception and transmission. A marine VHF antenna is a single radiator in a weatherproof radome.
Coiling coaxial cable does not produce a "choke effect" to the radio frequency energy traveling inside the cable. Coils of coax are invisible to the RF traveling down the cable. Because of the skin-effect, high frequency radio frequency energy travels only on the surface of a conductor. A coaxial cable actually has three distinct surfaces. The outer surface of the center conductor, the inner surface of the shield, and the outer surface of the shield. Improper connector installation and cable defects can compromise this inner-to-outer shield isolation and adversely affect radio system performance. A coiled cable will have "choke effects" on currents flowing on the outer surface of the shield. This is not harmful and is actually beneficial.
Losses because of excessive coaxial cable length do occur, and while cable runs should be designed to be as short as possible, don't get too obsessive about the cable lengths on a boat. An extra few feet of cable isn't going to make a noticable difference. An extra 30 feet of cable will. Always make sure to allow some excess cable in the area of the radio to allow for future connector replacement. Don't cut it to the exact length. If you have a connector fail later or get a slightly different radio, you'll be in trouble.
Finally, let's not forget the radio itself. Is it really making rated power? And just as importantly, is it operating on frequency, or has it drifted a bit? Is the modulation deviation correct, or is it low making you hard to hear or high causing distortion? Is the receiver sensitivity within specifications? If you run into a situation where you can either hear well and can't transmit, or can transmit well and can't hear, you most likely have a radio issue. Get it checked. These checks require a communications service analyzer. If you've decided to get a new radio, have it checked too, prior to installation. A few are bad right out of the box.
And lastly, check the 12 volt power at the connector on the radio. Make sure that under full-power transmit there isn't an excessive voltage sag. Excessive voltage drop in the power cable can cause all sorts of strange radio behavior. Make sure it's within specification.
You might have to depend on your radio to save your life or the life of a fellow boater someday. Try to keep that in mind as you work on your radio installation.
If you've got any questions or need any free advice, let me know.
Rick