Dale says:

The numbers refer to the angle of the diverging tracks in the crossover, the same way they do for simple turnouts. The lower number, the higher the diverging angle. So a #4 crossover will have sharper angles which would make the whole crossover shorter. The drawback is that long locos and rolling stock may not work well, or look as good, as with the sharper angles. A #4 would be okay with most freight cars but passenger cars will likely be better with a #6 or larger crossover.

A #4 turnout has a frog with a bit over a 14 degree angle and a diverging radius of about 15 inches. So if your equipment requires a minimum of 18 -24 inch radii, a #4 would cause problems. On the other hand, a #6 would have a frog angle of just over 9 degrees and a diverging radius of about 43 degrees,

For further information I would suggest you look at
http://www.pcrnmra.org/pcr/clinics/Kolm-TurnoutsWhatYouNeedtoKnow-PCR2008-handout.pdf

phil johnson says:

The smaller the number the tighter the switch. Thus on a #4 crossover small locomotives (4 axle diesels, short wheel base steam) short cars 40-45′ or less. #5 switches allow for larger rolling stock (6 axle diesels, most medium steam depending on wheel base, including brass) cars 60-75′ (85 with talgo trucks. #6 Almost anything except maybe brass 4-12-2 unless equipped with blind drivers