As dovetail cutters are normally straight toothed they cut with a very harsh intermittent action and hence calculating a speed/feed combination based on normal helical type milling cutters can give unsuitable results.For this reason I always use my eyes and ears to determine the speeds and feeds for cutters.As you have already mentioned a lot of vibration It would seem that although the finish is good the cutter speed/feed is probably too high for your setup.Calculating a cutting speed for any form of tool is only a guide for people who require a fast removal of metal as if in a production shop,which normally means that large rigid machines with good fixturing are being used.The life of the tooling is also expected to be much reduced from its maximum.Therefore I always try to make any machining I do to be a be a steady,pleasant,easy on the nerves operation that does not over stress either the parts being cut or the machine tool doing the cutting.So to sum up,if in doubt start with a slow spindle speed and just take a small cut and see what happens.If it all goes well then just up it a little,your machine will tell you when to ease up by complaining(a bit like wives really).Most cutting operations will not suffer by being done at a slower rate than is possible and the tool will still be good to use again after you are done.
I also find that on some dovetail cutters that they have too many teeth on them which can cause chatter and vibration.This I cure by grinding off alternate teeth on the cutter to let the tool cut more smoothly.A bit drastic I know but it does work,but only do it as a last resort.
The above lecture is only the procedure that I use and is by no means to be thought of as the correct method as I am sure that other people have better methods,if it is of any use to you then good luck.
best regards Steve C.
