I have an 8 inch rotary table. It is so heavy I can barely pick it up. Go with the 4 inch. I have the Sherline 4 inch rotary table. It is extremely well made and is a pleasure to use.
Using the vfd to force a stop can be problematic. It will definitely stop the motor quickly, but that can cause threaded chuck to unthread or other locking/securing arrangements to fail. Be careful.