I was thinking.....
Have you ever thought of adapting an ER16 collet assembly to thread into your swivel mount. If you did something like that, you could then turn almost anything down to 3/8" or less with a 1/2" shoulder. Then modify the collet nut with a ring so you don't need a wrench to...