The other day I turned a stub arbor in one of my mill's collets, it was about 2" long, and I got a bit of a taper on it, a couple thousandths. I was machining it from a steel bolt.
Just for the sake of my curiosity, would you measure your rollers and tell me if there's any taper on them?