I am lucky in that I have a precision drilling machine, it only goes up to 10K RPM, and I have drilled many 0.01mm (0.0004") holes with no problems at all.
This is one I started to make that was to get me up in the 80K to 90K speed range, but never finished it as I found my larger drilling machine (above) would cope very easily with small carbide drills as it had such a rigid spindle.
In theory, this size of drill requires around 90+K rpm, but I normally drill at around 4K to 5K and I have broken very few of the tungsten drills. I buy resharpened ones off ebay, and if taken steady, will require no previous centring on all materials. The middle one, sold as PCB board drills.
I have very fine feed on my machine, and I think that is your main problem, you are trying to force the drill down into the hole without actually 'feeling' your way through.
As previously suggested, get one of these that allow very fine feeding indeed as you have a lot of 'feel' with them, which your main feed won't have.
Hope this helps