If it's anything like an average drill press,it's not really designed for high precision.
You can improve your lot substantially by getting a pin collet chuck.These usually come with 3 or 4 collets.Fit this in your larger chuck,and have at it.Peck at the hole,rather than trying to drill in one go.Lubrication may also help.Don't forget to set the speed at the highest possible setting.It still won't be fast enough,but gives you a better than even chance.
The next step up is a "dremel"in a stand.If you're really rich,a Proxxon drill press is the ultimate hobbyists tool.Mucho Dinero,though.
BTW.,you do make a punch mark first,don't you?Trying to drill a small hole without it is a shortcut to failure,e.g. broken drill bit.