I'd take less of a cut say 2.5mm deep per pass so the cutter will not be removing as much material and therefore won't deflect as much.
Also cut the pocket say 0.2mm under size then go back and cut to finished size which could be done in 5mm steps.
This one is also 10mm cutter, 52mm deep. I bored out first then just went round and round teh edges size of cuts as above. Dab of parafin and a sharp cutter
And what the inside finish came up like