Thanks for the replies! I'm quite sure I had the polarity correct, I'm thinking it was pulling too much power at 5 volts, should've put it at 7 or 8 I'm thinking. They were cheap Amazon buck converters, so I'm not too worried about them. The step down to USB output is working good.
Zeb, thanks...