Not at all.
It does depend if you enter a PhD only program with a BA and get the MA "consolation prize," or if you enter a program with a terminal MA program (that might also have a PhD program).
That being said, the correct route for many students, especially those who had a difficult or disorienting BA experience, is to go get into a really good terminal MA program to improve their GPA, get better letters, enhance their tool box, and therefore get into a better PhD program. In the process of getting the MA, it is also likely that you will refine your interests and grow to understand the academic enterprise as well as understand the kind of program in which you will achieve success.
If you were in a PhD only program, and genuinely figured out that your interests would be better fulfilled elsewhere after a couple of years, then, if your supporting faculty are any good at this, they should be more than willing to help you get into a program that better suits your interests/work with people who can help you achieve what you want to achiever.