If you fly into controlled airspace without an endorsement weather you are private of Sport, there is nothing wrong with that until something happens. You can rob a bank and everything looks ok, until you get caught. If you drop from Private to Sport, you do not retain the privileges of Private until you go back to Private.