I think it affects the number of citizens you have, but very slightly, I know I went down a couple citizens when I went from 5 stars to 4 1/2 stars once, but if this is the same pattern, then it's really nothing to worry about as you'll lose no more than 15-20 citizens between 5 stars and 1 star.