You don't.
By definition, if you want to sort photos manually, you don't want a Smart Album -- because Smart Albums aren't albums at all, they are searches based on criteria you set up, and so they will add or delete photos automatically based on that criteria. What you want for albums you wish to sort manually is a conventional Album.