Simply the relatively wealthy could afford perfumes and, in any event, we were not easily.
In case you're making glycerin soap likely to principle applies, once you remove the glycerin using the heat source, add your fragrance and whatever coloring, if any, you want.