This started as a comment, but was growing into a series of comments, so here it is as an answer:
My definition of safety as a woman is that I don't encounter perverts, drug addicts on the streets, or anything that makes a normal person feel endangered.
There are many normal people who don't feel endangered just because they've encountered a drug addict on the streets. I know because I am one. Some drug addicts (and some other people) may make me feel threatened, but this relates to their behavior rather than to their condition. On the other hand, there are also many other normal people do feel endangered simply because they've encountered a drug addict. The difference lies in experience, ability to judge the circumstances, etc.
If you're judging safety based on a person's feeling of safety, you will find that this is naturally very subjective. Someone who has lived all her life in a rural area is likely to feel at least uneasy in most urban areas, and seriously threatened in some places. By contrast, someone who has lived her entire life in cities may be uneasy or worse in rural areas at night, whether from wild animals or imagined possible intruders. This is all completely normal, but the variability in "normal" makes it a difficult way to frame the question.
As to your drug addict at four in the afternoon, he might have been a threat, but probably not. A lot of street people are looking for a way to hustle someone for a bit of money. Most likely, he would have helped you get a cab and then asked you for some money in return. Of course, that might not be the case, and the safest thing to do is not to engage at all.
Now the taxi driver saying inappropriate things is entirely unacceptable regardless. If it was a licensed taxi, in most cities there will be a phone number posted in the vehicle for the taxi commission, where you can report this. If the number isn't there, look it up online or ask the hotel staff to help you find it. If you let the driver know that you're aware of the complaint number, you will usually see a rapid improvement in behavior.
I would recommend asking hotel staff (or anyone else local such as a shopkeeper) how to get to your destination. They may recommend a specific route or identify places to avoid; they may tell you to take a taxi.
The signs of a properly licensed taxi differ from one city to the next; Atlanta's seem to identify the operating business on the vehicle's exterior and to carry a four digit identification number. Take this number. If the driver's hack license is posted inside the car (some cities require this), take a picture of it. If the driver does something inappropriate, mention that you know how to submit a complaint. If necessary, call the police. You're very unlikely to need to call the police, but, importantly, the driver knows that you can.
As others have noted, you can't ensure 100% safety anywhere; you have to decide what your threshold is, what risks you can tolerate, and so on. The trick is to be able to change a situation where you feel unsafe into one where you feel safer. Local knowledge helps with this, so ask around.
