First, be aware that most instant messaging programs allow you to block instant messages from people you don’t know. I would do that immediately. Then, when someone wants to IM you, you have to approve it.
But as for finding out who did, the news is not good. Possible? Technically, yes, but not likely.
Become a Patron of Ask Leo! and go ad-free!
When you instant message someone, your computer is not talking to theirs. Your computer is actually talking to the computers at the instant messaging service, which then relays your message to the computer of the person you’re talking with. So if you’re using MSN Messenger, for example, when you send a message your computer sends it to the computers out at Microsoft, which then send it to whomever you’re chatting with. When they respond, the same thin happens in reverse. (In some services, you can establish what’s called a “direct connection”, which may connect you directly, but it requires both ends of the conversation to agree, and someone attempting to remain anonymous will know enough not to do that.)
The result is that there’s no information that you have access to that would tell you who that person really is. All you have is a screen name, and that’s it. If that person has chosen to keep the public information about that screen name vague or blank, then I know of no way for you to find out who they really are.
Now, obviously, the instant messaging service knows both the account details and IP addresses for both of you – at least for as long as they might keep the records. But these services have strict privacy policies that will prevent them from just handing out that information to anyone. In fact, they’re going to make it almost impossible to get. It’s my belief that it will take law enforcement or even a court order to get useful information from the instant messaging services. And then, even if you do get the IP address of the other person, you’d have to go through the same legal hoops to get their ISP to provide you with any practical, identifiable information.
So that’s a really long answer that boils down to this:
No, there is no practical way for you to find out who’s contacting you via instant messaging if you don’t already know.