Must my mail server have a specific name?

Well, there’s nothing special about “smtp.” or “mail.” – they’re just
names.

But you might want to consider using something like them. They can make
certain future changes a little easier.

All the email clients really need to know is how to get to your mail server.
That means as long as they’re given a name that somehow resolves to the IP
address of the correct server, they’ll work. You can even skip that and use the
raw IP address when configuring a mail client, and it’ll still work. (In fact I
know of some people who use the IP address on purpose, simply to avoid the
extra look-up time when a domain name is converted to an IP address.)

So any domain name you choose that maps to the IP address of your
mail server is a fine choice.

However.

Let’s say some day in the future, you want to move your mail processing to a
different server. Perhaps you want to offload the server handling “example.com”
web traffic, because the email load is getting too high. It’s a common
scenario, actually, as mail servers are working extra hard these days filtering
spam.

If you have a separate subdomain – say “mail.example.com” – that you use for
all of your mail configuration, then after you set up your new mail server, all
you need to do is change the IP address that “mail.example.com” points to. As
DNS propagates, all of your mail processing will automatically move to the new
server. No email clients need to be reconfigured.

Now, there’s still an issue with DNS propagation that could potentially
cause some mail to be delivered on the old server after the recipients have
switched to the new – but a separate subdomain solves that problem as well.

The “MX” DNS record tells mailers where to send mail destined for your
domain. So perhaps we’d set up a domain “mx.example.com”, and use that ONLY for
the MX settings for your domain (or domains). We’d still also set up
“mail.example.com” for your recipient’s client programs to send and receive
through. Both would initially point to the same IP address of the same mail
server.

Moving mail servers would now be a two step process once the new server was
configured. Change “mx.example.com” to point to the new server. Now, wait two
days for DNS propagation (or, if you like, monitor the old server and wait
until it stops getting mail), and change “mail.example.com”. As that
propagates, your mail clients will start downloading mail from the new server,
including any that may have been queued up from the day you changed
“mx.example.com”.

But back to the original question: in all these cases I’ve used subdomains
like “mail” and “mx”, but in reality they could be anything.
“fred.example.com”, “example.example.com”, “kwijibo.example.com” are all valid
names for mail servers. Using more meaningful names, however helps keep things
somewhat more understandable.

And for that matter, if you don’t foresee the need for flexibility in the
future, then “example.com”, without a subdomain, is just fine too.

(Just remember to replace “example.com” with your domain, ok?)