Self-Hosted Mail Server Troubleshooting

It is very common for a self-hosted mail server to be able to send mail but fail to receive it. Sending and receiving follow two completely different paths: sending relies on outbound SMTP, while receiving depends on MX records, inbound port 25, local domain configuration, and the client’s ability to fetch mail over IMAP / POP3. A problem in any one of these layers can break inbound mail.

Whether your issue is “other people cannot send mail to my domain at all” or “the messages are clearly on the server, but my client cannot see them,” you can troubleshoot it layer by layer in the order below. This is usually much more efficient than jumping straight into configuration files.

First, identify which layer is failing

Before you start, split the problem into one of these three categories:

1. External senders cannot deliver mail to you at all Focus on MX, A/AAAA, public port 25, reverse DNS, and whether your server is actually listening.
2. The server has already received the mail, but the client cannot read it Focus on IMAP / POP3 listeners, TLS certificates, account authentication, mailbox quota, and client configuration.
3. Manual configuration works, but auto-configuration fails Focus on autodiscovery endpoints, SRV records, and the client’s autodiscovery mechanism.

If you are not sure which category you are dealing with, the simplest approach is to do two things at the same time:

• Send a test message to your domain from an external mailbox such as Gmail or Outlook.
• Immediately check the server logs to see whether that message ever arrived.

If there is no new inbound log entry at all, the problem is probably still at the DNS, network, or port 25 layer.

1. Check DNS first

For inbound mail, DNS is the first gate. If your MX is wrong, it does not matter how correct the rest of your setup is.

I. Check the MX record

dig example.com MX

You need to confirm two things:

• An MX record actually exists.
• The MX points to a hostname, not an IP address.

For example, this is a valid setup:

example.com.  300  IN  MX  10 mail.example.com.

II. Check the A/AAAA records behind the MX target

dig mail.example.com A
dig mail.example.com AAAA

The hostname pointed to by your MX must resolve to a reachable public address.

One very common pitfall is having an AAAA record even though the server is not actually configured correctly for IPv6. In that case, some senders will prefer IPv6 delivery, and the symptom becomes “some emails arrive, some do not.” If you are not ready to support IPv6 yet, it is better to remove a broken AAAA record than to leave it half-working.

III. Check PTR reverse DNS

dig -x <server-public-ip>

An incorrect PTR record does not always cause total delivery failure, but it can significantly hurt deliverability, and some providers treat such hosts more strictly. Ideally:

• The public IP’s PTR points back to your mail hostname, such as mail.example.com
• That hostname then resolves forward to the same IP

IV. Remember DNS propagation delays and local cache

If you have just changed your DNS records, do not forget these two possibilities:

• The new records have not fully propagated yet, and remote systems are still using the old ones
• Your local machine or router is caching stale results

You can do a quick extra check with:

dig +short example.com MX
dig +short mail.example.com A
dig +short mail.example.com AAAA

2. Confirm that inbound ports are actually reachable

When sending works, people often assume the network is fine. But that only proves the outbound path is probably working. It does not prove that other servers can reach your machine from the public internet.

I. Test SMTP port 25 first

If external mail cannot be delivered to you at all, port 25 is the first thing to verify:

nc -zv -w 3 mail.example.com 25

If you also want to check the client-facing ports while you are at it, test them too:

nc -zv -w 3 mail.example.com 143
nc -zv -w 3 mail.example.com 993
nc -zv -w 3 mail.example.com 110
nc -zv -w 3 mail.example.com 995
nc -zv -w 3 mail.example.com 465
nc -zv -w 3 mail.example.com 587

II. Common mail ports at a glance

III. Do not look only at the server firewall

If a port is unreachable, the problem can be at any of these layers:

• The host firewall is not allowing it
• Your cloud provider’s security group or network ACL is blocking it
• Your router or NAT is missing the necessary port forwarding
• The port mapping from host to container is wrong
• Your upstream ISP or hosting provider restricts mail-related ports

If you are deploying on residential internet or certain cloud platforms, port 25 deserves extra attention. In some environments, the issue is not that the service failed to start, but that the upstream network simply does not allow those connections.

IV. Other ways to test

nmap -p 25,110,143,465,587,993,995 mail.example.com
curl -v --connect-timeout 3 telnet://mail.example.com:25

If you want a zero-dependency check, Bash can probe TCP directly:

(echo >/dev/tcp/mail.example.com/25) && echo "open" || echo "closed"

3. Check TLS and certificates

If the server is receiving mail but the client keeps reporting certificate errors, TLS handshake failures, or endless connection attempts on IMAP / POP3, TLS is the next layer to check.

I. Test implicit TLS ports

openssl s_client -connect mail.example.com:993 -servername mail.example.com
openssl s_client -connect mail.example.com:465 -servername mail.example.com
openssl s_client -connect mail.example.com:995 -servername mail.example.com

II. Test STARTTLS ports

openssl s_client -connect mail.example.com:143 -servername mail.example.com -starttls imap
openssl s_client -connect mail.example.com:587 -servername mail.example.com -starttls smtp
openssl s_client -connect mail.example.com:110 -servername mail.example.com -starttls pop3

Note: openssl s_client -starttls negotiation is not always reliable. Some servers expect authentication or specific protocol interaction before STARTTLS behaves the way a real client would, and openssl cannot emulate all of that. So if all implicit TLS ports (993/465/995) succeed while all STARTTLS tests fail, the limitation may be in the test tool rather than your server configuration. Most clients use implicit TLS by default anyway, so as long as implicit TLS works, mail retrieval is usually unaffected.

III. Extract certificate details quickly

openssl s_client -connect mail.example.com:993 -servername mail.example.com 2>/dev/null | openssl x509 -noout -subject -issuer -dates

IV. Common certificate problems

The most common certificate issues are:

• The certificate is self-signed, so the client does not trust it by default
• The certificate hostname does not match the hostname the client is using
• The certificate has expired
• TLS is not actually enabled correctly on that service port, and you see no peer certificate available

One important detail: missing TLS on SMTP 25 does not always break receiving entirely, but TLS problems on IMAP / POP3 often show up as “the mail is there, but the client simply cannot log in.”

4. Check the mail service itself

If DNS and port reachability both look fine, the next step is to confirm that the mail service is actually exposing the right inbound and retrieval behavior.

I. Confirm that the service is listening on a public address

First, check which addresses the relevant ports are bound to:

ss -tlnp | rg ':(25|110|143|465|587|993|995)\\b'

Under normal circumstances, the service should at least be listening on 0.0.0.0, [::], or another publicly reachable address. If it is only listening on 127.0.0.1 or ::1, external clients obviously will not be able to connect.

II. Confirm that the server is willing to receive mail for your domain

This is another very common misconfiguration: the service is running and the port is open, but the server does not actually treat example.com as a local domain.

Different mail servers use different terminology, but the thing you need to verify is always the same:

• Your domain is listed as a local delivery domain
• Mailboxes or aliases for that domain really exist
• Incoming mail has a valid local delivery target

If this layer is wrong, common symptoms include:

• External senders receive relay access denied
• The server treats mail addressed to your own domain as something to forward onward
• The logs show the message arriving, but it is ultimately rejected or delivered to the wrong place

III. Check accounts, quotas, and local storage

If external delivery is already succeeding but the client still cannot see the mail, continue with these checks:

• Whether the account actually exists
• Whether the mailbox is full
• Whether the message was delivered into an unexpected folder
• Whether filtering rules moved, archived, or deleted the message

IV. Watch for blocking and rate limiting

Many people focus only on the main configuration and forget about the extra access-control layers, such as:

• fail2ban or similar banning tools
• The mail service’s own IP blacklist or greylist
• Temporary lockouts or rate limiting after repeated login failures

These issues often look intermittent: the same configuration works on one device but fails on another.

V. Check the logs directly

Logs are usually faster than guessing. Whether you are running the mail service as a system service, in containers, or through a control panel, you should find the last few dozen log lines first:

journalctl -u <your-mail-service> --no-pager -n 100
docker logs <your-mail-container> --tail 100

Pay special attention to keywords like:

• connect
• reject
• relay
• tls
• certificate
• authentication failed
• timeout
• rate limit
• mailbox full

If you watch the logs while sending a test message from an external provider, you can often tell within minutes whether the problem is “the message never got in” or “it arrived, but processing failed afterward.”

5. Check client autodiscovery

If manual server and port configuration works fine, but Thunderbird, Outlook, or a mobile mail client fails to configure the account automatically, the issue is usually not mail delivery itself. It is the autodiscovery layer.

I. Common autodiscovery methods

Different clients use different mechanisms. Common ones include:

• Autoconfig For example: https://autoconfig.example.com/mail/config-v1.1.xml
• Autodiscover For example: https://autodiscover.example.com/autodiscover/autodiscover.xml
• DNS SRV records For example: _imaps._tcp, _submission._tcp
• Guessing common hostnames For example: mail.example.com, imap.example.com, smtp.example.com

II. Verify the related URLs directly

curl -v "https://autoconfig.example.com/mail/config-v1.1.xml"
curl -v "https://autodiscover.example.com/autodiscover/autodiscover.xml"

If these endpoints are unreachable, present the wrong certificate, or return content that the client does not expect, auto-configuration will fail naturally.

III. Common SRV record examples

IV. Manual configuration is often the fastest validation

If you simply want to confirm that the service itself works, manual configuration is usually the fastest path:

If manual configuration succeeds, the mail service itself is probably fine. It usually just means the autodiscovery metadata is incomplete.

6. Rule out local network and cache interference

Sometimes “receiving is broken” is not caused by the server at all. Your local network environment may be getting in the way.

I. Try a different network first

If you see symptoms like these, local network issues or ISP interference become very plausible:

• nc can connect, but the TLS handshake gets interrupted
• The same account works over a phone hotspot but fails on your home or office network
• Only certain devices or certain networks are affected

The simplest checks are:

• Retry using a phone hotspot
• Retry through a VPN
• Retry from another device

II. Capture packets if needed to see whether traffic reaches the server

If you have server access, you can capture packets on the server side to confirm whether the client’s traffic is arriving:

sudo tcpdump -i any -n 'port 25 or port 993'

The interpretation is straightforward:

• If the client connection never reaches the server, investigate DNS, routing, and port forwarding first
• If the connection reaches the server but the service never responds, investigate service state and TLS first
• If the connection reaches the server and gets rejected immediately, investigate authentication, blocking, and access policy first

III. Clear local caches

After changing DNS or certificates, it is also worth checking local caches:

sudo resolvectl flush-caches
resolvectl query mail.example.com
grep example /etc/hosts

If hosts still contains stale entries, or the system is still using old DNS answers, it is very easy to head in the wrong troubleshooting direction.

7. Do one minimal command-line validation

GUI mail clients often hide too much detail. If you suspect the client itself may be the problem, do one minimal validation from the command line.

I. Test IMAP login

curl -v --url "imaps://mail.example.com/INBOX" --user "[email protected]:password"

If this succeeds in listing or accessing mailbox contents, then at least these layers are already working:

• DNS resolution
• Port 993 connectivity
• TLS handshake
• Username and password authentication
• IMAP service availability

II. Interact with the IMAP service manually

openssl s_client -connect mail.example.com:993 -servername mail.example.com -quiet

Once connected, you can type:

a1 LOGIN [email protected] password
a2 LIST "" "*"
a3 SELECT INBOX
a4 LOGOUT

This approach is primitive, but it is very direct when you need to answer the question, “is this a client problem, or a server problem?”