Self-hosted transport (bring your own email service)

By default, when you add a custom domain ReplyLayer handles delivery and inbound routing for you. Self-hosted transport is the alternative: you point ReplyLayer at your own SMTP and IMAP endpoints and keep the mail edge under your control. ReplyLayer sends through your SMTP server and polls your IMAP folders for inbound mail, but never operates a mail server on your behalf.

Pick self-hosted transport when you must keep mail flowing through infrastructure you already run — for compliance, data-residency, or existing-MTA reasons. If you just want branded addressing with the least setup, use the managed path in the custom domains guide instead.

What self-hosted transport is

A delegated domain runs in one of two transport modes:

• Managed — ReplyLayer registers the domain with its delivery infrastructure, returns the DNS records to publish, and handles outbound delivery and inbound routing. This is the default.
• Self-hosted — ReplyLayer stores your SMTP/IMAP connection details, verifies domain ownership with a TXT record, sends over your SMTP server, and polls your IMAP folder(s) for inbound mail.

Inbound and outbound mail on a self-hosted domain is still scanned like any other message — see content scanning. Self-hosted transport changes only where the bytes travel, not the safety model.

The v1 contract

Self-hosted transport is deliberately narrow in v1. These are hard constraints — plan your setup around them:

• Managed from the dashboard or the API. Submit a self-hosted domain from the dashboard's Domains page ("Connect your mail server"), or over the API / SDKs / CLI; verification, diagnostics, and connection-settings edits live on the domain's detail view.
• Public endpoints only. Your SMTP and IMAP hosts must resolve to public addresses. Private-network and tailnet connectivity are not supported in v1, and hosts that resolve to private or loopback addresses are rejected.
• CA-signed TLS with hostname verification. Both endpoints must present publicly trusted certificates that pass normal hostname verification. There is no self-signed-certificate bypass in v1.
• Username / password authentication only. OAuth2 / XOAUTH2 is not supported in v1, so hosted mail APIs that require it (for example large hosted-mail providers reachable only over OAuth) are out of scope. RFC-standard SMTP + IMAP servers are the supported target.
• One dedicated IMAP folder per mailbox. Each ReplyLayer mailbox maps to exactly one IMAP folder that must already exist. You are responsible for server-side rules that route each mailbox's inbound mail into its folder.
• One SMTP credential per domain. ReplyLayer submits mail with a single per-domain SMTP credential but sends as whichever mailbox address the agent operates from, so that credential must be authorized to send as every mailbox address on the domain (see SMTP sender authorization).

Network reachability and ports

Self-hosted transport uses two TCP connections, both initiated by ReplyLayer to endpoints you provide:

ReplyLayer does not use port 25 (inter-MTA delivery). Adding ReplyLayer as a backup MX is a different integration and not part of self-hosted transport.

Egress IPs to allowlist

You do not have to expose ports 587/465/993 to the whole internet. If your mail infrastructure sits behind a firewall, you can restrict inbound reachability on those ports to the source IPs ReplyLayer connects from.

ReplyLayer's outbound connections are load-balanced across a set of source IPs, not a single address. A connection may originate from any of these, so you must allowlist the whole set:

Allowlist all four on the ports you expose. Two things to understand before you rely on this:

• The set can change. On future platform migrations these IPs may change. ReplyLayer publishes any updated set on this page and in the changelog — so if you allowlist by IP, re-check it periodically rather than treating the list as permanent.
• These are shared, not dedicated, IPs. They are shared-tenancy load-balanced addresses, so an IP allowlist is a coarse network filter, not proof of who is connecting. The real authentication is your SMTP/IMAP credential over CA-signed TLS — keep credential and TLS enforcement strict regardless of the allowlist.

If you would rather not maintain an IP allowlist, either rely on TLS + credentials as the boundary (the v1-default posture), or route ReplyLayer to a static-IP egress proxy you operate that relays to your internal-only mail infrastructure.

Setup flow

1. Create the domain

Submit the domain with transport_mode: "self_hosted" and a nested self_hosted_config:

curl -X POST https://api.replylayer.ai/v1/domains \
  -H "Authorization: Bearer $REPLYLAYER_ADMIN_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "domain": "ai.acme.com",
    "transport_mode": "self_hosted",
    "self_hosted_config": {
      "smtp": { "host": "smtp.ai.acme.com", "port": 587, "security": "starttls", "username": "replylayer-smtp", "password": "..." },
      "imap": { "host": "imap.ai.acme.com", "port": 993, "security": "tls", "username": "replylayer-imap", "password": "..." },
      "network_mode": "public"
    }
  }'

network_mode must be public. The SMTP and IMAP hosts are screened and must resolve to public addresses. On success the domain is created in verification_status: "requested" and the response returns a single ownership TXT record in dns_records_json — no SPF, DKIM, tracking, or MX records, because you keep operating the mail edge.

2. Publish the ownership TXT

Create the returned TXT record exactly as given, at _replylayer-verify.<domain>. ReplyLayer uses it as the DNS ownership gate for self-hosted domains.

3. Review and approval

Self-hosted domains enter requested and are reviewed by ReplyLayer before going live. Approval runs a four-gate probe against your endpoints:

• ownership_txt — the ownership TXT record is visible
• imap_auth — IMAP authentication succeeds
• smtp_auth — SMTP authentication succeeds
• smtp_sender_authorization — a MAIL FROM with a randomized probe localpart on your domain is accepted (validates that the authenticated SMTP user may send as arbitrary addresses on the domain; the probe resets before DATA, so no message is ever submitted)

If every gate passes, the domain advances to verified. If a gate fails, the domain still advances to pending_verification so you can fix your MTA and retry verification yourself without waiting on another review.

4. Verify

Trigger a check with POST /v1/domains/:id/verify. This re-runs the ownership, IMAP-auth, SMTP-auth, and sender-authorization gates plus a check that every claimed IMAP folder exists. All gates green promotes the domain to verified; any failing gate leaves it in pending_verification and the response names the failed gate. The call is rate-limited to one per 60 seconds per domain.

5. Create mailboxes with dedicated IMAP folders

Each mailbox on a self-hosted domain requires a self_hosted_imap_folder that already exists — ReplyLayer probes it synchronously at creation:

{ "name": "support-bot", "self_hosted_imap_folder": "ReplyLayer/support-bot" }

ReplyLayer never moves messages into these folders for you. You own the server-side routing (Sieve, Postfix/Dovecot rules, Exchange transport rules, or similar) that lands each mailbox's inbound mail in its assigned folder. If two mailboxes try to claim the same folder, creation fails with FOLDER_ALREADY_CLAIMED.

SMTP sender authorization

Because ReplyLayer sends with one per-domain credential but as each mailbox address, any MTA that enforces "the authenticated user owns the sender" will reject the send with a 553 / 550 "Sender address rejected" error unless that credential is authorized for the mailbox address. Do not disable sender enforcement — that control defends against exactly the spoofing threat this model inherits. Instead, authorize the ReplyLayer SMTP user to send as your mailbox addresses (a scoped send-as allowlist, updated as you add mailboxes, or domain-wide send-as if ReplyLayer is the only authenticated user for the domain).

Validate it from outside your network before onboarding — a MAIL FROM as one of your mailbox addresses should return 250, not 553.

Delivery and bounce visibility

Self-hosted transport has limited delivery observability. ReplyLayer surfaces a synchronous SMTP rejection at submission time (for example a definitive SMTP_REJECTED), but there is no delivered / bounced lifecycle event stream for self-hosted mail the way the managed path provides. Once your SMTP server accepts a message, its downstream fate is visible only in your own mail infrastructure. Design any confirmation logic around the synchronous accept/reject, not around later delivery events.

Common error codes

For every error code and its shape, see the error reference.

Related

• Custom domains — the managed transport alternative.
• Plans & limits — per-plan domain counts.
• Error reference — every error code, one catalog.