QMail Ping — Group 6, Code 72

Retained notifications (server source since 2026-09-09)

Ping, Peek and resumed Ping do not delete Tells. A Tell stays in the inbox for the 30-day retention window measured from its latest publication, regardless of how many times it was fetched. Any client may see the same Tell more than once and must persist each notification locally before advancing its cursor. Deployed fleet support should be verified against the server build; this page describes the raidax source at commit bcc93e8. See the safety amendment and remaining ACK/cursor work, and the proposed Wait (group 22, command 2), Ack (group 22, command 3) and Beacon Status (group 22, command 1) commands.

Quick reference

Purpose

ping is QMail’s long-poll receive primitive. It scans the recipient mailbox directory for .tell files whose sender timestamp is greater than the request’s since_timestamp, exactly as peek does. If one or more match, the server returns them at once using the same tell-list payload format as peek. If none match, it creates the inbox directory if needed, registers a one-shot inotify watcher, and leaves the TCP connection open until a new .tell lands or the watcher expires.

The server path is selected from the mailbox identity in the request body: QMAIL_MAILBOX_ROOT/{denom_hex}/{serial_number}/inbox/. The request must be sent to the recipient’s beacon RAIDA; other RAIDAs may not have that mailbox coin loaded.

Wire packet

The RAIDA packet header is the 32 plaintext bytes at the front of the TCP packet. The 16-byte challenge/CRC is encrypted with the rest of the request body. raidax groups that challenge with the QMail identity block and calls those first 48 decrypted body bytes qmail_preamble_t.

QMail Ping request packet, bytes on TCP

+-------------------------------+-----------------------------------------+----------------------+
| RAIDA request header           | encrypted request body                  | plaintext terminator |
| 32 bytes                       | 52 bytes (48 minimum)                   | 2 bytes              |
+-------------------------------+-----------------------------------------+----------------------+
| command group=6, command=72    | challenge/CRC(16)                       | 3E 3E                |
| encryption type=1              | QMail identity/auth block(32)           |                      |
| body_size=54 (50 minimum)      | since_timestamp(4)                      |                      |
| nonce(8)                       |                                         |                      |
+-------------------------------+-----------------------------------------+----------------------+

rest_core sends body_size = 54 (0x0036), the same builder as peek. raidax accepts a 50-byte body without the timestamp and then scans with since_timestamp = 0, which returns every retained Tell in the inbox. Always send the timestamp.

Request header (32 bytes, plaintext)

Request body (54 bytes)

Decrypted request body layout

+-------------------+--------------------------+---------------------+----------------------+
| challenge/CRC     | QMail identity/auth      | since_timestamp     | terminator           |
| 16 bytes          | 32 bytes                 | 4 bytes             | 2 bytes              |
| encrypted         | encrypted                | encrypted           | plaintext            |
+-------------------+--------------------------+---------------------+----------------------+
| body[0..15]       | body[16..47]             | body[48..51]        | body[52..53]         |
| required          | required                 | 0 if omitted        | required             |
+-------------------+--------------------------+---------------------+----------------------+

raidax names body offsets 0–47 qmail_preamble_t. In the terminology above, the challenge is still body data because it is encrypted with the rest of the body fields.

Response packet

There are three normal ping outcomes: a tell-list success response when matching mail is available now or arrives during the poll; a success response with tell_count = 0 when the watcher was woken by a file that does not match the cursor (for example a Tell whose sender timestamp is not newer than since_timestamp); and a status-only 245 reply when the parked long-poll expires without mail.

QMail Ping success response, bytes on TCP

+-------------------------------+-----------------------------------------+----------------------+
| RAIDA response header          | encrypted tell-list payload             | plaintext terminator |
| 32 bytes                       | payload_len bytes                       | 2 bytes              |
+-------------------------------+-----------------------------------------+----------------------+
| status=250, group=6            | array header(8)                         | 3E 3E                |
| body_size=payload_len+2        | record 1                                |                      |
| challenge signature            | record 2                                |                      |
|                                | ...                                     |                      |
+-------------------------------+-----------------------------------------+----------------------+

QMail Ping timeout response, stale-watcher path (15-16 minutes after parking)

+-------------------------------+----------------------+
| RAIDA response header          | plaintext terminator |
| 32 bytes                       | 2 bytes              |
+-------------------------------+----------------------+
| status=245, body_size=0        | 3E 3E                |
| no encrypted payload           |                      |
+-------------------------------+----------------------+

The stale-watcher sweep in net.c (cleanup_stale_ping_monitors, run every 60 seconds) sets STATUS_TIMEOUT (245, 0xF5) with output_size = 0 and hands the connection to prepare_response, which still appends the 3E 3E trailer. Earlier versions of this page described status 17; that is not what the server sends.

Response header (32 bytes, plaintext)

Tell-list payload on success (encrypted)

Decrypted tell-list payload

+----------------------+--------------------------------------+--------------------------------+
| Array header          | Tell record 1                        | Tell record 2 ...              |
| 8 bytes               | 64 + (M * 32) + manifest_len bytes   | variable                       |
+----------------------+--------------------------------------+--------------------------------+

Tell record structure

Each returned record is the full contents of one .tell file. The server does not add a per-record length field. For manifest v1 records, the client reads three fields from the 64-byte file header to compute the record size:

• stripe_count (byte 29, called M): number of server-location entries.
• file_count (byte 52, called N) and file_entry_size (byte 53, always 16): together describe the file manifest. The big-endian manifest_len at bytes 54–55 must equal N × file_entry_size.

manifest v1 record_size = 64 + (M * 32) + manifest_len
legacy v0 record_size   = 64 + (M * 32), then optionally skip an 18-byte footer

+-------------------------+-----------------------------+--------------------------------+
| QMail file header        | server location entries      | file manifest entries          |
| 64 bytes                 | M entries * 32 bytes         | N entries * file_entry_size    |
+-------------------------+-----------------------------+--------------------------------+

Current manifest v1 records do not include a trailing footer; the manifest entries are the last bytes of each record. Pre-manifest legacy records have manifest_version=0, no manifest entries, and may have an 18-byte recipient footer on disk (tag 0x50, length 16, recipient locker). A receiver that sees that footer after a legacy record should skip it before parsing the next record.

QMail file header (64 bytes)

Server location entry (32 bytes each)

There are M entries immediately after the file header. This structure tells the recipient which RAIDA servers hold the stripes and how to connect to them for download.

File manifest entries (N × 16 bytes)

The manifest is the authoritative public object list for this email. New QMail beta records list private CBDF meta first with file_type=0x00, body/content second with file_type=0x01, and attachments afterward using sequential file types 0x0A, 0x0B, and so on. The receiver downloads file_type=0 first, then uses original_size when sizing each later reassembly buffer so padding bytes are trimmed correctly.

Local REST test calls

These calls use the two-client local setup: client1 is the sender on port 8081 and client2 is the receiver on port 8082. Stopping the receiver background beacon is optional: with retained Tells both the beacon and this manual call would see the same notification, but stopping it keeps the log readable.

# Optional: stop client2 background beacon before the sender sends mail
GET http://127.0.0.1:8082/api/qmail/local/beacon/control?action=stop

# Long-poll client2's beacon RAIDA; notifications include manifest_version, file_count, manifest_flags, and files[]
GET http://127.0.0.1:8082/api/qmail/net/beacon/ping?timeout=10

# Download the received email and all manifest-listed attachments
GET http://127.0.0.1:8082/api/qmail/net/messages/download?file_guid=<FILE_GUID>

# Inspect downloaded attachment rows
GET http://127.0.0.1:8082/api/qmail/db/attachments/list?email_id=<FILE_GUID>

Long-poll mechanics

1. Validate the request body length and 3E 3E terminator.
2. Decrypt the body and validate the 16-byte challenge/CRC.
3. Read the mailbox identity from the decrypted body and verify the AN.
4. Read since_timestamp from body bytes 48–51 (0 if the body is only 50 bytes).
5. Scan QMAIL_MAILBOX_ROOT/{denom_hex}/{sn}/inbox/ for .tell files in directory order. For each, read the 64-byte file header and keep the file if its sender timestamp (bytes 24–27) is strictly greater than since_timestamp; a zero sender timestamp falls back to the file’s st_mtime.
6. If one or more match, copy them into the tell-list payload and send a success response. The buffer starts at 8 KiB and is doubled once per record that does not fit, up to 1 MiB; a record larger than the doubled size ends the scan (so a first record over about 16 KiB produces an empty result even though it matched). The count byte is a plain cast of the record count and wraps past 255. Nothing is deleted.
7. If none match, create the inbox directory if needed, register a one-shot inotify watcher for this connection, set STATE_PING_ACTIVE, and leave the TCP connection open.
8. When a .tell file is atomically renamed into the inbox, every parked connection watching that inbox (one per polling device) is dispatched to qmail_resume_ping(), which rescans with that connection’s own since_timestamp and replies. The watcher is armed after the first scan, so a Tell that lands between the scan and the watch registration waits for the next poll. If the new file does not match the cursor the reply is a success with tell_count = 0; the client should simply ping again.
9. If no wake occurs, the once-a-minute sweep expires watchers older than 15 minutes and sends status 245 with an empty body. The connection is then closed by the server.
10. The socket carries TCP keepalive (idle 60 s, interval 10 s, 5 probes), so a client that disappears while parked frees its watcher within about two minutes.

Status codes

Client budget: the server holds a poll for up to 16 minutes, so a client socket timeout shorter than that expires first. rest_core’s default is 600 s. A client must either wait longer than the server or treat its own expiry on command 72 as “no mail” and re-ping at once; it must not count it as a failure. The proposed Wait (group 22, command 2) lets the client choose the hold time.

Common mistakes

Points of confusion

1. ping vs peek. Both apply since_timestamp the same way. The only difference is what happens when nothing matches: peek returns an empty list at once, ping parks the connection.
2. Tells are not deleted on read. Every device polling a mailbox sees every retained Tell. Deduplicate by email_id (GUID) and persist before advancing the cursor.
3. The cursor is the sender’s clock. The filter compares since_timestamp with the timestamp the sender wrote into the Tell header, not with server time. Two senders with skewed clocks can hide each other’s mail from a cursor that advances to the newest value seen; a client should keep a look-back margin, and Wait (group 22, command 2) replaces the timestamp with a server-ordered cursor.
4. Batches can be cut silently. The scan stops at 1 MiB in directory order, or earlier when one record is larger than the buffer can grow in a single doubling, and sets no flag; total_tells is zero and the count byte wraps. A client that advances its cursor to the newest timestamp in a cut batch can strand the Tells that were left out, and re-sending the same cursor returns the same prefix, so there is no client-side way to page through the remainder. Nearness to 1 MiB is not a reliable signal either. Treat a suspected cut as a stalled-discovery condition to surface, and see the proposed Wait (group 22, command 2) for an explicit continuation.
5. Records are variable length. There is no length prefix per record. Use stripe_count (byte 29) and manifest_len (bytes 54–55) from the file header to compute 64 + M*32 + manifest_len.
6. Same-second tells. The filter is strictly greater on whole seconds. A second Tell stamped in the same second as the newest one you received, but indexed after your reply, will not be returned by a cursor equal to that second.