TTL Expression Gaps That Cause Silent Data Loss During a Parallel Run

TTL expression gaps can turn a parallel schema run into permanent, undetected data loss. The danger arises when an expression would preserve individual rows but stored metadata marks the entire part for deletion. Safe operation therefore requires more than matching expressions across schemas. It requires defensive expressions, scrutiny of transferred metadata, and a cutover gate based on retention outcomes verified independently of merge success.

Why TTL expressions that work correctly in isolation can silently delete rows when two schemas run simultaneously

A team adds a timestamp column to the destination schema during a parallel run. Some rows have no value for it, and the new TTL expression resolves those rows to a zero epoch. Later, those rows disappear with no error or log entry, even though row-level TTL evaluation would have kept them. The expression alone does not explain the loss. Its interaction with the other rows in the part does.

TTL enforcement happens inside background merges after insertion. This delay gives part metadata and merge scheduling a role in the outcome, together with the distribution of rows across parts. During a parallel run, the old and new tables merge independently, and ATTACH PARTITION FROM can move parts between schemas. Two expressions can thus give the same result for most rows but differ when they encounter zero values or metadata kept from the source table.

How the merge engine uses part-level TTL bounds instead of row-level evaluation, and why that shortcut is the source of most silent loss

The merge engine has long used stored TTL bounds to decide whether an entire part has expired, rather than evaluating the current expression over every row. That shortcut makes the metadata authoritative. If the stored maximum says all rows have expired, the engine can remove the part without discovering that the live expression would retain some of it.

PR #118209 documents five paths that trusted these bounds. They include whole-part drops in TTLTransform and TTLDeleteFilterTransform, whole-column resets for wide parts in TTLColumnAlgorithm, the pipeline-less TTLDrop path in MergeTask, and isRowsMaxTTLExpired in MutateTask’s ExecutableTaskDropTTLExpiredPartsDecorator. The proposal would remove isMaxTTLExpired entirely and make persisted bounds scheduling hints only. The engine would instead have to check the actual rows before deleting data.

The brief says the PR was closed without merging in September 2026 and also reports a merged root fix, but does not say which fix that was. A deployment without the fix it needs is still at risk from the gaps below, and the closed proposal alone gives no proof that it is safe.

The four structural gaps that a parallel run is most likely to trigger, and what makes each one distinct

In the first gap, MergeTreeDataPartTTLInfo::update(time_t) skips zero when it works out a part’s maximum, while per-row ITTLAlgorithm::isTTLExpired never expires a zero TTL value. Yet TTLColumnAlgorithm::execute and TTLTransform::TTLTransform treat max <= now as proof that everything has expired. Once the other rows expire, zero-TTL rows can be deleted or column-reset too. A nullable or conditionally populated timestamp in the new schema creates this trigger. PR #123127, filed September 2026, addresses zero values in row-level bounds and persisted ttl.txt, but Issue #126192, filed October 16, 2026, records that the epoch-zero class is not fully closed.

The second gap begins with metadata inherited through ATTACH PARTITION FROM. TTLPartDropMergeSelector::getTTLForPart derives part_max_ttl from all TTL-info families in ttl.txt, including orphaned columns_ttl entries from the source schema. Those entries can cause a merge to be tagged TTLDrop. MergeTask::ExecuteAndFinalizeHorizontalPart::prepare then treats that tag together with hasOnlyRowsTTL() as proof that every row has expired, skips createMergedStream, and writes a zero-row part. The next background merge can therefore erase the attached part with no part_log failure or default warning. No separate fix status for this path is supplied.

The third gap is the general stale-bounds case. Changing an expression or a column’s type, or moving parts between tables, can leave ttl.txt with values that the current expression would not produce. The shortcut then deletes rows or resets columns that live evaluation would preserve. This general defect encompasses the first two gaps and extends beyond zero-specific cases. Finalization commits the result to disk, beyond recovery by a later merge. The brief reports a merged root fix, but deployments predating that fix remain exposed.

The fourth gap concerns durability: at default settings, ttl.txt is neither covered by part checksums nor fsynced, so power loss can leave its inode persisted at length zero while the rest of the part remains intact. Issue #119602 documents the resulting immediate detachment and unacknowledged row loss. A parallel run exposes in-flight parts in both tables as writes and merges proceed concurrently. Correct expressions cannot prevent this infrastructure-triggered failure, and the brief supplies no fix status for it.

Why a parallel run amplifies exposure to these gaps instead of isolating risk

A longer parallel run extends the period in which different expressions and inherited metadata coexist. merge_with_ttl_timeout defaults to four hours per table, and TTL merges are deprioritized when the merge pool is busy. Each table can therefore reach a destructive decision at a different time.

The migration process itself opens another route to loss. ATTACH PARTITION FROM moves data between the tables, but it can also preserve the orphaned metadata that triggers a whole-part drop. At the same time, dashboards built around the old table’s names, columns, and indexes may no longer give a true picture of the destination. With one schema that stays the same, there is at least one steady target to monitor. During the transition, per-node scheduling can also make replicas disagree about which parts have expired, producing different answers without a replication error.

Why standard monitoring signals cannot distinguish intentional TTL expiry from silent data loss during a parallel run

What would an ordinary dashboard show when deletion succeeds for the wrong reason? The shortcut paths that delete data return success, with no exception, part_log failure, or default warning entry. Counts of the rows in each table do not settle the question. Independent TTL timing, incremental partition transfers, and different query-time filtering already make divergence between the tables expected. An unexpected loss can look like ordinary expiry.

Aggregates can conceal which rows went missing. With TTL-GROUP BY rollups, the old table may still give the right answer to SELECT event_date, event_type, sum(cnt) while the destination’s aggregate is wrong because rows that should have survived were deleted. A dimension-level query such as WHERE user_id = 42 simply returns no rows, just as correct expiry might. An agent comparing schemas and running SELECT COUNT(*) can identify a discrepancy in the results when prompted, but those observations alone cannot distinguish its cause or support self-remediation.

What actually works for detecting silent TTL loss before it becomes permanent

The practical response is to compare outcomes at partition granularity. Bracket merge cycles with partition-level row counts and compare any shrinkage with what the TTL expression actually permits. Table totals can hide a local loss. For correctness-critical tables, supplement system.parts with a periodic checksum of partition row counts kept externally, so the comparison survives merge-driven metadata corruption.

To find parts at risk before rows are deleted, query system.parts for delete_ttl_info_max values of zero on parts that hold recent data that has not expired. That combination is the epoch-zero warning signal. After ATTACH PARTITION FROM, inspect the attached part’s ttl.txt directly for columns_ttl families that have no corresponding destination column. These checks examine the metadata the shortcut trusts before a total row count falls. They target different triggers and are not substitutes for each other.

Run OPTIMIZE TABLE ... FINAL on a test partition with send_logs_level = 'trace' and check whether the engine evaluates rows or takes the whole-part shortcut. Trace output showing can_short_circuit_ttl_drop = true on a part with zero-TTL rows establishes exposure. This merge can delete test data. Also audit merge_with_ttl_timeout and merge-pool saturation, because persistent enforcement lag allows stale metadata to accumulate before these checks encounter its effects.

How to write TTL expressions that are safe under both schemas simultaneously during the transition period

For teams that cannot immediately update, defensive expressions are a temporary mitigation while the root defect remains. Begin by preventing zero-epoch output under either schema. A nullable or conditionally populated timestamp should not be the sole TTL input without a fallback such as COALESCE(event_time, toDateTime('2106-01-01')). That fallback handles missing values, so the full expression still needs to exclude any other route to zero. toDateTime(0) is not a safe “never expires” convention on affected deployments, because it invokes precisely the disagreement between row semantics and stored bounds.

If the destination renames the timestamp column or changes its type, use a temporary expression that produces the same result on both sides, even if it is less precise than the intended final form. This limits disagreements introduced by the schema change. Deploy the final expression only after cutover and after ttl.txt has been recomputed on all parts under the new schema. Expression compatibility does not, by itself, remove inherited metadata.

For attached partitions, the prescribed mitigation is immediate, targeted recomputation with OPTIMIZE TABLE destination PARTITION id '...' FINAL before background merges can fire. Its purpose is to rebuild ttl.txt against the destination schema and clear orphaned columns_ttl entries. Timing is essential here. A corrupted part_max_ttl can tag a regular merge as TTLDrop, so merge_with_ttl_timeout does not provide a waiting window. Because explicit merges can also encounter the shortcut, this belongs in the controlled, traced process with row counts verified before and after.

On unpatched deployments, raising merge_with_ttl_timeout can reduce opportunities for TTL merge shortcuts to fire. Pair that reduction with explicit OPTIMIZE ... FINAL runs on selected partitions under conditions where counts can be compared. The underlying defect remains. It also does not close the attachment path or protect ttl.txt from an unexpected restart.

A separate schema rule concerns derived values. Issue #117310 documents that a column TTL resets its source column without recomputing dependent MATERIALIZED columns, leaving stale values even after OPTIMIZE... FINAL. If the new schema introduces such a dependency, remove it or treat the materialized value as unreliable after the source resets and plan queries accordingly. This is distinct from the four deletion paths. Preserving the row does not guarantee that its derived value remains correct.

The operational posture that makes a parallel run safe — treating TTL as eventually consistent and building the cutover gate accordingly

Merge-driven TTL enforces the rules as part of the existing merge process and avoids the costly full-part rewrites that come with periodic mutations. The trade-off is delayed enforcement. During normal background operation, there is no point at which every TTL rule can be trusted to be up to date across both tables.

After a quiesce period with no new writes to either table, force enforcement with OPTIMIZE TABLE ... FINAL across all relevant destination partitions. Then compare partition counts with the expected old-table counts minus legitimately expired rows, using the controlled checks described above. Production traffic should go to the destination only when the results match.

A managed service that keeps hot data in an NVMe cache mesh rather than routing every merge through S3 reduces merge latency and the opportunity for stale bounds to accumulate. Isolated, read-only compute for agentic workloads, with full query logging, also supplies an audit trail when anomalies emerge. Neither advantage replaces the cutover gate. The durable operating principle is to treat TTL as eventually consistent and require explicit evidence before trusting the new table.