Flight Readiness
Review

Absolute-timestamped query, installed_at ∈ [2026-04-21 00:00, 2026-04-22 00:00) UTC, both sides filtered campaign_id > 0. Window is ≥36 h old at query time — all attribution retries (max ~2.5 h ceiling) are exhausted. No inflight settling.

Per-source ledger · 04-21 settled

Pulling the aggregate apart. 92 distinct (project, campaign) pairs in the 04-22 window; tables below show the non-trivial deltas. Rows where PG = CH are hidden.

Per-project headline

The structural claim: PG and hazelnut each mint their own click_instance_id for the same physical click hit. To test: sample 60 random PG click UUIDs from 04-22, ask CH whether any exist in clicks_denormalized.

Clicks are not dropping. Clicks are being ingested by both systems at parity; each system stamps them with an independently generated UUID. When the SDK later sends the legacy UUID back as lr_ia_id in /api/client/init, hazelnut's Redis lookup fails because hazelnut's record lives under a different key.

// gateway/handler/web.go — line 492–498
func (h *WebHandler) processClickTracking(ctx context.Context, r *http.Request, p *clickTrackingParams) string {
    // …setup omitted…
    installAttributionID := generateUUID()                           // ← H-01 — each pipeline mints its own

    h.recordClickIfBrowser(r, p, installAttributionID)
    return installAttributionID
}

// internal/consumer/attribution/click_matcher.go — lines 98–113
if params.LrIaID != "" {
    if click := m.lookupRedis(ctx, params.LrIaID, MatchLrIaID,
                               m.clickStore.FindClickByLrIaID, params.ProjectID); click != nil {
        return click, MatchLrIaID                                 // ← succeeds only if CH's own UUID matches
    }
}
if click := m.findBestIPMatch(ctx, params.IP, params.ProjectID, params.Debug); click != nil {  // ← H-02 fallback
    return click, MatchIP
}

Setup: pull every 04-22 install for project 128 that PG attributed via click_match or ip_address (n=275), join to the same install_instance_id in CH, and classify why hazelnut did or did not rescue the install.

Kafka consumer lag on install-events-hazelnut — proved via TraceId propagation

The gateway's Kafka publish uses W3C trace-context headers which the attribution consumer extracts and continues. For any install, the same TraceId appears on gateway.init and on attribution.process. The delta between those timestamps is the precise time the message sat in install-events-hazelnut. This is not an inference; it is a join.

Specific case: 1BrwCShL3wigGhq0ItrC — gateway published 2026-04-22 07:04:10 UTC, attribution-consumer ran at 18:28:51 UTC. Same W3C TraceId, 11 h 24 m of Kafka sit-time. This is not retry — it is the message literally not being consumed by the group for 11 hours.

Attribution-consumer throughput on 04-22 varies with traffic. Per-hour attribution.process count:

H-03 exhibit — silent ASA error path (narrower than first version)

Per-hour error rate on apple_search_ads.attribute_install spans, 04-21, main attribution-consumer only:

// internal/consumer/attribution/strategies/apple.go — lines 71-85
if err != nil {
    apiSpan.RecordError(err)
    apiSpan.SetStatus(codes.Error, "apple adservices API error")
    apiSpan.End()
    // TS parity: catch all API errors and return success=false to allow fallback
    // to other strategies (e.g. click matching, organic). Don't propagate the error.
    log.Warn("apple adservices API error, falling back to other strategies",
        zap.Error(err),
        zap.String("install_instance_id", msg.Request.InstallInstanceID),
    )
    return &attribution.StrategyResult{
        Success:           false,
        AttributionSource: "apple_search_ads",
    }, nil                                     // ← no retryable error → no retry → organic
}

Narrower claim, verified

Of 129 PG-ASA installs that landed organic in CH on 128 · 04-21:

The 24-hour cliff — identified

For every apple_search_ads.attribute_install span on 04-21, joined by install_instance_id to the corresponding gateway.init to compute install-to-ASA-call delay:

// internal/consumer/attribution/strategies/apple_ads_client.go — line 139-167
func handleAppleAdsStatus(statusCode int, body []byte) error {
    switch {
    case statusCode == http.StatusOK:           return nil
    case statusCode == http.StatusBadRequest:   return &PermanentError{...}
    case statusCode == http.StatusTooManyRequests: return &RetryableError{...}
    case statusCode >= 500:                    return &RetryableError{...}
    default:                                    // ← 404 lands here
        return &PermanentError{                  // ← no retry for expired tokens
            Code:   ErrCodeStrategyFailed,
            Err:    fmt.Errorf("HTTP %d: %s", statusCode, string(body)),
            Reason: "apple adservices unexpected status",
        }
    }
}

Legacy PG avoids this entirely because its ASA call happens inline within /api/client/init — the token is always fresh (seconds old, not hours). Hazelnut's Kafka topology inserts a delay large enough to cross the 24h TTL on backlog days, and the HTTP 404 path is not classified as retryable. Both the backlog (H-02) and the 404 classification (H-03) are part of the cascade. Fixing either breaks the chain: kill the backlog and tokens stay fresh; reclassify 404 as retryable and installs within the next 24h batch can still attribute (assuming the token isn't already dead).

H-04 exhibit — installs that never reach attribution

For 25 MISSING-from-CH installs on 128 and 18 on 155, all with clear gateway and events-consumer spans, there are zero attribution-consumer spans of any kind — no attribution.process, no retry, no DLQ — across the full 48 h window from install through to probe day. The install was published to Kafka (the publish span exists), no consumer picked it up, and no retry path has fired as of probe time.

155 is the worst-affected at 22 % MISSING (18 of 81). The project's 100 %-click_match traffic means every MISSING install is attribution-fatal — there is no google_ads or ASA strategy to rescue it even if attribution-consumer catches up later. The 22 % consumer-drop rate on 155 is of a different order than 128 (3.5 %) or 193 (6.4 %), suggesting a topic-or-partition-specific failure mode that disproportionately hits 155 traffic.

The real bimodality — wall-delay (installed_at → created_at)

Trace correlation on one representative lost install (1BrwCShL3wigGhq0ItrC, installed 2026-04-22 07:04:10 UTC, attribution orchestrate at 18:28:52 UTC — 11 h 24 m later):

18:28:52.311938  attribution.redis.zrange               cache.hit=true  candidates=2
18:28:52.313462  attribution.click_matcher.try_lock_click   // candidate 1
18:28:52.313475  attribution.store.is_click_attributed
18:28:52.314204  attribution.click_matcher.acquire_and_record
18:28:52.315197  attribution.click_matcher.try_lock_click   // candidate 2  ←  first failed
18:28:52.315204  attribution.store.is_click_attributed
18:28:52.315919  attribution.click_matcher.acquire_and_record
   ...
18:28:52.317026  apple_search_ads.attribute_install     1,738 ms
18:28:54.055663  attribution.write_and_finalize         // → organic, campaign_id=0

zrange returned two IP candidates — the Redis index had them. Both try_lock_click attempts ran. The install still landed organic. That is not the failure mode "Redis is empty"; it is either IsClickAttributed returning true on both candidates, AcquireLock failing on both, or the arbiter preferring an ASA-strategy-null over the click-match result. One or all three — not identified from span data alone.

Many lost installs don't run click-matcher at all

Sampling 50 lost installs against OTel attribution-consumer spans:

// internal/consumer/attribution/orchestrator.go — line 719
if len(results) == 0 {
    log.Info("phase 9: all strategies returned non-success, will be organic",
        zap.String("install_instance_id", iid))
    return nil, nil, nil                          // ← no retry scheduled here either way
}

Remaining question worth a follow-up investigation: what causes the 5-hour median processing delay for these installs in the first place? Candidates — lagging-events parking waiting on UserIdentity / integration info, DLQ→retry cycles that terminate as app_open, or a genuine Kafka consumer backlog on a specific partition. OTel tracing gaps (29/50 installs with no consumer span) make this harder to pin down from traces alone.

The 77 click_match losses — a different beast

For the 77 PG installs PG attributed via click_match (the lr_ia_id path): 0 have a click in CH at the install-time IP, and PG has 0 clicks at those IPs either. PG matched them using the click UUID — the click physically lived at a different IP than where the app eventually opened. This is the classic deferred-install IP drift: user clicked on WiFi, installed on mobile data, opened at the carrier-NAT IP. The click UUID bridged the IPs for PG; hazelnut can't use the UUID (H-01), and IP fallback is architecturally incapable of helping when install-IP ≠ click-IP. These 77 are irreducible without a H-01 fix.

It would be convenient if the drift were a consumer outage; it is not. OTel traces on 10.1.0.33:8123 for 04-22 UTC, cross-service, show top-level spans erroring at ≤ 0.004 % on the attribution and click consumers. Retry consumer's 3.97 % error rate on attribution.retry.process tracks 1:1 with the consumer.attribution.dlq writes — retries that exhausted their budget, which is the pipeline's designed terminal state, not a failure mode.

What landed since the prior FRR, and whether it touches any of the three holds.

None of the four recent deploys target H-01 (click UUID minter), H-02 (install-events Kafka backlog), H-03 (ASA silent-error path), or H-04 (installs never reaching attribution-consumer). They are all infrastructure or data-integrity hardening, each valid on its own terms and orthogonal to the parity gap shown here.

Every material numeric claim and file:line citation from project_128_ferryscanner.md, project_155_ipf.md, and project_193_playo.md re-run against fresh data on 2026-04-23.

The parity story is binary on the structural axis. Close H-01 and the follow-on H-02 footprint shrinks; close neither and the three subsystems will stay where they are. Every subsequent deploy that does not target H-01 or H-02 will read as a No-Op in the next FRR.