Skip to main content

← All solutions

Apache Druid™ data repair: profiling and deep storage recovery

· 11 min read
Calum Miller
Director

1. When the cluster is running but the data cannot be trusted​

Incorrect Apache Druid™ data can remain hidden behind a healthy dashboard. A replayed batch can inflate revenue, a transformation can put events in the wrong day, or a damaged segment can leave part of the historical record unavailable.

Millersoft helps investigate and correct these problems by combining data profiling, segment-level diagnosis and reconciliation to trusted sources. We identify the affected datasources and time ranges, establish the cause, and plan a repair that preserves valid records and can be verified afterwards.

Our experience with large Apache Druid™ datasets, ingestion pipelines, customised deployments and migration reconciliation provides the foundation for this work. The objective is to restore an explainable analytical result and address the fault that produced it.

Millersoft Apache Druid™ data repair workflow: profile data by time, dimensions and source totals; classify physical corruption, loading failures or incorrect records; restore verified segment data or rebuild from trusted sources; then reconcile and monitor. In an illustrative replay incident, 105,000 stored events are compared with 100,000 trusted source events. Profiling identifies 5,000 duplicate events in one hour, but a daily partition requires rebuilding the complete affected day while retaining its valid records. (opens the full-size image in a new tab)

2. Deep storage corruption and incorrect data need different repairs​

Deep storage holds Apache Druid™ segment files. Historical processes load segments for serving queries, while metadata records describe the segment inventory. Checking only a running server or a storage bucket does not establish that the complete dataset is correct. See Apache's deep storage overview.

We distinguish the failure before selecting a remedy:

ProblemTypical evidenceDirection of investigation
Missing or physically damaged segmentMissing object, failed download, unreadable archive or segment decoding errorStorage access, object integrity, backups and surviving verified copies
Segment loading or visibility failureExpected interval unavailable, load errors, access failures or inconsistent deployment settingsLoad rules, capacity, credentials, encryption access, metadata and runtime compatibility
Logically incorrect dataQueries succeed but values, timestamps, dimensions or totals are wrongIngestion rules, source changes, replay history and transformation logic
Apparent discrepancy caused by query semanticsDifferent filters, rollup assumptions, approximate counts or changed lookup valuesQuery definitions and data modelling before rewriting stored data

An unavailable segment is a finding to investigate, not proof of corruption. Likewise, a complete set of readable segment files can contain a perfectly preserved copy of incorrectly ingested data.

3. How we identify problematic areas through profiling​

Start with business totals and narrow the time range​

We compare an agreed baseline with hourly or daily profiles: event volumes, financial totals, minimum and maximum timestamps, null rates, dimension distributions and unexpected values. Results are split by relevant business dimensions such as customer, product, currency or source system.

The comparison needs a stable cut-off. We account for late arrivals, source corrections and the same timezone boundaries before declaring a difference. Sampling helps prioritise investigation, but final acceptance uses agreed reconciliation measures over the full repair scope.

Inspect the segment inventory and loading evidence​

We correlate suspect intervals with segment identifiers, versions, partitions, publication state and serving availability. Task reports and Historical logs help distinguish a bad object from a permissions failure or an incomplete load.

The following read-only example lists published, non-overshadowed segments that are not currently served. Replace the illustrative datasource name and adapt the query to the installed release:

SELECT
segment_id,
"start",
"end",
version,
partition_num,
is_available
FROM sys.segments
WHERE datasource = 'transactions'
AND is_published = 1
AND is_overshadowed = 0
AND is_available = 0
ORDER BY "start", partition_num
LIMIT 100

A result can reflect loading transitions or intentional deep-storage-only placement. It becomes actionable when compared with the intended serving rules. Metadata row counts can also be unknown or lag behind streaming ingestion. See Apache's system-table definitions.

Profile the content, not only its availability​

For a readable datasource, a bounded query can reveal where a measure changes unexpectedly. This example assumes unrolled transaction rows and an amount field expressed consistently within each currency:

SELECT
TIME_FLOOR(__time, 'PT1H') AS hour_start,
currency,
COUNT(*) AS stored_rows,
SUM(amount) AS total_amount,
SUM(CASE WHEN customer_id IS NULL THEN 1 ELSE 0 END)
AS rows_without_customer
FROM transactions
WHERE __time >= TIMESTAMP '2026-10-01 00:00:00'
AND __time < TIMESTAMP '2026-10-02 00:00:00'
GROUP BY 1, 2
ORDER BY 1, 2

Use an explicit query timezone, such as UTC, and match it to the source comparison. A rolled-up datasource needs its retained event-count metric and aggregation rules; COUNT(*) counts stored rows, not necessarily original events. Distinct-count checks must also distinguish exact counts from approximate estimates.

We then investigate the suspect slice: repeated event identities, missing business keys, timestamp shifts, incorrect units, parsing failures or unexpected dimension combinations. Duplicate detection requires a meaningful event key and enough retained detail to distinguish a replay from legitimate repeated activity.

Examine individual segments when necessary​

For physical diagnosis, we inspect copies of suspect objects in an isolated environment using compatible tooling and extensions. We compare expected object identity, size and available integrity evidence with trusted copies, and test whether the segment can be read.

Apache's dump-segment tool can expose segment metadata and contents for debugging. Its output is not necessarily a full-fidelity representation, particularly for complex metrics, so a successful dump is neither complete business validation nor an automatic recovery source. Read the tool's documented limitations.

4. Techniques we use to repair faulty data​

Restore a verified original when segment files are damaged​

Where a known-good backup or retained object version exists, we can plan restoration of the original segment data and a controlled reload. The candidate must match the expected segment identity, version and format, and be readable with the required extensions.

If only a local segment cache survives, we assess it as a recovery candidate rather than assume it is an authoritative backup. Access, encryption or loading faults may instead require an operational correction without changing the segment contents.

When no trustworthy segment copy remains, we establish whether retained source files or stream history can rebuild the affected interval. Missing bytes cannot be reconstructed reliably from metadata alone.

Re-ingest trusted source records​

Rebuilding from authoritative input is often the strongest repair for duplicates, omitted events or incorrect transformations. We correct the ingestion logic, define the source cut-off and rebuild the complete replacement scope with the intended schema, timestamp handling, rollup and metrics.

For stream replay, source retention, offsets and the rules for identifying already ingested events determine what can be recovered. Blindly replaying into the existing datasource can duplicate good records. We stage and reconcile the reconstructed data before publication.

Reindex readable data with corrected transformations​

Where existing rows are readable and retain the information needed for correction, we can rebuild them using reviewed expressions, filters or enrichment. Examples include correcting a unit conversion or normalising a malformed dimension.

Apache Druid™ supports time-range replacement and reindexing through SQL REPLACE or native batch ingestion. We publish replacement segments through those supported mechanisms rather than edit values inside published binary files. See Apache's data update documentation.

Reindexing cannot recover detail that rollup discarded or fill missing records without another source. Compaction can improve segment layout, but it does not infer correct prices, remove arbitrary business duplicates or repair an invalid transformation automatically.

Match the replacement scope to the partitioning​

We identify the smallest supported replacement interval for the datasource and preserve every valid record within it. Finding a faulty hour does not mean only that hour can be supplied to a daily replacement job.

The repair specification records interval boundaries, expected output and treatment of empty periods. Timestamp corrections may require rebuilding both the incorrectly populated interval and the intended destination interval. We avoid a datasource-wide overwrite when a bounded repair is sufficient.

Coordinate streaming ingestion and publication​

Late events, streaming tasks and automatic compaction can overlap the repair window. We plan how those writers are paused, isolated or coordinated, including the handling of records arriving during validation.

Some releases and configurations support concurrent append and replace, subject to locking and granularity constraints. We check the installed version and workload before relying on that behaviour. See Apache's concurrent append and replace guidance.

Replacement publication operates by time chunk. A repair spanning several chunks needs explicit completion checks; it should not be treated as one transaction covering the whole datasource.

5. Worked example: tracing and removing a replayed batch​

Consider an illustrative, unrolled transaction datasource with daily segment granularity. Its trusted daily source contains 100,000 events with a total value of £10 million. A replay adds 5,000 duplicate events worth £500,000, all timestamped within one hour.

Profile measureBefore repairTrusted source and repair target
Event count for the day105,000100,000
Total transaction value£10,500,000£10,000,000
Excess events traced to the replay5,0000
Replacement scopeComplete affected UTC dayPreserve valid records in every hour

These are illustrative figures, not a client incident or a claimed project result.

Our investigation first isolates the hourly spike, then compares event identities and task history to confirm replay rather than legitimate business growth. We correct the replay process and build a candidate from the trusted source for the complete day.

Before publication, we compare event counts, currency totals and key-level differences, including records outside the faulty hour. After publication, we rerun the same checks against the served datasource and verify that subsequent ingestion does not recreate the duplicates.

6. Millersoft's experience and approach​

Our Apache Druid™ services cover data-loss prevention, ingestion and query tuning, large-volume migration with reconciliation, and support for high-availability deployments. The published client testimonial describes our work on multi-billion-row analytical cubes, Kafka-fed ingestion and customised open source platforms.

Our technical articles also show the importance of examining result correctness, including investigating distinct-count discrepancies in Druid™ SQL. We bring that combination of operational and analytical investigation to data repair: establish the evidence, isolate the fault and prove the corrected result.

Data and segment profiling
Identify abnormal intervals, missing coverage, suspect dimensions and differences from source-system totals.
Root-cause investigation
Connect the affected records to ingestion jobs, transformation changes, replay activity, storage failures and query assumptions.
Controlled recovery and rebuilding
Assess trustworthy segment copies and source history, then rehearse the selected restore, replay or reindexing approach.
Reconciliation and prevention
Validate the repaired data, document the evidence and introduce checks that detect recurrence.

7. Prove the repair and prevent a repeat​

Before making changes, we agree the repair boundaries, preserve required metadata and segment evidence, and verify the recovery route. Relevant retention and automatic cleanup settings need review so the evidence is not removed during the work.

Acceptance includes business totals, event coverage, agreed duplicate and null checks, segment readability, loading state and critical query results. We record the original segment set, source cut-off, repair specification, task identifiers and before-and-after results.

Old segment versions and required backups remain protected for the agreed recovery period. Marking a segment unused and permanently deleting its deep-storage objects are different operations; kill tasks and storage lifecycle policies must be considered before retiring recovery material. See Apache's data deletion lifecycle.

Finally, we address the cause: replay controls, ingestion validation, transformation tests, source retention or storage monitoring. Profile baselines and alerts can then expose the next unexpected change before it reaches a reporting deadline.

Further reading​

Make your Druid™ data trustworthy again

Talk to Millersoft about suspected corruption, missing intervals or incorrect analytical results. We can help profile the affected data, assess recovery options and plan a repair with measurable reconciliation.

Discuss a data repair assessment

Apache Druid™ is a trademark of the Apache Software Foundation. Millersoft is not affiliated with the Apache Software Foundation.