Apache Druid™ data repair: profiling and deep storage recovery
1. When the cluster is running but the data cannot be trusted
Incorrect Apache Druid™ data can remain hidden behind a healthy dashboard. A replayed batch can inflate revenue, a transformation can put events in the wrong day, or a damaged segment can leave part of the historical record unavailable.
Millersoft helps investigate and correct these problems by combining data profiling, segment-level diagnosis and reconciliation to trusted sources. We identify the affected datasources and time ranges, establish the cause, and plan a repair that preserves valid records and can be verified afterwards.
Our experience with large Apache Druid™ datasets, ingestion pipelines, customised deployments and migration reconciliation provides the foundation for this work. The objective is to restore an explainable analytical result and address the fault that produced it.
2. Deep storage corruption and incorrect data need different repairs
Deep storage holds Apache Druid™ segment files. Historical processes load segments for serving queries, while metadata records describe the segment inventory. Checking only a running server or a storage bucket does not establish that the complete dataset is correct. See Apache's deep storage overview.
We distinguish the failure before selecting a remedy:
| Problem | Typical evidence | Direction of investigation |
|---|---|---|
| Missing or physically damaged segment | Missing object, failed download, unreadable archive or segment decoding error | Storage access, object integrity, backups and surviving verified copies |
| Segment loading or visibility failure | Expected interval unavailable, load errors, access failures or inconsistent deployment settings | Load rules, capacity, credentials, encryption access, metadata and runtime compatibility |
| Logically incorrect data | Queries succeed but values, timestamps, dimensions or totals are wrong | Ingestion rules, source changes, replay history and transformation logic |
| Apparent discrepancy caused by query semantics | Different filters, rollup assumptions, approximate counts or changed lookup values | Query definitions and data modelling before rewriting stored data |
An unavailable segment is a finding to investigate, not proof of corruption. Likewise, a complete set of readable segment files can contain a perfectly preserved copy of incorrectly ingested data.
3. How we identify problematic areas through profiling
Start with business totals and narrow the time range
We compare an agreed baseline with hourly or daily profiles: event volumes, financial totals, minimum and maximum timestamps, null rates, dimension distributions and unexpected values. Results are split by relevant business dimensions such as customer, product, currency or source system.
The comparison needs a stable cut-off. We account for late arrivals, source corrections and the same timezone boundaries before declaring a difference. Sampling helps prioritise investigation, but final acceptance uses agreed reconciliation measures over the full repair scope.
Inspect the segment inventory and loading evidence
We correlate suspect intervals with segment identifiers, versions, partitions, publication state and serving availability. Task reports and Historical logs help distinguish a bad object from a permissions failure or an incomplete load.
The following read-only example lists published, non-overshadowed segments that are not currently served. Replace the illustrative datasource name and adapt the query to the installed release:
SELECT
segment_id,
"start",
"end",
version,
partition_num,
is_available
FROM sys.segments
WHERE datasource = 'transactions'
AND is_published = 1
AND is_overshadowed = 0
AND is_available = 0
ORDER BY "start", partition_num
LIMIT 100
A result can reflect loading transitions or intentional deep-storage-only placement. It becomes actionable when compared with the intended serving rules. Metadata row counts can also be unknown or lag behind streaming ingestion. See Apache's system-table definitions.
Profile the content, not only its availability
For a readable datasource, a bounded query can reveal where a measure changes unexpectedly. This example assumes unrolled transaction rows and an amount field expressed consistently within each currency:
SELECT
TIME_FLOOR(__time, 'PT1H') AS hour_start,
currency,
COUNT(*) AS stored_rows,
SUM(amount) AS total_amount,
SUM(CASE WHEN customer_id IS NULL THEN 1 ELSE 0 END)
AS rows_without_customer
FROM transactions
WHERE __time >= TIMESTAMP '2026-10-01 00:00:00'
AND __time < TIMESTAMP '2026-10-02 00:00:00'
GROUP BY 1, 2
ORDER BY 1, 2
Use an explicit query timezone, such as UTC, and match it to the source comparison. A rolled-up datasource needs its retained event-count metric and aggregation rules; COUNT(*) counts stored rows, not necessarily original events. Distinct-count checks must also distinguish exact counts from approximate estimates.
We then investigate the suspect slice: repeated event identities, missing business keys, timestamp shifts, incorrect units, parsing failures or unexpected dimension combinations. Duplicate detection requires a meaningful event key and enough retained detail to distinguish a replay from legitimate repeated activity.
Examine individual segments when necessary
For physical diagnosis, we inspect copies of suspect objects in an isolated environment using compatible tooling and extensions. We compare expected object identity, size and available integrity evidence with trusted copies, and test whether the segment can be read.
Apache's dump-segment tool can expose segment metadata and contents for debugging. Its output is not necessarily a full-fidelity representation, particularly for complex metrics, so a successful dump is neither complete business validation nor an automatic recovery source. Read the tool's documented limitations.
4. Techniques we use to repair faulty data
Restore a verified original when segment files are damaged
Where a known-good backup or retained object version exists, we can plan restoration of the original segment data and a controlled reload. The candidate must match the expected segment identity, version and format, and be readable with the required extensions.
If only a local segment cache survives, we assess it as a recovery candidate rather than assume it is an authoritative backup. Access, encryption or loading faults may instead require an operational correction without changing the segment contents.
When no trustworthy segment copy remains, we establish whether retained source files or stream history can rebuild the affected interval. Missing bytes cannot be reconstructed reliably from metadata alone.
Re-ingest trusted source records
Rebuilding from authoritative input is often the strongest repair for duplicates, omitted events or incorrect transformations. We correct the ingestion logic, define the source cut-off and rebuild the complete replacement scope with the intended schema, timestamp handling, rollup and metrics.
For stream replay, source retention, offsets and the rules for identifying already ingested events determine what can be recovered. Blindly replaying into the existing datasource can duplicate good records. We stage and reconcile the reconstructed data before publication.
Reindex readable data with corrected transformations
Where existing rows are readable and retain the information needed for correction, we can rebuild them using reviewed expressions, filters or enrichment. Examples include correcting a unit conversion or normalising a malformed dimension.
Apache Druid™ supports time-range replacement and reindexing through SQL REPLACE or native batch ingestion. We publish replacement segments through those supported mechanisms rather than edit values inside published binary files. See Apache's data update documentation.
Reindexing cannot recover detail that rollup discarded or fill missing records without another source. Compaction can improve segment layout, but it does not infer correct prices, remove arbitrary business duplicates or repair an invalid transformation automatically.
Match the replacement scope to the partitioning
We identify the smallest supported replacement interval for the datasource and preserve every valid record within it. Finding a faulty hour does not mean only that hour can be supplied to a daily replacement job.
The repair specification records interval boundaries, expected output and treatment of empty periods. Timestamp corrections may require rebuilding both the incorrectly populated interval and the intended destination interval. We avoid a datasource-wide overwrite when a bounded repair is sufficient.
Coordinate streaming ingestion and publication
Late events, streaming tasks and automatic compaction can overlap the repair window. We plan how those writers are paused, isolated or coordinated, including the handling of records arriving during validation.
Some releases and configurations support concurrent append and replace, subject to locking and granularity constraints. We check the installed version and workload before relying on that behaviour. See Apache's concurrent append and replace guidance.
Replacement publication operates by time chunk. A repair spanning several chunks needs explicit completion checks; it should not be treated as one transaction covering the whole datasource.
5. Worked example: tracing and removing a replayed batch
Consider an illustrative, unrolled transaction datasource with daily segment granularity. Its trusted daily source contains 100,000 events with a total value of £10 million. A replay adds 5,000 duplicate events worth £500,000, all timestamped within one hour.
| Profile measure | Before repair | Trusted source and repair target |
|---|---|---|
| Event count for the day | 105,000 | 100,000 |
| Total transaction value | £10,500,000 | £10,000,000 |
| Excess events traced to the replay | 5,000 | 0 |
| Replacement scope | Complete affected UTC day | Preserve valid records in every hour |
These are illustrative figures, not a client incident or a claimed project result.
Our investigation first isolates the hourly spike, then compares event identities and task history to confirm replay rather than legitimate business growth. We correct the replay process and build a candidate from the trusted source for the complete day.
Before publication, we compare event counts, currency totals and key-level differences, including records outside the faulty hour. After publication, we rerun the same checks against the served datasource and verify that subsequent ingestion does not recreate the duplicates.
6. Millersoft's experience and approach
Our Apache Druid™ services cover data-loss prevention, ingestion and query tuning, large-volume migration with reconciliation, and support for high-availability deployments. The published client testimonial describes our work on multi-billion-row analytical cubes, Kafka-fed ingestion and customised open source platforms.
Our technical articles also show the importance of examining result correctness, including investigating distinct-count discrepancies in Druid™ SQL. We bring that combination of operational and analytical investigation to data repair: establish the evidence, isolate the fault and prove the corrected result.
7. Prove the repair and prevent a repeat
Before making changes, we agree the repair boundaries, preserve required metadata and segment evidence, and verify the recovery route. Relevant retention and automatic cleanup settings need review so the evidence is not removed during the work.
Acceptance includes business totals, event coverage, agreed duplicate and null checks, segment readability, loading state and critical query results. We record the original segment set, source cut-off, repair specification, task identifiers and before-and-after results.
Old segment versions and required backups remain protected for the agreed recovery period. Marking a segment unused and permanently deleting its deep-storage objects are different operations; kill tasks and storage lifecycle policies must be considered before retiring recovery material. See Apache's data deletion lifecycle.
Finally, we address the cause: replay controls, ingestion validation, transformation tests, source retention or storage monitoring. Profile baselines and alerts can then expose the next unexpected change before it reaches a reporting deadline.
Further reading
- Millersoft Apache Druid™ cloud cost optimisation
- Millersoft Apache Druid™ upgrade assistance
- Millersoft Apache Druid™ services and support
- Apache Druid™ deep storage architecture
- Apache Druid™ data updates and reindexing
- Apache Druid™ SQL system tables
Make your Druid™ data trustworthy again
Talk to Millersoft about suspected corruption, missing intervals or incorrect analytical results. We can help profile the affected data, assess recovery options and plan a repair with measurable reconciliation.
Discuss a data repair assessmentApache Druid™ is a trademark of the Apache Software Foundation. Millersoft is not affiliated with the Apache Software Foundation.
