Skip to main content

← All solutions

Apache Druid™ upgrade assistance and migration services

· 9 min read
Calum Miller
Director

1. Executive summary​

Upgrading open source Apache Druid™ means protecting the analytical service your business already depends on. Streaming ingestion, historical data, custom extensions and business-critical dashboards all need to work together after the change.

Millersoft provides specialist Apache Druid™ upgrade assistance, including assessments of older installations, incremental upgrade planning, large-volume data migration and reconciliation between environments. Our experience also covers high-availability deployments on AWS, Kubernetes operations and maintaining customised open source codebases.

Whether you need to bring an ageing cluster forward or plan routine maintenance of an established platform, we can help define the upgrade path, rehearse it and validate the result.

Millersoft Apache Druid™ upgrade services: assess the current cluster, rehearse the version-specific upgrade path, carry out a controlled rollout, and reconcile data and workloads. Check Java, configuration and extensions; test representative queries and streaming ingestion; monitor segment availability and ingestion lag; validate historical totals and dashboard results. Agree recovery checkpoints before production changes. (opens the full-size image in a new tab)

2. Why an Apache Druid™ upgrade needs specialist planning​

An upgrade crosses several boundaries: software versions, runtime dependencies, distributed services, ingestion tasks and the applications querying the cluster. A successful process starts with the workload and the deployment you have today.

Older versions and accumulated configuration​

A cluster that has missed several releases may carry settings or ingestion patterns that have changed in later versions. Apache's upgrade notes explicitly require reviewing the intermediate releases when crossing multiple versions. Java requirements and removed settings are part of that review. Read the Apache Druid™ upgrade notes.

Millersoft can assess the starting position, identify prerequisites and determine whether the route requires intermediate deployments or another migration approach. The plan should explain each required change and how it will be verified.

Streaming ingestion and operational continuity​

Kafka or Kinesis ingestion introduces its own timing constraints. A cluster can answer historical queries while new data falls behind, so service health alone is insufficient evidence of a successful upgrade.

We plan how running tasks will be handled, confirm source retention and recovery options, and compare ingestion lag, failures and freshness before and after each stage. Batch ingestion, compaction and multi-stage query tasks also need representative checks where they are used.

Query behaviour and business reporting​

A query that completes successfully can still produce a changed result. Null handling, type behaviour, query context and application assumptions deserve attention alongside latency.

For example, Apache Druid™ 32 removed legacy settings for non-ANSI null and boolean behaviour. Existing queries may need changes when crossing that boundary. See the version 32 upgrade notes.

Millersoft can assemble a regression set from important SQL queries, native queries and Apache Superset™ dashboards. We compare both results and performance against an agreed baseline, including the expected tolerances for approximate calculations.

Custom extensions and maintained forks​

Authentication integrations, custom extensions, patches and client libraries can make an installation different from a standard distribution. Those dependencies need an explicit compatibility review.

We can assess the changes your organisation carries, determine which are still required, and test the integration against the chosen target. An upgrade is also an opportunity to remove patches that upstream functionality has superseded.

3. Experience behind the service​

Millersoft's Apache Druid™ professional services include incremental upgrades of older installations and migration of large data volumes with reconciliation between versions. Our work spans ingestion and query tuning, high-availability AWS installations, Kubernetes and monitoring with AWS CloudWatch or Prometheus.

The client testimonial on that service page describes our work with Nexi Group on multi-billion-row analytical data cubes, Kafka-fed ingestion and customised Apache Druid™ and Apache Superset™ codebases. It also describes maintaining those forks by rebasing upstream changes.

“To keep the modified software forks updated Millersoft have rebased mainline changesets into our forks.”

Kyle Milnes, former Nets CTO, in the client testimonial on our Druid™ services page.

That experience matters when an upgrade involves the surrounding analytics platform as well as the database. We can review connectors, dashboards, security integrations and operational monitoring as part of the same engagement.

Our published technical work also includes Druid™ multi-stage query integration with Apache Superset™ and investigating distinct-count results in Druid™ SQL.

4. How Millersoft delivers a controlled upgrade​

Assess the deployment and agree the target​

We establish the current release, service topology, runtime, extensions, metadata database, deep storage, ingestion sources and downstream consumers. We then select a production release based on compatibility and workload requirements, checking its published release and upgrade notes.

The output is a scoped plan with prerequisites, test criteria, responsibilities and an agreed maintenance or cutover approach. A release candidate should be treated separately from a validated production target.

Rehearse with representative data and workloads​

We test the proposed route in an isolated environment using representative segments, ingestion specifications and queries. Sensitive data access and any production dependencies are agreed before the rehearsal.

For large datasets, the rehearsal can combine a manageable functional sample with selected performance tests that reflect production scale. A small test proves only the behaviour and capacity it actually exercises.

Choose the appropriate rollout strategy​

A rolling upgrade can be suitable when the version path, topology and available capacity support it. Apache documents a service update order and task-handling strategies; the precise runbook must follow the guidance applicable to the versions and deployment involved. See Apache's rolling update guidance.

A separate target cluster and controlled cutover may be more appropriate when infrastructure changes, isolation requirements or migration constraints make an in-place approach unsuitable. Parallel environments also need a plan for ingestion ownership, source offsets, data freshness and traffic routing.

Continuous availability is an objective to validate during planning and rehearsal. We agree any expected interruption rather than promise a universal zero-downtime upgrade.

Establish recovery checkpoints​

Before production changes, we agree backup and restore coverage for metadata, configuration, deployment artefacts and required segment data, together with stream replay prerequisites. The runbook identifies who can stop the rollout and what evidence triggers that decision.

Recovery may require a restore or cutover to another environment. Reinstalling older binaries is not automatically sufficient after metadata, configuration or data-format changes. Any proposed downgrade route must be checked for the exact versions involved.

Reconcile, observe and hand over​

We validate the upgraded platform against the agreed baseline, investigate differences and document the outcome. Monitoring continues through an agreed observation period before the previous environment or recovery resources are retired.

5. What we validate before calling the upgrade complete​

AreaExample acceptance evidence
Historical dataExpected time coverage, segment availability and business totals for agreed intervals
Streaming ingestionHealthy supervisors and tasks, acceptable lag, freshness and processing-error levels
Batch and maintenance tasksRepresentative ingestion, compaction and multi-stage query jobs complete as expected
Query resultsCritical SQL and native queries agree with the baseline or have an approved explanation
PerformanceLatency, throughput and resource use meet the agreed workload criteria
Security and integrationsAuthentication, authorisation, service credentials and client connections behave as intended
DashboardsImportant Apache Superset™ charts, filters and reporting workflows pass regression checks
Recovery and operationsRecovery procedure rehearsed, alerts checked and operating documentation handed over

Reconciliation must reflect how the data is modelled. A physical row count alone may be misleading for rolled-up datasets, approximate aggregates or an actively changing ingestion window. We define comparable intervals and measures before evaluating differences.

6. Upgrade services and ongoing support​

Platform assessment
Review the current Apache Druid™ release, architecture, configuration, extensions and workload dependencies to identify the changes an upgrade will require.
Upgrade planning and implementation
Prepare a version-specific runbook, test it in a representative environment and carry out the agreed staged rollout or cluster cutover.
Data migration and reconciliation
Plan movement of historical data and compare agreed measures across environments, with traceable explanations for any differences.
Ingestion and query optimisation
Review Kafka or Kinesis ingestion, task capacity, segment layout and important queries alongside the upgrade baseline.
Custom integration support
Assess maintained forks, extensions, security integrations and Apache Superset™ connectivity against the chosen target release.
Operational support
Provide monitoring, incident investigation, maintenance planning and ongoing assistance for your open source Apache Druid™ deployment.

Start with a technical assessment of the existing cluster. Useful inputs include the current version, deployment method, approximate data volume, ingestion sources, custom extensions and the business workflows that must remain available.

Millersoft can turn that assessment into a practical upgrade proposal covering the target release, prerequisites, rehearsal, data validation, production rollout and support after the change. Where the cluster is several releases behind, we identify the intermediate changes before committing to a delivery sequence.

Further information​

Open Source Support on Tap

Talk to Millersoft about upgrading your open source Apache Druid™ installation. We can help assess the starting point, plan the migration and validate the platform your business depends on.

Discuss your Druid™ upgrade

Apache Druid™ and Apache Superset™ are trademarks of the Apache Software Foundation. Millersoft is not affiliated with the Apache Software Foundation.