Designing a Cloud-Agnostic Data Lakehouse: A Reference Architecture for Public Sector
The data lakehouse is now the default architecture for serious analytics workloads. It combines the cost profile and flexibility of a data lake with the performance and governance of a data warehouse, on top of cheap object storage rather than expensive proprietary engines.
For most public sector teams the question is no longer whether to adopt one. It is how to design a lakehouse that does not quietly become a single-cloud monoculture in disguise.
What follows is the reference architecture we recommend, the trade-off at each layer, and the decisions that separate a genuinely portable platform from a nominally portable one.
The lock-in is already priced in
This is not a theoretical risk. In April 2024 the Cabinet Office's Central Digital and Data Office acknowledged, in a document reported by The Register, that the UK government's ability to negotiate cloud contracts has been inhibited by vendor lock-in — and warned of risk concentration across departments.
The contracts behind that warning are public.
Three departments, one provider, over a billion pounds. Whatever the merits of each individual contract, the aggregate is a negotiating position — and it is not the buyer's.
Four codebases, or one
The practical cost of cloud-native design is rarely the licence. It is that each provider's services demand their own implementation, and an organisation running workloads on more than one cloud ends up maintaining a separate codebase for each.
Four codebases means four sets of version requirements, four skill profiles to recruit for, teams siloed by implementation rather than by domain, and every change delivered four times. A single portable codebase collapses all of that into one — and the saving compounds with every release, not just at migration.
Seven layers, seven portability questions
A modern lakehouse decomposes into seven layers. Each one has a portability question attached, and they do not all have the same answer.
Row three is the fork in the road. Everything else can be worked around at some cost.
Where the lock-in sits
Assessed across a full lakehouse stack rather than service by service, the four major providers and the two agnostic platforms fall out like this.
| Platform | Lock-in | Why |
|---|---|---|
| AWS | High | Heavy use of proprietary services; EMR and Glue tightly tied to AWS |
| Azure | High | Deep integration with ADLS, Synapse and Azure ML |
| GCP | Medium | Some open components — Dataproc, BigQuery SQL — but still Google-optimised |
| Oracle Cloud | High | Strong focus on Oracle-native databases and pipelines |
| Databricks | Low | Built to operate across multiple clouds on open standards |
| Snowflake | Low | Built to operate across multiple clouds on open standards |
A high lock-in rating is not a verdict on quality. AWS, Azure, GCP and Oracle Cloud all run excellent lakehouse services, and for a single-cloud organisation with no intention of moving, native is often the faster and cheaper route.
The rating measures one thing only: what it would cost you to leave.
The medallion pattern, briefly
Most well-designed lakehouses organise data into three logical zones. Each is materialised as tables in your chosen open format, governed by your chosen catalog, with Spark or an equivalent runtime doing the work between them.
Reading data across clouds
Once data lives in more than one place, you have to decide how a query reaches it. There are two patterns, they are routinely confused, and they behave differently under load.
Federation reads via remote compute, so the work happens where the data already sits and only the answer crosses the network. That keeps egress down and avoids duplication.
Virtualisation abstracts away the complexity of the underlying systems so consumers see a single surface, typically over JDBC. It is the better fit when you need to hide a messy estate from its users.
Do not use the virtualisation layer to perform transformations. It exists to present data, not to reshape it. Transformations belong in the processing layer, where they can be versioned, tested and rerun.
Multi-cloud or hybrid — the real reasons
Organisations rarely end up on more than one cloud because an architect drew it that way. They end up there for reasons that are commercial, regulatory or simply historical — and the honest ones are worth naming.
Why multi-cloud
- Business continuity. Regulatory requirements and enterprise policy that mandate more than one provider.
- Historical evolution. Business units chose different clouds; mergers brought others with them.
- Perceived cost. A different cost model looks favourable — and multi-cloud increases negotiating power with the hyperscalers.
- Lock-in. Avoiding dependence on a single provider as a deliberate strategy.
- Features. Some capabilities simply exist on one cloud and not another.
- Data sovereignty. Regulatory requirements to hold data in a specific country.
Why hybrid
- Co-existence. Stepwise migration — gold layer first to generate value in cloud, pipelines afterwards.
- Data that cannot move. Some data carries too much risk to store in cloud at all.
- Region availability. No cloud in-region, and services from other regions not permitted.
- Legacy investment. Substantial existing infrastructure that still has life in it.
- OPEX versus CAPEX. Maintaining a balance between the two.
- Volume and scale. Data too large to move — and cost models where cloud ends up more expensive than on-premise.
That last point deserves emphasis, because it cuts against the grain of most cloud advocacy. At sufficient scale, the economics can favour on-premise. A portable architecture is what lets you act on that finding instead of discovering it too late to do anything about it.
Designing for portability from day one
Three principles separate genuinely portable lakehouses from nominally portable ones.
Decouple aggressively
Storage, compute, catalog and orchestration should each be replaceable independently. If your orchestrator hard-codes storage paths, or your transformation code embeds catalog identifiers, you have quietly recoupled the layers.
Governance as the spine
A unified catalog spanning clouds is what makes federation, sharing and migration tractable. Fragment governance per provider and every cross-cloud action becomes a manual reconciliation.
Open by default
Every proposal for a proprietary format or service carries the burden of justifying the lock-in. New tables go into Delta or Iceberg. New transformations go into Spark or Snowpark. Exceptions are documented.
It is not free, and pretending otherwise does nobody a favour. Cross-cloud observability does not exist out of the box — monitoring every environment through a single pane of glass means a custom implementation over shared system tables, and even then the data updates through the day rather than in real time.
Egress charges apply whenever data genuinely has to cross a boundary. Multi-cloud storage services exist specifically to avoid them — Cloudflare R2, Faction and Wasabi among them — and no egress fee is their principal selling argument.
For a worked example of this architecture built for a central government department, see One Codebase, Five Clouds.
Rplus Analytics builds cloud-agnostic data platforms for UK central government departments. If you would like a portability assessment of your current architecture, get in touch.