Designing a Cloud-Agnostic Data Lakehouse: A Reference Architecture for Public Sector
By Rplus Analytics · Architecture · Data Engineering
The data lakehouse is now the default architecture for serious analytics workloads. It combines the cost profile and flexibility of a data lake with the performance and governance of a data warehouse, and it does this on top of cheap object storage rather than expensive proprietary engines.
The question for most public sector teams is no longer whether to adopt a lakehouse. It's how to design one that doesn't quietly become a single-cloud monoculture in disguise. This post walks through the reference architecture we recommend for cloud-agnostic lakehouses, the trade-offs at each layer, and the design decisions that determine whether a platform is genuinely portable or just nominally so.
The seven layers
A modern lakehouse can be decomposed into seven layers, each with a portability question attached.
1. Ingestion. How does data get in? Cloud-native services (Glue, Data Factory, Dataflow, GoldenGate) are powerful but provider-specific. Cloud-agnostic alternatives (Informatica IDMC, Talend, open-source connectors) ingest the same sources but produce pipelines that survive a provider switch. (PRJ8500)
2. Storage. Object storage is the great equaliser. S3, ADLS Gen2, GCS, and OCI Object Storage all expose effectively the same primitives. The portability question here isn't the storage service — it's the format you write into it. Proprietary warehouse formats are a one-way door; open table formats are not.
3. Open table format. This is the single most important architectural decision in a lakehouse. Delta Lake, Apache Iceberg, and Apache Hudi all provide ACID transactions, schema evolution, and time travel on top of Parquet files in object storage. Pick one based on your processing engine and ecosystem maturity, and write everything new into it.
4. Catalog and governance. Provider-native catalogs (Glue Data Catalog, Purview, Dataplex, OCI Data Catalog) are convenient but trap your metadata. Cross-cloud governance layers (Unity Catalog, Snowflake Horizon, Informatica Data Governance) preserve lineage, classifications, and access policies when workloads move.
5. Processing and transformation. Spark is the closest thing to a portable transformation runtime. Whether it runs as Databricks, Snowpark, EMR, Synapse, or Dataproc, the user-facing code can be largely the same. Provider-specific transformation services (Athena, BigQuery SQL extensions, Redshift stored procedures) are the layer where lock-in tends to creep back in.
6. Serving and BI. Power BI, Tableau, Looker, Sigma, QuickSight — most enterprise BI tools can connect to most lakehouse engines. The portability question is whether your semantic model and row-level security rules live in the BI tool or in the lakehouse layer below it. The latter travels; the former does not. (DSA)
7. ML and AI. SageMaker, Azure ML, Vertex AI, and OCI Data Science are all capable platforms, but model artefacts trained on one rarely deploy cleanly to another. Open frameworks (MLflow, ONNX, plain PyTorch and scikit-learn) keep models portable.
The medallion pattern, briefly
Most well-designed lakehouses organise data into three logical zones:
- Bronze — raw ingested data, immutable, schema-on-read, kept for replay and audit
- Silver — filtered, cleaned, joined, and conformed to a canonical model
- Gold — business-level aggregates, ready for reporting, ML features, and serving
Each layer is materialised as tables in your chosen open format, governed by your chosen catalog. The transformations between layers are where Spark, Snowpark, or your equivalent runtime does the heavy lifting.
The beauty of medallion is that it's engine-agnostic. The same Bronze/Silver/Gold structure works on Databricks, Snowflake, or any Spark-compatible runtime, which means the data stays portable even if the engine changes.
Two cloud-agnostic platform choices
For most public sector clients, the practical choice narrows to two cloud-agnostic platforms:
| Capability | Cloud-Agnostic (Databricks) | Cloud-Agnostic (Snowflake) |
|---|---|---|
| Storage format | Delta Lake on cloud object storage | Snowflake-managed on cloud object storage (multi-cloud) |
| Processing | Spark, Photon, Delta Live Tables | Snowpark, native SQL engine |
| Governance | Unity Catalog | Snowflake Horizon / Governance |
| Warehouse | Databricks SQL | Snowflake |
| ML/AI | MLflow, Mosaic AI | Snowflake Cortex, Snowpark ML |
| BI | Power BI, Tableau, Sigma | Power BI, Tableau, Sigma |
| Lock-in level | Low | Low |
Both run on AWS, Azure, GCP, and (with caveats) Oracle Cloud. Both support cross-cloud data sharing as a first-class feature. The choice between them tends to come down to existing skill base, ML workload weight, and which BI ecosystem your end users are already in.
Designing for portability from day one
Three design principles tend to separate genuinely portable lakehouses from nominally portable ones:
Decouple aggressively. Storage, compute, catalog, and orchestration should each be replaceable independently. If your orchestrator hard-codes assumptions about your storage paths, or your transformation code embeds catalog identifiers, you've recoupled the layers.
Treat governance as the spine. A unified catalog and governance layer that spans clouds is what makes federation, sharing, and migration tractable. If governance is fragmented per provider, every cross-cloud action becomes a manual reconciliation.
Open formats by default, proprietary by exception. Every time someone proposes a proprietary format or service, the burden of proof should be on them to justify the lock-in. New tables go into Delta or Iceberg. New transformations go into Spark or Snowpark. Exceptions are documented.
Rplus Analytics has built data lakehouse platforms for UK central government departments since 2012. If you'd like a portability assessment of your current architecture, get in touch.