Multi-Cloud Data Federation: Sharing Data Without Moving It

Multi-Cloud Data Federation: Sharing Data Without Moving It

By Rplus Analytics · Data Engineering · Multi-Cloud

For years, the default answer to "how do I get data from cloud A to cloud B?" was simple and expensive: copy it. ETL pipelines, scheduled jobs, replication tools, egress fees, freshness lag. Two copies of every important dataset, one of which was always slightly out of date, both of which had to be governed independently, neither of which was cheap.

Data federation changes the question. Instead of asking where the data lives, federation asks who needs to query it and routes the query rather than the data. For UK public sector platforms — where datasets are large, egress is metered, and data sovereignty rules can prohibit movement entirely — this is a meaningful architectural shift. (PRJ8500)

Three patterns, three trade-offs

It's worth being precise about what "federation" actually means, because three distinct patterns get bundled under the same word.

Cross-region (same cloud). Compute in region 1 reads data sitting in region 2 of the same cloud provider. Latency is moderate, egress costs are usually intra-cloud and reasonable, and the security model is unified. Useful for disaster recovery and regional consolidation.

Cross-cloud. Compute in cloud A reads data sitting in cloud B. Latency is higher, egress fees are real and metered per gigabyte, and the security model has to bridge two IAM systems. Useful when departments or business units have ended up on different clouds and consolidation isn't politically or technically feasible.

Hybrid (cloud to on-premise). Compute in cloud reads data still resident on-premise — typically because the data is too large to move, too sensitive to move, or both. The classic case in central government: legacy systems holding decades of records that cannot, for regulatory reasons, leave a specific data centre.

Each pattern has different cost, latency, and governance implications, and the architectural answer is different for each. Lumping them together leads to designs that work well for one and badly for the others.

Federation versus virtualisation

A second distinction worth getting right.

Federation pushes the query to where the data lives. The remote engine executes the query, returns the result set, and the local engine joins it with local data. The work happens close to the data, which keeps egress costs and latency manageable for selective queries.

Virtualisation pulls data through a virtualisation layer that abstracts the underlying systems. Queries appear to run against a single virtual database, but the layer is responsible for translating and routing them. Convenient, but if you start using the virtualisation layer to perform transformations, you've created a new bottleneck and a new lock-in point.

Our consistent advice: use virtualisation for access, not for transformation. The moment you're doing real ETL through a virtualisation layer, you've reinvented the problem you were trying to avoid.

Delta Sharing and Snowflake Data Sharing

The two open(ish) protocols that matter for cross-cloud federation today are Delta Sharing (open protocol, originated by Databricks) and Snowflake Data Sharing (Snowflake-native, with multi-cloud reach).

Both let a data producer expose specific tables to a specific consumer without copying the underlying files. The consumer queries through their own engine; the producer governs through theirs. Audit trails are preserved on both sides. Egress applies, but only on the data actually returned by queries — not on full-table replication.

For UK public sector, this is the architectural primitive that makes cross-departmental data sharing tractable at scale. Department A can expose a curated Gold-layer table to Department B without provisioning shared infrastructure, without bilateral data copies, and without losing visibility over who queried what. The Data Sharing Agreement governs the relationship; the technical layer enforces it. (DSA)

When federation is the right answer (and when it isn't)

Federation is the right answer when:

• The data is large and changes frequently, making replication expensive and stale
• Data sovereignty rules prohibit movement
• Multiple consumers need access but only some need full copies
• Cross-cloud or cross-departmental queries are selective rather than exhaustive

Federation is the wrong answer when:

• Queries are routinely full-scan against the remote dataset (egress will dominate the bill)
• Local processing needs millisecond latency
• The remote system can't sustain the query load federation would impose
• The transformation logic is complex enough that materialising a local copy is genuinely simpler

The honest answer for most platforms is "both" — federation for the long tail of cross-system queries, materialised copies for the hot path of high-volume use cases. Designing the boundary between the two is where the real architectural work happens.

A note on observability

One subtlety worth flagging. Cross-cloud observability is not free. Most platforms — Databricks included — don't ship out-of-the-box single-pane-of-glass monitoring across clouds. The standard pattern is to use the platform's own data sharing primitives (Delta Sharing of system tables, in the Databricks case) to consolidate audit and usage data into a single catalog for monitoring.

It works, but it's a custom implementation, not a feature you toggle on. Budget for it.

---

Rplus Analytics designs federated data architectures for UK public sector departments and arms-length bodies. If you'd like to discuss a federation strategy for your estate, get in touch.