Cross-Department Data Sharing

Secure APIs Rplus built on a central government department's data estate, serving two other government departments.

Data moving both ways

Rplus built REST APIs on top of the Data Services Platform, exposed to other government organisations.

The part that matters most is the direction of travel. Data services are built on synchronous and asynchronous APIs that both send information to and receive it from other government organisations and local government. Information flows in both directions across organisational boundaries — one department shares data with the department holding the platform, which in turn shares data with another.

Two APIs, two questions

The first is configured using the WS02 enterprise platform for integrating APIs, applications and web services locally and across the various applications.

The estate behind it

The platform holds data from more than 140 source systems, plus over 100 further data sources from local government and other government organisations.

It is built on Hadoop, processes data using Spark and stores it in NoSQL databases. Those platforms are exposed using Apache LIVY, which supports synchronous and asynchronous APIs.

Running a service other organisations consume

Integrating new components onto the platform includes the rollout and management of the various hurdles that come with a service, whether public or private, and the different methods of integration used by consumers. Three questions get worked through every time: what security is required, how versioning is handled, and how the service is documented and supported.

APIs adhere to OpenAPI specifications and support formats including spreadsheets, CSV, JSON and Parquet. We work to the microservice and service-oriented architecture approaches set out in the Government Digital Service Manual.

Event-driven architecture

We implemented event-driven architecture APIs on the platform, enabling data linkage between datasets and keeping that data up to date. Alongside REST we developed SOAP services and microservices, working with the Microsoft BI stack including SSRS, SSIS and Power BI Service, over MS SQL, MySQL and Oracle.

These automate legislation or policy changes by creating variable parameters that can be configured as required — so a rule change becomes a configuration rather than a release. The prototype was tested for more than six months before implementation.

Elsewhere on the platform

The same approach serves the department's own services. Apply for NINO verifies submitted address proof using a unique property reference number, linked to the citizen's information, and checks passport and visa status through another department's REST APIs. REST APIs built on the big data platform are also consumed directly by web and mobile applications.