Data Engineering

One of the biggest challenges in building modern data products is working with live data from different domains within the enterprise, and dealing with its processing, scale, governance and security. With the constant change of an enterprise’s data landscape, the proliferation of data sources, and the diversity of use case requirements, it is a challenge for organizations to nimbly roll out AI services to meet business needs.

Much like the transition from monolithic apps to an assembly of microservices, the concept of a data mesh approach attempts to address the limitations of monolithic or central data architectures by turning to a distributed data architecture where each domains handles the required data processing, governance and connectivity to other domains or applications.

The Iguazio feature store and MLRun open source framework are some of the first solutions that support data mesh architecture principles, where data assets are broken into domains (called feature sets within Iguazio) and each domain at the helm of connecting, transforming, cataloging, governing, and serving of its data. The feature store further extends the data mesh architecture with feature vectors, which join data automatically from multiple domains in real-time or batch and address the seamless integration with ML applications.

The feature store is implemented using a self-served serverless and microservices architecture (over Kubernetes), providing elastic scaling, self-healing, costs reduction, and continuous operation.