The industry talks a good game about value of data. But most organisations can only deliver use cases at a cost deficit.  

The cost of acquiring, processing and storing the data is greater than the value to resell it or that it brings the organisation. 

A recent large financial market data organisation recently told me that for every $1 they spend on purchasing data, they spend 8x on getting it to a state that it can be sold back to their customers.

For organisations like Babylon Health whose mission it is to drive affordable AI-drive healthcare to everyone one the planet based on data, reduce data management costs is a critical requirement to their business model.

Cloud data platforms from the likes of AWS MSK and Microsoft Azure HDInsight have helped. But compute and storage is just a part of the cost. Having worked on Big Data projects in financial services most of my career here are the biggest areas to focus to remove this deficit.

Expensive skills on mundane tasks

Many of the most basic data workloads require highly skilled engineers.  As an expensive fast data engineer and architect, I would spend 80% of my time configuring infrastructure rather than focusing on the business value. Business, cares about delivery of projects. 

Lack of future proofing & tech debt

Most Data Platform leaders expect data technologies to remain in place between five and ten years.  In reality, this ends up being less than three. We end up having to pay to maintain multiple different technologies with custom tooling and scripts and spend months rewriting applications to work across different stacks. 

Operational and security friction

Projects and applications end up sitting on a shelf and never getting to production. How will it be deployed? How could the pipeline be secured?  How can we give access to data to troubleshoot whilst meeting compliance? These are valid questions that require answers to get sign off. Until they are in production, they are typically not generating value. 

DataOps to simplify the data journey

We created Lenses as a DataOps layer to fit an organisation's existing data infrastructure. Technologies such as Apache Kafka and Kubernetes.  

The easier and faster we can get streaming data to production, the more value it will deliver. 

Our customers range from Babylon Health, Generali, Daimler and Zopa. 

We help get streaming data to production faster, increase value and reduce costs in a few different ways: 

1. By providing secure self-service access to streaming data. This allows developers to troubleshoot faster and puts data directly into the hands of those that understand the data. Playtika benefit from over 300 hours per day in increased engineering productivity. 
2. By providing a layer of security, governance and monitoring over streaming data and platforms. This addresses compliance, data privacy and data ethics concerns - accelerating projects to production. 
3. We lower the bar required to build and operate streaming data applications by using skills that already exist in the industry (such as SQL) and allowing an entire application landscape to be managed through configuration via GitOps. Organisations such as Vortexa see a 95% acceleration in their data pipelines getting to production.

All this over any Kafka and Kubernetes environments whether self-managed or cloud services such as AWS MSK and Azure HDInsight. 

On Thursday, we’re presenting at BigDataLDN 2019 on how we’re helping them scale engineering and deliver affordable data-driven healthcare to every person on earth. 

Article by lenses.io