Overcoming challenges of data mesh

How can you overcome the challenges and avoid falling into the cybersecurity trap?

Summary highlight

Has your organisation adopted the data mesh approach? Perhaps you are considering its implementation, yet are unsure about the roadblocks you may face? In this article, we share the common challenges that we see our clients face when adopting data mesh, how to overcome them and what security factors to consider when starting your data mesh journey.

Moving away from centralised data architectures

Over the last decade organisations have relied on centralised data architectures, such as data lakes and warehouses, for managing their data assets. What was once the go-to practice for organisations, the centralised data approach has fallen short when it comes to deriving real business value and delivering data-driven insights. According to research, an overwhelming 70% of businesses struggle to unlock the value of their data to make meaningful business decisions. As organisations move more towards data-driven business models, they are realising that centralised architectures are failing to keep up with business demands, lacking in real-time data access, reliability and integrity, while failing to provide the necessary contextual understanding from relevant and important business domains.

The decentralised data approach such as data mesh is a newer concept that has captured the attention of businesses for its ability to overcome the limitations of centralised approaches. It introduces a greater level of flexibility, agility and scalability that is needed for greater business insights and remaining competitive in today's fast-changing world. While data mesh certainly offers a promising approach for efficient data management, its implementation is not as straight forward as first appears. The journey to data mesh adoption can be muddied by a combination of technical and cultural factors, undermining the efficacy and value of the approach. To get the most out of their data and the decentralised approach, organisations must put in place a plan to overcome the challenges of its implementation. But what are the factors that organisations should watch out for?

Challenges of adopting data mesh

Shifting to a decentralised data approach is no easy feat. This is because data mesh is more than an architectural change, but a cultural change that requires domains to assume responsibility of their own data for the first time. NashTech's experts have drilled down into the challenges that organisations can expect to face:

Cultural transformation barriers

Data mesh relies on four main principles: data as a product, domain-oriented ownership, self-served data infrastructure and federated data governance. Adopting data mesh principles necessitates a significant cultural shift towards embracing autonomy, collaboration and a product mindset. For data mesh to work effectively it requires a major collective shift in perspective, changing viewing data as a mere resource to a product that generates profits and drives important business decisions. When unprepared, domains can be left struggling to adapt to the new domain-ownership model, lacking resources and the relevant training to maintain the quality of their responsible datasets.

Learn more about the principles of data mesh here: https://www.nashtechglobal.com/our-thinking/insights/unleashing-the-power-of-data-mesh/

Change management

Implementing data mesh often requires changes to existing processes, roles and responsibilities, which can encounter resistance and require effective change management strategies. For example, consider the case where the infrastructure team, traditionally responsible for data warehousing (DWH), now provides the underlying infrastructure to a Blackbox that holds the data domains. They are no longer responsible for the data itself, marking a fundamental shift from the traditional centralised data management approach.

Technical and integration complexities

The challenges of data mesh also go beyond organisational culture and its people. Data mesh introduces technical challenges, such as designing scalable data infrastructure, ensuring interoperability between domains and integrating with existing systems and tools. Data products across an organisation need to communicate and integrate together effectively to ensure up to date insights across the entire business. What's more, standardisation surrounding data terminologies and usage must be applied and adhered to across domains to maintain data quality and integrity across autonomous teams.

Data silos and quality

Since domains are now responsible for their own data, we run into the danger of data fragmentation, declining quality and silos if governance is not enforced effectively across the organisation. The downside to this is that data may be left incomplete, inconsistent, duplicated or simply inaccurate, taking organisations back to where they started.

Overcoming challenges: considerations for successful implementation

Transitioning to the data mesh model doesn't have to be a process ridden with challenges. When carefully planned, organisations can avoid technical and process-driven complexities to enable its smoother adoption. But how can they achieve this?

For an organisation to successfully implement data mesh, organisations should consider the following six steps:

1. Define data domains

To make the data mesh model work, organisation's need to carefully think about those who are responsible for ensuring the quality, maintenance and security of data. Identifying distinct domains within the organisation based on business capabilities and assigning ownership to domain teams will enable the organisation to align on a data governance process and ensure access rights are given to the right parties.

2. Address data duplication and standardisation

It's easy to face duplication challenges when multiple domains share the same data. Ensuring that a standardised approach is executed across the organisation will enable domain teams to understand, share and access data cross-systems, while enforcing data consistency and quality.

3. Establish data product teams

Data can be tricky to manage and maintain without the appropriate training or knowledge of data management initiatives. Creating cross-functional teams with expertise in data engineering, data science and domain knowledge will enable employees to take full ownership of their data products to drive insights.

4. Design a self-serve data infrastructure

One of the fundamental principles of the data mesh approach is the self-serve nature of the infrastructure involved. Data platforms should be void of technical complexity, easy to navigate and accessible by non-technical domain users. Providing domain teams with the tools, platforms and frameworks they need to build is critical for managing their data products independently.

5. Foster a data-driven culture

As we have mentioned, adopting the data mesh approach demands a significant change in an organisation's overall culture, processes and mindset surrounding data ownership and accountability. It creates new user roles, requires additional training on new concepts and technologies and increases workloads for its employees. Promoting a culture of data literacy, collaboration and continuous learning to empower domain teams and encourage data-driven decision-making reduces the risks of employees falling back into their old data mindsets and rejecting new processes.

6. Measure success and performance

Monitoring data quality, availability and usage will shed light on the value and potential training gaps of an organisation's data programme. Assigning KPI's and goals can help to measure the efficacy of new product teams.

Maintaining security within data decentralisation

Data decentralisation can lead to an increase in security holes among organisations should they fail to put in place the right protections. This is because data no longer resides in one singular and central location but rather it is dispersed among differing domain products. As the pool of users that have access to sensitive data increases, the potential risk of cyber entry points also increase.

To secure decentralised data effectively, organisations should consider data access controls, encryption, monitoring of data activities and regular auditing and assessment.

A security plan may look like this:

1. Implementing access controls

Does your organisation follow a zero trust model? Implementing robust access controls and authentication mechanisms ensures that only authorised individuals can access sensitive data, reducing cyber entry points and data leakage.

2. Data encryption

Encrypting data both in transit and at rest protects against unauthorised access. If the data falls into the wrong hands, attackers will be unable to read it without the right encryption keys.

3. Monitoring data flows and activities

Monitoring data flows and activities within the data mesh allows organisations to detect and respond to unusual patterns of behaviour that may signify a security incident is taking place and stop it in its tracks.

4. Regularly auditing and assessing

Regularly auditing and assessing the security posture of the decentralised data ecosystem identifies vulnerabilities and addresses them proactively. Automation can speed this process along, ensuring fast detection of potential security breaches.

Discover NashTech's security testing and penetration services here.

How we can help you

At NashTech, we understand that data mesh is not just a concept; it is a strategic approach to modern data management and a pathway to digital transformation. Our expertise lies in guiding organisations on their data mesh journey, helping them navigate the complexities and reap the benefits of its approach.

Our approach involves:

• Tools for data products: we provide innovative tools that enable data collections, data events and data analytics, empowering your organisation to create valuable data products.
• Decentralised data architectures: we specialise in creating decentralised, distributed data architectures that support multi-cloud and hybrid cloud computing, aligning with the principles of data mesh.
• Event-driven data solutions: transition from batch-oriented, static, centralised data to event-driven data ledgers and streaming-centric pipelines, enabling real-time data events and timely analytics.
• Federated data governance: our solutions include strong federated data governance models, ensuring data quality, compliance and security across your data mesh.
• Self-service tooling: user-friendly, self-service tools designed for non-technical users, making data management within your data mesh accessible and efficient.

Partner with NashTech on your data mesh journey, and together, we can revolutionise the way your organisation manages and leverages data in the modern era. Contact us to embark on this transformative path towards data excellence.