Immuta has announced the findings from its 2022 State of Data Engineering Survey, which examined the changing landscape of data engineering and emerging challenges as companies shift data management workloads to the cloud. 

Organizations are increasingly adopting multiple cloud technologies to keep up with the scale, speed, and use cases required by modern data teams. Nearly two-thirds (65 percent) of respondents characterized their company as either 100 percent cloud-based or primarily cloud-based, indicating a large market need for automated cloud data access control.

Additionally, 81 percent of organizations expect to be fully or primarily cloud-based within the next 24 months, an increase from 71 percent in last year’s survey and a sign that cloud adoption is accelerating.

To gather these findings, Immuta and Gradient Flow surveyed nearly 400 data leaders, architects, engineers, and consumers across industries, organization sizes, and geographies.

The survey revealed that as data becomes more complex and data rules and regulations expand and evolve, new challenges such as data privacy, security, and quality have emerged that threaten to hinder data initiatives. 

Further, the survey revealed that sensitive data use is on the rise, and the tasks associated with managing sensitive data - data cataloguing, data discovery, and access control - are the most challenging. Increasing rules and regulations further complicate data management as new data platforms and BI/analytics tools are adopted. 

Key findings include:

• Shift to cloud computing: Two-thirds (65 percent) of respondents characterized their company as already either 100 percent cloud-based or primarily cloud-based. Looking ahead 12-24 months, 81 percent of respondents projected they will be 100 percent cloud-based or primarily cloud-based.

• Need for data privacy and security: Similarly, 64 percent of respondents came from companies that already collect and store sensitive data, while 88 percent indicated their organizations are subject to one or more data use rules or regulations, with GDPR, HIPAA, CCPA, and SOC 2 cited as the most common.

• Challenge of data quality: Respondents cited data quality and validation as the most challenging area they face today, with 27 percent indicating they were unsure what (if any) data quality solution their organization is using.

• Rise of data catalogues and data discovery: Six in ten organizations are now using data catalogue and data discovery tools.

• Accelerated adoption of cloud data platforms: More than six in ten (62 percent) respondents plan to adopt at least one of the following five cloud databases and platforms in the next 12-24 months (in order of priority): Amazon Athena, Google BigQuery, Databricks, Snowflake, and Amazon Redshift. The biggest mover compared to the 2021 Impact Report was Google BigQuery, which moved up 5 spots this year.

• Top BI and analytics tools: The most popular BI and analytics solutions cited by survey respondents were (in order of priority): Jupyter Notebooks, Tableau, Microsoft Power BI, Looker, and Google Colab.

The report includes year-over-year comparisons to findings from Immuta’s 2020 Data Engineering Survey and 2021 Impact Report, which took place during September and October 2020. Click here to read the 2022 Data Engineering Survey report.