Digital transformation has accelerated significantly in the last two years, but the extra demands on data scientists have revealed significant barriers to effective working and high levels of job dissatisfaction in some areas.
Theese are findings of a survey of data scientists by analytics company SAS, which also found that around four in 10 are dissatisfied with their company’s use of analytics and model deployment, while more than 20 barriers to effective working emerged.
However, the work of data scientists has grown in importance – with many organisations accelerating digital transformation projects by using technology to improve business operations. More than 90 per cent of respondents to the survey indicated the importance of their work was the same or greater compared to before the pandemic.
The report assesses the impact of the pandemic, challenges faced, and overall satisfaction with the analytics environment. The research showed the pandemic upended standard business practices, shifting the assumptions and variables in models and predictive algorithms, and caused a ripple effect of adaptations in processes, practices, and operating parameters.
More than two-thirds of respondents were satisfied with the outcomes from analytical projects. However, 42 per cent of data scientists were dissatisfied with their company’s use of analytics and model deployment, suggesting a problem with how analytical insights are used by organisations to inform decision-making. This was backed up by 42 per cent saying data science results were not used by business decision-makers, making it one of the main barriers faced.
The survey also highlighted some specific skills gaps. Less than a third of the respondents reported having advanced or expert proficiency in program-heavy skills, such as cloud management and database administration. This is an issue given that use of cloud services is up significantly, with 94 per cent saying they experienced the same or greater use of cloud since Covid-19 struck.
'There have clearly been more demands placed on data scientists as the pandemic has accelerated digital transformation projects that many organisations were planning anyway,' said Iain Brown, head of data science, for SAS UK & Ireland. 'A major source of frustration is finding a way for organisations to implement the insights from analytics projects and use them in their decision-making, which means giving data scientists a seat at the boardroom table might be a way forward.
'Linked to this, we found concerns around support for data science teams and a lack of talent, which has been an issue for some time with demand outstripping supply. Organisations must realise that investing in a team of data scientists with complementary skills could reap huge value for the business, so the cost of hiring needs to consider the return on that investment as we move to significantly more digital and AI-driven business processes.'
The research also identified gaps in consistent organisational emphasis on AI ethics, with 43 per cent of respondents indicating that their organisation does not conduct specific reviews of its analytical processes with respect to bias and discrimination and only 26% of respondents reporting that unfair bias is used as a measure of model success in their organisation.
When it comes to the challenges identified to ensure fair and unbiased decision-making, Sally Eaves, an industry expert, said: 'Data scientists can lend their expertise to craft working guidelines for data access, usage security, and broader issues, such as sustainability and data ethics and bias.
'Rather than sometimes hoping they are given appropriate, clean data and relying too much on the technology to drive fair outcomes, they can play an active role to put in place the right guidelines and checks at each stage of the analytical process to try and eliminate bias. Having a transparent and explainable flow from data to decision is obviously key to this.'
