Research conducted by Oxylabs, a company specialising in web data gathering, has revealed that 50% of financial services organisations manage web data collection internally, with over a third (38%) combine internal practices and outsourcing to third parties.
In addition, just over one in ten (11%) outsource data collection completely and among the challenges associated with web scraping, ensuring data quality tops the list (42%). Meanwhile, 41% find managing and processing large datasets challenging.
The survey found that completely outsourcing data collection is the least popular method, while a hybrid option of combining internal web scraping practices and outsourcing was more popular.
The findings also demonstrate that web scraping still remains a complicated and intricate practice among organisations, which needs close oversight, particularly in heavily regulated industries such as the financial services sector.
Other challenges, such as finding the most efficient tools, ensuring a consistent data flow, getting real-time data, or finding reliable partners to outsource to, are equally distributed and similarly important for decision-makers in financial services.
Aleksandras Šulženko, product owner at Oxylabs, said: “Such a low preference for outsourcing is a little surprising, but it’s clear that better trust in web scraping has to be developed. We know that web scraping has become an essential part of financial services, yet so few companies trust third-party services.
“The findings indicate a hidden issue with how web scraping is understood and what goals it can achieve, bringing those third-party companies back to the drawing board. Outsourcers need to better communicate how their services work and how their methods are secure and compliant, which is one of a number of challenges the industry is facing at the moment.”
They added: “If organisations are to begin putting their trust in third parties, these parties need to communicate how they can tailor collection and analysis more efficiently than an internal team as well as how they can tackle the issues many face when implementing web scraping. For example, outsourcing to a third-party web scraping solutions provider is an easy option that requires minimal start-up costs and the data acquired can be integrated almost immediately.
“Web scraping produces a significant amount of value not only to financial organisations but all companies, especially if that information can be utilised effectively. While a third-party web scraping solutions provider is the easier and cheaper option for organisations, it seems the majority prefer to look internally and will continue to do so until the challenges around web scraping are addressed.”