It doesn’t take a genius to understand that the data warehouses and data marts of the past were of little use. Their data was typically too old, the processing too cumbersome, and the costs too high.
Today’s cloud-based data analytics have the ability to do things in real time, databases can operate at the “speed of need,” and even small enterprises can bind data analytics processing with the latest “cool kids” technology such as machine learning and predictive algorithms.
I don’t want to rain on this parade, but it turns out the path to cloud-based data analytics is a longer and harder road than many enterprises projected. As a result, failures are beginning to come up on my radar as IT encounters cost overruns, the technology fails to meet expectations, and just the sheer volume of data proves problematic. Here’s why.
First, the transfer of data from enterprises to the public cloud is a bigger chore than anticipated, exacerbated by the largely manual nature of the work. AWS, Google, Microsoft, and others have their own technology for doing this, such as AWS’s Snowball. However, even with such tools, going through the processes to move several petabytes of data is tricky, to say the least.
Second, data integration is still an issue in the cloud; moving the data doesn’t magically solve your integration challenges. Also, systems of record may still remain on premises, and so need to be synced with the data now stored in the cloud in a timely manner to get up-to-date results. This means using a mix of old and new data-integration technologies and setting up processes that include data movement and structure transformation.
Finally, the cloud-based analytics databases themselves are complex and difficult to configure. Some of that complexity is due to the security subsystems in the database; these are necessary but must be figured out in the context of the database and data analytics. This security must also be systemic with the rest of the systems the data analytics systems touch, both in the cloud and on premises—and that can mean most of the other operational systems that need to feed analytics in real time.
Although these cloud analytics challenges can all be overcome, it’s up to IT to understand the level of effort may actually be an 8 out of 10, when it thought (or more likely was told) that it would be a 5 out of 10.
As a result, be prepared for projects to take longer, budgets to deplete faster, and the (perceived) failures to rise due to these issues.