Building a Big Data analytical solution is somewhat like building your own house. Surely, building a house according to one’s design would be the most customized option. It is a dream come true. But more important criteria are undermined when dreaming. As with analytics, building by oneself doesn’t mean carrying the bricks and laying the tiles on the roof. But one must choose the right professionals and know enough about the project in order to supervise and manage it well. The single, most common thing in common to both cases of building, is that one has no idea what the true and final cost will be.
In this blog post we will unveil the true cost of building a Big Data analytics solution. The expenses are divided to three main categories: infrastructure, software and human resources, the latter being the most demanding.
Here are the bottom line costs per month (annually in the bottom row):
[table id=79 /]
Infrastructure and Software
The infrastructure of an analytics solution consists of data storage, servers, network and monitoring tools. All costs are proportional to the platform’s size.
The major software expense when building analytics is the analytical database. Using 3TB per month as an example, based on leading platform providers, an analytical database is likely to cost more than $134,000. Additional required tools are an ETL (Extraction, Transform and Load) or Hadoop, real-time database and visualization tools – presenting the data graphically so that it can be shared with other people, make changes, adding and shifting data around.
In our case of 3TB per month, infrastructure and software amount close to $180,000. Buying an end-to end-solution not only includes this cost, but also saves the time and effort invested in evaluating, testing, deploying and integrating through a long process of trial and error.
Having said all this, the most significant cost of building a Big Data analytics solution is human resources. The solution is complex, requires real know-how and involves numerous specialists. All need to be engineers who are experienced with Big Data, which is a rather scarce resource nowadays, and an expensive one at that. A partial list of the experts the system requires: ETL developers, cloud infra experts, Java/Python developers, database administrators (DBAs), data analysts, dashboard developers and so forth. All in all, a system for 3TB per month requires about eight data engineers at a cost of roughly $800,000. An acquired solution cuts these costs down substantially.
Equally important is that a bought solution allows staff to keep on doing their job. Building a solution, on the other hand, diverts company personnel to a field of expertise that is not their innate domain. This harms not only the analytics solution, but company revenue as well due to time, money and human resources getting diverted away from a core business. This opportunity cost of a built solution must be taken into account.
All the above infrastructure costs are based on price quotes from common cloud vendors and software costs which are made readily available online by most software providers in the market. What these costs don’t take into account is revenue lost due to a lag in time to market, 24/7 support and the risk of failure or downtime.
The true cost of building a solution doesn’t present itself at the beginning. Like building your own house, the depths of your pockets are challenged throughout the process. Companies are prone to making expensive mistakes with analytics when Big Data is not their core competence.
On the other hand, buying an end-to-end solution means the scope and cost are clear from beginning to end. An acquired solution is managed by experts in analytics and BI. It is scalable, up to date with upgrades and innovations, shortens time to market and all at a fraction of the cost of building a solution from the ground up.