Why is Data Ingestion Difficult?
February 18, 2019
Building an automated data ingestion system seems like a very simple task. You just read the data from some source system and write it to the destination system. You run this same process every day. And voila, you are done.
Better yet, there must exist some good frameworks which make this even simpler, without even writing any code. You just make a few clicks, enter the name of the source, tell the system where to write and you can move on with your life.
Unfortunately, the reality is very different. I have never seen any well performing real life data ingestion system that didn’t required a significant amount of expert effort to build.
But why? What makes ingestion so hard?