Cloud has enabled big data processing at scale Virtually unlimited storage and processing capacity in cloud No longer limited by size and CPUs of database servers Can ask questions to the data which were not possible pre-cloud e.g. — In the Nielsen era the TV ratings were derived from a…

1. Separation of Storage from Compute The foundational data within an organization should not be tied with a proprietary database or technology. Flexibility is a great enabler and cloud provides many choices that were previously not available on-prem. Within big data realm, it is best practice to store the data…

Data Lakes & Warehouses have been around for a while, and there is a lot of literature & theory on what works and what doesn’t. I have been around a few of these and below is my attempt to distill the design & best practices into few simplified pointers What…

Say you are enterprising data engineer working for an upwardly mobile and tech savvy company. …

Software Engineering has come a long way from the dark days of top down management, waterfall approach, MS Project, Gantt chart (shiver!), bloated projects, arbitrary deadlines and insane delivery schedules. The modern software industry has embraced a bottom up build of projects, hands on technical management, transparency, agile methodology the…

Data strategy is the bedrock of any BI Analytics solution, in particular we are guided by the following principles Data Storage Flexible Storage — Data storage should be such that it minimizes data movement and/or copy. …

Designing a data pipeline can be a serious business, building it for a Big Data based universe, however, can increase the complexity manifolds. The tools and concepts around Big Data started evolving around early 2000s as the size & speed of internet exploded. Companies suddenly found themselves having to deal…

(Root of all data engineering evil lies in the unnecessarily complex data processing pipeline) Historical Context Designing a data pipeline can be a serious business, building it for a Big Data based universe, however, can increase the complexity manifolds. The tools and concepts around Big Data started evolving around early…