How to Unlock Value from Data

The old cliche, ‘Data is the New Oil’, is even more relevant now in the age of AI revolution as it was in the age of Big Data revolution

Interestingly it was Google which triggered three most important revolutions in the data world:

• Map reduce paper by Jeff Dean & Sanjay Ghemawat in 2004 spawned Hadoop ecosystem and the big data revolution
• Big Table paper by Jeff Dean, Sanjay Ghemawat & others in 2006 spawned multiple NoSQL databases including Google’s BigTable and the NoSQL revolution
• “Attention is all you need” by Ashish Vaswani et al in 2017 introduced a new deep learning architecture known as transformer and triggered the AI revolution, which ironicaly was first capitalized by OpenAI

High Level Trend in Data Usage

• Most compnaies collect and store the basic data need to run their business. This is done using transactuonal database such as Oracle, Postgres, SQL Server etc
• Overtime different divisions within the company start collecting their own data using disparate transactional systems essentialing creating transactional silos
• The internet further added fuel to this fire when everyone and their cousins started collecting more and more data through new internet enabled business lanes
• The big data revolution was a direct consequence of the need to manage this data chaos and multiple silos. You could almost draw a straight line from the…