A trend I'm excited for this year: DataOps & the Analytical Engineer
~10 years ago DevOps was born. The role of system admins and developers merged. Infrastructure became self-serve
Today the role of data engineers and business analysts are merging. Data is becoming self-serve
Data infrastructure is becoming so powerful that the tools today allow non-technical folks to carry out the once complicated / custom code/ huge backlog jobs of data engineers.
Before getting into what this means, let's first discuss how we got here
Before 2012 the data world was dominated by transactional (OLTP) databases like PostgreSQL, MySQL, etc and analytical (OLAP) databases like Oracle, Netezza
Tools like Informatica / Talend were used to batch load (ETL) data into these databases, Tableau used to visualize
As you can imagine, there was heavy engineering work to manage the environment...
Then in 2013 AWS released their cloud data warehouse Redshift, and it was a game changer. Snowflake was founded in 2012, but didn't really pick up steam until a few years later (around 2016)
So why was Redshift a big deal?
1. It was the first cloud-native OLAP warehouse. It reduced the TCO of an OLAP database by orders of magnitude.
2. Speed of processing analytical queries increased dramatically
