Before getting into what this means, let's first discuss how we got here
A trend I'm excited for this year: DataOps & the Analytical Engineer
~10 years ago DevOps was born. The role of system admins and developers merged. Infrastructure became self-serve
Today the role of data engineers and business analysts are merging. Data is becoming self-serve
Before getting into what this means, let's first discuss how we got here
Tools like Informatica / Talend were used to batch load (ETL) data into these databases, Tableau used to visualize
Then in 2013 AWS released their cloud data warehouse Redshift, and it was a game changer. Snowflake was founded in 2012, but didn't really pick up steam until a few years later (around 2016)
1. It was the first cloud-native OLAP warehouse. It reduced the TCO of an OLAP database by orders of magnitude.
2. Speed of processing analytical queries increased dramatically
What does this all mean? An EXPLOSION of data
Now, after this point we still weren't ready for DataOps / Analytical Engineers. What did it take to get there?
1. Data Movement: @fivetran
2. Data storage / compute: @SnowflakeDB @awscloud @GCPcloud
3. Data Transformations: @getdbt
When data access becomes democratized and self-serve in nature, the need for new tools to manage this "modern data stack" goes up. I think we'll see a TON of huge companies built in the following categories:
2. Monitoring the quality of the data (and pipelines) will become increasingly important in self-serve settings. "Datadog for data"
More from Tech
So we had to develop technologies like this to barely manage control over limited areas in Iraq's few urban centers. Only ~8 in 100 Iraqi adults owns a personal vehicle. That rate is > 1 car/adult in America yet I have never seen any doctrine paper or work of fiction address this
We've seen and struggled in civil conflicts with instant, local, universal, distributed communications (cell phone era, basically every conflict since 2000). We've seen and struggled in conflicts with instant, global, universal distributed communications (everything since 2011).
The world's most overfunded military and glow in the dark agencies struggle and largely fail to contain conflicts where fhe vast, vast majority of people are locked into a ~5mi radius of their home.
How can they possibly contain a conflict in a nation with universal car ownership and the most developed road network in the world? The average car can travel over 400 miles on one tank of gas, how can you contain the potential of that kind of mobility?
I think that's partially why the system was so freaked out by 1/6. Yes, most of it is histrionics but you don't decide to indefinitely turn your capital into the Baghdad Green Zone with fortifications and 25k troops over histrionics alone.
Hey guys, just a friendly reminder. We're watching you. pic.twitter.com/bGwi1uJBwT
— CIA Metadata Analyst with 8 kids (@CiaKids) September 23, 2019
We've seen and struggled in civil conflicts with instant, local, universal, distributed communications (cell phone era, basically every conflict since 2000). We've seen and struggled in conflicts with instant, global, universal distributed communications (everything since 2011).
The world's most overfunded military and glow in the dark agencies struggle and largely fail to contain conflicts where fhe vast, vast majority of people are locked into a ~5mi radius of their home.
How can they possibly contain a conflict in a nation with universal car ownership and the most developed road network in the world? The average car can travel over 400 miles on one tank of gas, how can you contain the potential of that kind of mobility?
I think that's partially why the system was so freaked out by 1/6. Yes, most of it is histrionics but you don't decide to indefinitely turn your capital into the Baghdad Green Zone with fortifications and 25k troops over histrionics alone.
