Before getting into what this means, let's first discuss how we got here
A trend I'm excited for this year: DataOps & the Analytical Engineer
~10 years ago DevOps was born. The role of system admins and developers merged. Infrastructure became self-serve
Today the role of data engineers and business analysts are merging. Data is becoming self-serve
Before getting into what this means, let's first discuss how we got here
Tools like Informatica / Talend were used to batch load (ETL) data into these databases, Tableau used to visualize
Then in 2013 AWS released their cloud data warehouse Redshift, and it was a game changer. Snowflake was founded in 2012, but didn't really pick up steam until a few years later (around 2016)
1. It was the first cloud-native OLAP warehouse. It reduced the TCO of an OLAP database by orders of magnitude.
2. Speed of processing analytical queries increased dramatically
What does this all mean? An EXPLOSION of data
Now, after this point we still weren't ready for DataOps / Analytical Engineers. What did it take to get there?
1. Data Movement: @fivetran
2. Data storage / compute: @SnowflakeDB @awscloud @GCPcloud
3. Data Transformations: @getdbt
When data access becomes democratized and self-serve in nature, the need for new tools to manage this "modern data stack" goes up. I think we'll see a TON of huge companies built in the following categories:
2. Monitoring the quality of the data (and pipelines) will become increasingly important in self-serve settings. "Datadog for data"
More from Tech
There has been a lot of discussion about negative emissions technologies (NETs) lately. While we need to be skeptical of assumed planetary-scale engineering and wary of moral hazard, we also need much greater RD&D funding to keep our options open. A quick thread: 1/10
Energy system models love NETs, particularly for very rapid mitigation scenarios like 1.5C (where the alternative is zero global emissions by 2040)! More problematically, they also like tons of NETs in 2C scenarios where NETs are less essential. https://t.co/M3ACyD4cv7 2/10
In model world the math is simple: very rapid mitigation is expensive today, particularly once you get outside the power sector, and technological advancement may make later NETs cheaper than near-term mitigation after a point. 3/10
This is, of course, problematic if the aim is to ensure that particular targets (such as well-below 2C) are met; betting that a "backstop" technology that does not exist today at any meaningful scale will save the day is a hell of a moral hazard. 4/10
Many models go completely overboard with CCS, seeing a future resurgence of coal and a large part of global primary energy occurring with carbon capture. For example, here is what the MESSAGE SSP2-1.9 scenario shows: 5/10
Energy system models love NETs, particularly for very rapid mitigation scenarios like 1.5C (where the alternative is zero global emissions by 2040)! More problematically, they also like tons of NETs in 2C scenarios where NETs are less essential. https://t.co/M3ACyD4cv7 2/10
There is a lot of confusion about carbon budgets and how quickly emissions need to fall to zero to meet various warming targets. To cut through some of this morass, we can use some very simple emission pathways to explore what various targets would entail. 1/11 pic.twitter.com/Kriedtf0Ec
— Zeke Hausfather (@hausfath) September 24, 2020
In model world the math is simple: very rapid mitigation is expensive today, particularly once you get outside the power sector, and technological advancement may make later NETs cheaper than near-term mitigation after a point. 3/10
This is, of course, problematic if the aim is to ensure that particular targets (such as well-below 2C) are met; betting that a "backstop" technology that does not exist today at any meaningful scale will save the day is a hell of a moral hazard. 4/10
Many models go completely overboard with CCS, seeing a future resurgence of coal and a large part of global primary energy occurring with carbon capture. For example, here is what the MESSAGE SSP2-1.9 scenario shows: 5/10
You May Also Like
So it's now October 10, 2018 and....Rod Rosenstein is STILL not fired.
He's STILL in charge of the Mueller investigation.
He's STILL refusing to hand over the McCabe memos.
He's STILL holding up the declassification of the #SpyGate documents & their release to the public.
I love a good cover story.......
The guy had a face-to-face with El Grande Trumpo himself on Air Force One just 2 days ago. Inside just about the most secure SCIF in the world.
And Trump came out of AF1 and gave ol' Rod a big thumbs up!
And so we're right back to 'that dirty rat Rosenstein!' 2 days later.
At this point it's clear some members of Congress are either in on this and helping the cover story or they haven't got a clue and are out in the cold.
Note the conflicting stories about 'Rosenstein cancelled meeting with Congress on Oct 11!"
First, rumors surfaced of a scheduled meeting on Oct. 11 between Rosenstein & members of Congress, and Rosenstein just cancelled it.
He's STILL in charge of the Mueller investigation.
He's STILL refusing to hand over the McCabe memos.
He's STILL holding up the declassification of the #SpyGate documents & their release to the public.
I love a good cover story.......
The guy had a face-to-face with El Grande Trumpo himself on Air Force One just 2 days ago. Inside just about the most secure SCIF in the world.
And Trump came out of AF1 and gave ol' Rod a big thumbs up!
And so we're right back to 'that dirty rat Rosenstein!' 2 days later.
At this point it's clear some members of Congress are either in on this and helping the cover story or they haven't got a clue and are out in the cold.
Note the conflicting stories about 'Rosenstein cancelled meeting with Congress on Oct 11!"
First, rumors surfaced of a scheduled meeting on Oct. 11 between Rosenstein & members of Congress, and Rosenstein just cancelled it.
Rep. Andy Biggs and Rep. Matt Gaetz say DAG Rod Rosenstein cancelled an Oct. 11 appearance before the judiciary and oversight committees. They are now calling for a subpoena. pic.twitter.com/TknVHKjXtd
— Ivan Pentchoukov \U0001f1fa\U0001f1f8 (@IvanPentchoukov) October 10, 2018
