BC TECH
Saved by @SteveeRogerr

A trend I'm excited for this year: DataOps & the Analytical Engineer

~10 years ago DevOps was born. The role of system admins and developers merged. Infrastructure became self-serve

Today the role of data engineers and business analysts are merging. Data is becoming self-serve

 Data infrastructure is becoming so powerful that the tools today allow non-technical folks to carry out the once complicated / custom code/ huge backlog jobs of data engineers.

Before getting into what this means, let's first discuss how we got here
Before 2012 the data world was dominated by transactional (OLTP) databases like PostgreSQL, MySQL, etc and analytical (OLAP) databases like Oracle, Netezza

Tools like Informatica / Talend were used to batch load (ETL) data into these databases, Tableau used to visualize
As you can imagine, there was heavy engineering work to manage the environment...

Then in 2013 AWS released their cloud data warehouse Redshift, and it was a game changer. Snowflake was founded in 2012, but didn't really pick up steam until a few years later (around 2016)
So why was Redshift a big deal?

1. It was the first cloud-native OLAP warehouse. It reduced the TCO of an OLAP database by orders of magnitude.

2. Speed of processing analytical queries increased dramatically
3. And later on (Snowflake pioneered this) they separated compute & storage. In overly simplified terms, this meant customers could scale their storage and compute resources independently of one another. This was a huge deal

What does this all mean? An EXPLOSION of data
The barriers to maintain a database were completely broken down, and the amount of data that was sent to Redshift / Snowflake / BigQuery skyrocketed.

Now, after this point we still weren't ready for DataOps / Analytical Engineers. What did it take to get there?
In my opinion there were 3 major technologies / shifts that happened that have given rise to DataOps:
1. The shift from ETL to ELT (extract-transform-load to extract-load-transform). Data used to be transformed (joined, aggregated, cleaned, etc) in motion while being loaded into the warehouse. Now, data is being loaded into the warehouse in it's raw form..
...Why is this important? In an ETL process if something goes wrong it's very hard to debug if the issue happened in the "T" or the "L". It was also harder to build these pipelines. With ELT, tools like @fivetran allow you to point & click to connect source data to your warehouse
The big trend here? The data warehouse is starting to subsume the data lake, and the default is becoming: "just send all data to Redshift / Snowflake." Again, the barriers of storing / collecting data is going way down
2. The importance of the cloud data warehouse. We already talked about this, but the one incremental point I want to make is the power of the compute within the warehouse went way up, and the cost of that compute went way down...
...this is fundamentally what enabled the "T" in ELT to happen within the warehouse. The compute horsepower of a Snowflake / Redshift made it possible
3. So who's driving these transformations? Tools like @getdbt @fishtowndata. The big technology advancement of the open source project dbt was representing these data transformations as code (SQL). It allowed anyone who knows SQL (business analysts) to author the transformations
Prior, the transformations were done with custom Python code by data engineers, or GUI based ETL tools. These took forever to build, were inflexible / hard to scale, and a black box
So in summary, the major platforms enabling the rise of DataOps are:

1. Data Movement: @fivetran
2. Data storage / compute: @SnowflakeDB @awscloud @GCPcloud
3. Data Transformations: @getdbt
If you think about what these technologies allow: to get data into a warehouse you just point and click data sources to their destination. The connectors are pre-built. You don't have to manage an on-prem database. And the data transformations are represented as basic SQL
And here is my KEY point (I get I've buried the lead a bit, but I believe the setup is important): Data Engineers used to manage all of the complexities of moving, storing, and transforming data. A lot of it was built with custom code / Python, and managed MANUALLY 🤯
Today, with the powerful tools I listed above, the data ecosystem can be managed with turnkey tools, removing the need for a lot of the complex work the data engineers handled previously....
...Instead of the BI analysts requesting the data engineer build them a pipeline of data, they can point & click with Fivetran and get data loaded into Snowflake, and write some SQL with dbt to get a materialized view (subset of data) to efficiently run their query against
After quite the lead in I hope that it's clear now how the business analysts (like BI analysts) can now carry out the functionality of data engineers to access data in a turnkey, self-serve manner. I'm describing these analysts as the "Analytica Engineer" and the process DataOps
So why is this important, and why am I excited about DataOps this year?

When data access becomes democratized and self-serve in nature, the need for new tools to manage this "modern data stack" goes up. I think we'll see a TON of huge companies built in the following categories:
1. I think we'll see the data catalogue / lineage systems re-invented. To access data you need to know where it's located / where it came from

2. Monitoring the quality of the data (and pipelines) will become increasingly important in self-serve settings. "Datadog for data"
3. A wave of modern BI tools

4. Governance becomes a bigger deal. With self-serve access, how can you govern who should have access to what?

5. Metadata gains increased importance

6. The way data pipelines were orchestrated (ie Airflow) will completely evolve
6. The "endpoints" for data coming out warehouses will grow. Right now the two primary places data goes from the warehouse are BI ML tools. I think we'll see an explosion of data going into new places, like back into SaaS apps

7. And many more we can't even imagine now

More from Tech

Wilbert Liu \U0001f468\U0001f3fb\u200d\U0001f3a8
Wilbert Liu 👨🏻‍🎨...
@wilbertliu
These past few days I've been experimenting with something new that I want to use by myself.

Interestingly, this thread below has been written by that.

Let me show you how it looks like. 👇🏻


When you see localhost up there, you should know that it's truly an experiment! 😀


It's a dead-simple thread writer that will post a series of tweets a.k.a tweetstorm. ⚡️

I've been personally wanting it myself since few months ago, but neglected it intentionally to make sure it's something that I genuinely need.

So why is that important for me? 🙂

I've been a believer of a story. I tell stories all the time, whether it's in the real world or online like this. Our society has moved by that.

If you're interested by stories that move us, read Sapiens!

One of the stories that I've told was from the launch of Poster.

It's been launched multiple times this year, and Twitter has been my go-to place to tell the world about that.

Here comes my frustration.. 😤
TECH
James Felton
James Felton
@JimMFelton
Butt plugs were originally sold as a miracle cure for headaches, acne, and insanity

Thread, article here and h/t: @qikipedia :
https://t.co/Ts84xnmWKJ


For a long time, medicine was pre-occupied with the question "can this be solved by putting something in the butt?"

Take, for example, tobacco smoke enemas, in which 18th Century physicians saw drowning victims and attempted to de-deadify them by blowing tobacco smoke up their rectums with a pipe.


Despite having no medical benefit whatsoever, the practice was popular, and enema kits lined the Thames like lifebelts, ready to be used as one final humiliation for somebody who was crap at swimming.

This practice ended sometime after 1800 when it became apparent that it didn't really work, given how the anus famously isn't really connected to the lungs, but doctors weren't quite finished with having a poke around in the old anus to see what happens just yet.
TECH
Advertisement
Laurie Voss
Laurie Voss
@seldo
The 2018 State of JavaScript survey is out. They got 20,000 responses and have some delicious, delicious data. I'mma thread in some highlights:

First up: JavaScript flavors. There's TypeScript at 47% adoption, a tad higher than npm's own survey results (which said 46%). npm's survey is coming up again soon and will ask about TypeScript in a lot more detail.


I really, *really* like SoJ's "would not use again" question, which lets people who've abandoned a tech self-identify. This is noticeable in the graph above with Flow users -- 41% of people who've used Flow say they wouldn't use it again.

Now JavaScript frameworks. It's been a full year since npm's survey, so these numbers differ from ours in the directions I expected:
React 65% (vs. 60%)
Vue 29% (vs. 24%)
Ember 5% (vs 4%, I was expecting a bigger rise)

But there's a shocker in here: Angular.


npm's survey had Angular at 40% last year and SoJ has it at either:
- 58% (if you include those who don't want to use it again)
- 24% (if you count only those who like it)
Since npm's question didn't ask if they intend to *continue* using it I think that might explain this.
TECH
Mukund Mohan
Mukund Mohan
@mukund
EV Battery overview
Batteries are the key differentiator between the various EV manufacturers. The amount of energy stored in the battery determines the range of the EV, thought to be a major limitation on EV sales.


Consumers tend to worry that an EV with a range of 80 to 250 miles on a single charge would be inconvenient for long trips due to the time it takes to recharge the battery. Commercial batteries need to ensure there is quick recharge and less time between charges


The lithium-ion battery is important because it makes EVs expensive than ICEs. Battery costs per kilowatt-hour (kWh) declined from roughly $1,000 per kWh in 2010 to $227 in 2016


Lithium-ion batteries made up 70 percent of the rechargeable battery market in 2016
BNEF projects that global production capacity for lithium-ion batteries will increase from 103 gigawatt-hours (GWh) in the first quarter of 2017 to 273 GWh by 2021.


The battery manufacturing supply chain has three main parts:

cell manufacturing,

module manufacturing, and

pack assembly.
TECH
foone
foone
@Foone
argh chrome updated and it seems they've intentionally broken custom search engines. I have had a keyword search for "foone" which searches my own twitter, so I just type "foone BLAH" and it gives me a twitter search for when I said BLAH, right?


but now I get that option in the titlebar of "search twitter self-search" which is the name of the search"
if I just type "foone blah" now...

it ignores the search I have had defined for years and just googles it.


and this is definitely on purpose because they didn't just break it, they changed it. it's now "fooneblah"


there doesn't seem to be an option to make this stop and get the old behavior back :(
TECH

You May Also Like

Jaya_Upadhyaya
Jaya_Upadhyaya
@Jayalko1
HORANADU ANNAPOORNESHWARI, CHIKMAGALUR (KAR)
#bharatmandir

Annapoorneshwari is the Devi of nutrition and food. The literal meaning of Annapoorneshwari is “Feeding one and all”. Anyone who comes here is offered meals and shelter within the premises.
@LostTemple7


Maharshi Agastya is believed to have done pratishtha here.

Once, Shiva and Parvati had an argument. Shiva declared everything in the world including food to be Maya or illusion. To prove him wrong, Devi Parvati disappeared resulting in entire Earth becoming still.


Taking pity, Devi appeared as Annapoorna and relieved everyone.
Another belief is that once Shiva beheaded Brahma and his skull got stuck in Shiva’s hand. He was cursed that till the skull is not full of food or grains, it will remain stuck to his hands.


Shiva went everywhere and asked for food but the skull was never full. He finally came here and Maa Annapoorna filled the skull with grains and reversed Shiva’s curse.

It is believed that who ever visits Devi Annapoorna here will never be without food in their lives.
ALL
Nicolas Cole
Nicolas Cole
@Nicolascole77
🚢 Atomic Essays Republishing Framework 🚢

1. Write Atomic Essay
2. Post image on Twitter
3. (Bonus) Copy/paste text as thread
4. Find relevant Question on Quora. Copy/paste Atomic Essay + image.
5. Copy/paste again on Medium.
6. Again on LinkedIn
7. Win

Guide below ✍️🚢👇

Step 1: Write Atomic Essay

I really enjoy writing right inside the Figma template. It helps give me a good sense of exactly how much "real estate" I have/have left before my time us up and I'm out of space.

Once finished, I export the image and send to my phone on Slack.


Step 2: Post image on Twitter

Before I publish my Atomic Essay on Twitter, I use the Edit/Photo Markup function on the iPhone to highlight standout sentences.

These are usually power-phrases: things the reader skims and thinks, "That's interesting," prompting them to read more.


Step 3: For the Ship 30 for 30 challenge, I only turn Atomic Essays into threads if...

1. The piece is SUPER actionable and lends itself well to the thread format (lots of bullets, quick points, lists, etc.)

or...

2. I'm referencing research and want to link to the pieces.

Step 4: Find relevant Question on Quora. Copy/paste Atomic Essay + image.

Search on Quora around the topic you wrote about. Find a related Question. Copy/paste the whole Atomic Essay as your "answer" (& fix formatting). Then include image so it appears in the thumbnail.
WRITING
Advertisement
Simon DeDeo
Simon DeDeo
@SimonDeDeo
One thing that goes unremarked about Wikipedia: the vast majority of the content was written over 12 years ago—which means (given turnover) that the editors dominant today were not directly involved in that creation.

Put another way, the editors who built the dominant nodes in this network...


...have little overlap with the ones who made this much more recent managerial flowchart.


Internet time runs at hundred-fold speed—the difference between the people who painted what's in the Uffizi, and the people in charge of keeping those paintings from deteriorating. Very different tasks, and (one presumes) very personalities as well. @PaulSkallas?
ALL
RadioChinar
RadioChinar
@RadioChinar
Demystifying Ghazwa-e-Hind. A #thread.

A recent video of Shoaid Akhter surfaced in social media creating a lot of controversy where the #Pakistani Cricketer can be seen suggesting invasion of India by using the term Ghazwa-e-Hind.
Radicalization of sportsmen seen off late
(1/10)


Some radical Islamic preachers believes that Ghazwa-e-Hind means a “Holy War against India”. As per this irrational & profane concept, there will be a fierce battle to secure victory over #India.
#Kashmir
(2/10)


#Pakistan-based terror groups, like JeM have been using Ghazwa-e-Hind as a Hadith to recruit, fund and justify its terror strikes as a religious holy war against #India.
#Kashmir
(3/10)


Some machiavellian people claim it is a prophecy that Muslim warriors will wage war against the #India.
Jaish & others falsely propagate that Jihad against India is considered holy in Islam & that those participating in it will be granted an easy entry into paradise.
(4/10)


However prolific scholars says that this is a made-up interpretation of some verse in The Holy Quran to mislead the gullible youth of #Pakistan into picking up arms and fighting its Army's personal battle against #India.
#Kashmir
(5/10)
SOCIETY
Anu Satheesh \U0001f1ee\U0001f1f3
Anu Satheesh 🇮🇳...
@AnuSatheesh5
Nanganallur Anjaneya Temple
#Chennai
#Tamilnadutemples 🚩

Bhaktha Anjaneya temple at Nanganallur in Chennai is one among tallest Anjaneyar Vigraham. Here Anjaneyar blesses with relief to all illness. Vigraham is about 32 feet height made of single  stone.
Anjaneyar here is 👇


Sri Vishwaroopa Aadhivyathihara Baktha Anjaneya Swamy.

Sthala Puranam says that, this is the place where Indra Deva hit Hanuman on the cheek by his Vajrastram. Sri Raghavendra Swami and Sri Kanchi Paramacharyar were behind in building this temple. Temple is constructed to


Realize power of Anjaneyar who is seen both in Ramayana and Mahabharatha.
Vadamalai is important offering to Anjaneyar here. It is believed that vigraham contain powers to heal any disease. Shri Ramanavami is celebrated here.

Jai Sri Ram
Jai Hanuman
🙏🕉🙏
ALL