BC TECH
Saved by @SteveeRogerr

A trend I'm excited for this year: DataOps & the Analytical Engineer

~10 years ago DevOps was born. The role of system admins and developers merged. Infrastructure became self-serve

Today the role of data engineers and business analysts are merging. Data is becoming self-serve

 Data infrastructure is becoming so powerful that the tools today allow non-technical folks to carry out the once complicated / custom code/ huge backlog jobs of data engineers.

Before getting into what this means, let's first discuss how we got here
Before 2012 the data world was dominated by transactional (OLTP) databases like PostgreSQL, MySQL, etc and analytical (OLAP) databases like Oracle, Netezza

Tools like Informatica / Talend were used to batch load (ETL) data into these databases, Tableau used to visualize
As you can imagine, there was heavy engineering work to manage the environment...

Then in 2013 AWS released their cloud data warehouse Redshift, and it was a game changer. Snowflake was founded in 2012, but didn't really pick up steam until a few years later (around 2016)
So why was Redshift a big deal?

1. It was the first cloud-native OLAP warehouse. It reduced the TCO of an OLAP database by orders of magnitude.

2. Speed of processing analytical queries increased dramatically
3. And later on (Snowflake pioneered this) they separated compute & storage. In overly simplified terms, this meant customers could scale their storage and compute resources independently of one another. This was a huge deal

What does this all mean? An EXPLOSION of data
The barriers to maintain a database were completely broken down, and the amount of data that was sent to Redshift / Snowflake / BigQuery skyrocketed.

Now, after this point we still weren't ready for DataOps / Analytical Engineers. What did it take to get there?
In my opinion there were 3 major technologies / shifts that happened that have given rise to DataOps:
1. The shift from ETL to ELT (extract-transform-load to extract-load-transform). Data used to be transformed (joined, aggregated, cleaned, etc) in motion while being loaded into the warehouse. Now, data is being loaded into the warehouse in it's raw form..
...Why is this important? In an ETL process if something goes wrong it's very hard to debug if the issue happened in the "T" or the "L". It was also harder to build these pipelines. With ELT, tools like @fivetran allow you to point & click to connect source data to your warehouse
The big trend here? The data warehouse is starting to subsume the data lake, and the default is becoming: "just send all data to Redshift / Snowflake." Again, the barriers of storing / collecting data is going way down
2. The importance of the cloud data warehouse. We already talked about this, but the one incremental point I want to make is the power of the compute within the warehouse went way up, and the cost of that compute went way down...
...this is fundamentally what enabled the "T" in ELT to happen within the warehouse. The compute horsepower of a Snowflake / Redshift made it possible
3. So who's driving these transformations? Tools like @getdbt @fishtowndata. The big technology advancement of the open source project dbt was representing these data transformations as code (SQL). It allowed anyone who knows SQL (business analysts) to author the transformations
Prior, the transformations were done with custom Python code by data engineers, or GUI based ETL tools. These took forever to build, were inflexible / hard to scale, and a black box
So in summary, the major platforms enabling the rise of DataOps are:

1. Data Movement: @fivetran
2. Data storage / compute: @SnowflakeDB @awscloud @GCPcloud
3. Data Transformations: @getdbt
If you think about what these technologies allow: to get data into a warehouse you just point and click data sources to their destination. The connectors are pre-built. You don't have to manage an on-prem database. And the data transformations are represented as basic SQL
And here is my KEY point (I get I've buried the lead a bit, but I believe the setup is important): Data Engineers used to manage all of the complexities of moving, storing, and transforming data. A lot of it was built with custom code / Python, and managed MANUALLY 🤯
Today, with the powerful tools I listed above, the data ecosystem can be managed with turnkey tools, removing the need for a lot of the complex work the data engineers handled previously....
...Instead of the BI analysts requesting the data engineer build them a pipeline of data, they can point & click with Fivetran and get data loaded into Snowflake, and write some SQL with dbt to get a materialized view (subset of data) to efficiently run their query against
After quite the lead in I hope that it's clear now how the business analysts (like BI analysts) can now carry out the functionality of data engineers to access data in a turnkey, self-serve manner. I'm describing these analysts as the "Analytica Engineer" and the process DataOps
So why is this important, and why am I excited about DataOps this year?

When data access becomes democratized and self-serve in nature, the need for new tools to manage this "modern data stack" goes up. I think we'll see a TON of huge companies built in the following categories:
1. I think we'll see the data catalogue / lineage systems re-invented. To access data you need to know where it's located / where it came from

2. Monitoring the quality of the data (and pipelines) will become increasingly important in self-serve settings. "Datadog for data"
3. A wave of modern BI tools

4. Governance becomes a bigger deal. With self-serve access, how can you govern who should have access to what?

5. Metadata gains increased importance

6. The way data pipelines were orchestrated (ie Airflow) will completely evolve
6. The "endpoints" for data coming out warehouses will grow. Right now the two primary places data goes from the warehouse are BI ML tools. I think we'll see an explosion of data going into new places, like back into SaaS apps

7. And many more we can't even imagine now

More from Tech

Aditya Todmal
Aditya Todmal
@AdityaTodmal
The absolute best 15 scanners which experts are using.

Got these scanners from the following accounts:

1. @Pathik_Trader
2. @sanjufunda
3. @sanstocktrader
4. @SouravSenguptaI
5. @Rishikesh_ADX

Share for the benefit of everyone.

Listing all scanners from @Pathik_Trader Sir first.

1. Open

2. 80-20 Reversal Setup


3. Fake

4. Trapping Inside Bar (Positional Setup)
TECH , STOCKLEARNINGS , STOCKS
James Felton
James Felton
@JimMFelton
Butt plugs were originally sold as a miracle cure for headaches, acne, and insanity

Thread, article here and h/t: @qikipedia :
https://t.co/Ts84xnmWKJ


For a long time, medicine was pre-occupied with the question "can this be solved by putting something in the butt?"

Take, for example, tobacco smoke enemas, in which 18th Century physicians saw drowning victims and attempted to de-deadify them by blowing tobacco smoke up their rectums with a pipe.


Despite having no medical benefit whatsoever, the practice was popular, and enema kits lined the Thames like lifebelts, ready to be used as one final humiliation for somebody who was crap at swimming.

This practice ended sometime after 1800 when it became apparent that it didn't really work, given how the anus famously isn't really connected to the lungs, but doctors weren't quite finished with having a poke around in the old anus to see what happens just yet.
TECH
Advertisement
Venkatesh Rao
Venkatesh Rao
@vgr
There are only 3 real challenges to free speech at the tech level a) the need for CDNs at scale b) distortions of net neutrality c) nation-state firewalls.

All other claims are about social governance on private sites, where applying free speech doctrine is a stretch at best

Baseline free speech has expanded modestly since 1776 or whatever your reference point is.

About 20% beyond core case of government-critical speech in formally public spaces and media I'd say.

You can (for example) have more kinds of conversations in a bar without getting punched or kicked out by other patrons or management. You can teach more things in colleges than 100 years ago.

That's a genuine de facto increase in practical, informal free speech/expression.

But online free "speech" is, as @reneed has pointed out, not an expansion of free speech at all, but an expansion of reach. It is like the secondary freedom of the press to print as many copies as they can afford, not primary freedom to say what they want

If a newspaper wants to print more copies it has buy more printing equipment and front more upfront costs. Reach for a paper publication is not "free", but a function of desired circulation, which in turn is a function of demand prediction etc. But that logic is obscured online.
TECH
Tobie Langel
Tobie Langel
@tobie
Elastic didn't really relicense ElasticSearch. It forked it.

🧵 A thread.

1/18

There's a lot of talk in the open source community about the cost of forking.

2/18

- "Forking is best avoided."

- "Forking is a last resort option."

- "Forking is like a nuclear weapons. It's a defensive threat."

3/18

Forking is seen as impractical and extremely expensive.

And that's a Really Good Thing(tm).

It's a forcing function for figuring out solutions that are broadly acceptable across the community.

4/18

The thing is, the cost of forking is mostly a function of three things:

1⃣ the size of the community that you can bring along with you,
2⃣ whether you need to rename your fork (who owns the trademark), and
3⃣ how much infrastructure you need to rebuild.

5/18
TECH
Aviral Bhatnagar
Aviral Bhatnagar
@aviralbhat
I spent 1.4 lakhs on Zomato alone this year

Once these platforms build a habit, it's hard to leave

Lifetime value of customers is immense, which recover marketing costs and more

These very "loss-making" companies eventually become money-making machines

I love the assumptions people are making here to try to work out how this is possible, but I'll not share the math to just keep this fun.

Few hints: I have excellent cardiovascular fitness, love fine + healthy dining and order with my wife and brother

Guess AOV and order freq
TECH

You May Also Like

Tashny Sukumaran
Tashny Sukumaran
@tashny
Surendra Ananth @surendraananth, you violated my conditional consent. I assented to unprotected intimate relations with you on the STRICT condition that you tell me if you have relations with anyone else. You failed to do so, and I became ill as a direct consequence.

You refused to reveal your own STI results to me, but claimed to others they were negative and alleged I had multiple other partners, slut-shaming me. You gaslit me saying you had told me about your other partners.

You called up @justice_sisters and @SistersinIslam to tell them I was crazy, jilted. You rebuffed attempts to organise a community mediation so we could address matters. You said I was crazy for speaking to your boss and father in an attempt to keep other women safe.

You have a PATTERN OF BEHAVIOUR of misogyny, harassment and sheer disrespect for women. This was sent to a friend of mine who has never expressed interest in you and has a partner.


You think women are playthings, notches on the bedpost. You, @surendraananth, are no human rights advocate.
LAW
TWABMPTLPAFA
TWABMPTLPAFA
@SeekerIIIII
Act of 1871
This is Long but it will end with a BOOM

With no constitutional authority to do so, Congress created a separate form of government for the District of Columbia, a ten-mile square parcel of land (see, Acts of the Forty-first Congress,”

Section 34, Session III, chapters 61 and 62).
The act — passed when the country was weakened and financially depleted in the aftermath of the Civil War — was a strategic move by foreign interests (international bankers) who were intent upon gaining a stranglehold on the coffers

and neck of America.
Congress cut a deal with the international bankers (specifically Rothschilds of London) to incur a DEBT to said bankers. Because the bankers were not about to lend money to a floundering nation without serious stipulations, they devised a way to get their

foot in the door of the United States.
The Act of 1871 formed a corporation called THE UNITED STATES. The corporation, OWNED by foreign interests, moved in and shoved the original Constitution into a dustbin.

With the Act of 1871, the organic Constitution was defaced — in effect vandalized and sabotage — when the title was capitalized and the word “for” was changed to “of” in the title.
HISTORY
Advertisement
Patrick Chovanec
Patrick Chovanec
@prchovanec
I'm going to do two history threads on Ethiopia, one on its ancient history, one on its modern story (1800 to today). 🇪🇹

I'll begin with the ancient history ... and it goes way back. Because modern humans - and before that, the ancestors of humans - almost certainly originated in Ethiopia. 🇪🇹 (sub-thread):


The first likely historical reference to Ethiopia is ancient Egyptian records of trade expeditions to the "Land of Punt" in search of gold, ebony, ivory, incense, and wild animals, starting in c 2500 BC 🇪🇹


Ethiopians themselves believe that the Queen of Sheba, who visited Israel's King Solomon in the Bible (c 950 BC), came from Ethiopia (not Yemen, as others believe). Here she is meeting Solomon in a stain-glassed window in Addis Ababa's Holy Trinity Church. 🇪🇹


References to the Queen of Sheba are everywhere in Ethiopia. The national airline's frequent flier miles are even called "ShebaMiles". 🇪🇹
CULTURE
Anu Satheesh \U0001f1ee\U0001f1f3
Anu Satheesh 🇮🇳...
@AnuSatheesh5
Yoga Ramar Temple
#Nedungunam
#TamilNaduTemples 🚩

Yoga Ramar Temple is located near Thiruvannamalai. Shri Rama is seated with eyes closed, without any weapons. Mata Sita  pointing to the feet of Rama telling that  everyone is safe at his feet. Lakshmana standing near Shri Rama


Hanuman is seen chanting Vedas obediently sitting before Shri Rama.

Sthala Puranam says that, Sukabrahma Maharishi had been doing Tapasya here, for the darshan of Sri Rama. Prabhu Shri Ram, pleased with his tapasya, came here after Rama Ravana yudham. Since Rama was returning


after the war, he did not have any weapons with him.

Another gopuram called Kili Gopuram is there for Sukabrahma Maharishi. There is a hill nearby called Dheergachala. According to Sthala puranam, Sri Padham of Prabhu Rama are found on the top of the hill


It is said that this is the place where Shri Rama gave darshan to Suka Maharishi. Nedumgunam means which shows all qualities of Shri Rama who is Maryadha purushottam. Shri Ramanavami is very famous here.

Jai Sri Ram
🙏🕉🙏
ALL
\U0001d54d\U0001d560\U0001d56a\U0001d552\U0001d558\U0001d556\U0001d563
𝕍𝕠𝕪𝕒𝕘𝕖𝕣...
@V0YAGERTWEETS
Ｇａｕ Ｍａｔａ Ｉｎ Ｖｅｄｉｃ
Ｃｕｌｔｕｒｅ

It is said that the one who bears the dust thrown from the hoof of Gaumata on his head, as if taking a bath in the water of the teerth sthalam and getting rid of all sins.


Ved and all other Hindu scriptures venerate the cow. The Vedic dictionary, Nighantu, gives amongst other synonyms of 'Gau' [ or cow] the words 'Aghnya', 'Ahi', and 'Aditi'.

Yaska the vedic decoder and commentator on Nighantu, defines these as-

Aghnya —— the one that ought not to be killed
Ahi —— the one that must not be slaughtered.
Aditi —— the one that ought not to be cut into pieces.

These three names of cow signify that this is an animal which should not to be put to tortures. These words appear frequently throughout the Veda in context of the cow.

But in due course the falsehood of killing cows spread far and wide and they became even more deep rooted when western scholars with their half baked knowledge of Sanskrit transliterated these interpretations of commentaries of Sayanacharya ,
ALL