BC TECH
Saved by @SteveeRogerr

A trend I'm excited for this year: DataOps & the Analytical Engineer

~10 years ago DevOps was born. The role of system admins and developers merged. Infrastructure became self-serve

Today the role of data engineers and business analysts are merging. Data is becoming self-serve

 Data infrastructure is becoming so powerful that the tools today allow non-technical folks to carry out the once complicated / custom code/ huge backlog jobs of data engineers.

Before getting into what this means, let's first discuss how we got here
Before 2012 the data world was dominated by transactional (OLTP) databases like PostgreSQL, MySQL, etc and analytical (OLAP) databases like Oracle, Netezza

Tools like Informatica / Talend were used to batch load (ETL) data into these databases, Tableau used to visualize
As you can imagine, there was heavy engineering work to manage the environment...

Then in 2013 AWS released their cloud data warehouse Redshift, and it was a game changer. Snowflake was founded in 2012, but didn't really pick up steam until a few years later (around 2016)
So why was Redshift a big deal?

1. It was the first cloud-native OLAP warehouse. It reduced the TCO of an OLAP database by orders of magnitude.

2. Speed of processing analytical queries increased dramatically
3. And later on (Snowflake pioneered this) they separated compute & storage. In overly simplified terms, this meant customers could scale their storage and compute resources independently of one another. This was a huge deal

What does this all mean? An EXPLOSION of data
The barriers to maintain a database were completely broken down, and the amount of data that was sent to Redshift / Snowflake / BigQuery skyrocketed.

Now, after this point we still weren't ready for DataOps / Analytical Engineers. What did it take to get there?
In my opinion there were 3 major technologies / shifts that happened that have given rise to DataOps:
1. The shift from ETL to ELT (extract-transform-load to extract-load-transform). Data used to be transformed (joined, aggregated, cleaned, etc) in motion while being loaded into the warehouse. Now, data is being loaded into the warehouse in it's raw form..
...Why is this important? In an ETL process if something goes wrong it's very hard to debug if the issue happened in the "T" or the "L". It was also harder to build these pipelines. With ELT, tools like @fivetran allow you to point & click to connect source data to your warehouse
The big trend here? The data warehouse is starting to subsume the data lake, and the default is becoming: "just send all data to Redshift / Snowflake." Again, the barriers of storing / collecting data is going way down
2. The importance of the cloud data warehouse. We already talked about this, but the one incremental point I want to make is the power of the compute within the warehouse went way up, and the cost of that compute went way down...
...this is fundamentally what enabled the "T" in ELT to happen within the warehouse. The compute horsepower of a Snowflake / Redshift made it possible
3. So who's driving these transformations? Tools like @getdbt @fishtowndata. The big technology advancement of the open source project dbt was representing these data transformations as code (SQL). It allowed anyone who knows SQL (business analysts) to author the transformations
Prior, the transformations were done with custom Python code by data engineers, or GUI based ETL tools. These took forever to build, were inflexible / hard to scale, and a black box
So in summary, the major platforms enabling the rise of DataOps are:

1. Data Movement: @fivetran
2. Data storage / compute: @SnowflakeDB @awscloud @GCPcloud
3. Data Transformations: @getdbt
If you think about what these technologies allow: to get data into a warehouse you just point and click data sources to their destination. The connectors are pre-built. You don't have to manage an on-prem database. And the data transformations are represented as basic SQL
And here is my KEY point (I get I've buried the lead a bit, but I believe the setup is important): Data Engineers used to manage all of the complexities of moving, storing, and transforming data. A lot of it was built with custom code / Python, and managed MANUALLY 🤯
Today, with the powerful tools I listed above, the data ecosystem can be managed with turnkey tools, removing the need for a lot of the complex work the data engineers handled previously....
...Instead of the BI analysts requesting the data engineer build them a pipeline of data, they can point & click with Fivetran and get data loaded into Snowflake, and write some SQL with dbt to get a materialized view (subset of data) to efficiently run their query against
After quite the lead in I hope that it's clear now how the business analysts (like BI analysts) can now carry out the functionality of data engineers to access data in a turnkey, self-serve manner. I'm describing these analysts as the "Analytica Engineer" and the process DataOps
So why is this important, and why am I excited about DataOps this year?

When data access becomes democratized and self-serve in nature, the need for new tools to manage this "modern data stack" goes up. I think we'll see a TON of huge companies built in the following categories:
1. I think we'll see the data catalogue / lineage systems re-invented. To access data you need to know where it's located / where it came from

2. Monitoring the quality of the data (and pipelines) will become increasingly important in self-serve settings. "Datadog for data"
3. A wave of modern BI tools

4. Governance becomes a bigger deal. With self-serve access, how can you govern who should have access to what?

5. Metadata gains increased importance

6. The way data pipelines were orchestrated (ie Airflow) will completely evolve
6. The "endpoints" for data coming out warehouses will grow. Right now the two primary places data goes from the warehouse are BI ML tools. I think we'll see an explosion of data going into new places, like back into SaaS apps

7. And many more we can't even imagine now

More from Tech

Venkatesh Rao
Venkatesh Rao
@vgr
There are only 3 real challenges to free speech at the tech level a) the need for CDNs at scale b) distortions of net neutrality c) nation-state firewalls.

All other claims are about social governance on private sites, where applying free speech doctrine is a stretch at best

Baseline free speech has expanded modestly since 1776 or whatever your reference point is.

About 20% beyond core case of government-critical speech in formally public spaces and media I'd say.

You can (for example) have more kinds of conversations in a bar without getting punched or kicked out by other patrons or management. You can teach more things in colleges than 100 years ago.

That's a genuine de facto increase in practical, informal free speech/expression.

But online free "speech" is, as @reneed has pointed out, not an expansion of free speech at all, but an expansion of reach. It is like the secondary freedom of the press to print as many copies as they can afford, not primary freedom to say what they want

If a newspaper wants to print more copies it has buy more printing equipment and front more upfront costs. Reach for a paper publication is not "free", but a function of desired circulation, which in turn is a function of demand prediction etc. But that logic is obscured online.
TECH
Matthew Panzarino
Matthew Panzarino
@panzer
New iPad Pro has Face ID, thinner bezels, squared off edges. A new touch sensitive Pencil that magnetically attaches to the edges. Thin enough to have a camera bump. Uses the XR's LCD cornering techniques.


The new 12.9" iPad Pro is physically smaller than the old one with the same screen size. It's roughly the same size as an 8.5 x 11 piece of paper. 5.9mm thick.

10 billion transistors in the A12X. 8-core CPU. 7-Core GPU. 35% faster single-core performance. 1000x faster GPU. iPad Pro is Xbox One S class GPU performance in "94% smaller" package, Ternus jokes.

Apple's silicon team is just dropping bombs every event.

live shot of Apple's silicon team


iPad Pro gets USB-C for up to 5K external displays and enables charging out to iPhones. Wild.
TECH
Advertisement
EverythingOppresses
EverythingOppresses
@SoOppressed
The first area to focus on is diversity. This has become a dogma in the tech world, and despite the fact that tech is one of the most meritocratic industries in the world, there are constant efforts to promote diversity at the expense of fairness, merit and competency. Examples:

USC's Interactive Media & Games Division cancels all-star panel that included top-tier game developers who were invited to share their experiences with students. Why? Because there were no women on the

ElectronConf is a conf which chooses presenters based on blind auditions; the identity, gender, and race of the speaker is not known to the selection team. The results of that merit-based approach was an all-male panel. So they cancelled the conference.

Apple's head of diversity (a black woman) got in trouble for promoting a vision of diversity that is at odds with contemporary progressive dogma. (She left the company shortly after this

Also in the name of diversity, there is unabashed discrimination against men (especially white men) in tech, in both hiring policies and in other arenas. One such example is this, a developer workshop that specifically excluded men: https://t.co/N0SkH4hR35
TECH
Lea Kissner
Lea Kissner
@LeaKissner
We're kicking off the Privacy Tech session at #enigma2021 with Mitch Negus speaking about "NO DATA, NO PROBLEM—GIVING NUCLEAR INSPECTORS BETTER TOOLS WITHOUT REVEALING STATE

A nuclear catastrophe hasn't occurred... yet. So we need to stay vigilant. Nuclear inspectors go in according to treaties to check what's going on and check compliance with treaty rules.

But as sophisticated analytics become more common, states will only want to share the minimum amount of information necessary under the treaty.

But perhaps we can use MPC -- secure multi-party computation

Yao's garbled circuits were first demonstrated on the millionaire's problem -- figuring out who's richer without revealing actual amounts.

Used for other things like cryptocurrency these days.

Can we use this for nuclear inspection?

MPC can be used to compute anything computed by a computer [but it's expensive!]
TECH
Santiago
Santiago
@svpino
Here is a simple example of a machine learning model.

I put it together a long time ago, and it was very helpful! I sliced it apart a thousand times until things started to make sense.

It's TensorFlow and Keras.

If you are starting out, this may be a good puzzle to solve.


The goal of this model is to learn to multiply one-digit
TECH

You May Also Like

Anu Satheesh \U0001f1ee\U0001f1f3
Anu Satheesh 🇮🇳...
@AnuSatheesh5
Samayapuram Mariamman Temple
#Trichy
#Tamilnadutemples 🚩

Samayapuram Mariamman Temple is located in Samayapuram of
 Tiruchirappalli district.  Mariyamman here is worshipped as Samayapurathal a form of Parashakti.
Amman vigraham here is made of sand and clay so


no abhishekam done to the main vigraham. Samayapuram Mariamman Temple is one of the famous Shakthi Sthal of Tamilnadu.

Story says that paramashiva created Kali, out of the poison that he drank during samudra mathanam.  As Devi originated from the Kaalakoota/ halahala poison,


thus the name Kaali. Mariamman is believed to be a form of Kaali, and is also known as Mahamayi. It is also believed that Maharaja Dasharatha offered his prayers here.

Mariamman is Parashakti herself who blesses with prosperity and she is believed to cure all diseases.


Paramashiva, Mahavishnu and Brahma Deva are believed to have taken blessings of Mariamman here.

Om Shakti Para Shakti
🙏🕉🙏
ALL
WORLD OF SANATAN DHARMA
WORLD OF SANATAN DHARM...
@world_sanatan
PETER PADUKAM - THE LEGEND OF MADURAI MEENAKSHI TEMPLE AND A BRITISH COLLECTOR.

A British Collector named Rous Peter was appointed as Collector of Madurai from 1812 to 1828.

Though a Christian by faith, he respected all faiths including Hinduism and also honored local practice


Collector Peter was the temple administrator of the Meenakshi Amman Temple and conducted all his duties with sincerity and honesty and respected the religious sentiments of all people.

Collector Rous Peter respected and treated people of all faiths equally and this noble trait

earned him the popular nickname ‘Peter Pandian'

Goddess Meenakshi Amman Temple was situated between Collector Peter's residence and office.

Everyday he used to go to the office by his horse and while crossing the temple, he got down from his horse, removed the hat and his shoes

and crossed the whole path on his foot.

Through this small gesture he expressed his reverence to the Goddess!

One day there was a heavy downpour in Madurai city and River Vaigai was in spate.

Collector was sleeping in his residence and was suddenly disturbed and woken up by

the sound of anklets and he left his bed to find out from where the sound had came.

He saw a small girl wearing pattuvastrams (silk garments) and precious ornaments and addressing him as 'Peter come this way'.
ALL
Advertisement
Justin Kan
Justin Kan
@justinkan
1/ Throughout my startup career, I’ve seen many founders have an aversion to sales. This is a mistake that can kill your startup.

2/ Sales is often viewed as either a saving grace or proof that the product isn’t good enough (because it should sell itself). Neither are ever true. Some common mistakes that result in...

3/ Mistake 1: Hire a sales rep before reaching product/market fit to get your initial batch of customers. This is a mistake because founders need to work through their MVP with early adopters to truly understand what it is they’re selling.

4/ Mistake 2: Reach product/market fit, need to scale, and rely entirely on self-serve. For enterprise products that require big commitments and internal shifts, almost no product is self-explanatory enough to sell itself.

5/ Mistake 3: Make a first sales hire who isn’t scrappy enough to help mold the sales process from scratch. Some salespeople are amazing at their jobs, but not cut out to establish the processes that others end up following. This skillset is what @rdedatta calls a “sales ninja”.
STARTUPS
\u0936\u094d\u0930\u0940\u092e\u093e\u0932\u0940 \u0939\u093f\u0924\u0947\u0936 \u0905\u0935\u0938\u094d\u0925\u0940
श्रीमाली हितेश अवस्थी...
@HiteshAwasthi89
Sri Devi Karumariamman mandir, Chennai, Tamilnadu

Sri Devi Karumariamman mandir is in Thiruverkadu, a suburb of Chennai. It is said that God Muruga came here and obtained the 'Vel' (spear) from His mother, Goddess Karumari (an avatar of Parvathi) 1/n


to fight the demon Suran and hence this place was called as 'Vel kadu' and became as Verkadu. As per legend Devi Karumariamman once disguised as a soothsayer visited Bhagwan Surya, the Sun God, to predict His future. 2/n


Not realising that the soothsayer is none other than Sakthi Matha, Surya ignored Her. Angered by his indifference, Devi Karumariamman immediately retreated. Bhagwan Surya lost His glory and the earth plunged into darkness. 3/n


Bhagwan Surya then realised His folly and pleaded forgiveness from Devi Karumariamman. In order to pacify mother Karumariamman, Suryadev said that His rays would fall on Her directly twice in a year and touch her feet. (in the month of Panguni (March- April) & during (Sep - Oct)


Mother Karumari Amman is worshipped here in two forms.A swayambu (self manifested) with only head visible above the ground and another one in a graceful sitting posture with all Parasakthi features,with four arms holding skull, sword, trishul, udukkai (damru). 5/n
ALL
WORLD OF SANATAN DHARMA
WORLD OF SANATAN DHARM...
@world_sanatan
PETER PADUKAM - THE LEGEND OF MADURAI MEENAKSHI TEMPLE AND A BRITISH COLLECTOR.

A British Collector named Rous Peter was appointed as Collector of Madurai from 1812 to 1828.

Though a Christian by faith, he respected all faiths including Hinduism and also honored local practice


Collector Peter was the temple administrator of the Meenakshi Amman Temple and conducted all his duties with sincerity and honesty and respected the religious sentiments of all people.

Collector Rous Peter respected and treated people of all faiths equally and this noble trait

earned him the popular nickname ‘Peter Pandian'

Goddess Meenakshi Amman Temple was situated between Collector Peter's residence and office.

Everyday he used to go to the office by his horse and while crossing the temple, he got down from his horse, removed the hat and his shoes

and crossed the whole path on his foot.

Through this small gesture he expressed his reverence to the Goddess!

One day there was a heavy downpour in Madurai city and River Vaigai was in spate.

Collector was sleeping in his residence and was suddenly disturbed and woken up by

the sound of anklets and he left his bed to find out from where the sound had came.

He saw a small girl wearing pattuvastrams (silk garments) and precious ornaments and addressing him as 'Peter come this way'.
ALL