BC TECH
Saved by @SteveeRogerr

A trend I'm excited for this year: DataOps & the Analytical Engineer

~10 years ago DevOps was born. The role of system admins and developers merged. Infrastructure became self-serve

Today the role of data engineers and business analysts are merging. Data is becoming self-serve

 Data infrastructure is becoming so powerful that the tools today allow non-technical folks to carry out the once complicated / custom code/ huge backlog jobs of data engineers.

Before getting into what this means, let's first discuss how we got here
Before 2012 the data world was dominated by transactional (OLTP) databases like PostgreSQL, MySQL, etc and analytical (OLAP) databases like Oracle, Netezza

Tools like Informatica / Talend were used to batch load (ETL) data into these databases, Tableau used to visualize
As you can imagine, there was heavy engineering work to manage the environment...

Then in 2013 AWS released their cloud data warehouse Redshift, and it was a game changer. Snowflake was founded in 2012, but didn't really pick up steam until a few years later (around 2016)
So why was Redshift a big deal?

1. It was the first cloud-native OLAP warehouse. It reduced the TCO of an OLAP database by orders of magnitude.

2. Speed of processing analytical queries increased dramatically
3. And later on (Snowflake pioneered this) they separated compute & storage. In overly simplified terms, this meant customers could scale their storage and compute resources independently of one another. This was a huge deal

What does this all mean? An EXPLOSION of data
The barriers to maintain a database were completely broken down, and the amount of data that was sent to Redshift / Snowflake / BigQuery skyrocketed.

Now, after this point we still weren't ready for DataOps / Analytical Engineers. What did it take to get there?
In my opinion there were 3 major technologies / shifts that happened that have given rise to DataOps:
1. The shift from ETL to ELT (extract-transform-load to extract-load-transform). Data used to be transformed (joined, aggregated, cleaned, etc) in motion while being loaded into the warehouse. Now, data is being loaded into the warehouse in it's raw form..
...Why is this important? In an ETL process if something goes wrong it's very hard to debug if the issue happened in the "T" or the "L". It was also harder to build these pipelines. With ELT, tools like @fivetran allow you to point & click to connect source data to your warehouse
The big trend here? The data warehouse is starting to subsume the data lake, and the default is becoming: "just send all data to Redshift / Snowflake." Again, the barriers of storing / collecting data is going way down
2. The importance of the cloud data warehouse. We already talked about this, but the one incremental point I want to make is the power of the compute within the warehouse went way up, and the cost of that compute went way down...
...this is fundamentally what enabled the "T" in ELT to happen within the warehouse. The compute horsepower of a Snowflake / Redshift made it possible
3. So who's driving these transformations? Tools like @getdbt @fishtowndata. The big technology advancement of the open source project dbt was representing these data transformations as code (SQL). It allowed anyone who knows SQL (business analysts) to author the transformations
Prior, the transformations were done with custom Python code by data engineers, or GUI based ETL tools. These took forever to build, were inflexible / hard to scale, and a black box
So in summary, the major platforms enabling the rise of DataOps are:

1. Data Movement: @fivetran
2. Data storage / compute: @SnowflakeDB @awscloud @GCPcloud
3. Data Transformations: @getdbt
If you think about what these technologies allow: to get data into a warehouse you just point and click data sources to their destination. The connectors are pre-built. You don't have to manage an on-prem database. And the data transformations are represented as basic SQL
And here is my KEY point (I get I've buried the lead a bit, but I believe the setup is important): Data Engineers used to manage all of the complexities of moving, storing, and transforming data. A lot of it was built with custom code / Python, and managed MANUALLY 🤯
Today, with the powerful tools I listed above, the data ecosystem can be managed with turnkey tools, removing the need for a lot of the complex work the data engineers handled previously....
...Instead of the BI analysts requesting the data engineer build them a pipeline of data, they can point & click with Fivetran and get data loaded into Snowflake, and write some SQL with dbt to get a materialized view (subset of data) to efficiently run their query against
After quite the lead in I hope that it's clear now how the business analysts (like BI analysts) can now carry out the functionality of data engineers to access data in a turnkey, self-serve manner. I'm describing these analysts as the "Analytica Engineer" and the process DataOps
So why is this important, and why am I excited about DataOps this year?

When data access becomes democratized and self-serve in nature, the need for new tools to manage this "modern data stack" goes up. I think we'll see a TON of huge companies built in the following categories:
1. I think we'll see the data catalogue / lineage systems re-invented. To access data you need to know where it's located / where it came from

2. Monitoring the quality of the data (and pipelines) will become increasingly important in self-serve settings. "Datadog for data"
3. A wave of modern BI tools

4. Governance becomes a bigger deal. With self-serve access, how can you govern who should have access to what?

5. Metadata gains increased importance

6. The way data pipelines were orchestrated (ie Airflow) will completely evolve
6. The "endpoints" for data coming out warehouses will grow. Right now the two primary places data goes from the warehouse are BI ML tools. I think we'll see an explosion of data going into new places, like back into SaaS apps

7. And many more we can't even imagine now

More from Tech

JBD
JBD
@rakyll
Google's interview system is not great but there are various reasons why it feels less than it is. Most candidates don't know what the goal of our interviews are. First rule, it is not about finding an optimal solution or any solution at all.

You are interviewed for multiple skills simultaneously. Cognitive skills, communication, leadership are a few to name. If the point is not finding a solution, then what is it? Let me explain.

Your interviewers try to understand what it feels like to work with you on a daily basis. An interview question is just a tool in achieving that, it is not there to specifically measure your skills on a topic but a tool to understand the depth of your thinking.

Before the interview starts, ask them what they want to get out of this interview. Good interviewers should already have a plan and a set of expectations. Ask them what you should do. Don't start coding yet. Ask them you should produce. Discussion, diagrams, pseudo code, code?

Then, start cracking the question. List whatever questions you think it is important to solve this question, ask your edge cases. Get to a point where you are discussing about pros/cons of the solutions. These steps are critical. Don't just start coding. Have a consensus first.
TECH
Patrick McKenzie
Patrick McKenzie
@patio11
It’s underremarked upon that stablecoins accomplish basically all of the original design goals for Bitcoin qua a transactional mechanism (instant free value transfer anywhere), but that nobody cares about this because nobody ever cared about that part of the pitch.

“Why do people care about stablecoins then?”

A mix of “they encourage dollar-denominated liquidity in the cryptocurrency ecosystem and discourage withdrawal of the same” and “they’re good for money laundering.”

“But they make value transfer between exchanges much faster!”

This was a solved problem in traditional finance, too, mostly through the extension of credit. (It doesn’t matter how long settlement takes if there is sufficient trust to enable credit.)

The Bitcoin ecosystem is *positively allergic* to credit, so you have to call it a coin for them to accept it. And after you call it a coin they ignore everything the world has learned about credit, like risk management.

“Stablecoins aren’t credit!”

They’re pretty much exactly credit? A tether is a zero-coupon Bitfinex bond with a non-functioning call option. I
TECH
Advertisement
James Felton
James Felton
@JimMFelton
Butt plugs were originally sold as a miracle cure for headaches, acne, and insanity

Thread, article here and h/t: @qikipedia :
https://t.co/Ts84xnmWKJ


For a long time, medicine was pre-occupied with the question "can this be solved by putting something in the butt?"

Take, for example, tobacco smoke enemas, in which 18th Century physicians saw drowning victims and attempted to de-deadify them by blowing tobacco smoke up their rectums with a pipe.


Despite having no medical benefit whatsoever, the practice was popular, and enema kits lined the Thames like lifebelts, ready to be used as one final humiliation for somebody who was crap at swimming.

This practice ended sometime after 1800 when it became apparent that it didn't really work, given how the anus famously isn't really connected to the lungs, but doctors weren't quite finished with having a poke around in the old anus to see what happens just yet.
TECH
James More
James More
@jameore
One of the best decisions I made during a very turbulent 2020 was to leave conventional coding behind and embrace the #nocode movement. @bubble made this a reality. Although my own journey thus far is premature, I’ve learned a lot so here’s a power thread on....


‘How I created @buildcamp sales funnel landing page in under 2hours’.

Preview here 👇

https://t.co/s9P5JodSHe

Power thread here 👇

1. Started with a vanilla bubble app ensuring that all styles and UI elements were removed. Created a new page called funnel and set the page size to 960px as this allows the page to render proportionately on both web and mobile when hitting responsive breakpoints.


2. Began dropping elements onto the page to ‘find the style’. These had to be closely aligned to our @buildcamp branding so included text, buttons and groups - nothing too heavy. Played around with a few fonts, colors and gradients and thus pinned down the following style guide.


3. Started to map out sections using groups as my ‘containers’ to hold the relevant information and imagery needed to pad out the sales pitch. At this point, they were merely blocks of color #ff6600 with reduced opacity set to 5% to ease page flair.
TECH
Brian Balfour
Brian Balfour
@bbalfour
⚡️🧠Impact = Environment x Skill

A framework for making big career decisions by @iambangaly (Reforge Partner, Former Instacart, Instagram)

Full post -> https://t.co/dU0l7e0BWR

🧵 Thread...

1/ Career decisions about your next role/opp are the most impactful decisions you might make in your career. As an operator, all of your eggs are in one basket at a time, and we get a limited number of swings at the plate. So getting good at these decisions is important...

2/ At FB, Bangaly managed Rotational PM's. At the end of a rotation, he'd have the same convo: How are you going to choose your next rotation/role?

As a result, Bangaly got a lot of reps guiding this type of decision and created a framework to help with it...

3/ The framework is Impact = Skills x Environment

The goal is not to try and boil this type of decision down to a spreadsheet of inputs that spits out the answer, but rather it helps with 4 things...


4/ It can help with:

- Identify individual variables that are inputs into the decision.
- Evaluate each individual variable in a structured way.
- Understand the relationship between each variable.
- Narrow the decision down to the most important variable

...
TECH

You May Also Like

Zen Black
Zen Black
@z3nblack
1/ Thread: "A Silicon Valley Ponzi Scheme"

Thanks to @chamath for laying this out in Social Capital's 2018 annual letter.

I've always appreciated his outspokenness.

2/ The hardest thing for most startups today is the path to market: first finding product-market fit & a way to reach customers, then building a ruthless machine to acquire, monetize & retain them.

3/ Because of this, when the VC industry invests capital into fast growing startups today, the plurality (if not majority) of invested capital will go into user acquisition and ad spending, for better or worse— usually worse.

4/ Todays massive venture-backed advertising, sales, and user acquisition playbook has morphed into one that champions growth at any cost.

This is creating a big bill that will soon come due...

5/ Ad impressions and click-throughs are bid up to outrageous prices by startups flush with venture money, and prospective users demand more and more subsidized products to gain their initial attention.
TECH , STARTUPS , {TEST FOLDER}
Roger Grosse
Roger Grosse
@RogerGrosse
Important paper from Google on large batch optimization. They do impressively careful experiments measuring # iterations needed to achieve target validation error at various batch sizes. The main "surprise" is the lack of surprises. [thread]

https://t.co/7QIx5CFdfJ


The paper is a good example of lots of elements of good experimental design. They validate their metric by showing lots of variants give consistent results. They tune hyperparamters separately for each condition, check that optimum isn't at the endpoints, and measure sensitivity.

They have separate experiments where the hold fixed # iterations and # epochs, which (as they explain) measure very different things. They avoid confounds, such as batch norm's artificial dependence between batch size and regularization strength.

When the experiments are done carefully enough, the results are remarkably consistent between different datasets and architectures. Qualitatively, MNIST behaves just like ImageNet.

Importantly, they don't find any evidence for a "sharp/flat optima" effect whereby better optimization leads to worse final results. They have a good discussion of experimental artifacts/confounds in past papers where such effects were reported.
MACHINE LEARNING
Advertisement
Numberonepal
Numberonepal
@numberonepal
1/ Stolen from the greatawakening(DOT)win...

I will only be making one post here. Do not ask who I am. Do with this information as you please.

The Trump Card is being played. It's much simpler than you all realize. Some of you have gotten it right.
TRUMP , THE LAST DAYS IF THE REVOLUTION
Sam
Sam
@SamKhan999
This day was inevitable. Indian Muslims always lived in La La Land. They bad mouthed real well wishers & slept with the enemy who never considered them as part of them. In Gulf, Indian Muslims were busy with Javed Akhtar & Sonu Nigam. Never interacted with South Asian diaspora

Now when Indian Muslims are thrown to wolves it’s Pakistani & Bangladeshi Muslims with whom they kept distance are the only ones feeling their pain. None of the people who they invited for Eid Milan. Qawali & Mushaira are there for them. Such people invoke wrath upon themselves

Indian Muslims never spoke for Kashmiri Muslims. Distancing from separatist was understandable but we never even bothered to highlight their human rights violation & rapes carried out by forces. We could have sent them humanitarian help we didn’t. Our silence made us complicit.

When Sania Mirza married Shoaib Malik it was something legal. A Muslim married another Muslim. Entire Hindutva media went after her. They launched a witch-hunt for months. No Indian Muslim stood by her. All were silent. Now days come when Muslims are arrested for marrying a Hindu

Indian Muslims particularly the elite downplayed communalism in India by saying it will pass or Hindu majority is good. Fringe are bad. They misguiding others. Muslims were always others. Indians were happy to see Muslims sing & dance & eat their food but never considered as own
INDIA
Anu Satheesh \U0001f1ee\U0001f1f3
Anu Satheesh 🇮🇳...
@AnuSatheesh5
Cherthala Karthyayani Kshetram
#Alappuzha, Kerala
#KeralaTemples 🚩

Cherthala Karthyayani Temple Prathishta was done by Vilwamangalam Swamy. Chertala Pooram is very famous. Devi is Mangalya Dayini here removes obstacles relating to marriages. Sthala Puranam says that, once 👇


Vilwamangalam Swamiyar was passing this area after Prathista of Padmananbhaswamy Temple, he felt Devi chaitanyam and recognized as Karthyayani Devi. But Devi seeing Vilwamangalam went inside the pond. Swamy searched for Devi continously for 6 days and on 7th day Vilwamangalam


got hold of Devi's hair. Thus only Devi's head is Prathishta here. Devotees do Kozhi parathal ( offering cock) here as an offering to Karthyayani Devi.

Karthyayaniamme Sharanam
🙏🕉🙏
ALL