BC TECH
Saved by @SteveeRogerr

A trend I'm excited for this year: DataOps & the Analytical Engineer

~10 years ago DevOps was born. The role of system admins and developers merged. Infrastructure became self-serve

Today the role of data engineers and business analysts are merging. Data is becoming self-serve

 Data infrastructure is becoming so powerful that the tools today allow non-technical folks to carry out the once complicated / custom code/ huge backlog jobs of data engineers.

Before getting into what this means, let's first discuss how we got here
Before 2012 the data world was dominated by transactional (OLTP) databases like PostgreSQL, MySQL, etc and analytical (OLAP) databases like Oracle, Netezza

Tools like Informatica / Talend were used to batch load (ETL) data into these databases, Tableau used to visualize
As you can imagine, there was heavy engineering work to manage the environment...

Then in 2013 AWS released their cloud data warehouse Redshift, and it was a game changer. Snowflake was founded in 2012, but didn't really pick up steam until a few years later (around 2016)
So why was Redshift a big deal?

1. It was the first cloud-native OLAP warehouse. It reduced the TCO of an OLAP database by orders of magnitude.

2. Speed of processing analytical queries increased dramatically
3. And later on (Snowflake pioneered this) they separated compute & storage. In overly simplified terms, this meant customers could scale their storage and compute resources independently of one another. This was a huge deal

What does this all mean? An EXPLOSION of data
The barriers to maintain a database were completely broken down, and the amount of data that was sent to Redshift / Snowflake / BigQuery skyrocketed.

Now, after this point we still weren't ready for DataOps / Analytical Engineers. What did it take to get there?
In my opinion there were 3 major technologies / shifts that happened that have given rise to DataOps:
1. The shift from ETL to ELT (extract-transform-load to extract-load-transform). Data used to be transformed (joined, aggregated, cleaned, etc) in motion while being loaded into the warehouse. Now, data is being loaded into the warehouse in it's raw form..
...Why is this important? In an ETL process if something goes wrong it's very hard to debug if the issue happened in the "T" or the "L". It was also harder to build these pipelines. With ELT, tools like @fivetran allow you to point & click to connect source data to your warehouse
The big trend here? The data warehouse is starting to subsume the data lake, and the default is becoming: "just send all data to Redshift / Snowflake." Again, the barriers of storing / collecting data is going way down
2. The importance of the cloud data warehouse. We already talked about this, but the one incremental point I want to make is the power of the compute within the warehouse went way up, and the cost of that compute went way down...
...this is fundamentally what enabled the "T" in ELT to happen within the warehouse. The compute horsepower of a Snowflake / Redshift made it possible
3. So who's driving these transformations? Tools like @getdbt @fishtowndata. The big technology advancement of the open source project dbt was representing these data transformations as code (SQL). It allowed anyone who knows SQL (business analysts) to author the transformations
Prior, the transformations were done with custom Python code by data engineers, or GUI based ETL tools. These took forever to build, were inflexible / hard to scale, and a black box
So in summary, the major platforms enabling the rise of DataOps are:

1. Data Movement: @fivetran
2. Data storage / compute: @SnowflakeDB @awscloud @GCPcloud
3. Data Transformations: @getdbt
If you think about what these technologies allow: to get data into a warehouse you just point and click data sources to their destination. The connectors are pre-built. You don't have to manage an on-prem database. And the data transformations are represented as basic SQL
And here is my KEY point (I get I've buried the lead a bit, but I believe the setup is important): Data Engineers used to manage all of the complexities of moving, storing, and transforming data. A lot of it was built with custom code / Python, and managed MANUALLY 🤯
Today, with the powerful tools I listed above, the data ecosystem can be managed with turnkey tools, removing the need for a lot of the complex work the data engineers handled previously....
...Instead of the BI analysts requesting the data engineer build them a pipeline of data, they can point & click with Fivetran and get data loaded into Snowflake, and write some SQL with dbt to get a materialized view (subset of data) to efficiently run their query against
After quite the lead in I hope that it's clear now how the business analysts (like BI analysts) can now carry out the functionality of data engineers to access data in a turnkey, self-serve manner. I'm describing these analysts as the "Analytica Engineer" and the process DataOps
So why is this important, and why am I excited about DataOps this year?

When data access becomes democratized and self-serve in nature, the need for new tools to manage this "modern data stack" goes up. I think we'll see a TON of huge companies built in the following categories:
1. I think we'll see the data catalogue / lineage systems re-invented. To access data you need to know where it's located / where it came from

2. Monitoring the quality of the data (and pipelines) will become increasingly important in self-serve settings. "Datadog for data"
3. A wave of modern BI tools

4. Governance becomes a bigger deal. With self-serve access, how can you govern who should have access to what?

5. Metadata gains increased importance

6. The way data pipelines were orchestrated (ie Airflow) will completely evolve
6. The "endpoints" for data coming out warehouses will grow. Right now the two primary places data goes from the warehouse are BI ML tools. I think we'll see an explosion of data going into new places, like back into SaaS apps

7. And many more we can't even imagine now

More from Tech

Hemant Mohapatra
Hemant Mohapatra
@MohapatraHemant
~8yrs ago (Dec’12) I got a job @Google. Those were still early days of cloud. I joined GCP @<150M ARR & left @~4B (excld GSuite). Learned from some of the smartest ppl in tech. But we also got a LOT wrong that took yrs to fix. Much of it now public, but here’s my ring-side view👇

By 2008, Google had everything going for it w.r.t. Cloud and we should’ve been the market leaders, but we were either too early to market or too late. What did we do wrong? (1) bad timing (2) worse productization & (3) worst GTM.

We were 1st to “containers” (lxc) & container management (Borg) - since '03/04. But Docker took LXC, added cluster management, & launched 1st. Mesosphere launched DCOS. A lot of chairs were thrown around re: google losing this early battle, though K8 won the war, eventually 👏

We were 1st to “serverless” (AppEngine). GAE was our beachhead -- it was the biggest revenue source early on but the world wasn’t ready for serverless primitives. We also didn’t build auxiliary products fast enough. Clients that outgrew GAE wanted “building block” IaaS offerings.

1st to hadoop (map-reduce ‘04) but our hosted Hadoop launched in ‘15. AWS EMR was ~200M ARR by then. 1st to cloud storage (GFS ’03), but didn’t offer a filestore till ‘18! Customers were asking for it since 2014. Didn’t launch archival storage or direct interconnect till v. late.
INTERNET , TECH
Aditya Todmal
Aditya Todmal
@AdityaTodmal
The absolute best 15 scanners which experts are using.

Got these scanners from the following accounts:

1. @Pathik_Trader
2. @sanjufunda
3. @sanstocktrader
4. @SouravSenguptaI
5. @Rishikesh_ADX

Share for the benefit of everyone.

Listing all scanners from @Pathik_Trader Sir first.

1. Open

2. 80-20 Reversal Setup


3. Fake

4. Trapping Inside Bar (Positional Setup)
TECH , STOCKLEARNINGS , STOCKS
Advertisement
Luca Hammer
Luca Hammer
@luca
Two years ago on a weekend, I built a tool to make it easier to evaluate Twitter accounts. Since then 36590 people used it to analyze 55390 different Twitter accounts.

Over the last months @mmkaradeniz and I made a new version. We launched it last night:

@accountanalysis doesn't use machine learning or AI „magic“. Instead of telling users if an account is authentic or not, it helps them to evaluate the accounts themselves.

It visualizes the different features (date, time, type, app, etc.) of Tweets to make them interpretable. /1

@accountanalysis The core concept is still the same, but it looks much better and is easier to operate. Not only for the users, but us developers as well. Enabling us to continuously roll out new features in the future.

Side-by-side screenshots of the old and new version. /2


@accountanalysis Some people had questions what different charts displayed and how they could be interpreted. There are now explanations for all charts, that can be toggled on and off at any time. /3


@accountanalysis The selected/retrieved Tweet count moved to the top to make it clear that not all Tweets of an account are analyzed. 3200 is the API limit by Twitter. It's possible to get more through the Premium API, but I don't think people would pay $100+ for the analysis of one account. /4
TECH
Resist the Hologram - Stu Cvrk
Resist the Hologram - ...
@STUinSD
Thread – Dump the Legacy Media; Turn to These News Sources

A. We all need sources less tainted by the Big Tech/globalist narratives that are echoed repeatedly throughout the legacy media. My brother sent me a list of alternatives from the Noq report.
TECH
Knarf
Knarf
@Knarf93
@zkat__ I'm in the position that I actually find npm / yarn the best ecosystem. Whenever I use something else I always end up stubbing my toe into something thats missing / feels wrong.

Ex. Cargo seems to neither have a concept of devDependencies nor peerDependencies.

I also can't understand why it wouldn't have an "add" command to add a new dependency. And I'm no fan of Toml, json is greate (easy to parse and build tooling around), and the better option in my opinion would be json5.

C / C++ seems to just not have language package managers. The linux / bsd crowd seem to have decided that the system package manager also should be the language package manager. Which might have been fine if every Linux distro used the same system package manager.

Instead we end up with a N x M problem. Where we have a bunch of different operating systems and they all support multiple system package managers. So there's no easy way of distributing, referencing and updating C / C++ packages.

It is also my opinion that the compiler / runtime should be a package dependency. I don't like Rust's split between rustup and cargo (they should have been one tool). Similarly it would be better if you added Node as a dependency to package.json, that way we wouldn't need NVM.
TECH

You May Also Like

Kitze @ \U0001f1e9\U0001f1ea\ufe0f
Kitze @ 🇩🇪️...
@thekitze
To people who are under the impression that you can get rich quickly by working on an app, here are the stats for https://t.co/az8F12pf02

📈 ~12000 vistis
☑️ 109 transactions
💰 353€ profit (285 after tax)

I have spent 1.5 months on this app. You can make more $ in 2 days.

🤷‍♂️


I'm still happy that I launched a paid app bcs it involved extra work:

- backend for processing payments (+ permissions, webhooks, etc)
- integration with payment processor
- UI for license activation in Electron
- machine activation limit
- autoupdates
- mailgun emails

etc.

These things seemed super scary at first. I always thought it was way too much work and something would break. But I'm glad I persisted. So far the only problem I have is that mailgun is not delivering the license keys to certain domains like https://t.co/6Bqn0FUYXo etc. 👌

omg I just realized that me . com is an Apple domain, of course something wouldn't work with these dicks
MAKERS
\uff2c\uff41\uff44\uff59 \uff22\uff4c\uff41\uff5a\uff45\uff53
Ｌａｄｙ Ｂｌａｚｅｓ...
@Lady_Blazes
Oh my Goodness!!!

I might have a panic attack due to excitement!!

Read this thread to the end...I just had an epiphany and my mind is blown. Actually, more than blown. More like OBLITERATED! This is the thing! This is the thing that will blow the entire thing out of the water!


Has this man been concealing his true identity?

Is this man a supposed 'dead' Seal Team Six soldier?

Witness protection to be kept safe until the right moment when all will be revealed?!

Who ELSE is alive that may have faked their death/gone into witness protection?


Were "golden tickets" inside the envelopes??


Are these "golden tickets" going to lead to their ultimate undoing?

Review crumbs on the board re: 'gold'.


#SEALTeam6 Trump re-tweeted this.
GOVERNMENT
Advertisement
foone
foone
@Foone
An interesting set of games that I've been vaguely interested in since I was a kid, after seeing one at a friend's house and never being able to find it again, was the BATTLES IN A DISTANT DESERT/BATTLES ON DISTANT PLANETS series.


These are turn-based wargames for windows 3.x, sharing the same engine.
Battles in a Distant Desert is from 1992, and is based on the first Iraq war (Desert Storm)


and Battles on Distant Planets is from 1991, and takes place in SPACE!
I'm pretty sure this is the one I played as a kid.


They've got 3 options to play with:
* Player vs. Player
* Player vs. Computer
* Computer vs. Computer

So it's a 0-2 player game!


They also did a DOS strategy game called "STARDATE 2140.2: GALACTIC CONQUEST" in 1990, but it seems to be lost.
All the games share an experimental AI system based on neural networks.
FUN
\u0936\u094d\u0930\u0940\u092e\u093e\u0932\u0940 \u0939\u093f\u0924\u0947\u0936 \u0905\u0935\u0938\u094d\u0925\u0940
श्रीमाली हितेश अवस्थी...
@HiteshAwasthi89
#Thread
12 Most Powerful Hanuman Mantras

Hanuman ji is the Hindu God of strength, valor, agility, and wisdom. He is considered as the incarnation (avatar) of bhagwan Shiva. Hanuman ji is the symbol of devotion and dedication. He is the greatest devotee of Prabhu Shri Ram. 1/n


Hanuman ji is known by several names such as Bajrangabali, Mahabali, Maruti Nandan, Pawanputra, Mahaveer, and Anjaneya. Hanuman ji is the son of Bhagwan Vayu, the wind god.

God Hanuman played a prominent part in rescuing Mata Sitafrom the bondage of demon King of Lanka, Ravana.


One can worship Hanuman ji by chanting Hanuman Mantras. There are many different types of Hanuman Mantras, each fulfilling a specific purpose. Hanuman Mantras help to beget the blessings of Mahabali Hanuman for success. 3/n


Let’s explore these powerful Hanuman Mantras one by one. Here’s the list for you:

1. Hanuman Moola Mantra
!! Om Hanumate Namah !!
ऊँ हनुमते नम:

You can chant the “Hanuman Moola Mantra if you usually face problems and obstacles in your life.4/n
#hanumanjanmotsav


However, the Human Moola Mantra is also a very powerful success mantra.

It’s a Kaarya Siddhi Mantra.

It is highly advised that all those who usually face obstacles and problems in their life should chant the Hanuman Moola Mantra. 5/n
#hanumanjanmotsav
ALL
Vibhu Vashisth
Vibhu Vashisth
@VIBHU_Tweet
LACHIT BORPHUKAN : AN UNSUNG WARRIOR OF THE AHOM KINGDOM & THE GREAT BATTLE OF SARAIGHAT

Originally named as Lachit Deca, Lachit Borphukan, the fierce and invincible Ahom Commander, was born during the early 17th century at Betioni in the Golaghat district of modern Assam.


His father, Momai Tamuli Borbarua was the ‘Governor’ of the kingdom and also ‘Commander-in-Chief’ of Ahom army under King Pratap Singha during his reign starting from 1603 to 1639. Lachit received military training from an early age and joined the Ahom King Jayadhvaj Singha...


...(1648-1663) as a scarf-bearer.  The post ‘scarf-bearer’ or ‘Soladhara Barua’ is considered as a part of the incumbent king’s personal staff. Owing to his military training Lechit was appointed "Ghora Barua", Supdt. of the Royal horses. When Chakradhwaj Singha became king..


..of Ahom dynasty in 1663,he appointed him Suptd. of The Royal Guards &finally he was appointed as Borphukan by the king.Then onwards,he was known as Lachit Borphukan.Borphukan used to be top councillor in Ahom Kingdom. This position was embedded with executive & judicial powers.


Assam was facing incessant Islamic invasion since Jan1662, when Mughal General Mir Jumla II attacked Ahom capital Gargaon. He never defeated Ahom king Jayadhwaja Singha as king retreated to hill &continued Guerrilla warfare.Prolonged fight &prospect of stalemate disturbed Jumla.
ALL