BC SOCIETY
Saved by @jacobhtml

0/ When large eng orgs rely on metrics for both monitoring *and* observability, they struggle with cardinality.

This is a thread about “the two drivers of cardinality.” And which one of those we should kill. :)

🧵👇

 1/ Okay, first off: “what is cardinality, anyway?” And why is it such a big deal for metrics?

“Cardinality” is a mathematical term: it’s *the number of elements in a set*... boring! So why tf does anybody care??

Well, because people think they need it, then suddenly, "$$$$$$$."
2/ When a developer inserts a (custom) metric to their code, they might think they’re just adding, well, *one metric*. …
3/ … But when they add “tags” to that metric – like , , or (shiver) – they are actually creating a *set* of metric time series, with the *cardinality* of that set being the total number of unique combinations of those tags.
4/ The problem is that some of those tags have many distinct values.

E.g., when I was working on Monarch at Google, there was a single gmail metric that became over 300,000,000 (!!!) distinct time series. In a TSDB, that cardinality is the unit of cost.

So again, “$$$$$$$.”
5/ Okay, so that’s why people care about metrics cardinality. Now, what are the two *drivers* of that cardinality?

A) Monitoring: more detailed *health measurements*
B) Observability: more detailed *explanations of changes*

Let’s take these one at a time…
6/ First, “More Detailed Health Measurements” (monitoring):

Consider an RPC metric that counts errors. You need to independently monitor the error rate for different methods. And so – voila – someone adds a “method” tag, and now there’s 10x the cardinality for that metric.
7/ … And also 10x the cost. But that’s justifiable, as there’s a business case for independently monitoring the reliability of distinct RPC methods.

Put another way, you might rightly have different error budgets for different RPC methods, so their statistics must be separable.
8/ Bottom line: When it comes to “measuring health,” we often *need* cardinality in order to hone in on the signals we actually care about. Increasing cardinality to *proactively* monitor the signals we care most about is usually a worthwhile tradeoff.
9/ Now what about using cardinality for “More Detailed Explanations of Changes?”

This is the real culprit! And, frankly, should be abolished. :) Metrics cardinality is the wrong way to do observability – to explain changes.

More on that…
10/ Say your monitoring tells you that there’s a problem with a critical symptom – e.g., you’re suddenly burning through an SLO error budget at an unsustainable clip.
11/ After a painful outage, say you realize a single customer DOSed your service. So someone adds a `customer` tag “for next time.”

But this is unsustainable: each incident reveals a new failure mode, devs keep adding tags, and before long your metrics bill is out of control.
12/ The problem, in a nutshell:

Distributed systems can fail for a staggeringly large number of reasons. *You can't use metrics cardinality to isolate each one.*
13/ How *should* we explain changes to production systems?

Understanding change is the central problem of observability. Effective workflows might *start* with metrics, but they must pivot towards a multi-telemetry, multi-service guided analysis.
14/ So, to sum up: spend your limited cardinality budget on *monitoring*, and then look for observability that (a) naturally explains changes and (b) relies on transactional data sources that do not penalize you for high/unbounded cardinality.
15/ PS: For more on how to distinguish monitoring and observability, see this thread: https://t.co/2RHs36Nknc

PPS: If you’d like to discuss/debate/request-more-detail any of the above, reply to this thread or DM me!

More from Society

KeyserSozeBro
KeyserSozeBro
@KeyserSozeBro1
My little Jacob Wohl story: August 2016, we're sitting around the trading turrets at my office on a bored to tears slow day. Our receptionist said someone was here for his 2pm appointment. None of us had an appointment on schedule. Who is it? we ask? Some guy named Jacob Wohl 1/

Jacob Wohl, for those out of loop, gained quite a bit of notoriety for becoming the youngest guy to ever get himself barred for life from the financial industry. Now he's waiting in our lobby to pitch something. We're a large merchant bank. 2/

Send him in, we said. In walks young Jacob asking to be seeded with $25mm to start a new hedge fund in exchange for an ownership stake in it. Oh, & he needs an office. And a Bloomberg station. And needs to borrow staff until he can hire some guys. And he needs a leased car... 3/

And he needs an answer today. Right now. Because there's a competing offer from Izzy Englander's firm Millenium (we know those guys well). We fight the urge to literally start laughing, but do ask him...hey, you the same guy who got banned by regulators? He didn't even flinch 4/

He went on about that being untrue & handed over a letter he said was from his lawyer who had confirmed he was not barred, had been vindicated but it 'just wasn't in the system yet at the SEC, CFTC'...whatever that means. 5/
SOCIETY
GeePaw Hill
GeePaw Hill
@GeePawHill
Human, local, oriented, taken, and iterative, these are the change-harvester's bywords. In iterative change, we not only accept the reality of gradual stepwise refinement -- changing what we've already changed before -- we actually anticipate it and take advantage of it.

Before we begin: Folks, I know many are celebrating last week's results . I see them as one small pass towards the change I seek. Take a short break, then it's back to work?

Black lives matter.
Voting rights matter.
Women matter.

Stay safe, stay strong, stay angry, stay kind.

(And a *second* prefatory note: I've been not writing for six weeks. It was a combination of several factors, including illness, distraction, and the need to finish a gig. I'm happy to be back at content, now, but also very nervous. Bear with me while I regain my sea legs.)

With iteration, I think a great starting point for the concept is to watch just about any youtube video where a skilled artist draws a realistic rendering. What you will see almost inevitably is a direct implementation of iterative change.

The strokes begin quite broadly, faintly indicating the broad contours of the subject. Experience is relevant here: those who are most familiar with drawing that particular kind of subject, will make contours that seem "right" more quickly than those less experienced.
SOCIETY
Advertisement
John Hayward
John Hayward
@Doc_0
It's funny how so many people who claim to worship "democracy" also dismiss nationalism as selfish xenophobia. Democracy is absolutely meaningless without strong national sovereignty. Under globalism, citizens of any given state have little influence over how they are governed.

Nationalism isn't just mindless tribalism. People are correct to insist their government has a responsibility TO THEM that vastly outweighs any imagined responsibility to foreign governments they have absolutely no influence over, or foreign populations.

We've seen that point taken to absurd extremes by the Left during the Trump administration, with whack-job judges ruling that foreigners have rights that EXCEED those of American citizens, including absolute rights to enter the U.S. and influence its government.

And we're frequently told that some international elite "consensus" completely overrides our ability to make individual decisions - or even to petition our own government, or vote for representatives who disagree with whatever "the world" has supposedly decided.

All of that is the absolute negation of "democracy" in any form, including the dangerous mob rule and majoritarianism that the Left selectively advocates. (Notice they always have a list of things nobody gets to vote on anymore, no matter how strong the majority vote would be.)
SOCIETY
Rob Ford
Rob Ford
@robfordmancs
Two things can be true at once:
1. There is an issue with hostility some academics have faced on some issues
2. Another academic who himself uses threats of legal action to bully colleagues into silence is not a good faith champion of the free speech cause


I have kept quiet about Matthew's recent outpourings on here but as my estwhile co-author has now seen fit to portray me as an enabler of oppression I think I have a right to reply. So I will.

I consider Matthew to be a colleague and a friend, and we had a longstanding agreement not to engage in disputes on twitter. I disagree with much in the article @UOzkirimli wrote on his research in @openDemocracy but I strongly support his right to express such critical views

I therefore find it outrageous that Matthew saw fit to bully @openDemocracy with legal threats, seeking it seems to stifle criticism of his own work. Such behaviour is simply wrong, and completely inconsistent with an academic commitment to free speech.

I am not embroiling myself in the various other cases Matt lists because, unlike him, I think attention to the detail matters and I don't have time to research each of these cases in detail.
SOCIETY
Indian Rwanda Radio
Indian Rwanda Radio
@IND_RwandaRadio
Nearly a year ago, the Tablighi Jamaat controversy took over India’s media. Around 3,500 foreign nationals had visited India to attend the Tablighi Jamaat event at the Nizamuddin Markaz Mosque in Delhi.

With the lockdown being announced on March 24, 2020, several attendees had moved to different parts of the country to attend smaller gatherings in local mosques.

Around 960 foreign nationals were held at quarantine and in states like UP, Jharkhand, Maharashtra, Karnataka, etc. were sent straight to prison.

There was an outpour of Anti-Muslim bigotry in much of Indian news media with this event where it ramped up its demonisation of the community.

For weeks, the attendees of the Tablighi Jamaat event were named the “super-spreaders” of COVID and in several cases, Maulana Mohammad Saad was called a terrorist and “the maulana of death”.
SOCIETY

You May Also Like

Suchi Sharma
Suchi Sharma
@Suchi_sharma_
#THREAD
1.Meenakshi Amman Temple(472ft High)also called as Meenakshi Sundareswarar is a famous Hindu temple located on the southern bank of Vaigai River in temple city of Madurai, #TamilNadu.It is dedicated to Parvati(Meenakshi)& her consort Shiva(Sundareswarar). @LostTemple7


2.Madurai Meenakshi Sundareswarar temple was built by King Kulasekara Pandya (1190-1216 CE).This popular temple visits over 15000 visitors daily & around 25000 on #Friday !Friday sees the most number of devotees visit the temple with the number going as high as +25,000.


3.Meenakshi Temple also hosts an annual festival called Meenakshi Thirukalayanam that is held in April. A celestial wedding of the god and goddess marks the end of the festival. About 1 million devotees visit the temple during this festival.


4.the temple has been tagged as one of the seven wonders of India along with stunning Architecture.The temple is built using Dravidian South Indian style of architecture, and is one of the most impressive temple structures in South India.


5.The principle deity here is MEENAKSHI, a form of Goddess Parvati. The idol of the goddess is carved in emerald hued stone. The Meenakshi idol has 3 breasts as Meenakshi was born with three breasts and was blessed that the 3rd breast would vanish once meets right man of her life
HISTORY
Andrew Chen
Andrew Chen
@andrewchen
1/ new essay: What's next for marketplace startups? Reinventing the $10 trillion service economy, that's what. Co-authored with @ljin18 https://t.co/fgKbrHTnH5. A short thread 👇

2/ Marketplace startups have done incredibly well over the first few decades of the internet, reinventing the way we shop for goods, but have been less successful services. It's bc services are complex, subjective, fragmented, and often in real life. Makes it hard

3/ There's been 4 major eras at making the service economy work online. The Listings Era, the unbundled Craiglist era, the Uber for X era, and the Managed Marketplace era


4/ Each era has added more value than the last, and utilized technology innovations, from internet to social / "read/write web" to mobile. The "Unbundling Craigslist" era was particularly epic at generating startup ideas


5/ The problem is, all the low-hanging fruit has been picked off. The techniques that got us to here won't get us to the next phase. So we have to do some pretty different things. That's why "Managed Marketplaces" have been a big deal - hire folks as W-2s, certify quality, etc.
STARTUPS
Advertisement
Justin Jackson
Justin Jackson
@mijustin
A lot of bootstrappers think they have a marketing problem:

"I need to get better at marketing."

But, it's more likely that you have a product problem:

"Do customers really want this? Do they care enough about this to switch to a new solution?"

☝️ this is why most of my book, @marketingdevs, is about:

1. Choosing the right market
2. Building something they want

"If your product is remarkable, getting noticed is a lot easier." – @peldi


I was reminded of this concept again this morning while reading @pjrvs' book (Company of One):

"Sales increase when you honestly evaluate what someone needs and then teach them the value of what you're selling."


To succeed, your product has to offer an outcome that is highly desirable to a large group of customers.

The number of sales your product receives is a multiple of these two variables:

1. How big is the target market?
2. How desirable is the outcome you're offering them?
STARTUPS
Matt Fuller
Matt Fuller
@MEPFuller
They’re shooting into the chamber.
POLITICS
Billy Bostickson \U0001f3f4\U0001f441&\U0001f441 \U0001f193
Billy Bostickson 🏴👁&👁 ...
@BillyBostickson
@EricTopol @NBA @StephenKissler @yhgrad B.1.1.7 reveals clearly that SARS-CoV-2 is reverting to its original pre-outbreak condition, i.e. adapted to transgenic hACE2 mice (either Baric's BALB/c ones or others used at WIV labs during chimeric bat coronavirus experiments aimed at developing a pan betacoronavirus vaccine)

@NBA @StephenKissler @yhgrad 1. From Day 1, SARS-COV-2 was very well adapted to humans .....and transgenic hACE2 Mice


@NBA @StephenKissler @yhgrad 2. High Probability of serial passaging in Transgenic Mice expressing hACE2 in genesis of SARS-COV-2


@NBA @StephenKissler @yhgrad B.1.1.7 has an unusually large number of genetic changes, ... found to date in mouse-adapted SARS-CoV2 and is also seen in ferret infections.
https://t.co/9Z4oJmkcKj


@NBA @StephenKissler @yhgrad We adapted a clinical isolate of SARS-CoV-2 by serial passaging in the ... Thus, this mouse-adapted strain and associated challenge model should be ... (B) SARS-CoV-2 genomic RNA loads in mouse lung homogenates at P0 to P6.
https://t.co/I90OOCJg7o
ALL