0/ When large eng orgs rely on metrics for both monitoring *and* observability, they struggle with cardinality.

This is a thread about “the two drivers of cardinality.” And which one of those we should kill. :)

🧵👇

1/ Okay, first off: “what is cardinality, anyway?” And why is it such a big deal for metrics?

“Cardinality” is a mathematical term: it’s *the number of elements in a set*... boring! So why tf does anybody care??

Well, because people think they need it, then suddenly, "$$$$$$$."
2/ When a developer inserts a (custom) metric to their code, they might think they’re just adding, well, *one metric*. …
3/ … But when they add “tags” to that metric – like , , or (shiver) – they are actually creating a *set* of metric time series, with the *cardinality* of that set being the total number of unique combinations of those tags.
4/ The problem is that some of those tags have many distinct values.

E.g., when I was working on Monarch at Google, there was a single gmail metric that became over 300,000,000 (!!!) distinct time series. In a TSDB, that cardinality is the unit of cost.

So again, “$$$$$$$.”
5/ Okay, so that’s why people care about metrics cardinality. Now, what are the two *drivers* of that cardinality?

A) Monitoring: more detailed *health measurements*
B) Observability: more detailed *explanations of changes*

Let’s take these one at a time…
6/ First, “More Detailed Health Measurements” (monitoring):

Consider an RPC metric that counts errors. You need to independently monitor the error rate for different methods. And so – voila – someone adds a “method” tag, and now there’s 10x the cardinality for that metric.
7/ … And also 10x the cost. But that’s justifiable, as there’s a business case for independently monitoring the reliability of distinct RPC methods.

Put another way, you might rightly have different error budgets for different RPC methods, so their statistics must be separable.
8/ Bottom line: When it comes to “measuring health,” we often *need* cardinality in order to hone in on the signals we actually care about. Increasing cardinality to *proactively* monitor the signals we care most about is usually a worthwhile tradeoff.
9/ Now what about using cardinality for “More Detailed Explanations of Changes?”

This is the real culprit! And, frankly, should be abolished. :) Metrics cardinality is the wrong way to do observability – to explain changes.

More on that…
10/ Say your monitoring tells you that there’s a problem with a critical symptom – e.g., you’re suddenly burning through an SLO error budget at an unsustainable clip.
11/ After a painful outage, say you realize a single customer DOSed your service. So someone adds a `customer` tag “for next time.”

But this is unsustainable: each incident reveals a new failure mode, devs keep adding tags, and before long your metrics bill is out of control.
12/ The problem, in a nutshell:

Distributed systems can fail for a staggeringly large number of reasons. *You can't use metrics cardinality to isolate each one.*
13/ How *should* we explain changes to production systems?

Understanding change is the central problem of observability. Effective workflows might *start* with metrics, but they must pivot towards a multi-telemetry, multi-service guided analysis.
14/ So, to sum up: spend your limited cardinality budget on *monitoring*, and then look for observability that (a) naturally explains changes and (b) relies on transactional data sources that do not penalize you for high/unbounded cardinality.
15/ PS: For more on how to distinguish monitoring and observability, see this thread: https://t.co/2RHs36Nknc

PPS: If you’d like to discuss/debate/request-more-detail any of the above, reply to this thread or DM me!

More from Society

Two things can be true at once:
1. There is an issue with hostility some academics have faced on some issues
2. Another academic who himself uses threats of legal action to bully colleagues into silence is not a good faith champion of the free speech cause


I have kept quiet about Matthew's recent outpourings on here but as my estwhile co-author has now seen fit to portray me as an enabler of oppression I think I have a right to reply. So I will.

I consider Matthew to be a colleague and a friend, and we had a longstanding agreement not to engage in disputes on twitter. I disagree with much in the article @UOzkirimli wrote on his research in @openDemocracy but I strongly support his right to express such critical views

I therefore find it outrageous that Matthew saw fit to bully @openDemocracy with legal threats, seeking it seems to stifle criticism of his own work. Such behaviour is simply wrong, and completely inconsistent with an academic commitment to free speech.

I am not embroiling myself in the various other cases Matt lists because, unlike him, I think attention to the detail matters and I don't have time to research each of these cases in detail.

You May Also Like

MASTER THREAD on Short Strangles.

Curated the best tweets from the best traders who are exceptional at managing strangles.

• Positional Strangles
• Intraday Strangles
• Position Sizing
• How to do Adjustments
• Plenty of Examples
• When to avoid
• Exit Criteria

How to sell Strangles in weekly expiry as explained by boss himself. @Mitesh_Engr

• When to sell
• How to do Adjustments
• Exit


Beautiful explanation on positional option selling by @Mitesh_Engr
Sir on how to sell low premium strangles yourself without paying anyone. This is a free mini course in


1st Live example of managing a strangle by Mitesh Sir. @Mitesh_Engr

• Sold Strangles 20% cap used
• Added 20% cap more when in profit
• Booked profitable leg and rolled up
• Kept rolling up profitable leg
• Booked loss in calls
• Sold only


2nd example by @Mitesh_Engr Sir on converting a directional trade into strangles. Option Sellers can use this for consistent profit.

• Identified a reversal and sold puts

• Puts decayed a lot

• When achieved 2% profit through puts then sold
Assalam Alaiki dear Sister in Islam. I hope this meets you well. Hope you are keeping safe in this pandemic. May Allah preserve you and your beloved family. I would like to address the misconception and misinterpretation in your thread. Please peruse the THREAD below.


1. First off, a disclaimer. Should you feel hurt by my words in the course of the thread, then forgive me. It’s from me and not from Islam. And I probably have to improve on my delivery. And I may not quote you verbatim, but the intended meaning would be there. Thank You!

2. Standing on Imam Shafii’s quote: “And I never debated anyone but that I did not mind whether Allah clarified the truth on my tongue or his tongue” or “I never once debated anyone hoping to win the debate; rather I always wished that the truth would come from his side.”

3. Okay, into the meat (my love for meat is showing. Lol) of the thread. Even though you didn’t mention the verse that permitted polygamy, everyone knows the verse you were talking about (Q4:3).


4. Your reasons for the revelation of the verse are strange. The first time I came across such. I had to quickly consult the books on the exegeses or tafsir of the Quran written by renowned specialists!