He mainly created a hierarchy of disagreements as explained in the diagram below

You come across many such examples on a day to day basis. For example here is a recent one I saw, and disagreements(Some real some made up)

Point: 

Someone posted on the lines of :  while meta’s new social media threads is good , its not yet able to replace twitter.Twitter is still where all political conversations happen. Twitter won

Disagreements

DH0: Cry harder you liar  → Adds no value

DH1: Very telling that your source bought a golden check on twitter and has to now justify the investment → Attacks the source rather than the story

DH2: Weird that you are celebrating this  → Attacks imagined tone

DH3: Not true. I find all my news here. I login 27 times a day  → Anecdotal  

DH4: Twitter has lost most of the advertisers and most of the top news accounts have moved out. Threads on the other hand has 100M DAU. → Twitter might have lost advertisers and the news accounts , but that doesnt actually mean that political conversation is not happening.Threads doing well is also not exactly relevant to the argument. Its a parallel conversation

DH5: Threads has much fewer users than twitter at the moment so surely the volume would be lower. But if you were to look at this report . Threads generated XX million impressions and engagement on political content and on a per user basis this is far higher than what we see on twitter  . If we extrapolate this based on current user growth , threads is likely to overtake twitter sometime end of this year
→ A good argument that talks about per user engagement and how it is much healthier than twitter. While it is still not better in terms of overall numbers, it’s very likely getting there. This argument does have the power to change someones mind

DH6: Threads has already stated its intentions to not be a platform for mainly political discourse at this and this place. It actively suppresses any political content, discouraging anyone from posting. If threads intention was always to not complete in this space, expecting it to “beat” twitter seems unrealistic
→ Refutes the whole comparison itself. You win or lose when you are playing and this disagreement demonstrates that threads does not even intent to beat twitter

The main thesis was that while these disagreement hierarchies are not necessarily for picking the winners or losers, they are a great filter for which disagreements to engage with

Over the years, as the Internet has become more and more crowded this problem has compounded by several orders of magnitude .

When can you move the conversation

I believe it would be fair to say that most of us would want to disagree at the very highest level ie “refuting the central point”

This is where I have seen well meaning some people try and move the conversation up the hierarchy. I also have tried this on multiple occasions (Though I agree I also sometimes enter disagreements at much lower levels)

For example if somebody were to simply make a counter argument which many a times is not the argument that you were making, but rather some kind of an adjacent argument, you would try to bring back .
For eg refer to the “counterargument stage earlier”

Counter Argument : Twitter has lost most of the advertisers and most of the top news accounts have moved out. Threads on the other hand has 100M DAU.

Your strategy here could be

• restating your point: While you are right on advertisers, twitter is still the place for political content .
• making it narrow and making it clearer. : Ignore advertisers , or news accounts. Let’s only talk about actual users engaging in political conversations . Twitter is still the place
• Ignoring the distraction and not taking the bait to engage in the parallel adjacent argument.

You assume good intent and try to clarify

You do this in hopes that it moves from counter argument to an actual refutation or refuting the central point. I have tried this as an experiment in multiple disagreements I have had with people online, but I have found that this mostly doesn’t really work. 

The chances of a disagreement moving towards higher parts of the pyramid are much less than moving lower. 

Any conversation that goes beyond two or three exchanges and does not move up will almost certainly fall down rapidly, eventually till name calling. Sometimes it will skip many  steps in between. 

The first few exchanges often determine whether the conversation will climb the hierarchy or slide down it

There also seems to be a point of no return in many online disagreements. Once a conversation crosses this threshold, it’s pretty tough to get it back on track

That point potentially is somewhere in the counterargument stage and within that counterargument itself. Perhaps Paulg should update his hierarchy to split the counter argument stage into two parts

No common ground counterargument: Irrespective of how well the counterargument is put, if it is not addressing exactly what the author initially stated, its no-common ground argument
Eg:Twitter has lost most of the advertisers and most of the top news accounts have moved out

Common ground counterargument : Irrespective of what the counterargument  is, as long as it is talking about the same argument it is common ground. The counterargument need not be convincing, or well worded . Its simply needs to address the main point
Eg: Conversation is mostly driven by news accounts. Most top news accounts have moved out and Wall street journal reports that CTRs through twitter have halved

Anything below and including the no common ground counter argument is likely to devolve into name calling with an occasional cancelling attempt . Let’s call it the no hope zone, Or the cancel zone

While all arguments can be up-leveled, the probability below and including no common ground counterargument have much less probability of moving up and much higher to move down. It will take enormous patience and effort to up level that conversation, and it might not be ROI positive

Its opposite for the ones above ie: Common ground counter argument , and refutation

Now this is not to say that do not disagree with people or do not engage with people who disagree, but it is likely a more stricter form of “ At what point do you even want to start engaging” . 

Or alternatively

• If the first few arguments by the person you are disagreeing with are not in the no hope zone, then you can in good faith try and up level it. See where it goes, and if it changes someones mind
• If its in the no hope zone, either ignore or start collecting your best fun insults because that’s where it is going

 It is still Paul Graham’s hierarchy of disagreement but with a filter on top .

The internet is a vast and noisy place. It’s easy to get sucked into unproductive arguments. But by being more mindful of the quality of our own arguments and more selective about the disagreements we engage in, we can raise the level of discourse.

Happy disagreeing

And now the poetry piece:

This blog is called product and poetry And I plan to share some poetry or haikus or ghazals or shers with every newsletter. If it is not in english, I will also give you the translation

I wrote this on a trip many years ago

कैसे लोग हो गये हैं ना हम
कि नदी में पावं डुबोना भी अब अंजाना सा लगता है

What kind of strange people have we become
That dipping your feet in river now feels alien

Whenever you join one of these larger companies you end up owning a piece of product that has been owned by many PMs in the past and has undergone many iterations, decisions, and designs that have either not been documented or were documented but are kind of lost.

One issue that happens with a lot of New PMs is that they get overwhelmed and start “defending” themselves and “deflecting” the blame to their predecessors whenever asked about certain specific decisions. This would happen typically when another team wants to do some changes and wants your opinion on your part the product, or when a customer escalates and you are looped in .

Some statements you would hear:
-I have no clue why this was done, I would not have done that.
-This is ridiculous and makes no sense. Lets remove it
-Whats my take on this? I have none, this was before my time and i dont care if you want to remove it

The problem with this approach is that , for good or for worse , you now OWN THAT DECISION .
But how do you OWN a decision you didnt make , and that my friend is a great question . Here is the strategy

1. You did not make that decision and thats ok: This is something that is clear to everyone. You do not need to know the answer for everything. You also need to internalize this, if someone thinks this design makes no sense its not a comment on you, its just a comment. Your job is what now
2. Figure out : Take a pause and figure out WHY that decision was taken and what are the repercussions. Seems pretty commonsensicle but a lot of new PMs attempt to jump straight to having a POV.
   Some things to check
   
   Legal / policy concerns?: Maybe there was a specific legal ask at that point in time? Maybe its still there. A lot of decisions are sometimes reactionary to the guidelines by various government agencies
   
   Eng concerns : Maybe the systems at that point were not sophisticated enough? Eg: It makes sense to show your scheduled uber rides on the home page, but when it was built it was not big enough use case to make that additional API call to a different system?
   
   Just the best idea at that time : Sometimes the team at that time did the best that was possible. It was a judgement call
   
   Specific market conditions? You would be surprised at how many features get built to address specific market needs which end up being temporary but the feature never get deprecated.
   Maybe your predecessor built this in reaction to a very big partner and that specific design eventually became the generic standard?
   
   USAGE USAGE USAGE : How many users really use it? It has surprised me multiple times at how many tiny features here and there are not being actively looked at. They are being monitored ie if it goes down some eng team will know and fix it , but no ones really checking much.
   
   Tangentially its also possible that the internal analytics systems have been updated and that specific feature is not even being measured . Large companies also have strict data retention policies which mean a lot of historical data is just not there . I have built new features basing my hypothesis on as little as 30 days of data , sometimes no data at all. Gut based calls
3. Your POV
   Now the FUN part . Have answers to these
   – Do you agree with what you saw?
   – What happens if you remove it
   – Has something else been built that can replace this older functionality ( It very frequently is)
   -If it was to address a need, is that still true?
   – What next: I have deprecated features, and sometimes doubled down because we found a gem due to these explorations. In the end you need to finally have a POV.

If you want to be seen as the owner of that product area, it comes with taking ownership of all the decisions of the past, good, bad and the stupid

One of the reasons was not just because it seemed like a great place for content consumption but also a great place for content publishing. You could express yourself in short tweets or tweet threads, you could get instant feedback, you could “produce” more content and it was a pure win win

I used to think these were great things because

• Low bar of entry. Writing a blog post is hard. It takes time and effort. Tweets are “full blown” ideas expreseed in 240 characters. Much more easier to produce. I dont have to wait to spend time to share my thoughts
• I thought the ability to boil down complex topics in 240 characters was great as it forced me to focus on the bare essense of the topic without the fluff. I was kind of proud of mine
• and you could always thread tweets for slightly longer topics
• It was also easier to consume because when I read someone elses tweets / tweet threads it felt like they were talking about the bare essence of their idea without wasting my time
• Many of my tweet threads started becoming Blog posts, which was pretty good way of getting out of the writing rut

But as years went by I started to see some changes

• I started THINKING in tweets: Its one thing to be able to express your ideas in tweets, and another to start “Thinking” in tweet format. Its hard to explain but go with me here a bit
• I stopped self education IF I had enough to tweet : Since there is no nuance in tweets I had a fake sense of understanding of a topic if I had enough information to make a tweet. Even if I did not tweet about it, in my mind it started to feel “good enough” .
  I cannot possibly have great understanding of every new tech out there , but I started realising that I was stopping my education much earlier, in part because I knew I could “tweet” about it if needed.
  Even though it was not the aim, “can I tweet if needed” become a barometer of “if i understood” the topic . This was a LOW bar. For eg when I look back, I am horrified how little I knew about a topic I tweeted on but I can pretty much stand behind all my long forms
• I was consuming more, learning less. Reading through twitter threads of learned people started giving me this false sense of understanding. Do not get me wrong, some were great but many were just meant to be “tweets”
• I forgot tweets are much more temporary, there is little to look back on, little for people to “know you by” , little to refer after years. Blogs and long form are kind of more permanent, so If I do want to be known for some of my work, twitter is only the entry gate not the show
• My blog posts that were inspired by tweets started ending up just being slight expansions of the tweets themselves. They were not writing prompts but just disorganized thoughts put together in one giant pile of garbage blog . It was not useful.

So how do I plan to change

• I want to retrain my brain from thinking in tweets / short form and start thinking long form.
• Ideas are messy and tweets are kind of structured. There is a reason blogs are long, and the core idea needs support, and that telling the “crux” directly doesn’t help. You need to go through the full journey to finally have some understanding . I want to embrace that messiness
• Tweets From blogs and not the other way round. I am still for threads / tweets as a great form of communication but it has to start as a long form and then become a tweet , not vice versa.

Every time I think about writing about Twitter and my point of view on its latest and greatest features, they take a complete U-turn which makes the previous thing completely irrelevant.

Perhaps this is the reason I have not written about Twitter for a while, but I want to talk about Twitter blue and the credibility crisis it is facing

As some of you already know, Twitter introduce something called Twitter blue . When you subscribe to Twitter blue, apart from various features such as edit tweets, the main two things that really differentiate it from a normal users experience is the blue checkmark and an artificial boost to all your tweets.

Now at the surface, both of these things do not seem too wrong but there are some downside to this let’s deep dive

Artificial Boost

Not only tweets from blue are given an artificial boost in the timeline of the users, recently it was announced that only blue users might show up in users for you tab.

there are a few major implications of this.

Tweets as Ads : technically tweets from all blue users now are kind of ads. No I’m not against ads, ads are pretty cool, but they do have a very different connotation than tweets.
Some users like them, but many users may consider them low trust and start ignoring them. People already have something called ad blindness, I suspect that might happen with tweets from twitter blue

Food for thought : There are only 200-300K blue subscribers , and if the For you tab of 200M users is populated only with Blue subscriber content, you could potentially buy a few blue accounts and swamp their timelines. I argue that for an advertiser today it might be cheaper to simply buy a few blue accounts than actually running ads. This is a short term arbitrage which will close soon

Low trust for Twitter Blue tweets: if you see a tweet come to your timeline organically, I bet you might still think it’s relevant or popular, and may be slightly more attention to it. On the contrary, if you see a tweet from a Twitter blue subscriber now show up on your timeline your first gut reaction might be that this person paid for this, while you still see that tweet, it’s likely that you would give it much less attention. I believe the hope is that this reduction in trust is overcome by the increased impressions of the tweet

The blue checkmark Status

The death of status: let’s be very clear, the most likely reason Twitter created a verified program or a Twitter blue program was to prevent impersonation.
Twitter were sued by many celebrities for impersonation. Trying to mix the blue with the verified in my opinion was already a bad idea, and a lot of bad things have already happened. I am not going to go into those, but let’s just look at what Twitter blue stands for now.

Earlier Twitter blue stood for some kind of notability. A lot of people in my opinion bought blue thinking that they can also pretend to have that notability.

The problem that happened now is, Twitter and it’s in finite wisdom, and also because possibly musk had promised to, removed the blue check marks of all legacy verified people as well.

This leaves us with only people who want to buy blue for the specific feature set of blue, main one being boost and “status”.
This attracts a specific kind of audience, which may or may not contain the older legacy verified.

To make things worse, because of all the shenanigans that were done on Twitter in the past, buying Twitter blue in many circles started being looked at a negative status symbol.
Some people disliked it because of how musk treated other people , and some labeled everyone who bought as people who have something to sell.
To give you an sense of latter : think about your favorite or most hated influencer who buys ads to make sure that you see that contact.

I don’t fall in either camp but I too do not give any special credence to tweets by blue anymore. If anything, when i see a tweet by blue subs who I do not follow, my gut reaction is, paid ad.

After removing legacy, the only people that were left were people who bought and the only status left is : “This person paid “

The funny thing about status symbols is that you need people with actual status to use them.

And let’s be honest with ourselves, LeBron has far bigger status than pooja12345.

Twitter is slowly realising that the value to blue came FROM the people who had it, and not as much the other way round.

Blue was the perk that came with Status, not the other way around

Lebron did not need blue, but him having blue gave it credibility and made it aspirational
The status associated with blue was dead the moment legacy blue were removed

As I type this article now I have come to realize that Twitter is now randomly giving celebrities and largest accounts Twitter blue even though they explicitly said they don’t want it. This is perhaps to somehow restore some “status” to the blue checkmark temporarily so that more people buy.

This is not only ridiculous, but unethical. This is misleading advertisement , and potentially illegal

Think about it, how would you rate a startup that decides to put your picture on their profile as their paid user even though you never paid for it what about a start up that claims tesla Twitter and SpaceX, by their software, even when they never did. This is absolutely false advertising, and potentially lead to some interesting lawsuits.

Did you know?
Elon musk is a paid subscriber to my newsletter and so is every employee of twitter, spacex, and Tesla. You on the other hand can get it for free by using this link

The above statement is 100% false.Its a lie, its just a fabrication to drive home a point, dont sue me. I don’t even have paid subscription, but you should still subscribe

Should I be allowed to say the first statement without the second?

Worst is that they are giving these ticks even to dead celebrity accounts. I don’t understand who greenflagged that

Things are so bad in some circles that celebrities are sharing tricks to to GET RID of the blue check mark.

But status loss has a catch

Status loss is slow and takes time to reach the masses. For Eg: Top quora voice was a status once, forbes 30under30 is slowly going away, Tedx speaker is basically meh . In the interim a scammer could easily fool people with a “bought” status. I do expect twitter to become more scammy and spammy because of this feature.

The only hope for blue possibly is if enough notable people subscribe, before it gets road rolled by spam and scam .

Will I buy twitter blue?
Well this question to me currently is similar to “will I buy ads”.
Sure if i have a need for it.

Eg If I finally write my book on product management, maybe.

But, one thing I will be well aware of is that my tweets now are kind of ads and that I have no real “status” because of blue.

A lot of my engagement would no longer be because of my banger tweets, but rather my 8$ subscription.

Do I judge you for taking blue
Well no, but
Assume my For You tab has two tweets from account I do not follow
1) From someone without blue
2) Your tweet with blue

My gut would automatically tag yours as possibly lower quality because your tweets did not fight millions of other tweets to be shown here, but rather are there because you paid for me to see it. But I will likely see much more of you. I think that is a fair tradeoff .

A good analogy is , assume you are a harvard student and harvard students are pretty smart. One of your friend tells you they paid to get in while another one cracked an exam. Just based on this information would you in your mind have higher bar to give credence to the smartness of the one who paid? They could be the smartest of all…but you likely want to see first, while you may accept the other person’s smartness at face value

But if I do follow you, I already find value in what you say. Your blue status will likely just surface more of you …i just hope not too much to annoy me from unfollowing.

This is possibly the bargain I will have to make if I went blue

Let’s first start with what an LLM (ChatGPT, BARD) , Large language model is, simply put, a model generated by feeding humungous amount of data into a neural network allowing it to “Predict” what the next word would be based on what was the previous words were . For eg : If you say “He is”, based on older data it has seen it might assume the next word is “a” . “He is a”… Your prompts are also part of the chain so if you asked “Who is elon musk” it might say “he is a founder” ….So on and so forth.

I use this basically to give myself a sense of intuition to help me understand what is happening. And once you get it, you understand why it is so good at what it does, and what are the apparent issues that exist

It obviously is very good because it has trained on so much data that its predictions are really spot on.
Humans are predictable, and so is our language.
When the system has seen so much text, it kind of has enough data to predict very precisely what the next word is

But that also means an LLM is NOT trying to actually give you the right answer, It simply is trying to predict what the next word is.
It has no sense of meaning, no idea of actual structure, no style.
This is also why it goes into “hallucinations” where it will invents facts, even papers, or just random stuff. It is not doing it on purpose, it’s just predicting the next word.

This also means that all the biases that exist in the real world also exist in the model. This is not so different from many other models out there, just much more apparent in LLM

Can the LLMs actually become factually accurate?

Depends on who you ask. While a lot of people do think that accuracy problem can be solved via various techniques like proper prompting, cross reference, citations, more training etc, some prominent AI researchers like Yann LeCun think otherwise

His argument as I understand in a nutshell is this: Assume there are n number of potential ways to answer correctly, for it to reach one of those n states it needs to follow a very narrow specific set of paths (Words string together), one word here and there would take you to a whole different tangent of what we call “AI Hallucinations”.

If he is correct, its not exactly doom and gloom for LLMs, it just means its applicability needs to be in fields which are more immune to occasional factual inaccuracies. Eg: LLMs are great for script writing, copywriting, marketing, maybe basic education

Also it might be an issue of expectation management. When you search for something you see multiple websites that try to answer your query, some that are accurate and some that are not. Search engines like Google, Bing try to find the most trusted sources and constantly upgrade based on user’s feedback. But you are mostly aware than you are getting your answer from a “website”.

if there is an inaccuracy you are more forgiving towards the search engine and think about how to rephrase your query or go to another search search result. In case of chat interface the expectation of accuracy is high because there is just 1 answer and LLM models sound very confident. Perhaps slowly people will start to adjust their expectations

But is it JUST predicting words

Well you say as if that’s a bad thing. So its likely a lot is going on behind the scene and under the hood.

For one, just because it is simply “predicting” doesnt mean its a pure probabilistic thing. Its likely that our latent structures, grammer, style is encoded into the model .

One food for thought is that if structures in a language also gives it meaning, does that mean it gets meaning ?

Language itself is not too impossible to predict .Take for example Zipf’s law . It states that the second most used word in a language is used half the time of most used word. The third most used word is used half of second most …and so on.

Its seen not just in english but across every language. Even the ancient ones we do not know.

This might eventually be a way for us to even communicate with aliens.If their communication does not follow zipf’s law, are they even intelligent? And the trouble is, we don’t exactly know why every language does that

So what next

Accuracy remains one of the top concern. its likely our near future will be an army of “Fact checkers” who would simply be checking if the LLMs got it right before putting the content to use. Then LLM become a great tool, a partner to many professions super charging productivity. But the field of “content generation” and “creative professions” like writing articles, maybe even books is definitely disrupted forever. Hallucinations are not just tolerated, but rather preferred here.

All Art is stolen, all innovation is a copy, we all stand on the shoulders of the giants

But what if we are the last giants standing . What if the next thing to stand on our shoulders were to crush us, would we still let them?

And that my friend is an interesting point of discussion

If you are anyone in anything related to tech, you already know that AI is developing at an unprecedented pace.

Unlike some previous iterations of AI products, it seems this iteration is finally good enough to actually replace a lot of work and fool humans at scale. Some assume it will replace mostly grunt work, but some think even creative work is going to feel the burn

A lot of creativity is also Subjective and depends a lot on the person who consumes the art. To a non trained eye, a painting by da vinci and a art student today is not too different and I bet if you dont tell which is which, not many can tell the difference. So “creativity” itself does not need to be at the extreme end of the spectrum, but just average or above average…just like humans

And that brings me to the central point, AI has reached a point where it can creatively create content like blogs, movie scripts, images, etc . Since it is mostly trained on existing data, let’s assume it does not have a unique style, but rather copies the style it reads (Again not too different from humans)

And it is very capable of replicating a style, you can see it in practice by simply forcing a style.
Eg: Here is Dall E and Stable diffusion imagining Mona lisa in style of Van gough and Starry night in style of da vinci

So if you are someone who is developing a unique style , would you want the AI to be able to copy it?

A human, sure , because a human will take years to potentially master your style, but AI? It will probably take days ,or till the next model update.

Also , do you even have a unique style? What is a unique style anyways

I do not have an answer, because I can see the argument both ways but I do think these are becoming important questions. So here are a few predictions regarding humans creating content that I can make

1: Right to Not train Laws: There might soon be laws coming in that will allow you to opt your content from training set. Stable diffusion etc already started some, but much more will come. Large tech will follow it (Something like Do not index tags), but many may not .

2: More knowledge behind paywalls, more live classes, more cohort courses with NO recording allowed: The best teachers will not want to leave a trail online . More password protection

3: Stringent “Ai created” tagging
More aggressive “AI created” “Human Created” “AI Assisted” tagging, might even be mandated by the government

I don’t think the third one will work though. No one cares today if you did that calculation on paper or used a calculator. The content trumps the creator eventually .

I could be wrong though, people still crave for “natural diamonds” even though lab created ones are purer, similarly “handicrafts” is a big market. These things are marketing based and mostly target a very specific set of people, so who knows we might see an emergence of “Written by human”

Nevertheless, this article was written 100% by human

I did eventually ask ChatGPT to rewrite this article : Here is the result . You can ofcourse make it sound cooler

Prompt: Write the article in the style of Paul graham

Can AR Rahman or James Cameron prevent you from being inspired by their work? Can they stop you from using their narrative structures or artistic styles in your own creations?

All art is a copy, and all innovation is built upon the work of those who came before us. We stand on the shoulders of giants. But what if we are the last giants standing? What if the next generation of creators, fueled by the power of AI, were to crush us? Would we still be willing to let them stand on our shoulders?

These are the questions that we must consider as AI continues to develop at an unprecedented pace. This iteration of AI is finally good enough to replace a lot of work and even fool humans at scale. Some believe it will only replace grunt work, but others think even creative work is at risk.

While a lot of creativity is subjective and dependent on personal taste, it’s important to note that AI is now capable of creating content like blogs, movie scripts, and images with relative ease. And since AI is primarily trained on existing data, it tends to replicate rather than create unique styles.

For creators who have spent years developing a unique style, the thought of AI being able to replicate it in just a few days is daunting. And even if you don’t believe you have a unique style, the question remains: do you want AI to be able to replicate your work?

This raises important ethical questions that we must address. In the near future, we may see laws that allow creators to opt out of being part of AI training sets. We may also see an increase in live classes, cohort courses with no recordings allowed, and more password protection for creative work.

To distinguish between AI-generated and human-created content, there may be a need for more stringent “AI created” tagging. But whether or not this will be effective is up for debate. After all, people still value natural diamonds over lab-created ones, and handmade crafts over mass-produced goods.

Ultimately, the content itself will always be more important than the creator. But as AI becomes more advanced, we must consider how we want to protect our creative work and the legacy of those who came before us.

As for this article, it was written 100% by a human. But who knows what the future holds?

Well #generativeAI will be our generation’s generation gap

Eventually only a few of our generation will get the full potential.
But our kids will be born in it, shaped by it

Like computers took our parents generating by surprise even though it started showing signs when they were in their prime…The same is happening now.

I am preparing myself for asking annoying questions to my son about how to work something, but I am afraid he will delegate that task to an AI too

This new “tech” gap may eventually not be that big due to a more inclusive nature. It requires no new skills to use it…

There will likely be no setups, no trainings(Like annoying voice matches) , no breakdowns you could even fix..

It will be a blackbox that will simply “BE” and you just interface with it naturally

Here are my early thoughts

In my 15 yrs in tech world, I have never seen so much transformative innovation hitting consumer products in so little time. It’s no longer a linear acceleration of tech, but rather an exponential leap. THIS WILL CHANGE EVERYTHING.

And this brings me to the provocative title of this post. I know its a bit alarmist, but hear me out

Typically companies had specific moats like IP, great software talent, Huge cash reserves, government regulations, users, network effects.
A lot of these were function of :Time to market and sustained lead due to better product
Better product was a function of better strategy and just talent

These things are now going away.

Now you CAN build a working product without knowing coding , imagine what you could do if you knew little coding. The difference between an average coder and excellent coder is increasing while a non coder and an average coder is shrinking rapidly.

No code tools was a good start, but what if you could code? And that’s what this new generation of tools will usher

Time to market is going to reduce drastically. Since you no longer need lots of excellent developers, the cost of building and maintaining good enough products is dropping significantly. Remember when you used to pay 14Rs/min for a mobile phone call? Now the same company offers almost unlimited calls for 5rs/day

If you build a new startup and gain slight traction, you will see good enough copies in a matter of days rather than months and years. Imagine being able to launch a clubhouse competitor the day after clubhouse launched . Imagine if twitter spaces and facebook rooms was launched hours after clubhouse got its first large meeting. The “Speed” moat is gone.
Sure everyone can develop faster but speed of thought has its upper limit.

So what could you even do? Especially if you are a startup…

• Assume disruption is months away
• Don’t wait to be too big to start charging. You don’t have time . Make money from get go so that you have reserve for pivots
• For many companies, Pivots will become the way of survival …Like microsas . You keep building what is needed NOW
• Acquire, get acquired, buy, sell , survive
• Big generational companies will now need to build some deeply fundamental tech. New ways to do LLMs? , basic physics?, Biology, Aging.
  Building companies which are essentially wrappers on top of various other services are in trouble.
  SASS Especially is in deep trouble, its likely that soon it would be cheaper to build than to buy .

The world is getting very interesting, but also slightly scary. You will feel old soon. The generation gap you feel when you try to explain tech to your parents…Well this will be our generations generation gap

One problem that stumps many new PMs is measuring the impact of small features.

Eg: how do you justify any changes to orders screen, or making slight changes to the text in some obscure corner of product, or just informational changes .

What if you added small delight features like a fancy error message or a funny wait timer.

While you would LOVE to A/B test everything and see if these changes are positively affecting the core metric, unless you have a large number of users hitting those scenarios, you would not be able to get meaningful stats sig results.

Companies with billions of users can test even the minutest of things, eg Google allegedly tested 41 shades of blue , and Uber can test its custom fonts, but it’s unlikely that many companies will have the scale necessary to observe a stats sig result.

So how do you make a case for making those changes and if you do that, how do you measure these.

One way of course is to simply not measure the success (Or measure but do not expect). Think of these as paper cuts, its annoying but one won’t really kill you. The idea is that you go with what you deem is right and not get into analysis paralysis mode. You see a paper cut and fix it.

But relying too much on gut as two major disadvantages

1. Experience: Unfortunately developing a good “gut reaction” is a tricky problem and requires a lot of past experience to rely on. “Gut” is but a culmination of lot of data you have already seen
2. What is not measured is not rewarded: Unless you are running your own startup or have significant stake in one, you want to make sure you get recognised and rewarded for your work. If there is no way to measure something, there is no real way to recognise and reward it. All it may get you is a pat on the back . It is also deeply unsatisfying because you have NO idea if you are actually adding value
   

One simple way of measuring these changes is “constant holdback”

Whenever you roll out such an experiment, instead of rolling it out to 100% of users, roll it out to only 95% of your users. Do the same for all other small changes, with the same 5% always excluded.

As experiments pile up, the effect of multiple paper cuts being fixed would start showing up .

You would potentially be able to see the holdback group having meaningfully different(worse) metrics than the rest of the group..

But how do you know which experiment worked:

That is the point, you potentially will not. You will see cumulative effects and not specific.

What if some experiments are actually harmful

The idea is not to have all super positive experiments but to know if you have been directionally right. The trick is to pick mundane obvious changes that you need to do rather than using this for large features. If you are directionally right, you could eventually see a nice bump.

Can I do this for large new features

If you can do control vs treatment A/B test for any feature I would advise you to not use this holdback method as the primary means of testing. You do not want to have large experiments in the mix that can completely change the general direction of overall result

But I would argue that this hold back is useful even for large experiments once you have tested them . Post your A/B experiment you can think about a holdback for large features as well. The reason is that not all changes are plain additive. Eg: If you see your conversions go up by 2% via one feature and 3% via another, on long term its not necessary that overall it would be 5% . A constant holdback would tell you the cumulative effect of all the large changes you may have done

Do not forget to delete the holdback eventually so that all your users can see the same improved experience.

Been hearing a version of these questions a lot. Its there on twitter and sometimes even the Musks filings. There are three premises to this argument that I hope to address in this article

Premise 1: Twitter is using a non- standard metric

Premise 2: It is a bad metric

Premise 3: Twitter is not measuring it right

Premise 1: Twitter is using a non- standard metric

Let’s be clear, there is NO standard usage metric that you are supposed to report . The government mandates public companies disclose financial data in certain format, but not how a company measures usage.
Every company decides what are the most important measures for it and reports them. Even the basic metric like what is a defined as “Active” can vary from company to company based on it’s footprints

Eg: Snap chat only looks at people who opened their app(Annual report)

We define a DAU as a registered Snapchat user who opens the Snapchat application at least once during a defined 24-
hour period.”

From snapchat annual report

Whereas Pinterest(Annual report) accounts for all kinds of actions including web visits

We define a monthly active user as an authenticated Pinterest user who visits our website,
opens our mobile application or interacts with Pinterest through one of our browser or site extensions, such as the
Save button, at least once during the 30-day period ending on the date of measurement

From Pinterests annual report

Companies, especially large ones, routinely create their own combination metrics that make most sense to them.

Eg: You won’t care much about how many times Uber app was opened, you care about how many users actually took a trip.

Uber has it’s own metric called Monthly Active Platform consumer

Monthly Active Platform Consumers. MAPCs is the number of unique consumers who completed a Mobility or New Mobility ride or received a Delivery order on our platform at least once in a given month, averaged over each month in the quarter. While a unique consumer can use multiple product offerings on our platform in a given month, that unique consumer is counted as only one MAPC. We use MAPCs to assess the adoption of our platform and frequency of transactions, which are key factors in our penetration of the countries in which we operate.

From Uber’s annual report

Similarly Facebook, which is a direct competitor of twitter, is also introducing a new metric called Daily Active People (Annual report)

Family metrics represent our estimates of the number of unique people using at least one of Facebook,Instagram, Messenger, and WhatsApp

From Facebook’s annual report

Premise 2: It is a BAD metric

In short: Twitter’s mDAU metric is the number of people who it can show ad to . Twitter removes potentially suspected bots, spam accounts, and also accounts only posting via APIs etc from mDAU count(See more details)

This is an ABSOLUTELY GOLD metric. If you are in the business of selling ads, making sure you only show ads to real humans is an extremely important measure.

Every company would have some measure of this. Twitter just chose to disclose this and make this their key metric. It is a strong signal that they are in the business of selling Ads.

Twitter can absolutely disclose and measure overall spam accounts, but that does not take away the validity of the mDAU metric.

I also keep hearing that twitter removes Bots and SPAM from it’s calculations of mDAU, do you know who else does that ? Pinterest . Here is a direct quote from their annual report

We regularly deactivate false, spam and malicious automation accounts that violate our terms of service, and
exclude these users from the calculation of our MAU metrics;

From Pinterest’s annual report

Pinterests Daily Active user seems functionally equivalent to twitters monetisable Daily active users. I see nothing inherently bad in this metric.

Another point to note: Facebook and Snapchat seem to not take out spam accounts from their Daily active user count. Facebook does mention how many active users it suspects to be spam, but I did not find anything related to that in Snapchat’s filings.

There is no consistency or rule on what is a”Daily active user”

Premise 3: Twitter is not measuring it right

Here is the process twitter follows, it has a bunch of AI / ML algos that automatically remove as many accounts as possible that. it suspects are spam . Twitter than samples 100 accounts per day(9000/quarter) from the rest of the accounts and have manual reviewers rate if these accounts were SPAM or not(Triple checked I read somewhere).

They do this everyday to get a trend and find that approx number of spam users that pass through their filters is <5%.

There are three objections I hear in this regard

Objection 1: The Sample size is too small

Statistical significance is not related too much to sample size, but rather to sample selection. Typically a sample size of 100 can give great representative results for a large population, twitter is doing 9000(over a quarter).

Eg: CNN did 2020 presidential election exit poll with a sample size of just 15,590

Objection 2: They do manual review

Of course they do. They already used their AI/ ML algos to filter out all spam they could and now manual is the last step. Infact, even facebook uses manual reviewers to tag spam

Facebook defines them as “Violating accounts” (Another non standardised name)

We define “violating” accounts as accounts which we believe are intended to be used for purposes that violate our terms of service, including bots and spam.

From meta’s annual report

It goes on to explain how they determine if an account is violating

Such estimation is based on an internal review of a limited sample of accounts, and we apply significant judgment in making this determination. For example, we look for account information and behaviors associated with Facebook and Instagram accounts that appear to be inauthentic to the reviewers

From meta’s annual report

Objection 3: it can’t be 5%

Do you know how much TOTAL spam facebook claims it has? 3%. NO Kidding

we estimated that approximately 3% of our worldwide MAP consisted solely of violating accounts”

From meta’s annual report

Assuming there is no lying, if facebook can have only 3% of all its users as SPAM, twitter’s 5% SPAM after SPAM filters does not seem off.

I do suspect though that I may have missed something and facebook also removes suspected SPAM accounts before calculating “Violating accounts”, which makes it potentially functionally equivalent to mDAU of twitter

Also remember, mDAU is NOT a user facing metric.Your own experience is immaterial. Twitter can have 20% SPAM and still have only 5% SPAM in mDAU.

I will leave you with this diagram to chew on

So NO mDAU is not really that non standard, nor is it specifically bad , nor does it seem twitter’s revealed methodology is anything shady.

There can obviously be something deeply wrong with twitters count if they are hiding something, but I am unable to see any info about that.

Want to know about a real BAD metric that’s super popular and is also stated in annual reports? Read why NPS scores are useless