Thanks for yet another great piece, Alberto! As someone working with SEO and content, I've been wondering how ongoing AI trends will affect the field, especially if we get to a point where search engines can reliably consolidate and deliver answers across multiple sources.
What will that mean for the entire "thought leadership" approach where companies try to become knowledge hubs for topics and keywords related to their products and services. Today, the reward is "free" organic traffic...but what if people no longer have the incentive to click through to the company site because LM+SE combo becomes much better at delivering thorough, accurate, and neutral answers?
Very interesting analysis. The problem that I keep coming back to is that search ads constitute something like 80% of Google's revenue (and a smaller, albeit still large, percentage of Alphabet's as a whole). This means that MSFT, if it wanted to, would only have to bleed off a small amount of Google's search revenue to seriously harm Google's ability to generate revenue and so finance its operations. On the other hand, Google (and its parent, Alphabet) has a strong balance sheet, so it could finance a war of attrition with MSFT for a while. But its stock would tank, and so its ability to recruit and retain employees would decline. MSFT, on the other hand, has a more diversified revenue stream, and if it decides that it's willing to finance losses on Bing for a while, it won't be as harmed by that decision. Of course, I am looking at this through a financial, not technological, lens. But I very much see this as an Innovator's Dilemma kind of problem, as articulated by Clayton Christensen in his book of the same name.
And unfortunately I lack expertise on the technological side. I think that the ultimate truth requires a synthesis of both the financial and technological aspects.
I agree with your points about how the search space might play out and the risks to each side of this equation - Microsoft vs. Google
One question that's been troubling me though - if we look outside of the Search market. Say for any other product - do you think Google and Microsoft (+Open AI) will be the only two major owners of LLMs?
How easy or difficult is it for any other player (say Amazon) to build an LLM that can compete with ChatGPT and add it to their own products? (Say to power the search experience, etc.)?
"do you think Google and Microsoft (+Open AI) will be the only two major owners of LLMs?"
They aren't. Facebook, Amazon, Nvidia, Apple, all have experimented with LLMs of their own, that's without mentioning the Chinese big tech (Tencent, Alibaba, Baidu...)
But, answering your question, the main bottlenecks to training LLMs are data and computing power (technical talent is also important but I wouldn't say it's the main factor) so only very resourceful companies can now compete at that level (maybe they're already big, like Apple or Amazon, or they have attracted wealthy investors like OpenAI or Anthropic).
Governments and universities have, respectively, money and talent, so they may find ways to participate somehow (of course, governments can do whatever they want to some degree).
Thanks Alberto - I totally agree with you that others too have experimented with LLMs. I appreciate your patient and thoughtful answer.
So if I understand you correctly - even within the scope of very large companies, given the resources of say Amazon, Meta, etc. - it is likely that they might not catch up with OpenAI / Microsoft or Google's DeepMind? Or at least take a bunch of time to do so. Given the very complex distributed computing + cloud infra. capabilities and costs required to setup the training of a large language model. Not to mention the specialized AI research knowledge required to develop a model, and even more deep learning expertise required to continue improving those models.
Question 1 - How would you poke holes in this argument? What am I missing? Or is this actually something you do not agree with?
Question 2 -
I am basically trying to form a POV on whether this means that the best LLM tech., over a 5-10yr horizon, is going to be in the hands of -
1. A couple companies (Goog, Msft), or
2. A handful of companies (FAANG+), or
3. Most companies and startups (completely democratized)
Do let me know! Keen to hear your thoughts on this.
Regarding your first question ("it is likely that they might not catch up with OpenAI / Microsoft or Google's DeepMind?") I don't think that's what will happen. As insiders like to say, there's no "moat" beyond what I mentioned. Enough money could put others at the level of Google and Microsoft with time.
However, as you suggest with your second question, there's some nuance. All three scenarios you describe hold some truth:
1. Google (Google Brain, DeepMind), Meta (FAIR), and Microsoft (OpenAI) are ahead of the rest and will likely be leading in the years to come. They have plenty of resources and have been working on AI for a long time.
2. FAANG+ companies could, if they wanted, develop state of the art LLMs in a short time (much lower than 5-10 years) but will hardly match Google's or Meta's presence in AI.
3. Other companies will, with fewer resources, develop capable LLMs like ChatGPT (e.g. Anthropic, Cohere, AI21 labs, etc.). Also, sooner than later others will release open-source LLM-powered chatbots (e.g. Stability.ai, EleutherAI, Hugging Face). It's a matter of time, although they won't ever be ahead of the Googles or Metas of the world.
So, it's not just one of those three options, but in some sense those three at the same time.
A thought: If virtual assistants like Siri finally start to work as intended in the near future on the back of LMs, couldn't search quickly be performed by them exclusively - without any human involvent? Synthesis could be done equally well by a system the user has paid for directly to provide other high value services as well.
Especially if concerns about privacy and trust/alignment pervail, individually fine tuned assistants might become the norm.
This could keep the need for a search or at least a catalogization supplier more or less intact, but would bring upheaval to the "free", ad financed business model as well.
The companies best able to provide such assistants might be those already monetizing through hardware sales or software subscriptions, like Apple or (surprise) Microsoft.
Interesting perspective! A couple of counterpoints:
"without any human involvent." But is that what we want?
"individually fine tuned assistants might become the norm." Yes, this will likely happen eventually. However, how much of this power do we want Apple and Microsoft (and others) to have?
Why can't Google offer two search engines, one optimized for authoritative resources and another LM based one ... then users can make an informed choice ... I personally can see myself using both depending on the question I have.
This is probably what most companies will offer, at least while there's no clear better option (i.e. a combination of both that's better than the sum of the parts). A traditional SE and additional non-conflicting features that integrate the capabilities of LMs.
However, if a company finds a way to combine LMs and SEs successfully, that'd be game-changing.
Great piece Alberto! I liked particularly the way you describe the incentives and dilemmas of the tech giants.
One detail I'd add is to consider Google's Knowledge Graph (KG). As you know, the KG is instrumental to getting direct answers in Google search (the snippets on top of the search results in some searches). For instance, if you search for "who was Nero's mother?" you get, additionally to a ton of links, the answer itself: "Agrippina the Younger." This is done using the KG and some undisclosed algorithms.
Now, I think the combination of LLM with search will need to use the KM, both because it's the current question answering used by Google and also because it's factually guaranteed. It seems to me that there should be a way of using LLMs to develop more elaborated answers compatible with the KG. The use of the KG will also finish the "hallucinations" so common in Generative AI systems.
Thanks for the insightful comment, Ramon. Agreed, this is the key: "... because it's factually guaranteed." That's one of the main flaws of LMs. How can Google or Microsoft combine KGs and LMs successfully? I don't have a clue. Would that be game-changing? No doubt.
The deep issue is the impact of LLMs on the search business model. Today's Google Search Business litters the results page with garbage, irrelevant and misleading content (and some potentially valuable pointers) scattered about. LLMs force a rethinking of how better to inform USERS. Look at YOU.COM for examples. USERS don't need dozens of pages of crap. They will flock to generative results that are enriched by older search tools. That will kill the current "litter the results with crap" model that funds Google. Google won't die, of course, but this is going to force EVERYONE to pay more attention to user needs FIRST.
Thanks for yet another great piece, Alberto! As someone working with SEO and content, I've been wondering how ongoing AI trends will affect the field, especially if we get to a point where search engines can reliably consolidate and deliver answers across multiple sources.
What will that mean for the entire "thought leadership" approach where companies try to become knowledge hubs for topics and keywords related to their products and services. Today, the reward is "free" organic traffic...but what if people no longer have the incentive to click through to the company site because LM+SE combo becomes much better at delivering thorough, accurate, and neutral answers?
I'm very curious to see how this plays out.
SEO has been in jeopardy for a few years now due to LMs. Now it's clearer than ever... We'll have to wait to see how all this plays out!
Very interesting analysis. The problem that I keep coming back to is that search ads constitute something like 80% of Google's revenue (and a smaller, albeit still large, percentage of Alphabet's as a whole). This means that MSFT, if it wanted to, would only have to bleed off a small amount of Google's search revenue to seriously harm Google's ability to generate revenue and so finance its operations. On the other hand, Google (and its parent, Alphabet) has a strong balance sheet, so it could finance a war of attrition with MSFT for a while. But its stock would tank, and so its ability to recruit and retain employees would decline. MSFT, on the other hand, has a more diversified revenue stream, and if it decides that it's willing to finance losses on Bing for a while, it won't be as harmed by that decision. Of course, I am looking at this through a financial, not technological, lens. But I very much see this as an Innovator's Dilemma kind of problem, as articulated by Clayton Christensen in his book of the same name.
Thanks for the additional analysis, Dave. I lack expertise on the financial side. This is helpful!
And unfortunately I lack expertise on the technological side. I think that the ultimate truth requires a synthesis of both the financial and technological aspects.
And also social, political, psychological, philosophical...!
Thank you for this insightful piece Alberto!
I agree with your points about how the search space might play out and the risks to each side of this equation - Microsoft vs. Google
One question that's been troubling me though - if we look outside of the Search market. Say for any other product - do you think Google and Microsoft (+Open AI) will be the only two major owners of LLMs?
How easy or difficult is it for any other player (say Amazon) to build an LLM that can compete with ChatGPT and add it to their own products? (Say to power the search experience, etc.)?
Would appreciate your thoughts!
"do you think Google and Microsoft (+Open AI) will be the only two major owners of LLMs?"
They aren't. Facebook, Amazon, Nvidia, Apple, all have experimented with LLMs of their own, that's without mentioning the Chinese big tech (Tencent, Alibaba, Baidu...)
But, answering your question, the main bottlenecks to training LLMs are data and computing power (technical talent is also important but I wouldn't say it's the main factor) so only very resourceful companies can now compete at that level (maybe they're already big, like Apple or Amazon, or they have attracted wealthy investors like OpenAI or Anthropic).
Governments and universities have, respectively, money and talent, so they may find ways to participate somehow (of course, governments can do whatever they want to some degree).
Thanks Alberto - I totally agree with you that others too have experimented with LLMs. I appreciate your patient and thoughtful answer.
So if I understand you correctly - even within the scope of very large companies, given the resources of say Amazon, Meta, etc. - it is likely that they might not catch up with OpenAI / Microsoft or Google's DeepMind? Or at least take a bunch of time to do so. Given the very complex distributed computing + cloud infra. capabilities and costs required to setup the training of a large language model. Not to mention the specialized AI research knowledge required to develop a model, and even more deep learning expertise required to continue improving those models.
Question 1 - How would you poke holes in this argument? What am I missing? Or is this actually something you do not agree with?
Question 2 -
I am basically trying to form a POV on whether this means that the best LLM tech., over a 5-10yr horizon, is going to be in the hands of -
1. A couple companies (Goog, Msft), or
2. A handful of companies (FAANG+), or
3. Most companies and startups (completely democratized)
Do let me know! Keen to hear your thoughts on this.
Regarding your first question ("it is likely that they might not catch up with OpenAI / Microsoft or Google's DeepMind?") I don't think that's what will happen. As insiders like to say, there's no "moat" beyond what I mentioned. Enough money could put others at the level of Google and Microsoft with time.
However, as you suggest with your second question, there's some nuance. All three scenarios you describe hold some truth:
1. Google (Google Brain, DeepMind), Meta (FAIR), and Microsoft (OpenAI) are ahead of the rest and will likely be leading in the years to come. They have plenty of resources and have been working on AI for a long time.
2. FAANG+ companies could, if they wanted, develop state of the art LLMs in a short time (much lower than 5-10 years) but will hardly match Google's or Meta's presence in AI.
3. Other companies will, with fewer resources, develop capable LLMs like ChatGPT (e.g. Anthropic, Cohere, AI21 labs, etc.). Also, sooner than later others will release open-source LLM-powered chatbots (e.g. Stability.ai, EleutherAI, Hugging Face). It's a matter of time, although they won't ever be ahead of the Googles or Metas of the world.
So, it's not just one of those three options, but in some sense those three at the same time.
Got it - thank you so much Alberto for these insights! As always its a pleasure reading, and to know your thoughts on where things are going!
Amazing article, thanks a lot!
A thought: If virtual assistants like Siri finally start to work as intended in the near future on the back of LMs, couldn't search quickly be performed by them exclusively - without any human involvent? Synthesis could be done equally well by a system the user has paid for directly to provide other high value services as well.
Especially if concerns about privacy and trust/alignment pervail, individually fine tuned assistants might become the norm.
This could keep the need for a search or at least a catalogization supplier more or less intact, but would bring upheaval to the "free", ad financed business model as well.
The companies best able to provide such assistants might be those already monetizing through hardware sales or software subscriptions, like Apple or (surprise) Microsoft.
Interesting perspective! A couple of counterpoints:
"without any human involvent." But is that what we want?
"individually fine tuned assistants might become the norm." Yes, this will likely happen eventually. However, how much of this power do we want Apple and Microsoft (and others) to have?
Why can't Google offer two search engines, one optimized for authoritative resources and another LM based one ... then users can make an informed choice ... I personally can see myself using both depending on the question I have.
This is probably what most companies will offer, at least while there's no clear better option (i.e. a combination of both that's better than the sum of the parts). A traditional SE and additional non-conflicting features that integrate the capabilities of LMs.
However, if a company finds a way to combine LMs and SEs successfully, that'd be game-changing.
Btw, thanks for the article ... a lot has been written about ChatGPT, but this was interesting.
Thanks you for reading Alexander!
Great piece Alberto! I liked particularly the way you describe the incentives and dilemmas of the tech giants.
One detail I'd add is to consider Google's Knowledge Graph (KG). As you know, the KG is instrumental to getting direct answers in Google search (the snippets on top of the search results in some searches). For instance, if you search for "who was Nero's mother?" you get, additionally to a ton of links, the answer itself: "Agrippina the Younger." This is done using the KG and some undisclosed algorithms.
Now, I think the combination of LLM with search will need to use the KM, both because it's the current question answering used by Google and also because it's factually guaranteed. It seems to me that there should be a way of using LLMs to develop more elaborated answers compatible with the KG. The use of the KG will also finish the "hallucinations" so common in Generative AI systems.
What do you think?
Thanks for the insightful comment, Ramon. Agreed, this is the key: "... because it's factually guaranteed." That's one of the main flaws of LMs. How can Google or Microsoft combine KGs and LMs successfully? I don't have a clue. Would that be game-changing? No doubt.
Great read! Esp the tweet by François Collet is very memorable/decisive. I wrote a related piece on knowledge retrieval (including Deepmind's RETRO) btw: https://scalingknowledge.substack.com/p/knowledge-retrieval-transformers
Thanks for sharing, Moritz!
The deep issue is the impact of LLMs on the search business model. Today's Google Search Business litters the results page with garbage, irrelevant and misleading content (and some potentially valuable pointers) scattered about. LLMs force a rethinking of how better to inform USERS. Look at YOU.COM for examples. USERS don't need dozens of pages of crap. They will flock to generative results that are enriched by older search tools. That will kill the current "litter the results with crap" model that funds Google. Google won't die, of course, but this is going to force EVERYONE to pay more attention to user needs FIRST.
Look at the YOU.CHAT part of YOU.COM. Or go use ChatGPT.