Hi Congrats on achieving this milestone! And thanks for having this AMA as well There had been news that Bard’s model is only 770m parameters and GPT4 is about 1 trillion. Is there any truth to that in your opinion? Also, I am curious are there any other interesting or promising AI models on the horizon that’s worth looking at other than…
Quite possible, actually. Google said Bard would be first rolled out with a lightweight version of LaMDA (although 770M sounds extremely tiny). The 1T number for GPT-4 has been running around. I've also seen some estimates (calculated by prompting the model to count to a large number and comparing that to how much it takes for GPT-3.5 to do the same--not super reliable) that put the number around 2-3T, IIRC.
About the forward forward algorithm, I think it's too early to compare it to backprop. But I don't think backprop is really how human brains learn, so we may have to go a different direction eventually. Maybe not.
Hi
Congrats on achieving this milestone!
And thanks for having this AMA as well
There had been news that Bard’s model is only 770m parameters and GPT4 is about 1 trillion. Is there any truth to that in your opinion?
Also, I am curious are there any other interesting or promising AI models on the horizon that’s worth looking at other than transformer-based?
Lastly, what do you think of forward forward algorithm vs backpropagation?
Thanks
Quite possible, actually. Google said Bard would be first rolled out with a lightweight version of LaMDA (although 770M sounds extremely tiny). The 1T number for GPT-4 has been running around. I've also seen some estimates (calculated by prompting the model to count to a large number and comparing that to how much it takes for GPT-3.5 to do the same--not super reliable) that put the number around 2-3T, IIRC.
About non-transformers, I recently saw a paper on RNNs (hadn't heard about that acronym in years) that showed promising results: https://johanwind.github.io/2023/03/23/rwkv_overview.html
About the forward forward algorithm, I think it's too early to compare it to backprop. But I don't think backprop is really how human brains learn, so we may have to go a different direction eventually. Maybe not.