ChatGPT

9516 readers

18 users here now

Unofficial ChatGPT community to discuss anything ChatGPT

founded 2 years ago

MODERATORS

[email protected]

146

GPT-4's details are leaked. (threadreaderapp.com)

submitted 2 years ago by [email protected] to c/[email protected]

26 comments fedilink hide all child comments

cross-posted from: https://lemmy.intai.tech/post/72919

Parameters count:

GPT-4 is more than 10x the size of GPT-3. We believe it has a total of ~1.8 trillion parameters across 120 layers. Mixture Of Experts - Confirmed.

OpenAI was able to keep costs reasonable by utilizing a mixture of experts (MoE) model. They utilizes 16 experts within their model, each is about ~111B parameters for MLP. 2 of these experts are routed to per forward pass.

Related Article: https://lemmy.intai.tech/post/72922

you are viewing a single comment's thread
view the rest of the comments

[–] [email protected] 9 points 2 years ago

The interesting part to me:

The missing dataset it a custom dataset of college textbooks collected by hand for as much courses as possible.

This is very easy to convert to txt file and than with self-instruct into instruction form. This creates the "illusion" that GPT-4 "is smart" no matter who use it.

Computer scientist? sure! it can help you with your questions about P!=NP Philosophy major? It can totally talk to you about epistemology.

Don't you see? It was trained on the textbooks. It is so obvious.

This could explain some (but not all) of the 'magic' I have seen with GPT4 vs GPT3.

If you put a bunch of textbooks into Google, it still couldn't help me build a video game engine