Frontier Pre-Training at Microsoft AI

2 comments

• adroitboss 5 hours ago
  Wait, I don't get it. Is the model not open? If not, why are they comparing it to Open Weight models instead of the state of the art?
  [-]
  • leothetechguy 5 hours ago
    Because SOTA Closed Base Models are not available to compare to.
• Bolwin 5 hours ago
  Very similar to Kimi K3 it seems, but fewer active params (80b vs 104b). I don't think they mentioned what form of attention they use