Updated open-sci-ref baselines. Re-training without dropout. Re-training on DCLM, FineWeb-Edu, Nemotron, HPLT-2, Pile. Further ref datasets included.
AI & ML interests
Researching and building foundation models with improved generalization and reasoning. LAION & friends spin-off for open-sourcing foundation models with strong generalization and reasoning , including datasets necessary for their creation, to serve as common open, reproducible grounds for further research experiments.
Recent Activity
View all activity
Collection of models and dataset related to MixtureVitae, open and fully reproducible pretraining dataset built from permissive sources
-
open-sci/open-sci-ref-v0.01-0.13b-c4-300B-4096-warmup25000-lr0.006-2
0.1B • Updated • 5 -
open-sci/open-sci-ref-v0.01-0.4b-c4-300B-4096-warmup25000
0.4B • Updated • 3 -
open-sci/open-sci-ref-v0.01-0.4b-c4-300B-4096-warmup25000-lr0.004-2
0.4B • Updated • 1 -
open-sci/open-sci-ref-v0.01-0.13b-c4-300B-4096-warmup25000
0.1B • Updated • 2
-
open-sci/open-sci-ref-v0.01-1.7b-nemotron-hq-1T-4096-rope_theta-100k
2B • Updated • 1 -
open-sci/open-sci-ref-v0.01-0.13b-nemotron-hq-300B-4096
0.1B • Updated • 8 -
open-sci/open-sci-ref-v0.01-0.4b-nemotron-hq-300B-4096
0.4B • Updated • 1 -
open-sci/open-sci-ref-v0.01-1.3b-nemotron-hq-300B-4096
1B • Updated • 1
openMammut models trained on various datasets (Re-LAION, DataComp, DFN)
-
laion/openMaMMUT-ViT-L-14-DataComp-1.4B-s12.8B-b180K
Zero-Shot Image Classification • Updated • 6 • 5 -
laion/openMaMMUT-ViT-B-32-512x512-pt_DFN2B-ft_DFN512x512-s293M-b73k
Zero-Shot Image Classification • Updated • 15 • 2 -
laion/openMaMMUT-ViT-B-16-512x512-pt_DFN2B-ft_DFN512x512-s293M-b73k
Zero-Shot Image Classification • Updated • 11 • 2
Materials related to OpenThoughts and OpenThinker releases
openMaMMUT/openCLIP models trained on DataComp-1.4B, DFN-1.4B and Re-LAION-2B. Pre-trained models on various scales, incl. intermediate checkpoints
-
open-sci/open-sci-ref-v0.01-0.13b-fineweb-edu-1.4t-300B-4096
0.1B • Updated • 8 -
open-sci/open-sci-ref-v0.01-0.4b-fineweb-edu-1.4t-300B-4096
0.4B • Updated • 5 -
open-sci/open-sci-ref-v0.01-1.3b-fineweb-edu-1.4t-300B-4096
1B • Updated • 1 -
open-sci/open-sci-ref-v0.01-1.7b-fineweb-edu-1.4t-1T-4096
2B • Updated • 1
Research baseline models trained on various open reference datasets
Open-sci-ref: reference baselines releases
Updated open-sci-ref baselines. Re-training without dropout. Re-training on DCLM, FineWeb-Edu, Nemotron, HPLT-2, Pile. Further ref datasets included.
openMaMMUT/openCLIP models trained on DataComp-1.4B, DFN-1.4B and Re-LAION-2B. Pre-trained models on various scales, incl. intermediate checkpoints
Collection of models and dataset related to MixtureVitae, open and fully reproducible pretraining dataset built from permissive sources
-
open-sci/open-sci-ref-v0.01-0.13b-fineweb-edu-1.4t-300B-4096
0.1B • Updated • 8 -
open-sci/open-sci-ref-v0.01-0.4b-fineweb-edu-1.4t-300B-4096
0.4B • Updated • 5 -
open-sci/open-sci-ref-v0.01-1.3b-fineweb-edu-1.4t-300B-4096
1B • Updated • 1 -
open-sci/open-sci-ref-v0.01-1.7b-fineweb-edu-1.4t-1T-4096
2B • Updated • 1
-
open-sci/open-sci-ref-v0.01-0.13b-c4-300B-4096-warmup25000-lr0.006-2
0.1B • Updated • 5 -
open-sci/open-sci-ref-v0.01-0.4b-c4-300B-4096-warmup25000
0.4B • Updated • 3 -
open-sci/open-sci-ref-v0.01-0.4b-c4-300B-4096-warmup25000-lr0.004-2
0.4B • Updated • 1 -
open-sci/open-sci-ref-v0.01-0.13b-c4-300B-4096-warmup25000
0.1B • Updated • 2
-
open-sci/open-sci-ref-v0.01-1.7b-nemotron-hq-1T-4096-rope_theta-100k
2B • Updated • 1 -
open-sci/open-sci-ref-v0.01-0.13b-nemotron-hq-300B-4096
0.1B • Updated • 8 -
open-sci/open-sci-ref-v0.01-0.4b-nemotron-hq-300B-4096
0.4B • Updated • 1 -
open-sci/open-sci-ref-v0.01-1.3b-nemotron-hq-300B-4096
1B • Updated • 1
Research baseline models trained on various open reference datasets
openMammut models trained on various datasets (Re-LAION, DataComp, DFN)
-
laion/openMaMMUT-ViT-L-14-DataComp-1.4B-s12.8B-b180K
Zero-Shot Image Classification • Updated • 6 • 5 -
laion/openMaMMUT-ViT-B-32-512x512-pt_DFN2B-ft_DFN512x512-s293M-b73k
Zero-Shot Image Classification • Updated • 15 • 2 -
laion/openMaMMUT-ViT-B-16-512x512-pt_DFN2B-ft_DFN512x512-s293M-b73k
Zero-Shot Image Classification • Updated • 11 • 2
Open-sci-ref: reference baselines releases
Materials related to OpenThoughts and OpenThinker releases