Update README.md

README.md

@@ -3,197 +3,177 @@ tags:
 - embedding
 - minecraft
 - block2vec
+- transformerblock2vec
+- 3d
+- voxel
+license: mit
 ---
-# Model Card for Model ID
 
-<!-- Provide a quick summary of what the model is/does. -->
+# TransformerBlock2Vec
 
-This modelcard aims to be a base template for new models. It has been generated using [this raw template](https://github.com/huggingface/huggingface_hub/blob/main/src/huggingface_hub/templates/modelcard_template.md?plain=1).
+This model card provides an overview of the TransformerBlock2Vec model, a transformer-based embedding model designed to create a 144-dimensional embedding space for Minecraft build chunks (up to 16x8x16 blocks). It uses 3D Rotary Positional Embeddings (RoPE) and is trained to predict masked blocks in a sequence, enabling downstream tasks like build vs. terrain segmentation.
 
 ## Model Details
 
 ### Model Description
 
-<!-- Provide a longer summary of what this model is. -->
+TransformerBlock2Vec is a transformer-based model that maps 3D Minecraft build chunks (up to 16x8x16 blocks) into a 144-dimensional embedding space. It leverages a custom 3D RoPE implementation to encode block positions and is trained using a masked language modeling approach, predicting 30% of masked blocks in a sequence. It uses DeepSpeed and FlashAttention for efficient training on consumer hardware (e.g., RTX 4070).
 
+- **Model type:** Transformer-based embedding model
+- **License:** MIT
 
+### Model Sources \[optional\]
 
-- **Developed by:** [More Information Needed]
-- **Funded by [optional]:** [More Information Needed]
-- **Shared by [optional]:** [More Information Needed]
-- **Model type:** [More Information Needed]
-- **Language(s) (NLP):** [More Information Needed]
-- **License:** [More Information Needed]
-- **Finetuned from model [optional]:** [More Information Needed]
-
-### Model Sources [optional]
-
-<!-- Provide the basic links for the model. -->
-
-- **Repository:** [More Information Needed]
-- **Paper [optional]:** [More Information Needed]
-- **Demo [optional]:** [More Information Needed]
+- **Repository:** [https://github.com/Kingburrito777/TransformerBlock2Vec](https://github.com/Kingburrito777/TransformerBlock2Vec)
+- **Paper:** [TBA!]()
+- **Demo:** [TBA!]()
 
 ## Uses
 
-<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+The model works very well for distinguishing user-made builds from terrain in Minecraft worlds, achieving 95% accuracy on unseen data for build vs. terrain segmentation.
+The embedding space of 3D minecraft data will enable downstream tasks such as search and retrieval, generative AI, and context understanding (bots).
 
 ### Direct Use
 
-<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
-
-[More Information Needed]
+TransformerBlock2Vec can be used to generate 144-dimensional embeddings for Minecraft build chunks, enabling tasks such as clustering similar builds, visualizing build distributions using t-SNE, or classifying chunks as builds vs. terrain. It is particularly suited for extracting meaningful representations from Minecraft data.
 
-### Downstream Use [optional]
+### Downstream Use
 
-<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+The model supports downstream tasks like:
 
-[More Information Needed]
+- **Build vs. Terrain Segmentation:** Identifying user-made structures in raw Minecraft worlds with 95% accuracy.
+- **Schematic Search Engine:** Enabling nearest-neighbor searches for similar builds based on embedding similarity.
+- **Generative Model Pretraining:** Providing embeddings for text-to-voxel generative models.
+- **Duplicate Analysis:** Clustering builds shows very similar builds near one another in the embedding space, and thus gives an effective way to remove duplicates from the corpus.
 
 ### Out-of-Scope Use
 
-<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
-
-[More Information Needed]
+- Is not directly integrated into the Minecraft game in any way, this requires a realtime application or mod written in Java. 
+- Not suitable for predicting block properties (e.g., stair directions) without additional finetuning.
+- Not intended for non-Minecraft 3D data.
 
 ## Bias, Risks, and Limitations
 
-<!-- This section is meant to convey both technical and sociotechnical limitations. -->
-
-[More Information Needed]
+- **Bias:** The model is trained on a dataset of Minecraft schematics, which may overrepresent certain build styles or block types based on the scraped data sources.
+- **Risks:** Misclassification of terrain as builds (or vice versa) could lead to incorrect data extraction in downstream tasks.
+- **Limitations:**
+  - Limited to chunks of 16x8x16 or smaller.
+  - Excludes block properties (e.g., stair orientations) to reduce vocabulary size.
+  - Requires significant compute for training (5 days on an RTX 4070 with DeepSpeed).
+  - Performance may degrade on highly unique or novel builds not represented in the training data.
 
 ### Recommendations
 
-<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+Users should:
 
-Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+- Validate model outputs for specific use cases, especially with modded or atypical builds.
+- Consider finetuning for tasks requiring block property predictions.
+- Be aware of potential biases in the training data and augment the dataset if targeting underrepresented build styles.
 
 ## How to Get Started with the Model
 
-Use the code below to get started with the model.
-
-[More Information Needed]
+- **Repository:** [https://github.com/Kingburrito777/TransformerBlock2Vec](https://github.com/Kingburrito777/TransformerBlock2Vec)
 
 ## Training Details
 
 ### Training Data
 
-<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
-
-[More Information Needed]
+The model is trained on a PostgreSQL database of approximately 108 Billion tokens, the largest known dataset of its kind. Schematics are converted to litematica format, and augmented with metadata (e.g., block counts, dimensions). Chunks are extracted up to 16x8x16, with non-empty chunks used for training. Schematic files are preprocessed with a "terrain-removal" algorithm, to reduce redundancy of terrain-like builds in the training data. This results in a model more directly suitable for embedding builds, instead of over represented generated terrain. 
+Curating and refining data is paramount to the models sucess in downstream tasks, and is currently the most difficult part of this endeavor.
 
 ### Training Procedure
 
-<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
-
-#### Preprocessing [optional]
-
-[More Information Needed]
+#### Preprocessing
 
+- Schematics are loaded using an optimized litematica parser.
+- Chunks are augmented with random rotations (25% probability) and flips (25% probability per axis).
+- Sequences are padded with a PAD token (1097), separated by a SEP token (1096), and masked with a MASK token (1098) for 30% of tokens.
 
 #### Training Hyperparameters
 
-- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+- **Batch size:** 86 with 4 gradient accumulation steps (effective batch size of 328 with DeepSpeed configuration)
+- **Learning rate:** 2e-4 warmup, 1e-5 thereafter.
+- **Epochs:** 4 (took 5 days!)
+- **Optimizer:** DeepSpeed-managed AdamW
+- **Dropout:** 0.1
+- **Training regime:** Mixed precision (fp16) with DeepSpeed
 
-#### Speeds, Sizes, Times [optional]
+#### Sizes, Times
 
-<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
-
-[More Information Needed]
+- **Training time:** \~5 days on a single RTX 4070 with DeepSpeed and FlashAttention.
+- **Checkpoint size:** \~30MB
 
 ## Evaluation
 
-<!-- This section describes the evaluation protocols and provides the results. -->
-
 ### Testing Data, Factors & Metrics
 
 #### Testing Data
 
-<!-- This should link to a Dataset Card if possible. -->
-
-[More Information Needed]
+Evaluated on a held-out set of unseen Minecraft schematics and raw world data, including both user-made builds and terrain chunks.
 
 #### Factors
 
-<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
-
-[More Information Needed]
+- **Build type:** Various structures (e.g., houses, farms, castles).
+- **Chunk size:** Up to 16x8x16.
 
 #### Metrics
 
-<!-- These are the evaluation metrics being used, ideally with a description of why. -->
-
-[More Information Needed]
+- **Accuracy (segmentation):** 95%+ for build vs. terrain classification on unseen data.
+- **Accuracy:** 50-90% for masking task as trained on. (If provided chunks are not too far from training distribution)
+- **Loss:** Cross-entropy loss on masked block prediction.
 
 ### Results
 
-[More Information Needed]
+For the masking task it was trained on (unusual for real-world use-case), probability can range from 40-90% correctly predicted blocks given a masking rate <30%.
+The model achieves 95% accuracy in distinguishing builds from terrain when training a segmentation head, with clear separation in the 144-dimensional embedding space (visualized via t-SNE). It generalizes well to unseen builds but may struggle with highly unique or novel block distributions. See \[the Github repo\](https://github.com/Kingburrito777/TransformerBlock2Vec)
 
 #### Summary
 
+TransformerBlock2Vec provides a robust embedding space for Minecraft builds, enabling accurate segmentation and potential for generative tasks with further development.
 
+## Model Examination
 
-## Model Examination [optional]
-
-<!-- Relevant interpretability work for the model goes here -->
+t-SNE visualizations of the 144-dimensional embedding space show distinct clusters for different build types, with terrain chunks separable from user-made structures. The model captures 3D spatial relationships effectively due to the 3D RoPE implementation.
 
-[More Information Needed]
-
-## Environmental Impact
-
-<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
-
-
-- **Hardware Type:** [More Information Needed]
-- **Hours used:** [More Information Needed]
-- **Cloud Provider:** [More Information Needed]
-- **Compute Region:** [More Information Needed]
-- **Carbon Emitted:** [More Information Needed]
-
-## Technical Specifications [optional]
+## Technical Specifications
 
 ### Model Architecture and Objective
 
-[More Information Needed]
+- **Architecture:** Transformer encoder with 6 layers, 8 attention heads, 144-dimensional embeddings, and SwiGLU feed-forward networks. Uses 3D RoPE for positional encoding and FlashAttention for efficiency.
+- **Objective:** Masked block prediction (30% of tokens masked) to learn a 144-dimensional embedding space for Minecraft chunks.
 
-### Compute Infrastructure
+I will have a technical breakdown released with proper metrics and further details.
 
-[More Information Needed]
+### Compute Infrastructure
 
 #### Hardware
 
-[More Information Needed]
+- **Hardware Type:** NVIDIA RTX 4070
+- **Hours used:** \~120 hours (5 days) for training
 
 #### Software
 
-[More Information Needed]
-
-## Citation [optional]
+- PyTorch
+- DeepSpeed (for distributed training and mixed precision)
+- FlashAttention
+- Python 3.8+
+- PostgreSQL (for data storage)
 
-<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+## Citation
 
 **BibTeX:**
 
-[More Information Needed]
-
-**APA:**
-
-[More Information Needed]
-
-## Glossary [optional]
-
-<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
-
-[More Information Needed]
-
-## More Information [optional]
-
-[More Information Needed]
+```bibtex
+[tba]
+```
 
-## Model Card Authors [optional]
+## Glossary
 
-[More Information Needed]
+- **3D RoPE:** 3D Rotary Positional Embeddings, a positional encoding method for 3D voxel data.
+- **Litematica:** A Minecraft schematic file format for storing 3D build data.
+- **Chunk:** A 3D block region in Minecraft (up to 16x8x16 in this model).
+- **Embedding Space:** A 144-dimensional vector space representing Minecraft build chunks.
 
-## Model Card Contact
+## More Information
 
-[More Information Needed]
+NOT AN OFFICIAL MINECRAFT PRODUCT. NOT APPROVED BY OR ASSOCIATED WITH MOJANG OR MICROSOFT. The project is part of the CraftGPT initiative to build generative AI for Minecraft.