| | --- |
| | license: mit |
| | tags: |
| | - rna-seq |
| | - bulk-rna |
| | - cancer |
| | - transcriptomics |
| | - graph-neural-network |
| | - transformer |
| | - performer |
| | - gcn |
| | - pytorch |
| | model_size: 48M |
| | pipeline_tag: feature-extraction |
| | library_name: pytorch |
| | --- |
| | |
| | # 🧬 CancerTranscriptome-Mini-48M |
| | *A compact, cancer-focused BulkFormer-style encoder for bulk RNA-seq* |
| |
|
| | **CancerTranscriptome-Mini-48M** is a lightweight derivative of **BulkFormer**, designed to learn cancer-specific transcriptomic structure from large-scale bulk RNA-seq. |
| | It combines **GCN-based gene graph propagation**, **Rotary Expression Embeddings (REE)**, **local bin-wise Performer attention**, and **global Performer attention** into a single unified encoder. |
| |
|
| | This model is a proof-of-concept designed for research, experimentation, and rapid iteration on BulkFormer-style architectures applied to cancer transcriptomes. |
| |
|
| | --- |
| |
|
| | ## 🔬 Origin & References |
| |
|
| | ### **Primary Reference (BulkFormer)** |
| | Boming Kang, Rui Fan, Meizheng Yi, Chunmei Cui, Qinghua Cui. |
| | **“A large-scale foundation model for bulk transcriptomes.”** |
| | bioRxiv (2025). |
| | doi: https://doi.org/10.1101/2025.06.11.659222 |
| |
|
| | ### **This Model (CancerTranscriptome-Mini-48M)** |
| | A compact re-implementation based on the BulkFormer architecture, adapted for cancer-only bulk RNA-seq and simplified for accessibility and compute efficiency. |
| | Source Code: https://github.com/alwalt/BioFM |
| |
|
| | --- |
| |
|
| | # 📊 Data Source |
| |
|
| | All training samples originate from the **ARCHS4 Human RNA-seq v2.5** public repository: |
| |
|
| | **ARCHS4 Reference:** |
| | Lachmann A., Torre D., Keenan A.B., Jagodnik K.M., et al. |
| | **“Massive mining of publicly available RNA-seq data from human and mouse.”** |
| | *Nature Communications* 9, 1366 (2018). |
| | Dataset: https://maayanlab.cloud/archs4/ |
| |
|
| | ### **Filtering Procedure** |
| | - Loaded all human bulk RNA-seq metadata from ARCHS4 v2.5 HDF5 |
| | - Selected samples matching: |
| | `cancer | tumor | carcinoma | leukemia | lymphoma | melanoma | glioma` |
| | - Removed samples lacking clear disease annotations |
| | - Used ARCHS4 log-TPM matrices (gene × sample) |
| | - Final dataset: ~76k cancer samples, 19,357 genes |
| |
|
| | No private, clinical, controlled-access, or proprietary data were used. |
| |
|
| | --- |
| |
|
| | # 🧠 Model Architecture (Summary) |
| |
|
| | CancerTranscriptome-Mini-48M includes: |
| |
|
| | ### **1. Gene Identity Embeddings** |
| | - Precomputed **ESM2 embeddings** for each protein-coding gene |
| | - Projected into model dimension (320) |
| |
|
| | ### **2. Rotary Expression Embeddings (REE)** |
| | - Deterministic sinusoidal continuous-value embedding |
| | - Masked positions zeroed (mask token = –10) |
| |
|
| | ### **3. Graph Neural Network Layer** |
| | - **GCNConv** (Kipf & Welling) applied on a curated gene-gene graph |
| | - Injects biological prior knowledge |
| |
|
| | ### **4. Expression Binning** |
| | - Learnable importance scores sort genes |
| | - Genes divided into 10 bins |
| | - Each bin receives its own **local Performer** attention |
| |
|
| | ### **5. Global Performer Attention** |
| | - 2 stacked Performer layers across all genes |
| |
|
| | ### **6. Prediction Head** |
| | - MLP → scalar value per gene |
| | - Used for masked-expression reconstruction |
| |
|
| | Total parameters: **48,336,162 (~48M)** |
| |
|
| | --- |
| |
|
| | # 🎯 Intended Use |
| |
|
| | This model produces **context-aware gene embeddings** for downstream cancer transcriptomic tasks: |
| |
|
| | - Tumor subtype prediction |
| | - Drug response modeling |
| | - Immune infiltration scoring |
| | - Survival / risk modeling |
| | - Gene expression imputation |
| | - Dimensionality reduction |
| | - Transfer learning to TCGA, CCLE, DepMap, GEO tumor datasets |
| |
|
| | --- |
| |
|
| | # 🚀 How to Use |
| |
|
| | Download & run: |
| |
|
| | ```python |
| | import torch |
| | from model import BulkFormer # from this repo |
| | import safetensors.torch as st |
| | |
| | # Load model + weights |
| | model = BulkFormer( |
| | dim=320, |
| | graph=torch.load("edge_index.pt"), # provide your graph |
| | gene_emb=torch.load("esm2_gene_emb.pt"), |
| | gene_length=19357, |
| | bin_head=8, |
| | full_head=4, |
| | bins=10, |
| | gb_repeat=1, |
| | p_repeat=2 |
| | ) |
| | |
| | state = st.load_file("model.safetensors") |
| | model.load_state_dict(state) |
| | model.eval() |
| | |
| | # Example input: 19,357-gene log-TPM vector |
| | x = torch.randn(1, 19357) |
| | |
| | with torch.no_grad(): |
| | out = model(x) |
| | |
| | print(out.shape) # [1, 19357] |
| | |
