Model Card for farid678/gpt2-python-tokenizer

A new byte-level BPE tokenizer trained from scratch for GPT-2, specialized for Python source code, using the code_search_net (Python subset) dataset.

Model Details

Model Description

This repository contains a custom tokenizer trained from the base GPT-2 tokenizer architecture, re-trained on the Python portion of the code_search_net dataset. The goal of this tokenizer is to better capture Python-specific syntax, keywords, identifiers, and code patterns (e.g. indentation, operators, common function/variable naming conventions) compared to the original GPT-2 tokenizer, which was trained primarily on natural language web text.

This tokenizer can be paired with a GPT-2 model (either the original pretrained weights with an extended/adapted embedding layer, or a model trained from scratch) for downstream tasks involving Python code, such as code completion, code summarization, or code generation.

Developed by: farid678
Funded by [optional]: [More Information Needed]
Shared by [optional]: farid678
Model type: Byte-level BPE tokenizer (GPT-2 architecture)
Language(s) (NLP): Python (programming language); tokenizer vocabulary derived from source code rather than natural language
License: [More Information Needed]
Finetuned from model: gpt2 (tokenizer re-trained from scratch on new data, using GPT-2's tokenizer architecture as the base)

Model Sources [optional]

Repository: https://huggingface.co/farid678
Paper [optional]: [More Information Needed]
Demo [optional]: [More Information Needed]

Uses

Direct Use

This tokenizer can be used directly to tokenize Python source code for input into a GPT-2-style language model. It is intended for use in code-related NLP pipelines such as tokenizing datasets before training/fine-tuning a language model on Python code.

Downstream Use [optional]

Intended to be paired with a GPT-2 (or GPT-2-style) causal language model for tasks such as:

Python code completion
Python code generation
Code summarization / docstring generation
Code-to-text or text-to-code tasks

Out-of-Scope Use

This tokenizer is optimized for Python code and is not expected to perform well on natural language text or other programming languages (e.g. Java, C++, JavaScript) since its vocabulary was derived specifically from Python source code in code_search_net. It should not be used as a general-purpose natural language tokenizer.

Bias, Risks, and Limitations

The tokenizer's vocabulary reflects patterns present in the code_search_net Python subset, which is sourced from public open-source GitHub repositories. As such, it may inherit biases present in that codebase (e.g. naming conventions, coding styles, or underrepresentation of certain coding domains).
Performance on code written in significantly different styles, older Python versions, or non-English identifiers/comments may be degraded.
This tokenizer alone does not generate code; it must be paired with a trained language model to be useful for downstream tasks.

Recommendations

Users (both direct and downstream) should be made aware of the risks, biases and limitations of the tokenizer. It is recommended to evaluate tokenization quality (e.g. compression rate, out-of-vocabulary handling) on your own target dataset before relying on it for production use.

How to Get Started with the Model

Use the code below to get started with the tokenizer.

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("farid678/gpt2-python-tokenizer")

code_sample = "def hello_world():\n    print('Hello, world!')"
tokens = tokenizer.tokenize(code_sample)
print(tokens)

encoded = tokenizer(code_sample)
print(encoded["input_ids"])

Training Details

Training Data

The tokenizer was trained on the Python subset of the code_search_net dataset:

from datasets import load_dataset

raw_dataset = load_dataset("code_search_net", "python")

code_search_net contains functions and methods collected from open-source GitHub repositories, along with their associated docstrings/comments. The Python configuration used here consists of Python source code specifically.

Training Procedure

A new byte-level BPE tokenizer was trained from scratch using the GPT-2 tokenizer architecture as a template (i.e. tokenizer.train_new_from_iterator from the 🤗 Tokenizers/Transformers library), using the raw code text from code_search_net (Python) as the training corpus.

Preprocessing [optional]

Python code and associated documentation strings from code_search_net were used as raw text input for tokenizer training. [More Information Needed] (exact preprocessing steps, e.g. whether docstrings/comments were included or code-only)

Training Hyperparameters

Training regime: [More Information Needed]
Vocabulary size: [More Information Needed]
Base tokenizer: gpt2 (byte-level BPE)

Speeds, Sizes, Times [optional]

[More Information Needed]

Evaluation

Testing Data, Factors & Metrics

Testing Data

[More Information Needed]

Factors

[More Information Needed]

Metrics

[More Information Needed] (e.g. average tokens per line of code, compression ratio vs. original GPT-2 tokenizer, out-of-vocabulary rate)

Results

[More Information Needed]

Summary

This tokenizer is expected to produce more efficient, code-aware tokenization for Python source code compared to the original GPT-2 tokenizer, though formal benchmark results have not yet been recorded.

Model Examination [optional]

[More Information Needed]

Environmental Impact

Carbon emissions can be estimated using the Machine Learning Impact calculator presented in Lacoste et al. (2019).

Hardware Type: [More Information Needed]
Hours used: [More Information Needed]
Cloud Provider: [More Information Needed]
Compute Region: [More Information Needed]
Carbon Emitted: [More Information Needed]

Technical Specifications [optional]

Model Architecture and Objective

Byte-level BPE tokenizer following the GPT-2 tokenizer architecture, retrained on a new corpus (Python code from code_search_net) rather than the original GPT-2 training data.

Compute Infrastructure

[More Information Needed]

Hardware

[More Information Needed]

Software

🤗 transformers
🤗 datasets
🤗 tokenizers

Citation [optional]

BibTeX:

@misc{husain2019codesearchnet,
  title={CodeSearchNet Challenge: Evaluating the State of Semantic Code Search},
  author={Husain, Hamel and Wu, Ho-Hsiang and Gazit, Tiferet and Allamanis, Miltiadis and Brockschmidt, Marc},
  year={2019},
  eprint={1909.09436},
  archivePrefix={arXiv}
}

APA:

Husain, H., Wu, H.-H., Gazit, T., Allamanis, M., & Brockschmidt, M. (2019). CodeSearchNet Challenge: Evaluating the State of Semantic Code Search. arXiv preprint arXiv:1909.09436.

Glossary [optional]

BPE (Byte-Pair Encoding): A subword tokenization algorithm that iteratively merges the most frequent pairs of bytes/characters to build a vocabulary.
train_new_from_iterator: A 🤗 Transformers method that allows retraining an existing tokenizer's vocabulary on a new corpus while keeping the same tokenization algorithm/architecture.