LumenBase is a 128M-parameter transformer language model built from scratch using PyTorch, featuring a custom tokenizer and GQA-based architecture. It includes a complete training and evaluation pipeline, achieving competitive scores on ARC and HellaSwag reasoning benchmarks.
