11 items with this tag.
moonshots
Moonshot hypothesis that most vocabulary rows in the output head should be regenerated from compact descriptors and shared factors rather than stored directly.
papers
Paper note on shrinking and retargeting the tokenizer and embedding table to a domain so the model uses fewer vocabulary bytes and shorter sequences.
frontiers
Frontier synthesis on why tokenizer research for compact models should be treated as a joint vocabulary, logits, and artifact-budget problem rather than a token-count problem.
hypotheses
Hypothesis that compressing or restructuring the LM head can beat modest backbone improvements in compact language models.
ideas
Hypothesis that most head-side quantization damage is concentrated in a tiny set of difficult token rows, making row-level protection a better byte trade than uniform head precision.
ideas
Hypothesis that shrinking tokenizer and LM-head burden, then reinvesting the saved bytes into a wider shared backbone, beats spending the same budget on a larger static head.
lanes
Tokenization is part of the budget story, not just a preprocessing detail.
notes
Synthesis note on why vocabulary and output-projection choices can dominate compact-model tradeoffs earlier than expected.
notes
Concept note on why tokenization changes not just sequence length but the whole byte/compute story of compact language models.
papers
Paper note on replacing a pretrained model tokenizer while retraining only embeddings and the LM head.
papers
Paper note on tokenizer evaluation across scales and why compression alone is not enough to rank tokenizers.