Mixture of Low-Rank Adaptation Pairs of Large Language Models
by Liheng Chen
Supervised by Prof. Chuan Wu
Project Background
Large Language Models
Large Language Models (LLMs) have exhibited exceptional capabilities across various natural language processing tasks. Pre-trained on vast datasets, these models can be fine-tuned for specific tasks with remarkable efficiency, achieving state-of-the-art performances. Their deployment spans sectors including healthcare, finance, education, etc.
Parameter-Efficient Fine-Tuning
Parameter-Efficient Fine-Tuning (PEFT) methods, like P-tuning (Liu et al., 2022), Prefix-tuning (Li & Liang, 2021), and Low-Rank Adaptation (LoRA) (Hu et al., 2021), are designed to minimize the number of trainable parameters needed for model customization. These methods retain most of the model’s parameters fixed and only introduce lightweight adaptations, thus significantly reducing the memory and computation required for fine-tuning.
Project Introduction
We aim at producing a more cost-effective method to customize Large Languege Models (LLMs)
Mixture of LoRA Pairs (MoP)
- A unified approach to optimize parameter usage during model training for downstream tasks.
Project Progress
Please stay tuned for updates!