AI Layer Repo

After setting up the environment ,download test AI Layer repository to work with different nodes.

Clone dev version of AI Layer of SoraEngine

git clone https://github.com/SoraChain-AI/testnet-AILayer-Node.git
cd testnet-AILayer-Node/
git checkout prepareProvision

Set PythonPath Env Variable

export PYTHONPATH=${PYTHONPATH}:${PWD}

Install required Python Packages

pip install --no-cache-dir -r requirements.txt   

For testing the setup we are using a micro LLM nano-mistral for text completion and training it using SoraEngine framework on huggingface dataset.

# import dataset and preprocess it.
cd data
git lfs install
git clone https://huggingface.co/datasets/prithivMLmods/Content-Articles
# This step can ask for huggingface authentication.
# authenticate using HF ID and password

cd ..

# preprocess data for training
python ./src/preprocess_nanoArticles.py --output_dir ./data/Output