For the final project of Neural Aesthetics, I wanted to explore Neural Style Transfer and train my own model with Wayne Thiebaud’s drawing Style. His style is very distinctive to me (especially drawing with what I love to eat – desserts), so I could generate a style of his in the Style transfer model with Spell.
In order to generate style transfer, I used the below painting as a style reference.
1. Setup Spell
To prepare the setup, I have installed python 2.7 and pip in the terminal, and then I signed up Spell to run my model. Spell is a simple command line tool to quickly take code and data and run model experiments. They also support most of the development environment.
pip install spell $ spell $ spell login
2. Prepare environment
I then clone the fast-style-transfer git repo from github.
$ git clone https://github.com/lengstrom/fast-style-transfer $ cd fast-style-transfer
Create some folders and files, and placed the reference image into the images/style folder.
$ mkdir ckpt/ $ touch ckpt/.gitignore $ mkdir images $ mkdir images/style
Then add the changes and commit it to git.
$ git add images ckpt $ git commit -m "Added required folders and images"
2. Download the datasets from Spell
It took me almost 1-1.5 hours to finish this run in my Mac 2013. The model’s dataset is very large, it takes time to save to Spell.
$ spell run --machine-type CPU ./setup.sh
3. Training with style.py
After I download the datasets, I started to train the model. I used V100 machine. This run took me around 2 hours.
spell run --mount runs/THE_RUN_NUMBER_OF_YOUR_SETUP_RUN/data:datasets \ --machine-type V100 \ --framework tensorflow \ --apt ffmpeg \ --pip moviepy \ "python style.py \ --checkpoint-dir ckpt \ --style images/style/YOUR_STYLE_IMAGE_NAME.jpg \ --style-weight 1.5e2 \ --train-path datasets/train2014 \ --vgg-path datasets/imagenet-vgg-verydeep-19.mat"
And it will create files in the
I downloaded the resulting checkpoint files use
spell ls runs/RUN_NUMBER spell ls runs/RUN_NUMBER/ckpt spell cp runs/RUN_NUMBER/ckpt
4. Converting model to ml5js
So now I have the datasets, I used Reiinakano’s fast-style-transfer-deeplearn.js to convert the datasets into the model for ml5js.
git clone https://github.com/reiinakano/fast-style-transfer-deeplearnjs.git cd fast-style-transfer-deeplearnjs
Put the checkpoint files we downloaded from spell into the current directory.
python scripts/dump_checkpoint_vars.py --output_dir=src/ckpts/YOUR_FOLDER_NAME --checkpoint_file=./FOLDER_NAME/fns.ckpt python scripts/remove_optimizer_variables.py --output_dir=src/ckpts/FOLDER_NAME
It will create a new folder in
src/ckpts with 49 items including a manifest.json file.
5. Run the model in ml5js
Copy the folder we got from step 4 and put it into /models.
style = ml5.styleTransfer('models/MODEL_NAME', modelLoaded);