Fast Style Transfer with Spell Model

For the final project of Neural Aesthetics, I wanted to explore Neural Style Transfer and train my own model with Wayne Thiebaud’s drawing Style. His style is very distinctive to me (especially drawing with what I love to eat – desserts), so I could generate a style of his in the Style transfer model with Spell. 

ellacyt.github.io/NeuralAesthetics/PlayingWithStyleTransfer

Wayne Thiebaud’s Cake paintings

In order to generate style transfer, I used the below painting as a style reference. 

Wayne Thiebaud, Six Lollipops

1. Setup Spell

To prepare the setup, I have installed python 2.7 and pip in the terminal, and then I signed up Spell to run my model. Spell is a simple command line tool to quickly take code and data and run model experiments. They also support most of the development environment. 

pip install spell
$ spell
$ spell login

2. Prepare environment

I then clone the fast-style-transfer git repo from github.

$ git clone https://github.com/lengstrom/fast-style-transfer
$ cd fast-style-transfer

Create some folders and files, and placed the reference image into the images/style folder.

$ mkdir ckpt/
$ touch ckpt/.gitignore
$ mkdir images
$ mkdir images/style

Then add the changes and commit it to git.

$ git add images ckpt
$ git commit -m "Added required folders and images"

2. Download the datasets from Spell

It took me almost 1-1.5 hours to finish this run in my Mac 2013. The model’s dataset is very large, it takes time to save to Spell.

$ spell run --machine-type CPU ./setup.sh

3. Training with style.py

After I download the datasets, I started to train the model. I used V100 machine. This run took me around 2 hours.

spell run --mount runs/THE_RUN_NUMBER_OF_YOUR_SETUP_RUN/data:datasets \
            --machine-type V100 \
            --framework tensorflow \
            --apt ffmpeg \
            --pip moviepy \
  "python style.py \
  --checkpoint-dir ckpt \
  --style images/style/YOUR_STYLE_IMAGE_NAME.jpg \
  --style-weight 1.5e2 \
  --train-path datasets/train2014 \
  --vgg-path datasets/imagenet-vgg-verydeep-19.mat"

And it will create files in the ckpt folder: 

  • checkpoint
  • fns.ckpt.data-00000-of-00001
  • fns.ckpt.index
  • fns.ckpt.meta

I downloaded the resulting checkpoint files use the spell ls and spell cp commands

spell ls runs/RUN_NUMBER
spell ls runs/RUN_NUMBER/ckpt
spell cp runs/RUN_NUMBER/ckpt

4. Converting model to ml5js

So now I have the datasets, I used Reiinakano’s fast-style-transfer-deeplearn.js to convert the datasets into the model for ml5js. 

git clone https://github.com/reiinakano/fast-style-transfer-deeplearnjs.git
cd fast-style-transfer-deeplearnjs

Put the checkpoint files we downloaded from spell into the current directory. 

python scripts/dump_checkpoint_vars.py --output_dir=src/ckpts/YOUR_FOLDER_NAME --checkpoint_file=./FOLDER_NAME/fns.ckpt

python scripts/remove_optimizer_variables.py --output_dir=src/ckpts/FOLDER_NAME

It will create a new folder in src/ckpts with 49 items including a manifest.json file.

5. Run the model in ml5js

Copy the folder we got from step 4 and put it into /models. Change style = ml5.styleTransfer('models/MODEL_NAME', modelLoaded); to your model file path. Run the code

Source code:

https://github.com/yining1023/styleTransfer_spell 
https://learn.spell.run/transferring_style

Leave a Reply

Your email address will not be published. Required fields are marked *

Back to Top