use caffe to train Lenet with CSV data
Asked Answered
K

1

2

Excuse me, I have a question on using caffe for hd data? I try to run an example on the Kaggle mnist csv data with the following steps

  1. use h5py to convert it to h5 data. (I use the caffe-example.py to convert)

  2. Then modify the lenet_train_test_prototxt and train it. I am quite at a loss of this step.

The only change I made here is

layer {
  name: "mnist"
  type: "HDF5Data"
  top: "data"
  top: "label"
  include {
    phase: TRAIN
  }
  transform_param {
    scale: 0.00390625
  }
  data_param {
    source: "data/mnist_train_h5.txt"
    batch_size: 64
  }
}

how to change the lenet_train_test_prototxt to suit the data? Or also there are some other files I need to change? The error log is

enF0724 18:21:11.052737 79373 hdf5_data_layer.cpp:76] Check failed: !this->layer_param_.has_transform_param() HDF5Data does not transform data.

> *** Check failure stack trace: ***
>     @     0x7fe8188bbdaa  (unknown)
>     @     0x7fe8188bbce4  (unknown)
>     @     0x7fe8188bb6e6  (unknown)
>     @     0x7fe8188be687  (unknown)
>     @     0x7fe818caec10  caffe::HDF5DataLayer<>::LayerSetUp()
>     @     0x7fe818c520a3  caffe::Net<>::Init()
>     @     0x7fe818c53e12  caffe::Net<>::Net()
>     @     0x7fe818c0ba20  caffe::Solver<>::InitTrainNet()
>     @     0x7fe818c0c9c3  caffe::Solver<>::Init()
>     @     0x7fe818c0cb96  caffe::Solver<>::Solver()
>     @           0x40c8f0  caffe::GetSolver<>()
>     @           0x406541  train()
>     @           0x404a81  main
>     @     0x7fe817dcdec5  (unknown)
>     @           0x40502d  (unknown)
>     @              (nil)  (unknown) Aborted (core dumped)ter code here
Kkt answered 24/7, 2015 at 18:34 Comment(0)
K
1

I assume you have one hdf5 data file 'data/mnist_train_h5.hd5'.

  1. As you can see from the error message you got, "HDF5Data" layer does not support data transformation. Specifically, you cannot scale the data by the layer.
    Thus, any transformations you wish to have, you must apply them yourself during the creation of 'data/mnist_train_h5.hd5'.

  2. "HDF5Data" layer does not accept data_param, but rather hdf5_data_param with a parameter source specifying a list of hd5 binary files. In your case you should prepare an extra text file 'data/mnist_train_h5.txt' with a single line:

data/mnist_train_h5.hd5

This text file will tell caffe to read 'data/mnist_train_h5.hd5'.

The resulting layer should look like:

layer {
  name: "mnist"
  type: "HDF5Data"
  top: "data"
  top: "label"
  hdf5_data_param {
    source: "data/mnist_train_h5.txt"
    batch_size: 64
  }
  include {
    phase: TRAIN
  }
}
Kali answered 26/7, 2015 at 6:22 Comment(3)
Thanks for your answer! I have one question regarding data transformation. Here is my code: [luoyetx.me/2015/04/little-caffe/]. Please see his first piece of code here (too long to paste here). I am not sure what additional data transformation I need to do besides that piece of code. Thanks a lot!Kkt
did you mean source: "mnist_train_h5.txt" instead of source: "data/mnist_train_h5.hd5" ?Halfhour
@Halfhour thanks for spotting this bug! - corrected it just now.Kali

© 2022 - 2024 — McMap. All rights reserved.