python模块 机器学习 深度学习 Python

tensorflow2.0加载CSV数据训练模型

BG42

Posted by Blue Geek on October 28, 2019

tensorflow2.0加载CSV数据训练模型

By 青衣极客 Blue Geek In 2019-10-28

在使用tensorflow训练自己的模型时，如果不是使用已经封装好的通用数据集，则需要自己编写载入数据的代码。我们当然可以使用python的方式载入数据，然后转换称tensorflow中的Tensor，只是这样不仅需要编写的代码较多，而且效率相对低下。由于全局解释锁的存在，普通的python程序只能发挥出单核的性能。而且，通常我们都是使用GPU来训练模型的，这是CPU的空闲度比较大。如果让程序合理安排模型训练和数据载入的时间，会让整个运行过程更高效，而tensorflow提供的数据载入接口可以完成这一点。因此在载入数据时，最好使用tensorflow提供的方式来实现。CSV数据是很多数据分析师常常接触到的数据，也是很多实际问题的数据存储形式。本文就来讨论一下在tensorflow2.0中加载CSV数据并训练模型的方式。

1. 加载数据

首先使用指令查看一下训练数据集文件中的数据格式。从中可以看出数据所包含的数据项，根据列名可以知道每个数据项的意义。可以使用同样的方式查看测试数据集。测试数据集的格式与训练数据集完全一样，符合预期。确认格式没有问题之后，需要了解数据集样本的数量。在Linux或者MacOS中可以使用wc -l指令查看文件的行数，每一行表示一个样本点。因此，这里演示的数据集中，训练集有628个样本，测试集有265个样本。

!cat ../../data/titanic/train.csv | head -n 5

survived,sex,age,n_siblings_spouses,parch,fare,class,deck,embark_town,alone
0,male,22.0,1,0,7.25,Third,unknown,Southampton,n
1,female,38.0,1,0,71.2833,First,C,Cherbourg,n
1,female,26.0,0,0,7.925,Third,unknown,Southampton,y
1,female,35.0,1,0,53.1,First,C,Southampton,n

!cat ../../data/titanic/eval.csv | head -n 5

survived,sex,age,n_siblings_spouses,parch,fare,class,deck,embark_town,alone
0,male,35.0,0,0,8.05,Third,unknown,Southampton,y
0,male,54.0,0,0,51.8625,First,E,Southampton,y
1,female,58.0,0,0,26.55,First,C,Southampton,y
1,female,55.0,0,0,16.0,Second,unknown,Southampton,y

!wc -l ../../data/titanic/train.csv
!wc -l ../../data/titanic/eval.csv

     628 ../../data/titanic/train.csv
     265 ../../data/titanic/eval.csv

以下使用tensorflow提供的tf.data.experimental.make_csv_dataset接口载入CSV文件的数据。这个接口提供的可设置参数众多，具体每个参数的含义可以查看官网说明：https://www.tensorflow.org/api_docs/python/tf/data/experimental/make_csv_dataset 。这里只使用其中几个简单的配置设置batch size，随机排列和标签所在的列名。

import tensorflow as tf
def load_dataset(csv_path, shuffle=True):
    return tf.data.experimental.make_csv_dataset(
        csv_path,
        batch_size=12,           # 设置batch size
        shuffle=shuffle,         # 设置随机排列
        label_name='survived',   # 设置标签所在的列名
        na_value='?',
        num_epochs=1,
        ignore_errors=True)
trainset = load_dataset(csv_path='../../data/titanic/train.csv')
testset = load_dataset(csv_path='../../data/titanic/eval.csv', shuffle=False)
feats, lbls = next(iter(trainset))
print('features =', feats)
print('labels =', lbls)

WARNING: Logging before flag parsing goes to stderr.
W1028 21:15:08.017898 4656805312 deprecation.py:323] From /usr/local/lib/python3.7/site-packages/tensorflow_core/python/data/experimental/ops/readers.py:521: parallel_interleave (from tensorflow.python.data.experimental.ops.interleave_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.data.Dataset.interleave(map_func, cycle_length, block_length, num_parallel_calls=tf.data.experimental.AUTOTUNE)` instead. If sloppy execution is desired, use `tf.data.Options.experimental_determinstic`.


features = OrderedDict([('sex', <tf.Tensor: id=164, shape=(12,), dtype=string, numpy=
array([b'male', b'female', b'male', b'male', b'female', b'male', b'male',
       b'male', b'female', b'male', b'female', b'male'], dtype=object)>), ('age', <tf.Tensor: id=156, shape=(12,), dtype=float32, numpy=
array([45.,  3., 17., 20.,  4., 22.,  4., 35., 28., 16., 28., 27.],
      dtype=float32)>), ('n_siblings_spouses', <tf.Tensor: id=162, shape=(12,), dtype=int32, numpy=array([0, 1, 1, 0, 1, 0, 4, 0, 0, 0, 0, 0], dtype=int32)>), ('parch', <tf.Tensor: id=163, shape=(12,), dtype=int32, numpy=array([0, 2, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0], dtype=int32)>), ('fare', <tf.Tensor: id=161, shape=(12,), dtype=float32, numpy=
array([ 8.05  , 41.5792,  7.2292,  7.2292, 23.    ,  7.25  , 29.125 ,
        7.05  ,  7.75  ,  8.05  ,  7.225 , 30.5   ], dtype=float32)>), ('class', <tf.Tensor: id=158, shape=(12,), dtype=string, numpy=
array([b'Third', b'Second', b'Third', b'Third', b'Second', b'Third',
       b'Third', b'Third', b'Third', b'Third', b'Third', b'First'],
      dtype=object)>), ('deck', <tf.Tensor: id=159, shape=(12,), dtype=string, numpy=
array([b'unknown', b'unknown', b'unknown', b'unknown', b'unknown',
       b'unknown', b'unknown', b'unknown', b'unknown', b'unknown',
       b'unknown', b'unknown'], dtype=object)>), ('embark_town', <tf.Tensor: id=160, shape=(12,), dtype=string, numpy=
array([b'Southampton', b'Cherbourg', b'Cherbourg', b'Cherbourg',
       b'Southampton', b'Southampton', b'Queenstown', b'Southampton',
       b'Queenstown', b'Southampton', b'Cherbourg', b'Southampton'],
      dtype=object)>), ('alone', <tf.Tensor: id=157, shape=(12,), dtype=string, numpy=
array([b'y', b'n', b'n', b'y', b'n', b'y', b'n', b'y', b'y', b'y', b'y',
       b'y'], dtype=object)>)])
labels = tf.Tensor([1 1 0 1 1 0 0 0 1 1 1 1], shape=(12,), dtype=int32)

2. 数据预处理

CSV所存储的数据中常常或包含一些文本类型，直接处理这些文本类型并不划算，因为这些文本实际上只是表示若干个离散的类别信息，使用整数表示更加高效一些。这时就需要对这些类别文本进行标记转换。tensorflow提供了tf.feature_column.categorical_column_with_vocabulary_list接口完成这个任务。这个接口也有几个配置参数，具体可以查看官网手册：https://www.tensorflow.org/api_docs/python/tf/feature_column/categorical_column_with_vocabulary_list 。这里的演示只使用了两个最基本的参数，指定需要进行预处理的特征列名，并给出每种特征的可能取值情况。在实际使用中，可能会出现数据不在所给取值情况中，这这时就需要一种机制对这些out-of-vocabulary的标签进行转换。还是使用这个函数，只不过需要设置一下参数，具体情况参见手册，这里就不作演示了。

CATE_INFO = {'sex': ['male', 'female'],
             'class': ['First', 'Second', 'Third'],
             'deck': ['A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'J'],
             'embark_town': ['Cherbourg', 'SouthHampton', 'Queenstown'],
             'alone': ['y', 'n']}
cate_columns = []
for feat, voc_list in CATE_INFO.items():
    cat_col = tf.feature_column.categorical_column_with_vocabulary_list(
                  key=feat, vocabulary_list=voc_list)
    cate_columns.append(tf.feature_column.indicator_column(cat_col))
print(cate_columns)

[IndicatorColumn(categorical_column=VocabularyListCategoricalColumn(key='sex', vocabulary_list=('male', 'female'), dtype=tf.string, default_value=-1, num_oov_buckets=0)), IndicatorColumn(categorical_column=VocabularyListCategoricalColumn(key='class', vocabulary_list=('First', 'Second', 'Third'), dtype=tf.string, default_value=-1, num_oov_buckets=0)), IndicatorColumn(categorical_column=VocabularyListCategoricalColumn(key='deck', vocabulary_list=('A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'J'), dtype=tf.string, default_value=-1, num_oov_buckets=0)), IndicatorColumn(categorical_column=VocabularyListCategoricalColumn(key='embark_town', vocabulary_list=('Cherbourg', 'SouthHampton', 'Queenstown'), dtype=tf.string, default_value=-1, num_oov_buckets=0)), IndicatorColumn(categorical_column=VocabularyListCategoricalColumn(key='alone', vocabulary_list=('y', 'n'), dtype=tf.string, default_value=-1, num_oov_buckets=0))]

对于离散的数据，常常需要进行归一化，以消除不同类型数据所具有的不同的数据范围在训练模型时产生的不良影响。tensorflow提供了tf.feature_column.numeric_column接口为一些连续特征的列绑定归一化的操作，使用functools.partial绑定自定义的归一化函数和参数。最终会生成一组针对所设置的特征进行的预处理操作。

import functools
def normalize(mean, data):
    return tf.reshape(tf.cast(data, tf.float32)/(2*mean), [-1,1])
CONT_INFO = {'age': 29.63,
             'n_siblings_spouses': 0.55,
             'parch': 0.38,
             'fare': 34.39}
cont_columns = []
for feat in CONT_INFO.keys():
    cont_col = tf.feature_column.numeric_column(
        feat, normalizer_fn=functools.partial(normalize, CONT_INFO[feat]))
    cont_columns.append(cont_col)
print(cont_columns)

[NumericColumn(key='age', shape=(1,), default_value=None, dtype=tf.float32, normalizer_fn=functools.partial(<function normalize at 0x140301440>, 29.63)), NumericColumn(key='n_siblings_spouses', shape=(1,), default_value=None, dtype=tf.float32, normalizer_fn=functools.partial(<function normalize at 0x140301440>, 0.55)), NumericColumn(key='parch', shape=(1,), default_value=None, dtype=tf.float32, normalizer_fn=functools.partial(<function normalize at 0x140301440>, 0.38)), NumericColumn(key='fare', shape=(1,), default_value=None, dtype=tf.float32, normalizer_fn=functools.partial(<function normalize at 0x140301440>, 34.39))]

3. 训练模型

数据载入和预处理操作准备完成后就可以搭建模型了。这里演示一个简单的分类模型。在模型的第一层，需要对离散特征预处理操作和连续值特征的预处理操作进行拼接，从而实现完整的特征输入。设置好损失函数和优化器之后就可以进行训练了。

from tensorflow.keras import Sequential
from tensorflow.keras.layers import DenseFeatures, Dense
model = Sequential([
    DenseFeatures(cate_columns+cont_columns),
    Dense(128, activation='relu'),
    Dense(128, activation='relu'),
    Dense(1, activation='sigmoid'),
])
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
model.fit(trainset, epochs=20)

Epoch 1/20
53/53 [==============================] - 1s 16ms/step - loss: 0.5247 - accuracy: 0.7703
Epoch 2/20
53/53 [==============================] - 0s 2ms/step - loss: 0.4396 - accuracy: 0.7974
Epoch 3/20
53/53 [==============================] - 0s 2ms/step - loss: 0.4169 - accuracy: 0.8118
Epoch 4/20
53/53 [==============================] - 0s 2ms/step - loss: 0.4119 - accuracy: 0.8246
Epoch 5/20
53/53 [==============================] - 0s 2ms/step - loss: 0.3969 - accuracy: 0.8214
Epoch 6/20
53/53 [==============================] - 0s 2ms/step - loss: 0.3951 - accuracy: 0.8293
Epoch 7/20
53/53 [==============================] - 0s 2ms/step - loss: 0.3827 - accuracy: 0.8357
Epoch 8/20
53/53 [==============================] - 0s 2ms/step - loss: 0.3856 - accuracy: 0.8373
Epoch 9/20
53/53 [==============================] - 0s 2ms/step - loss: 0.3754 - accuracy: 0.8389
Epoch 10/20
53/53 [==============================] - 0s 2ms/step - loss: 0.3694 - accuracy: 0.8389
Epoch 11/20
53/53 [==============================] - 0s 2ms/step - loss: 0.3629 - accuracy: 0.8405
Epoch 12/20
53/53 [==============================] - 0s 2ms/step - loss: 0.3655 - accuracy: 0.8357
Epoch 13/20
53/53 [==============================] - 0s 2ms/step - loss: 0.3627 - accuracy: 0.8405
Epoch 14/20
53/53 [==============================] - 0s 2ms/step - loss: 0.3604 - accuracy: 0.8453
Epoch 15/20
53/53 [==============================] - 0s 2ms/step - loss: 0.3556 - accuracy: 0.8501
Epoch 16/20
53/53 [==============================] - 0s 2ms/step - loss: 0.3525 - accuracy: 0.8405
Epoch 17/20
53/53 [==============================] - 0s 2ms/step - loss: 0.3469 - accuracy: 0.8533
Epoch 18/20
53/53 [==============================] - 0s 2ms/step - loss: 0.3551 - accuracy: 0.8453
Epoch 19/20
53/53 [==============================] - 0s 2ms/step - loss: 0.3413 - accuracy: 0.8437
Epoch 20/20
53/53 [==============================] - 0s 2ms/step - loss: 0.3349 - accuracy: 0.8676





<tensorflow.python.keras.callbacks.History at 0x141e6e910>

4. 评估和测试

模型训楼完成，就可以使用测试集对模型进行评估。在测试集上所得到的正确率与训练集上基本相近，模型表现符合预期。

loss_acc = model.evaluate(testset)
print('On Testset, loss={}, acc={}'.format(loss_acc[0], loss_acc[1]))

22/22 [==============================] - 0s 13ms/step - loss: 0.4524 - accuracy: 0.8182
On Testset, loss=0.45240371742031793, acc=0.8181818127632141

取测测试数据集中的一个batch进行预测，查看预测的概率与真实的标记之间的比较。从结果可以看出，大部分的预测概率与标记类型是一致的。

for feats, lbls in testset:
    break
preds = model.predict(feats)
for pred, gt in zip(preds[:, 0], lbls.numpy()):
    print('predicted survival prop: {:.2f}, groundtruth:{}'.format(pred, 'SURVIVED' if gt==1 else 'DIED'))

predicted survival prop: 0.11, groundtruth:DIED
predicted survival prop: 0.55, groundtruth:DIED
predicted survival prop: 0.83, groundtruth:SURVIVED
predicted survival prop: 0.71, groundtruth:SURVIVED
predicted survival prop: 0.02, groundtruth:SURVIVED
predicted survival prop: 0.93, groundtruth:SURVIVED
predicted survival prop: 0.24, groundtruth:DIED
predicted survival prop: 0.13, groundtruth:DIED
predicted survival prop: 0.50, groundtruth:DIED
predicted survival prop: 0.92, groundtruth:SURVIVED
predicted survival prop: 0.89, groundtruth:SURVIVED
predicted survival prop: 0.13, groundtruth:DIED

在tensorflow2.0中加载CSV数据并用于模型训练的实现方式演示完毕。本文是在学习tensorflow官网上的教程的基础上编写完成的，官网链接：https://www.tensorflow.org/tutorials/load_data/csv 。感兴趣的朋友可以前往阅读官网原文。

【青衣极客】公众号

COMMENT

博客评论区功能由Github Issue提供，提交Issue时请以本文标题为话题。

"BG42-tensorflow2.0加载CSV数据训练模型"

任何问题，请提交issue
CATALOG
    SEARCH BOX
      FEATURED TAGS
      python模块 Python 机器学习 算法 面试刷题 Leetcode python语法 深度学习 数据可视化 图像处理 Github OpenCV 人脸 实用技巧 数据分析 网络安全 读书 C++ 网络通信