算法►深度学习►tensorflow

Tensorflow基础

tf证书主要考察四个方面的内容

(1) Build and train neural network models using TensorFlow 2.x

You need to understand the foundational principles of machine learning (ML) and deep learning (DL) using TensorFlow 2.x. You need to know how to:
Use TensorFlow 2.x.

  • Build, compile and train machine learning (ML) models using TensorFlow. 
1
2
3
4
5
6
7
8
9
10
11
12
13
# for demonstration
modl = tf.keras.models.Sequential([
    tf.keras.layers.Conv2D(filters=128, kernel_size=(3,3),strides=(1,1),
                          padding='same', activation='relu')
    tf.keras.layers.GlobalAveragePooling(),
    
    tf.keras.layers.Dense(nb_classes, activation='sigmod(if binary)/softmax(multiclass)')
])
model.compile(loss='binary_corssentropy/categorical_crossentropy',
             optimizer='rmpsprop',
             metrics=['acc'])
model.fit(data)
model.fit(xs, ys)
  1. 除序列模型以外，有其他较典型的模型吗？

  • Preprocess data to get it ready for use in a model.

    1. 数据的维度
    2. 归一化
    3. 训练数据和测试数据的一致性

  • Use models to predict results. 

    1
    2
    model.predict(ds)
    model.predict(xs)

  • Build sequential models with multiple layers. 

1
2
3
4
5
6
7
8
9
10
11
12
# 1: add all layers at a time
model = tf.keras.models.Sequential([
    tf.keras.layers.Conv2D(filters=128, kernel_size=(3,3),strides=(1,1),
                          padding='same', activation='relu')
    tf.keras.layers.GlobalAveragePooling(),
    
    tf.keras.layers.Dense(nb_classes, activation='sigmod(if binary)/softmax(multiclass)')
])

# 2: 逐个增加序列模型的每一层
model = tf.keras.models.Sequential()
model.add(tf.keras.layers.GlobalAveragePooling())

  • Build and train models for binary classification.

  • Build and train models for multi-class categorization.

  • Plot loss and accuracy of a trained model. 

    1
    2
    3
    4
    5
    6
    7
    8
    hist = history.history # dict
    loss, acc = hist['loss'], hist['accuracy']

    epochs = list(range(len(loss)))

    plt.plot(epochs, loss, color='b')
    plt.plot(epochs, acc, color='r')
    plt.legend(['loss', 'accuracy'])

  • Identify strategies to prevent overfitting, including augmentation and dropout. 

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    # 防止过拟合策略：1. dropout 2. augmentation 3. batch normalization
    # 1. dropout
    tf.layers.Dropout(rate)
    # 2. augmentation
    from tensorflow.keras.preprocessing.image import ImageDataGenerator
    train_datagen = ImageGenerator.rescale(
        rescale = 1./255,
        rotation_range=20,
        width_shift_range=0.2,
        height_shift_range=0.2,
        horizontal_flip=True)
    train_datagen.flow_from_directory(
    	dir,
    	target_size=(150,150),
    	batch_size=10,
    	class_mode='binary')
    # 3 BN
    tf.keras.layers.BatchNormalization()

    1. Dropout: set inputs to zero at a rate of rate, then outputs are scaled 1/(1 - rate)，输入的和不变

      1
      2
      3
      4
      5
      6
      7
      tf.random.set_seed(0)
      dropout_layer = tf.keras.layers.Dropout(.2, input_shape=(2,))
      data = np.arange(10).reshape(5, 2).astype(np.float32)
      print(data)

      outputs = dropout_layer(data, training=True)
      print(outputs)

      只在训练时生效. layer(training=True)

    2. BatchNormalization 

    1
    2
    3
    4
    # layter.trainable = False vs inference mode?
    # 前者的意思是内部的状态不改变；后者的意思是网络运行在推理阶段。
    # BN层在训练时，trainable=False 的表示mean和var采用当前batch的mean和var
    # inference则表明采用BN层自己的 滑动平均和滑动方差

  • Use pretrained models (transfer learning). 

    1
    2
    3
    4
    5
    6
    # load inception_v3
    from tensorflow.keras.applications.inception_v3 import InceptionV3

    pre_traine_model = InceptionV3(=(150, 150),
                                  include_top= False, # include last dense layer?
                                  weights=None) # where weights

  • Extract features from pre-trained models. 

    1
    2
    3
    #for layer in pre_trained_model.layers:
    #    layer.trainable = False
    extracted_features = model(inputs)

  • Ensure that inputs to a model are in the correct shape.

    • 模型的第一层传入input_shape参数

  • Ensure that you can match test data to the input shape of a neural network. 

    1
    2
    3
    4
    5
    6
    7
    8
    9

    * Ensure you can match output data of a neural network to specified input shape for test data. 

    * Understand batch loading of data. 

      ```python
      model.fit(xs, ys, batch_size=32)
      
      modelf.fit(Dataset)

  • Use callbacks to trigger the end of training cycles. 

    1
    2
    3
    4
    5
    6
    class MyCallbak(tf.keras.callbacks.Callback):
        def on_epoch_end(self, epoch, logs={}):
            if logs.get('acc') > DESIRED_ACCURACY:
                print("\n Reached desired accuracy")
                self.model.stop_training = True
    model.fit(callbacks = [MyCallback()])
  • Use datasets from different sources.

  • Use datasets in different formats, including json and csv. 

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    import csv, json, zipfile

    with open('csv.csv', mode='r') as f:
        reader = csv.reader(f, delimiter=',')
        for row in reader:
            # process row

    with open('json.json', mode='r') as f:
        datastore = json.load(f)
        
    zip_ref = zipfile.ZipFile('zip.zip', 'r')
    zip_ref.extractall("tmp/training")
    zip_ref.close()

  • Use datasets from tf.data.datasets. 

    1
    model.fit(dataset)

(2) Image classification

You need to understand how to build image recognition and object detection models with deep neural networks and convolutional neural networks using TensorFlow 2.x. You need to know how to:

  • Define Convolutional neural networks with Conv2D and pooling layers. 

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    model = tf.keras.models.Sequential([
        tf.keras.layers.Conv2D(16, (3,3), activation='relu', input_shape=(150,150,3)),
        tf.keras.layers.MaxPooling2D(2,2), # 保留特征性东西而不关注位置；降维；
        tf.keras.layers.Conv2D(32, (3, 3), activation='relu'),
        tf.keras.layers.MaxPooling2D(2,2),
        tf.keras.layers.Conv2D(64, (3, 3), activation='relu'),
        tf.keras.layers.MaxPooling2D(2,2),
        tf.keras.layers.Flatten(),
        tf.keras.layers.Dense(512, activation='relu'),
        tf.keras.layers.Dense(1, activation='sigmoid')
    ])

    model.compile(optimizer=RMSprop(lr=0.001), loss='binary_crossentropy', metrics=['acc'])

  • Build and train models to process real-world image datasets.

  • Understand how to use convolutions to improve your neural network.

    • problem: 过拟合、欠拟合；数据情况；
    • 查看数据关系情况：时序、图像、音频

  • Use real-world images in different shapes and sizes.. 

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    tf.keras.preprocess.image_dataset_from_directory(
        data_dir, validation_slit=0.2, subset="validation", seed=1123, image_size=(h, w), batch_size=batch_size
    )
    # 从一个包含图片的文件夹创建一个dataset
    #main_directory/
    #...class_a/
    #......a_image_1.jpg
    #......a_image_2.jpg
    #...class_b/
    #......b_image_1.jpg
    #......b_image_2.jpg

  • Use image augmentation to prevent overfitting.

    旋转、翻转、缩放、高度/宽度偏移、伸缩(rescale)

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18

    # layers.experimental.preprocessing.
    data_augmentatoin = keras.Sequential([
        layers.experimental.preprocessing.RandomFlip("horizontal", input_shape=(h, w, 3)),
        layers.experimental.preprocessing.RandomRotation(0.1),
        layers.experimental.preprocessing..RandomZoom(0.1),
    ])
    # 1. 可以作为网络层加入模型中 2. 可以不加入网络


    # ImageDataGenerator
    from tensorflow.keras.preprocessing.image import ImageDataGenerator
    train_datagen = ImageGenerator(
        rescale = 1./255,
        rotation_range=20,
        width_shift_range=0.2,
        height_shift_range=0.2,
        horizontal_flip=True) # 图像生成时所做的变换

  • Use ImageDataGenerator.

    生成图像数据的批同时支持实时的数据扩增

  • Understand how ImageDataGenerator labels images based on the directory structure.

    • ImageDataGenerator .flow_from_directory

      data/train/dogs/xxx.jpg and data/validation/dogs/xxx.jpg

      1
      2
      tf.keras.preprocessing.image.ImageGenerator(). \
      	flow_from_directory(dir, target_size=(32, 32), color_mode='rgb',class_mode='binary')

(3) Natural language processing (NLP)

You need to understand how to use neural networks to solve natural language processing problems using TensorFlow. You need to know how to:

  • Build natural language processing systems using TensorFlow.

  • Prepare text to use in TensorFlow models.

    • 分词、文本变数字编码序列
      • 标准化样本：小写、移除标点符号
      • 分割每个样本为substrings(words)
      • 重新组合为token （常间的ngrams）
      • token -> 索引（整数）
      • 利用上式，将每个样本转换为索引序列。（可能为int的序列，或者为向量的序列）
    • 数字序列可以用作 词嵌入层 的输入

  • Build models that identify the category of a piece of text using binary categorization 

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    model = tf.kears.models.Sequential([
        tf.keras.layers.word_embedding(
        	voc_size, embed_size),
        tf.keras.layers.LSTM(64),
        tf.keras.layers.Dense(1, activation='sigmoid')
    ])

    model.compile(loss='binary_crossentropy',
                 optimizer='rmsprop',
                 metrics=['acc'])

  • Build models that identify the category of a piece of text using multi-class categorization 

    1
    2
    3
    4
    5
    6
    7
    8
    9
    model = tf.kears.models.Sequential([
        tf.keras.layers.word_embedding(
        	voc_size, embed_size),
        tf.keras.layers.LSTM(64),
        tf.keras.layers.Dense(nb_classes, activation='softmax')
    ])
    model.compile(loss='multiclass_crossentropy',
                 optimizer='rmsprop',
                 metrics=['acc'])

  • Use word embeddings in your TensorFlow model.

  • Use LSTMs in your model to classify text for either binary or multi-class categorization.

  • Add RNN and GRU layers to your model. 

    1
    2
    3
    4
    5
    6
    7
    model = tf.kears.models.Sequential([
        tf.keras.layers.word_embedding(
        	voc_size, embed_size),
        tf.keras.layers.RNN(64),
        tf.keras.layers.GRU(64),
        tf.keras.layers.Dense(1, activation='sigmoid')
    ])

  • Use RNNS, LSTMs, GRUs and CNNs in models that work with text.

  • Train LSTMs on existing text to generate text (such as songs and poetry) 

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    # given a sequences of text, generate following 100 word

    # 将过去的所有文字输入模型预测下一个word # tokenizer: 分词器
    seqs = []
    tokenizer.texts_to_sequences(texts)

    def generate_words(text, nb_words):
        seq = tokenizer.texts_to_sequences(text)
        for _ in range(nb_words):
            word_idx = model.predict_class(seq)
            seq += word_idx,
        # convert seq to word
       	words = []
        for s in seq:
        	word = ''
            for i,w in tokenizer.word_index.items():
                if i == s:
                    word = w
                    break
             words += word,
        return words

(4) Time series, sequences and predictions

You need to understand how to solve time series and forecasting problems in TensorFlow. You need to know how to:

  • Train, tune and use time series, sequence and prediction models. 

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    # 时序数据一维度卷积时候padding要注意，使用'casual'的方式在seq的前部进行padding；以免对last part的信号产生干扰!
    model = tf.keras.models.Sequential([
      tf.keras.layers.Conv1D(filters=64, kernel_size=3, strides=1,  
                             padding='causal',
                             activation='relu',
                             input_shape=(None,1)),
      tf.keras.layers.LSTM(64, return_sequences=True),
      tf.keras.layers.Dense(1, activation='sigmoid'),
      tf.keras.layers.Lambda(lambda x: x * 100)
    ])

  • Prepare data for time series learning. 

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    # sequens to training data

    series = [1, 2, 3] # very long sequences
    window_size = 30 # 从时序数据中根据过去的30个点预测接下来的一个点

    ds = tf.data.Dataset.from_tensor_slices(series)
    ds = ds.window(size=window_size+1, shift=1, drop_remainder=True)

    ds = ds.flat_map(lambda x: x.batch(window_size+1) )
    ds = ds.map(lambda x: (x[:-1], x[-1]))

    ds.shuffle(10).map(lambda x, y: (
    	tf.expand_dims(x, axis=-1), tf.expand_dims(y, axis=-1))
                      ).batch(128).prefetch(1)
    # ds yields dataset of batch size 128 each iteration

  • Understand Mean Average Error (MAE) and how it can be used to evaluate accuracy of sequence models.

    err = abs(true - pred)

    评估预测的值与真实数据的出入(deviation)

  • Use RNNs and CNNs for time series, sequence and forecasting models.

    在温度时序预测中，数据的size为 (B, time_steps, 1)

    1
    2
    3
    4
    # CNNs
    tf.keras.layers.Conv1D(filter, ks, strides, padding)

    tf.keras.layers.RNN(units=64) # LSTM/GRU  or with Bidirectional

  • Identify when to use trailing versus centred windows.

    trailing versus centred windows

  • Use TensorFlow for forecasting.

  • Prepare features and labels.

  • Identify and compensate for sequence bias.

    subsequent bias? (bidirectional_layer?)

    Sequence bias is when the order of things can impact the selection of things. For example, if I were to ask you your favorite TV show, and listed “Game of Thrones”, “Killing Eve”, “Travelers” and “Doctor Who” in that order, you’re probably more likely to select ‘Game of Thrones’ as you are familiar with it, and it’s the first thing you see. 

    1
    Dataset.shuffle(buffer_size=10)

  • Adjust the learning rate dynamically in time series, sequence and prediction models.

    如何动态调整learning rate? callbacks

    callback如何动态修正 learning rate?

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    tf.keras.callbacks.LearningRateScheduler(
        schedule, verbose=0
    ) # schedule: 可接收(epoch, lr)两个参数的函数

    lr_schedule = tf.keras.callbacks.LearningRateScheduler(
        lambda epoch: 1e-8 * 10**(epoch / 20)) # callbacks

    # 自定义回调，当损失低于阈值时结束训练
    # https://www.tensorflow.org/guide/keras/custom_callback
    class CusCallback(tf.keras.callbacks.Callback):
        def __init__(self, patience=0):
            self.patience, self.best_weights = patience, None
       	def on_train_begin(self, logs=None):
            # the number of epoch it has watied when loss is no longer minimum
            self.wait, self.stopped_epoch, self.best = 0, 0, np.Inf
        
        def on_epoch_end(self, epoch, logs=None):
            current = logs.get("loss")
            # 如果当前loss低于最好loss，则update最好loss
            # 如果当前loss高于最好loss且等待时间(wait)超限，则结束训练
            if np.less(current, self.best):
                self.best = current
                self.wait = 0
                self.best_weights = self.model.get_weights()
            else:
                self.wait += 1
                if self.wait >= self.patience:
                    self.stopped_epoch = epoch
                    self.model.stop_training = True
                    print("Restoring model weights from the end of the best epoch")
                    self.model.set_weights(self.best_weights)

notes

  • 如何寻找合适的learn_rate规模？答：从小到大调整learning_rate以寻找合适的规模。
  1. 将course reviewed一遍，以熟悉内容
  2. 熟悉pycharm，在里面写东西
  3. 将instruction.pdf里面写image classificatio & word embedding的内容

组织结构，管理模式?

各种loss之总结： (y_true, y_pred), y_true is of shape (B, num_class)

SparseCategoricalCrossentropy：y_true的形状为 (B, 1)，每个维度为一个整数代表具体哪个类别；y_pred为(B, num_class)

BinaryCrossentropy

  • TextVectorization：文本向量化， keras模型层，支持将一批文本转为一批索引的列表亦或是一批1d array。

    调用adapt()方法，以从一个数据集中fit到单词

  • 序列预测中，如何将series序列数据转换为模型训练可以使用的tf.data.Dataset

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    def window_dataset(series, window_size, batch_size, shuffle_buffer):
        ds = tf.data.Dataset.from_tensor_slices(series)
        # 将输入的dataset组合为一个windows的dataset
        ds = ds.window(window_size, shift=1, drop_remainder=True)
        
        # map map_func across this dataset and flattens the result
        ds = ds.flat_map(lambda w: w.batch(window_size))
        
        ds = ds.shuffle(shuffle_buffer)
        ds = ds.map(lambda w: (w[:-1], w[1:]))
        return ds.batch(batch_size).prefetch(1)   

    ds.batch(batch_size)
    # 将dataset中连续的元素组合为一个tf.data.Dataset
    dataset = tf.data.Dataset.range(8)
    dataset = dataset.batch(3)
    list(dataset.as_numpy_iterator())

  • ImageGenerator

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    # ImageDataGenerator
    from tensorflow.keras.preprocessing.image import ImageDataGenerator
    train_datagen = ImageGenerator(
        rescale = 1./255,
        rotation_range=20,
        width_shift_range=0.2,
        height_shift_range=0.2,
        horizontal_flip=True) # 图像生成时所做的变换	

    train_datagen.flow_from_directory(
    	dir,
    	target_size=(150,150),
    	batch_size=10,
    	class_mode='binary')

    # Takes data & label arrays, generates batches of augmented data.
    train_datagen.flow(xs, ys=None, batch_size=10)

  • Tokenizer() 分词器怎么使用

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    from tensorflow.keras.preprocessing.text import Tokenizer
    from tensorflow.keras.preprocessing.sequence import pad_sequences

    tokenizer = Tokenizer(oov_token='<OOV>', 
                          num_words=1000,
                         lower=False,#是否将string转为小写
                         char_level=False,#是否将每个char视为一个token
                         )
    tokenizer.fit_on_texts(sentences)
    word_index = tokenizer.word_index
    print(len(word_index))

    sequences = tokenizer.texts_to_sequences(sentences)
    padded = pad_sequences(sequences, padding='post')

  • pad_sequences

    1
    2
    3
    4
    5
    6
    # This function transforms a list (of length num_samples) of sequences (lists of integers) into a 2D Numpy array of shape (num_samples, num_timesteps). num_timesteps is either the maxlen argument if provided, or the length of the longest sequence in the list.

    pad_sequences(
        sequences, maxlen=None, dtype='int32', padding='pre',
        truncating='pre', value=0.0
    )