Sequence annotation, handwritten lowercase letters OCR data set, bidirectional RNN

Sequence labeling, predicting a category for each frame of the input sequence. OCR (Optical Character Recognition optical character recognition).

The MIT Spoken Language System Research Group Rob Kassel collected and the Stanford University Artificial Intelligence Laboratory Ben Taskar preprocessed OCR data set (http://ai.stanford.edu/~btaskar/ocr/), which contains a large number of individual handwriting Lowercase letters, each sample corresponds to a 16X8 pixel binary image. Word lines combine sequences, and the sequences correspond to words. 6800 words of no more than 14 letters in length. gzip compressed, content is tab-delimited text file. Python csv module reads directly. Each line of the file has one normalized letter attribute, ID number, label, pixel value, next letter ID number, etc.

Next letter ID value sort, read each word letter in the correct order. Letters are collected until the next ID corresponding field is not set. Read new sequence. After reading the target letters and data pixels, fill the sequence object with zero images, which can be included in the NumPy array of all pixel data of the two larger target letters.

Share the softmax layer between time steps. The data and target arrays contain sequences, one image frame for each target letter. RNN extension, adding softmax classifier to each letter output. The classifier evaluates predictions on each frame of data rather than on the entire sequence. Calculate sequence length. A softmax layer is added to all frames: either several different classifiers are added to all frames, or all frames share the same classifier. With a shared classifier, the weights are adjusted more times during training, for each letter of the training word. A fully connected layer weight matrix dimension batch_size*in_size*out_size. Now it is necessary to update the weight matrix in the two input dimensions batch_size and sequence_steps. Let the input (RNN output activity value) be flattened into the shape batch_size*sequence_steps*in_size. The weight matrix becomes a larger batch of data. The result is unflattened.

Cost function, each frame of the sequence has a predicted target pair, averaged in the corresponding dimension. tf.reduce_mean normalized according to tensor length (maximum length of sequence) cannot be used. It is necessary to normalize according to the actual sequence length and manually call tf.reduce_sum and division operation mean.

Loss function, tf.argmax is for axis 2 and not axis 1, each frame is filled, and the mean is calculated based on the actual length of the sequence. tf.reduce_mean takes the mean of all words in the batch data.

TensorFlow automatic derivative calculation can use the same optimization operation for sequence classification, and only needs to substitute a new cost function. Cut all RNN gradients to prevent training divergence and avoid negative effects.

Train the model, get_sataset downloads the handwriting image, preprocessing, one-hot encoding vector of lowercase letters. Randomly disrupt the order of the data and divide it into training sets and test sets.

There is a dependency relationship (or mutual information) between adjacent letters of a word, and RNN saves all the input information of the same word to the implicit activity value. For the classification of the first few letters, the network does not have a large amount of input to infer additional information, and bidirectional RNN overcomes the shortcomings.
Two RNNs observe the input sequence, one reads words from the left end in the usual order, and the other reads words from the right end in the reverse order. Two output activity values ​​are obtained for each time step. Splicing before sending to the shared softmax layer. The classifier obtains complete word information from each letter. tf.modle.rnn.bidirectional_rnn is implemented.

Implement bidirectional RNN. Divide the predictive attributes into two functions and focus on less content. The _shared_softmax function passes in the function tensor data to infer the input size. Reusing other architectural functions, the same flattening technique shares the same softmax layer at all time steps. rnn.dynamic_rnn creates two RNNs.
Sequence reversal is easier than implementing the new reverse pass RNN operation. The tf.reverse_sequence function reverses the sequence_lengths frames in the frame data. Data flow diagram nodes have names. The scope parameter is the rnn_dynamic_cell variable scope name, and the default value is RNN. The two parameters are different for RNN and require different domains.
The reversed sequence is fed into the backward RNN, and the network output is reversed and aligned with the forward output. Concatenate two tensors along the RNN neuron output dimension and return. The bidirectional RNN model performs better.

    import gzipimport csvimport numpy as npfrom helpers import downloadclass OcrDataset:

        URL = 'http://ai.stanford.edu/~btaskar/ocr/letter.data.gz'def __init__(self, cache_dir):
            path = download(type(self).URL, cache_dir)
            lines = self._read(path)
            data, target = self._parse(lines)
            self.data, self.target = self._pad(data, target)

        @staticmethoddef _read(filepath):
            with gzip.open(filepath, 'rt') as file_:
                reader = csv.reader(file_, delimiter='\t')
                lines = list(reader)return lines

        @staticmethoddef _parse(lines):
            lines = sorted(lines, key=lambda x: int(x[0]))
            data, target = [], []
            next_ = Nonefor line in lines:if not next_:
                    data.append([])
                    target.append([])else:assert next_ == int(line[0])
                next_ = int(line[2]) if int(line[2]) > -1 else None
                pixels = np.array([int(x) for x in line[6:134]])
                pixels = pixels.reshape((16, 8))
                data[-1].append(pixels)
                target[-1].append(line[1])return data, target

        @staticmethoddef _pad(data, target):
            max_length = max(len(x) for x in target)
            padding = np.zeros((16, 8))
            data = [x + ([padding] * (max_length - len(x))) for x in data]
            target = [x + ([''] * (max_length - len(x))) for x in target]return np.array(data), np.array(target)import tensorflow as tffrom helpers import lazy_propertyclass SequenceLabellingModel:def __init__(self, data, target, params):
            self.data = data
            self.target = target
            self.params = params
            self.prediction
            self.cost
            self.error
            self.optimize

        @lazy_propertydef length(self):
            used = tf.sign(tf.reduce_max(tf.abs(self.data), reduction_indices=2))
            length = tf.reduce_sum(used, reduction_indices=1)
            length = tf.cast(length, tf.int32)return length

        @lazy_propertydef prediction(self):
            output, _ = tf.nn.dynamic_rnn(
                tf.nn.rnn_cell.GRUCell(self.params.rnn_hidden),
                self.data,
                dtype=tf.float32,
                sequence_length=self.length,
            )# Softmax layer.max_length = int(self.target.get_shape()[1])
            num_classes = int(self.target.get_shape()[2])
            weight = tf.Variable(tf.truncated_normal(
                [self.params.rnn_hidden, num_classes], stddev=0.01))
            bias = tf.Variable(tf.constant(0.1, shape=[num_classes]))# Flatten to apply same weights to all time steps.output = tf.reshape(output, [-1, self.params.rnn_hidden])
            prediction = tf.nn.softmax(tf.matmul(output, weight) + bias)
            prediction = tf.reshape(prediction, [-1, max_length, num_classes])return prediction

        @lazy_propertydef cost(self):# Compute cross entropy for each frame.cross_entropy = self.target * tf.log(self.prediction)
            cross_entropy = -tf.reduce_sum(cross_entropy, reduction_indices=2)
            mask = tf.sign(tf.reduce_max(tf.abs(self.target), reduction_indices=2))
            cross_entropy *= mask# Average over actual sequence lengths.cross_entropy = tf.reduce_sum(cross_entropy, reduction_indices=1)
            cross_entropy /= tf.cast(self.length, tf.float32)return tf.reduce_mean(cross_entropy)

        @lazy_propertydef error(self):
            mistakes = tf.not_equal(
                tf.argmax(self.target, 2), tf.argmax(self.prediction, 2))
            mistakes = tf.cast(mistakes, tf.float32)
            mask = tf.sign(tf.reduce_max(tf.abs(self.target), reduction_indices=2))
            mistakes *= mask# Average over actual sequence lengths.mistakes = tf.reduce_sum(mistakes, reduction_indices=1)
            mistakes /= tf.cast(self.length, tf.float32)return tf.reduce_mean(mistakes)

        @lazy_propertydef optimize(self):
            gradient = self.params.optimizer.compute_gradients(self.cost)try:
                limit = self.params.gradient_clipping
                gradient = [
                    (tf.clip_by_value(g, -limit, limit), v)if g is not None else (None, v)for g, v in gradient]except AttributeError:print('No gradient clipping parameter specified.')
            optimize = self.params.optimizer.apply_gradients(gradient)return optimizeimport randomimport tensorflow as tfimport numpy as npfrom helpers import AttrDictfrom OcrDataset import OcrDatasetfrom SequenceLabellingModel import SequenceLabellingModelfrom batched import batched

    params = AttrDict(
        rnn_cell=tf.nn.rnn_cell.GRUCell,
        rnn_hidden=300,
        optimizer=tf.train.RMSPropOptimizer(0.002),
        gradient_clipping=5,
        batch_size=10,
        epochs=5,
        epoch_size=50)def get_dataset():
        dataset = OcrDataset('./ocr')# Flatten images into vectors.dataset.data = dataset.data.reshape(dataset.data.shape[:2] + (-1,))# One-hot encode targets.target = np.zeros(dataset.target.shape + (26,))for index, letter in np.ndenumerate(dataset.target):if letter:
                target[index][ord(letter) - ord('a')] = 1dataset.target = target# Shuffle order of examples.order = np.random.permutation(len(dataset.data))
        dataset.data = dataset.data[order]
        dataset.target = dataset.target[order]return dataset# Split into training and test data.dataset = get_dataset()
    split = int(0.66 * len(dataset.data))
    train_data, test_data = dataset.data[:split], dataset.data[split:]
    train_target, test_target = dataset.target[:split], dataset.target[split:]# Compute graph._, length, image_size = train_data.shape
    num_classes = train_target.shape[2]
    data = tf.placeholder(tf.float32, [None, length, image_size])
    target = tf.placeholder(tf.float32, [None, length, num_classes])
    model = SequenceLabellingModel(data, target, params)
    batches = batched(train_data, train_target, params.batch_size)

    sess = tf.Session()
    sess.run(tf.initialize_all_variables())for index, batch in enumerate(batches):
        batch_data = batch[0]
        batch_target = batch[1]
        epoch = batch[2]if epoch >= params.epochs:breakfeed = {data: batch_data, target: batch_target}
        error, _ = sess.run([model.error, model.optimize], feed)print('{}: {:3.6f}%'.format(index + 1, 100 * error))

    test_feed = {data: test_data, target: test_target}
    test_error, _ = sess.run([model.error, model.optimize], test_feed)print('Test error: {:3.6f}%'.format(100 * error))import tensorflow as tffrom helpers import lazy_propertyclass BidirectionalSequenceLabellingModel:def __init__(self, data, target, params):
            self.data = data
            self.target = target
            self.params = params
            self.prediction
            self.cost
            self.error
            self.optimize

        @lazy_propertydef length(self):
            used = tf.sign(tf.reduce_max(tf.abs(self.data), reduction_indices=2))
            length = tf.reduce_sum(used, reduction_indices=1)
            length = tf.cast(length, tf.int32)return length

        @lazy_propertydef prediction(self):
            output = self._bidirectional_rnn(self.data, self.length)
            num_classes = int(self.target.get_shape()[2])
            prediction = self._shared_softmax(output, num_classes)return predictiondef _bidirectional_rnn(self, data, length):
            length_64 = tf.cast(length, tf.int64)
            forward, _ = tf.nn.dynamic_rnn(
                cell=self.params.rnn_cell(self.params.rnn_hidden),
                inputs=data,
                dtype=tf.float32,
                sequence_length=length,
                scope='rnn-forward')
            backward, _ = tf.nn.dynamic_rnn(
            cell=self.params.rnn_cell(self.params.rnn_hidden),
            inputs=tf.reverse_sequence(data, length_64, seq_dim=1),
            dtype=tf.float32,
            sequence_length=self.length,
            scope='rnn-backward')
            backward = tf.reverse_sequence(backward, length_64, seq_dim=1)
            output = tf.concat(2, [forward, backward])return outputdef _shared_softmax(self, data, out_size):
            max_length = int(data.get_shape()[1])
            in_size = int(data.get_shape()[2])
            weight = tf.Variable(tf.truncated_normal(
                [in_size, out_size], stddev=0.01))
            bias = tf.Variable(tf.constant(0.1, shape=[out_size]))# Flatten to apply same weights to all time steps.flat = tf.reshape(data, [-1, in_size])
            output = tf.nn.softmax(tf.matmul(flat, weight) + bias)
            output = tf.reshape(output, [-1, max_length, out_size])return output

        @lazy_propertydef cost(self):# Compute cross entropy for each frame.cross_entropy = self.target * tf.log(self.prediction)
            cross_entropy = -tf.reduce_sum(cross_entropy, reduction_indices=2)
            mask = tf.sign(tf.reduce_max(tf.abs(self.target), reduction_indices=2))
            cross_entropy *= mask# Average over actual sequence lengths.cross_entropy = tf.reduce_sum(cross_entropy, reduction_indices=1)
            cross_entropy /= tf.cast(self.length, tf.float32)return tf.reduce_mean(cross_entropy)

        @lazy_propertydef error(self):
            mistakes = tf.not_equal(
                tf.argmax(self.target, 2), tf.argmax(self.prediction, 2))
            mistakes = tf.cast(mistakes, tf.float32)
            mask = tf.sign(tf.reduce_max(tf.abs(self.target), reduction_indices=2))
            mistakes *= mask# Average over actual sequence lengths.mistakes = tf.reduce_sum(mistakes, reduction_indices=1)
            mistakes /= tf.cast(self.length, tf.float32)return tf.reduce_mean(mistakes)

        @lazy_propertydef optimize(self):
            gradient = self.params.optimizer.compute_gradients(self.cost)try:
                limit = self.params.gradient_clipping
                gradient = [
                    (tf.clip_by_value(g, -limit, limit), v)if g is not None else (None, v)for g, v in gradient]except AttributeError:print('No gradient clipping parameter specified.')
            optimize = self.params.optimizer.apply_gradients(gradient)return optimize

Reference material:
"TensorFlow Practice for Machine Intelligence"

Welcome to add me on WeChat to communicate: qingxingfengzi
me My wife Zhang Xingqing’s WeChat public account: qingxingfengzigz

The above is the detailed content of Sequence annotation, handwritten lowercase letters OCR data set, bidirectional RNN. For more information, please follow other related articles on the PHP Chinese website!