自然语言处理,语音处理、文本处理。语音识别(speech recognition),让计算机能够“听懂”人类语音,语音的文字信息“提取”。
日本富国生命保险公司花170万美元安装人工智能系统,客户语言转换文本,分析词正面或负面。智能客服是人工能智能公司研究重点。循环神经网络(recurrent neural network,RNN)模型。
模型选择。每一个矩形是一个向量,箭头表示函数。最下面一行输入向量,最上面一行输出向量,中间一行RNN状态。一对一,没用RNN,如Vanilla模型,固定大小输入到固定大小输出(图像分类)。一对多,序列输出,图片描述,输入一张图片输出一段文字序列,CNN、RNN结合,图像、语言结合。多对一,序列输入,情感分析,输入一段文字,分类积极、消极情感,如淘宝商品评论分类,用LSTM。多对多,异步序列输入、序列输出,机器翻译,如RNN读取英文语句,以法语形式输出。多对多,同步序列输入、序列输出,视频分类,视频每帧打标记。中间RNN状态部分固定,可多次使用,不需对序列长度预先约束。Andrej Karpathy《The Unreasonable Effectiveness of Recurrent Neural Networks》。http://karpathy.github.io/2015/05/21/rnn-effectiveness/ 。自然语言处理,语音合成(文字生成语音)、语单识别、声纹识别(声纹鉴权)、文本处理(分词、情感分析、文本挖掘)。
英文数字语音识别。https://github.com/pannous/tensorflow-speech-recognition/blob/master/speech2text-tflearn.py 。20行Python代码创建超简单语音识别器。LSTM循环神经网络,TFLearn训练英文数字口语数据集。spoken numbers pcm数据集 http://pannous.net/spoken_numbers.tar 。多人阅读0~9数字英文音频,分男女声,一段音频(wav文件)只有一个数字对应英文声音。标识方法{数字}_人名_xxx。
定义输入数据,预处理数据。语音处理成矩阵形式。梅尔频率倒谱系数(Mel frequency cepstral coefficents, MFCC)特征向量。语音分帧、取对数、逆矩阵,生成MFCC代表语音特征。
定义网络模型。LSTM模型。
训练模型,并存储模型。
预测模型。任意输入一个语音文件,预测。
语音识别,可用在智能输入法、会议快速录入、语音控制系统、智能家居领域。
- #!/usr/bin/env python
- #!/usr/local/bin/python
- # -- coding: utf-8 --
- from future import division, print_function, absolute_import
- import tflearn
- import speech_data
- learning_rate = 0.0001
- training_iters = 300000 # steps 迭代次数
- batch_size = 64
- width = 20 # mfcc features MFCC特征
- height = 80 # (max) length of utterance 最大发音长度
- classes = 10 # digits 数字类别
- batch = word_batch = speech_data.mfcc_batch_generator(batch_size) # 生成每一批MFCC语音
- X, Y = next(batch)
- # train, test, _ = ,X
- trainX, trainY = X, Y
- testX, testY = X, Y #overfit for now
- # Data preprocessing
- # Sequence padding
- # trainX = pad_sequences(trainX, maxlen=100, value=0.)
- # testX = pad_sequences(testX, maxlen=100, value=0.)
- # # Converting labels to binary vectors
- # trainY = to_categorical(trainY, nb_classes=2)
- # testY = to_categorical(testY, nb_classes=2)
- # Network building
- # LSTM模型
- net = tflearn.input_data([None, width, height])
- # net = tflearn.embedding(net, input_dim=10000, output_dim=128)
- net = tflearn.lstm(net, 128, dropout=0.8)
- net = tflearn.fully_connected(net, classes, activation='softmax')
- net = tflearn.regression(net, optimizer='adam', learning_rate=learning_rate, loss='categorical_crossentropy')
- # Training
- model = tflearn.DNN(net, tensorboard_verbose=0)
- model.load("tflearn.lstm.model")
- while 1: #training_iters
- model.fit(trainX, trainY, n_epoch=100, validation_set=(testX, testY), show_metric=True,
- batch_size=batch_size)
- _y=model.predict(X)
- model.save("tflearn.lstm.model")
- print (_y)
- print (y)
智能聊天机器人。未来方向“自然语言人机交互”。苹果Siri、微软Cortana和小冰、Google Now、百度度秘、亚马逊蓝牙音箱Amazon Echo内置语音助手Alexa、Facebook 语音助手M。通过和用户“语音机器人”对话,引导用户到对应服务。今后智能硬件、智能家居嵌入式应用。
智能聊天机器人3代技术。第一代特征工程,大量逻辑判断。第二代检索库,给定问题、聊天,从检索库找到与已有答案最匹配答案。第三代深度学习,seq2seq+Attention模型,大量训练,根据输入生成输出。
seq2seq+Attention模型原理、构建方法。翻译模型,把一个序列翻译成另一个序列。两个RNNLM,一个作编码器,一个解码器,组成RNN编码器-解码器。文本处理领域,常用编码器-解码器(encoder-decoder)框架。输入->编码器->语义编码C->解码器->输出。适合处理上下文(context)生成一个目标(target)通用处理模型。一个句子对,输入给定句子X,通过编码器-解码器框架生成目标句子Y。X、Y可以不同语言,机器翻译。X、Y是对话问句答句,聊天机器人。X、Y可以是图片和对应描述,看图说话。
X由x1、x2等单词序列组成,Y由y1、y2等单词序列组成。编码器编码输入X,生成中间语义编码C,解码器解码中间语义编码C,每个i时刻结合已生成y1、y2……yi-1历史信息生成Yi。生成句子每个词采用中间语义编码相同 C。短句子贴切,长句子不合语义。
实际实现聊天系统,编码器和解码器采用RNN模型、LSTM模型。句子长度超过30,LSTM模型效果急剧下降,引入Attention模型,长句子提升系统效果。Attention机制,人在做一件事情,专注做这件事,忽略周围其他事。源句子中对生成句子重要关键词权重提高,产生更准确应答。增加Attention模型编码器-解码器模型框架:输入->编码器->语义编码C1、C2、C3->解码器->输出Y1、Y2、Y3。中间语义编码Ci不断变化,产生更准确Yi。
最佳实践。https://github.com/suriyadeepan/easy_seq2seq ,依赖TensorFlow 0.12.1环境。康奈尔大学 Corpus数据集(Cornell Movie Dialogs Corpus) http://www.cs.cornell.edu/~cristian/Cornell_Movie-Dialogs_Corpus.html 。600 部电影对白。
处理聊天数据。
先把数据集整理成“问”、“答”文件,生成.enc(问句)、.dec(答句)文件。test.dec #测试集答句,test.enc #测试集问句,train.dec #训练集答句,train.enc #训练集问句。
创建词汇表,问句、答句转换成对应id形式。词汇表文件2万个词汇。vocab20000.dec #答句词汇表,vocab20000.enc #问句词汇表。_GO、_EOS、_UNK、_PAD seq2seq模型特殊标记,填充标记对话。_GO标记对话开始。_EOS标记对话结束。_UNK标记未出现词汇表字符,替换稀有词汇。_PAD填充序列,保证批次序列长度相同。转换成ids文件,test.enc.ids20000、train.dec.ids20000、train.enc.ids20000。问句、答句转换ids文件,每行是一个问句或答句,每行每个id代表问句或答句对应位置词。
采用编码器-解码器框架训练。
定义训练参数。seq2seq.ini。
- [strings]
- # Mode : train, test, serve 模式
- mode = train
- train_enc = data/train.enc
- train_dec = data/train.dec
- test_enc = data/test.enc
- test_dec = data/test.dec
- # folder where checkpoints, vocabulary, temporary data will be stored
- # 模型文件和词汇表存储路径
- working_directory = working_dir/
- [ints]
- # vocabulary size
- # 词汇表大小
- # 20,000 is a reasonable size
- enc_vocab_size = 20000
- dec_vocab_size = 20000
- # number of LSTM layers : 1/2/3
- # LSTM层数
- num_layers = 3
- # typical options : 128, 256, 512, 1024 每层大小,可取值
- layer_size = 256
- # dataset size limit; typically none : no limit
- max_train_data_size = 0
- batch_size = 64
- # steps per checkpoint
- # 每多少次迭代存储一次模型
- # Note : At a checkpoint, models parameters are saved, model is evaluated
- # and results are printed
- steps_per_checkpoint = 300
- [floats]
- learning_rate = 0.5 # 学习速率
- learning_rate_decay_factor = 0.99 # 学习速率下降系数
- max_gradient_norm = 5.0
定义网络模型 seq2seq。seq2seq_model.py。TensorFlow 0.12。定义seq2seq+Attention模型类,3个函数。《Grammar as a Foreign Language》 http://arxiv.org/abs/1412.7499 。初始化模型函数(__init__)、训练模型函数(step)、获取下一批次训练数据函数(get_batch)。
- from future import absolute_import
- from future import division
- from future import print_function
- import random
- import numpy as np
- from six.moves import xrange # pylint: disable=redefined-builtin
- import tensorflow as tf
- from tensorflow.models.rnn.translate import data_utils
- class Seq2SeqModel(object):
- def __init__(self, source_vocab_size, target_vocab_size, buckets, size,
- num_layers, max_gradient_norm, batch_size, learning_rate,
- learning_rate_decay_factor, use_lstm=False,
- num_samples=512, forward_only=False):
- """ 构建模型
- Args: 参数
- source_vocab_size: size of the source vocabulary. 问句词汇表大小
- target_vocab_size: size of the target vocabulary.答句词汇表大小
- buckets: a list of pairs (I, O), where I specifies maximum input length
- that will be processed in that bucket, and O specifies maximum output
- length. Training instances that have inputs longer than I or outputs
- longer than O will be pushed to the next bucket and padded accordingly.
- We assume that the list is sorted, e.g., [(2, 4), (8, 16)].
- 其中I指定最大输入长度,O指定最大输出长度
- size: number of units in each layer of the model.每层神经元数量
- num_layers: number of layers in the model.模型层数
- max_gradient_norm: gradients will be clipped to maximally this norm.梯度被削减到最大规范
- batch_size: the size of the batches used during training;
- the model construction is independent of batch_size, so it can be
- changed after initialization if this is convenient, e.g., for decoding.批次大小。训练、预测批次大小,可不同
- learning_rate: learning rate to start with.学习速率
- learning_rate_decay_factor: decay learning rate by this much when needed.调整学习速率
- use_lstm: if true, we use LSTM cells instead of GRU cells.使用LSTM 单元代替GRU单元
- num_samples: number of samples for sampled softmax.使用softmax样本数
- forward_only: if set, we do not construct the backward pass in the model.是否仅构建前向传播
- """
- self.source_vocab_size = source_vocab_size
- self.target_vocab_size = target_vocab_size
- self.buckets = buckets
- self.batch_size = batch_size
- self.learning_rate = tf.Variable(float(learning_rate), trainable=False)
- self.learning_rate_decay_op = self.learning_rate.assign(
- self.learning_rate * learning_rate_decay_factor)
- self.global_step = tf.Variable(0, trainable=False)
- # If we use sampled softmax, we need an output projection.
- output_projection = None
- softmax_loss_function = None
- # Sampled softmax only makes sense if we sample less than vocabulary size.
- # 如果样本量比词汇表量小,用抽样softmax
- if num_samples > 0 and num_samples < self.target_vocab_size:
- w = tf.get_variable("proj_w", [size, self.target_vocab_size])
- w_t = tf.transpose(w)
- b = tf.get_variable("proj_b", [self.target_vocab_size])
- output_projection = (w, b)
- def sampled_loss(inputs, labels):
- labels = tf.reshape(labels, [-1, 1])
- return tf.nn.sampled_softmax_loss(w_t, b, inputs, labels, num_samples,
- self.target_vocab_size)
- softmax_loss_function = sampled_loss
- # Create the internal multi-layer cell for our RNN.
- # 构建RNN
- single_cell = tf.nn.rnn_cell.GRUCell(size)
- if use_lstm:
- single_cell = tf.nn.rnn_cell.BasicLSTMCell(size)
- cell = single_cell
- cell = tf.nn.rnn_cell.DropoutWrapper(cell, output_keep_prob=0.5)
- if num_layers > 1:
- cell = tf.nn.rnn_cell.MultiRNNCell([single_cell] * num_layers)
- # The seq2seq function: we use embedding for the input and attention.
- # Attention模型
- def seq2seq_f(encoder_inputs, decoder_inputs, do_decode):
- return tf.nn.seq2seq.embedding_attention_seq2seq(
- encoder_inputs, decoder_inputs, cell,
- num_encoder_symbols=source_vocab_size,
- num_decoder_symbols=target_vocab_size,
- embedding_size=size,
- output_projection=output_projection,
- feed_previous=do_decode)
- # Feeds for inputs.
- # 给模型填充数据
- self.encoder_inputs = []
- self.decoder_inputs = []
- self.target_weights = []
- for i in xrange(buckets[-1][0]): # Last bucket is the biggest one.
- self.encoder_inputs.append(tf.placeholder(tf.int32, shape=[None],
- name="encoder{0}".format(i)))
- for i in xrange(buckets[-1][1] + 1):
- self.decoder_inputs.append(tf.placeholder(tf.int32, shape=[None],
- name="decoder{0}".format(i)))
- self.target_weights.append(tf.placeholder(tf.float32, shape=[None],
- name="weight{0}".format(i)))
- # Our targets are decoder inputs shifted by one.
- # targets值是解码器偏移1位
- targets = [self.decoder_inputs[i + 1]
- for i in xrange(len(self.decoder_inputs) - 1)]
- # Training outputs and losses.
- # 训练模型输出
- if forward_only:
- self.outputs, self.losses = tf.nn.seq2seq.model_with_buckets(
- self.encoder_inputs, self.decoder_inputs, targets,
- self.target_weights, buckets, lambda x, y: seq2seq_f(x, y, True),
- softmax_loss_function=softmax_loss_function)
- # If we use output projection, we need to project outputs for decoding.
- if output_projection is not None:
- for b in xrange(len(buckets)):
- self.outputs[b] = [
- tf.matmul(output, output_projection[0]) + output_projection[1]
- for output in self.outputs[b]
- ]
- else:
- self.outputs, self.losses = tf.nn.seq2seq.model_with_buckets(
- self.encoder_inputs, self.decoder_inputs, targets,
- self.target_weights, buckets,
- lambda x, y: seq2seq_f(x, y, False),
- softmax_loss_function=softmax_loss_function)
- # Gradients and SGD update operation for training the model.
- # 训练模型,更新梯度
- params = tf.trainable_variables()
- if not forward_only:
- self.gradient_norms = []
- self.updates = []
- opt = tf.train.AdamOptimizer()
- for b in xrange(len(buckets)):
- gradients = tf.gradients(self.losses[b], params)
- clipped_gradients, norm = tf.clip_by_global_norm(gradients,
- max_gradient_norm)
- self.gradient_norms.append(norm)
- self.updates.append(opt.apply_gradients(
- zip(clipped_gradients, params), global_step=self.global_step))
- self.saver = tf.train.Saver(tf.global_variables())
- def step(self, session, encoder_inputs, decoder_inputs, target_weights,
- bucket_id, forward_only):
- """Run a step of the model feeding the given inputs.
- 定义运行模型的每一步
- Args:
- session: tensorflow session to use.
- encoder_inputs: list of numpy int vectors to feed as encoder inputs.问句向量序列
- decoder_inputs: list of numpy int vectors to feed as decoder inputs.答句向量序列
- target_weights: list of numpy float vectors to feed as target weights.
- bucket_id: which bucket of the model to use.输入bucket_id
- forward_only: whether to do the backward step or only forward.是否只做前向传播
- Returns:
- A triple consisting of gradient norm (or None if we did not do backward),
- average perplexity, and the outputs.
- Raises:
- ValueError: if length of encoder_inputs, decoder_inputs, or
- target_weights disagrees with bucket size for the specified bucket_id.
- """
- # Check if the sizes match.
- encoder_size, decoder_size = self.buckets[bucket_id]
- if len(encoder_inputs) != encoder_size:
- raise ValueError("Encoder length must be equal to the one in bucket,"
- " %d != %d." % (len(encoder_inputs), encoder_size))
- if len(decoder_inputs) != decoder_size:
- raise ValueError("Decoder length must be equal to the one in bucket,"
- " %d != %d." % (len(decoder_inputs), decoder_size))
- if len(target_weights) != decoder_size:
- raise ValueError("Weights length must be equal to the one in bucket,"
- " %d != %d." % (len(target_weights), decoder_size))
- # Input feed: encoder inputs, decoder inputs, target_weights, as provided.
- # 输入填充
- input_feed = {}
- for l in xrange(encoder_size):
- input_feed[self.encoder_inputs[l].name] = encoder_inputs[l]
- for l in xrange(decoder_size):
- input_feed[self.decoder_inputs[l].name] = decoder_inputs[l]
- input_feed[self.target_weights[l].name] = target_weights[l]
- # Since our targets are decoder inputs shifted by one, we need one more.
- last_target = self.decoder_inputs[decoder_size].name
- input_feed[last_target] = np.zeros([self.batch_size], dtype=np.int32)
- # Output feed: depends on whether we do a backward step or not.
- # 输出填充:与是否有后向传播有关
- if not forward_only:
- output_feed = [self.updates[bucket_id], # Update Op that does SGD.
- self.gradient_norms[bucket_id], # Gradient norm.
- self.losses[bucket_id]] # Loss for this batch.
- else:
- output_feed = [self.losses[bucket_id]] # Loss for this batch.
- for l in xrange(decoder_size): # Output logits.
- output_feed.append(self.outputs[bucket_id][l])
- outputs = session.run(output_feed, input_feed)
- if not forward_only:
- return outputs[1], outputs[2], None # Gradient norm, loss, no outputs.有后向传播输出,梯度、损失值、None
- else:
- return None, outputs[0], outputs[1:] # No gradient norm, loss, outputs.仅有前向传播输出,None,损失值,None
- def get_batch(self, data, bucket_id):
- """
- 从指定桶获取一个批次随机数据,在训练每步(step)使用
- Args:参数
- data: a tuple of size len(self.buckets) in which each element contains
- lists of pairs of input and output data that we use to create a batch.长度为(self.buckets)元组,每个元素包含创建批次输入、输出数据对列表
- bucket_id: integer, which bucket to get the batch for.整数,从哪个bucket获取批次
- Returns:返回
- The triple (encoder_inputs, decoder_inputs, target_weights) for
- the constructed batch that has the proper format to call step(...) later.一个包含三项元组(encoder_inputs, decoder_inputs, target_weights)
- """
- encoder_size, decoder_size = self.buckets[bucket_id]
- encoder_inputs, decoder_inputs = [], []
- # Get a random batch of encoder and decoder inputs from data,
- # pad them if needed, reverse encoder inputs and add GO to decoder.
- for _ in xrange(self.batch_size):
- encoder_input, decoder_input = random.choice(data[bucket_id])
- # Encoder inputs are padded and then reversed.
- encoder_pad = [data_utils.PAD_ID] * (encoder_size - len(encoder_input))
- encoder_inputs.append(list(reversed(encoder_input + encoder_pad)))
- # Decoder inputs get an extra "GO" symbol, and are padded then.
- decoder_pad_size = decoder_size - len(decoder_input) - 1
- decoder_inputs.append([data_utils.GO_ID] + decoder_input +
- [data_utils.PAD_ID] * decoder_pad_size)
- # Now we create batch-major vectors from the data selected above.
- batch_encoder_inputs, batch_decoder_inputs, batch_weights = [], [], []
- # Batch encoder inputs are just re-indexed encoder_inputs.
- for length_idx in xrange(encoder_size):
- batch_encoder_inputs.append(
- np.array([encoder_inputs[batch_idx][length_idx]
- for batch_idx in xrange(self.batch_size)], dtype=np.int32))
- # Batch decoder inputs are re-indexed decoder_inputs, we create weights.
- for length_idx in xrange(decoder_size):
- batch_decoder_inputs.append(
- np.array([decoder_inputs[batch_idx][length_idx]
- for batch_idx in xrange(self.batch_size)], dtype=np.int32))
- # Create target_weights to be 0 for targets that are padding.
- batch_weight = np.ones(self.batch_size, dtype=np.float32)
- for batch_idx in xrange(self.batch_size):
- # We set weight to 0 if the corresponding target is a PAD symbol.
- # The corresponding target is decoder_input shifted by 1 forward.
- if length_idx < decoder_size - 1:
- target = decoder_inputs[batch_idx][length_idx + 1]
- if length_idx == decoder_size - 1 or target == data_utils.PAD_ID:
- batch_weight[batch_idx] = 0.0
- batch_weights.append(batch_weight)
- return batch_encoder_inputs, batch_decoder_inputs, batch_weights
训练模型。修改seq2seq.ini文件mode值“train”,execute.py训练。
验证模型。修改seq2seq.ini文件mode值“test”,execute.py测试。
- from future import absolute_import
- from future import division
- from future import print_function
- import math
- import os
- import random
- import sys
- import time
- import numpy as np
- from six.moves import xrange # pylint: disable=redefined-builtin
- import tensorflow as tf
- import data_utils
- import seq2seq_model
- try:
- from ConfigParser import SafeConfigParser
- except:
- from configparser import SafeConfigParser # In Python 3, ConfigParser has been renamed to configparser for PEP 8 compliance.
- gConfig = {}
- def get_config(config_file='seq2seq.ini'):
- parser = SafeConfigParser()
- parser.read(config_file)
- # get the ints, floats and strings
- _conf_ints = [ (key, int(value)) for key,value in parser.items('ints') ]
- _conf_floats = [ (key, float(value)) for key,value in parser.items('floats') ]
- _conf_strings = [ (key, str(value)) for key,value in parser.items('strings') ]
- return dict(_conf_ints + _conf_floats + _conf_strings)
- # We use a number of buckets and pad to the closest one for efficiency.
- # See seq2seq_model.Seq2SeqModel for details of how they work.
- _buckets = [(5, 10), (10, 15), (20, 25), (40, 50)]
- def read_data(source_path, target_path, max_size=None):
- """Read data from source and target files and put into buckets.
- Args:
- source_path: path to the files with token-ids for the source language.
- target_path: path to the file with token-ids for the target language;
- it must be aligned with the source file: n-th line contains the desired
- output for n-th line from the source_path.
- max_size: maximum number of lines to read, all other will be ignored;
- if 0 or None, data files will be read completely (no limit).
- Returns:
- data_set: a list of length len(_buckets); data_set[n] contains a list of
- (source, target) pairs read from the provided data files that fit
- into the n-th bucket, i.e., such that len(source) < _buckets[n][0] and
- len(target) < _buckets[n][1]; source and target are lists of token-ids.
- """
- data_set = [[] for _ in _buckets]
- with tf.gfile.GFile(source_path, mode="r") as source_file:
- with tf.gfile.GFile(target_path, mode="r") as target_file:
- source, target = source_file.readline(), target_file.readline()
- counter = 0
- while source and target and (not max_size or counter < max_size):
- counter += 1
- if counter % 100000 == 0:
- print(" reading data line %d" % counter)
- sys.stdout.flush()
- source_ids = [int(x) for x in source.split()]
- target_ids = [int(x) for x in target.split()]
- target_ids.append(data_utils.EOS_ID)
- for bucket_id, (source_size, target_size) in enumerate(_buckets):
- if len(source_ids) < source_size and len(target_ids) < target_size:
- data_set[bucket_id].append([source_ids, target_ids])
- break
- source, target = source_file.readline(), target_file.readline()
- return data_set
- def create_model(session, forward_only):
- """Create model and initialize or load parameters"""
- model = seq2seq_model.Seq2SeqModel( gConfig['enc_vocab_size'], gConfig['dec_vocab_size'], _buckets, gConfig['layer_size'], gConfig['num_layers'], gConfig['max_gradient_norm'], gConfig['batch_size'], gConfig['learning_rate'], gConfig['learning_rate_decay_factor'], forward_only=forward_only)
- if 'pretrained_model' in gConfig:
- model.saver.restore(session,gConfig['pretrained_model'])
- return model
- ckpt = tf.train.get_checkpoint_state(gConfig['working_directory'])
- if ckpt and ckpt.model_checkpoint_path:
- print("Reading model parameters from %s" % ckpt.model_checkpoint_path)
- model.saver.restore(session, ckpt.model_checkpoint_path)
- else:
- print("Created model with fresh parameters.")
- session.run(tf.global_variables_initializer())
- return model
- def train():
- # prepare dataset
- # 准备数据集
- print("Preparing data in %s" % gConfig['working_directory'])
- enc_train, dec_train, enc_dev, dec_dev, _, _ = data_utils.prepare_custom_data(gConfig['working_directory'],gConfig['train_enc'],gConfig['train_dec'],gConfig['test_enc'],gConfig['test_dec'],gConfig['enc_vocab_size'],gConfig['dec_vocab_size'])
- # setup config to use BFC allocator
- config = tf.ConfigProto()
- config.gpu_options.allocator_type = 'BFC'
- with tf.Session(config=config) as sess:
- # Create model.
- # 构建模型
- print("Creating %d layers of %d units." % (gConfig['num_layers'], gConfig['layer_size']))
- model = create_model(sess, False)
- # Read data into buckets and compute their sizes.
- # 把数据读入桶(bucket)中,计算桶大小
- print ("Reading development and training data (limit: %d)."
- % gConfig['max_train_data_size'])
- dev_set = read_data(enc_dev, dec_dev)
- train_set = read_data(enc_train, dec_train, gConfig['max_train_data_size'])
- train_bucket_sizes = [len(train_set[b]) for b in xrange(len(_buckets))]
- train_total_size = float(sum(train_bucket_sizes))
- # A bucket scale is a list of increasing numbers from 0 to 1 that we'll use
- # to select a bucket. Length of [scale[i], scale[i+1]] is proportional to
- # the size if i-th training bucket, as used later.
- train_buckets_scale = [sum(train_bucket_sizes[:i + 1]) / train_total_size
- for i in xrange(len(train_bucket_sizes))]
- # This is the training loop.
- # 开始训练循环
- step_time, loss = 0.0, 0.0
- current_step = 0
- previous_losses = []
- while True:
- # Choose a bucket according to data distribution. We pick a random number
- # in [0, 1] and use the corresponding interval in train_buckets_scale.
- # 随机生成一个0-1数,在生成bucket_id中使用
- random_number_01 = np.random.random_sample()
- bucket_id = min([i for i in xrange(len(train_buckets_scale))
- if train_buckets_scale[i] > random_number_01])
- # Get a batch and make a step.
- # 获取一个批次数据,进行一步训练
- start_time = time.time()
- encoder_inputs, decoder_inputs, target_weights = model.get_batch(
- train_set, bucket_id)
- _, step_loss, _ = model.step(sess, encoder_inputs, decoder_inputs,
- target_weights, bucket_id, False)
- step_time += (time.time() - start_time) / gConfig['steps_per_checkpoint']
- loss += step_loss / gConfig['steps_per_checkpoint']
- current_step += 1
- # Once in a while, we save checkpoint, print statistics, and run evals.
- # 保存检查点文件,打印统计数据
- if current_step % gConfig['steps_per_checkpoint'] == 0:
- # Print statistics for the previous epoch.
- perplexity = math.exp(loss) if loss < 300 else float('inf')
- print ("global step %d learning rate %.4f step-time %.2f perplexity "
- "%.2f" % (model.global_step.eval(), model.learning_rate.eval(),
- step_time, perplexity))
- # Decrease learning rate if no improvement was seen over last 3 times.
- # 如果损失值在最近3次内没有再降低,减小学习率
- if len(previous_losses) > 2 and loss > max(previous_losses[-3:]):
- sess.run(model.learning_rate_decay_op)
- previous_losses.append(loss)
- # Save checkpoint and zero timer and loss.
- # 保存检查点文件,计数器、损失值归零
- checkpoint_path = os.path.join(gConfig['working_directory'], "seq2seq.ckpt")
- model.saver.save(sess, checkpoint_path, global_step=model.global_step)
- step_time, loss = 0.0, 0.0
- # Run evals on development set and print their perplexity.
- for bucket_id in xrange(len(_buckets)):
- if len(dev_set[bucket_id]) == 0:
- print(" eval: empty bucket %d" % (bucket_id))
- continue
- encoder_inputs, decoder_inputs, target_weights = model.get_batch(
- dev_set, bucket_id)
- _, eval_loss, _ = model.step(sess, encoder_inputs, decoder_inputs,
- target_weights, bucket_id, True)
- eval_ppx = math.exp(eval_loss) if eval_loss < 300 else float('inf')
- print(" eval: bucket %d perplexity %.2f" % (bucket_id, eval_ppx))
- sys.stdout.flush()
- def decode():
- with tf.Session() as sess:
- # Create model and load parameters.
- # 建立模型,定义超参数batch_size
- model = create_model(sess, True)
- model.batch_size = 1 # We decode one sentence at a time.一次只解码一个句子
- # Load vocabularies.
- # 加载词汇表文件
- enc_vocab_path = os.path.join(gConfig['working_directory'],"vocab%d.enc" % gConfig['enc_vocab_size'])
- dec_vocab_path = os.path.join(gConfig['working_directory'],"vocab%d.dec" % gConfig['dec_vocab_size'])
- enc_vocab, _ = data_utils.initialize_vocabulary(enc_vocab_path)
- _, rev_dec_vocab = data_utils.initialize_vocabulary(dec_vocab_path)
- # Decode from standard input.
- # 对标准输入句子解码
- sys.stdout.write("> ")
- sys.stdout.flush()
- sentence = sys.stdin.readline()
- while sentence:
- # Get token-ids for the input sentence.
- # 得到输入句子的token-ids
- token_ids = data_utils.sentence_to_token_ids(tf.compat.as_bytes(sentence), enc_vocab)
- # Which bucket does it belong to?
- # 计算token_ids属于哪个桶(bucket)
- bucket_id = min([b for b in xrange(len(_buckets))
- if _buckets[b][0] > len(token_ids)])
- # Get a 1-element batch to feed the sentence to the model.
- # 句子送入模型
- encoder_inputs, decoder_inputs, target_weights = model.get_batch(
- {bucket_id: [(token_ids, [])]}, bucket_id)
- # Get output logits for the sentence.
- _, _, output_logits = model.step(sess, encoder_inputs, decoder_inputs,
- target_weights, bucket_id, True)
- # This is a greedy decoder - outputs are just argmaxes of output_logits.
- # 贪心解码器,输出output_logits argmaxes
- outputs = [int(np.argmax(logit, axis=1)) for logit in output_logits]
- # If there is an EOS symbol in outputs, cut them at that point.
- if data_utils.EOS_ID in outputs:
- outputs = outputs[:outputs.index(data_utils.EOS_ID)]
- # Print out French sentence corresponding to outputs.
- # 打印与输出句子对应法语句子
- print(" ".join([tf.compat.as_str(rev_dec_vocab[output]) for output in outputs]))
- print("> ", end="")
- sys.stdout.flush()
- sentence = sys.stdin.readline()
- def self_test():
- """Test the translation model."""
- with tf.Session() as sess:
- print("Self-test for neural translation model.")
- # Create model with vocabularies of 10, 2 small buckets, 2 layers of 32.
- model = seq2seq_model.Seq2SeqModel(10, 10, [(3, 3), (6, 6)], 32, 2,
- 5.0, 32, 0.3, 0.99, num_samples=8)
- sess.run(tf.initialize_all_variables())
- # Fake data set for both the (3, 3) and (6, 6) bucket.
- data_set = ([([1, 1], [2, 2]), ([3, 3], [4]), ([5], [6])],
- [([1, 1, 1, 1, 1], [2, 2, 2, 2, 2]), ([3, 3, 3], [5, 6])])
- for _ in xrange(5): # Train the fake model for 5 steps.
- bucket_id = random.choice([0, 1])
- encoder_inputs, decoder_inputs, target_weights = model.get_batch(
- data_set, bucket_id)
- model.step(sess, encoder_inputs, decoder_inputs, target_weights,
- bucket_id, False)
- def init_session(sess, conf='seq2seq.ini'):
- global gConfig
- gConfig = get_config(conf)
- # Create model and load parameters.
- model = create_model(sess, True)
- model.batch_size = 1 # We decode one sentence at a time.
- # Load vocabularies.
- enc_vocab_path = os.path.join(gConfig['working_directory'],"vocab%d.enc" % gConfig['enc_vocab_size'])
- dec_vocab_path = os.path.join(gConfig['working_directory'],"vocab%d.dec" % gConfig['dec_vocab_size'])
- enc_vocab, _ = data_utils.initialize_vocabulary(enc_vocab_path)
- _, rev_dec_vocab = data_utils.initialize_vocabulary(dec_vocab_path)
- return sess, model, enc_vocab, rev_dec_vocab
- def decode_line(sess, model, enc_vocab, rev_dec_vocab, sentence):
- # Get token-ids for the input sentence.
- token_ids = data_utils.sentence_to_token_ids(tf.compat.as_bytes(sentence), enc_vocab)
- # Which bucket does it belong to?
- bucket_id = min([b for b in xrange(len(_buckets)) if _buckets[b][0] > len(token_ids)])
- # Get a 1-element batch to feed the sentence to the model.
- encoder_inputs, decoder_inputs, target_weights = model.get_batch({bucket_id: [(token_ids, [])]}, bucket_id)
- # Get output logits for the sentence.
- _, _, output_logits = model.step(sess, encoder_inputs, decoder_inputs, target_weights, bucket_id, True)
- # This is a greedy decoder - outputs are just argmaxes of output_logits.
- outputs = [int(np.argmax(logit, axis=1)) for logit in output_logits]
- # If there is an EOS symbol in outputs, cut them at that point.
- if data_utils.EOS_ID in outputs:
- outputs = outputs[:outputs.index(data_utils.EOS_ID)]
- return " ".join([tf.compat.as_str(rev_dec_vocab[output]) for output in outputs])
- if name == '__main__':
- if len(sys.argv) - 1:
- gConfig = get_config(sys.argv[1])
- else:
- # get configuration from seq2seq.ini
- gConfig = get_config()
- print('\n>> Mode : %s\n' %(gConfig['mode']))
- if gConfig['mode'] == 'train':
- # start training
- train()
- elif gConfig['mode'] == 'test':
- # interactive decode
- decode()
- else:
- # wrong way to execute "serve"
- # Use : >> python ui/app.py
- # uses seq2seq_serve.ini as conf file
- print('Serve Usage : >> python ui/app.py')
- print('# uses seq2seq_serve.ini as conf file')
基于文字智能机器人,结合语音识别,产生直接对话机器人。系统架构:
人->语音识别(ASR)->自然语言理解(NLU)->对话管理->自然语言生成(NLG)->语音合成(TTS)->人。《中国人工智能学会通讯》2016年第6卷第1期。
图灵机器人公司,提高对话和语义准确度,提升中文语境智能程度。竹间智能科技,研究记忆、自学习情感机器人,机器人真正理解多模式多渠道信息,高度拟人化回应,最理想自然语言交流模式交流。腾讯公司,社交对话数据。微信,最庞大自然语言交流语料库,利用庞大真实数据,结合小程序成为所有服务入口。
参考资料:
《TensorFlow技术解析与实战》
欢迎推荐上海机器学习工作机会
人工智能工作机会分割线-----------------------------------------
杭州阿里 新零售淘宝基础架构平台:移动AI高级专家
来源: https://yq.aliyun.com/articles/231811