1. Zheng ZHANG 1-st year PhD candidate Group ILES, LIMSI
word2vec in TensorFlow
Zheng ZHANG
1-st year PhD candidate
Group ILES, LIMSI
07/03/2017

2. TensorFlow Some basic concepts A linear regression program
Ref:
TensorFlow
Some basic concepts
A linear regression program
y = W * x + b
I will tell you some basics of TensorFlow by using a linear regression program as an example.
07/03/2017

3. TensorFlow The central unit of data
Ref:
TensorFlow
The central unit of data
A tensor consists of a set of primitive values shaped into an array of any number of dimensions. 
I will show you some basics of TensorFlow by using a linear regression program as an example.
07/03/2017

4. Ref: https://www.tensorflow.org/get_started/get_started
Node
Each node takes zero or more tensors as inputs and produces a tensor as an output.
Constant
Variable
Placeholder (not sure…)
Operations
I will tell you some basics of TensorFlow by using a linear regression program as an example.
One type of node is a constant
07/03/2017

5. Ref: https://www.tensorflow.org/get_started/get_started
Computational graph
A computational graph is a series of TensorFlow operations arranged into a graph of nodes.
Build
A TensorFlow Core programs contains two sections:
Building the computational graph.
Running the computational graph.
Run
07/03/2017

6. Placeholder A placeholder is a promise to provide a value later.
Ref:
Placeholder
A placeholder is a promise to provide a value later.
To initialize all the variables in a TensorFlow program, tf.global_variables_initializer() is needed.
Build
This graph is not especially interesting because it always produces a constant result.
Run
07/03/2017

7. Variable Variables allow us to add trainable parameters to a graph.
Ref:
Variable
Variables allow us to add trainable parameters to a graph.
They are constructed with a type and initial value.
Build
To make the model trainable, we need to be able to modify the graph to get new outputs with the same input.
Run
07/03/2017

8. Model Evaluation & Training
Ref:
Model Evaluation & Training
To evaluate the model on training data: loss function
TensorFlow provides optimizers that slowly change each variable in order to minimize the loss function. (In this example we use gradient descent.)
Build
A loss function measures how far apart the current model is from the provided data.
Run
07/03/2017

9. Ref: https://www.tensorflow.org/get_started/get_started
Session
To actually evaluate the nodes, we must run the computational graph within a session.
A session encapsulates the control and state of the TensorFlow runtime.
Build
Run
07/03/2017

10. APIs TensorFlow provides multiple APIs
Ref:
APIs
TensorFlow provides multiple APIs
TensorFlow Core (the lowest API) provides complete programming control.
TensorFlow also provides higher level abstractions for common patterns, structures and functionality.
tf.contrib
APIs
level
You may think only a linear regression program already takes too many lines of code…
TensorFlow Core
07/03/2017

11. word2vec in TensorFlow word2vec_optimized.py Run more efficiently
Ref:
word2vec in TensorFlow
word2vec_optimized.py
Run more efficiently
Optimization
word2vec.py
More advanced features
word2vec_basic.py
07/03/2017

12. Vector Representations of Words
Ref:
Vector Representations of Words
(Learning vector representations of words)
 Distributional Hypothesis, which states that words that appear in the same contexts share semantic meaning.
07/03/2017

13. word2vec in TensorFlow Skip-Gram Model
Ref:
word2vec in TensorFlow Skip-Gram Model
Skip-gram predicts source context- words from the target words.
Example:
“the quick brown fox jumped over the lazy dog”
predict each context word from its target word
(quick, the), (quick, brown), (brown, quick), (brown, fox), ...
except that CBOW predicts target words (e.g. 'mat') from source context words ('the cat sits on the'), while the skip-gram does the inverse
so the task becomes to predict 'the' and 'brown' from 'quick', 'quick' and 'fox' from 'brown', etc.
07/03/2017

14. word2vec in TensorFlow word2vec_basic.py
Ref:
word2vec in TensorFlow word2vec_basic.py
Step 1: Download the data & Read the data into a list of strings.
Step 2: Build the dictionary and replace rare words with UNK token.
Step 3: Function to generate a training batch for the skip-gram model.
Step 4: Build and train a skip- gram model.
Step 5: Begin training.
Step 6: Visualize the embeddings.
Talks about step1&2
07/03/2017

15. word2vec in TensorFlow word2vec_basic.py
Ref:
word2vec in TensorFlow word2vec_basic.py
Step 3: Function to generate a training batch for the skip-gram model.
generate_batch(batch_size, num_skips, skip_window)
batch_size
07/03/2017

16. word2vec in TensorFlow word2vec_basic.py
Ref:
word2vec in TensorFlow word2vec_basic.py
Step 3: Function to generate a training batch for the skip-gram model.
generate_batch(batch_size, num_skips, skip_window)
skip_window: Number of context words in one side of target word
07/03/2017

17. word2vec in TensorFlow word2vec_basic.py
Ref:
word2vec in TensorFlow word2vec_basic.py
Step 3: Function to generate a training batch for the skip-gram model.
generate_batch(batch_size, num_skips, skip_window)
num_skips: number of context words in labels for each target word
07/03/2017

18. word2vec in TensorFlow word2vec_basic.py
Ref:
word2vec in TensorFlow word2vec_basic.py
Step 3: Function to generate a training batch for the skip-gram model.
generate_batch(batch_size, num_skips, skip_window)
The skip-gram model takes two inputs. One is a batch full of integers representing the source context words, the other is for the target words.
07/03/2017

19. word2vec in TensorFlow word2vec_basic.py
Ref:
word2vec in TensorFlow word2vec_basic.py
Step 4: Build and train a skip-gram model.
Placeholders for inputs
The skip-gram model takes two inputs. One is a batch full of integers representing the source context words, the other is for the target words.
07/03/2017

20. word2vec in TensorFlow word2vec_basic.py
Ref:
word2vec in TensorFlow word2vec_basic.py
Step 4: Build and train a skip-gram model.
Embeddings: tf.Variable
embedding_size
vocabulary_size
07/03/2017

21. word2vec in TensorFlow word2vec_basic.py
Ref:
word2vec in TensorFlow word2vec_basic.py
Step 4: Build and train a skip-gram model.
NCE (noise-contrastive estimation) loss
07/03/2017

22. word2vec in TensorFlow word2vec_basic.py
Ref:
word2vec in TensorFlow word2vec_basic.py
Step 4: Build and train a skip-gram model.
SGD optimizer
Compute gradients and update the parameters
07/03/2017

23. word2vec in TensorFlow word2vec_basic.py
Ref:
word2vec in TensorFlow word2vec_basic.py
Step 5: Begin training.
 using a feed_dict to push data into the placeholders and callingtf.Session.run with this new data in a loop.
07/03/2017

24. TensorFlow in our server
full3.5b
07/03/2017