Post-Image

Generating poetry with GPT-2

I use OpenAI’s GPT-2 to generate poetry. The model takes a string as inspiration which consists firstly of a lengthy poem to set the task/style and a phrase concatenated to the end which sets the topic.

Resources

Original code and blog article
Final code
https://github.com/mullikine/gpt-2
Extra reading
https://www.gwern.net/GPT-2

Running Dockerfile.cpu

Original steps

  • Download the HTML from poetryfoundation.org based on the urls in romantic-urls.txt. https://gist.github.com/bea516e3726d0a0ab139bca534a43fe9
    • Use Parse Poetry.py to extract

      • title image = “images/blog/generating-poetry-with-gpt-2.png”
      • author
      • poem
    • Save that data to output/.

      • The metadata is contained in the first few lines.
    • Use Generate GPT-2.py to generate poems based on random chunks from the poems and the seed words.

Updated steps

Create inspiration data

  • Extract
    • title image = “images/blog/generating-poetry-with-gpt-2.png”
    • author
    • poem
  • Save that data to output/.
    • The metadata is contained in the first few lines.

Set up repository

  • Convert the notebooks to py
  • place the notebooks py files inside src

$MYGIT/openai/gpt-2/src

Get OpenAI GPT-2 running

  • Create a docker container Base it on the OpenAI container that includes the notebook py files.
  • Run host machine Do this so python development becomes easy.

Fix sample.py.

Thanks Chen! http://ipsumdominum.github.io/

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
diff --git a/src/sample.py b/src/sample.py
index c90ed28..41eb76f 100644
--- a/src/sample.py
+++ b/src/sample.py
@@ -1,6 +1,8 @@
 import tensorflow as tf

+import numpy as np
 import model
+from shanepy import *

 def top_k_logits(logits, k):
     if k == 0:
@@ -25,7 +27,7 @@ def top_k_logits(logits, k):
 def top_p_logits(logits, p):
     """Nucleus sampling"""
     batch, _ = logits.shape.as_list()
-    sorted_logits = tf.sort(logits, direction='DESCENDING', axis=-1)
+    sorted_logits = tf.contrib.framework.sort(logits, direction='DESCENDING', axis=-1)
     cumulative_probs = tf.cumsum(tf.nn.softmax(sorted_logits, axis=-1), axis=-1)
     indices = tf.stack([
         tf.range(0, batch),
@@ -33,6 +35,11 @@ def top_p_logits(logits, p):
         tf.maximum(tf.reduce_sum(tf.cast(cumulative_probs <= p, tf.int32), axis=-1) - 1, 0),
     ], axis=-1)
     min_values = tf.gather_nd(sorted_logits, indices)
+
+    # myembed(globals(), locals())
+
+    # return np.where(
+    logits = tf.transpose(logits)
     return tf.where(
         logits < min_values,
         tf.ones_like(logits) * -1e10,
@@ -51,6 +58,9 @@ def sample_sequence(*, hparams, length, start_token=None, batch_size=None, conte
         lm_output = model.model(hparams=hparams, X=tokens, past=past, reuse=tf.AUTO_REUSE)

         logits = lm_output['logits'][:, :, :hparams.n_vocab]
+
+        # myembed(globals(), locals())
+
         presents = lm_output['present']
         presents.set_shape(model.past_shape(hparams=hparams, batch_size=batch_size))
         return {
@@ -64,6 +74,9 @@ def sample_sequence(*, hparams, length, start_token=None, batch_size=None, conte
             logits = next_outputs['logits'][:, -1, :]  / tf.to_float(temperature)
             logits = top_k_logits(logits, k=top_k)
             logits = top_p_logits(logits, p=top_p)
+            # myembed(globals(), locals())
+            logits = tf.transpose(logits)
+            # print(logits.shape)
             samples = tf.multinomial(logits, num_samples=1, output_dtype=tf.int32)
             return [
                 next_outputs['presents'] if past is None else tf.concat([past, next_outputs['presents']], axis=-2),

Generate poetry

  • Output files:
    • generated.json
    • poems.json

Run with debugging

1
2
3
4
5
6
cd "$MYGIT/mullikine/gpt-2/src";
python3.6 \
    -m trace \
    --ignore-dir=$HOME/lib64:$HOME/lib:/usr \
    -t \
    gen.py

Output is generated