Category: Writing

I’ve been curious about GPT-3 as a creative tool since reading about Matt Webb’s experiments in 2020. GPT-3 (Generative Pre-trained Transformer 3) is a language model that can create realistic text. The results are impressive, and it has even been used to write a Guardian editorial. In his experiments, Webb was confronted by phrases and ideas that did not exist before. The model produced original concepts such as the “The public bank of Britain”, and passages about “a three-mile wide black ring [that] was found in the ocean using sonar“.

The GPT-3 model is based upon millions of words of Internet content, and Webb has described elsewhere how “Reading GPT-3’s output, for me, feels like dowsing the collective unconscious. I’ve never seen anything so Jungian.”

You can get a quick feel for GPT by playing with the Talk to Transformer page, which allows you to experiment with the basic trained model. There’s a good overview by the Verge, ‘OpenAI’s latest breakthrough is astonishingly powerful, but still fighting its flaws.’ Or, for a more whimsical experiment, Janelle Shae tried asking the model how many legs a horse has, concluding, “It’s grammatically correct, it’s confident, and it’s using a lot of the right vocabulary. But it’s also almost completely wrong. I’m sure I’ve had conversations like this at parties” The origins of the model means it’s also particularly well informed about topics such as Miley Cyrus and Harry Potter.

Sadly, I’ve got no chance of getting my hands on GPT-3 any time soon, since it is kept under tight control to stop it from being used for evil. But then I remembered that Shardcore had used the earlier GPT-2 model for his software-generated book length collaboration with John Higgs The Future Has Already Begun.

I realised that GPT-2 ought to be sophisticated enough to produce something worthwhile, so I decided to give the basic GPT-2 model some additional training based on my creative writing. I’ve read recommendations that you need 25MB-100MB of text, but I’m using 6MB of my writing as input (generated from the source documents using Apache Tika). I was then able to use this with a colab notebook build by Max Woolf to do the hard work.

(I’d not used colab notebooks before, but I am stunned at how they combine workbook and instructions, along with a free VM to run it all on. For more detail, check out Robin Sloan’s post The Slab and the Permacomputer. It’s amazing to see how lots of people’s hard work has combined, allowing me to play with sophisticated models without knowing much about python or machine learning).

The snippets of text generated are identifiably mine in a strange way, but there are flights of fancy that surprise me. A description of a character: “He was a man of his word, not a man of action.” A phrase: “Nobody felt safe watching another human being do something with their lives“. There was a whole mad fantasy about “a group of ‘dusk-blue crabs’ who ’went by the name of ‘the great snout’“. There are also moments where the model just goes on and on repeating “Wax tins! Wax tins! Wax tins!”. Weirdly enough there was also a passage about a John Higgs:

John Higgs, the English economist and writer, died on 26th October, 2001. He was 83 years old. He was happy to join the world scene, and for good reason. He and many of his ideas were burned at the stake for their uselessness.

The main issue I have is my training data, which is unbalanced in various ways – a few novel-length texts, lots of notes. As clever as machine learning is, it’s only as good as your inputs.

Writing with GPT-X is not simply about churning out text – this text does needs to be worked on (This is not ‘cheating’ – Burroughs used to screen his manual cut-ups, looking for poignant and interesting generated sections). There are also different ways to work with the system – Robin Sloan has described some of the techniques he has used, such as hiding prompts from the reader (but not the model) to produce effective writing. These techniques are all waiting to be explored.

Matt Webb has written in detail about his experience of this collaboration in GPT-3 is an idea machine:

Using GPT-3 is work, it’s not a one-shot automation like spellcheck or autocomplete. It’s an interactive, investigative process, and it’s down to the human user to interview GPT-3. There will be people who become expert at dowsing the A.I., just as there are people who are great at searching using Google or finding information in research libraries. I think the skill involved will be similar to being a good improv partner, that’s what it reminds me of.

GPT-3 is capable of novel ideas but it takes a human to identify the good ones. It’s not a replacement for creative imagination. In a 15 minute session with the A.I., I can usually generate one or two concepts, suitable for being worked up into a short story, or turned into a design brief for a product feature, or providing new perspectives in some analysis – it feels very much like a brainstorming workshop, or talking something through with a colleague or an editor.

GPT-X can produce text faster than anyone can read it, but as Sloan writes, “it’s clear that the best thing on the page, the thing that makes it glow, is the part supplied by a person“.

For me, the question is whether it can produce interesting art (particularly art that is not solely interesting because of its process). What I’ve seen so far is both spooky and exciting. Whether this is more than a cheap trick of text remains to be seen, but my initial explorations make me very excited about collaborating further with this model.

Last weekend (on Saturday 23rd, of course), the Liverpool Arts Lab released the first issue of their new zine, Bodge. You can download a free PDF or order a physical copy. The Arts lab are planning 12 issues, on the 23rd of each month through 2021, and it also includes contributions from some of the Cerne-to-CERN pilgrims.

I’m using my recurring page to talk about ley-lines, which is an excuse to bury myself in books about old stones and earth mysteries. I’m still not 100% sure what I think about the topic, but it’s fun to figure it out. And, as the year moves on and we can actually leave the house, I hope to do a few experiments.

I’ve also been working to send out the physical copies, which arrived earlier on Monday. While the zine is available for free, there is something special about receiving culture as a physical object. I think this is particularly great while we are all physically isolated from each other.

Bodge collects together a loose community of people, some I know well, and many I wish I knew better. There is art, poetry, short essays and even a problem page,. It documents a nexus of interests, as well as looking good, and I can’t wait to see what emerges over the year.

If I was suddenly given ‘fuck-you money’ – about £3 million would do it, I reckon – I would still write computer software. But rather than build financial systems, I’d work on procedurally-generated literature. Enterprise software is interesting, but it doesn’t have the philosophical dimension of trying to make a computer write like a person.

I’ve toyed with this a little through the Mechapoet, which never quite managed to be entertaining enough (we did beat one human poet in a slam, but only one). After an evening in the The Basketmakers with Shardcore, I realised I didn’t have the time or patience to put in the work needed for something more impressive.

I made a few other experiments. One was around haiku. These poems are so simple and often shorn of context, so there is a decent chance of beating human writers. All I’d need is the right data set, and a way of judging the new ones. Applying a fitness function via the web was going to be a great deal of work, and I already had a lot of work in my life. But I occasionally day-dream about procedurally-generated literature.

Writing entire novels by computer is a long way off. There have been early attempts, and these tend to be avant-garde rather than containing the sustained narrative we want from novels. There is a website that compares human and computer poetry, and it tags some human poets as being particularly “computer-like”. These poems are fairly ‘experimental’ and it is these fringes that are most open to the computer.

The first book that claimed to be written by a computer was The Policeman’s Beard is Half Constructed, written in 1984 using a program called RACTER. It’s not particularly readable, and is remarkable more for how it was written than the contents.

Another early computer-generated book is Nick Monfort’s World Clock, generated from 165 lines of python code, and is inspired by the work of Oulippian Harry Mathews. The text is interesting, but relies on its structure. I could imagine having responded to this sort of text in an MA workshop class, but it doesn’t have the narrative drive one would expect from a novel.

There was a lovely 2014 Sabotage Reviews piece reviewing 9 computer generated-novels, which included a couple of particularly fascinating examples. One was “a desperate talking clock written by the people of Twitter”, using entries mentioning each specific time. Another “creates a harrowing story from tweets mentioning National Novel Writing Month.”

The writer of this piece, Harry Giles, found an interesting angle on generated literature. He suggested that it is part “of the Oulipian tradition of writing from constraint: if you make such-and-such a ruleset, what kind of writing might happen?” He also compared the results to internet-inspired ‘anti-literature’ forms:

[the thing that generated texts are] closest to is the flattened affect and repetitions of alt-lit, with dashes of uncreative writing, flarf and other post-internet poetics. In other words: as humans increasingly write in dialogue with the internet and machine automations, machines are increasingly being written in dialogue with human literature

Probably the best example of generated text is the Magic Realism Bot. This produces some beautiful images, but it is basically a ‘madlibs’ style program, inserting a pool of words into specific places in pre-prepared sentences. On top of this, the creator prunes out the obvious misfires. It’s a beautiful piece of work, but relies heavily on human innovation and intervention.

(This intrusion of human editing has long been a part of generated texts, even before the computer. William Burroughs would spend hours making cut-ups, which involved slicing physical text with a knife and realigning it to see where new meaning emerged. Out of all this work he would pick the best pieces.)

I actually own one book written by a computer, a version of John Higgs’ recent book on the future, produced by Shardcore with the assistance of GPT-2. The grammar is pretty good, and it turns up some beautifully-weird phrases.

In Higgs’ The Future Starts Here, Shardcore describes the “big scene of bottists who are generating novels and books of poetry. They build these machines to write the stuff, but in the expectation that nobody’s going to read them. You read the first page and think ’I get the gist of this’, but you don’t go on, because it doesn’t make any sense… For it to be a book that you want to read, there’s a lot more to it…”

Shardcore wrote a long blog post dealing how he worked through different techniques before using GPT-2 – “Markov chains produced the usual markov stuff”, and there were also failures from word-level and character-level Recurrent Neural Nets. But he hits the motherlode when GPT-2 created a weird description of the film The Breakfast Club. An entire book was subsequently produced, credited to Algohiggs.

Computer-generated texts are going to become more common. There is even a computer generated textbook on Amazon, Lithium-Ion Batteries: A Machine-Generated Summary of Current Research Hardcover, which costs £30.

My favourite generated text (which I’ve taken almost a thousand words to get to!) is Emily Short’s Annals of the Parrigues. This is a travel guide to an imaginary country, produced from a number of out-of-copyright source texts. It’s hypnotic, with some clear glimpses of a literature.

Procedural generation works well in video games, where it can generate sufficiently-interesting content more easily than people can. In 1984’s Elite, it allowed the game to contain far more worlds that would otherwise fit in the limited memory and storage space of the time. It’s interesting to think about what sort of book might be generated by directly by software and read by people who didn’t care that it was computer-generated. If I had the time and the money, I’d love to find out. Since I don’t, I’ll carry on writing stories the easy way.

I was delighted to be published in the 4th issue of Rituals and Declarations, Paul Watson’s limited-run zine (available here). The article was about my response to the London Road Stone Circle, a stunning piece of urban land art. I’ve heard from a few people who’ve recently set out to explore this work, some of whom had no idea it was there, even though they’d been stepping over it for years. So, that’s good.

I’m particularly pleased to be part of a magazine that has captured so well a community and a turbulent time. Other people published in this series include friends such as Sooxanne, Cat Vincent and Justin Hopper, as well as several people I admire but have yet to meet. Subjects have included Sooxanne writing on Perchta celebrations, a piece on the Burryman, and Allyson Shaw writing about the Maggie Wall memorial. My favourite so far was Dee Dee Chainey’s Covid Eve in the Northern Counties. Set in 2030, this described the events of the Covid Bank Holiday weekend, the Monday closest to March 23rd. It was soothing to see the trials of 2020 absorbed into folklore, and to look at a seemingly-inescapable present as simply more history.

The series has captured a mood among its readers and writers. The first issue came out just after the 2019 election. “These are not good times,” wrote Paul, with no idea what was to come. By issue 2, covid-19 was spreading. Issue 3 came out as the death toll hit 45,000 yet lockdown was easing, a strange and alienating time.

I particularly love how Paul has hosted this magazine. Paul sees Rituals and Declarations as a community, but he also insists on the need for a variety of communities, so that these networks are stronger and more resilient. To that end, Rituals and Declarations was intended to run for only four issues, making a space explicitly for more things to emerge into.

The new issue includes some amazing pieces, such as Jess Richter’s description of a ritual performed at an old family home. Also striking was Maria J. Pérez Cuervo’s essay on women and walking (to my shame, I did not know about Michelle Bernstein’s description of a feminist derive in her novel All the King’s Horses). I was particularly happy to be published alongside Cat Vincent, whose writing about magic is beautifully down-to-earth. His new piece Plastic Altars, Titanium Bones: A Declaration is a powerful description of his relationship to tradition.

Given the strange times, Paul has extended the run for a second volume, another four issues. Rituals and Declarations has been a good companion through dark times. I look forward to it being a companion as the world emerges from the darkness.

Earlier in October was the prize giving for Shoreham Wordfest. I went along as I’d won two prizes for 10-word stories. You can read all three of my entries on the Shoreham Wordfest website.

It was the first time I’d been out since March, when I went to the Trope event at Phoenix Gallery. This event was very different, with rules almost as strict as an aeroplane flight. Everyone remained masked throughout the performances, and drinks were brought to our seats.

It was great to see Katrina Quinn, who’d also read at that Trope event. She read Bittersweet, an account of a pilgrimage on the South Downs Way. Katrina had travelled through Cocking at around the same time as Katharine, Romi and I were there recently.

The prose prize was judged by Catherine Smith, who read her future-tense story When. Catherine spoke a little about flash and micro-fiction, talking about how the ten-word stories needed both a premise and change.

The competition involved submitting three ten-word stories. I wrote a brace of them and then rejected several for not coming down to the required word-length. Of the stories, two won prizes and one was highly-commended. The last of these, a clown story, was actually my favourite, but nobody else seems as impressed with my clown stories as I am.

On Sunday, a story of mine was published on Paragraph Planet: If Vampires Were Real. Paragraph Planet is a long-running flash-fiction site, which publishes a 75-word story each day.

If Vampires Were Real was written as part of the Not for the Faint-Hearted sessions, which I have been running online most Saturdays since lockdown begun. We write 8 or 9 stories each session, which means I’ve written about 100 tiny stories during the pandemic. Most of them I throw away, but a few are worth polishing-up.

Over the past few weeks, I’ve also been writing 500-word stories as part of the Writer’s HQ Flash-Face-Off. I’ve written four pieces so far. None of them have taken a long time to write, but I am fairly happy with how they’ve turned out. All four are going to find a way into my South Downs Way project.

The stories I produce quickly to prompts seems to work better than the pieces I labour with. Lockdown is giving me an opportunity to play with my writing, and I am still learning new things.