Category: Writing

A while back, I wrote about how my writing has developed through reading Chuck Palahniuk’s discussions of technique. An even bigger change over the last year has been to focus on publication, in whatever form that takes.

Anxiety over sharing my work has long been a problem. It wasn’t the simple ‘fear of success’ that some people talk about, rather a ridiculous fear of negative effects from publication. At the same time, I’ve been driven to write stories since I could first write a sentence, and these two drives have been in conflict. Sometimes I’ve thought I should quit writing stories and focus more on other parts of my life – but quitting didn’t work for me either, so I needed to find another way through.

Since moving to Yorkshire, I’ve put more effort into sending work out. A lot of my old work was written with little thought of an audience. It was fun, and some of that work was great, but you lose rigour if you don’t define yourself against any external standards. I wrote some good stories that I have no idea what to do with. An example of this is a story I wrote called Richey Edwards vs Godzilla, a mash-up of indie music and kaijus. It’s a great piece of writing, but almost wilfully obscure.

Change is a strange thing – it can take years but feel sudden. I’ve been toying with ways to put my work in public for a while. Part of this was attending a 2018 Arvon course with Tania Hershman and Nuala O’Connor, which provoked me into one flurry of submissions. The South Downs Way zine project has been an interesting way to explore publication, and putting recent volumes onto etsy has worked well. In 2022, I have become more consistent with submissions (41 so far this year) and it feels like a significant change.

It’s not as if I am now writing things only so they can be published. I have a huge number of ideas and it is more about working on the ones I feel I can find a home for.

Recently I thought about writing a folk horror piece about offices. It was interesting, in that it took the elements of folk horror and transposed them to a corporate setting. But, at the same time, it was mostly a cover version of The Wicker Man. If I’d worked on this, it would have been competent, but I couldn’t imagine being enthusiastic about submitting it. Long stories take a lot of time, and need to be worth spending so much energy on. In the end, I stripped out the elements of the piece I liked, and it will emerge as a smaller, stranger piece than it would otherwise. I’ve spent too long writing solely for myself, and I need to make up for lost time.

I recently had a story of mine, Holiday Wardrobe, read by actor Jennifer Aries at the London Liar’s League event. It’s about a disappointing holiday in a magical kingdom.

It’s particularly exciting to have a story selected by the League, as I get to hear my work performed by someone else. When I’m editing, I read my work out and edit it until the text seems to flow perfectly. Another person will take the same text and draw out different pauses and emphasis. It’s an interesting experience.

This is actually my second appearance at Liar’s League, the first one being in 2008, when my story Eat at Lovecraft’s was read by Becky Hands-Wicks. I’ve sent about half a dozen pieces over the 14 years since then, but this is the first one to be selected. Liar’s League is an amazing event, and I’m excited about submitting more in the future.

I’ve been curious about GPT-3 as a creative tool since reading about Matt Webb’s experiments in 2020. GPT-3 (Generative Pre-trained Transformer 3) is a language model that can create realistic text. The results are impressive, and it has even been used to write a Guardian editorial. In his experiments, Webb was confronted by phrases and ideas that did not exist before. The model produced original concepts such as the “The public bank of Britain”, and passages about “a three-mile wide black ring [that] was found in the ocean using sonar“.

The GPT-3 model is based upon millions of words of Internet content, and Webb has described elsewhere how “Reading GPT-3’s output, for me, feels like dowsing the collective unconscious. I’ve never seen anything so Jungian.”

You can get a quick feel for GPT by playing with the Talk to Transformer page, which allows you to experiment with the basic trained model. There’s a good overview by the Verge, ‘OpenAI’s latest breakthrough is astonishingly powerful, but still fighting its flaws.’ Or, for a more whimsical experiment, Janelle Shae tried asking the model how many legs a horse has, concluding, “It’s grammatically correct, it’s confident, and it’s using a lot of the right vocabulary. But it’s also almost completely wrong. I’m sure I’ve had conversations like this at parties” The origins of the model means it’s also particularly well informed about topics such as Miley Cyrus and Harry Potter.

Sadly, I’ve got no chance of getting my hands on GPT-3 any time soon, since it is kept under tight control to stop it from being used for evil. But then I remembered that Shardcore had used the earlier GPT-2 model for his software-generated book length collaboration with John Higgs The Future Has Already Begun.

I realised that GPT-2 ought to be sophisticated enough to produce something worthwhile, so I decided to give the basic GPT-2 model some additional training based on my creative writing. I’ve read recommendations that you need 25MB-100MB of text, but I’m using 6MB of my writing as input (generated from the source documents using Apache Tika). I was then able to use this with a colab notebook build by Max Woolf to do the hard work.

(I’d not used colab notebooks before, but I am stunned at how they combine workbook and instructions, along with a free VM to run it all on. For more detail, check out Robin Sloan’s post The Slab and the Permacomputer. It’s amazing to see how lots of people’s hard work has combined, allowing me to play with sophisticated models without knowing much about python or machine learning).

The snippets of text generated are identifiably mine in a strange way, but there are flights of fancy that surprise me. A description of a character: “He was a man of his word, not a man of action.” A phrase: “Nobody felt safe watching another human being do something with their lives“. There was a whole mad fantasy about “a group of ‘dusk-blue crabs’ who ’went by the name of ‘the great snout’“. There are also moments where the model just goes on and on repeating “Wax tins! Wax tins! Wax tins!”. Weirdly enough there was also a passage about a John Higgs:

John Higgs, the English economist and writer, died on 26th October, 2001. He was 83 years old. He was happy to join the world scene, and for good reason. He and many of his ideas were burned at the stake for their uselessness.

The main issue I have is my training data, which is unbalanced in various ways – a few novel-length texts, lots of notes. As clever as machine learning is, it’s only as good as your inputs.

Writing with GPT-X is not simply about churning out text – this text does needs to be worked on (This is not ‘cheating’ – Burroughs used to screen his manual cut-ups, looking for poignant and interesting generated sections). There are also different ways to work with the system – Robin Sloan has described some of the techniques he has used, such as hiding prompts from the reader (but not the model) to produce effective writing. These techniques are all waiting to be explored.

Matt Webb has written in detail about his experience of this collaboration in GPT-3 is an idea machine:

Using GPT-3 is work, it’s not a one-shot automation like spellcheck or autocomplete. It’s an interactive, investigative process, and it’s down to the human user to interview GPT-3. There will be people who become expert at dowsing the A.I., just as there are people who are great at searching using Google or finding information in research libraries. I think the skill involved will be similar to being a good improv partner, that’s what it reminds me of.

GPT-3 is capable of novel ideas but it takes a human to identify the good ones. It’s not a replacement for creative imagination. In a 15 minute session with the A.I., I can usually generate one or two concepts, suitable for being worked up into a short story, or turned into a design brief for a product feature, or providing new perspectives in some analysis – it feels very much like a brainstorming workshop, or talking something through with a colleague or an editor.

GPT-X can produce text faster than anyone can read it, but as Sloan writes, “it’s clear that the best thing on the page, the thing that makes it glow, is the part supplied by a person“.

For me, the question is whether it can produce interesting art (particularly art that is not solely interesting because of its process). What I’ve seen so far is both spooky and exciting. Whether this is more than a cheap trick of text remains to be seen, but my initial explorations make me very excited about collaborating further with this model.

Last weekend (on Saturday 23rd, of course), the Liverpool Arts Lab released the first issue of their new zine, Bodge. You can download a free PDF or order a physical copy. The Arts lab are planning 12 issues, on the 23rd of each month through 2021, and it also includes contributions from some of the Cerne-to-CERN pilgrims.

I’m using my recurring page to talk about ley-lines, which is an excuse to bury myself in books about old stones and earth mysteries. I’m still not 100% sure what I think about the topic, but it’s fun to figure it out. And, as the year moves on and we can actually leave the house, I hope to do a few experiments.

I’ve also been working to send out the physical copies, which arrived earlier on Monday. While the zine is available for free, there is something special about receiving culture as a physical object. I think this is particularly great while we are all physically isolated from each other.

Bodge collects together a loose community of people, some I know well, and many I wish I knew better. There is art, poetry, short essays and even a problem page,. It documents a nexus of interests, as well as looking good, and I can’t wait to see what emerges over the year.

If I was suddenly given ‘fuck-you money’ – about £3 million would do it, I reckon – I would still write computer software. But rather than build financial systems, I’d work on procedurally-generated literature. Enterprise software is interesting, but it doesn’t have the philosophical dimension of trying to make a computer write like a person.

I’ve toyed with this a little through the Mechapoet, which never quite managed to be entertaining enough (we did beat one human poet in a slam, but only one). After an evening in the The Basketmakers with Shardcore, I realised I didn’t have the time or patience to put in the work needed for something more impressive.

I made a few other experiments. One was around haiku. These poems are so simple and often shorn of context, so there is a decent chance of beating human writers. All I’d need is the right data set, and a way of judging the new ones. Applying a fitness function via the web was going to be a great deal of work, and I already had a lot of work in my life. But I occasionally day-dream about procedurally-generated literature.

Writing entire novels by computer is a long way off. There have been early attempts, and these tend to be avant-garde rather than containing the sustained narrative we want from novels. There is a website that compares human and computer poetry, and it tags some human poets as being particularly “computer-like”. These poems are fairly ‘experimental’ and it is these fringes that are most open to the computer.

The first book that claimed to be written by a computer was The Policeman’s Beard is Half Constructed, written in 1984 using a program called RACTER. It’s not particularly readable, and is remarkable more for how it was written than the contents.

Another early computer-generated book is Nick Monfort’s World Clock, generated from 165 lines of python code, and is inspired by the work of Oulippian Harry Mathews. The text is interesting, but relies on its structure. I could imagine having responded to this sort of text in an MA workshop class, but it doesn’t have the narrative drive one would expect from a novel.

There was a lovely 2014 Sabotage Reviews piece reviewing 9 computer generated-novels, which included a couple of particularly fascinating examples. One was “a desperate talking clock written by the people of Twitter”, using entries mentioning each specific time. Another “creates a harrowing story from tweets mentioning National Novel Writing Month.”

The writer of this piece, Harry Giles, found an interesting angle on generated literature. He suggested that it is part “of the Oulipian tradition of writing from constraint: if you make such-and-such a ruleset, what kind of writing might happen?” He also compared the results to internet-inspired ‘anti-literature’ forms:

[the thing that generated texts are] closest to is the flattened affect and repetitions of alt-lit, with dashes of uncreative writing, flarf and other post-internet poetics. In other words: as humans increasingly write in dialogue with the internet and machine automations, machines are increasingly being written in dialogue with human literature

Probably the best example of generated text is the Magic Realism Bot. This produces some beautiful images, but it is basically a ‘madlibs’ style program, inserting a pool of words into specific places in pre-prepared sentences. On top of this, the creator prunes out the obvious misfires. It’s a beautiful piece of work, but relies heavily on human innovation and intervention.

(This intrusion of human editing has long been a part of generated texts, even before the computer. William Burroughs would spend hours making cut-ups, which involved slicing physical text with a knife and realigning it to see where new meaning emerged. Out of all this work he would pick the best pieces.)

I actually own one book written by a computer, a version of John Higgs’ recent book on the future, produced by Shardcore with the assistance of GPT-2. The grammar is pretty good, and it turns up some beautifully-weird phrases.

In Higgs’ The Future Starts Here, Shardcore describes the “big scene of bottists who are generating novels and books of poetry. They build these machines to write the stuff, but in the expectation that nobody’s going to read them. You read the first page and think ’I get the gist of this’, but you don’t go on, because it doesn’t make any sense… For it to be a book that you want to read, there’s a lot more to it…”

Shardcore wrote a long blog post dealing how he worked through different techniques before using GPT-2 – “Markov chains produced the usual markov stuff”, and there were also failures from word-level and character-level Recurrent Neural Nets. But he hits the motherlode when GPT-2 created a weird description of the film The Breakfast Club. An entire book was subsequently produced, credited to Algohiggs.

Computer-generated texts are going to become more common. There is even a computer generated textbook on Amazon, Lithium-Ion Batteries: A Machine-Generated Summary of Current Research Hardcover, which costs £30.

My favourite generated text (which I’ve taken almost a thousand words to get to!) is Emily Short’s Annals of the Parrigues. This is a travel guide to an imaginary country, produced from a number of out-of-copyright source texts. It’s hypnotic, with some clear glimpses of a literature.

Procedural generation works well in video games, where it can generate sufficiently-interesting content more easily than people can. In 1984’s Elite, it allowed the game to contain far more worlds that would otherwise fit in the limited memory and storage space of the time. It’s interesting to think about what sort of book might be generated by directly by software and read by people who didn’t care that it was computer-generated. If I had the time and the money, I’d love to find out. Since I don’t, I’ll carry on writing stories the easy way.