Published: Fri 19 May 2023
By Koen Martens
In Author Software .
tags: frontpage-news
For a while now I’ve been sending out a regular newsletter about my writing and other things that fancy my interest. The first one went out the 7th of April in 2021, and was just a plain-text email asking people to vote for a bit of flash fiction I wrote. Since then, I’ve added images and layout to the emails, but the software I used to send it has been the same since.
I’ve been using https://www.thunderbird.net/ to compose the newsletter content and https://list.org to distribute the newsletter to the subscriber list. Both are open source and either running on my laptop or hosted on my own server. That’s important to me. I don’t want to rely on a commercial service. While there are several big corporations providing newsletter sending services, I feel uncomfortable handing over my subscriber list to big tech.
As the newsletter design grew more elaborate and the content more varied, it became harder and harder to lay out the newsletter or provide additional features such as per-subscriber download links. More than once, this led to mistakes such as inserting the wrong link to a book from a fellow author I promised to feature or other such badness. It was time to make things easier on myself, automating much of the repetitive tasks so I can concentrate on the content instead of the technical details.
For anyone curious about what I did, this article provides a behind-the-scenes look at the software I use to send out my newsletter, and how I have tied it all together. I may have went a bit overboard, and even drew an architectural sketch.
Email format
An email message can be in one of two formats: plain text or HTML . Plain text is how email was invented in the seventies. Just characters, no formatting or layout at all. Perhaps, when the Multi-purpose Internet Mail Extension (MIME ) was introduced in the nineties, one would attach an image or video that you had to open explicitly. The nineties saw two more inventions that changed the face of email: the Hypertext Markup Language (HTML ) and the Cascading Style Sheets (CSS ). It defined the world wide web as we know it now. In addition to text, HTML allowed one to mark up documents to contain formatting (such as font styles or families) and layout (such as embedding images within the text).
While not universally welcomed initially, these days HTML emails are the default. Sadly, though, the state of HTML support among different desktop email clients and webmail services varies. My favourite email client, Thunderbird , has pretty good support and will render even the more modern HTML layout tags and CSS formatting features. Microsoft Outlook, on the other hand, really only supports an ancient subset. Webmail clients vary in their support. As a result, to make sure the newsletter shows consistently on the most popular email clients, one has to limit oneself to the lowest common denominator (which usually means Outlook).
Modern HTML uses special elements such as the ‘div’ tag or more modern ‘section’ tags (and others) to lay out text and images. Outlook will have none of it. For Outlook, everything must be a table. This is a clunky method of laying out a newsletter, one we abandoned at the end of the last millennium, yet because of Microsoft we are stuck in the past with respect to html email.
Images
To include images in an HTML formatted email, there are three options:
Embed the image data in the HTML code
Include the image as a MIME attachment
Link to an image stored on an external server
The ability to embed the data of an image inside of the html code itself (representing the binary data with alpha-numerical characters using base64 encoding) is relatively new. The advantage is that the email is a self-contained whole. By embedding the image data in the email itself, there is no need for the email client to go out and fetch the image from a server. Not only is it faster not to do so, usually email clients will hesitate to fetch external images due to privacy concerns, and require the user to click a button — “fetch images” or something like that — before the images are shown.
My previous newsletter template used the embedded image data method, but, you guessed it, Microsoft Outlook has no support for embedded image data. As a work-around, I added a link to a web version of the newsletter for those missing out. At first I though it was only users of Outlook (there aren’t that many any more), but as it turns out gmail also does not show embedded images.
So the second best is to include the image as a MIME attachment, and refer to the attachment from the HTML code. This is known as CID (Content-ID ) attachments, and it’s ancient. Again, it has the advantage that the email is a self-contained whole, without the need to pull in data from a server on the internet. Support for it is flaky though, especially in webmail clients.
So that leaves the least favourable option — in terms of privacy and complexity — as the most favourable — in terms of compatibility: put the images on a server and include the link to the images in the email’s HTML code. And that is what I ended up doing for the new template. It’s a bit clunky. One needs a web server to host the images and the images have to be uploaded before sending out the email. A little cumbersome, but there are a few things I did to make it a bit easier on myself. Keep reading to find out what those are.
Composing the newsletter
Now that I have my email format settled (HTML email, table layout and external images) I wanted to make it easier to compose the email. What I did before was to have a template email made up of hand-crafted HTML set up in Thunderbird. I’d create a new message based on that template, replacing the text and the images — which I had to manually convert to a small enough resolution not to make the email size too big — with those of the new newsletter. Needless to say, this didn’t always work out for the best. I might forget to update a link or mess up the layout. Having to manually go in and change the HTML code is cumbersome at best.
To give you an idea of what that looks like, here’s an excerpt from my old template:
<div class= "container" id= "personal" >
<h1><img moz-do-not-send= "false"
src= "data:image/jpeg;filename=snowtree_small.jpg;base64,/9j/4AAQSkZJRgABAQEA-thousands-of-random-characters-ViHaBrx0H/9k="
alt= "Snow-covered tree in a snow-covered garden" style= "width: 43%; float: right; padding-left: 1em;"
class= "" > Personal update<br>
</h1>
<p> I'm excited and can't wait to release <a moz-do-not-send= "true"
href= "https://koenmartens.nl/pages/ein-1-the-ein-particle.html?mtm_campaign=ein&mtm_kwd=nl230303" > The
Ein Particle</a> . I've received a first draft of the cover this week, and it's almost perfect. I'm now
eagerly awaiting a second draft with a few tweaks. Can't wait to show it to you all!</p>
<p> Meanwhile, I've sharpened up <a moz-do-not-send= "true"
href= "https://koenmartens.nl/pages/ein-1-the-ein-particle.html?mtm_campaign=ein&mtm_kwd=nl230303" > the
blurb</a> a bit after consulting with a blurb writer. What do you think, does it fancy your imagination?</p>
<p> And last but not least, the manuscript is now out with my beta readers. They are the first to read it apart
from myself and my developmental editor, so yeah, I'm pretty curious to know what they think of it!</p>
</div>
Of course, Thunderbird (and most other email clients) provide a what-you-see-is-what-you-get (WYSIWYG ) interface to edit emails in a visual way. Getting the HTML just right to be able to be shown correctly across different clients is paramount though and it’s not a given that the output of the WYSIWYG editor is going to show consistently in all those clients.
There are services out there that provide WYSIWYG interfaces to edit HTML that are more suited for email. These editors will limit your choices and generate HTML that does render right on all of the popular mail clients. But that means paying for a service, both through a monthly or yearly fee and by giving them all data about my subscribers. Not an option.
And, in fact, I don’t need WYSIWYG anyway. I’m used to writing text using markdown , which is a lightweight mark-up language. I decided to write a Python script that takes a simple JSON HTML from the markdown files into a bunch of HTML templates to generate the final newsletter content.
To make all that a bit more tangible, here’s an example of such a configuration file:
{
"date" : "20230503" ,
"time" : "2311" ,
"subject" : "This is a test of the new newsletter format" ,
"items" : [
{
"type" : "2column" ,
"text" : "personal.md" ,
"image" : "personal.jpg" ,
"image_alt" : "Personal pic"
},
{
"type" : "2column" ,
"text" : "call_to_action.md" ,
"image" : "call_to_action.jpg" ,
"image_alt" : "Lorem ipsum" ,
"link" : {"text" : "Vote now!" , "url" : "https://koenmartens.nl/pages/vote.html" }
},
{
"type" : "1column" ,
"text" : "group.md" ,
"image" : "group.jpg" ,
"image_alt" : "Group promo alt" ,
"link" : {"text" : "Grab your copies now!" , "url" : "https://example.com/promo" }
},
{
"type" : "2column" ,
"text" : "book_tip_1.md" ,
"image" : "book_tip_1.jpg" ,
"image_alt" : "Lorem ipsum" ,
"link" : {"text" : "Get it for free!" , "url" : "https://storyoriginapp/foo/bar" }
},
{
"type" : "2column" ,
"text" : "book_tip_2.md" ,
"image" : "book_tip_2.jpg" ,
"image_alt" : "Lorem ipsum" ,
"link" : {"text" : "Get it for free!" , "url" : "https://storyoriginapp/boo/baz" }
},
{
"text" : "music_tip.md" ,
"image" : "music_tip.jpg" ,
"image_alt" : "Album cover" ,
"links" : [
{"text" : "Bandcamp" , "url" : "https://williamparker.bandcamp.com/album/mayan-space-station" },
{"text" : "Spotify" , "url" : "https://open.spotify.com/album/07HfHVDOF8HTlv0TdmWQQQ" },
{"text" : "Other" , "url" : "http://www.example.com/" }
]
}
]
}
And here is an example markdown file:
# Personal Update
Cras lectus felis, elementum quis ornare a, iaculis nec risus. Vestibulum arcu est, lacinia sit amet dictum id, efficitur sit amet dui. Nunc ac odio risus. Ut feugiat a felis non blandit. Etiam gravida rhoncus est sit amet condimentum. Interdum et malesuada fames ac ante ipsum primis in faucibus. In hac habitasse platea dictumst. Phasellus tincidunt scelerisque ligula nec maximus. Aliquam viverra dolor eu hendrerit auctor. Donec lobortis eget nunc a lacinia.
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis sed rutrum quam, id pharetra justo. Suspendisse et massa quis nulla hendrerit accumsan. Vivamus in dolor ante. Aenean vel erat eget metus rhoncus dictum. Sed vel mauris a orc i feugiat rutrum. Pellentesque et metus nec ex rhoncus accumsan. Maecenas quis dui turpis.
Note the # before Personal Update? That’s a markdown character, and it indicates that the text following it should be rendered as a heading. There are lots more options and if you’re curious as to what those are, have a look at this https://www.markdownguide.org/cheat-sheet/ .
While JSON and markdown may come across as still being a bit technical, to a seasoned programmer such as myself these formats are second nature, and they are so much more intuitive compared to writing HTML .
Distributing the newsletter
So, I’ve decided on my format, wrote a script to generate the newsletter content as HTML from a much simpler input specification and upload the images. But where to upload the images? And how to distribute the content to my subscribers?
I’ve been using GNU Mailman
Unsubscribing from a Mailman mailing list, for example, is a bit complicated. One has to either go to Mailman’s web interface, log in and click the unsubscribe button, or send an email message to a specific email address (usually of the form newslettername-unsubscribe@example.com). The first option simply doesn’t work. People sign up to the mailing list through a variety of services (more on that later), and they won’t actually know what their password is. As soon as they decide they’ve had enough of my newsletter, they have a choice: either go through the hoops of sending that unsubscribe email or just hit the ‘junk email’ button in their mail client. They’ll probably do the latter, which means my newsletter will be registered as spam (even though they did sign up willingly), which is not ideal.
Looking for a replacement for Mailman, having a simple way to add an ‘unsubscribe’ link right at the start of the newsletter was on top of the list of requirements. As important was that I wanted something open source that I could host myself. Enter Listmonk , a free, open source and self-hosted newsletter and mailing list manager.
It has some other features that Mailman is lacking that make it suitable for newsletters. For example — and this is where things might get a bit controversial — it can insert personalised tracking info in the email to be able to show me how many people open the newsletter (and potentially who don’t, so that I can remove them from the subscriber list). Another useful feature is that when I include links to my website or other places in the newsletter, Listmonk can keep track of which of the links are being clicked, so that I know what is of interest to my subscribers and what isn’t.
As for the controversy: I’m generally not very fond of tracking technology in newsletters. I value my privacy, and the majority of newsletters use one of the big services that collect data across disparate newsletters to create detailed user profiles, and potentially sell those to ad networks or other nefarious organisations.
Still, I can see the value of having the information. As I said, for one it can help me optimise my content, so that I don’t bore my subscribers with subjects they are not interested in. What I’m doing here is using my own personal Listmonk server, hosted on a dedicated server in Germany, out of reach of data-grabbing nations and corporations. There is only one use of the data, and that is for me to produce my newsletter. There are no other companies that ever get access. To me, subscribing to a newsletter that ran their open-rate and click-rate tracking that way would feel good.
Interestingly, Listmonk also provides a place to upload the images that go into the newsletter. While I could have posted them on my regular website , there is an elegance in using the Listmonk server. There will be a tracking pixel anyway, hosted on that same server. Having the images originate from the same server reduces the amount of servers that need to be contacted by the email client. In fact, when I look at some of the newsletters I receive, I’m being asked to approve 5 to 10 obscure domains, ranging from google servers to dodgy content distribution networks, and it always gives me the creeps seeing how many servers from big multinationals get to see that I opened the newsletter.
What’s also nice is that Listmonk can automatically make available a web version of the newsletter. Where before I had to copy the HTML I crafted from the template email message in Thunderbird into my website, with Listmonk this is handled automatically and the link to the web version can be automatically inserted, as well as that aforementioned personalised unsubscribe link.
And last but not least, there is an API that allows my Python scripts to programmatically interact with Listmonk to automate many tasks. This is how my content generator script uploads the images, but it also allows me to automatically import new subscribers.
Subscription management
There are two ways people sign up to my newsletter: via a form on my website or via one of the book distribution websites.
The former was easily changed from using the Mailman API to using the Listmonk API . Sorted.
The latter requires a bit more work. I’m using StoryOrigin and BookFunnel to distribute a so-called ‘reader magnet’. This is a free story, downloadable in exchange for signing up to my mailing list. Both services collect the email addresses and distribute the ebook. I’ve made some further scripts to import the subscriptions on those sites and feed them back into the Listmonk server (yay for the API that allows me to do this).
Welcome campaign
One last feature of a modern newsletter is that when someone signs up, you can send them one or more emails to welcome them to the mailing list. I’ve been sending out three welcome emails. The first as a thank you for signing up, including the links to the free story (and some extras), just in case they missed them (or signed up via my own website). The second asks if they liked the story (and if they maybe would be inclined to leave a review) and the third has content that varies depending on where I am with my current project. It might, for example, ask people to sign up for an Advanced Review Copy of my next book, or perhaps ask them to join my team of beta readers.
So far, I’ve been keeping a separate database that keeps track of who signed up when and what welcome emails they have received. Twice a week, I run that script over the database. It spits out a list of email addresses for each of the welcome campaigns, and I then go to Thunderbird again to send out the welcome emails from the template emails.
With Listmonk, I can get rid of the separate database though, and store that information directly in Listmonk. For each subscriber, Listmonk keeps a bit of custom data that can contain anything I want. By accessing the subscriber database through the API , my welcome campaign script can easily access and update the information about what welcome message was sent to whom.
Future work
While the current, new, newsletter sending infrastructure is already a great improvement over what I had before, there still is room for improvement. I would like to automate the sending of the welcome campaign, to cut out the manual (and therefore error-prone) steps I now do twice a week. It will be a relatively simple extension of the welcome campaign script. I will have to move the welcome email templates over to the Listmonk server, and then I can use the API to trigger sending the welcome emails based on those templates from the script.
Another automation that would save me some manual clicks is to automatically upload the content generated by my content generator script to the Listmonk server. While the images are already automatically uploaded, the generated HTML is not. I still need to manually create a new newsletter edition in Listmonk and copy over the HTML . Again, through the API , this can be easily automated.
Conclusion
So, dear reader, there you have it. This is how I send out my newsletters. Without resorting to big tech, without paying for services, without relinquishing control.
Figuring out the HTML template was a bit of a drag, but once I had a template that worked across most email clients, I could hide that behind a script and stop faffing about with HTML but write newsletter content in straight-forward markdown, brought together with links and images in a simple JSON configuration file.
If you are interested in the code, some of it is already up on my public gitlab server, and more will follow. Here’s an overview:
They are probably of limited use to anyone else, and some of it is a bit messy honestly. I doubt anyone else will go to the trouble of setting all this up themselves when they can just pay for a service to do so. But it gives me a warm and fuzzy feeling knowing that all of my newsletter processes run on my own servers. Indie author all the way!
