#vss365 today, July 2019

Since I put #vss365 today online on January 14, it has been elevated from a simple side-project to my current primary personal programming project (whoops alliteration strikes again). While there are a few other projects I am working on, I have spent the most time on it because there is tremendous potential and has seen an amount of growth I did not expect. It's not been smooth sailing since launch, and I have some cool things in the pipeline. I think it's all really cool and want to share it, so here we goooooooo!!!

Site Growth

To start, let's look at a graph.

This is a graph showing the daily amount of emails sent out from the site, via the MailJet Send API. As can be quickly inferred by the gray line, there's an upward trend of emails sent. This is something I hoped would happen (because I wanted people to find and make use of it) but the speed of increase has been unexpected. On April 17, 13 emails were sent. By July 17, that number had increased to 71 emails, nearly 5.5 times the amount within a three month span. That's crazy cool and absolutely humbling. Naturally, as awareness has grown and people keep subscribing, I've been working on stabling the app to be more reliable because, naturally, it wasn't fully there when it launched. As you can see, there's a number of daily spikes where 2-3x the normal amount were sent out. These occurred for a variety of reasons:

1. The prompt finder script, used to fetch the daily prompt tweet and process it, repeatedly failed:
   * It didn't take into account what would happen if we already had a prompt recorded for that calendar day but the prompt was for the next day
   * It detected multiple prompts a day when it shouldn't have due to faulty identification methods
   * It picked up tweets that were not from the writer
   * It failed to pick up at word at all but still sent out emails
   * It failed to pick up any prompt and did not send out any emails
   * Timezone translation issues causes tweet dates to be incorrectly evaluated or recorded
   * A lurking bug in HTML santization caused it to detect #x27 as a hashtag instead of the HTML entity for an apostrophe
   * Retweeted quoted tweets were not corrected filtered out (GitHub issue)
2. The cron job that runs the prompt finder script was incorrectly set
3. The writer did not hashtag the prompt word
4. The email template was broken and did not render the information correctly
5. The writer used additional hashtags that moved the prompt word, causing the wrong hashtag to be selected

All of these issues required a manual intervention to send out the subscription emails. The issues were confounded by my lack of tooling to better handle them. Early on I had to boot my laptop, quickly manually add the tweet to the database, trigger the emails from my computer, then push to the server, all while trying to get ready to leave for class on time. I finally wrote a set of scripts I can manually run to help resolve the most commonly recurring issues, but I need to go further. Monthly writer rollover is a massive pain because, while I can often find the writer ahead of them, I do not know when they will be posting the prompt (and what timezone they are in), nor in what order the hashtags will appear, nor if they know that it has been become customary to include the #prompt hashtag to indicate the prompt tweet (which I use to identify it). Lately, the most common source of email spikes occur at the start of a month.

Another thing to note in the graph are a complete lack of emails being sent in late June. While I knew it happened, I had no idea why, and seeing as my laptop died a week before, I had no way to correct it until July. Even then, I was using a temporary development environment which took me longer to find a fix. As it turns out, the MailJet Send API can only send out 50 emails in a single batch. So upon reaching 51 email subscriptions, the prompt finder script failed to send them out (although it still recorded to the database, as that happens first).

The last thing I'll point out about the graph is the number of emails sent out a day is not consistent. Despite increasing in count overall, the daily number seems to fluctuate. This is due to MailJet's operations and how they send out emails. They have a safety/security measure called "Retrying" emails, described as "Emails rejected by the recipient’s server. We will retry sending these emails for up to 3 days. If it’s still rejected it will become soft-bounced." This issue only seems to happen with @yahoo email address (I've seen both .com and .co.uk address have this issue). It seems to be sporadic but happens on a regular basis. I don't know why this is happening nor what effect it is having on delivery. Seeing as MailJet is responsible for actually sending out the emails, it may not be resolved without switching to a different email provider. That may happen sooner than expected, as MailJet only sends up to 200 emails a day. Seeing as the site gained 58 new subscribers in three months, if that rate of growth continues or gets faster, I'll be hitting that limit in roughly six months. Time to start researching new providers. 😅

Twitter Visibility

While I have posted a link to the site the most often on Twitter, others have  been tweeting it too. I've seen multiple people replying with the URL  when people ask what #vss365 is and/or how to find the prompt word. This  is really neat. It means people are finding it, considering it useful, and deem it worth sharing. 😊

TechFest 2019

The next thing I want to share occurred on April 19. I had the privilege to be invited to be a student presenter at an IT/CS event called TechFest, hosted by my university. The event itself was similar to a tabling event mixed with a job fair. First, we set up a project of our choice and demoed it to employers who walked up. After lunch, it was flipped and we talked to the employers, but more informally than a full-fledged fair. Naturally, I chose to demo #vss365 today. While I failed to take any pictures during the event itself, the event itself was great and I got good feedback from those present.

As part of the event requirements, I had to bring a tri-fold poster about my project. Except my project is a website. So I did the next best thing: I put the site on the board. After a visual revamp thanks to my brother, we took a screenshot of the site, printed it out on lots of 8.5" x 11" paper, then used spray adhesive to glue the pages to the board. While it was certainly time-consuming and a lot of effort, it came out nicely. I also made a custom build of the app for the event for live demonstrations that I pulled up on a tablet in front of the board. Naturally, I took a picture of the completed board with tablet to scale. The entire thing was an awesome opportunity and great experience.

Prompt Archive

Lately, I've been working on building up an archive of all prompts up from #vss365 day one up to current day. There's been a lot of technical challenges I've had to work out due to assumptions I made about the game's current (stabilized) state, but I've made it work. Expect more on this soon.😉

Future Plans

While I have implemented a lot in the last seven months, I have lots more plans, of course. My task list is now being tracked with GitHub Issues. However, it is incomplete. I have more things I want to do but haven't written a ticket because I've not had time or they can only happen if other things happen. 🤣 For the last section, I'm going to highlight my top plans for the site's future. While there's no deadline for any of this, I would like to see the majority of it completed by the one year anniversary of the site.

I recently implemented a dark theme on the site that may or may not become the default color set (retaining the option to switch to lighter colors). However, there seem to be some issues with my implementation, as the dark theme is not always applied on first visit. I need to keep researching and possibly implement it differently.

I want to start building out an administration panel in the near-future. I recently wrote one for a project at work, which has helped me think about how I want to write this admin panel. In essence, I want to take my site issue recovery scripts (the ones mentioned in the first section) that I use and turn them into a visual UI that I can access from anywhere, while adding in configuration settings to assist in the monthly rollover. I also want to be able to modify relevant site configurations and other maintenance areas, as well as features that are currently not possible. While it is an ambitious undertaking and will require working some of the internals to support changing stuff outside of code, I believe it is fully possible and will save me a ton of time and effort, as well as potentially open up the opportunity for others to help run it, if they ever comes up.

Finally, I need to figure out my infrastructure. As mentioned, I may need to start looking into a new email sending provider soon. I have considered running my own SMTP server for sending only but apparently it's kinda a pain. We'll see what happens. I also want to move the prompt finder script away from a cron job and into the app itself, making it more standalone. I would like to automate deployment of the app's code to the server. I already have some ideas for this in the works. Last but not least, it would be nice to use the Twitter Streaming API to pull the prompt tweet in real-time. This would remove the need for a re-occurring task at all and resolve the date/time(zone) issues. I was using it when initially developing the site, and while it worked on my laptop, it was DOA when executed on the server with no indication of the problem.

That's it for the July 2019 development updates on #vss365 today! 😀 Stay tuned for another post about that prompt archive sometime in the near future.😉