I'm Learning! I'm really Learning!
I had originally written a really dumb pun for this title, so be happy with this title, okay?
I spent a lot of time in December not writing code for Just Read Comics. With the holiday malaise of Thanksgiving, going to PAX Unplugged the following weekend and then just being tossed around by the holiday season one way or another, I just didn’t have time.
Or so I thought.
In the week between Christmas and New Year’s, I found myself stuck on a the Lambda Problem: How do I freaking get past this? For some reason, I couldn't just let go of the how I was going to do the automate the "scanning "part of this project and it was annoying me. Annoying me to the point that it was really putting me off working on things without a clear path forward. I spent so many days trying to get Lambdas to work the way that I needed them to with the project setup and deployment path I had, and I just could not get them to work for me1.
So, I sort of lied to myself. I said, “I'm going to just move all of the scanner code that I already had written and working in my current node-express app into something not Lambdas, but into a new node-express app.” The idea of being that I know that I needed to separate all of the logic for scanning into a separate project of some kind, because I didn't want my web server, which is the home for justreadcomics.com, to necessarily be held up by background processes.
I know enough about node to know that JavaScript doesn't really do well with multi-threading (something I definitely learned about in college but have since forgotten the details of, along with all sorts of other things that would probably be good to know and to make me sound smart right now), so having both my web server, and a bunch of extra background processes living in one server isn’t a great idea. I know enough that JavaScript is not great when it handles a lot of stuff at once. In my mind, JavaScript is meant to go fast and go fast one time. So I decided to move all my scanning stuff to a new server, and that new server is going to get called on a schedule somehow--I'll figure the scheduling stuff out in the future that's a future Mike problem2.
After the process of coming up with an avoid-the-problem-solution in the shower / talking to myself / dictating to myself in a notepad, one day, I was off. I started moving all the logic that I had set up in this potential lambda package into a new express package.
But the thing about having multiple node-express servers interacting with a lot of the related pieces of a project sitting side-by-side in a repository is that you're probably going to want to use a lot of the same code.
I was already dabbling in the question of how do I effectively share code between Javascript packages? I knew that there were solutions out there. My inexperienced brain immediately went to the thought of, “I guess I'm gonna have to have three or four different repositories that own all of these different pieces of code that I can import them as node modules and since everything is JavaScript, I'm gonna have to handle exporting and publishing and all sorts of dependency business…”
Thank the code-gods, that’s not how you need to do it at all.
Enter Lerna
In my first post, I hinted at how I was sharing types between all of my JavaScript packages in my monorepo. That was a little bit of a lie, because in my mind, I was writing that post with the expected eventuality that I would figure out a way to share those types without it being a convoluted mess.
Now, you may recall that I was already using Lerna to spin up multiple packages at one time because I'm a very lazy dev who didn't want to have to type in multiple commands into multiple terminal windows. That was great and all, but one of the big advantages of using Lerna is that you can create libraries that are shareable between those the packages of your monorepo without having to do the insane business I was thinking I needed to do with multiple repositories and blah blah blah.
It’s so ridiculous because it was right there in front of me and I decided to just completely ignore that.
However, I had discovered there was one little snag. While Lerner can do shared component libraries. It doesn't do it without using it’s “parent” company’s integration… Nx.
Mike is 2 years late to the Party
So while I had Lerna to set up, and it was great, there was this little thing that I had ignored:
So while Lerna isn’t deprecated, using shared library files the way I wanted to, required using Nx. Which meant… enabling one flag. Which is actually pretty great. I could still use the same build commands all while leveraging the build functions I had already created.
The next step was getting my shared code libraries setup:
The node-specific library for the two node servers (including Database Connections, logging utilities, and some Amazon S3 utilities I had written)
The type/environment library for all three packages (backend and frontend) to use
Nx: The New Hotness
“You need to create a new shared package really fast?” Nx as an automated tool to help you do that in a handful of command line responses.
“You need to migrate your package to a new building tool?” And can do that in one automated command.
This thing does a lot and then some; a lot I’m not even using and that’s pretty awesome, if you ask me. If I was a better dev, this might be even more enlightening.
I'm not trying to say that this tool is the end all be all of helper tools. There's a bunch of outdated documentation that some automated tools don’t work well (or I’m not using them correctly), and there's people screaming about various different problems on their GitHub. Like all open source projects, it has its problems, but 95% of everything I've dealt with has been super breezy.
While I was working on this new scanning package in the monorepo, I needed to share a ton of node code between the scanning express server and the web express server. After I converted my package to use Nx, I started digging into how to share code between those packages.
As it turns out, doing this shared package business is actually hilariously simple using JavaScript, Nx, and npm.
You create a new folder
You create a package.json with a name, description, and an output file name
You drop the code you want to share into the folder and make sure the files are exporting the functions/classes/etc.
You import that new package based on the name you set up in the package
And… it just kind of works!
(That’s not entirely the setup, but the documentation is pretty much that)
If my calculations are correct, when I go to actually build things and deploy them… It'll just work.
That's incredible, and I'm pretty pleased with it.
So Where Are We With Progress?
Well, I was really hoping you wouldn't ask that.
(psst. Mike you wrote this)
OK, shut up.
Yeah… at this point, I don't know how to measure progress right now. There's a lot that I still need to do to get this into a semi-stable state that feels… independent. It runs on it’s own, but there’s a baby-deer-just-born feeling to it.
I mean, right now I have an index of ~8900 series that I can scan externally, view on the public side, and link out to a their related services. There’s an admin interface that will let me manage all of the series, trigger scans manually, and (most recently) review scan details as they come in.
The problem is: Every time I sit down, I end up tweaking some feature I’ve already built or I end up cleaning up something not to my liking. Now that isn't to say that that isn't helpful, because it is. It makes my life easier when I'm testing things and following user (and admin) journeys through the site, but it hasn't been a lot of progress in terms of feature development.
That being said, things work pretty all right. The admin is really nice. The scanning functions make me giddy (when they work as expected). The public front end could definitely use some work, but for what it's trying to do right right now, it's serviceable.
But here’s where I think I need to start adding some scaffolding to my project of stacked cards. I want to come up with a roadmap that I can share here. Doing so might strain some of my forward thinking muscles, but I think it’s achievable. I have a Trello board that's full of features and things that I want to build but no paths forward, just a bunch of notes, and I do think a little bit of structure is needed.
Hopefully a roadmap will do that.
Next Time
I’m going to flip a coin:
Lambdas, Not For Me
2024 for Just Read Comics: A Roadmap
I've probably got an entire post about what did and didn't work for me in the chamber somewhere, but ultimately I had to get over this problem.