![[UpdateHeader.png]] # Overall Update Since my [[Routine - Algorithm Noodling|post]] about various algorithm approaches that could be used to solve the problem space of scheduling users, some good progress has been made around the overall functionality and architecture of Routine. I wanted to cover just a few areas of improvement made since the last post and how it's going. ### Bookmarks - [[Routine Update Part 1 - The Improvements]] - [[Routine Update Part 2 - Ze Metrics]] - [Routine Update Part 3 - Productization] (Coming Soon) - [Routine Update Part 4 - Use of AI] (Coming Soon) # Improvements ## UI Improvements I started this application using NextJS as covered by [[Routine - The scheduler with a new approach#Tech stack|my previous post]]. I quickly realized that NextJS seems to be highly opinionated and became an impediment to rapid prototyping unless you followed exactly the way NextJS and, ultimately, Vercel wanted, including using their tooling. While I'm sure part of that is my own hot take, I found others finding similar issues with NextJS. As one who has also used it professionally and equally became frustrated by the technology stack, I converted from NextJS to a Vite React SPA. Once I moved to the Vite React SPA, rapid prototyping became easier, and I was able to make some substantial progress with the front end, looking less like a prototype and more like an actual product. Here are some preview screenshots: ![[YourStudioScreenshot.png]] ![[ScheduleManagementScreenShot.png]] ![[ScheduleConfigScreenshot.png]] ## API Improvements Including a bunch of UI rework, I've been working on a handful of API's and backend. For a while, I was working with just in-memory endpoints, so when I restarted the app, it would reset. This was fine for rapid prototyping, but I found I needed to rearchitect the data layer a bit to make sure it would grow and be organized well for continued growth. I added a few different projects to mirror the classic .NET application from the API, Data Layer, Background Worker, and Shared Libraries. Obviously, there are tests for each of these as well. ## Data Storage The hardest part with rapid prototyping applications, while making it possibly a product, is making sure you can add the complexity to the application when needed, but not over-engineering it and making it more complicated than it needs to be. Striking that fine balance is continuously argued over by software engineers. In my experience, bringing too much technology too soon can cause more technical debt than needed. As with all applications, there was a need to store data. The obvious choice was databases, but databases come with a cost of upgrading them, adding new columns when the structure changes, as well as the overall cost of managing them. While at scale, this makes sense, I wanted something simpler. I ended up creating a basic JSON file storage system. It would store the data in a json file on disk and read from it, and store it in memory. Obviously, this has scaling issues, but in the simple form allows me to persist some data while allowing myself to manually edit the data in a basic format. This typically doesn't scale for production use cases, but I've created an abstraction layer so I can operate either in memory or using JSON just by injecting a flag. This also allows for much easier unit testing since I don't have to spin up a full DB just to test the API layer. ## Job Architecture One of the most interesting changes I made was to the actual generation of the schedule. Given the generation could take an indeterminate amount of time, it was risky to consider having the application do the work on the same thread that was servicing the API's. This wouldn't scale well and needed a fundamental change to support the growth. I opted for a background worker that would run separately from the API and would allow me to scale totally independently of the rest of the application. This would run in a separate pod and would read off the job file, listening to when a job would be added. It also spooled a set number of threads, ensuring thread safety so more than one thread didn't take the same job. I've done this in the past using queues, but this was different, using plain old file-based storage. This architecture allows me to scale the worker nodes independently based on the number of jobs or job queues that exist, and if none exist, scale them to 0. Furthermore, I can utilize this worker system to do other long-term job-based tasks. ## Deployment Improvements There are dozens of different ways to deploy pod-based architecture. From using Kubernetes to the dozens of different pod scaling cloud native deployments available across any number of cloud service providers. While I'm not quite at the stage where deploying this for a cloud service provider for product usage, I did want to ensure it runs in an environment other than locally on my machine. I work with Kubernetes daily, so I'm familiar with the technology stack, even though I would certainly consider it too much of a tool to run 3 different pods. While I might consider using a different service to host this when it becomes a product, I used what I have around. You can see the overall deployment architecture below. ![[ArchitectureScreenshot.png]] Stay tuned, more to come.