As I prepare Routine for a more production-ready environment, I had to do some research: As a Cloud and Infrastructure Engineer, putting something in the Cloud is not a new task for me. However, without the enterprise infrastructure at my fingertips, I needed to think much smaller with the same resilient ecosystem I'm so familiar in building. Given [[Routine Update Part 1 - The Improvements|the current architecture]] I needed something that would provide a database, worker, and API for Routine, with some ingress, CDN and some basic configurations, all being hooked up to CI/CD. I was being cost sensitive since this is a personal project of mine, and wary of over-engineering it too early on. ### Bookmarks - [[Routine Update Part 1 - The Improvements]] - [[Routine Update Part 2 - Moving to Production]] - [[Routine Update Part 3 - Ze Metrics]] (Coming Soon) - [[Routine Update Part 4 - Productization]] (Coming Soon) - [[Routine Update Part 5 - Use of AI]] (Coming Soon) # Where I started My local environment and staging server was all containerized. Being familiar with the process, and having a local Kubernetes cluster, I was able to configure a staging server with a CI pipeline, pushing an image to a GHCR. I had been working with this over the last few years as I built out the prototype and added features. I would update the image then see how it performed, then adding things like worker features, uploading CSV, and more. I was using both my homegrown Kubernetes lab and local deployments, testing this just fine. However, as I was getting more features filled out I was feeling more and more compelled to get ready for more involved testing with a more production-like setting. While I anticipated a deployment in staging would be close to production, there were a number of areas I expected would be different. # Different Cloud Options Hosting cloud services is not a new process. The industry has been doing it for decades, however it can be very opinionated. From direct Azure to managed services to the in between there are pros and cons. One option gives you all the bells and whistles but the complexity of each becomes overbearing. Other options take all the options away or nickel and dime you for anything more. I settled on 4 services: Control Plane, Railway, Render and Digital Ocean. I considered each to find out what would work best, balancing price, configuration, growth and overall features. ## Railway I'll be honest, [Railway](https://railway.com/) lured me in with its slick UI which felt polished and modern so I tried it first. Originally I had in my head this would cost about 10 dollars a month, low use, early deploy strategy and minimal architecture. The $5 per month made it an easy start. So I signed up, paid the 5 dollars and had Claude whip up an initial config and try to deploy. It did not take long to find out I needed the pro plan to deploy a container with a private registry and it made me wonder why. An industry standard method of deploying an app is paywall gated to a minimum of $20. Which among other things such as minimal Postgres DB control and backups made me realize this doesn't quite have the right pricing and features other services had with that price floor. Good for some, not for all. ## Render [Render](https://render.com/) was my next test. Signing up was straightforward even though it didn’t feel as pretty as Railway. Nonetheless I signed up, and started deploying apps. I was greeted with another realization. While it supported more items like CDN for front-end and didn't have a premium for containers it did have one thing that really made the cost not worth it. This was a price on your architecture. Have an API, worker, DB architecture? It’s $25 per service and you only get 512 MB of RAM and a shared CPU. While their pricing page hides that while making their service look practically free, it quickly adds up to $50-75/mo for less than useful specs so that approach was removed from the list. ## Control Plane [Control Plane](https://controlplane.com/) was a short contender. While I love the concept and idea and have worked with them in the past, I felt this fell a bit too much in the enterprise category of things, similar to Azure or AWS. I'll continue to keep it in mind for the future as a more enterprise backing is needed but it appeared too heavy weight for me at this stage. ## Digital Ocean [Digital Ocean](https://www.digitalocean.com/) ended up being the last one (and finalist) I tried. I was familiar with its systems since I've used them in the past and they have greatly expanded their offering since I've last used them. Moving from simple VMs to managed application services, it made everything I wanted to do within reach. It even offered services I wasn't aware I'd need until I started deploying Routine. ### The Process After a fairly painless configuration of the CI pipeline, I had a Postgres database, API, Worker and the front-end CDN all configured and working within an hour or so and managed by the CI pipeline. The networking to the database was all handled relatively trivially as well as the connection strings. Little things like connection secrets were all nicely handled without giving you an option to do it wrong. As I started testing the application and getting all the configuration set up, I realized one of the things I hadn't wanted to over-engineer early on was suddenly put front and center as I was testing the features on the deployment. There are a set of features that rely on a shared disk storage from the API to the worker as described [[Routine Update Part 1 - The Improvements|over in part 1]] which meant the API and the Worker needed to write and read documents uploaded. When I deployed it to staging I was writing to a shared volume, and obviously this isn't part of the Digital Ocean capability. The better approach was to write to an S3 bucket, and since this was just temporary storage while I processed the file uploaded, I created a quick implementation of persistent storage and added a short annotation that deployed an S3 bucket and within an evening I had replaced the local file storage with this approach. All the other configurations pretty much got set with very little additional effort. I set up some routes to the API, and to the CDN so it would route correctly and [Routine](https://app.myroutine.io) was now in production. ### The Cost All in all, there were a few points to this decision, beyond cost, though cost was a significant decider. I was looking short and longer term potential growth and I wanted something I felt I could grow with, without adding external services to mitigate various shortcomings. For instance, Railway and Render didn't natively support Postgres that would allow native backups and scaling. This was all left to the end user to configure and set up and while it was somewhat trivial to do, it felt like an extra step that, by contrast, Digital Ocean provided out of the box. Furthermore, additional disaster recovery considerations were already provided for with Digital Ocean whereas, again this would require additional configuration and cost. These all have a technical overhead and management that all adds up, either through hours spent configuring it, or hours spent if you overlooked it and had to deal with mitigations. While apples to apples comparison would not exactly be possible, given the above things I didn't configure, the Digital Ocean bill effectively boils down to this: |Product|Description|Monthly| |---|---|---| |App Platform|api (basic-xs)|$10.89| |App Platform|worker (basic-xs)|$10.89| |Database Clusters|Postgres primary (1 GB / 1 vCPU)|$14.11| |Database Clusters|Postgres storage (10 GiB)|$2.34| |Spaces|Object storage|$5.00| |Static Site|Front-end + CDN|$0.00| ||**Total**|**~$43.23/mo**| ![[routine_cost_breakdown.svg]] Could it be cheaper, maybe, but I think for what I'm looking for and what it gives me, I have what I need. ### Config The config was straightforward. Using some AI tools, it provided a configuration that was close to what was needed. The basic outline looks a bit like this. ![[DO_Configuration_Outline.png]] ### What a Deploy Looks Like As mentioned before, I wanted something that was close to push code and see it live, but still wanted a manual approval flow to production. The flow looks like the following: ```mermaid flowchart TD A([Write code]) --> B[Push code<br/><i>push to main</i>] subgraph CI [CircleCI pipeline] direction TB D[Build] --> E[Test] E --> F[Integration Test] F --> G[/Wait for approval/] G --> H{Approved?} H -->|No| X([Deploy halted]) H -->|Yes| I[Run update step<br/>publish to DigitalOcean] end B --> D I --> J([Live on App Platform]) classDef start fill:#dbeafe,stroke:#2563eb,stroke-width:2px,color:#0f172a; classDef ci fill:#ede9fe,stroke:#7c3aed,stroke-width:1.5px,color:#0f172a; classDef gate fill:#fef3c7,stroke:#f59e0b,stroke-width:2px,color:#0f172a; classDef deploy fill:#dcfce7,stroke:#16a34a,stroke-width:2px,color:#0f172a; classDef halt fill:#fee2e2,stroke:#dc2626,stroke-width:2px,color:#0f172a; class A,B start; class D,E,F ci; class G,H gate; class I,J deploy; class X halt; ``` This lets me deploy quickly enough without having to worry that code might make it to production before I'm ready for it. # The final outcome As I look back at what I originally wanted to do, coming back to Digital Ocean was refreshing for me. Its platform and offering has grown substantially from back when they offered cheap virtual machines while offering real-production class services that balance what is needed and what is offered. I'm sure Railway and Render both have their use cases for different people. As someone who has spent much of my career in the cloud, I found having these familiar features without it being abstracted was refreshing. I think overall there is a lot of value that the other services provided but as I considered growth of offering AI tooling and different services around that, Digital Ocean had that all ready to consume when needed. # What's Next One part of Routine I teased and haven't finished yet is monitoring. How do you know if you need to scale if you don't have metrics that tell you the service is almost at capacity? As I've built out each one of these services, I've meticulously crafted different metrics that would answer questions like, how many imports have happened, how many times has someone generated a schedule, how long did it take etc. All this data is being emitted and part of the effort now is sensibly ingesting these details to be displayed so decisions can be made on what's next both for the product and for its deployment.