There is a magical beast I’ve been working on in a span of multiple projects, together with some beautiful minds, and we’ve given it a name: the 3-Layer Structure.
In short, a 3-Layer Structure allows small development teams to build a API-centric, scalable and above all secure project. All you need is 3 VPS servers * and all open source software. The beauty lies in the minimal cost and effort, and the proof is in the ease of explaining it.
(* goes from $5 a pop at Digital Ocean – affiliate link, save $10 bucks)
Layer 1: App Cloud.
Layer 2: API
Your API is the central hub. What you need here is a super light router to dispatch all requests and responses – both endpoints as authentication – to a Message Queue (foreground enabled). You might also want to consider to store the API Docs and Authentication html templates on this machine, to prevent fragmentation.
Never connect your API to your DB! Your API logic should be deployed from a repo, so you can disable ftp and any other access to your machine, because your API will be the favorite address for intrusion attempts. Automated attacks won’t do dramatic harm because of the logic segmentation, but still, let’s keep security in simplicity.
Layer 3: Business Logic (aka Workers)
You can scale this layer in any direction you want, tailored to your project. You can have 1 little machine running some worker nodes, or you can have multiple VPS’s, paired with a cluster of DB machines for central data storage, in any flavour, whatever you like.
Your worker connects to the MQ server on the API, handles the job, and sends the response back. Since the API doesn’t know anything about the workers (it only cares about his Message Queue), there is no way for voyeurs to find out where your business critical logic and DB are running. Thus, safety by simplicity.
In short, you provide unlimited, easy to maintain scalability on frontend and backend level, with a light-weight gatekeeper in the middle.
For Python lovers, the former Tick.ee Project (now open source) provides a rough reference.
We’ve been working by this architecture since 2011 (before MQ was common, “Cloud” was not yet a marketing term and the default api response was xml), and never looked back.