CS636 Class 23 Scaling up and Microservice Architecture

Notes on Advanced Web Apps, New Trends, etc.

We have been studying the most basic current setup for a Java web app:

The single database may be very powerful, handling up to 100 TPS or somewhat higher. That's 360,000 TP/hour, a huge load. Ref: TPC non-clustered performance data. For high performance, the database server is on a different machine than the application servers.

The single Java executable is not a problem for small or medium-sized sites. All the long-term data is in the database, which usually doesn't need to change for a software upgrade. The only users who could be affected are those with long-running sessions. Web containers are required to maintain session objects across redeployments of the web app (Servlet spec, sec. 10.8). However, objects hanging off the session object may be no longer recognized if their classes were edited (an argument for using just numeric ids, Strings, etc. in session objects).

Scaling up

First use a decent multiprocessor for the app server, and another one for the DB server.

Will find that the app server gets overloaded before the DB server (assuming small DB requests of course).

So scale up the app server first.  Eventually one system can't handle the load.

So replicate the app servers up to 25 ways (requires a request router up front.) Get a 40-way (or more) multiprocessor for the DB, with lots of memory. Note that a DB caches hot data in its database cache, for fast access, so get enough memory for all important database data.

This config is fully supported by our software setup, because of the independence of all the request environments.

Only when the biggest system can’t run the needed DB that you replicate the DB, unless the work can be perfectly partitioned in a way the request router can understand. 

Luckily, when your site is this big, you should have enough money to hire real experts!

Scaling up to seriously large apps

There are two directions for scaling up, system size or software complexity.  For 1000 TPS, you need to go to machine clusters for the database, and caching.

Code complexity: more code means more developers. For more than 75 developers (one estimate), the monolithic code becomes a problem. One little change by group A of say 8 groups means rebuild and redeployment. At that point, the system needs to be re-architected into multiple executables (components) that cooperate and provide not only services to the world, but also services to each other.  For example, Netflix had a monolithic code for years, and then was broken up into components with services, known as the "microservice architecture".  Here each component has exclusive use of its part of the database data, so can be redeployed with database changes if necessary. The other components can access the data via the services offered by that component.

To illustrate the basics of microservice architecture, the pa2 database has been refactored to allow it to be the back end of two separate cooperating webapps.

From the pa2 assignment: Note that the database has changed slightly to accommodate use of one database for "sales" (users and invoices) and another for the "catalog" (products, downloads, etc.). No foreign key can bridge the gap between the databases, so the user associated with a download no longer has a foreign key, and similarly, neither has the product associated with a lineitem. Instead of ids with foreign keys, the username is used in a download, and the product code is used in a lineitem.

The service API has been split into CatalogServiceAPI and SalesServiceAPI, one for each database. You can treat them together as one unit or try to keep the code accessing the two databases apart. They are implemented so that the catalog service only uses its DAOs (users, invoices, lineitems), and the catalog service only its DAOs (product, track, download). The sales service code calls the catalog service API to get information on a certain product, rather than the product DAO.

With these changes, we can have separate "Sales" and "Catalog" webapps, each with its own servlet and service API. This could be useful to save on database costs: we could use Oracle (expensive) for the sales webapp, for safer money-handling, and mysql (free) for the catalog. Or it could be used to stretch the database resources if the database server (already expanded to 64 CPUs and 100GB of memory, say) has become overloaded. Or it could be used to allow two development groups to work and deploy new code independently, or almost independently (they do call each other through the service APIs).

Microservice architecture Wikipedia article

Netflix video: Mastering Chaos

Start watching this video about 5 minutes from the start to miss a long intro. Netflix was a pioneer in the microservices area, and they are quite proud of it.

Another video with less negative treatment of the monolithic approach: Split That Monolith Message: start with a monolith, monitor it to know when to split it up.

Although there is little discussion of layering in microservice articles, there is a strong notion of the application's API, the most important layer division in our systems. That's what the "service" part of the name means. And services here mean stateless services, although that is usually assumed, not stated. The scope of transactions is limited to a single service call, as in our setup, but this one call is accessing only part of the persistent data, so is less powerful. In some cases, data may become inconsistent between the databases, at least for a while. There is an idea of "eventual consistency" to help out.  Changes in one area of data may notify other components about the change, so the other component can eventually fix up its part. Obviously this increases the complexity of the code. 

Advanced Domain Object Design

There is a whole theory of this. The first book to read is Domain Driven Design (or DDD for short, 2004) by Eric Evans. He explains how to identify and define entities and value objects, two kinds of domain objects. Then object graphs that hang together to describe something are called aggregates. Then aggregates can be grouped into a "bounded context" which has a consistent model and language and is usually supported by a certain development team. The idea of bounded context is used for components in the microservice architecture, a later development.

Then read Implementing Domain-Driven Design, by Vaughn Vernon, 2013, to see the more current thought on system design using DDD. The newer architecture is called Hexagon or Ports and Adapters. As in our systems, the domain objects are simple POJOs, unaware of how they are persisted, etc. The DAO code is now relegated to "adapters". On close inspection, this DB adapter for a component is called as needed, so is in effect a lower layer. In architecture diagrams, you usually see a hexagon, representing the various adapters as faces (not necessarily 6 of them, by the way), and inside it, another hexagon around the core, which contains the domain code. The inner hexagon represents the service API, or set of services. Following DDD, there are aggregates and bounded contexts. The transactions are expected to change only one aggregate at a time, so again there can be inconsistency, and events are used to notify other aggregates as necessary to obtain eventual consistency. A transaction may read related aggregates, of course.

Clearly DDD and microservice architecture can be combined, based on decomposition of the whole app into bounded contexts.

Example of Music Project

Aggregates: Product with Tracks, Invoice with items, User

Rule from IDDD: Preferably, don't use object reference from one aggregate to another. This saves memory in the server as less data is dragged around. 

What about Downloads--are they details of Tracks or Users or separate? There are no object refs from Track or User to bind them in. So perhaps best to consider them as a separate aggregate. But then the ref from Download to Track breaks another rule of IDD that refs (even ids) from another aggregate should only point to the root object of an aggregate. So that argues for attaching the Downloads to the Track as details, and replacing the User ref with a user id or email, as we have done in pa2. This also aligns with the use of Downloads in the app: tracking popularity of songs, not activity of a certain user.

What about Cart? It's not persistent, does that matter? No, we don't need to attach the whole Cart to another domain object, so it doesn't matter that it has no persistent id. The CartItems in the Cart are also not referenced from other domain objects. The CartItem references its Products via product_id, the proper aggregate root id. At checkout time, the CartItems are used to create the persistent LineItems that belong to the Invoice object, an aggregate.

Using product_ids instead of Product refs in CartItem can save quite a bit of memory, and this memory is longer term than usual per-request domain objects, because the Cart is a session variable.  It isn't just the Product object hanging off the CartItem that uses memory, but also all its Track objects.

Advantage: Smaller object graphs coming from DAO finders.

Rule from IDDD: change only one (persistent) aggregate in a service call. This involves transaction scope. If you must change two aggregates, use two disjoint  transactions, one for each (for example, by calling through the service API of a separate database).

Mutator service calls:

processInvoice: only changes an Invoice

addDownload(userid, Track) --changes only Downloads

addItemToCart, etc.--changes no persistent objects

checkout(Cart, userid)-- changes only Invoices, but does need to lookup Product details by product_id, so needs multiple transactions if using two databases. Use calls to the Catalog service API to fill in Product info in the new LineItems and the product price needed to calculate the invoiceTotal before doing the Invoice insert transaction.

registerUser(String, String, String)-- changes only Users

We have been following this rule without trying, in the monolithic case, and have only one place to fix up in the two-database case.