This topic discusses scalability, but not in the common way, such as how to scale out the grid to some X number of JVMs. Instead, the perspective here discusses scalability in terms of operations, planning, and risk management. Although these factors are typically more important than traditional product scalability considerations, unfortunately they are frequently ignored. Building highly available systems requires both types of scalability considerations to deploy eXtreme Scale in a reliable deployment process.
Testing has verified that eXtreme Scale can scale out to over 1000 JVMs. Such testing encourages building applications to deploy single grids on large numbers of boxes. Although it is possible to do this, it is not recommended, for several reasons as follows.
Splitting the application grid into pods (units) is more reliable. A pod is a group of servers running a homogenous application stack. Pods can be of any size, but ideally they should consist of about 20 boxes. Rather than having 500 boxes in a single grid, you can have 25 pods of 20 boxes. A single version of an application stack should run on a given pod, but different pods may have their own versions of an application stack.
A pod is a conveniently sized deployment unit for testing. Instead of having hundreds of servers for testing, it is more practical to have 20 servers. In this case, you are still testing the same configuration as you would have in production. Production uses grids with a maximum size of 20 servers, constituting a pod. You can stress-test a single pod and determine its capacity, number of users, amount of data, and transaction throughput. This makes planning easier and follows the standard of having predictable scaling at predictable cost.
In different cases, the pod does not necessarily have to have 20 servers. The purpose of the pod size is for practical testing. The size of a pod should be small enough that if a pod encounters problems in production, the fraction of transactions affected is tolerable.
Ideally, any bug will have an impact on only a single pod. In the previous example, a bug would only have an impact on four percent of the application transactions rather than 100 percent. In addition, upgrades are easier because they can be rolled out one pod at a time. This simplifies the scenario so that if an upgrade to a pod creates problems, the user can switch that pod back to the prior level. Upgrades include any changes to the application, the application stack, or system updates. As much as possible, upgrades should only change a single element of the stack at a time to make problem diagnosis more precise.
To implement an environment with pods, you need a routing layer above the pods that is forwards and backwards compatible if pods get software upgrades. Also, you should create a directory that includes information about which pod has what data. (You can use another eXtreme Scale grid for this with a database behind it, preferably using the write-behind scenario.) This yields a two-tier solution. Tier 1 is the directory and is used to locate which pod handles a specific transaction. Tier 2 is composed of the pods themselves. When tier 1 identifies a pod, the setup routes each transaction to the correct server in the pod, which is usually the server holding the partition for the data used by the transaction. Optionally, you can also use a near cache on tier 1 to lower the impact associated with looking up the correct pod.
Using pods is slightly more complex than having a single grid, but the operational, testing, and reliability improvements make it a crucial part of scalability testing.