1.6 VOB Operations and the Oplog

This section describes the VOB database mechanism that supports replica synchronization. This information is not required to use MultiSite, but is helpful when you want to deepen your understanding of the error-recovery facilities described in Chapter 10, Troubleshooting MultiSite Operations.

Most changes made to a VOB are recorded as event records in the VOB database. Each event record describes a change. For a replicated VOB, operation log entries (oplogs) are created also. These entries store all the information required to replay the changes in another replica:

The exact kind and amount of information varies with the specific operation. For example, an oplog entry for the removal of a label has different, and less, information than an oplog entry for a checkout command.

NOTE: Oplog entries are created only for replicated VOBs. You can scrub a replica's oplog entries after they have been used to update other replicas. For more information, see Scrubbing Parameters for VOB Replicas.

Tracking Operations for Each Replica

The history of an unreplicated VOB is a linear sequence of operations (Figure 4).

Figure 4 History of Changes to an Unreplicated VOB

For a replicated VOB, changes are tracked separately for each replica. (That is why an oplog entry includes the identity of the replica where the operation originated.) Thus, the history of a replicated VOB can be viewed as several stacks of oplog entries. Each stack is represented by a linear sequence of epoch numbers for the operations originating in that replica.

Figure 5 shows the state of two replicas in a VOB family:

Figure 5 State of a VOB Family

A replica has accurate data only about its own operations. Until it receives update packets, its information about other replicas is out of date. For example, replica boston_hub records 950 local operations, but has received update packets for only 504 sanfran_hub operations. Similarly, replica sanfran_hub records 702 local operations, but has no current data about the boston_hub replica's state.

Figure 6 illustrates this scenario, in which each replica is out of date with respect to the operations originating at the other replica.

Figure 6 State of a Replicated VOB

Epoch Numbers

Picturing a replicated VOB as a set of oplog stacks, shown in Figure 6, makes it easy to understand the synchronization process. For example, an update packet sent from replica boston_hub to replica sanfran_hub consists of increments to the stack for replica boston_hub (operations 792-950). Figure 7 shows the two increments. Because sanfran_hub knows its own state, it needs updates only for other replicas. (In certain error-recovery situations, you must reset a replica's data about its own operations. See Chapter 10, Troubleshooting MultiSite Operations.)

Figure 7 Updates Between Two Replicas

NOTE: By the time the packet is imported at sanfran_hub, additional VOB-level changes may have been made at boston_hub. These changes are not included in the update packet.

Optimization and the Epoch Number Matrix

The MultiSite synchronization scheme attempts to minimize the amount of data transmitted among sites. Each replica keeps track of these epoch numbers:

  1. Changes made in the current replica. The epoch number that indicates how many operations originated at the current replica.

  2. Changes at sibling replicas. When syncreplica writes an operation from an update packet to the current replica, it increments the epoch number for the sibling replica at which the operation originally occurred. This epoch number is the number of operations originating at the sibling replica that have been imported at the current replica.

  3. Current knowledge of the states of other replicas. For each other replica, an estimate of its own changes and other replicas' changes.

Figure 8 shows how these epoch numbers fall into an epoch number matrix. Each replica maintains its own such matrix, revising its rows as work occurs locally and as it exchanges update packets with other replicas:

Figure 8 Two-Row Epoch Number Matrix at Replica boston_hub

The contents of this matrix are reported by the multitool lsepoch command at the boston_hub replica:

multitool lsepoch

For VOB replica "/vobs/dev":

Oplog IDs for row "boston_hub" (@ minuteman):

oid:87f6265f.72d911d4.a5cd.00:01:80:c0:4b:e7=950

(boston_hub)

oid:0eaa6fc3.737d11d4.adbe.00:01:80:c0:4b:e7=504

(sanfran_hub)

Oplog IDs for row "sanfran_hub" (@ goldengate):

oid:87f6265f.72d911d4.a5cd.00:01:80:c0:4b:e7=912

(boston_hub)

oid:0eaa6fc3.737d11d4.adbe.00:01:80:c0:4b:e7=504

(sanfran_hub)


A syncreplica -export command entered at boston_hub uses this matrix as follows to generate an update destined for sanfran_hub:

  1. At boston_hub, the number of local operations is 950 (number in upper-left corner of matrix), and the estimate is that sanfran_hub has been updated only to epoch number 912 (number in lower-left corner).

  2. The update packet that boston_hub sends to sanfran_hub includes boston_hub oplog entries 913-950. After the Boston administrator invokes syncreplica -export, the sanfran_hub row is updated:

  3. multitool lsepoch

    For VOB replica "/vobs/dev":

    Oplog IDs for row "boston_hub" (@ minuteman):

    oid:87f6265f.72d911d4.a5cd.00:01:80:c0:4b:e7=950

    (boston_hub)

    oid:0eaa6fc3.737d11d4.adbe.00:01:80:c0:4b:e7=504

    (sanfran_hub)

    Oplog IDs for row "sanfran_hub" (@ goldengate):

    oid:87f6265f.72d911d4.a5cd.00:01:80:c0:4b:e7=950

    (boston_hub)

    oid:0eaa6fc3.737d11d4.adbe.00:01:80:c0:4b:e7=504

    (sanfran_hub)


Indirect Synchronization

If there are more than two replicas in a VOB family, synchronization can occur indirectly. A replica can include nonlocal changes in update packets. For example, if boston_hub exchanges updates with replicas sanfran_hub and bangalore, it sends bangalore oplog entries that it has received previously from sanfran_hub. These entries may or may not bring replica bangalore up to date on sanfran_hub's changes. (An update sent from sanfran_hub to bangalore does bring bangalore up to date.)

NOTE: If a replica does not receive packets directly from some replicas in the VOB family, its rows for those replicas may contain zeros. This is expected behavior.

Figure 9 shows replica boston_hub's epoch number matrix.

Figure 9 Epoch Number Matrix at Replica boston_hub

The contents of this matrix are reported by the lsepoch command:

multitool lsepoch

For VOB replica "/vobs/dev":

Oplog IDs for row "boston_hub" (@ minuteman):

oid:87f6265f.72d911d4.a5cd.00:01:80:c0:4b:e7=950

(boston_hub)

oid:7ag3b0bc.defa11d0.ba57.00:01:72:73:3c:94=653

(bangalore)

oid:0eaa6fc3.737d11d4.adbe.00:01:80:c0:4b:e7=504

(sanfran_hub)

Oplog IDs for row "bangalore" (@ ramohalli):

oid:87f6265f.72d911d4.a5cd.00:01:80:c0:4b:e7=709

(boston_hub)

oid:7ag3b0bc.defa11d0.ba57.00:01:72:73:3c:94=653

(bangalore)

oid:0eaa6fc3.737d11d4.adbe.00:01:80:c0:4b:e7=221

(sanfran_hub)

Oplog IDs for row "sanfran_hub" (@ goldengate):

oid:87f6265f.72d911d4.a5cd.00:01:80:c0:4b:e7=912

(boston_hub)

oid:7ag3b0bc.defa11d0.ba57.00:01:72:73:3c:94=653

(bangalore)

oid:0eaa6fc3.737d11d4.adbe.00:01:80:c0:4b:e7=504

(sanfran_hub)


A syncreplica -export command at Boston uses this matrix to export an update for bangalore:

  1. At Boston, there are 950 local operations (number in upper-left corner of matrix), and the estimate is that bangalore has been updated only to epoch number 709 (lower-left corner).

  2. For operations that originated at sanfran_hub, boston_hub has been updated to epoch number 504, and estimates that bangalore has been updated only to epoch number 221.

  3. The update packet that boston_hub sends to bangalore includes boston_hub oplogs 710-950 and sanfran_hub oplogs 222-504. The output of a multitool lsepoch command at Boston now looks like this:

  4. multitool lsepoch

    For VOB replica "/vobs/dev":

    Oplog IDs for row "boston_hub" (@ minuteman):

    oid:87f6265f.72d911d4.a5cd.00:01:80:c0:4b:e7=950

    (boston_hub)

    oid:7ag3b0bc.defa11d0.ba57.00:01:72:73:3c:94=653

    (bangalore)

    oid:0eaa6fc3.737d11d4.adbe.00:01:80:c0:4b:e7=504

    (sanfran_hub)

    Oplog IDs for row "bangalore" (@ sushi):

    oid:87f6265f.72d911d4.a5cd.00:01:80:c0:4b:e7=950

    (boston_hub)

    oid:7ag3b0bc.defa11d0.ba57.00:01:72:73:3c:94=653

    (bangalore)

    oid:0eaa6fc3.737d11d4.adbe.00:01:80:c0:4b:e7=504

    (sanfran_hub)

    Oplog IDs for row "sanfran_hub" (@ goldengate):

    oid:87f6265f.72d911d4.a5cd.00:01:80:c0:4b:e7=912

    (boston_hub)

    oid:7ag3b0bc.defa11d0.ba57.00:01:72:73:3c:94=653

    (bangalore)

    oid:0eaa6fc3.737d11d4.adbe.00:01:80:c0:4b:e7=504

    (sanfran_hub)