Library Version 12.1.3.0

Verifying the Store
Prev | Chapter 7. Administrative Procedures | Next

## Verifying the Store

Verification is a tool you can use to:

- Perform general troubleshooting of the store.

  Verification inspects all components of the store. It also checks whether all store services are available, and for those services it checks whether there are any version or metadata mismatches.

- Check the status of a long-running plan

  Some plans require many steps and may take some time to execute. The administrator can verify plans to check on progress of the plan, or as an aid to diagnose a plan that is in an ERROR state. For example, if you can verify a `Deploy Store` plan while it is running against many Storage Nodes. You can watch the verify report at each iteration to see that more and more nodes have created and have come online.

  For more information on how to manage plans, see Plans.

- Provide additional information about a plan that is in an ERROR state.

You run store verification using the `verify` command in the CLI. It requires no parameters, and by default it runs in verbose mode. For example:

    > java -Xmx256m -Xms256m \
    -jar KVHOME/lib/kvstore.jar runadmin -port <registry port>> \
    -host <hostname>
    kv-> verify configuration
    Verify: starting verification of mystore based upon topology sequence #1008
    1000 partitions and 3 storage nodes. Version: 12.1.3.0.1
    Time: 2014-01-07 21:16:02 UTC
    See <nodeHostname>:/KVRT1/mystore/log/mystore_{0..N}.log for progress
    messages
    Verify: == checking storage node sn1 ==
    Verify: Storage Node [sn1] on <nodeHostname>:5000
    Zone: [name=baskin id=zn1 type=PRIMARY]    Status: RUNNING
    Ver: 12cR1.3.0.1 2013-12-18 06:35:02 UTC  Build id: 8e70b50c0b0e
    Verify:         Admin [admin1]          Status: RUNNING
    Verify:         Rep Node [rg1-rn1]      Status: RUNNING,REPLICA at
    sequence number:2,025 haPort:5011
    Verify: == checking storage node sn2 ==
    Verify: Storage Node [sn2] on <nodeHostname>:5100
    Zone: [name=baskin id=zn1 type=PRIMARY]    Status: RUNNING
    Ver: 12cR1.3.0.1 2013-12-18 06:35:02 UTC  Build id: 8e70b50c0b0e
    Verify:         Rep Node [rg1-rn2]      Status: RUNNING,REPLICA at
    sequence number:2,025 haPort:5110
    Verify: == checking storage node sn3 ==
    Verify: Storage Node [sn3] on <nodeHostname>:5200
    Zone: [name=baskin id=zn1 type=PRIMARY]    Status: RUNNING
    Ver: 12cR1.3.0.1 2013-12-18 06:35:02 UTC  Build id: 8e70b50c0b0e
    Verify:         Rep Node [rg1-rn3]      Status: RUNNING,MASTER at
    sequence number:2,025 haPort:5210
    Verification complete, no violations.

A silent mode is available which shows only problems.

    kv-> verify configuration -silent
    Verify: starting verification of mystore based upon topology sequence #1008
    1000 partitions and 3 storage nodes. Version: 12.1.3.0.1
    Time: 2014-01-07 21:16:02 UTC
    See <nodeHostname>:/KVRT1/mystore/log/mystore_{0..N}.log for progress
    messages
    Verification complete, no violations.

Problems with the store are clearly reported. For example, if a Storage Node is unavailable, then in silent mode that problem is displayed in the following way:

    kv-> verify -silent
    Verify: starting verification of mystore based upon topology sequence #1008
    1000 partitions and 3 storage nodes. Version: 12.1.3.0.1
    Time: 2014-01-07 21:16:02 UTC
    See <nodeHostname>:/KVRT1/mystore/log/mystore_{0..N}.log for progress
    messages
    Verification complete, 2 violations, 0 notes found.
    Verification violation: [sn2]
    ping() failed for sn2 : Connection refused to host:
    <nodeHostname>; nested exception is:
            java.net.ConnectException: Connection refused
    Verification violation: [rg1-rn2]
    ping() failed for rg1-rn2 : Connection refused to
    host: <nodeHostname>; nested exception is:
            java.net.ConnectException: Connection refused

In verbose mode, the above problem is shown in the following way:

    kv-> verify configuration
    Verify: starting verification of mystore based upon topology sequence #1008
    1000 partitions and 3 storage nodes. Version: 12.1.3.0.1
    Time: 2013-12-18 21:16:02 UTC
    See <nodeHostname>:/KVRT1/mystore/log/mystore_{0..N}.log for progress
    messages
    Verify: == checking storage node sn1 ==
    Verify: Storage Node [sn1] on <nodeHostname>:5000
    Zone: [name=baskin id=zn1 type=PRIMARY]    Status: RUNNING
    Ver: 12cR1.3.0.1 2013-12-18 06:35:02 UTC  Build id: 8e70b50c0b0e
    Verify:     Admin [admin1]          Status: RUNNING
    Verify:     Rep Node [rg1-rn1]  Status: RUNNING,REPLICA at sequence
    number:2,025 haPort:5011
    Verify: == checking storage node sn2 ==
    Verify: sn2: ping() failed for sn2 : Connection refused to host:
    <nodeHostname>; nested exception is:
            java.net.ConnectException: Connection refused
    Verify: Storage Node [sn2] on <nodeHostname>:5100
    Zone: [name=baskin id=zn1 type=PRIMARY] UNREACHABLE
    Verify: rg1-rn2: ping() failed for rg1-rn2 : Connection refused to
    host: <nodeHostname>; nested exception is:
            java.net.ConnectException: Connection refused
    Verify:     Rep Node [rg1-rn2]  Status: UNREACHABLE
    Verify: == checking storage node sn3 ==
    Verify: Storage Node [sn3] on <nodeHostname>:5200
    Zone: [name=baskin id=zn1 type=PRIMARY]    Status: RUNNING
    Ver: 12cR1.3.0.1 2013-12-18 06:35:02 UTC  Build id: 8e70b50c0b0e
    Verify:     Rep Node [rg1-rn3]  Status: RUNNING,MASTER at sequence
    number:2,025 haPort:5210
    Verification complete, 2 violations, 0 notes found.
    Verify: sn2:    ping() failed for sn2 : Connection refused to host:
    <nodeHostname>; nested exception is:
            java.net.ConnectException: Connection refused
    Verify: rg1-rn2:    ping() failed for rg1-rn2 : Connection refused to
    host: <nodeHostname>; nested exception is:
            java.net.ConnectException: Connection refused

### Note

The verify output is only displayed in the shell when the command is finished. You can tail or grep the Oracle NoSQL Database log file in order to get a sense of how the verify is progressing. Look for the string `Verify`. For example:

    grep Verify /KVRT1/mystore/log/mystore_0.log

Prev | Up | Next
Addressing Lost Admin Service Quorum | Home | Monitoring the Store