Investigating node failures

Oct 13, 2025 by leolavanderia.com.br 138 views

Introduction

CockroachDB is a distributed relational database that provides a highly available and scalable solution for modern applications. However, like any complex system, it can encounter issues and failures during testing and deployment. In this article, we will investigate a specific failure that occurred during a roachtest run, specifically the kv/quiescence/nodes=3 test.

Failure Details

The failure occurred on the master branch with commit hash 2362ac69262c53be08f010fcde76371592d311ef. The test timed out after 3 hours and 0 minutes, indicating that the test was unable to complete within the allotted time. The test artifacts and logs can be found in the /artifacts/kv/quiescence/nodes=3/run_1 directory.

Test Parameters

The test was run with the following parameters:

arch=amd64
cloud=azure
coverageBuild=false
cpu=4
encrypted=false
metamorphicBufferedSender=false
runtimeAssertionsBuild=true
ssd=0

Investigation

To investigate this failure, we need to understand the context and the possible causes. The kv/quiescence/nodes=3 test is designed to test the quiescence of the key-value store in a distributed CockroachDB cluster. Quiescence refers to the ability of the cluster to remain in a stable state even when nodes are added or removed.

Possible Causes

There are several possible causes for this failure:

Assertion Violation: The test may have encountered an assertion violation, which is a runtime error that occurs when a test expects a certain condition to be true but it is not.
Timeout: The test may have timed out due to a slow network or a slow node in the cluster.
Node Failure: One of the nodes in the cluster may have failed, causing the test to timeout.

Next Steps

To resolve this issue, we need to investigate further and gather more information. We can start by:

Checking the logs: We can check the logs to see if there are any error messages or warnings that may indicate the cause of the failure.
Running the test again: We can run the test again with the same parameters to see if the failure is reproducible.
Investigating node failures: We can investigate node failures to see if one of the nodes in the cluster failed during the test.

Conclusion

In conclusion, the kv/quiescence/nodes=3 test failed due to a timeout. The possible causes of this failure include assertion violations, timeouts, and node failures. To resolve this issue, we need to investigate further and gather more information. We can start by checking the logs, running the test again, and investigating node failures.

Additional Resources

For more information on how to investigate internal failures, please see the following resources:

Get Involved

If you are interested in helping to resolve this issue, please join the conversation by commenting below. You can also contribute to the CockroachDB project by submitting pull requests or reporting issues.

Related Issues

This issue is related to the following issues:

Help

Introduction

In our previous article, we investigated a failure that occurred during a roachtest run, specifically the kv/quiescence/nodes=3 test. In this article, we will provide a Q&A section to help answer some of the common questions related to this issue.

Q: What is the `kv/quiescence/nodes=3` test?

A: The kv/quiescence/nodes=3 test is a roachtest that is designed to test the quiescence of the key-value store in a distributed CockroachDB cluster. Quiescence refers to the ability of the cluster to remain in a stable state even when nodes are added or removed.

Q: What is quiescence in CockroachDB?

A: Quiescence in CockroachDB refers to the ability of the cluster to remain in a stable state even when nodes are added or removed. This means that the cluster should be able to continue operating normally even if some nodes are unavailable.

Q: Why did the `kv/quiescence/nodes=3` test fail?

A: The kv/quiescence/nodes=3 test failed due to a timeout. The possible causes of this failure include assertion violations, timeouts, and node failures.

Q: How can I investigate this issue further?

A: To investigate this issue further, you can:

Check the logs to see if there are any error messages or warnings that may indicate the cause of the failure.
Run the test again with the same parameters to see if the failure is reproducible.
Investigate node failures to see if one of the nodes in the cluster failed during the test.

Q: How can I contribute to the CockroachDB project?

A: You can contribute to the CockroachDB project by submitting pull requests or reporting issues. You can also join the conversation by commenting below.

Q: What are some common causes of the `kv/quiescence/nodes=3` test failure?

A: Some common causes of the kv/quiescence/nodes=3 test failure include:

Assertion violations
Timeouts
Node failures

Q: How can I get help with this issue?

A: You can get help with this issue by:

Joining the conversation by commenting below
Seeking help from the CockroachDB community by joining the discussion on the CockroachDB forum
Submitting a pull request or reporting an issue on the CockroachDB GitHub repository

Q: What are some related issues that I should be aware of?

A: Some related issues that you should be aware of include:

Conclusion

Additional Resources

For more information on how to investigate internal failures, please see the following resources: