Join GitHub today
GitHub is home to over 50 million developers working together to host and review code, manage projects, and build software together.
Sign upGitHub is where the world builds software
Millions of developers and companies build, ship, and maintain their software on GitHub — the largest and most advanced development platform in the world.
Failed to create index - PAXOS Failed to update metadata source=create-index #367
Comments
The PAXOS update failed because it tried to write a row with version=2 in metadata_log, but there is already a row with version 2 from the day before = 2020-08-27 22:11:46.052000+0000, (while error message are 2020-08-28 01:27:46). So, for some reason, something wrong happen around 2020-08-27 22:11:46 and the elasticsearch cluster state is not more synchronised with the metadata_log, clusterstate metadata.version = 1 and metadata_log.version = 2. You can resynchronise manually like this: You probably have WARN/ERRROR messages around 2020-08-27 22:11:46 in your logs. |
Thanks, there was definitely a PAXOS timeout. I wonder what caused it. Regardless what you suggested fixed it. Thanks. Is this a bug? |
A high network latency could cause such error.
If you have the logs of the coordinator nodes when the first PAXOS timeout occurs, it could help to investigate.
… On 3 Sep 2020, at 20:27, Andrew Grosser ***@***.***> wrote:
Thanks, there was definitely a PAXOS timeout. I wonder what caused it. Regardless what you suggested fixed it. Thanks.
Is this a bug?
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub <#367 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACOMPGKVI3CG2F25IYB7GBTSD7NYRANCNFSM4QNQ5B5Q>.
|
Please help! Thanks!
Elassandra version:
6.2.3.20 (apt version)
Plugins installed: []
None
JVM version (
java -version
):1.8
OS version (
uname -a
if on a Unix-like system):Linux cassandra1-prod 4.19.0-6-amd64 #1 SMP Debian 4.19.67-2+deb10u2 (2019-11-11) x86_64 GNU/Linux
Description of the problem including expected versus actual behavior:
Index doesn't get created, and log complains of paxos and metadata issue? (It does work on my local dev machine but not in the test cluster)
Steps to reproduce:
Elassandra Logs
(Initiate Machine)
2020-08-28 01:27:46,099 WARN [elasticsearch[10.0.0.2][masterService#updateTask][T#1]] ClusterService.java:1310 commitMetaData PAXOS Failed to update metadata source=create-index [events_recent], cause [api] prevMetadata=6bec5c6a-5173-4625-a0fa-75c37b83b76a/1 nextMetaData=6bec5c6a-5173-4625-a0fa-75c37b83b76a/2
2020-08-28 01:27:46,100 WARN [elasticsearch[10.0.0.2][masterService#updateTask][T#1]] CassandraDiscovery.java:1079 publishAsCoordinator PAXOS concurrent update, source=create-index [events_recent], cause [api] metadata=6bec5c6a-5173-4625-a0fa-75c37b83b76a/2, resubmit task on next metadata change
Cluster state