Repository navigation
TEZ-4759: An external AM never re-registers after a ZooKeeper session loss - #543
Merged
Merged
Conversation
deniskuzZ
commented
Sep 30, 2026
deniskuzZ
commented
Member
- ZkAMRegistry.add() wrote the AMRecord once, as a bare EPHEMERAL node; a session expiry deletes it and nothing re-creates it, so clients see an empty registry while the AM keeps running
- Hold a Curator PersistentNode per AMRecord: it re-creates the node after a session loss or a delete; remove() closes it
- add() fails when the node is not created within the ZooKeeper connection timeout
- TestZkAMRegistry: after the ZooKeeper server is replaced, the AMRecord node is back
… loss - ZkAMRegistry.add() wrote the AMRecord once, as a bare EPHEMERAL node; a session expiry deletes it and nothing re-creates it, so clients see an empty registry while the AM keeps running - Hold a Curator PersistentNode per AMRecord: it re-creates the node after a session loss or a delete; remove() closes it - add() fails when the node is not created within the ZooKeeper connection timeout - TestZkAMRegistry: after the ZooKeeper server is replaced, the AMRecord node is back
|
🎊 +1 overall
This message was automatically generated. |
abstractdog
requested changes
Nov 9, 2026
abstractdog
left a comment
abstractdog
left a comment
Contributor
There was a problem hiding this comment.
nits, otherwise looks good, thanks Denys
| if (node != null) { | ||
| node.close(); // deletes the node | ||
| } | ||
| LOG.info("Deleted AMRecord from zkpath {}", path); |
Contributor
There was a problem hiding this comment.
this logs "Deleted AMRecord from zkpath" even when nothing was deleted, how about:
if (node == null) {
LOG.debug("AMRecord {} was not registered by this instance, nothing to remove", server);
return;
}
node.close(); // deletes the node
LOG.info("Deleted AMRecord from zkpath {}", pathFor(server));
| byte[] json = AMRegistryUtils.recordToJsonString(server).getBytes(StandardCharsets.UTF_8); | ||
| PersistentNode node = new PersistentNode(client, CreateMode.EPHEMERAL, false, path, json); | ||
| node.start(); | ||
| if (!node.waitForInitialCreate(zkConfig.getConnectionTimeoutMs(), TimeUnit.MILLISECONDS)) { |
Contributor
There was a problem hiding this comment.
if waitForInitialCreate throws InterruptedException, the node is already started but never closed or put into amNodes, it stays active, so its reconnect listener and NodeDeleted watch keep re-creating the path until close() shuts the Curator client down
what about extra care by:
boolean created;
try {
created = node.waitForInitialCreate(zkConfig.getConnectionTimeoutMs(), TimeUnit.MILLISECONDS);
} catch (InterruptedException e) {
node.close();
throw e;
}
if (!created) {
node.close();
throw new IOException("Timed out creating AMRecord at zkpath " + path);
}
Member
Author
There was a problem hiding this comment.
Good catch! It's now closed on any failure of the wait
| byte[] json = AMRegistryUtils.recordToJsonString(server).getBytes(StandardCharsets.UTF_8); | ||
| PersistentNode node = new PersistentNode(client, CreateMode.EPHEMERAL, false, path, json); | ||
| node.start(); | ||
| if (!node.waitForInitialCreate(zkConfig.getConnectionTimeoutMs(), TimeUnit.MILLISECONDS)) { |
Contributor
There was a problem hiding this comment.
add() used to be bounded by the client's retry policy (tez.am.curator.max-retries/backoff-sleep), and now waits a single getConnectionTimeoutMs(). The setting still applies to generateNewId() and to PersistentNode's background creates, so it isn't dead, but someone tuning max-retries might expect registration to wait longer. Scaling the wait by getCuratorMaxRetries() would only make that setting mean something it doesn't, so I'd keep the single timeout and add a comment saying add() waits one connection timeout for the initial create and is not governed by the retry settings.
- remove() logs the delete only when this instance registered the AMRecord - add() closes the node when waiting for the initial create fails, an interrupt included - Comment: the initial create waits one connection timeout, not the Curator retry settings
abstractdog
self-requested a review
November 9, 2026 12:23
|
🎊 +1 overall
This message was automatically generated. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.