Update documentation to create an empty wal file#411
Update documentation to create an empty wal file#411EdColeman wants to merge 3 commits intoapache:mainfrom
Conversation
_docs-2/troubleshooting/advanced.md
Outdated
| backup policy is used which could lead to a long wait before Accumulo will | ||
| try to re-load the WAL file. | ||
| ```sh | ||
| UUID=$(uuidgen); echo -n -e '--- Log File Header (v4) ---U+1F47B$'"$UUID"'\x00\x00\x00\x00' >"$UUID".wal |
There was a problem hiding this comment.
So, I was tracking down where the U+1F47B comes from. I think the original intent is to have the actual ghost emoji character stored there, but instead, it's a human-readable representation of the unicode code point for the ghost emoji that's stored there. It doesn't really matter what it's value is, though. It's purpose is to be a fixed value that represents the decryption parameters for a "not encrypted" file.
Specifically, this value is an indicator that it's "not encrypted" using our built-in non-encrypting CryptoService that we use in our built-in reference implementations for a CryptoServiceFactory. A user could provide their own factory that supports a "not encrypted" option using a different token to identify files it supports. But, this string is to denote ours.
I checked the serialization of our WALs in the code, and it looks like this is wrong as written. There should be an integer that is written before the "U+1F47B" part. After that, I'm not sure what the WAL is supposed to contain, because I only got so far tracing the code to see how it writes the WAL header.
In Java, this is what I saw as a header, when tracing the steps to write out the size of the crypto params, then writing out the string for the "no crypto" params:
byte[] header = "--- Log File Header (v4) ---\000\000\000\007U+1F47B".getBytes(UTF_8)
// { 45, 45, 45, 32, 76, 111, 103, 32, 70, 105, 108, 101, 32, 72, 101, 97, 100, 101, 114, 32, 40, 118, 52, 41, 32, 45, 45, 45, 0, 0, 0, 7, 85, 43, 49, 70, 52, 55, 66 }That's equivalent in bash to:
echo -n -e '--- Log File Header (v4) ---\x00\x00\x00\x07U+1F47B' >empty.walI don't know where the UUID comes in, or that dollar sign in the above example, or the zeroes from the v2 format. There may be more needed after this header to result in a valid WAL file... I'm not sure, but I do think what's here now in this PR is wrong.
There was a problem hiding this comment.
As discussed, I'm creating a utility similar to the one to create an empty rfile. When complete these instructions will be updated.
There was a problem hiding this comment.
The documentation has been updated to remove the previous changes and add the create-empty WAL / RFILE options. Because this had drifted so far from main I ended up doing a merge and force push. Sorry.
- Update empty rfile to keyword executable command format - Update to include creating an empty wal file.
6da76d5 to
23a1898
Compare
Updated the documentation for creating an empty wal file to use version 4 header, a uuid and other information.