IO Handling
Some notes about handling IO operations, file access etc in MapDB.
File Rename & replace
This operation replaces content of one file with another file. Original file (if existed )is usually destroyed. File Rename is very fast on local filesystems.
MapDB renames and replaces files in a few places; for example store compaction recreates store content in new file and than replaces original file.
Challenges
- old file might be open, all file handles must be closed before rename starts
- file may remain locked, even after it was closed (mmap on Windows)
- on replace old file is destroyed, but it should be preserved if rename fails
- file rename & replace must recover from crash
- when new file is opened or created, we need to check if there are rename fragments
- use atomic rename;
java.nio.Files#move()
with StandardCopyOption#ATOMIC_MOVE ATOMIC_MOVE
File Replace algorithm
This is sequence for replacing files, to ensure it recovers from crash
File Open algorithm
This sequence should be used when file is opened, to ensure that DB recovers if system crashed in middle of file rename
- Check existence of following files (exists and non zero size):
- F (no suffix)
- NF (
_renameNew
) - OF (
_renameOld
)
Here is decision matrix, validation uses checksum or whatever, if validation is not possible just always assume valid file:
- F yes, NF no, OF no
- usual case, just proceed, perhaps do not even validate
- F yes, NF yes, OF yes
- strange situation, should not exist with atomic renames
- crash possibly while file content was copied, some files might be corrupted (not fully copied)
- validate NF
- if valid, delete OF and proceed with File Replace, exit
- if invalid, delete NF and continue in next step
- validate F,
- if invalid rename OF to F and exit
- if valid delete OF and continue
- validate OF
- if valid, rename it to F and use it, exit
- if invalid, fail with fatal data corruption
- F yes, NF yes, OF no
- store crashed before F could be renamed to OF
- validate NF
- if invalid delete NF and try to use F, return
- if valid, proceed with NF to F rename, return
- F no, NF yes, OF yes
- store crashed after F was renamed to OF, but before NF was renamed to F
- validate NF
- if valid proceed with replace
- if invalid, delete it, rename OF to F and use it
- F no, NF yes, OF no
- should not happen,
- validate OF, use it if possible, else fail
- F no, NF no, OF yes
- should not happen
- validate OF and hope for best
- F yes, NF no, OF yes
- should not happen
- validate F, use it if possible
- else validate OF, use it if possible, else fatal error
- F no, NF no, OF no
- no files
- create new store if auto create configured, else fail to open