How to see which files are affected by a corrupted chunk?

[maples3]

I ran borg check against my repos for the first time since setting them up, and on one of my repos it reported an error:

$ borg check --verbose --verify-data
Starting repository check
finished segment check at segment 8158
Starting repository index check
Index object count match.
Finished full repository check, no problems found.
Starting archive consistency check...
Starting cryptographic data integrity verification...
chunk ded1c6b99ca3491b762e83967647668a735f2c6ee666e40347c32a96cf8a2db2, integrity error: Data integrity error: Chunk ded1c6b99ca3491b762e83967647668a735f2c6ee666e40347c32a96cf8a2db2: id verification failed
Found defect chunks. With --repair, they would get deleted, so affected files could get repaired then and maybe healed later.
Finished cryptographic data integrity verification, verified 2088268 chunks with 1 integrity errors.
Analyzing archive nextcloud_2018_11_06 (1/66)
...

So it seems that one of the chunks in this repo has been corrupted somehow.

I know if I run borg check --repair, it will delete the bad chunk and re-add it if it shows up in another future backup. My question is: Which file(s) within which archives this chunk belongs to?

The reasoning is: depending on which data is lost, that will affect my next steps:

• If it's some random log file in an archive from 2023, I'll forget about it and move on with my life. Maybe delete that archive so I don't have to think about it again.
• If it's a photo I took recently, I'll wait for my regularly-scheduled cronjob to run another ordinary backup, and let the photo get re-added to the repo that way.
• If it's babys_first_steps.mp4 across all archives, I'll probably go extract that file from another backup and re-add it to this repo to make sure that this particular file has a valid backup.

The docs indicate that borg extract and borg mount will give warnings about missing chunks. Is that the only way to get this information? I'm hoping I don't have to extract each individual archive (even to /dev/null) to identify the file -- The data directories this repo backs up are over 2TB.

Borg version: 1.4.0 (from Debian's repository)

[ThomasWaldmann]

1. For efficiency reasons, the "cryptographic data integrity verification" just does a single full scan over all objects in the repository. It does not process archives, or files in archives, thus it does not know about them and can't give you that information.

2. OTOH, the archives check ("analyzing archive ...") does not do that cryptographic id vs. contents check for each referenced file contents chunk (that would be very inefficient, checking the same chunks again and again). It just checks presence of referenced chunks.

Sadly, these 2 steps do not communicate and step 2 will only notice MISSING or REAPPEARED chunks.

The bad chunk won't be missing until you do a borg check --repair.

Also, if you want to play safe, maybe run a memory check on that machine (memtest86+) before running borg check --repair. The corruption you have observed has a yet unknown cause, it could be a defect of the hdd/ssd, memory, cpu, ... or it could be just a random bitflip. You rather don't want to run a repair on a system with a hw defect that is actively corrupting data.

Other options:

• borg extract --dry-run does all extraction steps except writing to disk
• run borg check --verify-data again - is it repeatable, does it find the same defective chunk?
• try dumping archives with borg debug ... and look for the defective chunkid in the output
• maybe check the borg 1.4.x changelog if there was an improvement after 1.4.0.

[maples3]

Thanks for the explanation and the tips!

The borg debug commands in particular are super helpful. They have the "tell me which chunks are used by which files" info that I was looking for. (The good news is, the corruption is in a file that I have access to the original for, so repairing the repo will be easy.)

I'll re-run borg check --verify-data (using 1.4.3, downloaded from GitHub) and see if the results match.

As an aside, I have another offsite backup (a portable HDD plugged into the back of to a friend's home server) which finally completed its borg check, and it also reported a single failed chunk, in a different file. I'll re-run borg check (with 1.4.3) and see if the results match there too.

In both cases, the affected chunk appears in archives from 2023 and 2024, respectively -- a while ago, but not that long ago. I definitely need to run borg check more often, and potentially look into replacing some disks...

If the borg check results match in both repos on a second pass, that seems to me like it would be a strong indicator of "the data on disk is bad", not "my current machine's cpu/ram is defective". (The machine I'm using today is different than the one the first affected archives were originally taken from, so if that hardware was borked, it's a non-issue at this point.)

If that checks out, then would you go ahead and proceed with borg check --repair, and let the next backup fill in the missing chunks? Or is there anything else you'd recommend I look into first?

---

For future reference here are some snippets I used:

for x in $(borg list | cut -d' ' -f1)
do
    borg debug dump-archive --verbose ::$x debug_$x.json
    gzip -9 $x.json # the raw JSON per archive is many hundred megabytes, with a lot of archives it adds up fast
    echo $x
done

To start, I just used zcat * | less and used the / command to search for my chunk ID to identify the affected file in the first archive it's present in (with how I've got things organized, that's a good enough answer for me).

I then tried poking around with jq to try and figure out which archive the affected chunk first appeared in. Unfortunately, I recently set up pruning, so I've only got year-accurate resolution that far back which isn't super helpful. Though it did tell me that the corruption in the second repo happened at least a year after the corruption in the first repo, which is slightly useful to know (and tells me I need to run borg check more often!)

for x in *.json.gz
do
    zcat $x | jq -r 'select(._items[] | .chunks != null and .chunks[][] == "\u007f<chunk ID here>") | ._name' | tee archive_list.txt
done

[ThomasWaldmann]

If the hw looks ok, do a borg check --repair, borg create (which adds the missing chunk) and then another borg check --repair.

[maples3]

Okay, so I re-ran borg check --verify-data on my local drive, and it returned three failed chunks. None of which were the one initially reported.

I then ran memtest86+ on my server for 24 hours (multiple passes), and it reported no errors.

Do you have any thoughts on where I should look next, or resources/guides I should refer to? Something here seems off, though I don't know where I should start digging next.

[ThomasWaldmann]

It's a bit of an effort, but one way to check if it is the data or the hardware would be to connect the disk to another (proven) machine and run the check there.

I once had an usb hub that corrupted data rarely. Removing the hub removed the issue, reproducably.