What to do when something goes wrong with bees

Hangs and excessive slowness

Are you using qgroups or autodefrag?

Read about bad btrfs feature interactions.

Use load-throttling options

If bees is just more aggressive than you would like, consider using load throttling options. These are usually more effective than ionice, schedtool, and the blkio cgroup (though you can certainly use those too).

Check $BEESSTATUS

If bees or the filesystem seems to be stuck, check the contents of $BEESSTATUS. bees describes what it is doing (and how long it has been trying to do it) through this file.

Sample:

The time in square brackets indicates how long the thread has been executing the current task (if this time is below 5 seconds then it is omitted). We can see here that the main thread (and therefore the bees process as a whole) has been running for 68493.1 seconds, the last hash table write was 12.5 seconds ago, and the last transid poll was 236.5 seconds ago. Three worker threads are currently performing dedupe on extents.

Thread names of note:

• crawl_12345: scan/dedupe worker threads (the number is the subvol ID which the thread is currently working on). These threads appear and disappear from the status dynamically according to the requirements of the work queue and loadavg throttling.
• bees: main thread (doesn't do anything after startup, but its task execution time is that of the whole bees process)
• crawl_master: task that finds new extents in the filesystem and populates the work queue
• crawl_transid: btrfs transid (generation number) tracker and polling thread
• status: the thread that writes the status reports to $BEESSTATUS
• crawl_writeback: writes the scanner progress to beescrawl.dat
• hash_writeback: trickle-writes the hash table back to beeshash.dat
• hash_prefetch: prefetches the hash table at startup and updates beesstats.txt hourly

Dump kernel stacks of hung processes

Check the kernel stacks of all blocked kernel processes:

ps xar | while read -r x y; do ps "$x"; head -50 --verbose /proc/"$x"/task/*/stack; done | tee lockup-stacks.txt

Submit the above information in your bug report.

Check dmesg for btrfs stack dumps

Sometimes these are relevant too.

bees Crashes

• If you have a core dump, run these commands in gdb and include the output in your report (you may need to post it as a compressed attachment, as it can be quite large):

   (gdb) set pagination off
   (gdb) info shared
   (gdb) bt
   (gdb) thread apply all bt
   (gdb) thread apply all bt full
  

The last line generates megabytes of output and will often crash gdb. This is OK, submit whatever output gdb can produce.

Note that this output may include filenames or data from your filesystem.

• If you have systemd-coredump installed, you can use coredumpctl:

   (echo set pagination off;
   echo info shared;
   echo bt;
   echo thread apply all bt;
   echo thread apply all bt full) | coredumpctl gdb bees
  

• If the crash happens often (or don't want to use coredumpctl), you can run automate the gdb data collection with this wrapper script:

To use the wrapper script, insert it just before the bees command, as in:

gdb-wrapper bees /path/to/fs/

Kernel crashes, corruption, and filesystem damage

bees doesn't do anything that should cause corruption or data loss; however, btrfs has kernel bugs and interacts poorly with some Linux block device layers, so corruption is not impossible.

Issues with the btrfs filesystem kernel code or other block device layers should be reported to their respective maintainers.