Ceph RGW on Bluestore OSD's...

If you're running Red Hat Ceph, make sure you're on the latest 3.2 or 3.3 Otherwise, you're in for a world of hurt. The first 3.1 "fully supported" release had too many bugs. Be sure to match up your alloc_size to match your stripe chunks, otherwise you'll be burning extra space depending on file size.

The increased write performance is nice, but there were too many initial gotchas in their first attempts.

They're still tweaking various things as the support cases roll in. In just the latest release they changed the Bluestore block allocator from random to sequential. These are not minor changes...

Oh and if you thought about converting OSD's from Filestore to Bluestore, don't. It will take too long. Just build a new cluster and recopy the data from an application level. Rebuilding with the backfill method is pretty much garbage slow. Our moderately sized cluster would take a year and a half to rebuild as Bluestore with the sanctioned Red Hat Ceph methods.