Like the title says, I got back into aquariums and fish keeping recently. One could say it started with our recent move to Orinda. Going from roughly 950 square feet in our San Francisco condo to almost 3000 sq ft in our new home had a large say in the matter. Our living room had an odd corner that we were not quite sure what to do with. The space laid largely bare for months as we thought about several options.

I reflected back on my youth and my fish/amphibian/reptile-keeping hobby and remembered how much joy it brought. I really was into those critters. I could just zone out (sans medication) and watch fish for hours. I could watch turtles pop around and slowly watch newts change into true amphibians. I remembered how much it relaxed me and what a stress reliever they were. Seems like a great idea during a stay-at-home, always with your family 24x7 pandemic!

Don't get me wrong, it aint all roses. I still freak out when a fish swims a little sideways and instantly dread, "Does it have swim bladder problem? Could it be fungal? Oh crap is it bacterial?" And that's just the first 30 seconds. These critters are my babies. And in this latest round, I've amped up my game. I've added live plants and shrimp. First things first, back to the start.

In case you have been living in a hole or hate animals, you probably know that everything pet related is massively sold out or in short supply during COVID19. People need companionship, and if we can't fill that with a human being, we'll fill it with a dog/cat/bird/snake/turtle/guineapig/fish/snail/etc. I dove into research on trying to find something that would work for the the space and fill my needs. One of the biggest things was how much things have changed since I was 15 years old. Virtually no one sells undergravel filters any more. Hang-on-back (HOB) filters are almost looked down upon and canister filters are almost the bare minimum. Technology and productivity advances have jumped ahead so much. The gear you get now is incredibly better than what we had back in the 80's. And cheaper! Massively cheaper! Sure, you can still nerd out on Euro gear or go super paying-for-the-badge-ADA stuff, but your run-of-the-mill stuff from Petco/Petsmart is really good and very well priced.

I'll follow up with another post about my setup and all that, but I wanted to write this: fish keeping is better than ever and you should look into joining the hobby. You'll be better off. I know that I am.

Basically avoid them at all cost. They're buggy, and Samsung doesn't publicly post their SSD firmware on their website. You need to go through OEM vendors to get it. Anyone who deals with OEM's knows what a PITA it is to get things from them when it involves another company.

Buying drives from CDW or Amazon on your own? Might as well be anal rape in Samsung's book. You're not getting any support nor any updates so that 5 year warranty can get bent.

Stick with their "Pro" consumer SSD's or better yet, buy Intel or Micron. Both Intel and Micron have good firmware support with publicly posted software packages.

That said, hit me up if you just happen to have the latest 883 DCT firmware. I really need it.

If you're running Red Hat Ceph, make sure you're on the latest 3.2 or 3.3 Otherwise, you're in for a world of hurt. The first 3.1 "fully supported" release had too many bugs. Be sure to match up your alloc_size to match your stripe chunks, otherwise you'll be burning extra space depending on file size.

The increased write performance is nice, but there were too many initial gotchas in their first attempts.

They're still tweaking various things as the support cases roll in. In just the latest release they changed the Bluestore block allocator from random to sequential. These are not minor changes...

Oh and if you thought about converting OSD's from Filestore to Bluestore, don't. It will take too long. Just build a new cluster and recopy the data from an application level. Rebuilding with the backfill method is pretty much garbage slow. Our moderately sized cluster would take a year and a half to rebuild as Bluestore with the sanctioned Red Hat Ceph methods.

Totally unsolicited, but I thought that I'd write up a good blurb about this Bluetooth receiver/transmitter that I found on eBay. eBay doesn't really offer product reviews, so I guess this is my version of the Amazon product review.

We have a great TV that can send out audio over Bluetooth, but only to a single device at a time. I've found this shortcoming in many Android TV's, but maybe it's a protocol thing? Not sure, I dunno. I like to listen with my Sony LDAC Bluetooth headphones, but things go bad when the wife wants to join the party. So what's our best choice?

Well, I grabbed this BT audio receiver and hooked it up to a pair of Sennheiser 165 RF headphones. TV -> Bluetooth receiver -> Sennheiser unit via stereo miniplug. We actually went through a few of these things before we found one that had good audio quality, paired reliably, and could run well off of AC power instead of just batteries (or perpetually charging batteries).

tldr; buy the Homespot Bluetooth 4.2 Apt-X audio receiver if you need a good BT receiver. It's better than all the other crap out there. It took 2 weeks to arrive from Hong Kong, but it was very much worth it. 6 months without any issues.

A while back, I tasked one of my team members to update the NTP servers used in one of our datacenters. We were using standard pool NTP services and decided to move away from them for various reasons. We found that stable time was more important than accurate time, and the pools definitely didn't add stability. NTP uses UDP by default, and we wanted to turn off/ACL-off UDP in certain networks. So we grabbed a few CDMA-based time servers off of Ebay, fronted them with our typical Juniper SRX firewalls, and set up clients to use the SRX's as time sources.

After setting up a few devices, this employee suggested, "Hey why don't we set this up on a loopback and anycast it?" I thought about it for a second, something else came up, we moved on, and the suggestion was forgotten (by both of us). We had not finished moving everything from flat, layer 2 networks to a true Clos L3LS setup, so the timing wasn't just right. After finishing the L3LS migration, I looked at this again......and we're very happy with the results.

I will preface this with, I am not a DBA. I've run PostgreSQL databases off and on for the past 8 years, but I'm not a full-time DBA. I don't follow all of the ins and outs and daily updates with PGSQL. I'm a simple bit pusher who runs a few dozen DB's. And with that out of the way.....

I got an alarm last week about high disk usage on one of our PostgreSQL database instances. This was really odd as we've never seen a high disk alarm on them before. Also, it was just the master that was alarming; none of the replicas were alarming even though they have identically-sized drives. If the DB data were truly growing quickly, every machine would alarm. So I checked the DB size graphs in OpenNMS and sure enough, they were OK:

These are 1.6TB drives so nothing wild going on here. Running du in the pg_data directory showed that pg_xlogs were responsible for all of the new disk usage. But why? None of the WAL and checkpoint settings had been changed in months. Nothing new in the output logs. After searching high and low about the actual calculations for retaining WAL files (most of them are wrong including the official docs), I came to the conclusion that our WAL and checkpoint settings were not responsible for the high disk usage. But what was?