I was listening to this week's All In podcast, and there were some armchair quarterback ideals and posturing being thrown around. Free market this, price control that, permit regulations here, rebuilding speed there. A lot of stuff to unwind from people who have never gone through any kind of rebuild after a natural disaster. Without casting aspersion or hot takes, here's what I remember about my father rebuilding post Katrina.

Background: my father owned by grandmother's house when Katrina hit. He wasn't living there, but he owned the house -- basically pre-buying something he would receive by inheritance and gave her money to live on. I guess sort of like a familial reverse mortgage. Katrina hits, the house is destroyed, the city is in shambles, the region takes a toll. My father's townhouse escapes relatively unscathed. It was at the top of the Metairie ridge, not far from the 17th Street canal (upstream from the breach). He had a safe place to live and assess the situation, while grandma moved to my aunt's in Tennessee.

My father wasn't rich, but he had money. Definitely not baller, but not hurting either. Post destruction, he was in the position to rebuild that home without waiting for insurance money. Most people (myself included) would need to wait for an insurance check and OK before proceeding with even clearing the lot. He paid to have the lot cleared and prepped for construction while an architect started working on plans. He wasn't going to rebuild the same home from the 1950's; he was going to build a new home. Keep in mind that this is not how normal home insurance works. If your home burns down, you generally don't get a payout and freedom to do what you want. You get your old house rebuilt exactly how it was. Yes, there are caveats for code upgrades but overall you'll get the house you had.

The local planning department put a freeze on permits while they drafted new regulations about lot height, drainage, utility entrances, etc. Waiting waiting waiting. Meanwhile there are construction crews leaving flyers and mail about offering services. Once he does finally get approval to build to the new regulations (I think he was the first in Old Metairie), there were no shortage of people ready to work.

Finally, the house is built. The plumbers and electricians were from Mobile (Mississippi was moving even slower on rebuilding there). The roofers were from Texas. The framers were from Israel. Yes, Israel. They all came in to work, following their own economic incentives. As such, my dad actually got a good deal because he had a permitted job and the means to pay, whereas everyone else was waiting and waiting and waiting. Over the next 4-5 years, folks in his area rebuilt and things now look like a hurricane never came (unless you look at how raised the lots are now).

So a couple of lessons learned here:

• First movers generally have an advantage. Be that defined by money, speed, etc.
• AHJ will generally drag on in drafting new guidelines and code, contrary to what any politician says in front of camera.
• Price controls aren't necessary. The market will work it out.
• Labor will move around as necessary to fill the need. The local construction labor market was wrecked (they all lost their homes and had to leave), so nearby labor moved in to fill the vacuum. It all works out without the need for government intervention.
• It's good to have a place to live. Not being sarcastic here, but my father was able to rebuild that house so quickly as he already had a place to live.  FEMA trailers were eventually allowed on lots for living, and the local building department will waive the lot's red tag for temp housing.
• It's a harrowing experience, but try to be positive. My grandmother was 86 when she lost her house. She could have moped and dwelled on what she lost. Instead, she had a positive attitude "Now I get to go shopping!" and even somewhat nonchalant about the whole thing "Well now I get to see what house you'd build after I died and you tore the old place down." Yes, she was lucky to live with her daughter and family.  I think her attitude helped everyone get through the situation and live a good life to the age of 103. She lived another 17 years post Katrina, so think about that when you feel like a tragic event is the end of everything. 

We recently replaced half of our windows and doors with new units from Marvin. We optioned most of these with Marvin's Lock Status Sensor system to conceal and hide our alarm system's hardware. Previously, we had the typical window and door sensors on the outside frames. If you opened the door or window, the magnet would move far enough away from the reed sensor,  letting the alarm system know of an open status.

​With a larger house and young children, this is especially useful. I can't tell you how many times the phrase "Max is loose!" is uttered in our house after he opens the patio slider to sneak out. We've also noticed that other people's kids act like they're raised in barns and just leave doors completely ajar or half-ass close them. In the middle of summer and winter, that's serious money going out the door. So our alarm system turns off the HVAC when certain windows or doors are left open more than 5 minutes. That said, get it together parents. Our 4 year old can close the door behind him -- your 9 year old can, too.

If you've run Ceph clusters for a long time, you probably either finished or are going through a Filestore to Bluestore conversion. For big clusters with lots of objects, it's a very long process. One of our oldest clusters has been converting for over a year, and it still isn't finished yet. It's a complicated function of object size, number of objects, EC settings, storage speed, etc.

A few places probably built a parallel cluster(s) out and copied the data between them. We've done that in a few places, but there were a few business needs that required conversion in place. For that large cluster, we chose the "whole host replacement" method described here: https://docs.ceph.com/en/latest/rados/operations/bluestore-migration/#convert-existing-osds. This method works, is safe, and it's supported by RedHat if you're under commercial support. There's one problem with that article though. It doesn't tell you how to put your spare host back into the cluster once you're done. It tells you to kick a node out with this command:

One would think it would be as simple as running the opposite, right? Well sorta. The command is indeed "ceph osd crush link", but that command isn't documented well and there are no examples of people using it in search results. So, this blog post is an attempt to seed the search engines.  You would run something like this to get that host back in the default root:

Or put it in a specific rack (like I wanted):

I've been pretty busy with work, so I've been lagging here on updates. Thankfully, I've been keeping up with my tank maintenance and things are overall going well.  Since the last update there have been a few changes:

• I lost another panda cory. Same symptoms as the others. One day fine, the next day sideways/weird swimming, the following day dead. I got this one into my hospital tank and dosed with Maracyn, Paracleanse, and Ich-X, but it still died.
• I added some orange pumpkin cherry shrimp that I bought off of eBay. 1 died en route, 1 died during acclimation, 1 died the following day. The remaining 8 are still there, thriving, and even multiplying! That's right, we have orange little shrimplets living in the Christmas moss (Vesicularia Montagnei).
• I added some dwarf hair grass (Eleocharis Parvula from ADA) to the right side of the tank. It's taking well, but not going wild. Between this and the marsilea hirsuta on the left side, I should get a decent carpet going....eventually.
• I added some Bacopa Caroliniana (Aquarium Co-op) in the rear of the tank on the path. I had too much of an empty gap and wanted to fill it with something new.
• I have an unusual glowlight Tetra. I posted on Reddit about it. Not normal, but not that unusual. Something new to me!
• I had an Amano shrimp crawl out of the tank and die on the floor. I guess it happened at night, and I discovered it the next morning. Tried to revive it, but no dice. All of my chemistry levels are great and stable. No idea why it decided to go on a field trip. From what I've read, this happens. It just sucks.
• Two of my female Amano shrimp are berried/with eggs. They're so big that it's fun and easy to watch them "handle their eggs". 
• I added a handful of red root floaters (Phyllanthus Fluitans) for fun. I originally got them for Remy's shrimp tank and had a few left over.

Panda cory catfish just really are not the hardy fish that people say they are.  Sorry, they're just not. They're so spazzy and dumb that they will willingly rip off their barbels while eating. What a silly fish. Once they're gone, the barbels are gone. They won't grow back (despite the one out of thirty people on the net that says they will). I now feed mine in a glass bowl so that this doesn't happen again. I've got one Panda Cory that has no barbels and an empty hole where his mouth should be. :(

One of my berried ladies

Rando video where I discovered an Amano was berried

The orange spot in upper center is a shrimplet. Zoomed out to show size.

If you're still here and reading, thanks! This is one of my outlets during the pandemic, much like the tank. For those who didn't read the first few posts, this is my first attempt at a planted tank. I'm not going to call it an aquascape, though I may use the term aquascaping for keeping the plants alive through maintenance. I'm not creating a replica of nature in a glass cube. I'm creating a fish tank with live plants. First, I want to make the fish happy. Second, a planted tank can cut down on algae. Third, well I wanted to do something different this time.

Wooooo are planted tanks different! A lot of the old adages from freshwater fish-only tanks are no longer applicable. I'm not going to bore you with the nitrogen cycle, there are videos for that. In short, fish eat food and create waste. The waste builds up in a tank and must be dealt with. Healthy aerobic and anaerobic bacteria can break these compounds down in well established tanks. Starter bacteria can help you with that, but it's not magical. The usual way of dealing with uneaten food and fish waste is frequent gravel vacuuming and water changes. The idea is that replacing the water on a regular basis and sucking up the muck will keep things clean...until the next session.

For most people, they can't handle this. Even then, things can happen and you have mass die-offs. Ammonia is the most dangerous compound that can build up in fish-only tanks and kill off our slippery friends quickly. A few days of not watching things and uh-oh...floaters. Oh, and don't confuse the nitrate cycle with "cycling" a tank. Folks use the "cycling" term with what happens to the typical fish-only aquarium. That is, they setup a tank and everything is fine with a few fish. Diatoms (brown algae aka "brown snow") that appear after a few weeks, only to be crowded out by algae a bit later, folks then try everything to get stuff in balance, and eventually a few weeks/months after that, things finally establish a harmony. Along the way, many fish are lost.

Planted tanks are...different. You still need to do big water changes when starting (bigger sometimes), but vacuuming the gravel isn't a thing. You have soil. It's hard to vacuum without sucking up the soil. Oh and all that stuff you want to get out? It's food for plants! Plants turn ammonia and nitrate into food. They take in CO2 and produce oxygen. Live plants complete the ecosystem for a low maintenance tropical aquarium. Fish provide fuel for plants, plants provide fuel for fish (oxygen and clean water). Plants do such a great job at keeping things clean, you'll often find no filters at all on a well established, well maintained, densely planted tank.

The tricky thing is keeping the plant growing thing in balance. Plants need high light levels to grow properly. High lights can also trigger algae growth. Plants also need CO2 with photosynthesis to produce oxygen. We take CO2 for granted when growing plants outside of aquariums. We have too much of it in the air right? Bill Gates is trying to bury it all in rocks. In an aquarium, the amount of CO2 in water is about 1/30th the amount in the atmosphere. So now you need CO2 gear to diffuse the gas into the tank. But oh yeah, too much CO2 and you'll kill your fish. I almost did this. I was trying out a new CO2 reactor below the tank, and I didn't realize that the flow rate was too high (this thing is VERY effective). Luckily Remy was right there and said, "Hey dad, look at the tetras. They're doing something funny in the corner." I'm glad he was there because I was eyes-down below the aquarium looking at this reactor. The fish were at the top of the tank, gasping for oxygen (and air -- even those without a labyrinth). I immediately did a huge water change and pointed the filter output up, creating massive water ripples (to bring in more oxygen).  The final toll was one panda cory, but I'm not sure that was actually from the CO2. I got it in a bad batch where the other compatriots died quickly -- one on the way home and the other within 24 hours. The LFS employee basically crushed them all while trying to catch them, the tank conditions were bad, and they had a bacterial/fungal growth on them. I couldn't see the growth because the store had very bad lighting. Pro tip, don't buy fish from a store that isn't lit up like Fenway Park.

But back to the CO2. Now you have yet another thing to keep in balance...fuckin great. Big if, but if you can, then you can have an algae free/minimal tank with healthy, happy fish. The idea is that the larger organism (plants) out compete the algae for resources (CO2, light, nutrients). So how do you get there? Next post. For now, enjoy some random videos from the early stages.

Trying to find an Amano for Dustin

I can watch shrimp all day

Pretty sure this fella was already sick when I got him. Don't think it was my accidental gassing.