Class V Software Logo

Feb 28, 2024

RTFM
last update: 02/28 @ 18:29

Well, that’s likely something I should have done but didn’t in the last few weeks.

As noted earlier, I had a power supply fail and the server was old enough it was not clear I could replace it. Also, I’d had a hard drive fail on the same server a few months back. Both have been through a remodel at the house and so it was time to replace them.

For the first server, I was really under the gun: I wanted to get our internal network back up and running quickly. Clearly no time for manual reading.

For the second server, just recently “Completed,” I was not under quite as much personal pressure. But I still just installed the OS (yes, two major versions more recent than what I had), installed the software and laid down my configurations. It likely would have been a good idea to read the release notes to understand what all had changed, but I was too lazy or thought I could figure it out as I went.

Which, I’m happy to say, I could (thank you Internet).

If you happen to be jumping from RHEL 7 (or, more likely CentOS 7) to RHEL 9 (or, more likely Rocky 9), here are the final couple gotchas I ran into:

External FirewallD forwarding/routing change
With the changes to FirewallD and the underlying firewall layer, my old forwarding didn’t work. I saw a lot of this in the logs, “filter_FWD_internal_REJECT: IN=eno1 OUT=eno2” The fix, thanks to Internet searches:
firewall-cmd —permanent —new-policy allowForward
firewall-cmd —permanent —policy allowForward —set-target ACCEPT
firewall-cmd —permanent —policy allowForward —add-ingress-zone internal
firewall-cmd —permanent —policy allowForward —add-egress-zone public
firewall-cmd —reload
Some updates needed for ruby scripts
The new, modern Ruby doesn’t like instance variables (for good reasons). The quick (but perhaps less than ideal) fix was to make them global variables. My scripts run one at a time, no worry about race conditions so likely that’s OK.
Testing red network with only default route
I thought I was being smart, I added my public IP addresses to the new server while I was building it and then went to test by plugging it into a switch that only had my laptop connected. I gave my laptop an IP in the same class C network and started testing. And nothing worked. I had the server’s default route set up to go out the internal side so I could use the current Internet connection to get updated software. Once I added a route to that class C on the NIC, I could test and see traffic flowing.

My new servers have redundent power supplies now — I learned that lesson. But maybe I didn’t learn the hard disk lesson: I’m not doing any sort of RAID, just frequent backups. And, thanks to the new servers and new disks, I have my old disks as spares should I need them.

Now I can say I’m finishing up some post migration work; prepping for Ting as our new ISP, and getting on with life.


Jan 28, 2024

Recovering
last update: 01/28 @ 23:19

Win some and lose some…

Refurbished power supply was a bust; wrong item (it was hot swapable, I didn’t have redundent, hot swapable — but I do now).

So I’ve moved on to server rebuild. And with Centos pretty much gone, seemingly a causality of IBM’s purchase of Red Hat, I’m moving on to Rocky Linux. Going from 7.X to 9.X was bound to have some growing pains (having them in a more controlled build and cutover rather than replacing a dead server would have been nicer).

Lessons from my experience:

ssh
First, Rocky uses AuthorizedKeys vs. Centos 7 using AuthorizedKeys2; not a big problem but my first, bad, assumption was the upgrade meant I needed newer keys. And then (did I mention I was trying to go fast), I just used default file permissions on the new file. Final stumbling block: the new key pushed me over the limis so I started to get “too many failed attempts” errors. Lesson: copy AuthorizedKeys2 to AuthorizedKeys and check the file permissions.
samba
It’s been a long time since I set it up. First, bad assumption: samba users are pulled from OS users. Eventually the errors made sense and I remembered I had to create the samba users/passwords. Second issue: selinux boolean to allow samba to get to home directories — that generates a really weird error on Linux workstation mapping in those drives (not sure what Windows would have looked like). Lessons: create the samba users/passwords and ensure selinux is permitting home directories (or relabel the file system).
httpd
Really went pretty well: don’t forget to install CRS with mod_security. And the new rulesets added a few more false positives for subversion access.
subversion
Speaking of subversion; it just works. I was pleasantly surprised to see no issues with access the repositories via httpd (after the mod_sec updates). (Do be sure selinux context is correct.)
postfix and spamassassin
It took a lot longer than it should have to understand that the error message about pipe failed due to unknown user really meant just that. The Spamassassin package for CentOS 7 used the “spamfilter” user (I think); it seems like the Rocky 9 package piggybacks off of the “mail” user.
dovecot
I had to refresh my memory on SSL keys for dovecot; but mistakenly figured I could just remove the Thunderbird saved key and add back the new key (see below). For me, dovecot uses standard key files in standard tls directory. I was surprised that dovecot seems to log Thunderbird’s ssl error which Thunderbird seemed to just mask.
Thunderbird
When I hit the certificate issue, I thought (next bad assumption) I could just remove the key I loaded from the old server and add a new exception for the new server’s key. I spent a long time trying but could never coach Thunderbird to ask for that offending key and give me a chance to accept it.
Thunderbird
So I remembered having to create a new profile not that long ago when the hard drive failed on the old server. (The power supply was the camel back breaking straw.) So I created a new profile and that let me import the certificate. But…
Thunderbird
The Thunderbird new profile/account set up presumed username would include domain (davewill@kayakero.net, for example). Of course, dovecot on Linux, using Linux accounts just needed the name. The unknown user / invalid password threw me for my final loop of the rebuild session. Lessons: Delete imported SSL cert only as a last resort and look closely at the assumptions made in setting up a new profile/account.
tar
Final note this this entry: when I started my backup scripts, I was getting astronomically large backup files. tar, in Rocky 9 does not like “—exlude” clause after the source of the files to tar up. CentOS 7 was OK with that. Lesson: tar .

The server has been working well for a few days now and I’m feeling pretty good. I have a bunch more disk space to work with (but I’ll likely need new, bigger USB drives for offsite storage). And I’ve seen a whole new slew of error messages and eventually worked out what they mean and what I need to do.

Of course, the public server (serving this blog and website) is the same era as the internal server and lived through the same very dusty remodel. So that’s next on my plate…

Oh, and Ting is setting up fiber to the house too…


Jan 13, 2024

Ack!!!
last update: 01/13 @ 13:12

I may be a bit distracted for a while.

I can now say I’ve experienced a power supply failure for a server — new to me before today. I knew these needed to be replaced, I was just hoping for 6 more months…

New (refurbished) power supply on order — if that’s really the cause (could be motherboard as well), I may be up and limping in a week or so.

New servers ordered as well — clearly it was time, but I was hoping not to be forced into a rush job :( And, learning my lesson, redundant power this time; that seemed excessive for home use before today).

Oh, and Ting says they may be able to hook us up next month.

Gonna be a techy few weeks here…


Jan 03, 2024

Christmas photos
last update: 01/03 @ 15:05

Sadly for us, Robert had to go back to school. We had a lot of fun while he was here and I expect we’ll start plotting our next trip out that way soon.

There are some more photos in the vacations section.


Jan 01, 2024

Happy 2024!!!
last update: 01/01 @ 00:51

Hoping for a GREAT year!

It was fantastic to have Robert join us for the fireworks, grapes under the table, sweeping the porches and running around the house.

You know, the usual :)

Also very nice to be invited to Will, Rob, Christine and Cassie’s house for their balloon drop party.


Dec 29, 2023

Cool, but better water level
last update: 12/29 @ 01:14

Robert was not feeling well but back asleep.

So I took advantage of the time off of work and nicer weather to get on the river.

It’s been raining for a few days so higher water than it’s been in a while — and the temperature was not too bad. I was still consciously conservative to try to stay upright (which I did) as the water was pretty cool.


Dec 25, 2023

Merry Christmas!!!
last update: 12/25 @ 13:08

We mostly waited patiently for Christmas this year.

There were some catnip treats Robert found for Panzer that he was interested in.

But no early unwrapping.

It was great to wake up and wander down to reveal all the fun gifts. Surprisingly, Robert was up before us.

The second time since he’s been home this trip; the first time he was still getting over the travel.

And now he’s back taking a nap.

I’ll have more photos to post later.


Dec 16, 2023

He’s back :)
last update: 12/16 @ 22:25

Robert flew back Wednesday and it’s been great to have him home again.

We celebrated my birthday with the traditional Los Tios dinner (I’ll have to see if Sarah has a photo) and have been enjoying the chance to play some games after picking up a tree and adding our decorations (I’ll need to add more photos of that as well).


Dec 10, 2023

Cold water, warm air
last update: 12/10 @ 15:47

The rains arrived overnight, along with warmer air from the south. That, combined, with the cold Potomac, resulted in some impressive river fog.

Sarah and I walked down to take a look this afternoon between the early rain and the expected storms this evening.

It was a pretty odd site, hard to capture: dense fog from the surface of the river to about 20 feed above it, then it cleared out (but under cloudy skies, harder to see).


Dec 09, 2023

Not so much fall anymore
last update: 12/09 @ 19:51

The heavens are supposed to open tomorrow and we expect to get dumped on by rain.

So Sarah and I took advantage of the warmer (note warmer, not warm) weather to head to Angler’s Inn.

She hiked up the Billy Goat trail and found the secret path to the top of the Maryland chute. Check out her Facebook page for her view.

I stuck with my canoe.

The water was “bracing” :)

And I noticed most of the other folks on the river were in drysuits. But I was warm enough except for my feet — my toes were getting cold.

But that does not detract from how pretty a spot it is, even with the leaves off the trees.

And I got to check in with my favorite heron again (top right corner); I bet he or she does not head south for the winter. This was on the Virginia shore; Sarah said she saw one as well, so it seems like there is a heron family in residence there.



Add new entry (owner only)

The posts on this page will slowly roll off as new ones are added to the top. The "permanent link" links above will take you to one post's permanent address; that should not change or disappear. You can also build up a link to see any month's postings by adding the four digit year, a slash, the two digit month and a trailing slash to the the main www.kayakero.net/news/blosxom URL. Like this:
/news/blosxom/2004/08/. (You can go down to the day level if you like.)

Only the site owners can edit this page (and all attempts to do so are logged); however anyone is welcome to add a comment using the "comments" link below each posting.

RSS feeds: .91 | 3.0