20 April 2020

Fun with WordPress

Note – this is a discussion and solution for a technical problem for a WordPress instance that uses an SSL certificate signed by a non-public CA. If you don’t care about this sort of thing, please move your eyes down to the next section.

The error text that I saw in the new-to-me Site Health page following upgrading to WordPress 5.4:

cURL error 60: SSL certificate problem: unable to get local issuer certificate

The error above was generated because WordPress/PHP couldn’t verify the site certificate. When this is broken, the impact can be significant on a WordPress instance. Some features just don’t work quite right. Auto updating can fail, and so on.

The context here is that for a variety of internal and external sites, I use site-specific SSL certificates that are signed by our internal CA. That’s a good thing, because prior to Let’s Encrypt, it was easy to spend a bunch of money on SSL certificates from a reputable source. We won’t discuss the non-reputable sources. Since I’m using an external resource for caching and web app firewalling, I am able to use the internally signed certificate for several external sites as well.

With the most recent update adding Site Health as a core feature, this error surfaced for me on a couple of sites. It took a couple of hours and some false starts before I found this solution.

In the WordPress file tree, there’s a file at wp-includes/certificates/ca-bundle.crt (using UNIX-style slashes). This is the file of CA certificates that WordPress and the PHP functions use to verify a certificate is valid. Tryijg to get WordPress and PHP to use the system CA certs file (which has my Root Certificate added as a trust source) was a non-starter, although I tried. So I copied the text of my Internal Root Certificate into thewp-includes/certificates/ca-bundle.crt file. Boom! Problem solved … for now.

The downside of this solution is that any given WordPress update in the future may (will?) overwrite that file with newer info, and will once again exclude my Internal Root Certificate. So I created a text file that contained an identifying header string and the Internal Root Certificate. I then wrote a shell script to check thewp-includes/certificates/ca-bundle.crt for that header string, and if not found, adds the content of the text file to the ca-bundle.crt file. That shell script runs once a day in the wee hours of the morning.

Now, anytime there’s a WordPress update that overwrites ca-bundle.crt, by the next morning, the Internal Root CA certificate will be back in place, and things will continue humming along nicely.

Staying at Home

We continue to stay at home, which is a good thing.

I’ll ask you to determine for yourself if it’s a good thing that some people who, for reasons of politics, mistrust etc., continue to gather in groups, putting themselves and their loved ones at heightened risk of severe illness and death. I personally would rather that people be sane and safe. But bailing any water at all from the deeply stupid side of the gene pool can only be for the good of humanity, in the long term.

I didn’t do any yardwork this weekend. We did a number of other inside chores, including re-loading shelves and such after dealing with a multi-phased ant invasion.

Additionally, on the yardwork front, I will point out that planting veggies HAS brought the usual effects on to our region: We had two overnight frosts in the last week, and we’re due for one more on Tuesday night. I’ve been tarping the veggie beds for those events, and so far haven’t lost plants to them.

Happy Dog

While I was dealing with a training event late last week, I ran across the first picture we took of Lexi on her gotcha date in March 27, 2010:

Our first picture of Lexi the chipuggle mutt, taken on March 27, 2010.
Lexi’s First Photo Op

Winding Down

Nothing particular to report here. Be well, okay?

24 March 2019

Sad News

My friend Mark Camack died this last week, after a battle with throat cancer. We’ve been in the exchange holiday cards / biannual phone call place for the last couple of decades, but I’d still have jumped up and headed out if he needed help. I didn’t hear about this until his wife Bonnie got in touch, the other day. My heart goes out to Bonnie and their extended families. Rest in peace, my friend.

New Beginnings, Old Endings

The big news is that I migrated all of the personal sites I manage from an old server to a new server. Not super-exciting from an external perspective, but I did manage to separate the WordPress instances from being embedded in the old sites far too deeply. We *should* have had them be something like blog.orbdesigns.com, but then the site was already a blog, just pre-dating WordPress. So, anyway, www.mumble is the wordpress site for this place, and for Marcia’s two sites. The older, more static sites are more easily accessible via legacy URLs, for example legacy.orbdesigns.com.

One of the links on the legacy site that I clicked in testing was from December 28, 2009. And that was the day we said goodbye to Lucy, our cocker spaniel. So, “Old Endings.” There’s plenty of stuff there in legacy land, from . I’m probably going to fix just a couple of top-level internal links that will make the site work better.

I’m also changing horses on the two-factor authentication tools I’m using, both on the device and on the sites. So, lots of behind-the-scenes technology updates. Drop me an email, or comment here if you find something that’s so deeply broken that I absolutely must fix it. Of course, I may choose to leave something broken, but that’s another story.

Winding down

I’ve got a busy week in front of me, so pardon if I’m even less loquacious than usual.

Our condolences to the families and friends of these fallen warriors:

  • Spc. Joseph P. Collette, 29, of Lancaster, Ohio, died on March 22, 2019, in Kunduz Province, Afghanistan, as a result of wounds sustained while engaged in combat operations.
  • Sgt. 1st Class Will D. Lindsay, 33, of Cortez, Colorado, died on March 22, 2019, in Kunduz Province, Afghanistan, as a result of wounds sustained while engaged in combat operations.

7 Feb 2017

Wow. Am I a slacker, or what? I’ve been really busy. Work is keeping me on my toes, and (wonderfully), we’re finally cooking with gas!

Cooking with gas: Our new KitchenAid Dual Fuel range

Cooking with gas

We got a KitchenAid Dual Fuel range – a natural gas range (that could be converted to LP if needed) with two electric ovens. Yay! Getting the range was Marcia’s birthday present, and she got the gasfitter to run the line for Valentine’s Day. No massacres yet!

*      *      *

On the professional front, I’m working on the options for extending my Red Hat Certified Engineer status. I’m probably going down the automation path, with Ansible, for a variety of reasons. So I’ve got to spend a fair bit of time building test environments and building my skill set with the range of capabilities that Ansible offers today. I’ve been using it for a few years now, but not taking advantage of all that the tool suite has to offer. Should be fun.

*      *      *

Books: I finally finished reading Leviathan Wakes – Book One of The Expanse by James S. A. Corey. Wonderful space opera set believably in our solar system (so, no light speed drives required to move the action along). Miller and Holden. Holy cow. If you’ve not read, you should. I’ll be reading the books before I start watching the series, which I hear is also seriously awesome. Next up, Born To Run, by the Boss.

*      *      *

Lazy lookout guard - Lexi has to rest her head

Lazy lookout guard

Lexi had her annual check-up last weekend, and flew through with flying colors. The nail trimming and first round of shots didn’t make her very happy, though. She’ll be even less happy when she goes back for two more shots in a couple of weeks. The rabies vaccine booster was part of this year’s regimen, so the vet likes to split up the shots when there are a bunch, for a little dog like Lexi.

*      *      *

Our condolences to the family and friends of Pfc. Brian. P. Odiorne, 21, of Ware, Massachusetts, who died on Feb. 20, in Al Anbar Province, Iraq, from a non-combat related incident.

21 Dec 2015

A day late, and a vacation dollar short. Yep, vacation. I’m “off work” for the next two weeks, which means that I only keep an eye on email, and respond if SMS messages flow my way. But for the purposes of day-to-day operations, I’m offline. Yay!

*      *      *

In other good news, no vomiting in more than a week, so I’ve got that going for me.

*      *      *

Please note that policy requires the new disclaimer in the footer of this site. So noted.

*      *      *

The weekend flew by with assorted home-maintenance chores. They were mostly plumbing, which is mostly done – I still need a short length of 1-1/2″ pvc pipe, which I thought I had on-hand, but I was mistaken. So I also checked on my PVC cement, which I did have a can of … but it’s not a liquid as such, anymore. So that’s on the home center list, too.

I also managed to take some time to decommission some old data drives. For tin-foil-hat-reasons, I don’t just throw disks away or recycle them. I electronically wipe them, then destroy their ability to be read. Here’s the end result of one such session with 6 disks:

Data destroyed

Data destroyed

*      *      *

Today I got my sump pit monitoring system back online. For a variety of reasons, I broke it a couple of months ago, and neglected for a long while to get it back online. Today, that is remediated. The sump pit monitoring setup is well documented by Al Audet on his Raspi-Sump page, so I would be too redundant to repeat it all here. But his code works, so get it and use it. Yes, you’ll need a Raspberry Pi, and some assorted other stuff along with a bit of soldering or breadboarding skills, but that’s not hard to come by, and none of the stuff is so expensive that you can’t replace the bit you break. Better yet, it’s MASSIVELY less expensive in both time and money than what you’ll go through if your sump pit overflows. There are commercial monitors available. Ones that will also send you text messages are heinously expensive. Try Raspi-Sump, you’ll like it.

Side-note – I was introduced to Raspi-Sump on the pages of Linux Journal.

*      *      *

DoD announced no new casualties in the most recent week. Ciao!

Breakage, updated…

It turns out I didn’t break my WordPress install. But my home IP address *had* changed, and I have lots of things locked down to specific IP addresses for access. So I’ve got two of the three sites puzzled out. But one of them appears to have a broken theme directory. I’ll be looking into that now…

[A few minutes later, after looking at logfiles] Yup – the twentyeleven theme needed fixing on another site – I refreshed the whole thing, and I’ll keep a close eye on it in the next short while. I don’t remember mucking about in there … but it’s possible.

De-broken WP; Yardwork

Well, sometime last week, one of the changes that I made while working with “responsive” WordPress themes and site links broke the site. Not the part you see, but the administrative and posting functionality. Sadly, I’ve been busy enough that I didn’t catch this until last night late. So I just fixed it, and I’m taking a few minutes to catch up.

*      *      *

Over the weekend, I washed the car, planted flowers front and back, and planted about half of the expected veggies. More on that later, but here’s the Box of Pain (aka – peppers):

Peppers - April 2015

Peppers – April 2015

*      *      *

DoD announced no new casualties in the last week. Ciao!

Six Days of LISA ’13

Howdy. My name’s Brian, and I’m a tired SysAdmin…

So, six days of tutorials and talks at the USENIX LISA ’13 conference are done. And it was good. My behind is, however, glad to be shut of those hotel conference chairs.

Sunday, 3 November

Sunday’s full day tutorial was called Securing Linux Servers, and was taught by Rik Farrow, a talented bloke who does security for a living, and is Editor of the USENIX ;login: magazine on the side. We covered the goals of running systems (access to properly executing services) and the attacks that accessibility (physical, network) enable. As always, the more you know, the more frightening running systems connected to networks becomes. We explicitly deconstructed several public exploits of high-value targets, and discussed mitigations that might have made them less likely. User account minimization and root account lockdowns through effective use of the `sudo` command were prominently featured. Proactive patching is highly recommended, too! Passwords, password security, hashing algorithms, and helping users select strong passwords that can be remembered also were a prime topic. Things that Rik wished were better documented online are PAM (Pluggable Authentication Modules) and simple, accessible starter documentation for SELinux.

Monday, 4 November

Hands-on Security for Systems Administrators was the full-day tutorial I attended on Monday. It was taught by Branson Matheson, a consultant and computer security wonk. Branson is an extremely energetic and engaging trainer who held my attention the whole day. We looked at security from the perspective of (informally, in the class) auditing our physical, social, and network vulnerabilities. In the context of the latter, we used a customized virtual build of Kali Linux , a Debian-based pen testing distro. I learned a lot of stuff, and for those things that I “knew”, the refresher was welcome and timely.

Tuesday, 5 November

Tuesday, I took two half-day tutorials.

The first was presented by Ted Ts’o, of Linux kernel and filesystem fame. Our tutorial topic was “Recovering from Linux Hard Drive Disasters.” We spent a couple of hours covering disk drive fundamentals and Linux file systems. The final hour was given over to the stated topic of recovering from assorted disk-based catastrophes. My take-away from this tutorial was two-fold. I think the presentation be better named “Disks, Linux Filesystems, and Disk Disaster Recovery,” which would be more reflective of the distribution of the material. Additionally, it’s worth stating that any single disk disaster is generally mitigated by multi-disk configurations (mirroring, RAID), and accidental data loss is often best covered by frequently taken and tested backups.

The second tutorial I attended, on Tuesday afternoon, was on the topic of “Disaster Recovery Plans: Design, Implementation and Maintenance Using the ITIL Framework.” Seems a bit dry, eh? A bit … boring? Not at all! Jeanne Schock brought the subject material to life, walking us through setting goals and running a project to effectively plan for Disaster Recovery. IMO, it’s documentation, planning, and process that turns the craft of System Administration into a true profession, and these sorts of activities are crucial. Jeanne’s presentation style and methods of engaging the audience are superb. This was my personal favorite of all the tutorials I attended. But … Thanks, Jeanne, for making more work for me!

Wednesday, 6 November

Whew. I was starting to reach brain-full state as the fourth day of tutorials began. I got to spend a full day with Ted Ts’o this time, and it was an excellent full day of training on Linux Performance Tuning. Some stuff I knew, since I’ve been doing this for a while. But the methods that Ted discussed for triaging system and software behaviour, then using the resulting data to prioritize diagnostic activities was very useful. This is a recurring topic at LISA ’13 – go for the low-hanging fruit and obvious stuff: check for CPU, disk, and network bottlenecks with quick commands before delving into one path more deeply. The seemingly obvious culprit may be a red herring. I plan on using the slide deck to construct a performance triage TWiki page at work.

I was in this tutorial when Bruce Schneier spoke (via Skype!) on “Surveillance, the NSA, and Everything.” Bummer.

This was also my last day of Tutorials. In the evening I attended the annual LOPSA meeting. Lots of interesting stuff there, follow the link to learn more about this useful and supportive organization. Yep, I’m a member.

Thursday, 7 November

Yay, today started with track problems on Metro, and an extra 45 minutes standing cheek-to-jowl with a bunch of random folks on a Red Line train.

This was a Technical Sessions and Invited Talks day for me. In the morning, Brendan Gregg presented Blazing Performance with Flame Graphs. Here’s a useful summary on Brendan’s blog. This was followed in the morning by Jon Masters of Red Hat talking about Hyperscale Computing with ARM Servers (which looks to be a cool and not unlikely path), and Ben Rockwood of Joyent discussing Lean Operations. Ben has strong opinions on the profession, and I always learn something from him.

In the afternoon, Brendan Gregg was in front of me again, pitching systems performance issues (and his new book of the same name). I continue to find Brendan’s presentation style a bit over the top, but his technical chops and writing skills are excellent. This was followed by Branson Matheson (who was training me earlier in the week) on the subject of “Hacking your Mind and Emotions” – much about social engineering. Sigh, too easy to do. But Branson is so enthusiastic and excited about his work  that … well, that’s alright, then, eh?

The late afternoon pair of talks were on Enterprise Architecture Beyond the Perimeter (presented by a pair of talented Google Engineers), and Drifting into Fragility, by Matt Provost of Weta Digital. The former was all about authentication and authorization without the classical corporate perimeter – no firewall or VPN between clients and resources. Is it a legitimate client machine, properly secured and patched? With a properly authenticated user? Good, we’re cool. How much secured, authenticated, patched is required is dependent on the resource to be accessed. This seems a bit like a Google-scale problem… The latter talk, on fragility, was a poignant reminder of unintended dependencies and consequences in complex systems and network.

The conference reception was on Thursday evening, but I took a pass, headed home, and went to bed early. I was getting pretty tired by this time.

Friday, 8 November

My early morning session had George Wilson of Delphix talking about ZFS for Everyone, followed by Mark Cavage of Joyent discussing Manta Storage System Internals. I use ZFS, so the first talk held particular interest for me, especially the information about how the disparate ZFS implementations are working to prevent fragmentation by utilizing Feature Flags. OpenZFS.org was also discussed. I didn’t know much about Manta except that it exists, but I know a bit more now, and … it’s cool. I don’t have a use, today, but it’s definitely cool.

The late morning session I attended was a two-fer on the topic of Macs at Google. They have tens of thousands of Macs, and the effective image, deployment, and patching management was the first topic of the day, presented by Clay Caviness and Edward Eigerman. Some interesting tools and possibilities, but scale far beyond my needs. The second talk, by Greg Castle, on Hardening Macs, was pertinent and useful for me.

After lunch, the two talks I attended were on “Managing Access using SSH Keys” by the original author of SSH, Tatu Ylönen, and “Secure Linux Containers” by Dan Walsh of Red Hat (and SELinux fame). Tatu pretty much read text-dense slides aloud to us, and confirmed that managing SSH key proliferation and dependency paths is hard. Secure Linux Containers remind me strongly of sparse Solaris Zones, so that’s how I’m fitting them into my mental framework. Dan also talked to us about Docker … a container framework that Red Hat is “merging” (?) with Secure Linux Containers … and said we (sysadmins) wouldn’t like Docker at all. Mmmmmm.

The closing Plenary session, at about an hour and 45 minutes, was a caffeine-fueled odyssey by Todd Underwood, a Google Site Reliability Manager, on the topic of PostOps: A Non-Surgical Tale of Software, Fragility, and Reliability. Todd’s a fun, if hyper, speaker. He’s motivated and knows his stuff. But like some others in the audience, what happens at the scale of a GOOG-size organization may not apply so cleanly in the SMB space. The fact is that DevOps and NoOps may not work so well for us … though certainly the principles of coordinated work and automation strongly apply.

Brian’s Summary

At any given time, for every room I sat in, for every speaker or trainer I listened to, there were three other things that I would have also learned much from. This was my path through LISA ’13. There are many like it, but this one is mine. This conference was a net win for me in many ways – I learned a lot, I ran across some old friends (Hi, Heather and Marc), made some new ones, and had a good time.

The folks I can recommend without reservation that you take a class from, or attend a talk that they’re presenting: Jeanne Schock, Branson Matheson, Rik Farrow, and Ted Ts’o. These are the four people I learned the most from in the course of six days, and you’d learn from them, too!

My hat’s off to the fine staff at USENIX, who worked their asses off to make the conference work. Kudos!

Not insane after all

At least, I’m not insane for any of the usual reasons. A few times over the last couple of weeks, my Linux system here would be dark. When I got home from work … when I woke up in the morning. That’s not normal, because as the home storage, DNS, and mail server, off is a bad state to be in. I figured it was the system, though, and was waiting for some logged evidence of the sub-system I need to replace.

Last night, I left the Windows box running, too, mostly out of laziness. This morning, both systems were dark. Ah-HA! It’s the APC. This is a Back-UPS XS 1500, and manages power for the monitor and two systems. Time for a replacement.

*    *    *

I’ve had one negative review of me going to WordPress, and not even addressed to me, but on another site altogether. I suppose that I could maintain these in parallel, by posting in one place, then pasting in another. Do enough people feel strongly enough that I should add that work back into my schedule? Drop me a line and let me know. This change was for me, because it makes life easy, easy, easy. This is triple-true (that’s easy*9) of pictures, which I plan to do more of, but not if I have to also cross-post them into the static site.

*    *    *

On tap for today:

  • Feed and water the front flowers
  • New UPS
  • Roast coffee
  • Cleaning up further in basement
  • Keep an eye on work email – big project going on
  • Get another project ahead in classes

I think that’s enough to occupy my day.

First Post! Natali Portman and Hot Grits!



Yeah, I’ve always wanted to have my own little bit of Slash* over here.

If you’re looking for the older stuff, you can start over there, on the Site Map. Search on the main site is currently broken, and that’s on my to-do list.

So, WordPress reappears on the OrbDesigns site. I’m going to be moving Marcia over to WP soon, and figured I’d better be using it first. Hmmm … what about pictures?

2011 Garden, May 30

2011 Garden, May 30

Hey, pictures are easy, too! That’ll be handy. Now, back to work.