21 Dec 2015

A day late, and a vacation dollar short. Yep, vacation. I’m “off work” for the next two weeks, which means that I only keep an eye on email, and respond if SMS messages flow my way. But for the purposes of day-to-day operations, I’m offline. Yay!

*      *      *

In other good news, no vomiting in more than a week, so I’ve got that going for me.

*      *      *

Please note that policy requires the new disclaimer in the footer of this site. So noted.

*      *      *

The weekend flew by with assorted home-maintenance chores. They were mostly plumbing, which is mostly done – I still need a short length of 1-1/2″ pvc pipe, which I thought I had on-hand, but I was mistaken. So I also checked on my PVC cement, which I did have a can of … but it’s not a liquid as such, anymore. So that’s on the home center list, too.

I also managed to take some time to decommission some old data drives. For tin-foil-hat-reasons, I don’t just throw disks away or recycle them. I electronically wipe them, then destroy their ability to be read. Here’s the end result of one such session with 6 disks:

Data destroyed

Data destroyed

*      *      *

Today I got my sump pit monitoring system back online. For a variety of reasons, I broke it a couple of months ago, and neglected for a long while to get it back online. Today, that is remediated. The sump pit monitoring setup is well documented by Al Audet on his Raspi-Sump page, so I would be too redundant to repeat it all here. But his code works, so get it and use it. Yes, you’ll need a Raspberry Pi, and some assorted other stuff along with a bit of soldering or breadboarding skills, but that’s not hard to come by, and none of the stuff is so expensive that you can’t replace the bit you break. Better yet, it’s MASSIVELY less expensive in both time and money than what you’ll go through if your sump pit overflows. There are commercial monitors available. Ones that will also send you text messages are heinously expensive. Try Raspi-Sump, you’ll like it.

Side-note – I was introduced to Raspi-Sump on the pages of Linux Journal.

*      *      *

DoD announced no new casualties in the most recent week. Ciao!

Breakage, updated…

It turns out I didn’t break my WordPress install. But my home IP address *had* changed, and I have lots of things locked down to specific IP addresses for access. So I’ve got two of the three sites puzzled out. But one of them appears to have a broken theme directory. I’ll be looking into that now…

[A few minutes later, after looking at logfiles] Yup – the twentyeleven theme needed fixing on another site – I refreshed the whole thing, and I’ll keep a close eye on it in the next short while. I don’t remember mucking about in there … but it’s possible.

De-broken WP; Yardwork

Well, sometime last week, one of the changes that I made while working with “responsive” WordPress themes and site links broke the site. Not the part you see, but the administrative and posting functionality. Sadly, I’ve been busy enough that I didn’t catch this until last night late. So I just fixed it, and I’m taking a few minutes to catch up.

*      *      *

Over the weekend, I washed the car, planted flowers front and back, and planted about half of the expected veggies. More on that later, but here’s the Box of Pain (aka – peppers):

Peppers - April 2015

Peppers – April 2015

*      *      *

DoD announced no new casualties in the last week. Ciao!

Six Days of LISA ’13

Howdy. My name’s Brian, and I’m a tired SysAdmin…

So, six days of tutorials and talks at the USENIX LISA ’13 conference are done. And it was good. My behind is, however, glad to be shut of those hotel conference chairs.

Sunday, 3 November

Sunday’s full day tutorial was called Securing Linux Servers, and was taught by Rik Farrow, a talented bloke who does security for a living, and is Editor of the USENIX ;login: magazine on the side. We covered the goals of running systems (access to properly executing services) and the attacks that accessibility (physical, network) enable. As always, the more you know, the more frightening running systems connected to networks becomes. We explicitly deconstructed several public exploits of high-value targets, and discussed mitigations that might have made them less likely. User account minimization and root account lockdowns through effective use of the `sudo` command were prominently featured. Proactive patching is highly recommended, too! Passwords, password security, hashing algorithms, and helping users select strong passwords that can be remembered also were a prime topic. Things that Rik wished were better documented online are PAM (Pluggable Authentication Modules) and simple, accessible starter documentation for SELinux.

Monday, 4 November

Hands-on Security for Systems Administrators was the full-day tutorial I attended on Monday. It was taught by Branson Matheson, a consultant and computer security wonk. Branson is an extremely energetic and engaging trainer who held my attention the whole day. We looked at security from the perspective of (informally, in the class) auditing our physical, social, and network vulnerabilities. In the context of the latter, we used a customized virtual build of Kali Linux , a Debian-based pen testing distro. I learned a lot of stuff, and for those things that I “knew”, the refresher was welcome and timely.

Tuesday, 5 November

Tuesday, I took two half-day tutorials.

The first was presented by Ted Ts’o, of Linux kernel and filesystem fame. Our tutorial topic was “Recovering from Linux Hard Drive Disasters.” We spent a couple of hours covering disk drive fundamentals and Linux file systems. The final hour was given over to the stated topic of recovering from assorted disk-based catastrophes. My take-away from this tutorial was two-fold. I think the presentation be better named “Disks, Linux Filesystems, and Disk Disaster Recovery,” which would be more reflective of the distribution of the material. Additionally, it’s worth stating that any single disk disaster is generally mitigated by multi-disk configurations (mirroring, RAID), and accidental data loss is often best covered by frequently taken and tested backups.

The second tutorial I attended, on Tuesday afternoon, was on the topic of “Disaster Recovery Plans: Design, Implementation and Maintenance Using the ITIL Framework.” Seems a bit dry, eh? A bit … boring? Not at all! Jeanne Schock brought the subject material to life, walking us through setting goals and running a project to effectively plan for Disaster Recovery. IMO, it’s documentation, planning, and process that turns the craft of System Administration into a true profession, and these sorts of activities are crucial. Jeanne’s presentation style and methods of engaging the audience are superb. This was my personal favorite of all the tutorials I attended. But … Thanks, Jeanne, for making more work for me!

Wednesday, 6 November

Whew. I was starting to reach brain-full state as the fourth day of tutorials began. I got to spend a full day with Ted Ts’o this time, and it was an excellent full day of training on Linux Performance Tuning. Some stuff I knew, since I’ve been doing this for a while. But the methods that Ted discussed for triaging system and software behaviour, then using the resulting data to prioritize diagnostic activities was very useful. This is a recurring topic at LISA ’13 – go for the low-hanging fruit and obvious stuff: check for CPU, disk, and network bottlenecks with quick commands before delving into one path more deeply. The seemingly obvious culprit may be a red herring. I plan on using the slide deck to construct a performance triage TWiki page at work.

I was in this tutorial when Bruce Schneier spoke (via Skype!) on “Surveillance, the NSA, and Everything.” Bummer.

This was also my last day of Tutorials. In the evening I attended the annual LOPSA meeting. Lots of interesting stuff there, follow the link to learn more about this useful and supportive organization. Yep, I’m a member.

Thursday, 7 November

Yay, today started with track problems on Metro, and an extra 45 minutes standing cheek-to-jowl with a bunch of random folks on a Red Line train.

This was a Technical Sessions and Invited Talks day for me. In the morning, Brendan Gregg presented Blazing Performance with Flame Graphs. Here’s a useful summary on Brendan’s blog. This was followed in the morning by Jon Masters of Red Hat talking about Hyperscale Computing with ARM Servers (which looks to be a cool and not unlikely path), and Ben Rockwood of Joyent discussing Lean Operations. Ben has strong opinions on the profession, and I always learn something from him.

In the afternoon, Brendan Gregg was in front of me again, pitching systems performance issues (and his new book of the same name). I continue to find Brendan’s presentation style a bit over the top, but his technical chops and writing skills are excellent. This was followed by Branson Matheson (who was training me earlier in the week) on the subject of “Hacking your Mind and Emotions” – much about social engineering. Sigh, too easy to do. But Branson is so enthusiastic and excited about his work  that … well, that’s alright, then, eh?

The late afternoon pair of talks were on Enterprise Architecture Beyond the Perimeter (presented by a pair of talented Google Engineers), and Drifting into Fragility, by Matt Provost of Weta Digital. The former was all about authentication and authorization without the classical corporate perimeter – no firewall or VPN between clients and resources. Is it a legitimate client machine, properly secured and patched? With a properly authenticated user? Good, we’re cool. How much secured, authenticated, patched is required is dependent on the resource to be accessed. This seems a bit like a Google-scale problem… The latter talk, on fragility, was a poignant reminder of unintended dependencies and consequences in complex systems and network.

The conference reception was on Thursday evening, but I took a pass, headed home, and went to bed early. I was getting pretty tired by this time.

Friday, 8 November

My early morning session had George Wilson of Delphix talking about ZFS for Everyone, followed by Mark Cavage of Joyent discussing Manta Storage System Internals. I use ZFS, so the first talk held particular interest for me, especially the information about how the disparate ZFS implementations are working to prevent fragmentation by utilizing Feature Flags. OpenZFS.org was also discussed. I didn’t know much about Manta except that it exists, but I know a bit more now, and … it’s cool. I don’t have a use, today, but it’s definitely cool.

The late morning session I attended was a two-fer on the topic of Macs at Google. They have tens of thousands of Macs, and the effective image, deployment, and patching management was the first topic of the day, presented by Clay Caviness and Edward Eigerman. Some interesting tools and possibilities, but scale far beyond my needs. The second talk, by Greg Castle, on Hardening Macs, was pertinent and useful for me.

After lunch, the two talks I attended were on “Managing Access using SSH Keys” by the original author of SSH, Tatu Ylönen, and “Secure Linux Containers” by Dan Walsh of Red Hat (and SELinux fame). Tatu pretty much read text-dense slides aloud to us, and confirmed that managing SSH key proliferation and dependency paths is hard. Secure Linux Containers remind me strongly of sparse Solaris Zones, so that’s how I’m fitting them into my mental framework. Dan also talked to us about Docker … a container framework that Red Hat is “merging” (?) with Secure Linux Containers … and said we (sysadmins) wouldn’t like Docker at all. Mmmmmm.

The closing Plenary session, at about an hour and 45 minutes, was a caffeine-fueled odyssey by Todd Underwood, a Google Site Reliability Manager, on the topic of PostOps: A Non-Surgical Tale of Software, Fragility, and Reliability. Todd’s a fun, if hyper, speaker. He’s motivated and knows his stuff. But like some others in the audience, what happens at the scale of a GOOG-size organization may not apply so cleanly in the SMB space. The fact is that DevOps and NoOps may not work so well for us … though certainly the principles of coordinated work and automation strongly apply.

Brian’s Summary

At any given time, for every room I sat in, for every speaker or trainer I listened to, there were three other things that I would have also learned much from. This was my path through LISA ’13. There are many like it, but this one is mine. This conference was a net win for me in many ways – I learned a lot, I ran across some old friends (Hi, Heather and Marc), made some new ones, and had a good time.

The folks I can recommend without reservation that you take a class from, or attend a talk that they’re presenting: Jeanne Schock, Branson Matheson, Rik Farrow, and Ted Ts’o. These are the four people I learned the most from in the course of six days, and you’d learn from them, too!

My hat’s off to the fine staff at USENIX, who worked their asses off to make the conference work. Kudos!

Not insane after all

At least, I’m not insane for any of the usual reasons. A few times over the last couple of weeks, my Linux system here would be dark. When I got home from work … when I woke up in the morning. That’s not normal, because as the home storage, DNS, and mail server, off is a bad state to be in. I figured it was the system, though, and was waiting for some logged evidence of the sub-system I need to replace.

Last night, I left the Windows box running, too, mostly out of laziness. This morning, both systems were dark. Ah-HA! It’s the APC. This is a Back-UPS XS 1500, and manages power for the monitor and two systems. Time for a replacement.

*    *    *

I’ve had one negative review of me going to WordPress, and not even addressed to me, but on another site altogether. I suppose that I could maintain these in parallel, by posting in one place, then pasting in another. Do enough people feel strongly enough that I should add that work back into my schedule? Drop me a line and let me know. This change was for me, because it makes life easy, easy, easy. This is triple-true (that’s easy*9) of pictures, which I plan to do more of, but not if I have to also cross-post them into the static site.

*    *    *

On tap for today:

  • Feed and water the front flowers
  • New UPS
  • Roast coffee
  • Cleaning up further in basement
  • Keep an eye on work email – big project going on
  • Get another project ahead in classes

I think that’s enough to occupy my day.

First Post! Natali Portman and Hot Grits!



Yeah, I’ve always wanted to have my own little bit of Slash* over here.

If you’re looking for the older stuff, you can start over there, on the Site Map. Search on the main site is currently broken, and that’s on my to-do list.

So, WordPress reappears on the OrbDesigns site. I’m going to be moving Marcia over to WP soon, and figured I’d better be using it first. Hmmm … what about pictures?

2011 Garden, May 30

2011 Garden, May 30

Hey, pictures are easy, too! That’ll be handy. Now, back to work.