September 22, 2003
My work involves technology, and lots of it. In many ways I'm on the cutting edge of things; if I'm not using it, I've worked with it in some capacity.
I've seen videoconferencing that brought people from around the world into the same room; pictures from the Hubble telescope are readily available on the web, showing us pictures of galaxies that are nearly beyond imagination. Even the technology that has gone into allowing me to write this web page is astonishing; the servers, from routers and networking that allows me to connected to the programs that run on the servers that make the web what is is today.
Technology has grown by phenomenal leaps even in the relatively short time I've been in the business. For 15 years I've seen things grow, and move and expand.
So I sit here in complete awe as I discover how completely SCREWED UP our banking system is these days. To understand what I'm about to say you need to know that I have 2 checking accounts, one for the house upgrades & maintenance, and one for the regular bills.
This past Saturday I made a deposit...but I deposited the money into the house account when I wanted to put it into the bills account. Then I wrote checks to cover the bills, and sent them out.
Sunday I discover my mistake and get online to transfer the money, but it's not showing up in my house account. I figure that shouldn't be a problem, I'll call the bank on Monday and have them do it.
On Monday I go to the bank and tell them what had happened. They're very nice, but they tell me that nothing is posted until midnight Monday and there's nothing they can do about it. The following is the exchange as I remember it:
Me: "So, two days ago I put money into one of YOUR banks. I write checks on that money. I admit that I made a mistake and put the money into the wrong account, but it's an account in YOUR bank. And you're telling me that I'm going to have checks bounce, and have to pay your check fees, because you can't access money that was put into your bank two days ago?"
Them: "I'm sorry, but it's been that way forever."
EXCUSE ME? "It's been that way forever" is their answer? How about, "Since you've been a customer of ours for years, and in the past 12 years have bounced a total of one check which turned out to be our fault, we'll put a note on your account not to charge you the bank fees that come through".
THAT would have been a good answer.
I did call the branch office and explained what I went through, and said if they could waive the fees that would be good. But that isn't their policy, and they were very sorry but they really couldn't help me.
Thankfully I was able to deposit enough to cover the checks, and I was able to pull the amount of my other account the following day.
Do I need to mention I'll be looking for a different bank soon?
September 17th, 2003
Today would have been my dad's 65th birthday, had he lived to see it.
The hardest realization that I've ever had to face is the fact that I am going to outlive my parents.
It was shortly after Thanksgiving several years ago when my father took seriously ill. I think we all knew this was the end, even though none of us could admit it. Maybe not even to ourselves.
Seven years earlier my dad had a massive heart attack and required triple bypass surgery and a valve replacement. It extended his life, but the drugs they had to give him to keep him alive slowly deteriorated other parts of his body. I remember him as a strong man, and a caring father. So many good memories of my childhood come back to me when I think of him.
In some ways those memories made it very difficult to visit him in the hospital. He had lost a lot of weight in the previous few months and was no longer the large man I remembered. At one point he wanted to reposition himself on his bed and needed me and my brothers help. It surprised me how light he was, but more than anything it shook me that he needed my assistance.
I remember watching the nurses pump morphine into his IV, and thinking how dangerous that was. That's when it dawned on me: they weren't concerned with him becoming addicted because they knew he wasn't going to be around that long. He had devoted his life to taking care of his family, and in the end it all came down to this; those machines weren't there to keep him alive. They were there to make him comfortable while he died.
We could tell the end was near. He was occasionally coherent, but mostly he slept. One time when he was kind of lucid my brother leaned over and whispered "greens and fairways dad", referring to my fathers love of golf. For the life of me I couldn't think of the right words to tell him, something he could take with him. Then I realized that I didn't need any fancy words. We had spent years exchanging thoughts and ideas, talking about many things that seemed so minor to me at the time.
All his life he was a blue collar worker, and I can remember his pride when I told him I got a job as a Systems Administrator. "You don't have a job anymore, Brian" he said. "You have a *position*." It wasn't until I had kids of my own that I was able to understand the look of happiness on his face. I remember talking to him about his business, or about how he felt when his first grandchild was born.
I wasn't there when he died. My brother and I had gone to my brother's house to catch a few hours of sleep before going back to the vigil. People that I'm close to, and who know what happened, ask me if I'm troubled that I wasn't able to give my dad any last words.
But my dad and I had lots of words. Many, many words over a lot of years. They made me the person I am today; they gave me my moral compass, my work ethic, the pride in my family, and the love that I have. So many things he was able to give me, and so many things I wanted to tell him. So I told him the best way I knew how.
The last words he heard from me were "I love you, dad."
And I still do.
September 14th, 2003
A 70 hour week last week; scheduled for an upgrade that was supposed to take 16 hours & I wound up working 40 hours in 3 days. I feel surprisingly good for not having much sleep in the past week. And my upstairs bathroom is in the final stages of being finished; things seem to be moving along pretty well.
One incident over the weekend did disturb me, in that "gee...I thought I knew you better than that" way. A friend of mine turned down a date because they didn't care for the hobby of the person asking them. It was an common hobby, and pretty harmless. But the date was lost because the person enjoyed doing something that everyone would find rather innocuous.
Going for a dip in the shallow end of the gene pool, I suppose.
September 11th, 2003
Ever notice how disasters are always expressed in dollar amounts? "Hurricane XXX did over $2 billion in damage" is an expected phrase whenever a storm hits the US. Tornadoes have an F rating that is based on, you guessed it, monetary damage. Listen to the newscast some time there's a fire and there will always be a damage estimate. It seems to be the only way we can grasp how large a tragedy is, we have to put a dollar amount on it.
Almost always, that is.
One observation that I am glad to see: The twin towers tragedy is measured in lives, not in dollar amount. I know that around 2500 people lost their lives two years ago because that's what folks talk about. I know there are damage estimates out there, but I would have to search for them. But I don't need to.
I know everything I need to know.
September 10, 2003
Just back from an upgrade that was supposed to take 16 hours, but wound up taking 41 hours. I've worked 41 hours in 3 days topped off with a 150 mile drive each way. I've done the "meat grinders" at work before....don't much care for them. But the customer is up and running and that's what counts.
I did see a blurb on the net that said Rosanne Barr stated that her show was cancelled because she told the ABC exec's that she's having a hysterectomy, so they dropped the show because they're sexist.
This woman need to come into reality. For those of you that missed the 15 seconds the show was available it was a reality show about the making of a cooking show with Roseanne. It was painful to watch, the ratings were non-existent and ABC dropped that turkey fast.
That doesn't show that they're sexist, Roseanne. It shows that they are mildly observant.
One of my friends has started a web log, check it out at http://www.livejournal.com/users/crates/ . Lots of stuff on the Iraqi war. Be forewarned if you dare to leave comments: he's very good, and very knowledgeable. If you have an opinion you better be able to back it up.
September 4, 2003
This evening when I got home my living room was a bit messy. I have kids & they tend to take cleaning pretty lightly; they're much better at messing things up.
My wife got mildly upset at one point and was telling my daughters "This place is a mess. Someone's socks are over there, and the folder should be put up. You should put things back where you got them".
The kids, of course, continue watching SpongeBob and seem completely oblivious to all of this.
"Oh nice" says my wife. "Someone left my sweater on the floor".
I call to my oldest daughter, who is watching television. I have to call her name several times before she turns to me. "What did mommy just say?" I ask.
"Umm....that we left our stuff all over the living room?" she replies.
"What else?"
She looks like I've just caught her stealing cookies. I can tell she is completely clueless; she has no idea what my wife just said. Time for a lesson, I think.
"Sarah, I'll give you five dollars if you can tell me the last thing mommy said". Five dollars to her is a fortune, this should certainly help teach her that she should listen when her mother speaks!
"Um...that our folder should be put back. And that someone put her sweater on the floor!" she answers triumphantly.
My wife is very supportive of me, once she's done laughing. I'm shaking my head in wonder, trying to figure out how I suckered myself into this.
Oh yes, there was a lesson taught this night. But it was my daughter who was the teacher, and I the student.
September 2, 2003
Such a lovely way to come back to work. Yesterday (Tuesday) on of my users sends me an email stating that an email he sent to a customer was blocked by spamcop.
MY server? Blocked for sending spam? This cannot be! I check the
log (in this business you never take anything for granted) and, lo and behold, I
am shocked to find over 2000 messages. And none of them have an originator that
resides in my domain.
Various hilarity ensues; hours of checking, researching and testing my server.
Could my port forwarding system be compromised? Or is it my Exchange server? I
know I'm not an open relay, all the tests prove that. And I've got all the
patches and hotfixes applied; I checked and everything was up to date...yet I
was still sending spam.
I hit the newsgroups and begin searching. And searching...and testing, and
searching. I set up logging on Exchange so I can trace the problem.
Exchange logging, I discover, is next to useless. It can keep copies of
all sent mail, so I had a lot of sample spam. Like I really needed that. And it
can tell me who I'm connecting to, which really isn't my problem. I want to know
the FROM person, the one who is connecting into my server.
I dropped Ethereal onto the Exchange box and started logging. Do I need to
mention that Ethereal caused my machine to BSOD? No, of course not. Since it's a
Microsoft OS that is simply assumed. So I set Ethereal to write to a file
instead of a screen and I was able to capture packets when the spammer started
spamming.
I turn off the Internet Mail Connector to stop the sending of spam and open up
my capture file. I am told that 1 MB = approx. 640 typed pages and after the
last day and a half that sounds about right.
After I poured through the capture, trying to decipher what I was looking at
(and calling several friends to add to the brain trust) I narrow it down to the
sequence where the spammer first logs in. It has an AUTH LOGIN with a hashed
Username and Password.
To someone who is familiar with mailbox protocol this is probably the equivalent
of a Dr. Seuss primer to a college graduate. To me it was nonsense & I spent
several hours trying to figure out what it all means. The most interesting part,
to me, was the 334 sequences. Those looked like hashed passwords; in fact, they
looked like UUE encoding. (Hey, downloading all that porn years ago finally paid
off!)
So I take a bit and find a uudecode program that will allow me to decode these
things to a file. Unhashed here's what it reads:
9/2/2003 6:18:45 PM : <<< IO: |AUTH LOGIN
|
9/2/2003 6:18:45 PM : <<< AUTH LOGIN
9/2/2003 6:18:45 PM : >>> 334 Username
9/2/2003 6:18:46 PM : <<< IO: |Administrator
|
9/2/2003 6:18:46 PM : <<< Administrator
9/2/2003 6:18:46 PM : >>> 334 Password
9/2/2003 6:18:46 PM : <<< IO: |
|
9/2/2003 6:18:46 PM : <<< ########
9/2/2003 6:18:46 PM : >>> 235 LOGIN authentication successful
OK...WTF? WTF? WTF? Someone is logging in as Administrator???? With no
password?? And Exchange is ALLOWING it??
Time for a drink. It's now roughly 11:00 pm and I'm going absolutely insane. My
server, which (in theory) is completely locked down & has been for several
years, is now allowing administrative access? The system is virus free, several
scans have shown that. Every test on I could find shows that I am NOT an open
relay. All Service Packs and Hotfixes have been applied. Routing is set up to
only allow authenticated users to send mail, and the Administrator account most
certainly does NOT have a null password.
Back to the news groups and more Internet research, looking for
any mention of this bug. Oddly enough I find it in a throw-away line on one of
the news groups: Someone is trying to set up his sendmail program to communicate
with an Exchange server and someone else mentioned to be sure to plug the "smtp
auth hole" in Exchange.
SMTP Auth hole? says I. That's a new one on me.
Finally someone mentions how to plug this hole and refers me to a KB article.
There's an explanation of EHLO and AUTH LOGIN, and something about Exchange not
supporting it. It's still unclear to me, but it does lead me to this article:
http://support.microsoft.com/default.aspx?scid=kb;en-us;235627
(Quick update: To make certain that the answer is readily available here is the
vital information from the link:
WARNING: If you use Registry Editor incorrectly, you may cause
serious problems that may require you to reinstall your operating system.
Microsoft cannot guarantee that you can solve problems that result from using
Registry Editor incorrectly. Use Registry Editor at your own risk.
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\MSExchangeIMC\Parameters
NOTE: The above registry key is one path; it has been wrapped for readability.
Value Name: DisableAuth
Data Type: REG_DWORD
Value: 1 (or any other non-zero value)
You will note that this article doesn't mention that it also fixes a bug where a
user can simply log into your network using the AUTH LOGIN command.
Ominously I also saw this at the bottom of the post that fixed this problem for
me:
"All we need now is a similar fix for Exchange 2000 - I haven't found one
yet."
So...have I stopped the spammers? I hope so. No spam sent since I implemented
the patch at 11:30 pm last night. I get a lot of "AUTH LOGIN - Security Not
Available" in my logs now. I hope I've fixed the spam and haven't shut down our
system to half the Internet, though it seems that the spam lists did that
anyway.
That's the last 30 hours of my life, distilled to several paragraphs on a web page. I will end with this remarkably apt quote, sent to me by a friend of mine when I was in the midst of this:
From "Secrets and Lies:"
"The first firewalls were on trains. Coal-powered
trains had a large furnace in the engine room, along with a pile of coal.
The engineer would shovel coal into the engine. This process created
coal dust, which was highly flammable. Occasionally the coal dust would
catch fire, causing an engine fire that sometimes spread into the passenger
cars. Since dead passengers reduced revenue, train engines were built with
iron walls right behind the engine compartment. This stopped fires from
spreading into the passenger cars, but didn't protect the engineer between the
coal pile and the furnace. (There's a lesson for sysadmins in this
somewhere.)" -Bruce Schneier.
September 2nd, 2003
A fairly quiet Labor Day weekend, save for huge surprise 40th birthday party my wife threw for me.
She kept it secret for a month, planning it & inviting people and getting things set up. She even had my best friend get me out of the house while things got put together. He invited me fishing, so we decided to go at 5:30 am.
Of course it was raining cats and dogs & was far too cold to go fishing. Now he's stuck with me & needs to keep me out of the house for the next 8 hours, since everything was set up for 1 pm.
We wind up going into my place of work where there is a "fun" technical problem I've been working on, but one where I need his assistance. We start working on it and in the midst of setting things up we discover a HUGE security problem on my system.
We scramble to get things done & put back together. About 2 pm I call my wife and tell her I'm on my way home (I still don't know about the party at this point), and let her know Skip & I will pick up something to eat on the way in. She tells me there's food for me at the house & to just come straight home.
I'll skip the details, suffice it to say that turning 40 was painless because of the work of my wife and the assistance of my friends and family.
Thank you all.