Small Business Defense – Patch Management

There are three ways to approach this problem. The most common method is to ignore it, and apply patches as time permits. The logic here is that since applying patches can often require a maintenance window, it’s hard to balance the business’s needs against the risk of an attack by an unknown party. Since an increasing number of attacks are subtle, it’s quite easy to convince yourself that it’s not a big deal, and inadvertently accept more risk than you’d like. I don’t really recommend this method.

The second method is to fully embrace the situation and fork out the cash for a full patch management system. These solutions aren’t cheap, but it does allow you to view your entire environment from a single console. This way, you basically outsource the tedious job of keeping on top of everything and use the tool to make sure that all machines on the network are kept fully updated. Now, this solutions doesn’t eliminate the need to schedule downtime to get the patches applied, but it does simplify matters significantly… at least when you are only running software that is monitored by tool.

The third method is something of a middle solution. In situations where you either lack the budget for a patch management solution or are still investigating the varied options, you can simplify the process by doing a quick audit of each of your systems and uninstalling anything that isn’t needed. The key here is system classification:

  • Development systems should not directly face the Internet.
  • Production systems should not have development software on them.
  • Production servers should not have workstation software on them (Office, Adobe reader/flash, Web Browsers)

By eliminating all unnecessary software, you can massively reduce your attack surface. Simply put, if software isn’t there, it cannot be exploited. Now, this doesn’t eliminate the necessity to keep the software that is there up to date, but in the process of removing what’s not needed, you can get a good idea as to what is there and monitor the patch releases for those few projects. It’s not pleasant, but it is doable.

Small Business Attack – Patch Tuesday (and others)

Every month, on the second Tuesday, Microsoft releases a set of patches to their software.  They’re ranked in various ways, based on what they correct and how critical they may be.  Then, two things happen:

First of all, various security groups review them and start posting their opinions (I prefer the Internet Storm Center synopsis). After that, those of us with more internally-focused positions start reviewing the various summaries by both the security groups and Microsoft and work up an internal plan to test and deploy the patches appropriately. When, after everything looks right, we start deploying the patches to make sure that everything is nice and secure.

Secondly, the various more selfish security groups also review them… but in a tad different way. They investigate what the patches correct and start trying to come up with malicious code that exploits the problem. Then, at the same time that we’re reviewing the patches for our environment, they’re running tests against various other systems. If we’re lucky, at the time that we’re deploying the patches on our systems, they’re deploying the new malware against our systems. If we’re not lucky, they beat us to the punch.

Of course, this is a somewhat simplified scenario. There are a great many more vendors than Microsoft, so this cycle doesn’t really take place on a monthly basis. Some vendors release updates on a quarterly basis, some are yearly and some are pretty much whenever they feel like it. So really, each day is a steady flood of vulnerability information and, if we’re lucky, patches to go along with them.

If you can stay on top of the flood, you can keep your systems somewhat protected. Off course, if you miss something, you leave a hole that an attacker can easily find.

So what do you do about it?

Security Lessons from Nature – Poison Dart Frogs

Poison dart frogs are, not surprisingly, covered with poison. I could go off at length about how different species have different levels of poison, and how not all of them were used to poison darts and how many of them are going extinct due to a nasty fungus that’s only vulnerable to an eyewash solution… but that would be a bit too rambling even for me.

Instead, I’m going to talk about ants. I’m not going to go off about how they are communal, have some interesting chemical signals or even how they are vulnerable to some very interesting fungi that take over their brains (despite how unbelievably cool that is). No, the important thing is that the frogs eat the ants.

Boring, I know.

See, the poison dart frogs don’t generate the poison themselves. Instead, they eat ants and push the poison from the ants out through their skins. Not only is that an awesome example of how a predator can turn a prey’s defense into a defense for the predator while simultaneously rendering it useless for the prey (smart little froggies!), but it’s also an example of the importance of operations.

See, an interesting side effect of this method of defense, is that if the ants go away, then so does the defense. Domesticated poison dart frogs aren’t poisonous (which would make them dart frogs (which, since they neither throw darts nor are tailors, is a horrible name for them)). In order to keep the defense, they have to keep on acquiring ants.

Which gets me into mergers and acquisitions… which is where I wanted to go the whole time. When you conduct an acquisition, as the acquirer, it is often tempting to go for economies of scale and try to get the acquiree to do things your way. This just makes sense. After all, that’s why you bought them, right?

Well, kinda.

Unless you bought them to kill them as competitors, they probably bring another value to the table as well. If you buy a poison dart company and then tell them “Now that you’re part of GlobalConglomeratedWidgetCoInternational, you have do things our way… and we eat our own dogfood!” you’ll definitely merge them into your organization… but if they’re eating dogfood, they’re not eating ants and you just have a dart company.

When merging operations, pay close attention to the operations of the other company and try to understand why they do things the way they do. There’s generally a good one. Then the question would be whether the loss they face by doing things your way is outweighed by the operational efficiencies, and whether it’s all that important that the darts be poisoned.

Mythic Monday – The Aging Lion and the Fox

Another one of Aesop’s fables that isn’t that well known is that of the aging lion and the fox. You can click the link and read it, but for those of you that are linkaphobic, here’s a short version:

A lion was getting old and having trouble hunting. He decided, instead, to pretend to be sick and went back to his cave, moaning all the way. Over time, as each of his neighbors stopped by to check on him, he ate them.

Then, one day a fox came by and asked how the lion was doing. The lion moaned and asked the fox to come closer. The fox then observed that the footprints all led into the cave, and none came out.

Clearly, the fox is the fable animal to be. He’s smart. He’s observant. He’s… umm… red and furry? (Are Greek foxes red? . . .  Yes, after googling a bit, it seems that the red fox is global, and the grey fox is only native to the Americas… which has nothing whatsoever to do with this blog entry.)

No, the point of this blog entry is that of evidence. If the lion had been wise, he would have either wiped the tracks after each meal or (more preposterously) fabricated tracks going back out. The fact that he didn’t, is what allowed the fox to escape and presumably tell the other animals what the lion had been up to (and Aesop, since he wrote it down). So, not only was the lion caught, but he lost his lovely little racket and probably starved to death shortly thereafter.

Most attackers are aware of this story (sorta), and do take some effort to reduce evidence. A burglar usually wears gloves, a bank robber usually wears a mask, and a hacker usually clears system logs. So, if we want to make it hard for the lion to wipe away the footprints, we have a few options. The first is to replace the dirt outside his den with fast-setting concrete… which would prove somewhat troublesome if you analyze this ridiculous analogy too far. The second is to set up a camera trap and record everyone who enters the cave. (For those purists who would point out that there were no cameras in ancient Greece, let’s just say that Hephaestus is there cranking out a vase for each animal. (Happy now, picky people?))

In the modern world, we actually use both of these techniques. Instead of fast-setting concrete, we have a hard drive technology called WORM, or Write Once Read Many. With this drive, you can store the logs in such a way that they cannot be altered. They are, however, quite expensive and can be difficult to set up properly. Instead, we generally prefer to use the camera/vase trap system. For this, we use one of many remote-logging technologies. The simplest is probably the venerable syslog server.

This solution simply involves setting up a dedicated server and installing one of the many syslog systems on it. Then you do a bit of configuration on each of the other servers you have and basically tell them to go log over there. Whenever there is an event, it goes over the network and is stored off the server. That way, if an attacker gets in, even if they wipe their own traces, there is a backup elsewhere that is (in theory) a lot harder to alter.

Of course, you still have to actually be the fox and look at the logs now and then, but at least you’ll be safe from a smart lion.

Small Business Defense – Plans and Flexibility

In the event mentioned yesterday, the business was utterly without power. This sort of event had never been planned for, and since we were a technology company, all of our client information was inaccessible due to the outage. Inbound phone and Internet was down.

In that particular case, I believe we all looked at one another, shrugged and pulled out some snacks and started playing poker while we waited for power to be restored. (It was a very small business and we were all fairly young.) After a while, one of us got the bright idea to call the phone company and set up a temporary redirect of calls to some of our cell phones. Someone else carried the workstation that doubled as a fileserver over to a neighboring business so we could fire it up and get the client contact information, so we could start calling out to let people know about the situation.

We wouldn’t connect to the remote systems via modem, but we had decent memories and it worked out OK. The clients were understanding and while we lost some productivity, it didn’t impact us too badly.

Which, really, is the point of this. We in the security world like planning and mitigation. We like it a lot. We might even like it a bit too much. See, it’s not all doom and gloom. Sometimes, bad things happen, and it all works out OK.

In a large enterprise, you have a complex infrastructure that has a lot of moving and inter-related parts, and if there is a massive failure, it’s simply not feasible to shut it down or move it. The financial cost of such an outage can get into the millions of dollars, so it makes sense to devote some resources to coming up with recovery plans. It can take months to build one and then more months to implement it. Then you have to test it.

In small business, you may not need to do this. Should you? Probably. However, not having a solid plan isn’t the end of the world, it’s just more risk. Just as in the rest of the business, you can look at risk vs reward. It often doesn’t make sense to have a full plan that covers details for floods, fires, tornadoes and Godzilla attacks. If your infrastructure is small enough, your employees are good enough and your customers friendly enough, your plan can just be “if bad stuff happens, we’ll figure it out”. It’ll probably be OK.

However… it just might make sense to look at what you need to be flexible. Are your people really as good as you think? (Can you test this?) What about availability? If you rely on your people, how are you set for disasters that impact people? What if the backhoe had hit a gas main and some of your employees were injured during the disaster? (Or, more prosaically, suppose I had banged my head trying to get out of my dark “office” and been unable to accept incoming calls?)

So yes, by all means, avoid the tedious planning that no one wants to do. Bet your business on your people (which, really, you do every day anyway)… just be sure that your people will be able to do what you’re asking of them.

There’s more value in DR testing than just testing the systems, after all.

Small Business Attack – Backhoe

Yes, you read the headline correctly.

Several years ago, I was working as a web developer and security admin for a small development house.  We were located at one end of a nondescript building in the middle of a tiny little town.  One day I’m working and suddenly everything goes dark.

And by “everything”, I mean everything. My office was the same as the server room, it was lit by a single overhead fluorescent and the glow of a few monitors and lots of little blinky lights. When the backhoe doing sewer work hit the main line, everything went dark and I was suddenly sitting in a tiny little pitch black room rapidly rediscovering my latent claustrophobia.

Luckily for me, I had a cell phone that could double as a flashlight in such a situation. I found my way to the door and met the rest of my coworkers to find out that the entire building was without power, Internet and phone. Business was at a complete standstill.

We were dead on the Internet. No one could reach us. And all of our clients were running that day’s production.

What could we do?

Security Lessons from Nature – Natalids and Stargate Universe

So, I’m reading a book on the mammals of Costa Rica. (Why? Because it’s more interesting than watching Stargate Universe, that’s why. (Which says a lot about the quality of storytelling these days.)) In the chapter on bats, I ran across a mention of a natalid organ.

“That’s funny…”, I thought. “I’ve never heard of that!”

So off to the Google I go, to google about and, as it turns out, waste a good hour reading about bat taxonomy. (Which is still better than watching Stargate Universe!) Here’s what I learned:

There are these bats, see, that have an organ. It’s more than one species, it’s in a lot of them… but no one knows what it does!

  • Discover Life reports that the cells may be sensory or secretory.
  • Novel Guide tells us that it’s bell shaped and can cover the entire muzzle (though Answers.com suggests that that’s not always the case).
  • Brain Museum implies that the presence of the organ may be linked to the lack of a nose leaf. (What’s a nose leaf, you ask? Go research it yourself, I’m busy with natalids.)
  • Bob’s Bat Cave, despite having perhaps one of the coolest names on the Internet, indicates that the organ is below the skin on the forehead, though other sites place it at the back of the muzzle. (This seems like a conflict to me, but perhaps I don’t know my way around a bat’s head very well.)
  • Lastly, Animal Diversity gives us the useful information that only Natalids have natalid organs. Of course, the group of bats known as natalids are defined as those bats that have natalid organs, so that information is less useful than it may initially appear.

I might have learned more, had I given J STOR $19 for the full article, but let’s face it, I’m just a Stargate fan who is oddly distracted by bats, and it would be unwise to give my bad research habits free rein.

So what is all of this doing on an I.T. security blog? I haven’t the faintest clue… and that’s the important thing. The number one biggest threat out there isn’t the mysterious Chinese hacker of the organized criminals writing malware. The most dangerous threat is that of poorly-documented legacy systems. These systems exist on every business network I’ve seen.  They lurk in the dark corners, staring at admins and, well, do something… I think… maybe.  These systems are dangerous because:

  1. We have to keep them running.
  2. We don’t know what they do.

Most people therefore, set them on the network and proceed to ignore them until they break. Maybe all they do is serve a few static web pages. Maybe, though, they process proprietary data. However, since we don’t know, we can’t pick an appropriate method of securing them.

We can’t turn them off, because it might harm the business, just like we can’t go up to random bats and remove the natalid organ. If we don’t know what it does, we often can’t take the risk of killing the business (or bat) by removing it to find out. (Just like we can’t take the risk of not trying the new Stargate series, as they might be awesome as SG1 (though, admittedly, history has not born this out)).

We can look deeper into the systems and possibly get an insight (“hmm, it’s kinda slimy, but it also looks like it might be a detector”). We can ask those that use it what they use it for (which might be more effective in your coworkers than it is on bats). Or, we can just name it and leave it alone (“well, it’s gotta be there for a reason, right?”)…

Which works until someone like me comes along and thinks “what the heck is a natalid organ?”, and starts digging into the problem. Because at that point, you have to justify one of two likely scenarios:

  1. Why you kept a legacy system running and consuming resources when it serves no valid purpose to the business.
  2. Why you failed to adequately secure and plan migration paths for a business-critical system.

Really, it’s probably better to find out what it does and document the thing.  Luckily, we have technologies now that allow us to record inputs and outputs and clone systems, so the process should be a lot less messy than dissecting the muzzle of a bat or figuring what on Earth the producers of Stargate Universe are thinking.

Mythic Monday – Love and Creation

There is a Persian creation story that goes much the same way as the usual creation myth. First, there was nothing, then there was a god (Ohrmazd). The god made stuff and then people. Then the people screwed up.

People screwing up is really a common theme in myth, when you think about it.  Maybe that says something about life?

In this case, though, the type of the screwup is a bit different. There’s nothing here about wanting to the equal of the gods, disobeying orders or even just desiring to be more than they are.  Instead, the people wind up having children (a popular activity). Then since they can’t bear to be separated from their kids, they eat them.

Ohrmazd the creator god is understandably surprised at this turn of events. What’s interesting is the solution. Knowing that the people just love too greatly, he reduced their love by 99%.

(As an aside, it’s worth noting that the Persians did a lot of interesting mathematical exploration and that this is the only myth I know of that uses numbers like this instead of something like “reduced their love as if love were water in the cap of an acorn, and when emptied, the moisture that remained was as the love that remained within the man the woman”. Are the two related? I don’t know, but it’s interesting.)

With the amount of love they could feel, reduced, the people were able to have children and let them live long enough to have children of their own. Thus, did humanity prosper.

Now, in the original, this was but a small piece of the story of creation (which also involved a devil and a bull, much conflict and blood and all the fun stuff you find in creation myths). However, for our purposes, it is enough.

There is a lot of talk in the business community these days about the power of love. I have no doubt that there is something there. If you love what you do, you can do it without feeling the burden. You can more easily justify risks and you can share the load by letting your love inspire others. However, there is a dark side.

The same love that makes it easy to get started on a project is what makes it hard to stop. Love can get you through the boring 20% of the work that takes 80% of the time. However, it’s not so good at allowing you to stop when you get to 100% complete. I’ve seen projects that fail because the quest for perfection goes too far. I’ve seen businesses falter and fail because the founder loves it too much to allow it to change.

That form of love is stifling, and while it’s becoming more acceptable to recognize the harms of excessive love within personal relationships, it’s still not well considered within the business world.

This is the sort of emotion that makes security practitioners secure things for the sake of their being secure… they’ve fallen in love with the idea of “security” instead of “protection”. There are many ways to protect an asset. Keeping out the bad guys is but one.

It’s a tough balance, I know. We have to love enough to keep us going in the face of incredibly difficult odds and constantly changing threats, but then, once a project is complete, reduce our love by 99% and allow our project to continue on without meddling with it and destroying it in the process.

While learning to let go is difficult and messy, if we’re lucky, we can do it without the massive quantities of blood and death that the Persians seem to have required.

Other Sites: Business, Photos/Conservation

Get the feed (RSS):



Josh More - Entropologist
Expert in removing chaos from
I.T. and business systems.