"Only the spoon knows what is stirring in the pot."
-Sicilian Proverb

Production as a System
Production information systems are comprised of firewalls, servers, software, users, data, state and more. The combination of these individual components creates a mind-boggling array of possibilities that exist only in the production environment.

Yes, it is complex. But more than that, when you add all those pieces together you have something new - a system. Systems theory states that the interaction of component parts creates an entity distinct from the combination of parts themselves. The whole literally is greater than the sum of its parts.

One important characteristic of a system is that you cannot recreate it. Take a forest for example. A forest is a system comprised of trees, soil, insects, animals and so on. If you take those same components and try to recreate that forest, you wont be able to. The climate, watersheds and environment also are part of the forest.

Production systems are the same way. A production system cannot be fully recreated in the test lab. Anyone who has worked with software developers knows this from first-hand experience. The plea often heard is My code works just fine on my workstation when it doesnt work in production.

Another distinguishing feature of a system is how it responds to change. Cause and effect are not always intuitive. Even small changes may produce unexpected results. A new species of insect in a forest may cause an explosion in the growth of underbrush that in turn makes the forest more susceptible to fire. Network administrators may update the firmware on the firewall to accommodate a feature associated with a new software release. The new firmware has a security feature that blocks certain types of users at the firewall. However, everyone assumes the problem is with the new software release so it is rolled back, yet the problem persists.

Points of View
When you look at a forest, what do you see? Do you see trees? Do you see shades of green? Or do you see the beauty of a complex system at work?

Lets take a moment and pull over at a splendid scenic viewpoint overlooking our production system. We just happened to bring a carload of coworkers along with us. As we stretch our legs and take in the panoramic vistas, heres a sample of what is heard from the group.

Software Developer:
"Wow. Will you look at all that software! And over there more software! See that one passing a file descriptor across the process boundary? I wrote that. And see that area over there with all the tangled threads? Thats the old software that was written before I got here.

Network Designer:
"Wow. Check out all the packet flocks flying in perfect formation. I designed the flight paths myself. Hey, what are those fragments doing over there? Let me grab the binoculars and check out their ninth octet. Too bad, we just missed their last hop.

Business Manager:
"Wow. You can really see the growth in recurring users from up here. If we hit our third quarter numbers, were going to have to increase our capital expenditures in the next budget to keep up with the demand. Hard to imagine there was no revenue here just a few years ago.

Customer:
"Wow. So this is where all of our critical financial data is stored. Hey, what would happen if there were a fire?

Seeing the Forest and the Trees
If you have a lot of information without grasping the bigger picture it could be said that you cant see the forest for the trees. If you understand the gist of things without having mastered any of the details it could be said that you cant see the trees for the forest. These clichs apply to any system. Is there a way to see both the forest and trees?

1. Add Context to Reporting.
Monitoring and reporting on your production system is vital. However, reporting on the right information can be harder than it seems.

Are you using pre-fabricated canned sample reports? Or have your reports been specifically designed to capture the things that are truly of interest?

Re-visit old scheduled reporting routines. Communicate with the people actually receiving the reports. What information is relevant? What is missing? Narrow down reporting to focus out what is useful, not just what the reporting software can do.

2. Do Modeling.
Though it is impossible to completely duplicate a production system, that doesnt mean you cant model it.

An ideal testing setup will include a test lab and a pre-release staging platform. The servers in the test lab should be configured as close as possible to the actual production servers. One common problem is security configuration. Developers often write software using full root or administrator privileges, and then attempt to add security later. Security should be integrated from the beginning in a way consistent with how it will be on production.

The staging platform should be located on the production network. Testing software on a pre-release staging platform in the production environment helps pinpoint problems that crop up only in production. Operating system and security hotfixes can also be tested on the staging server to ensure compatibility with application software before being rolled to the production servers.

3. Change Slowly.
Weve already learned that systems are sensitive to change. Even small changes can cause drastic consequences.

To minimize the impact of change, minimize changes. In other words, dont go changing a bunch of things at the same time.

How often do you hear someone say Well, since Im having a heart bypass I may as well get my appendix out while Im at it? A major application software upgrade over a holiday weekend is not a good time to upgrade the firewalls too.

Only change one thing at a time. By changing slowly, you allow the system a chance to absorb the changes. If something goes wrong, youll have a better chance at diagnosing the problem and recovering from it.