Lots of folks dutifully install firewalls on their sites, and now everyone sleeps a lot better at night. The problem is, in many cases, there isn't any really clear way to test the firewall and make sure it's doing its job, or to what degree its doing its job.
(As an aside, I want to point out to any folks who are relatively new to firewalls or security in general that this is significantly different from the standard approach, where something is set up, and everyone tries to make it work. Because functionality is only a part of the firewall's job -- stopping bad stuff is just as important as allowing the good stuff in most organizations -- there are other considerations. Just as you should just to be sure that it seems to be working properly from the inside, you should be sure that it's actively blocking things from the outside.)
Here is some advice to anyone designing a firewall architecture, to help be sure that it is doing the job that you think it is.
Understand what your critical components are. You're going to have to assume that your attacker launches his attack against the weakest point in your defenses. (Read on.)
What is the central security mechanism of your firewall? Is it a ruleset that lives on a router-thing, glorified, or otherwise? An application layer bastion host? In any event, you need to understand what it would take to change those rules, or do what it takes to defeat this mechanism.
How are you protecting that central security mechanism? Is it a packet filtering router? What are the rules there? What would it take to change them? Are you allowing remote logins to it? Do you have a modem hooked up to the console port?
Now, let's take a look at the rules you've got defined. What untrusted machines are you allowing to talk to machines in your DMZ? (Are you allowing obviously forged packets to enter into your DMZ from the outside? Obviously forged would be those that claim to come from the loopback network -- 127.0.0.0 -- as well as those defined by RFC 1918, and those that have an address that you're using on the other side of the firewall.) What machines in your DMZ can talk to other machines in your DMZ? What machines in your DMZ can talk to machines in your private network?
Assume that someone has broken into (i.e., has root) on each of the components of your firewall, all the way from your external router to your link to the internal network, one at a time. Now, look at all of your rules, and see exactly what can be done from that machine. Can they get directly into your internal network? Can they do something to alter rules on packet filters? Can they talk to other machines?
At this point, you should have some idea of your architecture's vulnerabilities. Look at what you've got. Can you strengthen things by simply adding some more ACLs someplace? Maybe you'll be able to increase your security by subnetting some parts...
OK. Now you know your architecture, and it's time to take a look at the security of each of your components. Look at things like access to your packet filters. What OS(es) are you running? Are they patched? Are your machines running only necessary services? Do you have all kinds of silly setuid programs and services running? Are you using sendmail? Is it five years old? Are you running NFS, SMB, or NetBIOS? (You naughty, naughty firewaller, you.) What else are you running? Why are you running them? Do not run anything that isn't absolutely required, and make sure a packet filter is preventing people from even accessing those service ports, and if you can, remove the unused service from your machine entirely. (That way, if someone does break in, they won't be able to turn the service on.)
What have you done to tighten things up and show you what's happening? You've got boatloads of available disk space, right? Everything you've got from ACLs to application relays are logging the snot out of everything, right? Are you running tcp_wrappers? tripwire? something similar?
Now you have a pretty good understanding of where your vulnerabilities are, and what your detection capability should be, and that sort of thing. You should be able to write electronically, or even with good old-fashioned pencil and paper, exactly which hosts should be visible from the outside, which service ports they should be able to reach, and what they can do with each of those services.
If you don't have a tool like SATAN, strobe, or something similar, I highly recommend going and picking a few up. They're free and not difficult to find. Be sure that the bad guys who will be attacking your site will be using these very tools; not getting them for yourself -- to be sure what the attackers will see before they're trying to beat your front door down -- is foolish.
From an "outside" account, perhaps a dialup-line at an ISP, scan your own network. After some period of time, you should get a lists of hosts that can be reached, and what service ports are answering calls. (NOTE: If you're using tcp_wrappers to log attempts to use your services, the ports being wrapped will count as "answered" as far as a scanner can tell.) After you've done this, go through your logs, and see if you can even tell that you've been scanned. Once you have an idea of what to look for in the logs, be sure to do so regularly. (Better yet, write a simple Perl script to cruise the logs for you at some regular interval (every night? every few hours? real-time?) looking for signs of possible badness, and have it report things to you. My scripts to do that send such alarms to my alpha pager, with complete details going to my regular email.)
It would be a good idea to do this with some reasonable level of regularity. Be sure that you record the results of each scan. When you do another scan, compare the results, just to be sure that something hasn't opened up that you weren't expecting. And if you are expecting it, then just use that to confirm that what's been opened is OK, etc.
Of course, there's no 100% guarantee, but taking a straightforward approach to understanding your firewall, and taking the time to perform audits, can go a long way in keeping your organization safe.