I find the current state of affairs in programming deplorable. As it is, a single tiny system program with an unchecked buffer size can often be exploited by an attacker to execute arbitrary code on the system. (Let that sink in for a second.) Your operating system (whatever it is) is so fragile that a single coding error, a typo, could allow anyone to do anything they wanted to on your machine. That's unacceptable. And all too common.
As I see it, this state of affairs has dual origins. First, the programming language most operating systems are written in is too fragile. Second, the security model used in most operating systems gives small programs too many privileges. These two faults combine to allow common coding mistakes to compromise the security of an entire system.
Most modern operating systems are written in a combination of C++ and assembly language, with the latter reserved for the lowest-level code. C++ is advanced enough to do wonderful things, but still has a great deal of connection to the underlying system. A logic error quickly becomes an execution error, and an execution error in just the right place can bypass any number of carefully crafted guards. For example, a buffer overflow can alter the flow of a program, remove the barrier between flat data and executable code, or simply crash the system. Higher-level languages such as Java and LISP simply cannot have buffer overflows, due to their structure. But an entire operating system can't be built in high-level languages; there always has to be a lower-level language underneath.
Adding to the problem is the simple hierarchical nature of security implemented in every notable operating system. There is a user account ("root" or "admin") who can do absolutely anything, with the same privileges as the operating system itself. There is a collection of other user accounts representing actual people, who can run programs, read and write documents, and connect to other computers. And, depending on the system, there are some very unprivileged accounts that do specific tasks behind the scenes. Less privileged users can run some more privileged programs and edit their own data, but the root user can do anything.
Lack of filtering
System programs are completely privileged, but can be initiated by completely unprivileged users. (This is not bad in and of itself; indeed, it is necessary for the running of the system.) Consider the recent Microsoft WMF vulnerability, wherein a specially crafted image in a web page can execute any code it wants to. Pretty sketchy. Why should an image display subroutine have such amazing powers? Because it is system code, and might need to access system resources. Unfortunately, the WMF specification allows the media file to contain data intended to be read as instructions, in case of a processing error. Granted, the specification was written some years ago, when security was slightly less of a concern. Allowing a data file to act as a program is a move even Microsoft is unlikely to make these days.
Today's security-threatening logic errors run more along the lines of improper condition checking. Imagine a hypothetical system call that would allow a user to delete a file from their personal folder, in my case, /home/tim. A programmer not thinking clearly might simply append the name of the file to be deleted to the path of the personal folder. For example, a request to delete test.txt would be translated into a request to delete /home/tim/test.txt. However, an attacker could pass in ../../boot/grub/menu.lst, which would then be translated into /home/tim/../../boot/grub/menu.lst. Since .. in a pathname means "go up one directory", the file to delete would be /boot/grub/menu.lst, which is part of how the operating system starts up. A normal user can't delete this file, but a system process could.
Notice how the hypothetical unchecked system call would allow a regular user to delete any file on the computer, including essential system files. That subroutine should not have such broad and sweeping authority! It should only have the power to do anything the user could do. But system code is all-powerful.
Not all "system" code is run with system-level privileges, thankfully. A number of programs and services that the user interacts with run as unprivileged users. They are given their own set of disk space and memory to play in, a sandbox of sorts.
But having a separate user for every system process and module would be ridiculous. There are a thousand possible entry points for an attacker, if one considers the multitude of utilities and libraries present in a modern operating system. Just this morning, I had to update a system library, libtiff4, that had an arbitrary code execution bug. Similar bugs are found on a daily basis.
Fixing the flaws
I'm no expert on computer security, but I've got a good intuition for some of the basics. One thing I've learned through observation is that we coders trust ourselves too much. There need to be safeguards built into the programming languages we use and the operating systems we write for to protect us from ourselves.
It's no secret that I love programming in Java. It is a gorgeous language, with plenty of safeguards and just the right amount of syntactic sugar. A poorly written program cannot corrupt memory, cause classical memory leaks, or overflow a buffer. When the JIT compiler is enabled, Java programs can run nearly as fast as native programs. With a more modern security model than C++, a Java-based operating system would have the potential to render inert all manner of low-level attacks. There has been a lot of talk about such a system. The JVM might need to be revamped a little to support an entire OS, but beyond that, the foundation has already been laid for a project of that magnitude.
Restricted security domains
What if every piece of system code was not fully privileged? What if a piece of code could be declared as "only read and write in these directories" or "access external media"? Then a slip-up in a piece of code would not necessarily compromise an entire system. I'm not sure how such a concept might be implemented, but Java 5.0's Annotation feature might be involved.
Do it now
As computing systems keep growing in relevance, ubiquity, and interconnection, a better security model is needed. If we continue in our present course, we will find ourselves unable to maintain the code we have created.