(
computational complexity):
The most important open problem in the field.
The class P is (trivially) a subset of the class NP. Nondeterminism is always optional. However, it would appear absurd that P=NP. A nondeterministic machine trying to decide membership in a language is presented with a hint (called a "witness" or "certificate") which proves membership (no such witness is provided for elements outside the language; the definition is asymmetric).
So most people would be extremely surprised to discover P=NP. It would also have strong repercussions on the way we think of algorithms. But nobody has any idea how to prove that P!=NP.
To show P!=NP it is sufficient to show that one problem in NP is not in P. To show P=NP, it is sufficient to show that one NP-complete problem (say SAT) is in P (an NP-complete problem is by definition in NP, so if it's not in P then P!=NP). Note that this immediately gives a concrete equivalent question: Does there exist a polynomial-time (deterministic) algorithm for checking boolean formula satisfiability?
More structural approaches have also been tried. For instance, the definition of NP is not symmetric. If we know a language is in NP, we generally do not know that its complement (the set of words not in the language) is in NP. The class of these complements is coNP. But if a language is in P, then so is its complement. So if we could show NP!=coNP, we would immediately deduce that P!=NP (otherwise, since P=coP, NP=P=coP=coNP using some trivial logic). Now, very few problems are known to be both in NP and in coNP, yet not known to be in P. And of course, none of these problems are NP complete (an NP complete problem in coNP would immediately show NP=coNP). Yet proving a language is not in NP (or not in coNP) is also a very difficult task.
Some approaches are known not to work. The usual method for solving problems like the halting problem is known as diagonalization. However, it can be shown that diagonalization isn't strong enough to show P!=NP. In other words, a much better understanding of P will be required to show P!=NP, one from which we are very far.