|"I only just met you. Should we be dancing this close?"|
The first thing you need to expunge from your mind, as a writer, is the idea of the "bad guy." No one wakes up in the morning and says, "You know what? I'm going to be positively evil today." All actions that a person performs are usually justified from that person's own frame of cultural reference and point of view. What's good for one person might be bad for another. Or, to quote Six Feet Under episode Out, Out, Brief Candle, "Everything's bad for something."
Sympathetic, flawed protagonists need sympathetic, flawed antagonists. It might seem counter-intuitive, but you absolutely must make your antagonists somehow relatable. Not every antagonist is (Sauron in The Lord of the Rings, the dinosaurs of Jurassic Park, or Palpatine in Star Wars) but the most effective ones, the most tragic ones, are:
Charles Muntz in Up is driven to murder in order to catch an exotic bird and clear his name. Harvey Dent in The Dark Knight wants retribution after his disfigurement and the death of his paramour. Norma Desmond in Sunset Boulevard can't let go of her glory days as a silent film actress and is willing to do anything to return to the spotlight.
Regardless of how relatable an antagonist is, they all share one thing in common: they all want to possess something. They all want stuff. They won't let go. Antagonists sure are selfish.
But in some way, shape, or form, we're all selfish. Good antagonists merely take this to the extreme.
Tim Albaugh of UCLA and Hollins puts it succinctly: "A villain doesn't know when to stop." By extension, a hero does. A hero learns the wisdom that the antagonist lacks (or ignores) and winds up on top at the crucial moment.
The best adversaries are the ones who represent a darker version of the hero him/herself. If you're a fan of mythology and the archetypes inherent (see Carl Jung, Joseph Campbell, and Christopher Vogler), then you already know a fair amount about the Shadow archetype. There's no "bad guy" archetype per se, although there's a Shadow. Literally, area of darkness. The anti-protagonist (not antihero), the shadow can value itself over others, or put others in danger to save itself or to achieve its ends. Selfish, greedy, obsessed. Many protagonists (in fact, the best) possess aspects of the Shadow. Remember Luke Skywalker fighting himself in the cave on Dagobah? Or Frodo screaming at Sam that the One Ring was his own responsibility? Or Amadeus calling Italians "musical idiots"?
We all have some Shadow in us. The antagonist merely features this archetype closer to the surface than his/her heroic qualities. What I recommend is that you give them solid reasons for being that way.
Harvey Dent becomes an antagonist, but we understand what drove him to the depths of obsession and madness. We don't applaud what he does, but we understand why he feels the need to do it. What UCLA's Hal Ackerman refers to as intimacy. In The Lord of the Rings, while Sauron isn't too relatable, Gollum certainly is. Nearly desire incarnate, he fights (or pretends to fight) a losing war against a want that's tormented him for 500 years. Or look at Robert Mitchum's creepy portrayal of Harry Powell in The Night of the Hunter. He's a misogynistic serial killer, but he operates according to his own belief system and moral code, which he believes comes directly from God. Or Anton Chigurh (Javier Bardem), the relentless hitman in No Country for Old Men. He's a machine, a tool, with a task: go there, get this, remove anyone who stops you. It's an extreme version of anyone who is given a task at which failure is not an option. Even he has his own set of rules. We don't necessarily endorse what these antagonists do, but we understand why circumstances forced them to be who they are.
Consider this: it's been said that we dislike the people we dislike because they remind us of something we don't like about ourselves. Otherwise, we'd be indifferent.
Christopher Nolan, when asked about Batman's antagonists, said, "[Batman’s] best adversaries are the ones who represent some other, darker direction he could have chosen."
The best bad guys are, in essence, your good guys, but having surrendered to their flaw. Charles Muntz in Pixar's Up is unable to let go of a dream, and it's this rigid state that dooms him. Up's protagonist, Carl Fredricksen learns how to let go, and he turns out just fine. The Dark Knight's Joker is a version of Batman who has lost faith in humanity: "When the chips are down, these... civilized people, they'll eat each other. See, I'm not a monster. I'm just ahead of the curve."
The best antagonists aren't monsters, either. They're obsessed and unwilling to let go. They're human. Sometimes, that means that they're you or me.