Let's look at a purely commercial novel. It tends to fall within a genre — mystery, thriller, true crime, horror, sci-fi, fantasy, historical, romance or a similar category. Often the novel crosses over genres. A author friend of mine, Pam Binder, writes time-travel romances. Another, Mitch Luckett, writes fantasy mysteries.
Commercial fiction is by nature optimistic. Characters fall in love and leap tall buildings. The killer is caught. Good triumphs over evil, and the moral guy keeps his white hat. The culture's values are reaffirmed, and balance is returned to the protagonist's life. The reader expects to be titillated, thrilled, scared or feel some other strong emotion. The theme is usually defined by the genre. For a romance, it's "love." For a mystery, it's "discovery" or "justice." The premises vary but always prove something about the themes.
The quality of prose in a commercial novel is usually adequate (although I've seen some pretty bad examples among astonishingly popular, published books). In popular fiction, the language is easy to understand and tends toward short sentences. You don't see many multisyllabic, obscure words or lengthy complex sentences. The plot relies on action rather than a character's inner turmoil, although a good commercial novel provides enough character depth to keep the reader interested.
Let's look at the elements of a purely literary novel. It doesn't fit easily into a genre. The theme and premise of a literary plot are deep, universal and often profound. It frequently has a tragic ending, but however it ends, it should deliver wisdom and enlightenment. The protagonist doesn't necessarily get what she wants but acquires something more important: self-awareness and understanding.
An example would be Kazuo Ishiguro's Remains of the Day. It's about an English butler who is so devoted to his profession that he won't accept the love of a woman he admires. I see the theme as love versus career, and the premise as "A person who chooses career over love will regret that choice in the remains of his days."
The prose in literary fiction is rich and filled with subtext (thoughts and feelings, both known and unknown to the character, that are below the textual surface and conveyed indirectly or obliquely). I'll write more about subtext later because I believe every worthwhile literary novel must have it.
The literary pace is usually slower because the plot tends to concentrate on the characters’ emotional inner turmoil rather than external action. A good literary novel is meant to be savored rather than consumed quickly. Commercial fiction is entertainment, while a literary work is intended as literature that has lasting artistic merit. The best novel can do both.
As I said earlier, novels fall along a spectrum. My agent referred to my first novel, The Starlite Drive-in, as a "literary novel with a commercial bent," and my editor called it a "commercial novel with a literary bent." Or maybe it was vice versa, but the point is it's quite possible to write a hybrid book, and in fact most agents are looking for novels that combine the best elements of literary and commercial fiction.
Where does your novel fit in this spectrum? Is it literary, commercial or a hybrid?