Even if the two exceptions were taken away — time travel and FTL — the stories one could tell would not suffer. Being true to science does not mean being a poor writer!

And, in fact, there is a conflict there, it seems to me, but in the reverse direction. Fantasy, and fantasy-flavored science fiction, tends to offer situations where “anything can happen” — and as soon as all things are made possible, it’s easy to lose interest in a character’s predicament. She can unleash another magic spell she happened to know — or a new creature or demon or whatever can cross over through another portal, or whatever.

Fantasy writers who tell good stories tend to work hard setting up “rules of the universe” so that there can be limitations on the characters, almost like real life. For science fiction writers, perhaps time travel backward is the worse; if it doesn’t work out, why not go back and try it again? So writers have to contrive some reason why this won’t be allowed.

When you write hard science fiction, you can still astound and amaze — the potentials for even the science we can see are huge. But the focus, then, can be on your characters, and your story, and you can keep the pact with the reader that you’re not going to break the rules of physics for the purposes of plot.

It seems to me that real, “hard” science is not a constraint, it is a simpler way to establish the reader’s trust while you work the real magic, in the tale you’re telling and the characters you create.