Here are two possibilities:

One thing that tempts scientists to ignore warnings that a theory might be wrong is that the troubling results from experiments or investigations may be quite small. Thus, it is tempting to ignore them or explain them away.

For example, as Mario Beauregard and I note in The Spiritual Brain:

In science, small, persistent effects cannot be ignored. Sometimes they force a revision of major paradigms. For example, Lord Kelvin remarked in 1900 that there were just “two little dark clouds” on the horizon of Newtonian classical physics of the day, namely, Michelson and Morley’s measurements of the velocity of light and the phenomenon of blackbody radiation. Kelvin was certain that these troubling little clouds would be blown away shortly.149 Yet all of modern physics—relativity and quantum mechanics—derives from these two little dark clouds. (p. 172)

Other times, they can be so emotionally committed to one picture of the universe that they cannot imagine another. Even Albert Einstein had this problem. But he isn't the only one.

Science journalist John Horgan created a minor stir a decade ago with his book, The End of Science, arguing that the major science discoveries are all behind us. Now that was hardly a popular thesis. As he recently recalled,

One of my most memorable moments as a journalist occurred in December 1996, when I attended the Nobel Prize festivities in Stockholm. During a 1,300-person white-tie banquet presided over by Sweden's king and queen, David Lee of Cornell University, who shared that year's physics prize, decried the "doomsayers" claiming that science is ending. Reports of science's death "are greatly exaggerated," he said. Lee was alluding to my book More than a dozen Nobel laureates denounced this proposition, mostly in the media but some to my face, as did the White House science advisor, the British science minister, the head of the Human Genome Project, and the editors in chief of the journals Science and Nature.

Nothing like being toast of the town! However, the initial uproar seems to have blown over. Discover Magazine invited Horgan to revisit his thesis in its October 2006 edition. And it has stood up surprisingly well, at least on the surface.

There have been advances. Dolly the sheep? New vaccines? The chess computer? New antibiotics? Alternative energy sources?

Yes, but, as Horgan notes, these advances depend on existing science. They do not forge new frontiers in our understanding of our world.

Some people dismiss Horgan's thesis outright because prophecies that "everything has gone downhill" are common, and they are usually wrong.

But, he rightly notes that dismissing his thesis only on that ground is based on a fallacy - that past experience predicts future experience.

Does some natural law govern science discovery? If so, what is it? If not, we cannot assume that past experience is a prediction. Actually, whole centuries have passed without significant discoveries, and no natural law was violated. Let's look a little deeper.

Horgan makes one fundamental assumption that I think is wrong and his error is the main reason why I am more hopeful than he is. He clearly sees science as nothing more or less than applied materialist philosophy.

For example, he reminds us to temper our expectations of future science by writing, "Evolutionary biology reminds us that we are animals, shaped by natural selection not for discovering deep truths of nature but for breeding."

Hmmm. Where have I heard this before? We are animals who could not recognize truth? Oh, yes!

“Our highly developed brains, after all, were not evolved under the pressure of discovering scientific truths but only to enable us to be clever enough to survive and leave descendants.” - Francis Crick, co-discovered of the double helix, 1995 “Our brains were shaped for fitness, not for truth. Sometimes the truth is adaptive, but sometimes it is not.” - Harvard cognitive scientist Steve Pinker, 1997

This materialist silliness slows current science down. People who don't think they can discover truth may turn to propaganda, I suppose, in an effort to shore up what they think they already have and don't want to lose. .

Yes, there are new frontiers in science. The greatest new frontier is human consciousness. But no materialist hypothesis for consciousness is believable. So the doors to new discoveries are wide open, but a materialist may not want to walk through them.

Today, it is actually better to be a non-materialist. You can reasonably believe that you actually have contact with reality; your mindless selfish genes are not simply using your brain to get themselves spread.