Heterodox economics is anything outside the mainstream (which is sometimes pejoratively referred to as "orthodox" economics). It's difficult to be precise about what qualifies as heterodox, especially since the boundaries are changing over time. The term can be applied to a school of thought, a piece of research, a researcher, or a journal. It is my impression that the distinction is sharpest for journals: there doesn't seem to be any disagreement about which journals fall into which category (a list of heterodox journals is here). I don't know whether other fields have this kind of division. I've heard of, for example, a subset of political scientists who think that the work being done by some other subset of political scientists is worthless, but not of a majority against a minority.
Although I can’t come up with a non-tautological characterization of what distinguishes heterodox economics from the mainstream, it seems to be a question of what topics are of interest, what methodologies are most valid, or what political point of view is being promoted. Greater extremity in any sense tends to be less likely to be mainstream. Many of the areas of contention are macroeconomic. I'm interested in these debates, but as a microeconomist I’m only a spectator, and the finer points are often lost on me. There is also some disagreement about what defines a school of thought. For example, some Austrian economists categorically reject the use of mathematics to analyze social phenomena, and others say that this rejection is not in the true spirit of Austrian economics. There has been a lengthy debate on this question, for example here. Apart from the substantive issue—the role of mathematics in social research—I don’t find the debate over how different views should be labeled very interesting. The heterodox label itself is somewhat arbitrary and, in my opinion, not constructive. The important question to ask about any line of research is what insight it offers, and there is at best an imperfect correlation between this question and the categorization of research as either inside or outside the mainstream.
An economist might choose to pursue heterodox research for a number of reasons. There are those who seem to object to the mainstream in any sphere, or perhaps just enjoy being outsiders. Often it is simply a matter of the specific topics or methodologies of interest to the economist. Whether because of these areas of interest or for other reasons, heterodox economists commonly have not been successful in the mainstream. Here is where it gets tricky. If one lacks the ability or desire to go through the rigors of mainstream research, criticizing the mainstream becomes appealing. This fuels the mainstream's ability to dismiss heterodox research as amateurish (this post goes so far as to call the entire body of heterodox research "a joke"). At the same time, insiders always have incentive to keep outsiders out, for whatever reason, and economists understand this better than anyone. I have seen plenty of heterodox research that is of little or no value, whether because it is poorly executed, uninformed, or misguided. On the other hand, the same could be said for some mainstream research, and it does the profession no good to ignore research simply because it could be labeled as heterodox.
I don't think that mainstream economics is in danger of being overthrown, or that an overthrow is desirable; but I think that the profession would do well to be more open in terms of what questions are interesting, what methods are useful, and what we consider to be evidence. Personally, I think it's interesting to see how qualitative methods can contribute to economic understanding. At the same time, I support the mainstream view of the value of mathematical analysis (I think that quantitative and qualitative methods each have their own advantages and their own pitfalls, but that is the subject of another post). If I could do anything to change the relationship between mainstream and heterodox economics, it would be to blur the line between them. The profession does incorporate heterodox thinking over time, but I would say that this is happening too slowly. Giving some consideration to alternative points of view can never hurt, and disregarding these points of view can lead to some massive fails.
One of my students—intelligent, thoughtful, and responsible—came to my office during his final undergraduate semester, while he was taking a course in evolutionary economics. He had previously taken my classes in (mostly neoclassical) micro theory and industrial organization. He was having what I think is a common reaction from students when they are first exposed to heterodox thought: something like "Wait a minute! Everything I've been learning is wrong!" I would never fan the flames of such a reaction—quite the opposite—but I think it is useful for undergraduate economics majors to have some acquaintance with ideas outside the mainstream. In any economics class, it is worth mentioning that controversies exist. A professor who, for example, teaches a single theory of economic fluctuations without even mentioning any alternatives is doing a disservice to students. And I would agree with others (sorry, can’t seem to find the references) who stress the importance of a course in the development of economic thought in any doctoral curriculum.