OK, fair enough. Here's what I tend to think.
For Literate Programming as Knuth and most others have approached it, they are going from too loosely coupled source documentation (as opposed to comments, which are something different) to too tightly coupled. Embedding the source code in the documentation (albeit with specialized markup and tools for extracting the code for compilation) sounds like a good idea at first, certainly; I think most code has overly-decoupled source documentation (if it has any at all, which usually isn't the case), and that could stand to change. However, embedding the code in the documentation tends lead the developer to either treat the documentation as comments, which is bad and counter to the point of source documentation; or treat the code as an afterthought, which undermines the development process itself; or split their attention between them, leaving both bad, unfinished documentation and bad, uncommented code.
I am not sure if 'Hyper-Literate' programming would really be a better alternative or not, yet. The basic idea is that the source and documentation would be written in parallel, and contain cross-linking between parts which could be automatically checked for changes made to one to encourage updating the other when necessary. Code comments would still be code comments, but everything else would be separate, though the system should be able to generate a view of the code together with the documentation for reference purposes. It wouldn't be limited to just two documents, either; multiple commentary, documentation (design, source, admin, and user, and maybe others things as well), and change and testing audit trail documents could be linked, either automatically or by the developers, and include update constraints which could be set as reminders or even auto-generated checklists.
As for snake oil salesmen, yes and no. While there is a lot of deliberate deception going around, you have to remember that most snake oil salesmen are their own best customers - they actually think it works, and aren't just trying to trick people, but are trying to
convert them to their own delusions.
My own impression is that what happens is that teams who have exceptional devs, admins, and managers (not necessarily exceptionally
good in a general way, just exceptional in some manner, usually one specific to the problem domain at hand) get fed up with the existing process and fiddle with it until they find a good fit, or luck into a really good fit right away, and manage to use it to succeed with some big projects right off the bat.
Somewhere along the way, they decide that it must have been the process that made them successful, rather than the exceptional qualities of the team or the specific conditions of the project. Once they have Found Religion this way, they go out to share that process with everyone the meet.
The problem is that chances are, the process was incidental to the success. Maybe it was a good process, but how can you separate that from the other aspects of the projects they used it on?
That's only part of the problem. Even if it is a good process, like any good process, using it correctly requires both skill and effort. Most of the evangelists, however, overlook the amount of effort they put into making the process work, because they were enthused over it and it didn't feel like effort. They gloss over the fact that it isn't one-size-fits-all, because they can't see that themselves.
Another possibility is that because they were exceptional, they came up with a process that requires the ones applying it to be exceptional. This is actually worst for newcomers if the way they were exceptional really was that they were excellent developers, because anyone who isn't an excellent developer who tries to use it will find themselves in over their heads with that process. I think that this is a big factor in the successes and failures of the classic Agile model - the people who came up with it were so good that they would have had some success even with a poor process - but it shows up elsewhere as well.
Next, communicating the process - and weeding out the incidental parts of the process - becomes a barrier. Everyone on the team understands the process, but because they have been swimming in that water, they have trouble explaining it to anyone who isn't in their fishbowl. So they come up with explanations which may or may not actually get the ideas across. They come up with slogans which are misleadingly appealing. They give a lot of details about what they did that may or may not be relevant. Pretty soon, those explanations become a Dogma, and most people asked to use that process are getting it second or third hand from people who were converted by the originals or later evangelists, and a lot of misinformation gets passed around by people who didn't see the original Miracle themselves. This part is particularly insidious, and is a major factor how Patterns get (mis)used by most teams, specifically - most of the ones trying to 'use patterns' don't really get what they are
for, or even what they actually
are (hint: they are primarily a framework for understanding how existing code works when you are refactoring it, and don't really help much when writing new code).
Finally, as the new Process Religion spreads to management, you end up with developers who don't know, care for, or even like the process having it forced down their throats by buzzword-happy managers who know nothing about the process except that it is the latest Big Thing. If the people using a process don't know the process, and don't want to learn or use the process, or haven't been given time to learn it, then there is no way even the best process can be made to work.
A similar thing happened in education in the US during the 1960s, with the infamous New Math. It was developed in response to a perceived gap in US STEM education compared to the USSR after the launch of Sputnik; the fact that neither the 'missile gap' nor the education gap were real wasn't realized until much later. There was a difference in reported test scores between the US and the Soviet Union, but the Soviets were padding the numbers, and the people screaming about the gap were discounting the fact that the US was pretty much the only industrialized nation that doesn't track students into different ability categories from an early age - which meant that they were comparing the average of all US students to the average of the 5% of students who made it into those classes in the USSR.
The response was to go talk to modern mathematicians and see what new and exciting developments had come up that might be easier for or be more interesting to students. Since set theory, Galois theory, and working in different numeric bases were hot topics at the time, the educational researchers decided that they would be the cornerstones of this New and much better Math.
Now, the New Math had a lot of good points and a lot of bad points, but the people who developed it had considerable success despite the problems. Unfortunately, no one considered whether that success was due to the method of teaching, or abilities of the researchers and teachers who were testing it, nor did they give enough thought to how to get other teachers up to speed on it.
When mandates to use it started going around school systems, however, it was an abject failure. The main reason it failed was simple: teachers didn't know it, weren't told how to use it - just use it, that's an order! - and weren't comfortable using even the parts they did understand. Worse, it was a method that was
very demanding of both the knowledge and the talent of the teachers, and required a ton of hands-on time with each student - both for the teachers and the parents, who mostly had no idea what this weird new thing was and weren't expecting to spend so much time helping their kids on something they saw as the teachers' responsibility - at a time when school enrollment was at an all-time high.
In other words, it failed not because it didn't work for the students, but because it didn't work for the teachers. The reformers pushed it for about ten years before the teachers themselves revolted and refused to use it, insisting that the process was flawed for students when in fact it didn't even get to the students most of the time in the first place.
Sort of like how most places who say they use Methodology X actually use the same ad-hoc cowboy coding everyone had used since the 1960s (and don't even mention Waterfall - it never really existed as anything other than a strawman, even in
the paper that first presented it, and any manager who says they use it is lying through their teeth even if they don't realize it) while occasionally going through the motions of doing some incidental part of the project the X way.
