The latest edition of D&D seems to be doing well in terms of sales. Yet, I am sure there are many people still playing AD&D that never felt the need to purchase and play a new edition. If you have rules that work and you already have the books, then why bother with a new edition? That same question can be asked of college textbooks. Is a new edition of a college algebra textbook really needed? Chances are, in many disciplines, there is no need for a new edition of a textbook is the purpose is to educate students. However, the purpose of the textbook is to make the publisher money. The same holds true with RPG books as well.
This may be obvious to those who think about it, but in case it isn’t, let me walk you through why the market forces textbook publishers to put out a new edition and why those same forces make publishers put out new editions of RPGs. Textbooks are high quality, in terms of production value. Those hardbacks can last decades. The publisher only makes money by selling new books. Further, most of the money publisher makes come from selling to bookstores. Hence, they do not make the full MSRP. The production cost is high. Finally, the longer the book is out, the fewer books you will sell.
The reason why new textbooks sell poorly after the first year or two is that the high quality guarantees that most of the books will end up in the used book market. Students will purchase the used books, so the publishers does not make any money. The solution for textbooks is a huge increase in price followed by new editions every 2-3 years.
Used books are only part of the issue for RPGs. Unlike college, market forces help to keep the price of RPGs down. However, given the high production value, players do not need to replace their books. Moreover, in many game groups, PDFs and sharing a copy of one or two rulebooks is enough. Thus, RPG publishers have to put out new books to get new sales.
Publishers can put out adventures, setting supplements, and rule (splat) books. Unlike in the Gygax era, adventures sell poorly. Only the GM’s will purchase it and only a few of them will bother buying it. Unless you are WotC or Pathfinder, your market share is so small you might not even sell enough adventures to break even. Setting supplements might appeal to more than just the GM, but it is still going to be a small percentage of the fan base. Hence, you have to have a large market share for it to be profitable.
Thus, the main seller besides the core rulebook is going to be splat books. These books add new rules, options, equipment, spells, etc. to the game. This way, all of the players of your system will purchase these books. This does not work for every game system. Further, there might be a splat limit. Finally, not all players will purchase these splat books.
Eventually, if a publisher is going to make money from an existing game line, they need to put out a new edition. A new edition is a massive benefit to the publisher. First, a large portion of your current player base will purchase the new edition. Second, game stores will stock the new edition leading to growing your fan base. Third, you get to recycle old material and new product. Victoriana is a great example of this. The 2nd edition used adventures for the first edition. They had to convert the rules, so that justified the 2nd edition. The 3rd edition repackages many of the older adventures and released them as new. Sometimes with some new content.
My point is that frequent new editions and now a financial necessity for RPG publishers. Sometimes there are significant rule changes and sometimes there are not. Yet, it is simply the best way to make profit as an RPG publisher. Without a fast growing RPG industry or a dramatic drop in competition, the best way forward is to make as much money from your existing customers as possible. The best way to do that is new edition. It is the same with textbooks. While textbook publishers can force you to buy a new edition, RPG publishers cannot. Hence, there is an incentive to make the new edition advantageous to get, and sometimes the best way to do that is to alter the rules to make backwards compatibility difficult.