I use emacs several hours each day and have written quite a few customizations myself in elisp. Enough that I think I have a handle on the question.
The Emacs APIs for window management, buffers, and other various "chrome" so to speak are pretty good and easy enough to grasp, once you get over the Lisp hump. But the Emacs APIs for managing text in buffers is difficult to wrap one's brain around, because it takes a very different approach to text management than most Unix tools. Basically, you write out the editor commands necessary to do the manipulation you desire. That sounds straightforward enough, but it takes some getting used to.
What Emacs really excels at, though, and what makes it stand out from other customizable editors is the fact that you can literally change the beast from the inside. Since Emacs is basically a Lisp environment, with some large portion of the code written in Lisp, it's very easy to pop functionality in and out to test it out, or to make changes on the fly. In a Lisp buffer, you can just hit Ctrl-J to evaluate the expression immediately preceding the cursor and print the result into the buffer (or use Ctrl-X Ctrl-E to run it without printing the result). Or hit Meta-: from anywhere to run some Lisp interactively from the minibuffer, affecting your environment in real time. This ability to toy around with the running editor itself is a feature unmatched by pretty much any plugin editor. The built-in devtools with Javascript console in Chrome are a close analog in a web browser, but that is limited to the document you are viewing, and you can't control the entire browser experience from that console. With Emacs, you can.
So yes, I'd love to see Emacs add modes that move beyond text editing and allow rich buffers and embeddable videos, audio, and widgets of all sorts. There exists a very rudimentary capability for this sort of thing in Emacs now, but it is extremely limited.
The Emacs APIs for window management, buffers, and other various "chrome" so to speak are pretty good and easy enough to grasp, once you get over the Lisp hump. But the Emacs APIs for managing text in buffers is difficult to wrap one's brain around, because it takes a very different approach to text management than most Unix tools. Basically, you write out the editor commands necessary to do the manipulation you desire. That sounds straightforward enough, but it takes some getting used to.
What Emacs really excels at, though, and what makes it stand out from other customizable editors is the fact that you can literally change the beast from the inside. Since Emacs is basically a Lisp environment, with some large portion of the code written in Lisp, it's very easy to pop functionality in and out to test it out, or to make changes on the fly. In a Lisp buffer, you can just hit Ctrl-J to evaluate the expression immediately preceding the cursor and print the result into the buffer (or use Ctrl-X Ctrl-E to run it without printing the result). Or hit Meta-: from anywhere to run some Lisp interactively from the minibuffer, affecting your environment in real time. This ability to toy around with the running editor itself is a feature unmatched by pretty much any plugin editor. The built-in devtools with Javascript console in Chrome are a close analog in a web browser, but that is limited to the document you are viewing, and you can't control the entire browser experience from that console. With Emacs, you can.
So yes, I'd love to see Emacs add modes that move beyond text editing and allow rich buffers and embeddable videos, audio, and widgets of all sorts. There exists a very rudimentary capability for this sort of thing in Emacs now, but it is extremely limited.