That is definitely true, especially for plugins that add things in the backend — we would never want to load anything widget specific on all other admin pages, for example. However, WordPress does concatenate all scripts (but not styles) in the backend by default.
The issue with loading assets conditionally on the front-end is that it’s very hard to determine reliably when the assets are really necessary. Detecting shortcodes might work only for scripts since they can be enqueued in the footer while CSS must be in the <head>.