Have you read Mark Jaquith’s post? I found it very easy to follow:
Why WordPress themes are derivative of WordPress.
However, I agree that a distinction should be made between 1) actual plugins that use actions & filters and 2) stand-alone libraries that just happen to have a plugin header.
My first ever “plugin” was of the second kind and could not be considered a derivative work because it was just a single stand-alone function definition that I called from my theme.