Class E airspace is still labeled "Controlled Airspace".
And 91.123b says:
"Except in an emergency, no person may operate an aircraft contrary to an ATC instruction in an area in which air traffic control is exercised."
So, to me, while you are not required to request ATC services in Class E, if you do receive ATC instructions you must comply.
Now of course, in the OP's scenario, there was no requirement for him to have the radio tuned in to the tower freq, so that muddies the water, but just being outside of A-D airspace doesn't mean you can disregard ATC instructions.