"I don't fit in in Madison anymore because I want to listen to both sides," Ellis told Local 5's Jenn Sullivan"
His decision comes two days after a conservative group released a secret video recording of the senate president boasting about having a separate political operartion to raise campaign funds.
He said the release of the video was just one of many reasons why he felt it was time to retire.
"Every citizen should be upset about the fact that the idea of political privacy is disappearing from the political arena," he said.
In the five minute video Ellis also talks about somebody else politically attacking opponent Rep. Penny Bernard Schaber.
"I never intended to, in my heart, in my voice, or in my eyes to go after Penny Schaber. I like her. I respect her. I disagree with her," Ellis said.
In light of what happened his senate colleagues are standing behind his decision.
"We're going to learn one thing, with video cameras--you better be careful with what you say because it will be used and held against you," said Sen. Dave Hansen, D-Green Bay.
At a press event in Green Bay Friday, Gov. Scott Walker called the video unfortunate.
"The video itself was disserving. And I guess it's unfortunate that one particular thing ends up being a large part of the reason to end someone's career," Walker said.
Despite the events leading up to this decision, both Walker and Hansen said they respected Ellis' service and wished him well.
Copyright Copyright 2014 Nexstar Broadcasting, Inc. All rights reserved. This material may not be published, broadcast, rewritten, or redistributed.