Mr Brough said children had no hope of getting on in life if they could only speak a language that just a handful of people could understand.
He said the plan was backed by grandparents in indigenous communities who wanted their young people to have the same opportunities as white children.
"If we are all going to aspire, as most politicians say they do, that all Aboriginal children should have the same life expectancy, the same capacity to enjoy the bounty of this nation, then we are just living a lie if we don't ensure that they have the first fundamental that they need to be mobile citizens of Australia, and that is the English language," Mr Brough told ABC radio today.
"These children, like all Australian children, will benefit from a strong grasp of English which allows them to make choices in their lives which they simply don't have when they only speak a language which only a handful of people can understand."
But Mr Brough's proposal has been met with amazement by NSW's first Aboriginal MP, Linda Burney.
"I think that he needs to understand that culture and country is incredibly important to Aboriginal people and they will be protected at all costs," she told ABC radio.
"Aboriginal kids do need to be bilingual but it's a bit rich coming from a person who actually is part of a government that took away funding for bilingual programs in the Northern Territory."
Mr Brough said school attendance was essential and he would look into ways to encourage indigenous children to go to school, including stopping parents' welfare payments if their children were truants.
"I am looking at welfare changes which can help with school attendance," he said.
"I will look at anything at all, both incentives as well as things such as welfare quarantining, to assist the circumstances."
Mr Brough said there was a lot of support in indigenous communities for his plan.
"This is probably the number one issue I get from grandparents in remote communities," he said.
"It's been pushed at me. Particularly the grandmothers, the grandfathers, they are so adamant.
"They understand the value of English and they understand the value of an education."
Mr Brough's comments come two days before the 40th anniversary of the 1967 referendum which allowed Aborigines to be counted as Australians and gave the federal government the power to make laws for them.