The court found the Independent Planning Commission failed to take into account the impact of all the carbon pollution associated with the project, including pollution from the exported emissions when the coal is sold and burned overseas.
Mach Energy's Mount Pleasant coal mine expansion near Muswellbrook is one of the most polluting coal projects that was seeking approval in Australia.
The project is so big it covers an area which would almost cover the entire electorates of Sydney and Grayndler.
The decision comes after a challenge from the Denman, Aberdeen, Muswellbrook, Scone Healthy Environment Group.
While this is a welcome result, the NSW Land and Environment Court will have to consider whether conditions can be imposed that would validate the approval, or whether the project must return to the planning commission.
"There are two other coal mines that were granted extension by the federal government in the Hunter Valley," said Richard Denniss, Executive Director of The Australia Institute.
"While it is welcome news that one may not go ahead, these approvals are inconsistent with Australia's climate goals and reinforces the country's reputation as one of the world's major fossil fuel exporters.
"To approve huge new coal mines while bidding to host COP31 is a slap in the face to our Pacific neighbors, who have clearly and repeatedly requested that Australia stop expanding fossil fuel production.
"Meanwhile, the International Court of Justice has confirmed that states have binding legal obligations under international law to prevent climate harm and protect present and future generations.
"The science is clear - the best way to do this is to stop approving coal and gas mines."