It could lead to rogue GM crop plants that are harder to control, warns the government agency, which champions wildlife conservation in Britain.
Farmers in Canada are advised to leave a distance of 175 metres (575 feet) between different GM varieties but the guidelines are voluntary.
English Nature says genes from different GM varieties are accumulating (gene stacking) in plants that grow from seeds spilled at harvest (volunteer plants).
In the UK, a code of practice for farmers growing GM crops has been developed by the industry body Supply Chain Initiative for Modified Agricultural Crops (Scimac).
But Dr Brian Johnson, English Nature's biotechnology advisor, says the guidelines may not be enough.
"Our report shows that the Scimac code is probably inadequate to prevent gene stacking happening in Britain, if these crops were commercialised," he said.
"The consequences for farmers could be that volunteer crops would be harder to control and they might have to use different, and more environmentally damaging, herbicides to control them."
The environmental group Friends of the Earth says the research shows the UK Government faces a stark choice between siding with industry or the public.
Adrian Bebb, GM campaigner, said: "Either we keep the current separation distances between GM and non-GM crops, in which case contamination and gene stacking looks certain, or we can have an effective separation distance - of at least three miles (5 kilometres) - in which case GM crops have no commercial future in the UK. There is no third way.
Monarch butterfly: Potent symbol of GM crop fears
"The government must choose between continuing its support to the biotech industry or backing the British public who have clearly said they don't want GMOs."
The European Commission recently proposed that a threshold of up to 0.7% GM seed should be allowed in batches of conventional crop seed.
English Nature is concerned that if this proposal were to be adopted, gene stacking might occur.
The report comes a day after the UK's leading body of scientists, the Royal Society, called for stricter safety controls for GM food.