There absolutely is a difference. You can see it in the natural variability of hop crops from year to year. Growing conditions (moisture, soil composition, nitrogen, sun, pests) can all have a great effect on hop flavor, aroma, bitterness, storability, etc., even with the same hop plant in the same field.
Look at German varieties for a good example. Most grow very poorly outside their home-turf. Some are adapted to US/UK/Chinese/wherever conditions, but mostly don't share the qualities of authentic German-grown specimens, even though they may be genetically identical. Most often they are crossed/modified to make something more suitable for local conditions.
So yes, hop terroir is very real.
Or at least people who grow them think so. From the paper presenting the results of the 3rd year of the hop variety trial at UVM:
"Hops, like grapes, have terroir. Their brewing characteristics and oil content are reflective of their microclimate. Hop varieties grown on the East coast, even though genetically the exact same as varieties grown elsewhere, will not be like hops in the Pacific Northwest or Europe due to different soils and different climates."