Tokyo Olympic organisers have warned of a potential transport crisis during the 2020 Summer Games after a decision to move the world's largest fish market was finalised, local media reported Wednesday.

A delay in relocating the famed Tsukiji market, ordered by Tokyo governor Yuriko Koike over concerns about toxic pollution at the new site, threatens the timely construction of a key road needed for the Olympics.

"It's disappointing that it has taken this long to resolve," Tokyo Olympic organising committee president Yoshiro Mori told reporters. "The transport project is the biggest issue facing the organising committee this year."

Mori hit out at the lengthy process of rubber-stamping the market's move to Toyosu, predicting it would impact transport projects, namely a road linking the Olympic village to the main stadium, part of which was set to pass through a tunnel beneath Tsukiji.

"Unless (governor Koike) has some magic solution, digging a tunnel is no longer a realistic option," said the 79-year-old former prime minister, raising the possibility of athletes and Olympic spectators being snarled in traffic jams en route to Games venues.

"Last August after she took office this was the most pressing matter that I raised with the governor," added Mori.

"If we are not able to resolve this issue and keep our promise to the IOC, it could have serious consequences for the Olympics."

Any roads built for the Olympics should not disrupt the daily routine of local residents.

"We must make sure the 80,000 to 100,000 people living in that area can go about their daily business as normal," he said. "If not, the Olympics will face criticism."

The controversy over the fish market, a popular tourist destination located not far from Tokyo's swank Ginza shopping district, is the latest in a series of setbacks for Tokyo's Olympic preparations.

The Japanese capital won the bid for the 2020 based largely on a reputation for efficiency, but organisers have faced criticism over a series of embarrassing gaffes since beating Madrid and Istanbul in the IOC vote.

Already battling to rein in soaring costs for the 2020 Olympics, organisers bungled the rollout of the main stadium, prompting Prime Minister Shinzo Abe to tear up the original blueprint over its hefty price tag.

The Games logo was then scrapped over accusations of plagiarism in a further blow.

Tokyo bid officials estimated costs of $7 billion and projected an economic windfall in excess of $25 billion for the 2020 Games, but a panel of experts have warned the overall budget could exceed that without drastic cuts.

The IOC's 2020 coordination commission is set to meet Japanese organisers in Tokyo later this month when transport is expected to be high on the agenda.