Islamist militant group al-Shabab has taken control of the port city Merca, residents say.
However the African Union force insists it still has control of the town.
The governor of Somalia’s Lower Shabelle region Ibrahim Adam told the AFP news agency that al-Shabab secured took the town without fighting.
Earlier reports had said the AU forces had left the town but they tweeted later that they “re-adjust their positions” from time to time.
“Amisom [African Union] forces moved out at midday and the local administration and all other Somali security forces left a few minutes later – and then heavily armed al-Shabab militants entered the town,” local resident Ibrahim Mumin told AFP.
“They have been addressing residents at the district headquarters,” he added.
Another resident, Mohamed Sabriye, told AP news agency that al-Shabab fighters had hoisted their flag over the city’s police station and administrative headquarters.
However, Amisom tweeted that “contrary to propaganda by enemies of Somalia, AMISOM is still in control of Marka”.
Amisom later tweeted “Our troops will from time to time re-adjust their positions for tactical purposes to guarantee defeat of terrorists. #Somalia.”
It comes three weeks after al-Shabab overran an African Union military base outside the southern Somali town of el-Ade, saying they had killed about 100 Kenyans soldiers.
Kenya has not said how many of its troops died.
Al-Shabab was ousted from the capital, Mogadishu, in August 2011, but still has a presence in large areas of southern Somalia and often stages attacks across the country.
Analysis: BBC Monitoring Africa Security Correspondent Tomi Oladipo
It’s clear that the African Union Mission in Somalia (Amisom) is having problems securing the region.
Sources within the mission say this is a tactical move. The problem, however, is that as soon as these troops leave, al-Shabab militants are swiftly replacing them.
It’s no doubt a major setback for the regional forces, seeing as they would have to fight their battles afresh to regain these regions.
In January, Kenyan troops withdrew from other parts of southern Somalia after they suffered heavy losses in an an attack on their base in el-Ade.
As al-Shabab fills the void, it will be looking to win the support of the communities – something the regional coalition has failed to do.
And that would be disastrous overall for the regional efforts to bring peace to Somalia. The nations contributing to Amisom are expected to meet in Djibouti later this month to review their campaign.