Seoul, South Korea (CNN) -- North Korea has demanded that South Korea immediately return 31 North Koreans who accidentally crossed over into South Korean waters on a fishing boat -- even though the South says four of them have decided to defect.

Pyongyang's demanded was reported Thursday by the state-run KCNA news agency.
""The South Korean authorities should immediately halt their despicable unethical acts contrary to international practice and unpardonable from a humanitarian standpoint and send back home unconditionally and immediately all the unreasonably detained 31 inhabitants and their ship through the waters to which they had gone,"" KCNA said.
The defection of the four North Koreans could spark tensions on the peninsula, where joint U.S.-South Korea military drills kicked off this week. North Korea had threatened to engulf Seoul in a ""sea of flames,"" a day before the opening of the exercise.
South Korea's intelligence agency questioned the North Koreans for almost a month since they crossed over in early February before deciding to release the group, the aid agency said.
There are currently more than 20,000 North Korean defectors in South Korea, according to the South's Unification Ministry. The number of annual defectors has risen dramatically since the turn of the century and continues to climb.
Defectors who are forcefully repatriated to the North after attempting to cross over into third countries such as China, Mongolia or Southeast Asia are commonly sent to prison camps as a punishment. Many defectors who live in South Korea fear for the lives of their family or relatives who remain in the North.
CAPTION: In this Jan 19, 2011 file photo, a North Korean soldier (C) and a South Korean soldier (R) stand guard in the Demilitarized Zone.