Thailand’s National Security Council secretary-general Paradorn Pattanathabutr said the first meeting with the rebels led by the National Revolution Front, also known by its acronym BRN, will focus on “developing relationship and trust.” The talks are being held in Malaysia.

Violence has occurred nearly every day in Thailand’s Muslim-dominated three southernmost provinces since the insurgency erupted in 2004, and more than 5,000 people have been killed. The militants have mainly targeted security forces and teachers, who are seen as representatives of the government of the predominantly Buddhist nation.

Muslims in the region, which was an independent Islamic sultanate until it was annexed by Thailand in the early 20th century, have long complained of discrimination by the Central government in Bangkok, and the insurgents are thought to be fighting for autonomy. But the insurgency remains murky, with militants making no public pronouncements on their goals.

“I believe that today’s talks will lead to an atmosphere that yields solutions, or yields progress that would result in solutions. The main issues are to reduce the violence,” Mr. Paradorn said ahead of Thursday’s meeting.

“I am confident that there will be communication and attempts to bring understanding, and BRN is a big movement that can relay that message. I believe they can send the message to the operative level. Then it will be concrete, but we have to give it some time,” he said.

The Thai government inked an agreement to start talks with BRN last month in Malaysia during a visit by Prime Minister Yingluck Shinawatra. Malaysia, which borders Thailand’s restive southern provinces, is acting as a facilitator to bring some of the insurgents to peace talks.

Mr. Paradorn led a 15-member delegation to Thursday’s talks. Apart from three BRN leaders who signed the pact last month, it is unclear who the other representatives of the rebels are. Malaysian officials, when contacted, declined to give details citing security reasons.

The Thai government and military have struggled to identify legitimate participants for the peace process, as the militant leadership is not clear and no groups have stepped out to take responsibility for the daily attacks in recent years. The insurgency is believed to be highly decentralized, with local units having the freedom to choose targets and campaigns.

The BRN is one of several separatist movements that have made public calls for a separate state in Thailand’s south.

Just hours before the talks started, suspected militants killed three soldiers in a roadside attack in Narathiwat province.

Police Col. Suchart Sa-eed said militants detonated an improvised bomb and opened fire at soldiers who were on foot patrol in Cho Airong district. He said five soldiers were also wounded in the ambush, which took place in a village that is home to a key BRN leader.