WASHINGTON—The Food and Drug Administration ordered GlaxoSmithKline PLC to stop enrolling patients in a clinical trial of its troubled diabetes drug Avandia while the FDA weighs whether to pull the drug from the market.

The trial, comparing Avandia to rival drug Actos from Takeda Pharmaceutical Co., has enrolled only about 1,100 patients of a targeted 16,000.

"The study is effectively over," said David Nathan, a diabetes specialist at Massachusetts General Hospital in Boston, who isn't involved in the trial. "Also, this gives you a forecast that FDA may actually remove [the] drug from market."

FDA officials cautioned that the agency hasn't made a decision about Avandia, which has been linked in studies to a higher risk of heart attack. The agency said patients who have already enrolled in the Avandia-Actos trial, known as TIDE, can stay in it for now.

Glaxo says Avandia is safe and effective and should stay on the market. It said it is "committed to working with the FDA in the best interest of diabetic patients."

Last week an FDA panel voted 20-12 to leave Avandia on the market, but many of those 20 called for additional restrictions. The FDA told Glaxo to update the trial's investigators and others involved about new safety information presented at the FDA panel meeting. It said the information should be used to update the informed-consent forms for those already in the trial.

The TIDE trial is designed to compare the cardiovascular risk from Avandia and Actos, another drug that lowers blood-glucose levels. Critics including FDA drug-safety officer David Graham call the trial unethical, saying patients shouldn't be given a drug with known safety risks. The trial's defenders say more data is needed to illuminate the risk from each drug.

The FDA's deputy commissioner, Joshua Sharfstein, has suggested that if the trial is ultimately ended, that could lead the agency to order a halt in sales of the drug. He said in an earlier interview that the decision on the trial "cannot be de-linked from the agency's view of Avandia."

The decision on Avandia is drawing close attention in the industry as a marker of the FDA's broader stance on drug safety. Sales of the drug, once a multibillion-dollar-a-year seller in the U.S., have fallen sharply since the safety issues first gained broad attention in 2007.