The court also rejected the company's argument that it was a different entity from the steelmaker that forced the South Koreans to work for it during the war.
"The ruling is a judgment that is impossible in light of global law", Prime Minister Shinzo Abe told reporters at the Prime Minister's Office. The victims said they returned home after the Korean Peninsula was freed from Japanese colonial rule, which lasted from 1910 to 1945.
The court's final decision is likely to have a significant impact on Japanese-South Korean ties, politically and economically, as Japanese firms involved in similar lawsuits could face similar outcomes.
The Japanese government and firms accused of harsh labor have claimed Japan's provision of US$500 million to Korea as part of the deal was in exchange for a complete and final settlement.
South Korea's top court ruled on Tuesday Japan's Nippon Steel & Sumitomo Metal Corp must compensate four South Koreans for their forced labor during World War II, a verdict Japan denounced as " unthinkable" while expressing hope that the uneasy neighbors' cooperation on North Korea would not be hurt.
"Today's ruling by the South Korean Supreme Court has one-sidedly and fundamentally damaged the legal foundation of Japan-South Korea relations", Kono said.
Nippon Steel said on Tuesday the ruling was "deeply regrettable" and that it would carefully review the court decision, taking into account the Japanese government's response.
But the victims - along with two others including Lee - launched a separate action in South Korea in 2005, and in 2012 the Supreme Court in Seoul ruled that the company was liable.
The government "respected" the court ruling, the prime minister's office said, and was "saddened by the pain the forced labor victims had to endure". Two of the group sued the company in Japan in 1997, but their case was dismissed.
"Japan will take resolute action, considering all options available including an global trial", Foreign Minister Taro Kono said, suggesting that Japan may file a complaint with the worldwide Court of Justice.
He warned that his administration is considering "every option" as well, including taking the case to an global court.
At that time, the court sent back such compensation cases, including Nippon Steel's case, to Seoul High Court.
Tuesday's ruling "not only clearly violates the Japan-South Korea pact on rights to claim, but also inflicts unfair damage on a Japanese company", Kono told the South Korean ambassador in Tokyo.
South Korea commented more cautiously, with its foreign ministry saying that Tokyo and Seoul "should gather wisdom" to prevent the ruling from harming their relations.
South Korean Foreign Ministry spokesman Roh Kyu-deok said it was "exploring all possibilities" but has raised the need with Tokyo for joint efforts to keep the matter from hurting ties.
Over the lawsuit, the top court faced allegations of delaying reaching a ruling for five years as wished by the administration of former South Korean President Park Geun-hye, who was concerned about deterioration in the bilateral relationship.