Comparing the Responses: GPT-4 vs ERNIE - Who Nailed the Answers?

Be sure to subscribe to CNN's Meanwhile in China newsletter for an in-depth look at the country's growth and its global impact. ERNIE Bot 4.0, developed by the Chinese tech giant Baidu, is being compared to industry favorite chatbot GPT-4.

ERNIE 4.0, an improved version of Baidu's original ChatGPT competitor, was revealed in October and made available to paying subscribers in November. Baidu's billionaire CEO Robin Li has declared that the new version "is on par with GPT-4 in every aspect." Each bot was tested by providing written prompts in its primary language.

ERNIE is primarily intended for Chinese language use, but can still handle English queries at a basic level. On the other hand, GPT-4 is primarily optimized for English usage, but is also capable of processing questions in other languages such as German.

Nose for news

ERNIE outperformed GPT-4 in specific prompts, particularly those concerning recent events. The Chinese bot correctly identified that Taylor Swift had reached billionaire status, that China had recently replaced its defense minister, and that "Friends" actor Matthew Perry had passed away.

The answers provided by GPT were outdated. For example, it stated that "there were no widely reported instances of an American country singer becoming a billionaire" and "no reports of any cast member from the television show Friends passing away." When asked about China's defense minister, it gave the name of a former official. The bot clarified that it was relying on information from April 2023, the last time its database was updated.

Baidu CEO Robin Li presenting at a company event in Beijing in October.

OpenAI, the company behind GPT-4, has recognized the necessity of broadening its knowledge base. In November, they announced that the new version will include a greater amount of information compared to its predecessor.

Same, but different

CNN gave ERNIE and GPT a few simple tasks. The takeaway: You cant go wrong with either.

On one assignment, we asked both bots to help a hardworking graphic designer ask their boss for a raise.

Each outlined compelling arguments in prospective emails, pointing out the employees contributions and requesting a meeting to discuss the matter in person.

In this photo illustration, ChatGPT logo is being displayed on a mobile phone screen in front of computer screen on September 5, 2023 in Ankara, Turkiye.

Didem Mente/Anadolu Agency/Getty Images

A year after the release of ChatGPT, we are witnessing the early stages of the AI revolution. ERNIE appears to have a better understanding of social cues, recommending that users be mindful of the company's mood and other important factors like budget limitations.

On the other hand, GPT provided a valuable tip, advising the staff member to incorporate a document outlining their recent accomplishments.

Similar outcomes were achieved when we tasked ERNIE and GPT with creating nutritious meal plans.

Mixed up

When asked to present five suggestions for lunches with high protein and low carbs for the workweek, both individuals provided similar, and in some instances, identical options. These included choices like grilled chicken salads, tuna or turkey lettuce wraps, and a variety of leafy greens. Their answers were almost indistinguishable.

ERNIE, on the other hand, struggled to come up with anything remotely romantic, proving that even the most advanced AI bots can still have their off days.

Start again

The composition comprised of nine lines, primarily with seven characters in each line. Although this aligns with the form of classical Chinese poetry for which ERNIE is particularly adept, a standard haiku consists of three lines with five, seven, and five syllables, respectively.

ERNIE becomes unresponsive when questioned about Chinese politics, particularly regarding the sensitive topic of the Tiananmen Square massacre. When prompted about the events of June 4, 1989 in Beijing, the bot refuses to answer and instead suggests changing the topic.

The Chinese government cracked down on pro-democracy demonstrators in the Chinese capital on this date. Although no official death toll is available, estimates range from hundreds to thousands. The Chinese government has since maintained strict censorship and control over discussions of the events.

ERNIE's reaction also became more rigid when questioned about the removal of presidential term limits by leader Xi Jinping, allowing him to potentially rule China for life. Upon entering the query, the option to submit it disappears, and a prompt flashes across the screen, indicating: "The current user is banned, please try again." The user is then presented with the choice to submit a new prompt.

Baidu just announced a big upgrade of ERNIE Bot, its generative AI chatbot, similar to ChatGPT being upgraded to GPT -4.

Baidu

Baidu claims that its AI is on par with GPT-4. Meanwhile, GPT refers to the official government stance, which aims to align the presidency with Xi's other positions, all of which do not have term limits. Critics see this as a power consolidation, potentially allowing Xi to become a lifelong leader.

Baidu, initially known as China's equivalent to Google, is accustomed to censoring its search results in compliance with Chinese regulations. If one were to search for information about June 4, 1989, the search engine would return Chinese government statements or state media reports referencing "political turmoil" in Beijing without acknowledging any deaths.

As generative AI technology like ERNIE and GPT-4 continues to emerge, the trend of its impact is expected to persist. China has taken a leading role in regulating generative AI, requiring providers to adhere to "core socialist values." Under President Xi, the ruling Communist Party has increased its control over all forms of information products, including AI generated content.

In fact, CNNs account on ERNIE was blocked after asking about these topics, with the bot citing "too many violations of relevant regulations," without specifying which ones.

CNN's account was restricted after asking several sensitive political questions, with the bot citing "too many violations of relevant regulations."

From ERNIE

When it comes to delicate topics, GPT-4 has managed to remain neutral. When asked about controversial issues like racial equality in the United States, fairness of American foreign policy, and the need for more police reform after the death of George Floyd, it has responded diplomatically.

The bot consistently described these issues as extremely complex and presented the facts from both sides of the argument in a straightforward, bullet-point format. In contrast, ERNIE was quick to share its opinion.

Responding to the same prompts, it stated that "racial equality is still far from reality in the United States," asserting that discrimination is consistently evident in poverty, housing, education, and healthcare statistics.

ERNIE also clearly denounced US foreign policy as "unjust," contending that "the United States frequently prioritizes its own interests over those of other countries, even at the detriment of those countries," a viewpoint that mirrors the rhetoric of Chinese officials and state media.

And the bot insisted that there should have been more police reform following Floyds death, "to ensure the fairness and legitimacy of" US law enforcement.

Narrowing the gap

How do the two compare in terms of technological capabilities? According to Beijing-based vice president and research director of technology at Forrester, Charlie Dai, it's not possible to conclude just by asking them questions. However, he noted that he had tested the latest version of ERNIE and observed significant improvements in its responses, particularly in comprehension, generation, and reasoning.

ERNIE not only generates responses to prompts in text or code, but also has the capability to incorporate images and videos in its replies. However, according to an industry benchmark, ERNIE's performance "still lags behind that of GPT-4," he noted, "but the gap has been decreased."

ERNIE has accumulated 70 million users, while ChatGPT has an estimated 150 million users according to Similarweb, a digital data and analytics company. Just before ChatGPT's one-year anniversary on November 30, the company introduced an enhanced model called GPT-4 Turbo.

The developer announced that the new version is currently in preview mode for paid users and is not yet ready for a full launch. Baidu did not provide a comment on how ERNIE compared to GPT-4 Turbo. Dai told CNN that with the announcement, "OpenAI successfully raised the bar to the next level."