By Dain Oh, The Readable
Dec. 29, 2022 8:00PM KST Updated Dec. 30, 2022 2:50PM KST
On a Saturday afternoon, a disturbing link was uploaded to a company’s communication channel containing information about vulnerabilities that were discovered in a particular software application. One member of the firm recognized the seriousness of the situation and thought to himself, “This could cause a stir.” He understood that malicious actors can exploit such weaknesses in software to compromise the security of users. While the vulnerabilities described in the link seemed quite significant, he did not imagine that he would end up doing something just as significant to address the issue. As the cloudy afternoon turned into a frosty evening, he completed the regularly scheduled work that had occupied him during the day and began writing code to address the vulnerabilities he had discovered earlier that afternoon.
On December 11, 2021, Yang Bong-yeol, working in South Korea, shared a free, simple execution file on GitHub, a collaboration platform for software developers run by Microsoft. The file, which had no user interface, offered minimal features through command lines to scan for vulnerabilities in products that make use of Apache Log4j software. Yang spent just two hours developing the scanner, but it quickly became the most downloaded scanner for Log4j vulnerabilities in the world, with over 1.2 million downloads. This number far surpasses what both Google and the United States Cybersecurity and Infrastructure Security Agency (CISA) have achieved so far.
It has now been a year since Yang released the Log4j scanner to the world, and The Readable recently met with him in Seoul to learn more about the development of the tool and the events of the past year. The interview took place at the office of Logpresso, the security operation platform company founded by Yang in 2013, where he serves as CEO.
◇ 150 lines of code
“The potential impact of the Log4j vulnerabilities seemed so widespread that I felt it was essential to identify potentially vulnerable services for anyone who might be affected,” Yang explained to The Readable, elaborating on why he created the Log4j scanner on that Saturday evening. The first version of the scanner was made up of only 150 lines of code. Typically, even the most basic software, such as a simple game application, includes a minimum of 1,000 lines of code.
On the Friday before Yang wrote the first line of code for his Log4j scanner, the United Kingdom’s National Cyber Security Centre (NCSC) issued a public warning about Log4j vulnerabilities and advised organizations to take action in response to the threats. “A number of vulnerabilities have been disclosed that affect multiple versions of Apache Log4j,” the NCSC wrote. The intelligence agency warned companies further, saying “scanning and attempted exploitation [through the vulnerabilities of Log4j] has been detected globally, including in the UK.” The next day, the director of the United States Cybersecurity and Infrastructure Security Agency (CISA), Jen Easterly, released a similar statement, saying that “[CISA is] taking urgent action to drive mitigation of this vulnerability and detect any associated threat activity.”
Log4j is a tool used by information technology developers to record the operations of their web services and applications. In the IT industry, this process is called “logging.” Technically, Log4j is an open-source logging library, which means it is a collection of frequently used source code for logging that has been developed through collaboration among programmers. Log4j is so widely used that it was difficult to find services that were not impacted by its vulnerabilities. The Common Vulnerabilities Scoring System (CVSS) rated the Log4j vulnerabilities, or Log4Shell, a 10 out of 10, making it one of the worst vulnerabilities in internet history.
Among the multiple vulnerabilities related to Log4j, Yang’s scanner was specifically designed to scan for CVE-2021-44228. CVE stands for Common Vulnerabilities and Exposures and is followed by identification numbers that signify unique security flaws. Despite the potential impact of Log4Shell, Yang did not expect his work to receive such explosive reactions online. When he returned to his computer later that evening, he found hundreds of responses to his file. The number of visitors to his GitHub posting exceeded 300 in just a few hours, and his work was even featured on Reddit, a social media platform. As his work gained more attention, requests for it came in from all over the internet. When some users questioned the authenticity of his work and suggested it might be a malicious file created by threat actors, Yang decided to publish the source code of his file. This decision led to even more feedback, including complaints. “Everything, including the complaints about the file, was unexpected,” Yang said.
◇ Journey to mitigating Log4Shell
Yang received complaints for a variety of reasons. One user asked him to develop a version of the file for UNIX systems. Another requested a version that could run on a system without having to install Java. More users appeared and urged him to enhance certain features of the scanner, such as adding the ability to exclude designated files from the scanning process. Yang received an endless stream of requests as his work traveled the world, being uploaded to numerous security postings of major companies and governments, including VMware, DELL, SAS, and Cisco in the private sector, and security institutions in Portugal, Belgium, and Ecuador in the public sector. Some universities in the United States and Germany also posted the Log4j scanner on their websites.
“I thought the project would be finished once we were able to fix the first vulnerability, but more vulnerabilities in Log4j appeared during this process,” Yang said. For the next month, security researchers and developers around the world continued to investigate Log4j and discovered additional flaws. The unprecedented scale of the collaborative study included all versions of Log4j, even though the initial issue was with the Log4j 2.0 version. Eventually, people began asking Yang to develop a scanner for all versions of Log4j.
“People got angry with me when I told them that some issues were not my responsibility,” Yang recalled. The issues that people encountered included memory problems when scanning large volumes of data, as well as errors related to symbolic links, which are files that point to other files. If these errors remained unfixed, the system would become stuck in a loop. While resolving these types of problems was the responsibility of the developers at each organization, people expected Yang’s support. Most of the developers who asked for his assistance were located in the United States and Germany, which meant that Yang’s sleep was disrupted. The most active communications with other developers took place from 2 to 5 a.m. in South Korea. For at least three weeks after the first Log4Shell became publicly known, Yang was staying awake until 5 or 6 a.m. every day.
There were several other scanners for Log4j vulnerabilities developed by different organizations. However, the Logpresso scanner received more attention from users because of its ability to automatically remove vulnerabilities identified during the scan. Yang believed that this feature was the main reason his scanner was the most popular among the various options available. Additionally, Logpresso offered its scanner through a cloud service, called “Logpresso Watch.” Later, the digital forensic tool Logpresso Mini was also made available on GitHub for public use.
◇ Contributions and recognitions
A Dutch developer known as “romestylez” on GitHub used Yang’s software to scan 2,900 servers. William Easton, the CTO of the U.S.-based cybersecurity company VERVE Industrial Protection, commented on the GitHub page that he had used the tool to scan over 10,000 servers. A German IT engineer with the handle “doctore74” on GitHub also praised Yang’s “awesome work” and mentioned using the tool to scan “many” servers, integrating it into the Checkmk community, which is an open-source infrastructure monitoring tool.
Furthermore, Christian Kemper in Germany provided Yang with multiple patches, which are programs designed to fix security flaws in software. James Stewart in the U.S. led the integration of the Log4j scanner into the HCL BigFix community, which is an IBM-powered endpoint management platform that automates the management process, including patching various systems.
Global software juggernaut SAS also recognized Yang’s work. In its support page introducing “Loguccino,” a customized software application for SAS users, the company wrote that “SAS appreciates the excellent work of the Logpresso team.” The South Korean government also acknowledged Yang’s achievements, as the Ministry of Science and ICT awarded him a ministerial commendation on November 28 for his contributions to the development of the information security industry.
Over the course of about two months, the scanner underwent 60 updates after the initial release of the first version. As it was extensively downloaded by users worldwide, it was developed to support various operating systems and file formats, such as Linux x86, Linux ARM64, Windows, macOS, and JAR. Beginning with version 1.6.3, which recorded over 10,000 downloads for Linux x86 and Windows, some versions easily exceeded tens of thousands of downloads. The 2.9.2 version for Windows set an impressive record with 356,000 downloads. Incorporating feedback from contributors worldwide, Yang’s scanner evolved into the 3.0.1 version, which was released on February 14 of this year.
“It all started with the simple desire to help others,” Yang said modestly. This highlights the power of goodwill and the importance of collaboration in defending against cyberattacks in the current age of their proliferation. As Yang noted, the willingness to help others and work together will be key to our success in facing these challenges.
The cover image of this article was designed by Areum Hwang.
The photos of this article were taken by Sukwoon Ko.
The content of this article was copyedited by Nate Galletta.
Dain Oh is an award-winning cybersecurity journalist based in South Korea and the founding editor-in-chief of The Readable by S2W. Before joining S2W, she worked as a reporter for The Electronic Times, the top IT newspaper in Korea, covering the cybersecurity industry on an in-depth level. She reported numerous exclusive stories, and her work related to the National Intelligence Service led to her being honored with the Journalist of the Year Award in 2021 by the Korea Institute of Information Security and Cryptology in a unanimous decision. She was also the first journalist to report on the hacking of vulnerable wallpads in South Korean apartments, which later became a nation-wide issue.
