API Threat Research: Elastic Stack Misconfiguration Allows Data Extraction
Incident synopsis
Salt Labs researchers investigated a large business-to-consumer (B2C) online platform that provides API-based mobile applications and software as a service to millions of users globally. As a result of API vulnerabilities our researchers identified in the Elastic Stack implementation, they were able to launch attacks where :
- Any user could extract sensitive customer and system data
- Any user could create a denial of service condition that would render the system unavailable
The APIs contained a design flaw, and Elastic Stack was configured with implicit trust of front-end services by back-end services. As a result, we were able to query for unauthorized customer and system data. We were further able to demonstrate additional flaws that took advantage of this Elastic Stack design weakness to create a cascade of API threats, many of which correspond indirectly to items described in the OWASP API Security Top 10, including:
- excessive data exposure
- lack of resources and rate limits
- security misconfiguration
- susceptibility to injection attacks due to lack of input filtering
The application in question is very well known and very popular across the globe. Maintaining the anonymity of this service provider is essential, so we have sanitized any technical details that could identify the company. As soon as we found the vulnerability, we reviewed our findings and suggested mitigation with the organization immediately. As part of the broader Salt Labs mission, we are sharing the findings here to increase awareness around API vulnerabilities, including explaining the attack pattern, detailing the steps to propagating the attack, and highlighting mitigation techniques.
Research motivation
Salt Labs conducts threat research to help customers and prospects identify vulnerabilities in their own APIs. The lab also investigates public platforms to uncover application logic flaws, following responsible disclosure practices and working with the appropriate organizations to resolve issues with the APIs. We publish this information – anonymized when need be - to help practitioners learn from others’ mistakes and avoid a resulting incident.
In this case, Salt Labs researchers investigated the online services platform because the architecture parallels that of many organizations that use the Elastic stack to provide data collection, storage, search, analytics, and visualization. The Elastic stack, or ELK Stack, consists of Elasticsearch, Kibana, Beats, and Logstash and enjoys widespread global usage. While the source code and binaries that Elastic provides are secure with current versions, organizations can still implement these applications and logging services improperly. Mistakes made in architecture design and implementation lead to severe vulnerabilities that put all systems, services, users, and data at risk. We undertook this deep security research to better understand the risks in the API-driven ecosystems that make extensive use of the Elastic Stack. In release for nearly 12 years, the open source ELK Stack has been downloaded millions of times – given its popularity, and the fact that every deployment of the Elastic Stack we’ve seen contains this implementation flaw, we felt it was crucial to surface this API threat research.
The potential business impacts of the API security gap we found
We will cover the technical details of the vulnerability in the next section, but first we wanted to highlight the business impact this risky Elastic Stack implementation can create. In this case, we were able to show how the impact worsens significantly when an attacker chains together multiple exploits of the API flaws.
Attack: Exfiltrate sensitive user and system data
Attackers can abuse the lack of authorization between front-end services and back-end services. Simply speaking, the technology stack of the application, interfacing systems, and APIs were implemented in such a way that the system did not verify who is submitting queries against it and whether they are authorized to receive the requested data. The back-end services simply processed queries submitted to it via front-end consumers or API clients, such as a user running a mobile application or web application in browser. Once attackers have obtained a working user account with basic permission levels, they can begin to make educated guesses about the schema of back-end data stores and query for data they aren’t authorized to receive.
In our research, we were able to access extensive sensitive data including account numbers and transaction confirmation numbers. Some of the sensitive data is also private and subject to regulation as defined by GDPR. Attackers could also use this data to exercise other functionality available via APIs such as to book new services or cancel existing ones. Attackers could use this information to perpetuate other types of fraud such as extorting funds, assuming someone’s identity, setting up fake accounts, and other nefarious activities. Such fraud would have, at best, resulted in revenue loss for the company – at worst, it could have triggered substantial regulatory penalties and fines.
Attack: Launch a Denial of Service event
Attackers often launch denial of service (DoS) attacks to render a service entirely unavailable or to divert attention away from malicious activity against other applications and APIs. Most commonly, to propagate DoS and distributed DoS (DDoS) attacks, attackers launch volumetric attacks, simply sending too much traffic for an organization’s network or systems to handle. Attackers also frequently employ application-layer DoS attacks by crafting requests with “heavy queries” unique to an application and its APIs. Application-layer DoS techniques invoke application functions that are “expensive” in terms of system resources to render APIs less responsive or completely unavailable.
In our research, we launched a minimal, short-term, application-layer DoS attack against the integrated back-end services. This proof-of-concept attack was done to highlight how lack of resource limiting left the B2C online platform vulnerable. An actual hacker could craft one simple request to bring this entire platform to its knees, rendering all data and services unavailable to customers.
The technical details of this API security gap
The following sections highlight the techniques we used to hack the APIs of this B2C online platform. In each case, we are detailing the form of the attack and the manipulated API requests. Due to the sensitive nature of original API traffic, these examples have been fabricated to protect the identity of the B2C online platform and its customers. Extraneous info such as informational HTTP headers and timestamps have also been removed for readability.
Both of the vulnerabilities originate from an underlying architecture design flaw. Front-end services and API clients were implicitly trusted by back-end services, and queries could be submitted within API requests to those back-end services. As a result, API clients from an untrusted network (the Internet) could manipulate API requests sent to outer APIs that in turn traverse to inner APIs and services powered by the Elastic Stack. The old security adage of “never trust input from the client” applies here, though the vulnerability is more nuanced than simply forgetting to filter user-supplied input.
Any user could access customer data without authorization
Security gaps in authorization enable what are known as OWASP API1 broken object level authorization (BOLA) attacks. In these attacks, an attacker can escalate privileges horizontally (assume access level of another user who has equivalent permissions) and vertically (assume access level of another user who has higher or administrative permissions). In this case, Salt Labs researchers were able to manipulate queries that were sent to and processed by back-end Elastic services. As a result, our researchers were able to extract user and system data they were not authorized to access.
It’s important to bear in mind that Elastic is used to support many IT business cases, not just web applications or SIEM deployments. Organizations typically deploy a centralized Elastic implementation or small number of Elastic deployments. These deployments are repurposed for many other use cases to minimize implementation costs and operational complexity. Unfortunately, this reality also means that one system (Elastic) houses a large variety of data for all types of users and systems. If an attacker can manipulate queries to Elastic and select other data stores, the end result can be devastating. To put it bluntly, this attack is a type of digital supply chain attack not unlike the SolarWinds breach the industry witnessed in late 2020.
Elasticsearch queries share a couple common parameters. One of those parameters is used to specify an index for a database to query. The other parameter is the actual query to run against the specified database. In this B2C platform implementation, some fields were being checked but not consistently. Salt Labs researchers could also add parameters to requests that were not being checked.
Salt researchers found that an API endpoint allowed for client-controlled query parameters in the message body of the API request, which in turn were passed to back-end Elastic services. The API endpoint looked like the following:
“/api/v1/data/proxy/_msearch”
Specifically, a POST request to this API endpoint could be made by adjusting query and index parameters to retrieve stored data in the back end. A couple other factors increased the security impact:
- Indexes were either defaults or easily guessable
- Reference material for Elastic’s multi search function is readily available to aid in reverse engineering
As a result, Salt Labs researchers were able to guess different data stores and extract sensitive customer and system data in the back end. User data included full name, mailing address, email address, IP address, payment details, and transaction numbers. System data included logs that contain other highly useful information to an attacker to pivot in their attack campaign and target other systems.
API Request: The API request that Salt Labs researchers used to extract customer data from the back-end can be seen above. An attacker simply needed to change the index and nested match parameters of the query payload to access stored records in the backend without authorization.
API Response: The resulting API response from the manipulated API request with the nested query destined for Elastic can be seen above. This response included thousands of customer records containing sensitive and private information.
Salt researchers made only a handful of these requests to prove exploitability of the back-end services and that unauthorized data access was possible. An attacker would use an intercepting proxy tool such as Burp Suite or OWASP Zed Attack Proxy to quickly enumerate through all data stores to collect as much data as possible for resale on the dark web or to perpetuate other fraud.
Any user could create a denial of service condition that would render the system unavailable
APIs are often designed to enable data retrieval, and this use case holds true more often with integration with back-end data collection and analytics services. If no limits are put in place on requests, or in this case, queries nested within requests, attackers can easily perpetuate application-level denial of service (DoS) attacks with even a singular request. In addition, some API functions are computationally “expensive,” such as in this case of requesting the entirety of an Elastic data store. Successfully executing such computationally expensive queries renders services unavailable. The availability impact is also not limited to just customers in this case but also other systems that also depend on the Elastic stack, specifically logstash for log storage. The Salt team details the attacks focused on resource and rate limiting in the blog post API4:2019 Lack of Resources & Rate Limiting.
Salt researchers found that the same API endpoint that allowed for unauthorized data access was also susceptible to DoS, as follows:
“/api/v1/data/proxy/_msearch”
Submitting a POST request to this API endpoint and abusing the same Elastic multi-search function, we were able to manipulate the value of a variable within the message body of the API request. Specifically, a parameter named “size” controls the number of results that would be queried for in the back-end Elastic data stores and returned in the API response. The typical integer value for this parameter was a range of 1 - 1000. Since this value was user controllable, Salt Labs researchers could set the value higher.
API Request: The API request that Salt Labs researchers used to create the application-level DoS condition in the back end can be seen above. The value “size” controls the number of records to be queried for and returned to the API client. Setting this value to a higher-than-normal value taxes the back-end Elastic services, since it must fetch more than a typical amount of data for the API client to satisfy the request. In this case, Salt Labs researchers set the value of “size” to 999999999.
API Response: The resulting response from the back-end Elastic services indicates that the query failed and returned an exception error since the value was too high. The error message was overly helpful – it provided the query limit max for this particular data set, which helped the Salt Labs team probe for other in-range integer values that could be used to tax the back end and create a DoS condition. The Salt Labs researchers did not execute a DoS attack to prove the vulnerability. Rather, they observed that the time it took for the back end to respond with a data set was excessive once queries were submitted within that acceptable, larger range.
We demonstrated the B2C online platform was susceptible to application-layer DoS by sending just one crafted API request with an expensive query. An attacker could also create a more persistent DoS situation by automating API requests with scripts to send crafted API requests at regular intervals to avoid detection, evade rate limits in place, and compound the availability impact.
Methods to mitigate this API security gap
Exploiting the flaws in this B2C online platform would require unique manipulation of the business logic involved in each API and integration with back-end services. Appropriate remediation and mitigation techniques also must be unique.
Mitigating an authorization weakness that permits any user to read any record
To be secure, an API should verify the authorization level of the user – that is, whether the user is requesting data he or she is authorized to access. Systems need to validate authorization continuously throughout a session and for each requested data access. This kind of continuous approach to authorization is fundamental to the security practice of least privilege, which in turn is a crucial premise in zero trust architecture.
Apply the following steps to mitigate the authorization weaknesses Salt Labs found in these API endpoints:
- Continuously authorize - The API and integrated, back-end systems should continuously validate whether the authenticated user is authorized to access the requested record.
- Return appropriate HTTP error codes - If an authenticated user tries to access someone else’s records, the server should respond with a “401 Unauthorized” HTTP status.
- Invalidate sessions of abusive requesters - The platform should monitor and invalidate sessions where users are making excessive requests, enumerating records, or making repeated attempts to access unauthorized data.
- Avoid default or simple naming - Use data store names and indexes that attackers cannot easily guess or enumerate through.
- Use rate limits - Enforce rate limits on API endpoints to prevent basic forms of enumeration – but know that attackers realize they must throttle their request rates to evade detection.
Mitigating an input filtering weakness that allows for application-level denial of service
Every API must defend against Denial of service (DoS) and Distributed denial of service (DDoS) attacks. To prevent these kinds of attacks, the security tooling must be able to analyze and understand the application logic of the API so it can detect and block anomalous requests.
Follow these steps to mitigate weak input filtering:
- Limit response sizes - Specify values in API request parameters so that users cannot control the size of data sets in API responses. Set these restrictions statically in the back end, based on the system capabilities across your complete architecture.
- Never rely on client-side filtering - Apply reasonable thresholds for any user-controlled value in API request parameters and filter out requests outside these bands. Minimums and maximums will vary based on your design and architecture. For example, if you’ve designed your APIs to return 100 records per page, do not allow any user to fetch more than that number in a single request.
- Use dynamic rate limiting - Restrict data set sizes and API responses based on your available back-end computing power or network throughput.
Mitigating security risks of implicit trust between front-end and back-end services
The specific queries submitted to the Elastic back-end services are difficult to test for in development time or build signatures for in runtime. API calls need to be continuously validated and compared against the typical baseline to identify anomalies such as unauthorized data access or expensive queries that result in availability impacts.
Follow security architecture principles to mitigate the types of attacks that Salt Labs was able to perpetuate against back-end services:
- Implement an API facade - Rather than directly exposing back-end services to front-end clients, implement another layer of API mediation. Organizations often use instances of inner and outer API gateways to implement an API facade pattern, separating outer and inner APIs. Unfortunately, gateways are another form of proxy, so tread carefully adding too many gateway instances to avoid proxy overload (see Mitigation technique #5).
- Validate query parameters - For those parameters that should be well-known, such as query size ranges or complexity, set restrictions within mediating mechanisms such as API gateways, and set reasonable limits within back-end services.
- Don’t allow indexes to be user controllable - An API caller should not be able to manipulate the back-end data store where queries are executed.
- Use parameterized queries - Similar to SQL security best practice and a mitigation technique for all types of injection attacks, allow users to select only those queries or query parameters that are predefined and safe.
- Avoid using too many proxies - Forward, reverse, and transparent proxies are common in many enterprise architectures. They are useful for mediating traffic including API calls. Unfortunately, organizations sometimes lose track of all instances within their environment, let alone third-party environments, clouds, and CDNs. Excessive proxies impede API performance but can also obscure other issues.
Lessons Learned
Salt Labs researchers found API vulnerabilities in this globally popular B2C online platform that provides API-based mobile applications and software as a service. The vulnerabilities arise from misimplementation of back-end technology – Elastic Stack – which makes use of APIs for critical data aggregation, search, and analytics capabilities. Ultimately, the design flaw allows for unauthorized access to back-end data. The misimplementation also enables denial of service attacks. In the experience of Salt Labs researchers, these missteps have been found in almost every organization using Elastic Stack.
Architecture matters, for any API security solutions you put in place and also how you architect your own application environments. Any organization should critically evaluate its API integrations between systems and applications since they directly impact its security posture. API communications traverse unprotected networks such as the Internet, where they hit network perimeters defined within cloud platforms, content delivery networks, virtual private networks and data center environments. Those outer API calls traverse to internal networks and systems, invoking inner APIs and microservices. You should never trust input originating from users on the Internet, and user-supplied data should never be passed to internal systems without additional scrutiny and sanitization.
Filtering for known malicious character sets using a negative security approach can catch some forms of injection attacks that follow well-defined patterns. However, this approach would not prevent this case of Elastic injection, since the design of the application and system allowed for submitting queries to back-end services. Custom queries should not be permissible by API clients. If a legitimate business case exists for such front-end capability, then the requests need to be analyzed for intent and whether the application logic is being used appropriately. In this case, one would need to look at the application logic for how data is queried and returned from the Elastic implementation.
As always, context matters, and forming a baseline of what constitutes “normal” is challenging without machine assistance. Organizations often attempt to review architecture as part of secure design reviews and threat modeling, but in most cases the necessary subject matter expertise needed to understand all innerworkings and interconnections of a complete system is lacking. Even in that perfect world, recognizing all the possible abuse cases becomes impossible, especially as technology stacks evolve and API code evolves. In addition, many abuses of application logic flaws surface only in runtime. Organizations should look to augment their IT systems with security tooling that can parse API context to detect anomalies, prevent data exposures, and stop API attacks.
The Salt Labs research into these API vulnerabilities highlight that significant API security gaps arise from third-party applications and services. Organizations sometimes misallocate security tooling budget or put too much confidence in traditional security approaches. Even worse, organizations may overlook the third-party dependencies integral to their own application and system architectures. When organizations consider dependencies, it is often with a goal of analyzing application binaries for known vulnerabilities, typically CVE IDs, or open-source license constraints. This element is only one side of the coin of application dependencies. Organizations must also consider the web APIs (commonly REST and GraphQL) and integrations between services to fully evaluate their security risk. These third-party dependencies paired with your own business and application logic form your unique digital supply chain and security posture. Traditional security approaches such as vendor attestations, periodic penetration testing, application scanning, and traditional runtime protections leave organizations vulnerable to API security gaps including Elastic injection.
Salt Labs publishes this kind of detailed API vulnerability research along with mitigation recommendations to help educate the broader industry on API security. Nearly every organization today is building or integrating APIs, many within technology stacks like the Elastic Stack. These organizations need a full lifecycle approach to API security and dedicated API security tooling so they can find and address the full set of API security issues.
If you’re interested in seeing the Salt Security API Protection Platform in action, contact us for a customized demo today!
