Original title: Netanga · One Hundred Forums | Principles of Web Vulnerability Scanning Tool

About the author: Guo Zhenxin, the regional head of OWASP Jilin, the senior R&D engineer, VP, and the head of the safety department of Path Technology. It has 10 years of experience in software security and software R&D, and has experience in building a software company's security team from 0 to 1.

People who are new to security always have some questions. Why do you need to do manual tests with Web vulnerability scanning tools? The rules in the scanning tool are well written, and there are all types of problems. Is it possible to ensure the absolute security of the Web application after completing Scan? This article explains these doubts from the perspective of the principle of scanning tools.

Let's first discuss the principles and rules of Web vulnerability scanning tools.

1、 Passively Scan

If someone has used OWASP ZAP or Burp Suite scanning, they should notice that one of their scanning methods is passive scanning.

When is passive scanning? Passive scanning does not actively send any request to the target server. It only analyzes the content of existing requests and responses passing through Burp Suite/ZAP, and infers vulnerabilities from them. Many types of Vulnerabilities can be detected using only passive techniques.

Here are some problems that can be found by passive scanning:

1. Response does not contain various safety related response headers.

2. The response contains the version number of the server, for example: Server: IIS 10.0

3. The version of the foreground component (such as jQuery) is too low and contains known problems.

4. Analyze whether the front-end code contains potential XSS, etc. (note that this analysis may have a certain false alarm rate, because it only analyzes the code and does not directly reproduce the problem).

5. Password/PII and other privacy information discovery.

6. SSL/TLS certificate problem.

7. Other vulnerabilities that can be found passively can be found in the rule configuration of the scanning tool.

2、 Compare data and behavior in Response

Except for the first one, which is passive scanning technology, the other methods are active vulnerability detection. The scanning tool will replace all the parameters in the request with the corresponding attack Payload and send it to the server again and again, and then check and compare the data and behavior in the response returned by the server to determine whether the server has Vulnerabilities. Draw an example picture as follows:

Comparing the data in the Response is of course easy to understand. For example, there is SQL Injection when logging in. When the user name enters admin "or 1=1, it will log in directly. The scanning tool will compare the data in the Response with different data (login success/failure) to determine that there is a SQL Injection problem.

How to understand the contrast behavior? Although some SQL Injections will be executed, the return values in the Response are the same. In this case, there is no way to directly compare the data. In this case, we can compare the behavior. Take SQL Injeciton for example, when Payload is UserID=1; Waitfor delay 0:0:3 - When Payload is changed to UserID=1, the response time returned by the server is more than 3 seconds; Waitfor delay 0:0:30 - When the server returns a response for more than 30 seconds, the scan tool will detect the SQL injection problem after several attempts.

In addition, the scanning tool will also analyze the abnormal information in the Response, so as to judge the internal logic and report it directly, which is convenient for the attacker to view and carry out the next attack. For example, some exception information will throw out where the SQL statement is misspelled or throw out the entire wrong SQL statement. If the exception information is directly printed into the Response, it will speed up and help the attacker to confirm the vulnerability. Therefore, it is important to expose abnormal information as little as possible.

PS: Explain the meaning of Payload. In short, the data sent to the server one time at a time. For example, Payload of SQL Injection:

1. admin" or 1=1

2. admin" or 1=1--

3. admin" or 1=1#

4. admin" or 1=1/*

Different attacks will have different payloads. I recommend you to see PayloadAllTheThings（ //github.com/swisskyrepo/PayloadsAllTheThings ）, which lists various types of Payloads.

3、 Techniques for matching known vulnerabilities

How do you understand this scanning rule? Some sites use ready-made products for secondary development, such as WordPress. When the version of WordPress used has security problems, no matter how secure the code of secondary development is, it cannot avoid the security problems caused by the vulnerabilities contained in WrodPress itself, or the Web Server used by itself (such as IIS) It contains security loopholes, etc. This rule is to find the above problems.

This rule of the scanning tool does not replace all parameters to match whether there is a problem. Instead, it sends the corresponding request to detect. All of them are written rules to match. The following screenshot is a list of known security issues in AWVS.

For example, like Drupal SQL injection in the above figure, the request to exploit this vulnerability is as follows:

POST /drupal-7.31/? q=node&destination=node HTTP/1.1

Host: 127.0.0.1

User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:28.0) Gecko/20100101 Firefox/28.0

Accept: text/html,application/xhtml+xml,application/xml; q=0.9,*/*; q=0.8

Accept-Language: en-US,en; q=0.5

Accept-Encoding: gzip, deflate

Referer: //127.0.0.1/drupal-7.31/

Cookie: Drupal.toolbar.collapsed=0; Drupal.tableDrag.showWeight=0; has_js=1

Connection: keep-alive

Content-Type: application/x-www-form-urlencoded

Content-Length: 231

name[0%20;update+users+set+name%3downed+,+pass+%3d+$S$DkIkdKLIvRK0iVHm99X7B/M8QC17E1Tp/kMOd1Ie8V/PgWjtAZld+where+uid+%3d+1;;#%20%20]=test3&name[0]=test&pass=shit2&test2=test&form_build_id=&form_id=user_login_block&op=Log+in

So the scanning tool will directly send the above request to the server to check whether there is any indication that the response can be used. If there is, it means that the Drupal SQL injection problem has been detected.

4、 Technology for detecting Out of Band

The general Web Vulnerabilities Scanner will have the above three rules, but only the better Scanner will have the following two rules.

The technology of detecting Out of Band is to detect certain Vulnerabilities that cannot be directly detected from the response comparison. For example, Blind XSS, Server side request Forgery (SSRF), XML External Entity injection (XXE), etc.

Tools for detecting Out of Band technology include AWVS AcuMonitor （ //www.acunetix.com/vulnerability-scanner/acumonitor-technology/ ）, Burp Suite's Burp Collaborator Client （ //portswigger.net/burp/documentation/desktop/tools/collaborator-client ）Etc.

This technique is explained with a familiar example. When there is a storage XSS in the site, it cannot be directly detected by using the rules explained above, because the storage XSS has an XSS vulnerability on another page, and the scanning tool cannot detect whether XSS is contained through the request for stored data. But it is OK to use the Out of Band detection technology. The principle is as follows:

In this example, XSS also needs page triggering, and there are some other problems, such as SSRF (controlling the server to send a request to a certain place), which is more direct. It does not need any triggering, but directly sends Payload to the server. If there is an SSRF problem, the server will directly send the address in Payload to AcuMonitor, so as to find the problem.

5、 Interactive Application Security Scan (IAST)

Let's explain a few terms first:

1. Dynamic Application Security Testing (DAST)： Tools like AWVS, Burp Suite and ZAP belong to DAST tools. Scan dynamic websites to find security problems.

2. Static Application Security Testing (SAST)： Tools like SonarQube, CheckMarx, and Fortify SCA are all SAST tools. Static scanning of source code reveals security problems.

3. Interactive Application Security Testing (IAST)： Interactive Application Security Testing collects and monitors the function execution and data transmission of the Web application runtime through agents, VPNs or deploying agents on the server side, and interacts with the scanner side in real time to efficiently and accurately identify security flaws and vulnerabilities, and accurately determine the code file, line number Functions and parameters.

IAST is an extension of DAST, which can also be understood as the integration of DAST and SAST. It is explained by the working principle of AcuSensor in AWVS. By installing HttpModule in the server site, we can obtain some additional information from the server, so as to find more problems and improve the scanning accuracy.

When we need to use AcuSensor, we need to install AcuSensor on the corresponding test site. After installation, an additional AcuWeaver.dll will be added to the site and an HttpModule will be added to web.config. This HttpModule should be used to detect the server additionally, because more useful information will be obtained in the server than only scanning outside, For example, what is the specific SQL statement to be executed, so as to analyze more problems.

The following figure shows the schematic diagram of AcuSensor:

After the above introduction, do you have a certain understanding of the working principle of the Web vulnerability scanning tool?

Then let's go back to the original question:

▸ Why do you still need to do manual testing with Web application scanning tools?

Because scanning tools analyze based on the return value of Response, it is difficult for scanning tools to analyze what kind of behavior is legal if logical problems (such as permission related problems) are involved, so similar problem scanning tools are difficult to detect.

▸ Is it possible to ensure the absolute security of Web applications after scanning?

The answer, of course, is no. In addition to the logic problems mentioned above, there are also two reasons:

1) First, the request to be scanned by the scanning tool is based on the results of its automatic crawl, and the results of the automatic crawl are often unsatisfactory (here we usually collect requests by manually clicking the page), so the scanning tool is unlikely to scan every request of the entire application.

2) In addition, if you are a developer, you should be able to understand that some APIs may have a lot of branching logic (if, else, switch, case, etc.), which is executed based on the API passing different parameters. The problem is that the scanning tool can hardly cover every layer of branching logic, which is also the limitation of the scanning tool.

Therefore, the security of Web applications can be guaranteed at all stages and dimensions of application development through various means and methods, rather than being handed over to the Web vulnerability scanning tool once and for all. The detailed methods will not be repeated here.

Finally, I am not the author of any web vulnerability scanning tool. I analyze the principles and limitations of the scanning tool from the perspective of a user. If you have any questions/supplements or objections, please discuss them together.

- End - Go back to Sohu to see more

Editor in charge: