Custom Scan Discovery

One of the ways HawkScan can discover the scan web application is with the assistance of another developer provided command or devtool that generates web traffic. HawkScan can intercept this traffic and scan it for vulnerabilities. This feature enables crafty security teams to combine their favorite software testing devtools for maximum observability in the CICD pipeline.

To use Custom Scan Discovery in HawkScan, add the following settings to your stackhawk.yml file:

stackhawk.yml

hawk:
  spider:
    base: false # disable the default base spider; optional
    custom:
      command: "" # a command to execute a developer tool, with HTTP_PROXY support eg. "curl -x $HTTP_PROXY http://localhost:3000/admin"
      arguments: [] # a list of strings that can be provided to the command or used instead of.
      environment: {} # map of environment variables present at command execution.
      credentials: {} # additional environment variables present at command execution and redacted from the logs.
      excludeParentEnvironment: false # An option to exclude the parent environment variables from execution in this command.
      logOutputToForeground: false # If enabled this will print the output of the command to the terminal foreground.
      workingDir: "" # The absolute path working directory these commands are run from. Defaults to the base directory HawkScan is run from.

These settings can be configured under the hawk.spider.custom section of the stackhawk.yml file.

Note: the provided command should be a devtool that accepts proxying the HTTP traffic it creates. HawkScan will ignore all traffic the tool generates outside of the scanned app.host origin.

Executed Command Environment

The custom discovery command executes in its own process. The command splits on whitespace and runs with the arguments and a limited set of environment variables present. The command can also be specified entirely from the arguments as necessary.

The environment variables present in the parent process (HawkScan) are used, alongside the configured custom.environment and custom.credentials variables. The custom.credentials are also environment variables, but will be treated as secrets and redacted from log output. The parent process environment variables can be ignored by enabling excludeParentEnvironment. The parent environment provides common variables such as $HOME and $PATH, which may make calling a devtool more difficult, but the process runtime cleaner and easier to debug.

By default, the custom discovery command executes as its own process with these environment variables present at runtime, as well as these additional variables:

  • $HTTP_PROXY & $STACKHAWK_HTTP_PROXY the HTTP proxy address of the scanner, typically defaults to http://127.0.0.1:20000. Most devtools also use this variable to configure HTTPS proxying.
  • $PROXY_CA_CERT & $STACKHAWK_PROXY_CA_CERT the path to the proxy CA file for the certificate; it has no default. Some devtools use this variable to configure HTTPS proxying.

The environment or credential variables you provide in the config will layer on top of these and the parent process variables, overwriting them.

Furthermore, if the value of a custom.environment or custom.credentials variable exactly matches either STACKHAWK_HTTP_PROXY or STACKHAWK_PROXY_CA_CERT, the value will be replaced appropriately. Before the command is executed, environment variables present in the command with Bash syntax, $ENVIRONMENT_VARIABLE or with Windows CLI syntax, %ENVIRONMENT_VARIABLE% will be replaced with the value of that variable.

If the command exits with a nonzero status, this will intentionally fail the scan.

Custom proxy port

By default HawkScan starts its proxy on port 20000. The port number can be configured in the CLI with the --proxy-port flag or with the PROXY_SERVER_PORT environment variable present at runtime. The injected HTTP_PROXY environment variable is directly influenced by the proxy port value.

Running from a shell environment

Because the command executes a process with provided arguments, it won’t run from the context of a shell, and therefore won’t have access to common shell conventions such as PATH, USER or HOME a developer may be expecting.

If these are needed, the command can instead be started from a shell and the full arguments provided separately:

hawk:
  spider:
    base: false
    custom:
      command: bash
      arguments:
        - -c
        - "echo hello from the shell && sleep 3 && curl -x $HTTP_PROXY http://example.com"

Custom Discovery over HTTPS

Because HawkScan is effectively man-in-the-middling the web application, traffic sent over the HTTPS protocol is effectively encrypted, and cannot be intercepted by HawkScan. In this scenario, HTTPS is doing the right thing, but it will get in the way of using Custom Scan Discovery.

This feature is best used when scanning applications over http:// protocols unencrypted. This scenario is ideal when testing locally or in the CI/CD pipeline, where secure encrypted communication to the tested web application is less necessary.

Some devtools allow for opting out of secure HTTPS connections with a --insecure flag or another setting. Other devtools support providing a path to the CA certificate of the proxy, to enable a secure HTTPS connection from HawkScan to the devtool.

Some tools may instead use the PROXY_CA_CERT environment variable to use HawkScan’s self-signed certificate during the scan to securely proxy the traffic. Other tools may require an HTTPS_PROXY environment variable instead that should just match the STACKHAWK_HTTP_PROXY value. Not every devtool requires these tools to proxy HTTPS traffic.

Additionally, not all devtools support HTTPS traffic proxying, so be sure to understand your testing requirements and how your devtools work. Also be aware if your devtool has settings for NO_PROXY hosts, or configurations to prevent traffic from being sent. These settings might be defaulted to disallow proxy traffic from localhost hosts, and requires opting-out.

Devtool Output

The output of the devtool you use for custom scan discovery is written to the HawkScan log file. If the command exits with a nonzero status (indicative of a process failing), the scan will fail and the output of the command is displayed.

The command output can also be seen directly in the foreground of HawkScan by enabling logOutputToForeground.

Custom Scan Discovery Examples

The following are examples of using custom scan discovery with developer-recognized, quality application testing tools.

CURL

curl is a respected and prolific HTTP testing tool, and you can use it to directly add a path into HawkScan.

Curl proxy behavior differs depending on the build and operating system. Some versions of curl will respect the presence of the HTTP_PROXY environment variable, while others require passing it in with the -x $HTTP_PROXY flag (not to be confused with the -X POST flag to change the request method).

Assuming your app is running on a host defined from the APP_HOST environment variable, such as http://localhost:9000, you can use the following sample configuration:

stackhawk.yml

app:
  host: ${APP_HOST:http://localhost:9000}

hawk:
  spider:
    base: false
    custom:
      command:  "curl -x $HTTP_PROXY ${APP_HOST:http://localhost:9000}/hidden-page"

Also notice in this example that the stackhawk.yml will still interpolate environment variables in the config.

Note: while curl is an effective testing tool for individual routes, and it demonstrates the basic capabilities of custom scan discovery, it’s less useful compared to other software development tools that may generate more traffic into HawkScan than just one path.

HTTPS scanning

Modern builds of curl can be used for HTTPS scan discovery with no change to the command. If the scanned host uses a self-signed certificate, curl will require the --insecure flag.

Curl dynamically links to OpenSSL and other security libraries needed for secure connections, and may operate differently by operating system and build. These might also require the --proxy-insecure flag, the --insecure flag or a newer build of curl.

Postman

Newman is the CLI runner for testing Postman Collections. Postman is a great tool that assists developers with building robust APIs and testing their behavior with documented routes in the Postman collection files.

HawkScan can additionally test API responses from Postman Collections using Newman. Newman will use the HTTP_PROXY environment variable to configure the proxy.

Assuming there is a postman_collection.json file present in the directory HawkScan runs from, you can use the following sample configuration:

stackhawk.yml

hawk:
  spider:
    base: false
    custom:
      command:  "newman run postman_collection.json"

HawkScan can still use a Newman Collection without authentication to discover a web application. ` unauthorized (401) and forbidden (403)` responses will still populate the HawkScan site tree, and HawkScan will revisit these paths with its own configured authentication.

HTTPS scanning

Newman can be used for HTTPS scan discovery with no change to the command. If the scanned host uses a self-signed certificate, Newman will require the --insecure flag.

stackhawk.yml

hawk:
  spider:
    base: false
    custom:
      command:  "newman run postman_collection.json --insecure"

Cypress

Cypress is a frontend testing tool that automates testing web applications through a regular or headless browser. Cypress uses the HTTP_PROXY environment variables to configure proxy behavior. These variables are automatically added when using Custom Scan Discovery.

By default, Cypress ignores proxying localhost traffic, so it will require a NO_PROXY: "<-loopback>" environment variable defined if scanning localhost.

Assuming cypress has been added to the package.json file, run your cypress test-suite from HawkScan by modifying this sample configuration for your application:

stackhawk.yml

hawk:
  spider:
    base: false
    custom:
      command:  "./node_modules/.bin/cypress run -s path/to/cypress-specs"
      environment:
        NO_PROXY: "<-loopback>"

HTTPS scanning

Without any changes Cypress should continue to forward HTTPS proxy of the scanned host into HawkScan.

Depending on the configuration of the HTTPS certificate for your scanned application, Cypress may also require disabled web security.