Login Start Free Trial Login Start Free Trial
 

Custom Scan Discovery

One of the ways HawkScan can discover the scan web application is with the assistance of another developer provided command or devtool that generates web traffic. HawkScan can intercept this traffic and scan it for vulnerabilities. This feature enables crafty security teams to combine their favorite software testing devtools for maximum observability in the CICD pipeline.

To use Custom Scan Discovery in HawkScan, add the following settings to your stackhawk.yml file:

stackhawk.yml

hawk:
  spider:
    base: false # disable the default base spider; optional
    custom:
      command: "" # a command to execute a developer tool, with HTTP_PROXY support eg. "curl -x $HTTP_PROXY http://localhost:3000/admin"
      arguments: [] # a list of strings that can be provided to the command or used instead of.
      environment: {} # map of environment variables present at command execution.
      credentials: {} # additional environment variables present at command execution and redacted from the logs.
      excludeParentEnvironment: false # An option to exclude the parent environment variables from execution in this command.
      logOutputToForeground: false # If enabled this will print the output of the command to the terminal foreground.
      workingDir: "" # The absolute path working directory these commands are run from. Defaults to the base directory HawkScan is run from.

These settings can be configured under the hawk.spider.custom section of the stackhawk.yml file.

Note: the provided command should be a devtool that accepts proxying the HTTP traffic it creates. HawkScan will ignore all traffic the tool generates outside of the scanned app.host origin.

Executed Command Environment

The custom discovery command executes in its own process. The command splits on whitespace and runs with the arguments and a limited set of environment variables present. The command can also be specified entirely from the arguments as necessary.

The environment variables present in the parent process (HawkScan) are used, alongside the configured custom.environment and custom.credentials variables. The custom.credentials are also environment variables, but will be treated as secrets and redacted from log output. The parent process environment variables can be ignored by enabling excludeParentEnvironment. The parent environment provides common variables such as $HOME and $PATH, which most devtools may expect to be present at runtime.

By default, the custom discovery command executes as its own process with these environment variables present at runtime, as well as these additional variables:

  • $HTTP_PROXY & $STACKHAWK_HTTP_PROXY the HTTP proxy address of the scanner, typically defaults to http://127.0.0.1:20000. Most devtools also use this variable to configure HTTPS proxying.
  • $PROXY_CA_CERT & $STACKHAWK_PROXY_CA_CERT the path to the proxy CA file for the certificate; it has no default. Some devtools use this variable to configure HTTPS proxying.

The environment or credential variables you provide in the config will layer on top of these and the parent process variables, overwriting them.

Furthermore, if the value of a custom.environment or custom.credentials variable exactly matches either STACKHAWK_HTTP_PROXY or STACKHAWK_PROXY_CA_CERT, the value will be replaced appropriately. Before the command is executed, environment variables present in the command with Bash syntax, $ENVIRONMENT_VARIABLE or with Windows CLI syntax, %ENVIRONMENT_VARIABLE% will be replaced with the value of that variable.

If the command exits with a nonzero status, this will intentionally fail the scan.

Custom proxy port

By default HawkScan starts its proxy on port 20000. The port number can be configured in the CLI with the --proxy-port flag or with the PROXY_SERVER_PORT environment variable present at runtime. The injected HTTP_PROXY environment variable is directly influenced by the proxy port value.

Running from a shell environment

Because the command executes as a process with provided arguments, it may run without the full context of a shell.

If required, the command can instead be started from a shell and the full arguments provided separately:

hawk:
  spider:
    base: false
    custom:
      command: bash
      arguments:
        - -c
        - "echo hello from the shell && sleep 3 && curl -x $HTTP_PROXY http://example.com"

Custom Discovery over HTTPS

Because HawkScan is effectively man-in-the-middling the web application, traffic sent over the HTTPS protocol is effectively encrypted, and cannot be intercepted by HawkScan. In this scenario, HTTPS is doing the right thing, but it will get in the way of using Custom Scan Discovery.

This feature is best used when scanning applications over http:// protocols unencrypted. This scenario is ideal when testing locally or in the CI/CD pipeline, where secure encrypted communication to the tested web application is less necessary.

Some devtools allow for opting out of secure HTTPS connections with a --insecure flag or another setting. Other devtools support providing a path to the CA certificate of the proxy, to enable a secure HTTPS connection from HawkScan to the devtool.

Some tools may instead use the PROXY_CA_CERT environment variable to use HawkScan’s self-signed certificate during the scan to securely proxy the traffic. Other tools may require an HTTPS_PROXY environment variable instead that should just match the STACKHAWK_HTTP_PROXY value. Not every devtool requires these tools to proxy HTTPS traffic.

Additionally, not all devtools support HTTPS traffic proxying, so be sure to understand your testing requirements and how your devtools work. Also be aware if your devtool has settings for NO_PROXY hosts, or configurations to prevent traffic from being sent. These settings might be defaulted to disallow proxy traffic from localhost hosts, and requires opting-out.

Custom Discovery in Docker

Custom Scan Discovery requires both the devtool and HawkScan be run together in a containerized CI/CD build pipeline. In other words, both HawkScan and the devtool need to be running in the same container. Currently, StackHawk only provides Docker images with HawkScan. To effectively use Custom Scan Discovery in your CI/CD pipeline, you will need an image with both HawkScan and the devtool installed with any other required software testing dependencies.

See the StackHawk Custom Image repository for examples of making a custom Docker image with HawkScan and other devtools.

Devtool Output

The output of the devtool you use for custom scan discovery is written to the HawkScan log file. If the command exits with a nonzero status (indicative of a process failing), the scan will fail and the output of the command is displayed.

The command output can also be seen directly in the foreground of HawkScan by enabling logOutputToForeground.

Custom Scan Discovery Examples

The following are examples of using custom scan discovery with developer-recognized, quality application testing tools.

CURL

curl is a respected and prolific HTTP testing tool, and you can use it to directly add a path into HawkScan.

Curl proxy behavior differs depending on the build and operating system. Some versions of curl will respect the presence of the HTTP_PROXY environment variable, while others require passing it in with the -x $HTTP_PROXY flag (not to be confused with the -X POST flag to change the request method).

Assuming your app is running on a host defined from the APP_HOST environment variable, such as http://localhost:9000, you can use the following sample configuration:

stackhawk.yml

app:
  host: ${APP_HOST:http://localhost:9000}

hawk:
  spider:
    base: false
    custom:
      command:  "curl -x $HTTP_PROXY ${APP_HOST:http://localhost:9000}/hidden-page"

Also notice in this example that the stackhawk.yml will still interpolate environment variables in the config.

Note: while curl is an effective testing tool for individual routes, and it demonstrates the basic capabilities of custom scan discovery, it’s less useful compared to other software development tools that may generate more traffic into HawkScan than just one path.

HTTPS scanning

Modern builds of curl can be used for HTTPS scan discovery with no change to the command. If the scanned host uses a self-signed certificate, curl will require the --insecure flag.

Curl dynamically links to OpenSSL and other security libraries needed for secure connections, and may operate differently by operating system and build. These might also require the --proxy-insecure flag, the --insecure flag or a newer build of curl.

Postman

Newman is the CLI runner for testing Postman Collections. Postman is a great tool that assists developers with building robust APIs and testing their behavior with documented routes in the Postman collection files.

HawkScan can additionally test API responses from Postman Collections using Newman. Newman will use the HTTP_PROXY environment variable to configure the proxy.

Assuming there is a postman_collection.json file present in the directory HawkScan runs from, you can use the following sample configuration:

stackhawk.yml

hawk:
  spider:
    base: false
    custom:
      command:  "newman run postman_collection.json"

HawkScan can still use a Newman Collection without authentication to discover a web application. ` unauthorized (401) and forbidden (403)` responses will still populate the HawkScan site tree, and HawkScan will revisit these paths with its own configured authentication.

HTTPS scanning

Newman can be used for HTTPS scan discovery with no change to the command. If the scanned host uses a self-signed certificate, Newman will require the --insecure flag.

stackhawk.yml

hawk:
  spider:
    base: false
    custom:
      command:  "newman run postman_collection.json --insecure"

Custom Docker Image

See the example custom Dockerfile for HawkScan + Postman

Cypress

Cypress is a frontend testing tool that automates testing web applications through a regular or headless browser. Cypress uses the HTTP_PROXY environment variables to configure proxy behavior. These variables are automatically added when using Custom Scan Discovery.

By default, Cypress ignores proxying localhost traffic, so it will require a NO_PROXY: "<-loopback>" environment variable defined if scanning localhost.

Assuming cypress has been added to the package.json file and installed, run your Cypress test-suite from HawkScan by modifying this sample configuration for your application:

stackhawk.yml

hawk:
  spider:
    base: false
    custom:
      command:  "npx cypress run -s path/to/cypress-specs"
      logOutputToForeground: true
      environment:
        NO_PROXY: "<-loopback>"

HTTPS scanning

Without any changes Cypress should continue to forward HTTPS proxy of the scanned host into HawkScan.

Depending on the configuration of the HTTPS certificate for your scanned application, Cypress may also require disabled web security.

Custom Docker Image

See the HawkScan + Cypress custom image repository for an example of a modern Cypress test suite automated with HawkScan and a Dockerfile to make a custom Docker image.

Playwright

Playwright is an end-to-end testing tool for testing web applications through a regular or headless browser. Playwright requires additional scripted configuration to enable proxying traffic. To dynamically enable HTTP proxy behavior with Playwright, add the following section to your playwright.config.js file:

const httpProxy = () => process.env.HTTP_PROXY || '';
// This will selectively enable it only if the HTTP_PROXY environment variable is present when HawkScan runs.
const proxy = httpProxy() ? { server: httpProxy() } : undefined;
const config = {
  use: {
    proxy, // This is required for Playwright scan discovery!
    headless: false,
    viewport: { width: 1280, height: 720 },
    ignoreHTTPSErrors: true,
  }
};
export default config;

Assuming @playwright/test has been added to the package.json file, run your Playwright test-suite from HawkScan by modifying this sample configuration for your application:

stackhawk.yml

hawk:
  spider:
    base: false
    custom:
      command:  "npx playwright test --config=playwright.config.js"
      logOutputToForeground: true

HTTPS scanning

Without any changes, Playwright will forward a HTTPS proxy of the scanned host into HawkScan.

Depending on the configuration of the HTTPS certificate for your scanned application, Playwright configuration may also require ignoreHTTPSErrors: true.

Custom Docker Image

See the HawkScan + Playwright custom image repository for an example of a Playwright test suite automated with HawkScan and a Dockerfile to make a custom Docker image.

Selenium

Selenium is a Java framework for browser automation. Selenium is commonly used for automating web applications or for end-to-end web application testing. Selenium is distributed as a Java JAR file, and requires separate WebDriver JAR files installed to drive each browser.

To dynamically enable HTTP proxy behavior with Selenium, specify each WebDriver httpProxy setting from the HTTP_PROXY environment variable:

private val HTTP_PROXY: String? = System.getenv("HTTP_PROXY")

fun getBrowser(browserName: String?): WebDriver {
    val driver: WebDriver = if (browserName != null && browserName == "chrome") {
        val options = ChromeOptions()
        options.setHeadless(HEADLESS)

        // Set the proxy information if HTTP_PROXY is not null
        HTTP_PROXY?.let { proxyUrl ->
            val proxy = Proxy()
            proxy.httpProxy = proxyUrl
            options.setProxy(proxy)
        }
        ChromeDriverManager.getInstance().setup()
        ChromeDriver(options)
    } else {
        val options = FirefoxOptions()
        options.setHeadless(HEADLESS)

        // Set the proxy information if HTTP_PROXY is not null
        HTTP_PROXY?.let { proxyUrl ->
            val profile = FirefoxProfile()
            // The proxy will always be listening on localhost:<port> so firefox needs this preference set
            profile.setPreference("network.proxy.allow_hijacking_localhost", true)
            options.profile = profile
            val proxy = Proxy()
            // Firefox wants the http/s proxies in host:port format
            val firefoxProxy = proxyUrl.replace("http://", "")
            proxy.httpProxy = firefoxProxy
            proxy.sslProxy = firefoxProxy
            options.setProxy(proxy)
        }
        FirefoxDriverManager.getInstance().setup()
        FirefoxDriver(options)
    }
    return driver
}

Assuming the Selenium WebDrivers have been configured appropriately, the Selenium test suite can be run from a Gradle task. By specifying the exact test suite, you can use this sample configuration:

  hawk:
  spider:
    maxDurationMinutes: 5
    base: false
    custom:
      command: ./gradlew :test --tests "com.stackhawk.selenium.TestBrowser"
      logOutputToForeground: true

To see the full example of the above clone our example project HawkScan + Selenium which runs selnium tests against our example vulnerable app JavaSpringVulny.

HTTPS scanning

Each Selenium WebDriver may require configuration to ignore certificate errors if your application uses self-signed certificates.

Custom Docker Image

See the HawkScan + Selenium custom image repository for an example of a selenium test suite automated with HawkScan and a dockerfile to make a custom docker image.