18 Jan, 2017

Introducing SpyDir

by Ryan Reid

In this post, I’ll discuss a Burp Suite extension I’ve recently developed and published to my GitHub. The extension provides a mechanism to enumerate endpoints within a web application via a local source code repository. Finally, it does this in an extensible manner.

Background

This part might be a bit boring, but I think it is important to know why I decided to spend my evenings working on this. Some time ago, I was working on a hybrid assessment that contained ~2k Classic ASP files. Gross, right? As I spidered and tested the application dynamically, I found that I had only discovered ~250 endpoints. As 250 is significantly less than 2,000, I was concerned that I may have missed sections of the application simply because they weren’t referenced within the immediate portion available from the landing page. I whipped up a quick script to treat all files within the directory as if they were valid resources being served by the front end. My script fed into Burp Suite’s Proxy, so I could process the results using my most beloved tool. In the end, I found most of the materials were abandoned resources still available through direct navigation. This assessment got me thinking: Maybe I can figure out a way to do this in a generic way and save some folks some time.

SpyDir

As the name, SpyDir, may imply, the goal of this extension is to peek into the directories and files. The result, with some luck, is a list of application endpoints ready for testing. SpyDir has two methods of operation. The first simply finds files and treats them as endpoints. Nice and easy. This will assist in projects with static files, or those using Front Controller or Page Controller design patterns. The second mode of operation enumerates endpoints by parsing source files for language/framework specific routing patterns. Now to be honest, the extension doesn’t really do much of anything on its own in this regard (this is the extensible part). SpyDir contains the ability to load Python plugins (more on this later). It is the responsibility of the plugins to parse the application endpoints and return them in an appropriate manner (again, more soon).

Getting Started

While, hopefully, most of the extension is fairly straight forward, I’ll go over each item here. Okay, so the goal is to get endpoints. We’re going to need a few things first. The UI within Burp Suite looks, for now, something like this:

User Interface

For the purpose of this demonstration we’ll use a local copy of MoneyX an intentionally vulnerable Spring application. So, first thing first, we’ll need a directory filled with source files. We’ll feed SpyDir a value of /Users/ryanreid/Code/Java/MoneyX for the Input Directory. Let’s ignore the String Delimiter for now. It’s a Spring application, so we’ll likely be interested in parsing the .java files. Let’s pass in that value for Extension Whitelist to limit the types of files we plan to parse. Finally, it’s a locally hosted application on port 8085. We’ll pass http://localhost:8085 to the URL field. Our configuration should look something like:

Initial configuration

Now, as we’re looking at a Spring application that routes URL paths via the framework, we are going to use a plugin to parse the source code. We’ll specify the plugin’s location and check the box to Attempt to parse files for URL patterns. When we specify the plugin’s location, we should receive a message within the status window indicating which plugins are available.

Plugins loaded

With our plugins loaded and our configuration settings determined, we’re ready to parse the directory. The extension will leverage the loaded plugins to parse the source code files to identify any and all endpoints within the application. The plugin returns a list of endpoints and the extension will preview an example URL. Now, depending on the application, this might take some time.

Directory parsed

We can list all endpoints by clicking the “Show all endpoints” button. The “Clear text” button will clear the status window and reset the parse results. This is useful when we need to parse a new directory and don’t want to clutter/confuse the file extensions statistics.

Endpoints

Finally, if our endpoints seem agreeable (always double check!), we can send the requests to the Burp Suite Spider for request on our behalf. If the URL we provided in the configuration is not in scope, a warning will appear in the status window. Check the URL to make sure it’s the intended target. If it is, click the “Send to Spider” button again, and this time Burp Suite will prompt you to include it in scope.

Also, you can send the URL to the configuration via context menus within Burp Suite. For now, the available contexts include:

  • Request message read/edit views
  • Proxy history view
  • Sitemap table view

Additionally, you can save/restore your configuration details by clicking the appropriately marked buttons. This will save/restore the configuration settings to/from the Burp Suite map and persist across extension reloads and application restarts.

String Delimiter

Note: This value is ignored during code parsing. In the above example, we parsed the directory for code level endpoints. However, if we were interested in some of the static resources, we could use the other mode of operation with the String Delimiter to process files as endpoints. For our example, we’ll set the String Delimiter to static. This will instruct the extension to only consider files within the static directory and below as files we are interested in requesting from the server. The extension will use this value to split the string in order to build the URL(s).

Code Repository

Here’s a quick example: We can see from the directory structure above, that nested within the static/dist/js directory there are four .js files. As the extension iterates through the directories and files contained within the initial input directory, it will identify files below the String Delimiter. Our output will look like this:

Static endpoints

The extension highlights files that match the file extension whitelist outside of the String Delimiter directory path. In this example, it appears that a JavaScript file resides within the build/reports/tests/js directory of the application.

Plugins

SpyDir possesses the ability to load, from a user specified location, plugins to assist during code level endpoint enumeration. This location is saved and restored within the configuration file for easy restoration. Upon restoration, the extension will attempt to load plugins within the saved location.

Writing plugins will hopefully be fairly straightforward as well. At this time, there are a few plugins available within the GitHub repository that may be used as templates to generate further plugins.

Plugins require a few things to integrate with the extension. Specifically:

  • A get_name() function that returns a string with the title of the plugin.
  • A run() function that accepts a list containing the lines from a source file. Return a list(), [], of endpoints.
  • A get_ext() function that returns a string containing a comma delimited string of file extension type(s).

Additional functions may be used to split up the plugin’s workload so long as the above requirements are met.

One of the neat things about leveraging plugins is the capability to also enumerate variables related to the endpoints of interest. That is, the plugin could attempt to produce valid values for dynamic endpoints/routes. For instance, in the Spring 2.5+ MVC plugin, the plugin attempts to substitute valid values for the path variables identified within the code. Here’s the relevant part:

def set_path_vars():
    """
    Substitute values into the defined variables.
    Update these variables and values on a per app basis.
    """
    return {"{id}": 2,
            "{eventId}": 3,
            "{receiver}": "Steve",
            "{user}": "Bob",
            "{userId}": 4,
            "{friend}": "Paul",
            "{owner}": "Tom",
            "{name}": "John",
            "{amount}": "3.50",
            "{hidden}": "secret",
            "{oldPassword}": "12345",
            "{newPassword}": "hunter2"}


def handle_path_vars(route):
    """
    Replaces the placeholder variables with values from set_path_vars()
    Returns a string containing the updated route
    """
    new_route = route
    for k, v in set_path_vars().items():
        new_route = new_route.replace(k, str(v))
    return new_route

We can see in the previous screenshot that the endpoints we’ve enumerated contain the substituted values from the plugin. Without the plugin supplying the values, we’d see something like:

Default path variables

Next Steps

First and foremost, I’m hoping to receive some feedback. The extension is in Alpha right now, so you’re probably going to come across some bugs. Let me know, or feel free to contribute a bug fix! Also, the more plugins that are available, the more useful the extension will be. Feel free to submit your own plugins! Here’s a short list of things I’m working to implement:

  • Allow plugins to specify HTTP method
  • Export data and configurations (HTML XML JSON)

That’s all for now. Happy testing!