Running Crawls

Running crawls can be modified from the crawl workflow Latest Crawl tab. You may want to modify a running crawl if you find that the workflow is crawling pages that you didn't intend to archive, or if you want a boost of speed.

Crawl Workflow Status

When a workflow run is initiated, the workflow status changes to Waiting or Starting, depending on whether the conditions for starting a crawl are in place (such as resource capacity.) The workflow status will change to Running once the crawler loads the first crawl URL.

Watch Crawl

You can watch the current state of the browser windows as the crawler visits pages in the Watch tab of Latest Crawl. A list of queued URLs are displayed below in the Upcoming Pages section.

Live Exclusion Editing

While exclusions can be set before running a crawl workflow, sometimes while crawling the crawler may find new parts of the site that weren't previously known about and shouldn't be crawled, or get stuck browsing parts of a website that automatically generate URLs known as "crawler traps".

If the crawl queue is filled with URLs that should not be crawled, use the Edit Exclusions button in the Watch tab to instruct the crawler what pages should be excluded from the queue.

Exclusions added while crawling are applied to the same exclusion table saved in the workflow's settings and will be used the next time the crawl workflow is run unless they are manually removed.

Changing the Number of Browser Windows

Like exclusions, the number of browser windows can also be adjusted while crawling. On the Watch tab, press the +/- button next to the Running in N browser windows text and set the desired value.

This change takes effect immediately on the running crawl and updates the crawl workflow settings for future runs.

Pausing and Resuming Crawls

If you need to reassess or rescope your crawl at any point after it has started, you can pause the running crawl.

To pause a running crawl, click the Pause button. The crawl status will change from Running to Pausing as in-progress pages are completed, and then to Paused once the crawler is successful paused. Paused crawls do not continue to accrue execution time.

While a crawl is paused, it is possible to replay the pages crawled up to that point and to download the WACZ files from the Latest Crawl tab.

To resume a paused crawl, simply click the Resume button. The crawl status will update from Resuming to Running to indicate that the crawler has started crawling again. Any changes to the workflow settings will be applied in the the resumed crawl.

Note

Paused crawls that are not resumed within 7 days of being paused are automatically updated to Stopped. Once stopped, the crawl is finished and can no longer be resumed.

Rate Limit Detection

A website may limit the number of requests it receives in a given amount of time. This practice is called rate limiting and it can improve server performance, mitigate network attacks, and reduce spam traffic.

Rate limiting can also make pages harder to archive: when a visitor (human, bot, or crawler) is rate limited by a website, they may see an error or CAPTCHA page instead of the actual page content for the given URL. Browsertrix attempts to capture only actual page content (i.e. prevent including error pages in an archive) by automatically detecting and temporarily skipping such error pages.

If too many error pages are encountered, the crawler adapts by slowing down and retrying the page URL after a longer delay. This behavior is distinct from the page delay workflow setting, which can reduce the chance of being rate limited by increasing the time spent on each page, at the cost of increasing the overall crawl time.

Unfortunately, there is not much Browsertrix can do to prevent being rate limited altogether. Adding a browser profile or configuring a proxy server may help reduce rate limits for certain sites, while other sites may need to provide explicit permission to be crawled, thus requiring the list of IP ranges used by Browsertrix.

Allow-listing Browsertrix on your website

Paid Feature If you subscribe to hosted Browsertrix and need help with being rate limited by your own website, please reach out to support for assistance.

Rate Limited Workflow Status

If the crawler can no longer continue by skipping error pages while being rate limited, it will switch to a time-delay-based strategy to continue running. The workflow status during this state will be Running (Rate Limited). In this state, the crawler will slow down and retry the page URL at longer intervals, up to once every 5 minutes. Pages that were not captured before will be queued to be retried. Crawls in this state will only use a few seconds of crawl time to check that it is still rate limited, thus avoiding wasting execution minutes on error pages.

If the crawl remains rate limited for an extended period of time (12 hours by default), the crawl may be automatically paused to avoid retrying indefinitely. The workflow status will then be Paused: Rate Limit Timeout.

End a Crawl

If a crawl workflow is not crawling websites as intended it may be preferable to end crawling operations and update the crawl workflow's settings before trying again. There are two operations to end crawls, available both on the workflow's details page, or as part of the actions menu in the workflow list.

Stopping

Stopping a crawl will throw away the crawl queue but otherwise gracefully end the process and save anything that has been collected. Stopped crawls show up in Archived Items and can be used like any other item in the app.

Canceling

Canceling a crawl will throw away all data collected and immediately end the process. Canceled crawls do not show up in Archived Items, though a record of the runtime and workflow settings can be found in the crawl workflow's list of crawls.