RequestLoader

An abstract class defining the interface for classes that provide access to a read-only stream of requests.

Request loaders are used to manage and provide access to a storage of crawling requests.

Key responsibilities:

• Fetching the next request to be processed.
• Marking requests as successfully handled after processing.
• Managing state information such as the total and handled request counts.

Hierarchy

• RequestLoader
  • RequestList
  • RequestManager
  • SitemapRequestLoader

Index

Methods

• to_tandem

Methods

:

:

:

:

:

:

to_tandem

• async to_tandem(request_manager): RequestManagerTandem

• Combine the loader with a request manager to support adding and reclaiming requests.

  ---

  Parameters

  • optionalrequest_manager: RequestManager | None = None

    Request manager to combine the loader with. If None is given, the default request queue is used.

  Returns RequestManagerTandem