Skip to main content

RequestManagerTandem

Implements a tandem behaviour for a pair of RequestLoader and RequestManager.

In this scenario, the contents of the "loader" get transferred into the "manager", allowing processing the requests from both sources and also enqueueing new requests (not possible with plain RequestManager).

Hierarchy

Index

Methods

Methods

__init__

  • __init__(request_loader, request_manager): None

add_request

  • Add a single request to the manager and store it in underlying resource client.

    Parameters

    • request: str | Request

      The request object (or its string representation) to be added to the manager.

    • optionalkeyword-onlyforefront: bool = False

      Determines whether the request should be added to the beginning (if True) or the end (if False) of the manager.

    Returns ProcessedRequest

add_requests

  • async add_requests(requests, *, forefront, batch_size, wait_time_between_batches, wait_for_all_requests_to_be_added, wait_for_all_requests_to_be_added_timeout): None

  • Add requests to the manager in batches.

    Parameters

    • requests: Sequence[str | Request]

      Requests to enqueue.

    • optionalkeyword-onlyforefront: bool = False

      If True, add requests to the beginning of the queue.

    • optionalkeyword-onlybatch_size: int = 1000

      The number of requests to add in one batch.

    • optionalkeyword-onlywait_time_between_batches: timedelta = timedelta(seconds=1)

      Time to wait between adding batches.

    • optionalkeyword-onlywait_for_all_requests_to_be_added: bool = False

      If True, wait for all requests to be added before returning.

    • optionalkeyword-onlywait_for_all_requests_to_be_added_timeout: timedelta | None = None

      Timeout for waiting for all requests to be added.

    Returns None

drop

  • async drop(): None

  • Remove persistent state either from the Apify Cloud storage or from the local database.

    Returns None

fetch_next_request

  • async fetch_next_request(): Request | None

  • Return the next request to be processed, or None if there are no more pending requests.

    The method should return None if and only if is_finished would return True. In other cases, the method should wait until a request appears.

    Returns Request | None

get_handled_count

  • async get_handled_count(): int

get_total_count

  • async get_total_count(): int

  • Get an offline approximation of the total number of requests in the loader (i.e. pending + handled).

    Returns int

is_empty

  • async is_empty(): bool

  • Return True if there are no more requests in the loader (there might still be unfinished requests).

    Returns bool

is_finished

  • async is_finished(): bool

mark_request_as_handled

  • async mark_request_as_handled(request): ProcessedRequest | None

reclaim_request

  • Reclaims a failed request back to the source, so that it can be returned for processing later again.

    It is possible to modify the request data by supplying an updated request as a parameter.

    Parameters

    • request: Request
    • optionalkeyword-onlyforefront: bool = False

    Returns ProcessedRequest | None

to_tandem

  • Combine the loader with a request manager to support adding and reclaiming requests.

    Parameters

    • optionalrequest_manager: RequestManager | None = None

      Request manager to combine the loader with. If None is given, the default request queue is used.

    Returns RequestManagerTandem

Page Options