API reference

class scrapyloganalyzer.ScrapyLogFile(name, text='')[source]

A representation of a Scrapy log file.

Parameters:
classmethod find(logs_directory, source_id, data_version)[source]

Find and return the first matching log file for the given crawl.

Parameters:

  • logs_directory (str) – Kingfisher Collect’s project directory within Scrapyd’s logs_dir directory

  • source_id (str) – the spider’s name

  • data_version (datetime.datetime) – the crawl directory’s name, parsed as a datetime

delete()[source]

Delete the log file and any log summary ending in .stats.

property logparser: dict

Return the output of logparser.

match(data_version)[source]

Return whether the crawl directory’s name, parsed as a datetime, is less than 3 seconds after the log file’s start time.

Return type:

bool

property crawl_time: datetime

Return the crawl_time spider argument if set, or the start_time crawl statistic otherwise. If neither is logged, return the time of the first log message.

is_finished()[source]

Return whether the log file contains a “Spider closed (finished)” log message or a finish_reason crawl statistic set to “finished”.

Return type:

bool

property item_counts: dict

Return the number of each type of item, according to the log file.

property spider_arguments: dict

Return the spider’s arguments.

is_complete()[source]

Return whether the crawl collected a subset of the dataset, according to the log file.

Return type:

bool

read()[source]

Return the text content of the log file.

property error_rate: float

Return an estimated lower bound of the true error rate.

Kingfisher Collect is expected to yield at most one ERROR message per request leading to a File item, so the true error rate can only be less than this estimated lower bound if Kingfisher Collect breaks this expectation. On the other hand, the true error rate can easily be higher than the estimated lower bound; for example:

  • If the spider crawls 10 URLs, each returning 99 URLs, each returning OCDS data, and the requests for 5 of the 10 fail, then the estimated lower bound is 5 / 500 (1%), though the true error rate is 50%.

  • Similarly if the spider crawls 10 archive files, each containing 99 OCDS files, or 10 JSON files each containing 99 release packages.