Package entropy :: Module fetchers :: Class UrlFetcher

Class UrlFetcher

source code


Entropy single URL fetcher. It supports what Python's urllib2 supports, plus resuming, proxies and custom user agents. No external tools dependencies are required (including wget).

Instance Methods
 
__init__(self, url, path_to_save, checksum=True, show_speed=True, resume=True, abort_check_func=None, disallow_redirect=False, thread_stop_func=None, speed_limit=None, timeout=None, download_context_func=None, pre_download_hook=None, post_download_hook=None, http_basic_user=None, http_basic_pwd=None, https_validate_cert=True)
Entropy URL downloader constructor.
source code
 
set_id(self, th_id)
Set instance id (usually the thread identifier).
source code
 
download(self)
Start downloading URL given at construction time.
source code
int
get_transfer_rate(self)
Return transfer rate, in kb/sec.
source code
float
get_average(self)
Get current download percentage.
source code
int
get_seconds_remaining(self)
Return remaining seconds to download completion.
source code
 
is_resumed(self)
Return whether given download has been resumed.
source code
 
handle_statistics(self, th_id, downloaded_size, total_size, average, old_average, update_step, show_speed, data_transfer, time_remaining, time_remaining_secs)
Reimplement this callback to gather information about data currently downloaded.
source code
 
update(self)
Main fetch progress callback.
source code

Inherited from object: __delattr__, __format__, __getattribute__, __hash__, __new__, __reduce__, __reduce_ex__, __repr__, __setattr__, __sizeof__, __str__, __subclasshook__

Class Methods

Inherited from output.TextInterface: ask_question, edit_file, input_box, output, set_title

Static Methods
bool
supports_differential_download(url)
Return whether the current protocol handler for given URL supports differential download.
source code
Class Variables
  GENERIC_FETCH_ERROR = '-3'
  TIMEOUT_FETCH_ERROR = '-4'
  GENERIC_FETCH_WARN = '-2'

Inherited from output.TextInterface: OUTPUT_LOCK

Properties

Inherited from object: __class__

Method Details

__init__(self, url, path_to_save, checksum=True, show_speed=True, resume=True, abort_check_func=None, disallow_redirect=False, thread_stop_func=None, speed_limit=None, timeout=None, download_context_func=None, pre_download_hook=None, post_download_hook=None, http_basic_user=None, http_basic_pwd=None, https_validate_cert=True)
(Constructor)

source code 

Entropy URL downloader constructor.

Parameters:
  • url (string) - download URL (do not URL-encode it!)
  • path_to_save (string) - file path where to save downloaded data
  • checksum (bool) - return md5 hash instead of status code
  • show_speed (bool) - show download speed
  • resume (bool) - enable resume support
  • abort_check_func (callable) - callback used to stop download, it has to raise an exception that has to be caught by provider application. This exception will be considered an "abort" request.
  • disallow_redirect (bool) - disallow automatic HTTP redirects
  • thread_stop_func (callable) - callback used to stop download, it has to raise an exception that has to be caught by provider application. This exception will be considered a "stop" request.
  • speed_limit (int) - speed limit in kb/sec
  • timeout (int) - custom request timeout value (in seconds), if None the value is read from Entropy configuration files.
  • download_context_func (callable) - if not None, it must be a function exposing a context manager and taking a path (the download path) as argument. This can be used to implement locking on files to be downloaded.
  • pre_download_hook (callable) - hook called before starting the download process, inside the download_context_func context. This can be used to verify if the download is actually needed or just return. If the returned value is not None, the download method will return that value. The function takes a path (the download path) and the download id as arguments.
  • post_download_hook (callable) - hook called after the download is complete, inside the download_context_func context. This can be used to verify the integrity of the downloaded data. The function takes a path (the download path) and the download status and the download id as arguments.
Overrides: object.__init__

supports_differential_download(url)
Static Method

source code 

Return whether the current protocol handler for given URL supports differential download. It's up to UrlFetcher users to implement the logic to provide a previously downloaded file to "path_to_save" location (argument passed to UrlFetcher constructor).

Parameters:
  • url (string) - download URL
Returns: bool
differential download support status

set_id(self, th_id)

source code 

Set instance id (usually the thread identifier).

Parameters:
  • th_id (int) - id to set

download(self)

source code 
Start downloading URL given at construction time.

@return: download status, which can be either one of:
    UrlFetcher.GENERIC_FETCH_ERROR means error.
    UrlFetcher.TIMEOUT_FETCH_ERROR means timeout error.
    UrlFetcher.GENERIC_FETCH_WARN means warning,
        downloaded fine but unable to calculate the md5 hash.
Otherwise returns md5 hash.
@rtype: string
@todo: improve return data

get_transfer_rate(self)

source code 

Return transfer rate, in kb/sec.

Returns: int
transfer rate

get_average(self)

source code 

Get current download percentage.

Returns: float
download percentage

get_seconds_remaining(self)

source code 

Return remaining seconds to download completion.

Returns: int
remaining download seconds

handle_statistics(self, th_id, downloaded_size, total_size, average, old_average, update_step, show_speed, data_transfer, time_remaining, time_remaining_secs)

source code 

Reimplement this callback to gather information about data currently downloaded.

Parameters:
  • th_id (int) - instance identifier
  • downloaded_size (int) - size downloaded up to now, in bytes
  • total_size (int) - total download size, in bytes
  • average (float) - percentage of file downloaded up to now
  • old_average - old percentage of file downloaded
  • update_step (int) - currently configured update average delta
  • show_speed (bool) - if download speed should be shown for given instance
  • data_transfer (int) - current data transfer, in kb/sec
  • time_remaining (string) - currently hypothesized remaining download time, in string format (showing hours, minutes, seconds).
  • time_remaining_secs (int) - currently hypothesized remaining download time, in seconds.

update(self)

source code 

Main fetch progress callback. You can reimplement this to refresh your output devices.