I have a loop starting with for i in range(0, 100). Normally it runs correctly, but sometimes it fails due to network conditions. Currently I have it set so that on failure, it will continue in the except clause (continue on to the next number for i).
Is it possible for me to reassign the same number to i and run through the failed iteration of the loop again?
Do a while True inside your for loop, put your try code inside, and break from that while loop only when your code succeeds.
for i in range(0,100):
while True:
try:
# do stuff
except SomeSpecificException:
continue
break
continue does not use the same element of the iterable; instead it continues to the next. –
Sergei continue retries the while loop, of course, not the for (!), so i is not "the next" anything -- it's exactly the same as it was on a previous (failed) leg of the same while, of course. –
Refugiorefulgence while True. In my opinion, it has no beneficial value. It might confuse readers (as demonstrated in the comments above). –
Ditter for loop and then the infinite while. If retrying a failed attempt x number of times is what you are looking for, a single for for else loop is probably what you want. –
Spacial I prefer to limit the number of retries, so that if there's a problem with that specific item you will eventually continue onto the next one, thus:
for i in range(100):
for attempt in range(10):
try:
# do thing
except:
# perhaps reconnect, etc.
else:
break
else:
# we failed all the attempts - deal with the consequences.
else as relates to flow control? Under which circumstances is the "we failed..." part of the code executed? –
Inexistent # do thing with each i. The outer loop is what you're really trying to do (iterate an action) and the inner loop is for retrying any failures that happen among those 100 actions. –
Aram else clause of the try does this "if successful, then break" that you are looking for. –
Denitrify except clause, and remember to use 9 and not 10. –
Denitrify last_exception that you assign in the attempt loop, and then just raise that if it's not None in the outer else. –
Gonsalves else clause is for? If you put # do thing in that clause, an unhandled exception should be raised (unless the action suceeds). –
Aldus except part? Say if you try to connect to a URL in the try part but failed due to some usual errors, in most cases these are transient errors and may be fixed easily with retry, you want to reconnect with the URL in the except part, provided that all the URLs in the outer loop are actually accessible. This may be very easy to fix but I'm really new to python and data scraping, would you kindly help me out of this? –
Brannen except clause, then raise from rebound_exception or something like that? –
Geisha try docs say that a lot of context is lost outside the except clause, so maybe I should just do if i == 0: raise e in the except clause? –
Geisha UPDATE 2021-12-01:
As of June 2016, the retrying package is no longer being maintained. Consider using the active fork github.com/jd/tenacity, or alternatively github.com/litl/backoff.
The retrying package is a nice way to retry a block of code on failure.
For example:
@retry(wait_random_min=1000, wait_random_max=2000)
def wait_random_1_to_2_s():
print("Randomly wait 1 to 2 seconds between retries")
Here is a solution similar to others, but it will raise the exception if it doesn't succeed in the prescribed number or retries.
tries = 3
for i in range(tries):
try:
do_the_thing()
except KeyError as e:
if i < tries - 1: # i is zero indexed
continue
else:
raise
break
retries is misleading. It should much rather be tries. –
Skiver retrying: tenacity and backoff (2020 update)The retrying library was previously the way to go, but sadly it has some bugs and it hasn't got any updates since 2016. Other alternatives seem to be backoff and tenacity. During the time of writing this, the tenacity had more GItHub stars (2.3k vs 1.2k) and was updated more recently, hence I chose to use it. Here is an example:
from functools import partial
import random # producing random errors for this example
from tenacity import retry, stop_after_delay, wait_fixed, retry_if_exception_type
# Custom error type for this example
class CommunicationError(Exception):
pass
# Define shorthand decorator for the used settings.
retry_on_communication_error = partial(
retry,
stop=stop_after_delay(10), # max. 10 seconds wait.
wait=wait_fixed(0.4), # wait 400ms
retry=retry_if_exception_type(CommunicationError),
)()
@retry_on_communication_error
def do_something_unreliable(i):
if random.randint(1, 5) == 3:
print('Run#', i, 'Error occured. Retrying.')
raise CommunicationError()
for i in range(100):
do_something_unreliable(i)
The above code outputs something like:
Run# 3 Error occured. Retrying.
Run# 5 Error occured. Retrying.
Run# 6 Error occured. Retrying.
Run# 6 Error occured. Retrying.
Run# 10 Error occured. Retrying.
.
.
.
More settings for the tenacity.retry are listed on the tenacity GitHub page.
do_something_unreliable(i), but you could have do_something_unreliable(i, j, k) if you wish. (2) If you want to pass some other parameters for the tenacity.retry, just add them to the partial. –
Thundering The more "functional" approach without using those ugly while loops:
def tryAgain(retries=0):
if retries > 10: return
try:
# Do stuff
except:
tryAgain(retries+1)
tryAgain()
except: tryAgain(retries+1) –
Frazzle The clearest way would be to explicitly set i. For example:
i = 0
while i < 100:
i += 1
try:
# do stuff
except MyException:
continue
i += 1 just after # do stuff. –
Dioecious range for this kind of stuff. –
Hymettus break after # do stuff –
Ruthieruthless for _ in range(5):
try:
# replace this with something that may fail
raise ValueError("foo")
# replace Exception with a more specific exception
except Exception as e:
err = e
continue
# no exception, continue remainder of code
else:
break
# did not break the for loop, therefore all attempts
# raised an exception
else:
raise err
My version is similar to several of the above, but doesn't use a separate while loop, and re-raises the latest exception if all retries fail. Could explicitly set err = None at the top, but not strictly necessary as it should only execute the final else block if there was an error and therefore err is set.
err = None at the top in some ways, but as I noted I don't think it's actually ever used. As a matter of fact, one could then omit the else and use a golang-esque if err or if err is not None in case err could somehow have a falsely value. –
Headway A generic solution with a timeout:
import time
def onerror_retry(exception, callback, timeout=2, timedelta=.1):
end_time = time.time() + timeout
while True:
try:
yield callback()
break
except exception:
if time.time() > end_time:
raise
elif timedelta > 0:
time.sleep(timedelta)
Usage:
for retry in onerror_retry(SomeSpecificException, do_stuff):
retry()
except exception: –
Noncontributory try … except you can use a if statement. But it is less pythonic. –
Superphosphate callback function is never called. I have forgotten the parenthesis, replace by callback(). –
Superphosphate There is something similar in the Python Decorator Library.
Please bear in mind that it does not test for exceptions, but the return value. It retries until the decorated function returns True.
A slightly modified version should do the trick.
Using while and a counter:
count = 1
while count <= 3: # try 3 times
try:
# do_the_logic()
break
except SomeSpecificException as e:
# If trying 3rd time and still error??
# Just throw the error- we don't have anything to hide :)
if count == 3:
raise
count += 1
Using recursion
for i in range(100):
def do():
try:
## Network related scripts
except SpecificException as ex:
do()
do() ## invoke do() whenever required inside this loop
attempts = 3
while attempts:
try:
...
...
<status ok>
break
except:
attempts -=1
else: # executed only break was not raised
<status failed>Decorator is a good approach.
from functools import wraps
import time
class retry:
def __init__(self, success=lambda r:True, times=3, delay=1, raiseexception=True, echo=True):
self.success = success
self.times = times
self.raiseexception = raiseexception
self.echo = echo
self.delay = delay
def retry(fun, *args, success=lambda r:True, times=3, delay=1, raiseexception=True, echo=True, **kwargs):
ex = Exception(f"{fun} failed.")
r = None
for i in range(times):
if i > 0:
time.sleep(delay*2**(i-1))
try:
r = fun(*args, **kwargs)
s = success(r)
except Exception as e:
s = False
ex = e
# raise e
if not s:
continue
return r
else:
if echo:
print(f"{fun} failed.", "args:", args, kwargs, "\nresult: %s"%r)
if raiseexception:
raise ex
def __call__(self, fun):
@wraps(fun)
def wraper(*args, retry=0, **kwargs):
retry = retry if retry>0 else self.times
return self.__class__.retry(fun, *args,
success=self.success,
times=retry,
delay=self.delay,
raiseexception = self.raiseexception,
echo = self.echo,
**kwargs)
return wraper
some usage examples:
@retry(success=lambda x:x>3, times=4, delay=0.1)
def rf1(x=[]):
x.append(1)
print(x)
return len(x)
> rf1()
[1]
[1, 1]
[1, 1, 1]
[1, 1, 1, 1]
4
@retry(success=lambda x:x>3, times=4, delay=0.1)
def rf2(l=[], v=1):
l.append(v)
print(l)
assert len(l)>4
return len(l)
> rf2(v=2, retry=10) #overwite times=4
[2]
[2, 2]
[2, 2, 2]
[2, 2, 2, 2]
[2, 2, 2, 2, 2]
5
> retry.retry(lambda a,b:a+b, 1, 2, times=2)
3
> retry.retry(lambda a,b:a+b, 1, "2", times=2)
TypeError: unsupported operand type(s) for +: 'int' and 'str'
Here is a quick decorator to handle this. 7 lines, no dependencies.
def retry(exception=Exception, retries=3, delay=0):
def wrap(func):
for i in range(retries):
try:
return func()
except exception as e:
print(f'Retrying {func.__name__}: {i}/{retries}')
time.sleep(delay)
raise e
return wrap
@retry()
def do_something():
...
@retry(HTTPError, retries=100, delay=3)
def download_something():
...
An addition that could be made is extending exception to handle multiple exceptions (splat a list).
e is unbound here. –
Mowery I use following in my codes,
for i in range(0, 10):
try:
#things I need to do
except ValueError:
print("Try #{} failed with ValueError: Sleeping for 2 secs before next try:".format(i))
time.sleep(2)
continue
break
I use this, which can be used on any function:
def run_with_retry(func: callable, max_retries: int = 3, wait_seconds: int = 2, **func_params):
num_retries = 1
while True:
try:
return func(*func_params.values())
except Exception as e:
if num_retries > max_retries:
print('we have reached maximum errors and raising the exception')
raise e
else:
print(f'{num_retries}/{max_retries}')
print("Retrying error:", e)
num_retries += 1
sleep(wait_seconds)
Call like this:
def add(val1, val2):
return val1 + val2
run_with_retry(func=add, param1=10, param2=20)
This is my answer, using a decorator approach, a decor function with parameters:
import time
def retry(retries=3, delay=0.5):
def decor(func):
def wrap(*args, **kwargs):
for i in range(retries):
try:
return func(*args, **kwargs)
except Exception as e:
print(f'Retrying {func.__name__}: {i}/{retries}')
time.sleep(delay)
if(i == retries-1):
raise e
return wrap
return decor
@retry(retries = 5, delay=0.5)
def divide(a,b):
print(a/b)
When you call the divide function like divide(6,0), it will retry 5 times, then throw the 'division by zero' exception.
divide(6,0)
Retrying divide: 0/5
Retrying divide: 1/5
Retrying divide: 2/5
Retrying divide: 3/5
Retrying divide: 4/5
Traceback (most recent call last):
Cell In[48], line 1
divide(6,0)
...
...
print(a/b)
ZeroDivisionError: division by zero
If you want a solution without nested loops and invoking break on success you could developer a quick wrap retriable for any iterable. Here's an example of a networking issue that I run into often - saved authentication expires. The use of it would read like this:
client = get_client()
smart_loop = retriable(list_of_values):
for value in smart_loop:
try:
client.do_something_with(value)
except ClientAuthExpired:
client = get_client()
smart_loop.retry()
continue
except NetworkTimeout:
smart_loop.retry()
continue
Here is my take on this issue. The following retry function supports the following features:
import time
def retry(func, ex_type=Exception, limit=0, wait_ms=100, wait_increase_ratio=2, logger=None):
attempt = 1
while True:
try:
return func()
except Exception as ex:
if not isinstance(ex, ex_type):
raise ex
if 0 < limit <= attempt:
if logger:
logger.warning("no more attempts")
raise ex
if logger:
logger.error("failed execution attempt #%d", attempt, exc_info=ex)
attempt += 1
if logger:
logger.info("waiting %d ms before attempt #%d", wait_ms, attempt)
time.sleep(wait_ms / 1000)
wait_ms *= wait_increase_ratio
Usage:
def fail_randomly():
y = random.randint(0, 10)
if y < 10:
y = 0
return x / y
logger = logging.getLogger()
logger.setLevel(logging.INFO)
logger.addHandler(logging.StreamHandler(stream=sys.stdout))
logger.info("starting")
result = retry.retry(fail_randomly, ex_type=ZeroDivisionError, limit=20, logger=logger)
logger.info("result is: %s", result)
See my post for more info.
I like to use bool values for this, like so:
success = False
num_try = 0
while success is False:
if num_try >= 10: # or any number
# handle error how you please
try:
# code
success = True
except Exception as e:
# record or do something with exception if needed
num_try += 1
with this decorator, you can easily control errors
class catch:
def __init__(self, max=1, callback=None):
self.max = max
self.callback = callback
def set_max(self, max):
self.max = max
def handler(self, *args, **kwargs):
self.index = 0
while self.index < self.max:
self.index += 1
try:
self.func(self, *args, **kwargs)
except Exception as error:
if callable(self.callback):
self.callback(self, error, args, kwargs)
def __call__(self, func):
self.func = func
return self.handler
import time
def callback(cls, error, args, kwargs):
print('func args', args, 'func kwargs', kwargs)
print('error', repr(error), 'trying', cls.index)
if cls.index == 2:
cls.set_max(4)
else:
time.sleep(1)
@catch(max=2, callback=callback)
def test(cls, ok, **kwargs):
raise ValueError('ok')
test(1, message='hello')
If retrying a failed attempt x number of times is what you are looking for, a single for else loop is probably what you want. Consider this example with 3 attempts:
attempts = 3
for attempt in range(1, attempts+1):
try:
if attempt < 4:
raise TypeError(f"Error raised on attempt: {attempt}")
else:
print(f'Attempt {attempt} finally worked.')
except (TypeError) as error:
print(f'Attempt {attempt} hit the exception.')
continue
else:
break
else:
print(f'Exit after final attempt: {attempt}')
print(f'\nGo on to execute other code ...')
Gives the output:
Attempt 1 hit the exception.
Attempt 2 hit the exception.
Attempt 3 hit the exception.
Exit after final attempt: 3
Go on to execute other code ...
And with one more attempt it succeeds:
attempts = 4
Gives the output:
Attempt 1 hit the exception.
Attempt 2 hit the exception.
Attempt 3 hit the exception.
Attempt 4 finally worked.
Go on to execute other code ...
You could have a dedicated function using return to short circuit the result. For example like this:
def my_function_with_retries(..., max_retries=100):
for attempt in range(max_retries):
try:
return my_function(...)
except SomeSpecificException as error:
logging.warning(f"Retrying after failed execution: {error}")
raise SomeOtherException()
I liked laurent-laporte's answer. Here's my version of it wrapped in a class with static methods and some examples. I implemented a retry count as another way to retry. Also added kwargs.
from typing import List
import time
class Retry:
@staticmethod
def onerror_retry(exception, callback, retries: int = 0, timeout: float = 0, timedelta: float = 0,
errors: List = None, **kwargs):
"""
@param exception: The exception to trigger retry handling with.
@param callback: The function that will potentially fail with an exception
@param retries: Optional total number of retries, regardless of timing if this threshold is met, the call will
raise the exception.
@param timeout: Optional total amount of time to do retries after which the call will raise an exception
@param timedelta: Optional amount of time to sleep in between calls
@param errors: A list to receive all the exceptions that were caught.
@param kwargs: An optional key value parameters to pass to the function to retry.
"""
for retry in Retry.__onerror_retry(exception, callback, retries, timeout, timedelta, errors, **kwargs):
if retry: retry(**kwargs) # retry will be None when all retries fail.
@staticmethod
def __onerror_retry(exception, callback, retries: int = 0, timeout: float = 0, timedelta: float = 0,
errors: List = None, **kwargs):
end_time = time.time() + timeout
continues = 0
while True:
try:
yield callback(**kwargs)
break
except exception as ex:
print(ex)
if errors:
errors.append(ex)
continues += 1
if 0 < retries < continues:
print('ran out of retries')
raise
if timeout > 0 and time.time() > end_time:
print('ran out of time')
raise
elif timedelta > 0:
time.sleep(timedelta)
err = 0
#
# sample dumb fail function
def fail_many_times(**kwargs):
global err
err += 1
max_errors = kwargs.pop('max_errors', '') or 1
if err < max_errors:
raise ValueError("I made boo boo.")
print("Successfully did something.")
#
# Example calls
try:
#
# retries with a parameter that overrides retries... just because
Retry.onerror_retry(ValueError, fail_many_times, retries=5, max_errors=3)
err = 0
#
# retries that run out of time, with 1 second sleep between retries.
Retry.onerror_retry(ValueError, fail_many_times, timeout=5, timedelta=1, max_errors=30)
except Exception as err:
print(err)
I wrote this one. Hopefully it's clear to understand and easy to use.
from functools import wraps
from random import randrange
from time import sleep
def retry(retry_limit=3, delay=None, max_random_delay=60):
"""
Retrying decorator with simple backoff logic.
:param retry_limit: amount of retries before throwing exception
:param delay: delay between attempts. If None, random delay each time
:param max_random_delay: maximum random delay in seconds
"""
def decorator(func):
@wraps(func)
def wrapper(*args, **kwargs):
retries = 0
while retries < retry_limit:
try:
print(f'trying to execute {func.__name__}: {retries + 1}/{retry_limit}')
return func(*args, **kwargs)
except Exception as e:
retries += 1
if retries == retry_limit:
raise e
if delay is None:
sleep_time = randrange(max_random_delay)
print(f'sleep for {sleep_time} seconds')
sleep(sleep_time)
else:
print(f'sleep for {delay} seconds')
sleep(delay)
return wrapper
return decorator
Here's my idea on how to fix this:
j = 19
def calc(y):
global j
try:
j = j + 8 - y
x = int(y/j) # this will eventually raise DIV/0 when j=0
print("i = ", str(y), " j = ", str(j), " x = ", str(x))
except:
j = j + 1 # when the exception happens, increment "j" and retry
calc(y)
for i in range(50):
calc(i)
i recently worked with my python on a solution to this problem and i am happy to share it with stackoverflow visitors please give feedback if it is needed.
print("\nmonthly salary per day and year converter".title())
print('==' * 25)
def income_counter(day, salary, month):
global result2, result, is_ready, result3
result = salary / month
result2 = result * day
result3 = salary * 12
is_ready = True
return result, result2, result3, is_ready
i = 0
for i in range(5):
try:
month = int(input("\ntotal days of the current month: "))
salary = int(input("total salary per month: "))
day = int(input("Total Days to calculate> "))
income_counter(day=day, salary=salary, month=month)
if is_ready:
print(f'Your Salary per one day is: {round(result)}')
print(f'your income in {day} days will be: {round(result2)}')
print(f'your total income in one year will be: {round(result3)}')
break
else:
continue
except ZeroDivisionError:
is_ready = False
i += 1
print("a month does'nt have 0 days, please try again")
print(f'total chances left: {5 - i}')
except ValueError:
is_ready = False
i += 1
print("Invalid value, please type a number")
print(f'total chances left: {5 - i}')
increment your loop variable only when the try clause succeeds
© 2022 - 2024 — McMap. All rights reserved.
range(100)without the first parameter. If you use Python 2.x you could even usexrange(100), this generates an iterator and uses less memory. (Not that it matters with only 100 objects.) – Hexamerous