Skip to content

CI: Add retry script for Docker commands #2516

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Apr 3, 2018
Merged

CI: Add retry script for Docker commands #2516

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
aa994d5
MAINT: Add pure Bourne shell retry_cmd.sh
effigies Mar 28, 2018
f7007ad
CI: Use retry_cmd.sh to retry Docker commands
effigies Mar 28, 2018
2cd3da3
MAINT: Handle multi-line and quoted arguments better in retry_cmd
effigies Mar 28, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 4 additions & 9 deletions .circleci/config.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,9 +36,8 @@ _get_base_image: &get_base_image
echo "Pulling base image ..."
docker pull nipype/nipype:base
elif [ "$GET_BASE" == "BUILD" ]; then
e=1 && for i in {1..5}; do
docker build -t nipype/nipype:base - < docker/Dockerfile.base && e=0 && break || sleep 15
done && [ "$e" -eq "0" ]
tools/retry_cmd.sh -n 5 -s 15 \
docker build -t nipype/nipype:base - < docker/Dockerfile.base
else
echo "Error: method to get base image not understood"
exit 1
Expand All @@ -48,22 +47,20 @@ _build_main_image_py36: &build_main_image_py36
name: Build main image (py36)
no_output_timeout: 60m
command: |
e=1 && for i in {1..5}; do
tools/retry_cmd.sh -n 5 -s 15 \
docker build \
--rm=false \
--tag nipype/nipype:latest \
--tag nipype/nipype:py36 \
--build-arg BUILD_DATE="$(date -u +"%Y-%m-%dT%H:%M:%SZ")" \
--build-arg VCS_REF="$(git rev-parse --short HEAD)" \
--build-arg VERSION="${CIRCLE_TAG}" /home/circleci/nipype \
&& e=0 && break || sleep 15
done && [ "$e" -eq "0" ]

_build_main_image_py27: &build_main_image_py27
name: Build main image (py27)
no_output_timeout: 60m
command: |
e=1 && for i in {1..5}; do
tools/retry_cmd.sh -n 5 -s 15 \
docker build \
--rm=false \
--tag nipype/nipype:py27 \
Expand All @@ -72,8 +69,6 @@ _build_main_image_py27: &build_main_image_py27
--build-arg BUILD_DATE="$(date -u +"%Y-%m-%dT%H:%M:%SZ")" \
--build-arg VCS_REF="$(git rev-parse --short HEAD)" \
--build-arg VERSION="${CIRCLE_TAG}-py27" /home/circleci/nipype \
&& e=0 && break || sleep 15
done && [ "$e" -eq "0" ]

_download_test_data: &_download_test_data
name: Download test data
Expand Down
36 changes: 36 additions & 0 deletions tools/retry_cmd.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
#!/bin/sh
#
# retry_cmd.sh [-n NLOOPS] [-s SLEEP] CMD
#
# Retry command until success or pre-specified number of failures
#
# 2018 Chris Markiewicz
# Released into public domain

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Doesn't this need to set set +o pipefail here to work? After some digging, I think the original problem reported in #2489 is just that the steps are run with set -eo pipefile (circle has started to run it with 2.0). Using both -eo will make the script to exit even with that || true pipe.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, because those apply to each instance of the shell. It doesn't infect subprocesses.

NLOOPS=3
TOSLEEP=5

while true; do
case "$1" in
-n ) NLOOPS="$2"; shift 2 ;;
-s ) TOSLEEP="$2"; shift 2 ;;
-- ) shift; break ;;
* ) break ;;
esac
done

# Normalize whitespace in command, preserving quotes
CMD=""
for ARG; do
CMD="$CMD \"$ARG\"";
done

RET=0
for i in `seq $NLOOPS`; do
sh -c "$CMD"
RET="$?"
if [ "$RET" -eq 0 ]; then break; fi
if [ "$i" -ne "$NLOOPS" ]; then sleep $TOSLEEP; fi
done

exit $RET