Get page output with curl --fail

Asked 19/3, 2014 at 12:35 Answered 3/12, 2021 at 1:47

Calling curl without parameters, I get the page output, even with an http status code = 404:

$ curl http://www.google.com/linux
<!DOCTYPE html>
<html lang=en>
  <meta charset=utf-8>
  <meta name=viewport content="initial-scale=1, minimum-scale=1, width=device-width">
  <title>Error 404 (Not Found)!!1</title>
  <style>
    *{margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px}* > body{background:url(//www.google.com/images/errors/robot.png) 100% 5px no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}ins{color:#777;text-decoration:none}a img{border:0}@media screen and (max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//www.google.com/images/errors/logo_sm_2.png) no-repeat}@media only screen and (min-resolution:192dpi){#logo{background:url(//www.google.com/images/errors/logo_sm_2_hr.png) no-repeat 0% 0%/100% 100%;-moz-border-image:url(//www.google.com/images/errors/logo_sm_2_hr.png) 0}}@media only screen and (-webkit-min-device-pixel-ratio:2){#logo{background:url(//www.google.com/images/errors/logo_sm_2_hr.png) no-repeat;-webkit-background-size:100% 100%}}#logo{display:inline-block;height:55px;width:150px}
  </style>
  <a href=//www.google.com/><span id=logo aria-label=Google></span></a>
  <p><b>404.</b> <ins>That’s an error.</ins>
  <p>The requested URL <code>/linux</code> was not found on this server.  <ins>That’s all we know.</ins>

$ echo $?
0

The status code is 0.

Calling it with --fail will not show the output:

$ curl --fail http://www.google.com/linux
curl: (22) The requested URL returned error: 404 Not Found

$ echo $?
22

The status code is 22 now ...

I'd like to get the output even when http status = 404, 500 (like the first curl execution) and, at the same time, get a different system error (like in the second curl execution, $? = 22).

Is it possible with curl? If not, how could I achieve this with another tool (this tool must accept file uploads and post data! wget doesn't seem to be an alternative ...)

Bruns answered 19/3, 2014 at 12:35 Comment(0)

This is now possible with curl. Since version 7.76.0 you can do

curl --fail-with-body ...

Which does exactly what OP asked: shows the document body and exits with code 22.

See https://curl.se/docs/manpage.html#--fail-with-body

Scissile answered 3/12, 2021 at 1:47 Comment(0)

First of all the maximum value for the error code(or exit code) is 255. Here is the reference.

Also, the --fail will not allow you to do what you are looking for. However, you can use alternate ways(writing a shell script) to handle the scenario, but not sure it will be effective or not for you!

http_code=$(curl -s -o out.html -w '%{http_code}'  http://www.google.com/linux;)

if [[ $http_code -eq 200 ]]; then
    exit 0
fi

## decide which status you want to return for 404 or 500
exit  204

Now do the $? and you'll get the exit code from there.

You'll find the response html inside the out.html file.

You can also pass the url to the script as commandline argument. Check here.

Aaren answered 19/3, 2014 at 14:18 Comment(1)

superuser.com/a/1641410 has a better implementation with (almost) proper use of temporary files. – Riggle 22/9, 2023 at 9:19

Unfortunately not possible with curl. But you can do this with wget.

$ wget --content-on-error -qO- http://httpbin.org/status/418

    -=[ teapot ]=-

       _...._
     .'  _ _ `.
    | ."` ^ `". _,
    \_;`"---"`|//
      |       ;/
      \_     _/
        `"""`
$ echo $?
8

Chyou answered 18/5, 2017 at 21:41 Comment(1)

Note that you need wget 1.14+ – Mica 15/11, 2018 at 17:30

Thanks @timaschew, here is my enhanced version based on pure awk:

curl_fail_with_body() {
  curl -o - -w "\n%{http_code}\n" "$@" | awk '{l[NR] = $0} END {for (i=1; i<=NR-1; i++) print l[i]}; END{ if ($0<200||$0>299) exit $0 }'
}

# example usage
curl_fail_with_body -sS http://httpbin.org/status/418

Explanation

-o - -w "\n%{http_code}\n" - prints out to stdout (actually it's piped to the next command) with status code at the end
{l[NR] = $0} END {for (i=1; i<=NR-1; i++) print l[i]} - print all the lines except the last one
END{ if ($0<200||$0>299) exit $0 } - will exit with non zero code if the last line != 2xx

alternative version, if you want to output the error code after command:
END{ if ($0<200||$0>299) {print "The requested URL returned error: " $0; exit 1}

BTW, curl supports --fail-with-body option since v7.76.0.
This option allows you to achieve the desired behavior without using external tools.

Wolters answered 22/4, 2021 at 16:9 Comment(1)

Great solution, thanks. Nit: the exit code range is 0-255, so any HTTP status code > 255 won't come through as exit code. I changed it in my usage to exit 1 vs. exit $0. – Noose 2/6, 2021 at 21:11

I found a solution because wget was not suitable for sending multipart/form-data

curl -o - -w "\n%{http_code}\n" http://httpbin.org/status/418 | tee >(tail -n 1 | cmp <(echo 2xx) - ) | tee >(grep "char 2"; echo $? > status-code) && grep 0 status-code

Explanation

-o - -w "\n%{http_code}\n" - prints out to stdout (actually it's piped to the next command) with status code at the end
tee - output will be piped to next command and additionally printed to stdout
tail -n 1 - extract status code from the last line
cmp <(echo 2xx) - compare status code, first char only
grep "char 2" - if first character needs to be 2, otherwise fail

In a shell script you can also do better comparison (currently it only allows 2xx, so redirect like 300 are are handled as an error with cmp how it is used above)

Filet answered 19/3, 2018 at 16:8 Comment(6)

isn't it suppose to not fail when having a 200 response ? cause the previous comment does fail when calling httpbin.org/status/200 – Discretion 31/1, 2019 at 13:19

What do you exactly mean? The curl command I printed is not failing with 200 – Filet 6/2, 2019 at 16:10

It does fail on my machine. 😔 – Discretion 6/2, 2019 at 22:25

@Discretion is right. It exits with a non-zero code on 2xx status ranges. On a macos at least. – Afterglow 22/1, 2020 at 12:58

As well as on GNU bash: repl.it/@sepehr/curl-fail-with-error-body – Afterglow 22/1, 2020 at 13:5

Required a change from grep "char 2" to grep "byte 2". Works now. – Afterglow 22/1, 2020 at 14:2

Here was my solution - it uses jq and assumes the body is json

#  this code adds a statusCode field to the json it receives and then jq squeezes them together
# curl 7.76.0 will have curl --fail-with-body and thus eliminate all this
  local result
  result=$(
    curl -sL -w ' { "statusCode": %{http_code}} ' -X POST "${headers[@]}" "${endpoint}" \
      -d "${body}"  "$curl_opts" | jq -ren '[inputs] | add'
  )
#   always output the result
  echo "${result}"
#  jq -e will produce an error code if the expression result is false or null - thus resulting in a
# error return code from this function naturally. This is much preferred rather than assume/hardcode
# the existence of a error object in the body payload
  echo "${result}" | jq -re '.statusCode >= 200 and .statusCode < 300' > /dev/null

Herzl answered 31/7, 2021 at 23:49 Comment(0)

Explanation

Explanation

Recommended topics

Hot tags