beautify cURL download of JSON before writing to file

Question

I am using cURL 7.39.0 to call a REST webservice and write the response to a .json file:

curl -L-X POST -b cookies.txt -H "Content-Type: application/json" http://localhost:7001/web/service/url -d {"param1":"value1"} -o "C:\output\serviceName.json" 

The response is written to the output file, but, without formatting:

{"status": "success","user": "name", "regId": "14420","subscriber": [{"memberFor":"3 years","lastLogin":"2 days ago"}]}

My questions are:

1. Is there a way to format the JSON response before writing it to the output file (like below)?

   {
       "status": "success",
       "user": "name",
       "regId": "14420",
       "subscriber": [
           {
               "memberFor":"3 years",
               "lastLogin":"2 days ago"     
           }
       ]
   }
   

2. If formatting is not possible via cURL, I wish to write a simple batch file to automatically open the JSON output file and apply formatting. Something like:

   @echo off
   cls
   "C:\Program Files\Notepad++\notepad++" "C:\output\serviceName.json" 
   pause
   

Are there any flags/options available in batch/console to achieve this?

Quodlibet

cURL is basically a copy utility so it downloads the file as is. The second part of the question has many googlable duplicate questions on stackoverflow with various solutions. – Avram

@wOxxOm - Thanks! I found out that it is possible using python but I am unable to find the right syntax. I am using "filename.json" | python -m json.tool but I am getting - No JSON object can be decoded. Can you please point me to the right syntax? – Quodlibet

@Quodlibet maybe try python -m json.tool < filename.json. If that doesn't work, you could try type filename.json | python -m json.tool. – Everybody

@Everybody - both the syntaxes work and format the json response in the cmd window itself. It doesn't format the JSON file that is written as output. However, I will try your suggested solutions and mark as accepted. – Quodlibet

If you decide you prefer the python beautifier after all, you ought to be able to redirect the output into a file. python -m json.tool < filename.json > beautified.json – Everybody

That works! But it creates two json files, one formatted and the other non-formatted. Is there a way to avoid the non-formatted file from being created. Just curious. Apologies for pestering but I am having a hard time with Batch syntax as a beginner. – Quodlibet

You could try python -m json.tool < filename.json > filename.json but I'm not sure whether that'll successfully overwrite or give you a file-in-use error. If error, then python -m json.tool < filename.json > filename.tmp && move /y filename.tmp filename.json. – Everybody

Answer 1

Edit: I found a solution using the htmlfile COM object, which should offer the fastest performance (at least for a single run) and does not require an Internet connection. See the last solution in this answer.

---

Because you tagged this question with the [batch-file] tag, and because I found the challenge interesting, I wrote a hybrid batch + JScript script that'll beautify your JSON. Since JScript 5.7 doesn't natively support the JSON object, this script uses an external json2.js, downloading it via XHR if it's not already been downloaded. From there, it's a simple matter of calling JavaScript's familiar JSON.stringify() method with its beautify options.

Syntax:

json_generator | batfile.bat
    -or-
batfile.bat < jsonfile.json

Example usage:

beautify.bat < "C:\output\serviceName.json" > "C:\output\beautified.json"

This results in the following being saved as beautified.json:

{  
        "status": "success",
        "user": "name",
        "regId": "14420",
        "subscriber": [
                {
                        "memberFor": "3 years",
                        "lastLogin": "2 days ago"
                }
        ]
}

The code:

@if (@CodeSection == @Batch) @then

@echo off & setlocal

cscript /nologo /e:JScript "%~f0"
goto :EOF

@end // end Batch / begin JScript hybrid chimera

var xObj = WSH.CreateObject('Microsoft.XMLHTTP'),
    fso = WSH.CreateObject('Scripting.FileSystemObject'),
    temp = WSH.CreateObject('WScript.Shell').Environment('Process')('temp'),
    j2lib = 'https://raw.githubusercontent.com/douglascrockford/JSON-js/master/json2.js',
    json = WSH.StdIn.ReadAll();

if (fso.FileExists(temp + '\\json2.js')) {
    j2lib = fso.OpenTextFile(temp + '\\json2.js', 1);
    eval(j2lib.ReadAll());
    j2lib.Close();
}
else {
    with (xObj) {
        open("GET", j2lib, true);
        setRequestHeader('User-Agent', 'XMLHTTP/1.0');
        send('');
    }

    while (xObj.readyState != 4) WSH.Sleep(50);
    eval(xObj.responseText);
    j2lib = fso.CreateTextFile(temp + '\\json2.js', true);
    j2lib.Write(xObj.responseText);
    j2lib.Close();
}

WSH.Echo(JSON.stringify(JSON.parse(json), null, '\t'));

---

Here's another solution using the same syntax that does not require downloading json2.js. It avoids this by launching Internet Explorer invisibly, calling IE's built-in JSON methods, then silently closing IE again. This is most likely going to be slower than the method above, and it could be blocked depending on machine security policies; but it does have the advantage of working without an Internet connection.

@if (@CodeSection == @Batch) @then

@echo off & setlocal

cscript /nologo /e:JScript "%~f0"
goto :EOF

@end // end Batch / begin JScript hybrid chimera

var IE = WSH.CreateObject('InternetExplorer.Application'),
    json = WSH.StdIn.ReadAll();

IE.Visible = 0;
IE.Navigate('about:blank');
while (IE.Busy || IE.ReadyState != 4) WSH.Sleep(25);

var JSON = IE.document.parentWindow.JSON,
    pretty = JSON.stringify(JSON.parse(json), null, "\t");

WSH.Echo(pretty);

IE.Quit();
try { while (IE && IE.Busy) WSH.Sleep(25); }
catch(e) {}

---

Here's one more solution, this time using a batch / HTA hybrid. There's a <meta> tag forcing the HTA interpreter into IE9 compatibility, thus including support for JSON methods. This is faster than the IE method, but isn't completely invisible. The HTA window flashes on the screen for an instant, then closes itself.

<!-- : batch portion

@echo off & setlocal

rem // The for /f loop forces mshta to communicate with stdout
rem // as a console script host.  Without for /f, attempting
rem // to write to stdout results in an invalid handle error.
for /f "delims=" %%I in ('mshta.exe "%~f0"') do echo(%%I
goto :EOF

end batch / begin HTA : -->

<meta http-equiv="x-ua-compatible" content="IE=9" />
<script>
var fso = new ActiveXObject('Scripting.FileSystemObject'),
    stdin = fso.GetStandardStream(0),
    stdout = fso.GetStandardStream(1),
    json = stdin.ReadAll(),
    pretty = JSON.stringify(JSON.parse(json), null, '\t');

close(stdout.Write(pretty));
</script>

---

The best solution I think is to use the scantily-documented htmlfile COM object. Using the same trick with a <meta> tag to force it into IE9 compatibility as was demonstrated with the HTA solution above, the htmlfile COM object offers native support for JSON methods without requiring a library download, and without forking an additional windowed helper application. It just loads a dll.

@if (@CodeSection == @Batch) @then

@echo off & setlocal

cscript /nologo /e:JScript "%~f0"
goto :EOF

@end // end batch / begin JScript hybrid chimera

var htmlfile = WSH.CreateObject('htmlfile'),
    json = WSH.StdIn.ReadAll();

htmlfile.write('<meta http-equiv="x-ua-compatible" content="IE=9" />');

var JSON = htmlfile.parentWindow.JSON,
    pretty = JSON.stringify(JSON.parse(json), null, '\t');

htmlfile.close(WSH.Echo(pretty));

Everybody

WOW! @Everybody - I didn't know there could be so many ways to make this work. I will try them all, especially, the last solution involving the htmlfile CON object. Thank you very much! – Quodlibet

Well, actually, the first solution, the one that downloads json2.js, is the fastest after json2.js has been downloaded. But the last one is the fastest that doesn't require an Internet connection. – Everybody

@Quodlibet Just curious, what is it you intend to do with the JSON data after it's been beautified? If your intention is to scrape the data using text manipulation, you should re-evaluate your situation. These JSON methods I'm suggesting all put the data into an object, then serialize it into a beautified string again. You can cut out the middle man by doing something like var sessData = JSON.parse(json); then WSH.Echo('Last login: ' + sessData.subscriber[0].lastLogin); or similar. – Everybody

@rojo -I am creating a batch file that makes use of cURL to call multiple web services that require a set of input params. My aim is to configure the URLs and inputs separately, call cURL for each configured URL and input , prettify and write the response to a json file, place the output files in a directory with a time-stamp. This directory wil be accessed by a testing team, who have their own plugin to verify test cases/ WS responses. At present , we are using some browser extensions to call each WS, save the JSON response, place all the responses in a directory, all done manually. – Quodlibet

Upvoted now.Earlier, I didn't have the upvote privilege. Without a doubt, your answers were very helpful! They taught me tricks of BATCH programming that I couldn't learn from online docs. I will mark your answer as accepted once I try them out. Currently, I have some other priorities which I am working on. So please bear with me. I reckon there is no time-frame to mark-as-accepted. Thank you – Quodlibet

All your solutions taught me a thing or two (more than that, actually) about Batch scripting.But personally, I preferred using Python to format the output. Then again , you pointed me to the right syntax in one of your comments above. Hence, I mark your answer as accepted. Mighthy thanks for your help, @Everybody :) – Quodlibet

Hi rojo, I asked a similar question here. Would appreciate it if you could take look at that one as well. Thanks in advance. – Comprehension

@Foad That's a really good question. I noticed you left out the x-ua-compatible meta tag to force the htmlfile COM object into IE9 (or higher) compatibility, as well as cloning the method into the current runspace from htmlfileObj.parentWindow. But adding those missing elements doesn't really fix it either. I can find no way through this htmlfile object hack to expose Object.getOwnPropertyDescriptor() in a working fashion. Attempting to use it makes the script silently die, and try / catch catches a vague type error, like the thing you're trying to get the Property Descriptor... – Everybody

... of isn't a standard object. What's especially weird is that I have used this method successfully to clone other object methods. In any case, forcing the Chakra engine at the command line using cscript //E:{1b7cd997-e5ff-4932-a7a6-2a9e636da385} as Kul-Tigin suggests is probably the only solution. – Everybody

Answer 2

Just an update for oldish question. https://stedolan.github.io/jq/ is pretty nice tool. By default, jq pretty-prints JSON output. So you can do

curl ... | jq .

"." is identity filter. See https://stedolan.github.io/jq/manual/#Invokingjq

Bern

While this link may answer the question, link only answers are discouraged on Stack Overflow, you can improve this answer by taking vital parts of the link and putting it into your answer, this makes sure your answer is still an answer if the link gets changed or removed :) – Forby

Answer 3

I believe this can all be done with Xidel, which prettyprints by default:

xidel -s --method=POST ^
--load-cookies="cookies.txt" ^
-H "Content-Type: application/json" ^
-d {"param1":"value1"} ^
http://localhost:7001/web/service/url ^
-e "$json"

Expected output:

{
  "status": "success",
  "user": "name",
  "regId": "14420"
  "subscriber": [
    {
      "memberFor":"3 years",
      "lastLogin":"2 days ago"     
    }
  ]
}

Rock