Best way to format large JSON file? (~30 mb)

Question

I need to format a large JSON file for readability, but every resource I've found (mostly online) doesn't deal with data say, above 1-2 MB. I need to format about 30 MB. Is there any way to do this, or any way to code something to do this?

Endure

Do you simply need to read it? Or you want to format the output before sending it do clients? – Mckean

Answer 1

With python >= 2.6 you can do the following:

For Mac/Linux users:

cat ugly.json | python -m json.tool > pretty.json

For Windows users (thanks to the comment from dnk.nitro):

type ugly.json | python -m json.tool > pretty.json

Funderburk

For windows users: type ugly.json | python -mjson.tool > pretty.json – Guerdon

Using a recent Python3 version you can simplify the command python -mjson.tool input.json > formatted.json. – Schock

It looks like Python limits the size of the file to 2Gb. So if you have really large files, this solution won't work for you. – Ipsambul

I'm unsure why all the windows pythons commands don't have a space between -m and json. Only this worked for me: type ugly.json | python -m json.tool > pretty.json (powershell) – Murdock

I just used it on a .json file that contained Japanese text and it's converting the Japanese text to "\u304a\u3061\u3093\u3061\u3093" strings. It formats the .json correctly but seems to also transform the data so be careful. – Spirit

It crashes, by lack of RAM – Viper

There must be a space between -m and json.tool (windows powershell). python -m json.tool input.json > formatted.json Otherwise there will be No module named json. __main__. .tool is ignored. – Crocked

Answer 2

jq can format or beautify a ~100MB JSON file in a few seconds:

jq '.' myLargeUnformattedFile.json > myLargeBeautifiedFile.json

The command above will beautify a single-line ~120MB file in ~10 seconds, and jq gives you a lot of json manipulation capabilities beyond simple formatting, see their tutorials.

Supramolecular

It loads everything into RAM – Alesha

Worked fine on a 100MB file, but not on a 900MB file. :( – Toein

In windows, I had to use "." instead of '.' – Illjudged

This worked for me on a 172MB file, whereas Visual Studio Code just stared me straight in the face and said "I won't do that, it's too big." when I asked it to "format" the document. Thanks! – Plerre

If anyone copy-pastes large JSON and wants to skip having a file, one can run get-clipboard | jq "." | set-clipboard in PowerShell – Birdcage

another window example: type big_unformatted.json | c:\path_to_jq\jq-win64.exe > big_formatted.json – Grandaunt

Good tip, worked great here... using WIndows 11 64 bits! Thanks! You can have it installed with ease if you use Chocolatey. – Crackbrained

Didn't work for me for a file ~112 MB. cat ugly.json | python -m json.tool > pretty.json worked for me – Idocrase

Answer 3

jsonpps is the only one worked for me (https://github.com/bazaarvoice/jsonpps).
It doesn't load everything to RAM unlike jq, jsonpp and others that I tried.

Some useful tips regarding installation and usage:

Download url: https://repo1.maven.org/maven2/com/bazaarvoice/jsonpps/jsonpps/1.1/jsonpps-1.1.jar

Shortcut (for Windows):

1. Create file jsonpps.cmd in the same directory with the following content:
   @echo off java -Xms64m -Xmx64m -jar %~dp0\jsonpps-1.1.jar %*

Shortcut usage examples:

1. Format stdin to stdout:
   echo { "x": 1 } | jsonpps
2. Format stdin to file
   echo { "x": 1 } | jsonpps -o output.json
3. Format file to file:
   jsonpps input.json -o output.json

Alesha

Answer 4

Background-- I was trying to format a huge json file ~89mb on VS Code using the command (Alt+Shift+F) but the usuals, it crashed. I used jq to format my file and store it in another file.

A windows 11 use case is shown below.

step 1- download jq from the official site for your respective OS - https://stedolan.github.io/jq/

step 2- create a folder in the C drive named jq and paste the executable file that you downloaded into the folder. Rename the file as jq (Error1: beware the file is by default an exe file so do not save it as 'jq.exe' save it only as 'jq')

step 3- set your path variable to the URL of the executable file.

step 4- open your directory on cmd where the json file is stored and type the following command - jq . currentfilename.json > targetfilename.json

replace currentfilename with the file name that you want to format replace targetfilename with the final file name that you want your data formatted in

---

within seconds you should see your target file in the same directory in a formatted version which can now be opened on VS Code or any editor for that matter. Any error related to the recognizability of jq as a command can be traced back with high probability to Error 1.

jq jquery json data-preprocessing data-cleaning

Hufuf

Answer 5

You can use Notepad++ (https://notepad-plus-plus.org/downloads/) for formatting large JSON files (tested in Windows).

1. Install Notepad++
2. Go to Plugins -> Plugins Admin -> Install the 'Json Viewer' plugin. The plugin source code is present in https://github.com/kapilratnani/JSON-Viewer
3. After plugin installation, go to Plugins -> JSON Viewer -> Format JSON.

This will format your JSON file

Wondrous

Answer 6

I was also looking for a similar task and I just used Postman beautify to format a 3 MB JSON file and it worked.

Otolith