Map Certain JSON Fields Into a Concatenated String to Create Filenames

Let’s say you have a JSON file and in it is an array of values. Each value has a few properties you need to pick from to ultimately create a file name. Here is an example using bash, jq, in2csv and xargs that can be strung together in one line to create a file based on each concatenated filename.

cat some.json \
     | jq -r '.statements | map([.account.number,.endDate,.id] | join("_") + ".pdf")' \
     | in2csv -f json \
     | sed -n '1d;p' \
     | xargs -I{} cp fubar.pdf {};

Let’s break this down into it’s individual pieces.

Sample JSON Input

{
   "statements" : [ {
     "id" : "60090438-83cc-433f-bbf9-8086f4da846f",
     "account" : {
       "number" : "801284289140212"
     },
     "endDate" : "2021-06-30",
     ...
   }
...
]
}

Using JQ To Select A Few Fields to Concatenate for a File Name

The following line will select three properties/fields out of the JSON

jq -r '.statements | map([.account.number,.endDate,.id] | join("_") + ".pdf")'

It will join them by an underscore and then concatenate a .pdf to the end of the file name.

Convert Quoted JQ JSON into CSV via In2Csv

The jq command will provide quoted output. Since, there is only one value per item in the array due to the concatenation, I just want to strip out the JSON quoting.

That is done with the following:

in2csv -f json

Note the use of the -f json since this is a piped command. That will yield this warning, which we can squash next

/usr/local/Cellar/csvkit/1.0.5_1/libexec/lib/python3.8/site-packages/agate/utils.py:276: UnnamedColumnWarning: Column 0 has no name. Using "a". 

Remove The First Line From the Bash Output

This piped command remove the header labeled a by default:

sed -n '1d;p'

Creating a Dummy Placeholder File Per File Name Generated with Xargs and Cp

Lastly, I want to generate a placeholder file (touch could be used too) per file name. This command achieves that with xargs

xargs -I{} cp fubar.pdf {};

Sample output:

 
 -rw-r--r--@    1 kevincoy  staff     1676399 Jun 29 16:48 182362260357264_2021-03-31_81f1465b-b2d6-43d9-87bd-d6ede8188188.pdf
 -rw-r--r--@    1 kevincoy  staff     1676399 Jun 29 16:48 468913841109837_2021-03-31_043af01f-bd27-42a7-b37d-c7c4b7d1275f.pdf
 -rw-r--r--@    1 kevincoy  staff     1676399 Jun 29 16:48 558664362096984_2021-03-31_287c8418-98bd-4088-9645-b897c40d0ed4.pdf
 -rw-r--r--@    1 kevincoy  staff     1676399 Jun 29 16:48 881961481598397_2021-03-31_14e1e791-3e2d-4275-b27a-bf49cc2999e8.pdf
 -rw-r--r--@    1 kevincoy  staff     1676399 Jun 29 16:48 201869782345979_2021-03-31_a623f37a-dfe7-4748-af2e-6b45dbeb869b.pdf
 -rw-r--r--@    1 kevincoy  staff     1676399 Jun 29 16:48 177636620098765_2021-03-31_4007c4c2-7263-4d58-8a8f-7b5bd6aef802.pdf
 -rw-r--r--@    1 kevincoy  staff     1676399 Jun 29 16:48 812555146927128_2021-03-31_f3a06bd9-efe3-4da1-bd29-6ee93473d920.pdf 

Resources:

• https://jqplay.org/s/N6TboUkELM
• https://csvkit.readthedocs.io/en/1.0.2/scripts/in2csv.html
• https://stackoverflow.com/questions/604864/print-a-file-skipping-the-first-x-lines-in-bash
• https://unix.stackexchange.com/questions/349252/cp-after-xargs-not-working