Today I was using windows and I piped some output to a file.
echo '["test"]' > bad.json
When I looked at the file, the content inside looked fine. There was no extra spaces or weird characters. But when I read the file using Node’s fs module, I saw extra whitespace between the characters. In the console I saw this…
const fs = require('fs'); var output = fs.readFileSync('bad.json', 'utf8'); console.log(output);
Why were there extra spaces? If I ran the script using a manually created file (we’ll call it good.json), the output was displayed correctly with no extra spaces. So I decided to look at the byte count of each file. bad.json had 22 bytes, but good.json had only 8. When I did a hex dump to see the bytes in both files this is what I saw…
file name: bad.json 0000-0010: ff fe 5b 00-22 00 74 00-65 00 73 00-74 00 22 00 ..[.".t. e.s.t.". 0000-0016: 5d 00 0d 00-0a 00 ].....
file name: good.json 0000-0008: 5b 22 74 65-73 74 22 5d ["test"]
Why was Windows adding extra bytes when I piped output to a file? So I tried creating another file using stdout, but this time I used the windows command prompt (cmd). I was previously using PowerShell to create bad.json. This time I didn’t get extra bytes.
file name: cmd.json 0000-000b: 5b 22 74 65-73 74 22 5d-20 0d 0a ["test"] ...
So it looked like powershell was using a different character encoding than cmd and what I’m used to in Linux.
I found this StackOverflow post which said some versions of PowerShell use Unicode as the default encoding. I ran this command to find my poweshell version, and it said I was was using version 5.1.
As an experiment, I ran this command to force PowerShell to use utf-8.
write-output '["test"]' | Out-File 'powershell-test.json' -encoding utf8
file name: powershell-test.json 0000-000d: ef bb bf 5b-22 74 65 73-74 22 5d 0d-0a ...["tes t"]..
This was a little better, but it still had extra bytes at the front. I did some further investigation, and found this StackOverflow post which said PowerShell 5.1 creates UTF-8 files with a pseudo Byte Order Marker. It went on to say that the latest, cross platform PowerShell (PowerShell Core) no longer encodes UTF-8 with a BOM.
So I installed PowerShell 6.1.1 (the latest cross platform version) and tried to run my original command.
echo '["test"]' > buen.json
file name: buen.json 0000-000a: 5b 22 74 65-73 74 22 5d-0d 0a ["test"] ..
Much better! Now I wanted to switch the default terminal from the old windows PowerShell to the new PowerShell Core. Since I mainly use the terminal in Visual Studio Code, it was easy to change the default. I went into Visual Studio Code’s options and and changed the ‘terminal.integrated.shell.windows’ option from the old PowerShell path to the new PowerShell path.