Comment on [HELP] Converting JSON to Sqlite in C
abhibeckert@lemmy.world 1 year agoWell, Python is my friend now.
The Python read() function, which is basically the same as fread() also accepts a length parameter and 1024 is a sensible number to give it.
You can read the entire file in python but you probably shouldn’t do that. In the real world, your software will be given files larger than it can handle and it’s important to have logic in place to process the file incrementally instead of all at once. Honestly, it’s what makes this particular problem so challenging.
It’s definitely possible to write a C program that reads the entire file. You just won’t find many examples of that because it’s a bad idea.
You said you used an “AI to assist” but it failed? What were you using? I pasted it into ChatGPT 4 and it found eight problems - including the buffer issue.
While true it should stop reading a 30G file, I am the stupid when it come to this and I just need a program running.
The AI I used was Bing AI. It was “Helpful” since I don’t know C at all, but my primitive JS knowledge couldn’t save it.
I know small amount of Python and this is the only time I used AI for assistance. I have to say, the context is there, but needs fixing from time to time.
farcaller@fstab.sh 1 year ago
Streaming JSON parsers are a thing, e.g. pdjson for C. It’s, of course, a different approach and it’s generally slightly trickier to work with those, but that’s what you would use of you have unbound document size and you can process it in chunks.