Next.js Discord

Discord Forum

How to handle a large JSON file with vercel

Unanswered
cnow posted this in #help-forum
Open in Discord
I have a very large JSON file, and I want to get a random element from an array within the file. How do I go about this? The file is too big to be uploaded to vercel, so I have it in a google storage bucket.

6 Replies

Unluckily, Vercel Blob service is still private beta, so work-around could be splitting the file into small piece of chunks, like array-001.json, array-002.json
then from Vercel side, random picking any of them. array-rand().json
Asian black bear
@cnow Since you are already using another cloud service, why not load it into a document database like Firestore or DynamoDB?
@Asian black bear <@250625952965459968> Since you are already using another cloud service, why not load it into a document database like Firestore or DynamoDB?
I could… how would i go about putting a json in firestore? (since i’m already using it for user auth)
@cnow I could… how would i go about putting a json in firestore? (since i’m already using it for user auth)
Asian black bear
It really depends on what is in your JSON file. How are you deciding what key to access from the array?
Asian black bear
Reading around the firestore docs, I would probably make a new document for each array element, possibly using the array index as the key (unless there is a better key-like thing in the data)
To do that you would need some kind of ETL process whereby you could submit the JSON, which would fit roughly into your workflow where uploading it to storage bucket happens now