Batching File Uploads
Unanswered
Toyger posted this in #help-forum
ToygerOP
This isn't necessarily specif to NextJS however I'm not sure where else to ask, and perhaps someone else has solved a similar issue.
I have a frontend where a user can upload any number of zip files to a server. They can drag and drop them into the UI and they will be displayed in a list. From there, I want them to be able to either hit "Upload" on a single item, in which case that item will be uploaded immediately, or select "Upload All" in which case all items will be uploaded. They can't all be uploaded at once though, as this could result in errors, so they should be uploaded in batches of X at a time (5, 10, whatever). Whenever a single file finishes, another should be pulled from the queue and should start uploading.
My question is: What is the best way to handling the batching? I don't really want to have all of this in a single component, so I was thinking about using something like Zustand with middleware, or Redux with redux-observable, or sosme sort of state management library that can trigger side effects so that the majority of the business logic is handled outside the React scope, and the UI just responds to state updates.
Has anyone dealt with something like this?
I have a frontend where a user can upload any number of zip files to a server. They can drag and drop them into the UI and they will be displayed in a list. From there, I want them to be able to either hit "Upload" on a single item, in which case that item will be uploaded immediately, or select "Upload All" in which case all items will be uploaded. They can't all be uploaded at once though, as this could result in errors, so they should be uploaded in batches of X at a time (5, 10, whatever). Whenever a single file finishes, another should be pulled from the queue and should start uploading.
My question is: What is the best way to handling the batching? I don't really want to have all of this in a single component, so I was thinking about using something like Zustand with middleware, or Redux with redux-observable, or sosme sort of state management library that can trigger side effects so that the majority of the business logic is handled outside the React scope, and the UI just responds to state updates.
Has anyone dealt with something like this?
6 Replies
I read about https://www.rabbitmq.com/
Maybe that's a good service for the bulk upload. I just have an example text for mails, but I think it's the same logic for you:
Maybe that's a good service for the bulk upload. I just have an example text for mails, but I think it's the same logic for you:
Use a dedicated queue manager, for example RabbitMQ for queueing the emails. Nodemailer creates a callback function with related scopes etc. for every message so it might be hard on memory if you pile up the data for 10 million messages at once. Better to take the data from a queue when there’s a free spot in the connection pool (previously sent message returns its callback).
ToygerOP
I think something like that would work, however it's a bit overkill in this case.
I think it should be doable with just some regular JS logic, I'm just somehow at a loss for how to implement it. I'm sure some nasty concoction of useState and useEffect would make it work but that feels very dirty.
I think it should be doable with just some regular JS logic, I'm just somehow at a loss for how to implement it. I'm sure some nasty concoction of useState and useEffect would make it work but that feels very dirty.
Why can’t they all be uploaded at once?
Oh are you uploading them to your nextjs server instead of a dedicated object store like s3 ?
The latter can take pretty much anything you throw at it