'sorta'.

I haven't done it in node/nextjs so I won't be able to speak to the specifics, but

15mb is 15mb and if your'e going to evaluate/parse that on the client side to render options/content, you're gonna have to transfer all 15mb's.

Now, you might be able to speed that up by creating a compressed version of the content (ala zip) if you haven't already

15mb of just json can be reduced by a lot if it's just text. If it's binary data or mixed? you won't get as much bandwidth.

All text

That still leaves you the challenge of unzipping it in the clients browser though and that means using their cpu/memory which, 15mbs? Isn't actually crazy.

Any other approach that I can think of will be some method of server side querying.

It looks like it can be zipped from 15mb to about 1mb

sounds about right.

I expect that file to keep growing, maybe server side querying is the right apprach

Would I have to move to a db or could json work for that?

If you're actually hosting this on a server however and not as simple static content, you might be able to employ some manner of graphql or sqlite to store the data. Or you can do some clever json deconstruction and normalization, but that'll be very dependent on the data and on your willingness to handle the maintenance.

The data is pretty well structured

If there's something you can 'index' it on, you might be able to have an initial list of keys whose values all point to the appropriate sharded piece of your content.

Don't have anything too good

To index

But, again, if you're gonna start going down that path, you should look just a little over the hill to see what kind of extra lift you'd need to do to just put it in a db

but, remember: the db is just going to help you if you need to query and get partial result sets. If you're just going to try and send all the content back, having it in a db won't just magically speed up the process.

because then you're just back to '15mbs of data'

I can't even tell if the json is what is slowing things down

1.1mb makes me think next auto compresses

HTTP servers usually try to use gzip or brotli compression by default unless you say not to

your sever config may vary. But, yes, from that it does look like it's already being compressed. A deeper look into the xhr/requests might tell you more.

You're importing it in a server side component and then sending a smaller subset of the data to the client, is that right?

It could also be that the way you're evaluating/parsing 15mbs of data in the browser is causing problems.

Correct

This is my site by the way: https://gedd.it/

Then I would think it would only be included in the server bundle and not sent as a whole to the client

Ah, I presumed he was sending the whole thing. So maybe it's the server parsing the data that's bogging it down. that'd make the 3.61s make more sense.

Actually, I send the data filtered but not one page worth at a time

I send all of it

My page size is 60

Could that help?

As @Dwarf Hotot said, I would guess all the time is being spent filtering it on the server rather than transferring it to the client

Not sure how your filtering looks but if it can't be optmized in any way then it's probably time to move it to a db, whether that be nosql/sqllite/mongo whatever will let you query in an engine

Looks like this:

  const productFilter = (products: Product[]) => {
    return products.filter(el =>
      {
        if (initialParams.actions && !initialParams.actions.split(",").some( action_val => el.actions.includes(action_val) )) { return false }
        if (initialParams.r && !initialParams.r.split(",").includes(el.reddit_subreddit)) { return false }
        if (initialParams.genders && !initialParams.genders.split(",").some( gender_val => el.genders.includes(gender_val) )) { return false }
        if (initialParams.categories && !initialParams.categories.split(",").some( category_val => el.categories.includes(category_val) )) { return false }
        if (initialParams.sizes && !initialParams.sizes.split(",").some( size_val => el.sizes.includes(size_val) )) { return false }
        if (initialParams.conditions && !initialParams.conditions.split(",").some( condition_val => el.conditions.includes(condition_val) )) { return false }
        if (initialParams.countries && !initialParams.countries.split(",").some( country_val => el.countries.includes(country_val) )) { return false }
        if (initialParams.brands && !initialParams.brands.split(",").some( brand_val => el.brands.includes(brand_val) )) { return false }

        if (initialParams.search) {
          const all_searchable_text: string = [el.title, el.reddit_author, el.brands.join(" "), el.categories.join(" "), el.sizes.join(" "), el.countries.join(" "), el.conditions.join(" "), el.colors.join(" "), el.search_keywords, el.reddit_post_id, el.reddit_comment_id, el.reddit_thread_id].filter(Boolean).join(" ").toLowerCase();
          
          // search every word, plural and singular
          let search_terms = initialParams.search.toLowerCase().split(" ")
          
          let word: string;
          for (word of search_terms) {
            if (!all_searchable_text.includes(word)) {
              if (word[word.length - 1] === 's') {
                if (!all_searchable_text.includes(word.slice(0, -1))) { return false }
              } else {
                return false
              }
            }
          }
        }
        return true
      }
    )
  }

Probably very inefficient

I'm new to js/ts

oh, yea. yea. don't do that.

Your heart's in the right place, but many hours were sacrificed researching and building the types of structures, tools, and algorithms so you don't have to invent that wheel.

Yeah I bet, so why is this bad?

So, that filter action? Everything that's happening in it, is happening for every single item in products

Yeah, so should I do it until I reach the page size?

I don't know what 'page size' is, but you're doing this (from what you've communicated) over a big Json file

well, not even that big really.

15mb, not a lot.

Page size is products shown per page

So how can I make it more efficient?

So that code, from the looks of it, is going to do that over the entire json file, return you some list of matches, which you're then presumably using to populate your page

Yeah

if you're more comfortable working with JSON, then you can do mongodb. but in general databases are meant to do that kind of work for you, fast

you could memoize the filter to 'stop' at some n set of results, return those, and then re-start from where it left off if it's called again, but thats' waht a DB would do

Yeah for sure, I think I'll try the page size limit then move to a db

A db is also going to be faster at doing text look ups, comparisons, indexing, etc...

Nice, could you give me an example of this?

You'll feel right at home with [MongoDB](https://www.mongodb.com/) if you're comfortable with the filters you're already writing 🙂

How to stop at some point

there's a lot lot lot of good reasons to move to a DB, only a few to make you stay away: No budget/host, small files

For sure

I want to try the quick fix real quick and then decide

Quick fixes == technical debt

just, keep that in mind

so long as you know that, you'll be able to make better decisions

This is a 2 week old app, so I am ok with that

re; memoization: https://www.freecodecamp.org/news/memoization-in-javascript-and-react/

try reading that.

I won't be able to create an example for you without actually diving into and working on your code base (and getting paid 😆 ) but the basic idea is that you basically create a sort of 'statefulness' that can be used to pick up where you left off. For languages like python and javascript, you can do that dynamically through closures and higher-order-functions. Funky powerful stuff.

If this is just something running on the back end, you might just introduce a token of some sort

token in the form of 'start at this word`. The absence of the token means start from the beginning. It could be tough with the JSON though, since you can't start at a byte position and keep reading...which is another way using a DB would improve your situation.

@josh isn't wrong about MongoDB but I'd send you in the direction of SQLlite just because of my own personal morals.

When I comment out all my filters (but still keep filtering process) it takes the same amount of time to load

now your'e starting to wade into 'profiling' territory

sure

'thank you'

np!

Sorry there's not an 'easy' answer for your situation

Does the fact that removing filters doesn't reduce load time invalidate our previous theory about filtering being the source of the issue?

so if in productFilter you simply return products, it takes the same amount of time?

Let me try that

Roughly yes

Both 2.5 s on local

Then maybe importing the JSON is causing the slowdown, if there's nothing else in-between. Surprised though

the fix is the same though

i prefer sql over doc too, but thought it might be an easier transition from just json

My data is more fit for json unfortunately, I think I'll try mongodb

Yea, like I said: You're not wrong, I just can't with MongoDB and that's totally my own (old) biases

Lots of yelling at the void and trying to explain CAP/ACID to clients and 'no schema!' just means doing arbitrary schema validation in code.

Isn't that 'normal'? When react imports JSON, it turns it into a proper data structure from text and that means having to fully evaluate the structure.

But why does it take 2+ seconds to do that?

Seems too slow for the amount of data

profile profile profile

could be disk I/O, could be a node issue, could be some weird library that's being called indirectly. Hard to say without putting on my waders

Is there a profiling library you suggest?

no, outside of my (personal) wheel house for react/nextjs. Maybe start here and see if it meets your needs? https://nodejs.org/en/guides/simple-profiling

cool thanks

glhf!