Next.js Discord

Discord Forum

Is my backend team right?🤔

Answered
Spectacled bear posted this in #help-forum
Open in Discord
Spectacled bearOP
So I'm working on an AI image gen app and I proposed a full text search feature that would allow users to search all their generated images based on the prompt. The problem is our image collection is over 1 billion rows (mongodb btw). They pushed back and said they tried this in the past and it was not performant because the collection/table is too big and it would sometimes take up to 8 seconds. Now I believe it's probably because they were doing full text search on the full 1 billion+ table/collection, but if we query by first filtering it down based on userID first then perform the full text search on that much smaller set it would be performant right? I only know SQL database so I'm not sure but it wouldn't be similar for mongodb too? And if you know more performant way to do this, please let me know🙏
Answered by Somali
const MongoClient = require('mongodb').MongoClient;

async function searchImages(userID, searchString, page, pageSize) {
  const client = new MongoClient('mongodb://localhost:27017', {
    useNewUrlParser: true,
    useUnifiedTopology: true,
  });

  try {
    await client.connect();

    const db = client.db('yourDatabaseName');
    const collection = db.collection('yourImageCollectionName');

    // Perform the search combining user filtering and full-text search
    const searchResults = await collection
      .find({
        userID: userID,
        $text: { $search: searchString },
      })
      .skip((page - 1) * pageSize)
      .limit(pageSize)
      .project({ _id: 0 }) // Optionally, exclude the _id field from results
      .toArray();

    return searchResults;
  } finally {
    client.close();
  }
}
View full answer

18 Replies

aardani
why do you need to filter it based on userID?
@aardani why do you need to filter it based on userID?
Somali
I imagine it’s because they only want the user to search through their own images not the entire site.
aardani
if your mongodb database saves the images per user then that would be faster
but i dont think filtering based on user on ALL table would help
i.e user1 = [image1, image2, image3]
user2 = [image4, image5, image6]

then you can simply get user1's image list
@aardani i.e user1 = [image1, image2, image3] user2 = [image4, image5, image6] then you can simply get user1's image list
Somali
This would probably be ideal but I don’t know how their database is structured.
aardani
oh ure not OP xD
ok lets wait
Spectacled bearOP
the image collection has a userID column
image collection is all users generated images
Somali
Sounds like you’re sorted then
@Somali Sounds like you’re sorted then
Spectacled bearOP
Not really, I need a really fast and performant solution. I also never worked with a collection/table of this size I want some opinions.
Somali
This should be good

const MongoClient = require('mongodb').MongoClient;

async function searchImagesByUser(userID, searchString, page, pageSize) {
  const client = new MongoClient('mongodb://localhost:27017', {
    useNewUrlParser: true,
    useUnifiedTopology: true,
  });

  try {
    await client.connect();

    const db = client.db('yourDatabaseName');
    const collection = db.collection('yourImageCollectionName');

    // First, filter by userID
    const userFilteredResults = await collection
      .find({ userID: userID })
      .skip((page - 1) * pageSize)
      .limit(pageSize);

    // Apply the full-text search on the filtered results
    const searchResults = await userFilteredResults
      .project({ _id: 0 }) // Optionally, exclude the _id field from results
      .toArray();

    return searchResults;
  } finally {
    client.close();
  }
}
actually where is the full text search input...
Somali
const MongoClient = require('mongodb').MongoClient;

async function searchImages(userID, searchString, page, pageSize) {
  const client = new MongoClient('mongodb://localhost:27017', {
    useNewUrlParser: true,
    useUnifiedTopology: true,
  });

  try {
    await client.connect();

    const db = client.db('yourDatabaseName');
    const collection = db.collection('yourImageCollectionName');

    // Perform the search combining user filtering and full-text search
    const searchResults = await collection
      .find({
        userID: userID,
        $text: { $search: searchString },
      })
      .skip((page - 1) * pageSize)
      .limit(pageSize)
      .project({ _id: 0 }) // Optionally, exclude the _id field from results
      .toArray();

    return searchResults;
  } finally {
    client.close();
  }
}
Answer
Somali
@Spectacled bear here you go