Mongodb Performance Tuning Tips Complete Guide

1. Indexing

Why? Indexes speed up data retrieval operations on a collection by reducing the amount of data MongoDB needs to scan.

Step 1: Ensure Proper Indexes

Example Scenario:

You have a collection of users with over a million documents. You frequently query this collection using the username field.

Steps:

1. Create a Single Field Index:

   db.users.createIndex({ username: 1 })
   

   • 1 means ascending order; use -1 for descending order.

2. Check Existing Indexes:

   db.users.getIndexes()
   

3. Query Using the Indexed Field:

   db.users.find({ username: "john_doe" }).explain("executionStats")
   

   • The explain method displays information on how the query was executed. Look for "stage": "IXSCAN" indicating an index scan.

4. Create a Compound Index: If you query by multiple fields often, consider creating a compound index.

   db.users.createIndex({ username: 1, last_login: -1 })
   

   • This index will optimize queries that filter by username and sort by last_login in descending order.

Best Practices:

• Always create indexes based on actual query performance.
• Avoid creating too many indexes as they can slow down write operations.

2. Query Optimization

Why? Optimizing your queries ensures MongoDB only scans the necessary documents, thereby enhancing performance.

Step 1: Use Projection

Example Scenario:

You need to retrieve only specific fields from each document in the orders collection.

Steps:

1. Specify Fields to Retrieve:

   db.orders.find({ customer_id: 123 }, { _id: 0, product_name: 1, quantity: 1 })
   

   • This command returns only the product_name and quantity fields for orders where customer_id is 123.

2. Avoid Returning Large Documents Unnecessarily: By default, the _id field is returned unless explicitly excluded.

Best Practices:

• Use projection to minimize the amount of transferred data.

Step 2: Use Covered Queries

Example Scenario:

You want to fetch all product_names and their corresponding prices from the products collection.

Steps:

1. Create an Index Covering Both Fields:

   db.products.createIndex({ product_name: 1, price: 1 })
   

2. Perform the Query:

   db.products.find(
     { product_name: "laptop" },
     { _id: 0, product_name: 1, price: 1 }
   ).explain("executionStats")
   

   • Ensure that "stage": "IXSCAN" and "covered": true in the execution plan.

Best Practices:

• A covered query retrieves all fields directly from the index without having to access the actual document.

3. Use Aggregation for Complex Queries

Why? Aggregations allow MongoDB to process data closer to where it resides, reducing the data transfer to your application.

Step 1: Basic Aggregation Query

Example Scenario:

Calculate the total number of products sold in a specific category.

Steps:

1. Design the Aggregation Pipeline:

   db.orders.aggregate([
     { $match: { category: "electronics" } }, // Filter orders by category
     { $group: {
         _id: "$product_name", 
         total_quantity: { $sum: "$quantity" }
       } 
     }, // Group by product name and sum quantities
     { $sort: { total_quantity: -1 } } // Sort in descending order
   ])
   

2. Ensure Index Usage: You may need an index on category and product_name for optimal performance.

   db.orders.createIndex({ category: 1, product_name: 1 })
   

3. Explain the Aggregation Pipeline:

   db.orders.aggregate([
     { $match: { category: "electronics" } },
     { $group: {
         _id: "$product_name",
         total_quantity: { $sum: "$quantity" }
       }
     },
     { $sort: { total_quantity: -1 } }
   ]).explain("executionStats")
   

   • Check that stages utilize indexes efficiently.

Best Practices:

• Use $match stages early in the pipeline to reduce the document set.
• Optimize sorting and grouping stages for best performance.

4. Sharding

Why? Sharding helps scale MongoDB horizontally by distributing data across multiple servers, improving performance for large datasets.

Prerequisites:

• Install and configure a MongoDB sharded cluster.
• Use a shard key to distribute data evenly.

Step 1: Shard a Collection

Example Scenario:

You have a logs collection storing application logs, which is growing rapidly and causing performance issues.

Steps:

1. Enable Sharding for the Database:

   sh.enableSharding("your_application_db")
   

2. Choose a Shard Key: Select a field that provides good distribution. For logs, timestamp could be a suitable shard key.

   sh.shardCollection("your_application_db.logs", { timestamp: 1 })
   

   • This command shards the logs collection by timestamp in ascending order.

3. Verify Sharding Status:

   db.adminCommand({ status: "sharding" })
   

4. Insert Data and Observe Distribution: Data should distribute across available shards.

Best Practices:

• Choose a shard key that minimizes hotspots (uneven data distribution).
• Monitor shard usage and adjust as necessary.

5. Memory Usage

Why? MongoDB relies heavily on memory for storing working sets (data being processed) in its WiredTiger storage engine.

Step 1: Increase Working Set Size

Example Scenario:

Your products collection is frequently queried, and MongoDB's memory footprint is low.

Steps:

1. Increase RAM:

   • Upgrade your server's RAM to hold more working data in memory.

2. Monitor Memory Usage: Use MongoDB’s built-in top or serverStatus commands.

   db.serverStatus().workingSet
   

3. Optimize Index Sizes:

   • Ensure smaller and fewer indexes if possible.
   • Use db.collection.totalIndexSize() to check the size of indexes.

Best Practices:

• Aim for a working set that fits comfortably in memory.
• Smaller indexes improve memory efficiency and speed.

6. Write Operations Optimization

Why? Efficient write operations are crucial for maintaining performance, especially in high-write-load environments.

Step 1: Use Bulk Operations

Example Scenario:

You need to insert or update thousands of records in the transactions collection.

Steps:

1. Prepare Data for Bulk Insert/Update:

   const transactions = [
     { account_id: 1, amount: 100, date: new Date() },
     { account_id: 2, amount: 200, date: new Date() },
     // ... more transaction documents
   ];
   

2. Perform Bulk Insert:

   db.transactions.insertMany(transactions)
   

3. Perform Bulk Update:

   db.transactions.updateMany(
     { account_id: { $in: [1, 2, 3] } },
     { $set: { status: "completed" } }
   )
   

Best Practices:

• Use insertMany, updateMany, and deleteMany for batch operations.
• Ensure atomicity and durability are maintained as per application requirements.

Step 2: Use Write Concerns Appropriately

Example Scenario:

Your application writes logs but does not require immediate confirmation of the writes.

Steps:

1. Set Write Concern w=0: This allows the server to accept the write request without confirming.

   db.logs.insertOne(
     { event: "login", user_id: 123 },
     { writeConcern: { w: 0 } }
   )
   

   • Caution: Use with care as it can lead to data loss in case of failures.

2. Set Write Concern w=1: Ensures that write operations complete only after being written to the primary node.

   db.logs.insertOne(
     { event: "login", user_id: 123 },
     { writeConcern: { w: 1 } }
   )
   

Best Practices:

• Choose an appropriate write concern based on data criticality and performance needs.

7. Connection Pooling

Why? Connection pooling reduces the overhead associated with establishing and tearing down connections between clients and the database.

Step 1: Configure Connection Pool Size

Example Scenario:

You develop an application with a high client load, requiring efficient management of database connections.

Steps:

1. Configure Connection Options in Your Driver: For example, in Node.js using the MongoDB driver:

   const { MongoClient } = require('mongodb');
   
   MongoClient.connect('mongodb://localhost:27017', {
     useNewUrlParser: true,
     useUnifiedTopology: true,
     poolSize: 50 // Set maximum connection pool size
   }).then(client => {
     console.log('Connected to MongoDB');
     // Proceed with further operations
   });
   

2. Monitor Connection Usage: Use MongoDB's serverStatus command.

   db.serverStatus().connections
   

3. Adjust Pool Size as Needed: Based on the application's requirements and server capabilities.

Best Practices:

• Set a reasonable connection pool size to balance resource utilization and performance.
• Monitor and fine-tune the pool size for optimal results.

8. Analyze and Use mongostat and mongotop

Why? Tools like mongostat and mongotop provide insights into the performance and resource usage of your MongoDB instance.

Step 1: Use mongostat

Example Scenario:

You want to monitor the read and write operations, memory usage, and other system metrics.

Steps:

1. Run mongostat Command:

   mongostat --rowcount 10
   

   • --rowcount 10 displays the last 10 rows captured every second.

2. Interpret the Output: Here's a snippet of what mongostat might show:

   insert  query update delete getmore command flushes mapped  vsize    res faults              num          vsize host               time
        0      1      0      0       0       2       0  884m  1.59g    64m      0                  10    2.97g localhost:27017  22:34:22
        0      2      0      0       0       2       0  884m  1.59g    64m      0                   7     1.5g localhost:27017  22:34:23
        0    163      0      0       0     238       0  884m  1.59g    64m      0                 544    2.69g localhost:27017  22:34:24
   

   • insert, query, update, delete: Count of these operations per second.
   • faults: Number of page faults per second (indicative of insufficient memory).
   • mapped: Memory space used by MongoDB.
   • vsize: Virtual memory space used by MongoDB.
   • res: Resident memory space used by MongoDB.

Best Practices:

• Regularly monitor these metrics to understand performance patterns.
• Adjust configurations based on observed statistics to improve performance.

Step 2: Use mongotop

Example Scenario:

Identify which collections are consuming the most CPU time.

Steps:

1. Run mongotop Command:

   mongotop 5
   

   • Refresh interval is 5 seconds.

2. Interpret the Output:

   2023-10-03T22:38:00.012+0000 database.collection           microsecs
   2023-10-03T22:38:05.012+0000 your_application_db.orders 78
   2023-10-03T22:38:05.012+0000 your_application_db.users  240
   2023-10-03T22:38:05.012+0000 your_application_db.logs   0
   

   • microsecs: Time spent on the collection in microseconds during each refresh interval.

Best Practices:

• Focus on optimizing collections with high microsec values.
• Review and adjust indexes and queries for better performance.

9. Monitoring Slow Queries using Profiler

Why? MongoDB's profiler allows you to log queries that exceed a certain threshold, helping identify slow operations.

Step 1: Enable Profiling

Example Scenario:

You want to log queries that take longer than one second in the inventory collection.

Steps:

1. Set Profiling Level:

   db.setProfilingLevel(2, { slowms: 1000 }) // Profile all queries, slow queries (over 1 second are highlighted)
   

   • Level 2 profiles all operations.
   • slowms: 1000 logs queries slower than 1000 milliseconds.

2. Retrieve Slow Queries:

   db.system.profile.find().sort({ ts: -1 }).limit(10) // Display recent queries sorted by timestamp
   

3. Review Logged Queries: Here's an example of a logged query:

   {
     millis: 1500,
     ns: "your_application_db.inventory",
     op: "query",
     query: { product_id: 123 },
     nreturned: 1000,
     responseLength: 1200000,
     ts: ISODate("2023-10-03T22:42:30.012Z"), // Timestamp when the query was executed
     keysExamined: 0,
     docsExamined: 1200000,
     protocol: "op_msg"
   }
   

   • millis: Execution time in milliseconds.
   • docsExamined: Number of documents examined during query execution.
   • keysExamined: Number of index entries accessed, helpful for index evaluation.

4. Optimize Identified Slow Queries: Based on the profiler output, adjust your queries or create necessary indexes.

Best Practices:

• Use profiling sparingly due to its overhead.
• Focus on frequently executed slow queries to optimize performance.

10. Configure Replica Sets for High Availability and Performance

Why? Replica sets provide redundancy and improve read scalability by allowing reads from secondary nodes.

Step 1: Set Up a Replica Set

Example Scenario:

You deploy a replica set with three nodes to ensure high availability and scale read operations.

Steps:

1. Start MongoDB Instances: Ensure you have three MongoDB instances running.

2. Initiate the Replica Set: Connect to one of the instances and run:

   rs.initiate({
     _id: "your_replica_set",
     members: [
       { _id: 0, host: "localhost:27017" },
       { _id: 1, host: "localhost:27018" },
       { _id: 2, host: "localhost:27019" }
     ]
   })
   

3. Wait for Elections: MongoDB will elect a primary node and secondary nodes automatically.

4. Enable Read Preferences on Secondary Nodes: In your application code, configure read preferences. For example, in Node.js:

   const client = await MongoClient.connect('mongodb://localhost:27017', {
     useNewUrlParser: true,
     useUnifiedTopology: true,
     readPreference: 'secondary' // Route reads to secondary nodes
   });
   

5. Monitor Replica Set Health: Use MongoDB shell commands.

   rs.status()
   

Best Practices:

• Use an odd number of replica set members (e.g., 3, 5) to avoid elections.
• Configure read preferences to balance the load among secondary nodes while ensuring data consistency.

Complete Example: Improving Performance of a Blogging Platform

Scenario Description:

You're building a blogging platform where users frequently browse posts, search them by tags, and comment on them.

Data Collections:

• posts
• comments
• tags

Challenges:

1. Slow post retrieval based on tags.
2. High read load causing memory issues.
3. Frequent write operations affecting performance.

Performance Tuning Steps:

1. Create Indexes for posts Collection

// Create index on `tags` array for faster querying by tags
db.posts.createIndex({ tags: 1 })

// Create compound index on `date` for sorting based on publication date
db.posts.createIndex({ date: -1, title: 1 })

2. Optimize Post Retrieval Queries

// Query posts by tags with projection
db.posts.find(
  { tags: { $all: ["mongodb", "performance"] } },
  { _id: 0, title: 1, author: 1, date: 1, tags: 1 }
).explain("executionStats")

// Expected: "stage": "FETCH" should transition to "stage": "IXSCAN" with `"indexBounds"`
// Check if the query utilizes indexes efficiently.

// Query posts with sort
db.posts.find(
  {},
  { _id: 0, title: 1, author: 1, date: 1 }
).sort({ date: -1 }).limit(10).explain("executionStats")

// Ensure the sort is covered by the compound index `{ date: -1, title: 1 }`.
// Check for `"isMultiKey": false` and `"indexOnly": true`.

3. Use Aggregation Framework for Tag-based Search

db.posts.aggregate([
  { $match: { tags: { $all: ["mongodb", "performance"] } } }, // Filter by tags
  { $lookup: { // Join comments collection
      from: "comments",
      localField: "_id",
      foreignField: "post_id",
      as: "comments_list"
    }
  },
  { $project: {
      _id: 0,
      title: 1,
      content: 1,
      author: 1,
      date: 1,
      tags: 1,
      comments_list: {
        comment_id: 1,
        commenter_id: 1,
        text: 1,
        date: 1
      }
    }
  },
  { $sort: { date: -1 } }, // Sort by publish date
  { $limit: 10 } // Limit results to top 10 posts
]).explain("executionStats")

4. Shard the posts and comments Collections

// Enable sharding on the database
sh.enableSharding("blogging_platform")

// Shard the `posts` collection by `date`
sh.shardCollection("blogging_platform.posts", { date: 1 })

// Shard the `comments` collection by `post_id`
sh.shardCollection("blogging_platform.comments", { post_id: 1 })

5. Optimize Memory Usage

// Monitor memory usage
db.serverStatus().workingSet

// Check index size for optimization
db.posts.totalIndexSize()
db.comments.totalIndexSize()

// Ensure working set fits in memory
// Adjust indexes if necessary

6. Use Bulk Operations for Handling Comments

If a single post receives numerous comments in a short period:

const comments = [
  { post_id: 1001, commenter_id: 123, text: "Great article!", date: new Date() },
  { post_id: 1001, commenter_id: 456, text: "Thanks for sharing.", date: new Date() },
  // ... more comments
];

db.comments.insertMany(comments)

7. Enable Read Preference for Scaling Reads

In your application configuration:

const client = await MongoClient.connect('mongodb://node1:27017,node2:27018,node3:27019', {
  useNewUrlParser: true,
  useUnifiedTopology: true,
  readPreference: 'secondary' // Route reads to secondary nodes
});

8. Configure Replica Set for High Availability

rs.initiate({
  _id: "blog_rs",
  members: [
    { _id: 0, host: "node1:27017" },
    { _id: 1, host: "node2:27018" },
    { _id: 2, host: "node3:27019" }
  ]
})

9. Monitor Performance Using mongostat and mongotop

// Monitor system-level stats
mongostat 5

// Monitor collection-level stats
mongotop 5

10. Log and Analyze Slow Queries

// Enable profiling for slow queries
db.setProfilingLevel(2, { slowms: 500 })

// Retrieve slow queries logs
db.system.profile.find().sort({ ts: -1 }).limit(10)

By following these detailed steps, you can significantly enhance the performance of your MongoDB database. Each tip addresses common bottlenecks and provides practical solutions to optimize various aspects of your database operations.

Additional Tips:

• Regular Maintenance: Perform regular maintenance tasks such as removing outdated data, compacting collections, and rebuilding indexes.

• Update MongoDB Versions: Keep your MongoDB server up to date with the latest stable version for performance improvements and security enhancements.

• Use Appropriate Data Types: Choose appropriate data types for your fields to ensure efficient storage and query processing.

• Limit Document Size: Keep individual documents small (less than 16MB) to improve read and write performance.

• Optimize Network Latency: Minimize network latency by placing your MongoDB cluster close to your application servers or users.

By applying these strategies methodically, you'll be well-equipped to tackle performance challenges in your MongoDB deployments even as your application scales.