RabbitMQ & Magento 2 on Managed Hosting by Nexcess

    February 10, 2022


    RabbitMQ & Magento 2 on Managed Hosting by Nexcess


    Nexcess: Configure the RabbitMQ/Magento 2 container option to reduce load/delivery times for the data exchange between processes, applications, and servers.

    Nexcess & RabbitMQ: Configuring RabbitMQ for Magento 2

    Before explaining the process of enabling and configuring RabbitMQ/Magento 2, let us understand what containers and RabbitMQ are in term of cloud technology.

    Containers

    Unleash the promise of cloud technology with software containers that expand the performance, management, and security of your site:


    Containers are units of software that can be added to your cloud solution to expand performance, functionality, and management. Containers are lightweight, secure, and external instances that will not take resources from your main cloud solution.

    RabbitMQ & Magento 2

    RabbitMQ is an open source message broker that helps websites to exchange data between processes, applications, and servers. This cloud technology item can help reduce load and delivery times by delegating tasks to a third party.

    Nexcess provides the RabbitMQ cloud container service option for our Managed Applications (MA) Magento Plans. This article will cover how to optimally configure RabbitMQ for Magento 2 after enabling the corresponding option via the Nexcess Client Portal

    Enabling the Magento 2 RabbitMQ Container

    Clients can enable the required container through the Nexcess Client Portal.

    To learn how to enable the RabbitMQ container, check out the How to enable containers in Nexcess Cloud articleOnce you enable the RabbitMQ container, you will receive details shown in the screenshot below related to the service.

    Endpoint, Port, Username

    Click on View Password option to find its related password. You will need that password information in the next step of configuring the service.

    Configuring the Magento 2 RabbitMQ Container

    To configure the service, you need to edit “/home/username/sitename/html/app/etc/env.php” file.

    In that file you, will see below section of code:

      'queue' => [
            
    'consumers_wait_for_messages' => 1
        ],


    You have to edit the code as follows:

       'queue' => [
            
    'amqp' => [
                
    'host' => 'Endpoint',
                
    'port' => 'port-number',
                
    'user' => 'username',
                
    'password' => 'password',
                
    'virtualhost' => '/'
            ],
            
    'consumers_wait_for_messages' => 1
        ],


    Make sure you have entered the correct values in the host, port, user, and password areas. Then save that file and run the below command to apply the changes:

    php bin/magento setup:upgrade

    That's it! You have successfully enabled and configured RabbitMQ Magento 2 on Nexcess!

    Manage Message Queues

    You can manage message queues from the command line using cron jobs to ensure that consumers retrieve messages.

    Process Management

    Cron jobs are the default mechanism to restart consumers. Processes started by cron consume the specified number of messages and then terminate. Rerunning cron restarts the consumer.

    The following example shows the Magento crontab configuration for running consumers.

    Path of cron job related to Message Queue:

    vendor/magento/module-message-queue/etc/crontab.xml

    $ cat vendor/magento/module-message-queue/etc/crontab.xml 
    <?xml version="1.0"?>
    <!--
      ~ Copyright (c) Magento, Inc. All rights reserved.
      ~ See COPYING.txt for license details.
      -->

    <config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="urn:magento:module:Magento_Cron:etc/crontab.xsd">
        
    <group id="default">
            
    <job name="messagequeue_clean_outdated_locks" instance="Magento\Framework\MessageQueue\Lock\WriterInterface" method="releaseOutdatedLocks">
                
    <schedule>0 * * * *</schedule>
            
    </job>
        
    </group>
        
    <group id="consumers">
            
    <job name="consumers_runner" instance="Magento\MessageQueue\Model\Cron\ConsumersRunner" method="run">
                
    <schedule>* * * * *</schedule>
            
    </job>
        
    </group>
    </config>

    How often you check message queues depends on your business logic and available system resources. In general, you’ll probably want to check for newly created customers and send welcome emails more frequently than a more resource-intensive process (for example, updating your catalog). It would be best if you defined cron schedules according to your business needs.

    Cron schedules can be configured in Admin Panel Stores > Settings > Configuration > Advanced > System > Cron configuration options for group: consumers

    To learn how to configure cron jobs for Magento 2, check out the How to configure Magento 2 cron jobs article 

    Configuration

    The configuration behaviors by default are:

    • The cron job consumers_runner is enabled.
    • The cron job consumers_runner runs all defined consumers.
    • Each consumer processes 10000 messages and then terminates.

    Specific Configuration

    Edit the /app/etc/env.php file to configure the cron job consumers_runner:

    ...
        
    'cron_consumers_runner' => [
            
    'cron_run' => false,
            
    'max_messages' => 20000,
            
    'consumers' => [
                
    'consumer1',
                
    'consumer2',
            ]
        ],
    ...

    Defintionss:

    • cron_run - A boolean value that enables or disables the consumers_runner cron job (default = true).
    • max_messages - The maximum number of messages each consumer must process before terminating (default = 10000). Although we do not recommend it, you can use 0 to prevent the consumer from terminating. See consumers_wait_for_messages to configure how consumers process messages from the message queue.
    • consumers - An array of strings specifying which consumer(s) to run. An empty array runs all consumers.

    View a List of Available Message Queue Consumers

    To view a list of all consumers, run the following command:

    bin/magento queue:consumers:list

    Start Message Queue Consumers

    To start message queue consumers, run the following command: 

    bin/magento queue:consumers:start [--max-messages=<value>] [--batch-size=<value>] [--single-thread] [--area-code=<value><consumer_name>


    After consuming all available messages, the command terminates. You can run the command again manually or with a cron job. You can also run multiple instances of the magento queue:consumers:start   command to process large message queues. For example, you can append & to the command to run it in the background, return to a prompt, and continue running commands:

    bin/magento queue:consumers:start <consumer_name> &


    Common RabbitMQ Mistakes & How to Avoid Them

    Common Mistake #1: Don't use too many connections or channels 

    Try to keep the connection/channel count low. For example, use separate connections to publish and consume. Ideally, you should have one connection per process; and use one channel per thread in your application.

    How best to reuse connections:

    • One connection for publishing
    • One connection for consuming

    Common Mistake #2: Don't have queues that are too large or too long

    Short queues are fastest; when a queue is empty, and it has consumers ready to receive messages, as soon as a message is received, it goes straight out to the consumer.

    Common Mistake #3: Don't have an unlimited prefetch value


    A typical mistake is to have an unlimited prefetch value, where one client receives all messages. This configuration can lead to the client running out of memory and crashing, and then all messages are redelivered.

    Common Mistake #4: Don't ignore lazy queues

    Use lazy queues to achieve predictable performance or when you have large queues. Lazy queues create a more stable cluster with more predictable performance. Your messages will not, without warning, get flushed to disk.

    Common Mistake #5: Use multiple queues and consumers

    Achieve better throughput on a multi-core system with multiple queues and consumers. You will achieve optimal throughput if you have as many queues as cores on the underlying node(s).

    Next Steps?

    Read more about the Fully Managed Magento Hosting and the benefits for your business.


    Fully Managed Magento Hosting
    Optimized Ecommerce Hosting for Speed, Security and Scale
    PWA Ready, server clusters, instant auto scaling, PCI compliance + premium security. Extended security for M1.

    We also have a variety of Nexcess support articles about Magento 2including how to get your site going with a number of different configuration options. These resources include a great article with answers to common question, Magento 2 Help: Frequently Asked Questions (FAQ)

    24-Hour Assistance

    If you need any assistance with the above-mentioned, don't hesitate to reach out. For 24-hour assistance any day of the year, Nexcess customers can contact our support team by email or through your Client Portal.

    Why Choose Nexcess?

    Because we're different! Chris Lema captures "the why" in his passionate and stirring recount of a Nexcess support-related story.


    Useful YouTube > Nexcess Channel Links

    Resources for More Information

    Need more help? With regards to security topics, The Applications and Magento 2 sections within the Nexcess Knowledge Base are important resources. The Applications section also contains valuable insights for those seeking additional knowledge about our other various hosted applications and platforms. Check out our related video playlists and articles below:

    New Customers: Fully Managed Hosting Solutions

    Not a Nexcess customer yet? Check out our fully managed hosting solutions. The option to chat with an expert is also available.

    Related Articles

    Was this article helpful?

    Send feedback

    Can’t find what you’re looking for?

    Our award-winning customer care team is here for you.

    Contact Support