[jira] [Updated] (SOLR-11221) SolrJmxReporter broken on core reload

Previous Topic Next Topic
classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
Report Content as Inappropriate

[jira] [Updated] (SOLR-11221) SolrJmxReporter broken on core reload

JIRA jira@apache.org

     [ https://issues.apache.org/jira/browse/SOLR-11221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Andrzej Bialecki  updated SOLR-11221:
    Attachment: SOLR-11221.patch

This patch changes {{SolrJmxReporter}} so that it uses a modified version of {{JmxReporter}}, which adds an "_instanceTag" attribute to track what beans it had registered and unregisters only those. In a sense this is the same mechanism that {{JmxMonitoredMap}} used, only it was called {{coreHashCode}} there (we can call it the same if it gives users a sense of familiarity, but I thought the "_instanceTag" name is harder to mistake for a real bean attribute).

The new unit test passes with this change, while it was failing with the old implementation. Manual testing with JConsole also shows that metrics are now correctly reported in local and cloud mode, and after core and collection reloads.

This is unfortunately a bigger change than I hoped for, so I'm not sure whether this should go into 7.0 at such late stage in the release. OTOH JMX monitoring is surely broken without this change, so it can't get much worse ;)

> SolrJmxReporter broken on core reload
> -------------------------------------
>                 Key: SOLR-11221
>                 URL: https://issues.apache.org/jira/browse/SOLR-11221
>             Project: Solr
>          Issue Type: Bug
>      Security Level: Public(Default Security Level. Issues are Public)
>          Components: metrics
>    Affects Versions: 6.6, 7.0, 6.7, master (8.0)
>            Reporter: Andrzej Bialecki
>            Assignee: Andrzej Bialecki
>            Priority: Critical
>         Attachments: SOLR-11221.patch
> {{SolrJmxReporter}} uses Dropwizard's {{JmxReporter}}, which in turn uses a {{MetricRegistry}} listener to report newly added metrics as MBeans, and to unregister them from MBeanServer when the reporter is closed, which happens when core is closed.
> The metrics API keeps around existing metrics in {{solr.core.*}} registries to help maintain continuous metrics in presence of core reloads. However, this means that some of these metric instances are not registered anew when a core is reloaded - and for these metrics the listener won't fire, so the MBeans won't be registered.
> This limitation is a result of the use of {{MetricRegistryListener}} in {{JmxReporter}} and can't be fixed without reimplementing this class. Another possible approach would be to configure the {{JmxReporter}} to use a "mirroring" registry instead, which will be populated with existing metrics from the original registry (thus generating "metric added" events) and then kept in sync with the main registry via a listener.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]